Genies
Senior Technical Artist - Genies - 2021-2026
Overview
I started at Genies in 2021 as the second person in an Avatar Tech R&D department of two. We grew into a small, scrappy team ready to take on any challenge. My years at Genies were deeply academic; constantly pushing my limits and learning new things.
Much of my work over the last three years grew from a simple idea: can we take facial animation channels and retarget them as needed? Can brow and mouth animations drive standardized expression channels on animal ears? Can we retarget eye channels to multiple eyes?
This question launched my work at Genies in two directions: generative character rigging, and generative animation via LLM tagging. On the rigging side, I went from defining our face-rigging pipelines to building an AI-driven facial auto-rigger, and eventually owned synthetic character data generation, powering both the auto-rigger and generative character mesh creation. On the animation side, I went from evangelizing the utility of simple, category-based animation channels to building Unity animation systems designed to catch LLM tags and drive procedural animation.
Generative Characters
Face Rig Pipeline
After building a system in Unity that let us parse out facial animation channels as described above, I launched a campaign to expand the team's capacity to rig non-standard faces for ARKit. I adopted Face-It, mastered it, then trained our artists to use it — greatly improving the pace of "Doll" character production (short, cartoony-themed characters).
AI Facial Auto-Rigger
In 2023, generative AI was becoming more than a buzzword. As our ML team expanded, I began building an AI-driven facial auto-rigger around a simple prompt: given OpenCV-style landmarks projected onto a 3D character face, could that face be procedurally rigged to 80–90% fidelity? Fidelity here didn't mean speeding up an artist's work — it meant production-ready blendshape rigs that performed perfectly with zero white-glove tweaks.
The theory was simple: wrap an existing ARKit face topology to the novel head using the landmark data, scale the deltas to fit the new proportions, transfer the blendshapes, done. In practice, that wrap-and-transfer approach only got us to about 60% fidelity — most facial features needed real development to get right, largely through masking systems that prevented feature bleed by correctly splitting and isolating the eyes, lids, lips, brows, teeth, tongue, eyelashes, and any other separate mesh on the face.
Splitting upper and lower lips made opening a mouth fairly easy. Closing an eye convincingly, with no visible seam, was a much harder problem. After several iterations, I landed on a process that averaged landmark positions into a center line, curved that line to match natural eye proportions, skinned joints along the lids using those landmarks, rotated the lids together at the seam, and used the eyeball mesh itself to gently repel the lids and prevent interpenetration.
Landmark Dataset
An OpenCV-style facial landmark model drove the auto-rigger, with its points serving as the foundation for all downstream rigging. I worked closely with the head of ML to identify where that model needed to improve, and with no staff devoted to synthetic data, I built a local image-generation pipeline myself — bulk re-imagining a pool of head renders and turning my own feature requests directly into training data.
The clearest example came as our target cohort expanded toward cartoony faces with larger eyes, which demanded new eye landmarks. I designed a solution that split the eye corner into two landmarks: for humanoid faces, the two points simply overlapped, preserving backward compatibility, while for cartoony eyes they separated to give better surface coverage and higher-fidelity blink shapes. As a key benefit, the gap between those two landmarks became a simple, AI-determined boolean the auto-rigger could use to detect whether a given eye was realistic or cartoony. I then built the datasets to train the model on this new feature.
Synthetic Character Generation
From here, I owned synthetic character asset generation and led the Tech Art department's efforts to supply datasets for the ML team. I refactored and expanded our modular hair system, built a Houdini pipeline for our first wave of face-generation and segmentation datasets, and created a Blender facial variation tool that generated attractive head variants to improve our auto-gen character pipeline.
The variation tool was the standout piece. It took our base Genie head and, through a mix of blendshapes and facial deformation joints, generated variations directly on the timeline, with real-time texture projection and mesh updates so I could scrub and QC each result instantly.
A web GUI drove the system, letting me define variation points, generation targets, and procedural constraints, then save them as reusable profiles to sculpt output toward specific facial cohorts.
Inside the web GUI I added a Value Trace menu that allowed each value to be simulated, with stepwise value readouts. With so many value tweaks happening under the hood, this clear debug menu was essential to tracking down the cause of unexpected results.
A snapshot system let me export or load heads as JSON metadata, which served two purposes: bulk-exporting good and bad examples for an LLM to find correlations and recommend generation guardrails, and exporting standout heads into an "attraction system" that averaged nearby high-performing heads into a weighted delta, nudging new variations toward attractiveness automatically.
Tagging and Buckets
As the project expanded, we began ingesting a large pool of authored head variations. I designed and implemented a batch LLM API prompt to evaluate and tag each face blendshape based on its features. After building a tag framework and running my first tests, the key difficulty I faced was style adherence. The LLM would consistently describe the Genies art style rather than the individual features of the target head. I was able to build a secondary prompt that leveraged face atlas images to anchor the LLM to the Genies art style. This improved tagging greatly.
The purpose of this tagging system was to further improve our ability to bucket facial features within a pool of 600+ heads. The ML team was going feature by feature, expanding capacity and refining quality. For example, the model had difficulty producing thinner heads, and more data fitting that cohort was required. The tagging system allowed us to agentically build buckets of correct facial features with simple written requests. These facial features were then further varied using the synthetic head variation tool.
Generative Animations
Modular Facial Animation Ingestion
The purpose of this project was to reroute facial animation clips onto any face. Endpoint characters could have variable facial features, including multiple eyes, and nonstandard blendshape names. I built a modular ingestion and retargeting layer in Unity: a custom retargeting job that could initialize against any blendshape set on the endpoint face and map the animation onto the matching features. The first use case was swappable facial accessories, such as beards.
Self Contacts
I was tasked with solving self-contact on our customizable characters. They varied in both shape (mesh) and proportion (skeleton), so any animation with self-contact, such as clapping or hands on hips, would break. Unity's humanoid system does not handle self-contact natively. To solve this, I mapped surface contacts to UV positions in Maya, then used those positions to drive UV locators in Unity, which drove the hand IK.
The locators sat on a low-poly proxy of our base character that carried the same customization blendshapes. The proxy stayed aligned with the avatar through any deformation.
The proxy mesh's UV was designed so (U,V) traversal would be intuitive.
Hands attached via IK to that UV contact point. The UV positions, and a weight for the contact, were baked in Maya as custom properties and read into Unity to drive the locators and the hand IK.
I also built a hand-collision setup that gently nudged the hands off their animated position. That kept unexpected self-contact from penetrating the character's body.
LLM Driven Animation
At Genies, I worked with the team on a system that drove a Unity animation state machine from chat. The goal was conversation with an avatar that felt alive, which meant generating animation in real time. Generative motion models could not do that well at the time, so we pivoted to engineering prompts that delivered animation metadata and tags along with its responses, and systems that ingested that data into animation calls. This approach allowed our artists to own animation clip/pose quality while opening up a whole world of ways we could have LLM control animations.
Pose to Pose Gestures
This system had to scale quickly, and keep scaling. With limited animation resources and ambitious goals for expressiveness and customizability, I proposed a pose-to-pose gesture animation system. Instead of calling full animation clips they could call single-frame poses, and transition between them gracefully. I built a custom playable that transitioned from one pose or clip to another. Curve-driven windup, follow-through, and layered additive idles made a single sampled frame read as alive.
I can't show the results. The demos used a character I don't have the rights to publish.
Locomotion
Our gesture system was built as an Animation Graph state machine. This graph allowed us to load in collections of animations from server at runtime and play them, a feature not available in traditional Mecanim state machines. It is however possible to create nested Mecanim state machines as layers inside of an Animation Graph state machine. This allows layers of an Animation Graph to benefit from the rapid prototyping/tweaking capabilities of Mecanim while keeping the fine control and customizability of custom animation graph nodes.
I built a 3-layer masked locomotion system that allowed for smooth blending in of gesture animations, full locomotion, full gesture animation, and combined gesture locomotion. This swapped on the fly based on the gesture status.
