Setting up LipSync asset for a RPG mobile game

The Mission

Animating every dialogue sequence for an indie game can be a very expensive and tedious process, so we decided to approach this problem with another perspective.
​
In this case study I want to talk about how I implemented a systemic lip sync in Ravensword Legacy for Crescent Moon Games in Unity.

Resources

• MayaLT 2018
• Unity 2018.2
• Lip Sync Pro from the Unity Asset Store

---

This post is about a mobile premium game we are working along with Crescent Moon Games, called Ravensword Legacy, which is currently on development.

This week I had to work revamping the awesome characters that were already made by another team member in order to allow them to talk. After some research, I found a plugin for Unity called LipSync Pro that allows you to add some keyframes for Audio Clips, so that the character that is talking moves their mouth accordingly (it allows for some other blendshapes, like blinking or yawning, and even some presetted expressions like angry and happy among others, so you can assign each expression to each line of the character).

---

The core of spoken languages

A phoneme is one of the minimum units of sound that distinguish one word from another in a particular language. For example, in English there are 44 phonemes. ​Similarly to VRChat's system, LipSync uses phonemes to choose between the different mouth shapes that represent a specific sound. This way, we can assign each keyframe from the Audio Clip to one phoneme, and the mouth will adapt.

For this kind of work, game developers usually group the phonemes together, for example the letter "k" in "key" sounds the same as the "c" in "car", hence needing only one phoneme for that sound. Same with "m", "b", and "p" and so on.

This is the simplified list that LipSync asks for us to work its magic. You don't need to fill all of them at all, in fact we're just using 3 blendshapes (A/I, E, O)+ the resting one.

---

Adapting to the new needs

I proceeded to modify the models and open their mouths, add the inside of the mouth (commonly called mouthbag), tongue and teeth. I also had to modify the textures so the teeth, tongue and mouthbag were textured.

After this, I duplicated and modified three times the resting pose for the A, E/I and O phonemes. As the game is low poly and has pixel post processing and a limited colour palette (sometimes even as low as 8 bits!), too much fidelity and/or fluidity would make it look uncanny. 

Each of these heads were exported as a single head with 4 blendshapes, using the modified mouth's ones as targets for the said blendshapes.

Then I repeated this process for a couple of NPCs and the other 8 head variations of our hero. Once that was done, I headed towards Unity and imported the new heads replacing the old ones. I also imported one character line from one of my favourite videogames for testing purposes.

---

Setup of the system

I created a LipSync info .asset file from that Audio Clip via LipSync's Clip Editor (shortcut Ctrl + Alt + A) and started adding the phonemes that matched with what the line was saying. Having only 3 phonemes really sped up this process, otherwise it'd have been too tedious. After that was done, I saved the LipSync info .asset file in the same folder as my Audio Clip.

Each of these black markers means that the mouth will change to the specified phoneme at the specified time. Once this was done, I went back to the prefab of the character head, added the LipSync script and assigned the head mesh as the main mesh, and the teeth as the secondary mesh. This means that the head blendshapes will drive the teeth ones too. I also assigned the Audio Output of this character to be the origin of the sound of the line, and dropped it into the slot.

I then specified which blendshapes were to be assigned to which phonemes so that LipSync knew what blendshape it had to change everytime the timeslider passed through a phoneme marker. 

I then modified the rest of the settings a little bit more towards what I was looking for, rapid opening and closing of the mouth after each phoneme, and Loop while I setup everything up and changed the settings.

---

Conclusion

And so this is the end result! It was a very fun experiment and I'll probably end up using this method again in the future for personal projects. 

Please be aware the audio clip was a test one to make sure the plugin worked and it's not intended to be used in the final product, since it's a dubbed line from another game.

If this was helpful to you in any way please consider sharing it with your gamedev friends, we really appreacite your support!

Post by:

Alex @Vontadeh​

Alejandro Bielsa is a junior 3D artist working at Polygonal Mind's  in-house team.
Passionate about videogames, vivid tutorial drinker and cat lover.
​​