Polygonal Mind
  • Home
  • Metaverse Builder
  • CryptoAvatars
  • MegaCube
  • Blog
  • Decentraland
  • Projects
  • Assets

Blog

Setting up LipSync asset for a RPG mobile game

2/14/2019

0 Comments

 
4 heads with different mouth positions

The Mission

Animating every dialogue sequence for an indie game can be a very expensive and tedious process, so we decided to approach this problem with another perspective.
​
In this case study I want to talk about how I implemented a systemic lip sync in Ravensword Legacy for Crescent Moon Games in Unity.

Resources

  • MayaLT 2018
  • Unity 2018.2
  • Lip Sync Pro from the Unity Asset Store

This post is about a mobile premium game we are working along with Crescent Moon Games, called Ravensword Legacy, which is currently on development.
This week I had to work revamping the awesome characters that were already made by another team member in order to allow them to talk. After some research, I found a plugin for Unity called LipSync Pro that allows you to add some keyframes for Audio Clips, so that the character that is talking moves their mouth accordingly (it allows for some other blendshapes, like blinking or yawning, and even some presetted expressions like angry and happy among others, so you can assign each expression to each line of the character).

The core of spoken languages

Picture
A phoneme is one of the minimum units of sound that distinguish one word from another in a particular language. For example, in English there are 44 phonemes. ​Similarly to VRChat's system, LipSync uses phonemes to choose between the different mouth shapes that represent a specific sound. This way, we can assign each keyframe from the Audio Clip to one phoneme, and the mouth will adapt.
For this kind of work, game developers usually group the phonemes together, for example the letter "k" in "key" sounds the same as the "c" in "car", hence needing only one phoneme for that sound. Same with "m", "b", and "p" and so on.
Picture
This is the simplified list that LipSync asks for us to work its magic. You don't need to fill all of them at all, in fact we're just using 3 blendshapes (A/I, E, O)+ the resting one.

Adapting to the new needs

I proceeded to modify the models and open their mouths, add the inside of the mouth (commonly called mouthbag), tongue and teeth. I also had to modify the textures so the teeth, tongue and mouthbag were textured.
Head model with tongue and tooths
After this, I duplicated and modified three times the resting pose for the A, E/I and O phonemes. As the game is low poly and has pixel post processing and a limited colour palette (sometimes even as low as 8 bits!), too much fidelity and/or fluidity would make it look uncanny. 
4 heads with different mouth positions
Each of these heads were exported as a single head with 4 blendshapes, using the modified mouth's ones as targets for the said blendshapes.
Hero 3d model complete assembly
Then I repeated this process for a couple of NPCs and the other 8 head variations of our hero. Once that was done, I headed towards Unity and imported the new heads replacing the old ones. I also imported one character line from one of my favourite videogames for testing purposes.

Setup of the system

I created a LipSync info .asset file from that Audio Clip via LipSync's Clip Editor (shortcut Ctrl + Alt + A) and started adding the phonemes that matched with what the line was saying. Having only 3 phonemes really sped up this process, otherwise it'd have been too tedious. After that was done, I saved the LipSync info .asset file in the same folder as my Audio Clip.
LipSync program
Each of these black markers means that the mouth will change to the specified phoneme at the specified time. Once this was done, I went back to the prefab of the character head, added the LipSync script and assigned the head mesh as the main mesh, and the teeth as the secondary mesh. This means that the head blendshapes will drive the teeth ones too. I also assigned the Audio Output of this character to be the origin of the sound of the line, and dropped it into the slot.
Picture
Picture
I then specified which blendshapes were to be assigned to which phonemes so that LipSync knew what blendshape it had to change everytime the timeslider passed through a phoneme marker. 
Picture
I then modified the rest of the settings a little bit more towards what I was looking for, rapid opening and closing of the mouth after each phoneme, and Loop while I setup everything up and changed the settings.

Conclusion

And so this is the end result! It was a very fun experiment and I'll probably end up using this method again in the future for personal projects. 
Please be aware the audio clip was a test one to make sure the plugin worked and it's not intended to be used in the final product, since it's a dubbed line from another game.
If this was helpful to you in any way please consider sharing it with your gamedev friends, we really appreacite your support!

Post by:

Alex @Vontadeh​

Alejandro Bielsa is a junior 3D artist working at Polygonal Mind's  in-house team.
Passionate about videogames, vivid tutorial drinker and cat lover.
​​
0 Comments

    Categories

    All
    Blender
    CryptoAvatars
    Decentraland
    Decentraland En Español
    Maya
    Metaverse
    Mixamo
    Morphite
    Substance Painter
    The Sandbox
    Totally Reliable Delivery Service
    Unity 3D
    Updates
    Vrchat

    Archives

    March 2022
    July 2021
    June 2021
    May 2021
    April 2021
    March 2021
    February 2021
    January 2021
    December 2020
    October 2020
    August 2020
    July 2020
    June 2020
    May 2020
    April 2020
    March 2020
    February 2020
    December 2019
    October 2019
    September 2019
    August 2019
    June 2019
    May 2019
    February 2019
    January 2019
    December 2018
    November 2018
    October 2018
    September 2016

    Picture
Home
Projects
Assets

Picture
Crypto
Friendly
Picture

Subscribe to get some 💚 in your inbox once in a while.

Follow us and your visit will never be forgotten!
Picture
Picture
Picture

 © 2015-2022 POLYGONAL MIND LTD. ALL RIGHTS RESERVED.
  • Home
  • Metaverse Builder
  • CryptoAvatars
  • MegaCube
  • Blog
  • Decentraland
  • Projects
  • Assets