Drive Anim W/audio?

[pixl]

To get a "cheap" lipsynch, it possible to drive the animation of a jawbone or blendshape based on the instensity of an audio recording?
Thanx.

[joojaa]

Sure why not but that doesnt really reflect whats happening. So its not only off its way off. Actually typing in the letters onot a timeline is allmost as fast and way better-

Please note sound intensity is a bit of a hard thing to do since it interleaves many intensities the same time.

[user_deleted]

Yea you will get absolute garbage. It doesn't work at all.

If you need to try for yourself there is a mel script that does what you want, called audioanimate.mel or something.

Or there was a program called MagPie Pro some time ago that had a pseudo automated lipsync system.

But even then it's outside maya so it's a pain. And the results were still mostly garbage.

Look. With lypsinc there is only one great truth: make yourself some nice Osipa style controls, and do it by hand. It's faster and better. It will take you a couple days to get it set up, and after that it is honestly way faster than you would think. Dare I say it, it's even kind of fun.

Even with tons of shots and looong screen time - lipsync, if set up properly, becomes the least of your problems.

Don't try to beat the system on this one...

What you should really worry about is head rotations! Ha.

[mzartler]

Yeah, intensity based stuff isn't so good. Even good intensity measurements create bad jaw flappers. "m" is pretty loud on the intensity scale, but the mouth is closed.

In full disclosure, I work for an automatic lipsync company, annosoft. I'm a programmer and actively working on better integration with Maya - I learn a lot from everyone on these message boards. Don't get much opportunity to post anything useful.

Automatic Lipsync technology still can't compete with conscientious manual work, but it is a little bit closer. It has really
improved since those first generation programs.

our business is primarily in video games, and typically hours of audio and/or multiple languages. There are some good quality options when faced with budget killer problems. Other companies, such as Oc3 entertainment, also serve this market.

For CG in movies, we aren't doing nearly as well. A little splash here and there, but nothing consistent. Not that i understand too much here, but I think it's part perception (been there, tried that) and part reality (By hand is still better)

A reasonable middle ground is something that gives a solid first-pass using an automatic lipsync technique, but provides the data in such a way that it is easily editable by the artist. The automatic pass needs to be good, close, but then the tools need to be available to make it very easy to spot fix. I'm not saying I'm is completely on top of this, just saying this is what a good tool would need to be.

thoughts?

Regards,
Mark Zartler
Annosoft

[user_deleted]

Hey Mark,

Well honestly I always wonder why not just drive the animation with a typed sentence text input. I mean then it's just a matter of slipping the keyframes to match the timing of the audio track, and that's not too rough. I would think with text, you could do a true deduction of what needs to happen to the mouth, no?

That's minimum user input and effort, too.

Magpie pro used to have both, sound + text. But I don't know that the sound half of it did any good.

[pixl]

Thanks for the replies.
I've used both audio-file-intensity driven and Osipa style approach to lipsynch in the past, but not w/Maya.
When I started this thread, I was bidding on a job which didn't have a budget for lipsynch, so I was looking for a way to create something that would acceptable, not accurate.
I didn't get the job, but as I was researching Maya-based lipsynch, I came across this, which looks to have a lot of promise:
http://www.voice-o-matic.com/Demo/Maya/

[mzartler]

Hi holographix,
Having the text makes a huge difference in for "automatic" lipsync for the reason you state. It provides you more information. It's still a technical challenge figuring out where in the audio file the changes are, the results are much better because the phonemes come from the audio. More information makes it easier for software to solve.

Good idea about starting with the text and then sliding them around. That makes a lot of sense since you dont have to continously scrub. Eventually, the scrubbing still has to get done, but it's at different, probably more productive, step.

It's actually a lot like what our "Text based Lipsync" software does. convert text to phonemes, and then line them up with the audio. Good, but as before, it's still not the same as "by hand". Words can be enunciated very differently, different emphasis, or even sound, those nuances are not easy to capture.

Regards,
Mark
Annosoft

[mzartler]

Hey Pixl,
Glad the problem worked itself out.

Di-o-matic are great guys. They use an old version of our speech stuff, but offer a good interface and plugin. We have a melscript which converts from our tool, it takes more effort than a plugin but may be more flexible. They are artists, I'm a speech nerd, it does show...

That said, I am trying to get a handle on the art side, it's not been an easy ride, learn a LOT from these boards...

Cheers,
Mark Zartler
Annosoft

PS: I am looking for beta testers for the new maya Lipscript. Anyone with lipsync work to do? It should work fine, but I could sure use a few expert eyeballs to get rid of the squeamish feeling.

[pixl]

Hi Mark:
The "problem" didn't exactly work itself out favorably...

I didn't discover your lipsync tool when I was looking for a solution for the job I was bidding on. It looks interesting. A bit more information on how it works would be helpful. For example, can we import our meshes and blendshapes?

Several years ago I used a lipsynch tool for Lightwave called TAFA http://www.macreitercreations.com/community/), which was very well designed and effective. Sadly, it has not flourished.
I wrote an article about using TAFA to create Osipa style facial animation, its on the web here:
http://www.hdri3d.com/xtra/07rosefelt/index.htm

The idea of placing text under the waveform is interesting, but from my limited experience, the most useful capability is to be able to drive your deformations over the audio in realtime. I'm a bit busy at the moment, but maybe at some point I could try out your software.
What I like about the Di-o-matic approach (which it appears that you also offer) is that it allows the quick creation of graph curves which can then be tweaked to add the "art". For a lower budget job, it shouldn't take much to get acceptable results. For a higher end project, I would expect to finish it, perhaps meticulously, by hand, but again a good tool, which TAFA was, should make it easy to get a great first pass.

my 2c

[mzartler]

Hey Pixl,

A very well written article, kudos on that.

The Lipsync Tool supports mesh import, .obj and .x files. I'd like to be able to import blend shapes, but am not sure which file format to support. Generically, it's a daunting task figuring out how to integrate with various 3d programs, and worse wet with various rigs... supporting bones, for instance. FBX seems like it might be the answer, but I donno.

TAFA is sweet! I'm salivating over that level of functionality. Lipsync Tool is much more utilitarian. Our lipsync data fits nicely into blend shapes. The 'openness' though is implied more than explicit. The visemes are modelled at "full open", the speech system decides how much to contribute. Although still editable by shifting up the curve on the primary viseme, having a separate "open-closed" viseme/track makes a lot of sense. Anything that can reduce the viseme count increases editability. I need to add that functionality soon. It clarifies some compatibility issues i've seen with downloadable rigs.

Thanks for following up with that information.

Regards,
Mark

[pixl]

TAFA is withering on the vine. The odds of another version being released are extremely low. Look at the dates and frequency of activity on the message boards. The author, Mac Reiter created it as a side project for a prominent LW animator and author (Timothy Albee). I'm sure there are a varity of reasons it didn't sell enough copies to maintain itself - the small installed base of LW & the low demand for lipsynched characters being among them.

Mac (the TAFA coder) also found it difficult to figure out how to support "blendshapes" in other 3D apps. I think bones would be very difficult to support natively (within an app like yours), though a workaround procedure could be developed where the bone "poses" would have to be exported as blend shapes and then animation curves would be returned to animated between the poses. This approach could also be used for a facial animation system that uses wire deformers. I'm pretty new to Maya, so I'm not sure how that is done, but I assume that is what Di-o-matic (and perhaps you) are doing if the face is animated w/bones.

The beauty and usefulness of using an external app for lipsynch is that it removes the overhead of the native app, so it should be realtime. The fact that the output of the lipsynch program is independent of the actual models/blendshapes/bone poses is an advantage, because all those can then be tweaked without losing the time put into the facial animation (though theoretically, this could also be done within an app). One brilliant aspect of TAFA, is that, like Jason Osipa's approach, it doesn't separate lipsynch from facial animation, which is really the correct way to think about it... we don't lipsych, we talk, and our faces reflect our personality and what we are thinking...

TAFA is a brilliant bit of coding, from an artist's perspective, its sad to watch it go extinct - maybe you should contact Mac and see if you could work out a deal to acquire or license his technology...

[mzartler]

Hey Pixl,
Great idea. It seems worth pursuing. Right now, my magic application be arranged around FBX, import FBX and then save it back to FBX. Theoretically, this would support more 3d packages. In practice, who knows...I may start another thread asking for "nightmare" situations with FBX.

It seems to be the big problem with specialized tools - finding smart integration paths. TAFA found it with Lightwave.

Our Lipsync SDK would fit perfectly into TAFA. Just add a button and a phoneme-to-viseme assignment window. Our data is a perfect fit for n-way blending. Pretty cool.

Thanks for all your ideas. Big help.

Regards,

Mark Zartler
Annosoft

[pixl]

... just remember me when you go public...
Really, it would be fantastic to be able to use something like TAFA w/Maya. Keep us posted.
Also, I might be able to dig up some old email w/recommendations for features...

[joojaa]

I think the geratest problem with automatic lipsych is in the title.

Automatic

If automatic WOULD work out then we would already all be using automatic. Now offcourse we use a some kind half automatic and discard most of it anyway. Reason for this is the control and noise, a capture animation is noisy so much taht integrating it with non capture is pure pain.

This especially for lipsynch, because this is the most important part of our animation. It conveys most of our messages. So for me it seems brain dead to let the computer do the job tahts MOST important in your production. Now while it would help to get it right ist not there yet.

As for text yes i agree text itself is no solution as pheromones vary from written text (not much in Finnish tough), especially in English. But having phoneme type is actually worthwhile (yes there's rules for this).

So at the end of the day i would choose to do lipsynch manually and automate something else, for the same time benefit. I mean who cares if you save time form wherever as long as what you saves makes it worth while. And i think lipsych is the WORNG holy grail here.