My multi-thread plugin runs into race condition

[miguel-winfield]

Hi    Currently I use winapi fuction to create a thread in the command plugin's doit method. and the thread will stay sleeping background until new job comes . Here are fuctions I use:

HANDLE WINAPI CreateThread(...);   //create thread
in thread pro I use:
SleepEx(..);   //make it sleep until QueueUserAPC is happen

The job is sent by a MAnimMessage callback in mainthread , which will use this function to add job into  queue :
QueueUserAPC (funtion,threadHandle,ULONG_PTR).

Here comes the problem. I dont want maya's main thread to be halt or sleep, but the callback in mainthread will update a global MObjectArray variable occasionally, and the function in the second thread need to iterate through that MObjectArray varibale, and do some jobs in background. Once these jobs are not done quickly enough, the varibale may be renewed in main thread,  which cause the socalled "Race condition" in maya's manual.  

I read some workround such as mutex, but it will stop one thread while the other thread is working. but I dont want to halt the main thread.  As long as I can make a copy of that MObjectArray and sent them by QueueUserAPC , problem may be solved. But unlucky QueueUserAPC  only accept ULONG_PTR as a argument,which is a pointer to unsigned long.

I haven't  tried maya 2008's new threadPool and other thread method, cos I want he plugin  to be used in maya 7.0-2012.

Any suggestion,guys?

[joojaa]

Engineers call situations like this race conditions, yes. The strategy you describe inherently does lead to race condition no question about it. QueueUserAPC would do exactly the same thing as the mutex locking, or have the exact same race condition problem, which you describe you dont want.

To be honest mutex locking is slightly better than you make it sound, since it frees the computer to do other things. But again you have no control over such things.

Theres no way around for this unfortunately IF you really design thisway. See the MObject must ensure you access while nothing else has access. And only running in same thread, or locking the other thread, ensures this. In fact theres little you can do about this in older maya implementations. So whetever code you have now needs a LOT of change to become therad safe, even possibly 99% rewirite.

On a second note you should NEVER make a persistent list of MObjects, you dont own the object. See whet might happen is that maya deletes whatever the MObject points to while your hibernating, unless you can 100% guarantee that your list is always up to date. Which you dont want to do, because it would slow down you EVEN MORE, if you dont do it tough you will cause maya to crash, go unstable or something on that line. Fighting instability by introducing more instability is clearly not the way to go. instead use a list of MObjectHandles as they provide you with the functionality you need for persistent storage.

only accept ULONG_PTR as a argument,which is a pointer to unsigned long.

ULONG_PTR is way of saying a pointer to anything arbitrary, cast your handle.

In essence you should not be doing it this way, since you can not just win it. To do this properly you MUST know what the hell takes time and what is not taking up time. So you should profile all your calls. Then redesign your loop so that.

Everything that does not take time is taken in main thread, but also ALL HEAVY MAYA INTERFACING CALLS MUST BE IN THE MAIN THREAD, if you want same code to work across maya generations. And everything that is NOT maya internals and that does take time is put the the second tread. Now then you may need to start thinking fo caching to minimize maya api calls if indeed thats the problem.

However its much more likely that the call to storage of the data is whats costing you the actual bulk of time, in which case you retrieve data you need, possibly just what changed, into your own memory structure, then pass that to a separate thread that does the heavy lifting. (tcpip, com and disk images carry HUGE time penelties per call.)

whishing good luck is in order. My advice would be to think really hard if your solving the real problem, or just the symptom. I mean really, you probably get much better bang for the buck if you take a look at what causes your crash and autosave just before that, also defeering things for minutes or seconds can not be taht big a deal.** Anyway since you say it frequently crashes saving, its a indicator that theres something worng iwth some plugin, possibly hardware.

But yeah if you do viligant caching, profile well, and write well designed code then i dont really see a problem for succeeding. Just not entirely optimal use of time tough. Say it crashes 1 time for a 3 minute discomfort and 1 minute loss, versus 3 minute startup and 0 second loss, yoru looking at a minute gain. For say, a very optimistic, 2 weeks of more coding then you need 2*8*60/2 = 960 crashes to pay your efforts back in the most naive situation. With say 5 crashes a day (ifmore seriously look at the stuff you have much closer) 192 days to bay back. Which is remotely feasible if not for the fact that you really cant expect it to be all this clean. Especially since you can probebly make the 1 minute loss one in a hour.

• I dont actually think you have much choice in all the nifty threading things, since maya 7.0 to my recollection is nowhere nearly thread safe enough for you to succeed, without a lot of hard work. Sicne you cant make internals behave better you probably cant leverage all possibilities either. They only became ready with the major threading rehaoul in maya 2008

** it is after all a tradeof, you can not possibly save everey state of a computer even if you wished to without slowing it to a crawl. See each second the i7 computer maxing capacity out shuffles hundreds of gigabytes of data. Maybe it would be cool, but you know if it was even remotely easy and feasible they would have done it.

PS: programming multithreaded applications IS HARD on so many levels, so be preopared to work for much longer that you think is needed now.

PPS: it might not be animator friendly but if its not engineering friedly it might take years to make it animator friencly.

[joojaa]

If i may i seem to have come up with a pretty good solution that may be easy to implement.

Listen to the change oevents of the curves. and issue ONE and only one idle high event thet handles all the changed curves one at a time witha  idle high one to next curve etc.

Thuisway you will lock maya, but only fractionally and whan user does nothing, and you automaticllly end up witha  solution that caches what has not changed.

[miguel-winfield]

Hey Joojaa
Thank you for such a long reply.  I know the threading problem is not that simply as I thought. But I still need to try to avoid animators lossing their tons of hours of effort Cos I am an animator, too. 

So here is what I do to get around the race conditons:
in MAnimMessage's callback function,I copy the MObjectArray,using

MObjectArray * temp = new MObjectArray();
And I copy the original MObjectArray data to this temp variable .send the temp point through ULONG_PTR into the threadproc.(Just as you reminded me, ULONG_PTR is a pointer to anything arbitrary)

In my second thread , what I does is extracting the keyframe data using MFnAnimCurve. And write the keyframe data out to ascII files.  My thread proc is kind like so:

VOID CALLBACK threadBackUpProc(ULONG_PTR dwParam)
{
void * data=(void*)dwParam;
MObjectArray animCurvesToDealWith = ((MObjectArray)data);
MFnAnimCurve curveFunc; 
...  //iterate through the MObjects and export ascII files.
delete [] data;  //I intend to release the memory I took by using new keyword before.
data=NULL;
SleepEx (INFINITE,true);
}

So what I use in the thread is only MFnAnimCurve. And I clean the memory allocation once it finished the task. 
I wonder whether or not it is thread safe? Am I doing something dangerous? Cos I test the code ,seems works fine in Maya.

Please revise,thank you

[joojaa]

Maybe partially. the mfn operates on the node the node is operated by other parts of maya, so something else may be accessing that node. However it MAY work in mayas 8.5 - 2012, snce they have some therad safety in place that may mean you mutex up stuff while your working with them.

remeber MObjects are just void pointers.

Cos I test the code ,seems works fine in Maya.

Well if it works fine it works fine, do some heavy load testing.

tons of hours of effort

Well yes but consider doing this kind of stuff every 2 minutes, now you only loose 2 minutes of work, and its not having this problem. Dont design stuff form

total crapshit solution -> perfect shiny thing.

Or you get into teh area of diminishing returns. Read on YAGN principle. You may luck out every now and thsn but dont count on it. remeber you also need to deliver stuf each and every time too.

[miguel-winfield]

OK~ I will do some heavy load testing.
But what I will do in background is dealing with animcurves that changed.So may be lucky there won't be too much heavy data to deal with.

thanks  

[joojaa]

Well howabout a scene with 20000 curves all shifted simulatenously?

[miguel-winfield]

Hmm.. that will happen definitely, that is why I use multithread.
I will test that