A relatively empty maya can easily use 500 megs of memory tough, once youve loaded a few scripts/plugs in memory (thisiswhy you dont want to just ennable all plugs all the time)! Altough some users running maya 6.0 and 6.01 have reported memory leaks, in maya especialy on linux and some on windows.
Altough to be honest loading all maya plugs i have and modelling for a while never brings my maya over 500 megs. But maybe you have unlimitted undo on! Also bulding a poly object without ever flushing old history away can in about 30 minutes of working eat a gigabyte of memory easily.
what build of maya are you running (save as a ascii filer and read the header using wordpad or whatever editor)
Altough i cant verify this as i dont have the problem.
The batchrenderer does indeed have a problem sometimes wit dissallocating memory (this is nothing new, its a standard render problem), or rather its the fact that it tries to recycle things without hope of doing so. so you should allways run it form a commandline to save you memory of launching a full maya, and then run it inside a for loop that restarts the process every 20 frames or so thereby flushing the mem.
It is impossible to foresee such occourences however, so this is on of the major reasons no one ever renders anything out to avis. Its just not maya that has theese kinds of problems but all renderers are prone to this if misconfigured or mistreatted. Infact its rether easy to have most renderers to leak, a bit. With the possible exception of those renderers that launch a new process for every frame (this is actualy smart on the long run).
also try disabling reuse tesselations.