That's a little more useful, thank you.
I'm not entirely sure what "grainy edges" means to you in this instance. Is it aliased? Is there a lot of value noise?
Little tutorial: depthmap shadows are made by rendering a depth pass from the view of the light. For a spot light the cone angle and the light linking of the light determines what objects are included in that pass.
Depending on the cone angle of your light you'll need to adjust the resolution of that depthmap so that there is enough resolution in the shadow to supply a decently smooth shadow. The filter size determines how much the depthmap file is filtered before it's applied as a shadow.
It's entirely possible that the cone angle of your light is too high for the depthmap resolution to cover all the objects that need shadows.
The very first thing I would try is raising the resolution of the depthmap. Then adjusting the filter size (although this will affect the look of the shadows which may not fit with the look you're going for) Then i'd go into the render settings and mess with the sampling values and the filtering values there. Do these steps one at a time and render inbetween so you can isolate which setting is affecting what aspect of the look.
I must admit that I'm not familiar with all the specifics of how Mental Ray handles lighting. Your problem may have something to do with that.
Hope this helps.
-James