Sometimes roto gets all messed up if you have a comp containing clips with mixed framerates. So if you have 24 frame per second comp and you also have some 30 fps elements, bad things happen in roto, even if these aren't the layers your trying to apply a mask to. They just have to be somewhere in your comp.
maybe look into that?
jon a