Nope, it should work like you expect. The transform constraints aren't set as soft are they? If so, you'll need to set the stiffness really really high for it to not lag (I was using values like 1000, 2000).
If they're hard constraints, check you're only turning on the constraints once the cloth is in place. Cloth transform constraints will maintain their offset from the target as soon as they're turned on, so if the cloth has to settle first, key the transform constraints to 'on' maybe 5 frames after the settling is done.
(edit) One other thing, see if you can construct your setup using panels rather than a poly object converted to cloth. Cloth gets unstable unless you use that irregular tesselation cloth likes. (/edit)
-matt