Note: I'm using 4.5
After a bit more investigation I found out that the problem is connected to the skeleton I'm using. It's a pretty basic setup with IK SC for the legs, IK RP for the arms and set driven for the fingers. I just exported my unbound skeleton and bound a simple nurbs sphere to it. Created a cloth collission object and the moment I created a garment Maya crashed on me again.
This was without PV constraints.
One thing that could cause it are the local rotation axes, I've been messing around with them a lot so that may be the cause. I'll rebuild the skeleton today using the old one as a template and see where that leads.