Ok found the issue, posting FYI.
I had baked my simulation in stages (simulate the "big" stuff, bake into passive Rigidbodies, then simulate the small stuff)
and under my "big" RB's I had parented new shapes and turned those into passive RBs.
played fine, but baking was freezing.