Let's see if I remember this.
Turn off the simulation. Delete the cache. Scale your cloth. Set the U and V scale to the same amount you are scaling the cloth, and also set the solver scale to reflect how much you are scaling your cloth.
I find that it's easier to scale your collision object instead.