The first frame when the sim starts 'for real' always takes longer than the others. Maybe you've got a combination of start frame and relax length that means your cloth starts properly draping on frame 7?
You might also want to look for any weird interpenetration, if the cloth is getting itself tangled, that can take longer to calculate than other frames.
-matt