try to contrast a bit your plate before and try 2 point shake tracker which i found strangely better than than the direct stabilise trakers, you copy them to your stabilise ( don't forget there is a rotation and scale parameter that you can enable/disable in add of your just pan stabilise).
it generaly work not to bad.
if your shot is very grainy you could have to degrain it before, that can be the problem.
Boujou or an other track soft generally work, if not, find the disturbing problem of your shot and do a matte to get rid of in boujou or whatever
good luck