What do you mean by 'it doesn't look natural" ?
Could it be that you need to track & match the textures to the face imagery prior to doubleing up.
If you're not doing this already it could be that you're noticing errors introduced by 'spline' interpolation between say frames 1.1 1.2, 2.1 2.3, 3.1 3.2 etc.
Another fix may be to ensure the tracking is using 'linear' interpolation....
Please don't blast me if I'm way off, I've never done this in Soft directly, as I have access to Eddie, which I believe sadly isn't available if you're running on a Win or Lin box....but there's always the lower end compositor packages you could try.