In my opinion, I'd model the inside of the banana of course then make a "hand" type object that represents the peel of the banana and build an IK chain going up each peel.
Then use the IK handles to pull down each peel.
I figure that would work, haven't tried it myself but it would get you away from having to use deformers.
Just my .0000002 cents,
Scott Luscombe