Yeah theres some init that maya does that don't get done in the script unless you call this once trough the poly selection constraining dialog*. Very hard thing to catch because it can work fine when developing and not when not. So if you open the poly selection constraint dialog run the code then it runs fine and does what you do. Need to investigate.
Ok i know what the problem is still trying to figure out how to reliably flush mayas state so it understands in the script im in face mode (maya updates this after my script is run, but i can't eval deferred or your code would not work)
(thats why the select is there it flushes my old version of maya but not say 2011)
how can I query the object to see if it has holes
you really can not, not in mel anyway. Without doing a lot more work then you would realistically want to do and thus slowing maya to a crawl. The option is to work with poly constraints or get the error. Lest you want to loop each edge split text and do about 6 crossproducts. On the otherhand in the api its 4 lines of code, just reflects the difference of what the tools are meant for.
Anyway you can do it with the constraints too just need to find the hack form my notes somewhere.
*polySelectConstraint -w 1; == select thngs next to open edges
