had the same question some time ago .. here is an answer from Lynn Miller, A|W...
snipp-
For the Outliner:
Create a 20x20 .xpm file (8-bit, transparent background.)
Name the file out_.xpm where is the exact name
of your node.
Place this file along your icon path. (Creating a 'icons' directory
under your maya/4.0/prefs/ directory will work.)
Start Maya. The icon will get automatically picked up for your node
in the Outliner.
The Hypergraph is harder. You need to create a .fti file. The naming
convention for the hypergraph is .fti to have the icon used
automatically.
-- snapp ---
there is a tool to create fti icons under unix, if you are on nt, open some of the fti icons in maya's icons dir. they are in ascii format and you'll see that you've got to provide 2d-vertex informations to build the polys for your vector icon. you can do it by hand or write a mel to convert maya polys to fti polys.
cheers, carsten