I never encountered this problem. When trying to make a group of two simple cubes, maya 5.0 refuses to do so. It also doesn't give any error or whatsoever. I tried al different combination in the optionbox and I am in the hierarchy mode. Normally when you group some object they al get green but that also doesn't happen. Actually nothing happens when trying to group.
Umm... only thing i can think of it is that someone has tinkered with the mel procedures the menus call try doing the gorup by the builtin mel command to se if that works?
Maybe youve sourced someyhing that overrides tha procedure.