Hi Oliver,
Rotation of a point/vector about an arbitrary axis* can be represented with
Quaternion math. Maya supports quaternions in the MQuaternion class.
To do the rotation you want, you can create a quaternion with the function:
MQuaternion &MQuaternion:: setAxisAngle (const MVector &axis, double theta)
Then you can get the equivalent matrix for this rotation with the function:
MMatrix MQuaternion:: asMatrix () const
You can then use this matrix to post-Multiply your point.
There are a few caveats:
First, the vector representing the axis to be rotated around must be a unit
vector. Second, it should pass through the origin.
Account for the first case is easy, just normalize the vector before calling
setAxisAngle.
For the second part, before you multiply by the matrix from asMatrix, you'll
need to first translate the point to be rotated by the amount that your axis is
offset from the origin. Then after multiplying by the asMatrix result, translate
the point back out to the offset.
You can find out more about quaternions in the Maya API Class docs for the
MQuaternion class, probably elsewhere in the Maya docs, and in any decent 3D
graphics textbook.
Hope this helps.
Paul