A procedure which gets called in order to notify an event.
Take Paint Effects for example. When you turn on Tubes for a stroke, you get the option to specify "User MEL Scripts". One for creation and one for runtime. The words you enter there are the names of procedures which you have implemented, for example "myCreation" and "myRuntime". Then, when Maya renders the strokes, for each tube it calls myCreation when it gets created, and myRuntime afterwards.
Both these functions are callback functions because Maya calls back to you, where you are represented by the two functions you implemented and specified as the callback function.