Detecting Broken Message Maps

• Visual Studio .Net 2002 or later.
  If you are building applications with Visual Studio .Net 2002 or later this tutorial does not apply to you – your compiler will not build broken message maps.
• Visual Studio 6 or earlier.
  If you are building applications with Visual Studio 6 or earlier this tutorial does apply to you – your compiler will build broken message maps if you supply incorrect function prototypes to it.

   

One of the problems with MFC programming that often causes trouble for programmers is the broken message map. The symptoms of a broken message map are the application works perfectly in Debug mode, but in release mode the application crashes with a corrupt stack at an apparently random point after choosing a menu option or interacting with a user interface control.

Programmers often look elsewhere for the problem becuase of the corrupt stack and the release mode nature of the bug. You can spend a lot of time looking for uninitialised variables when the real problem is an incorrectly coded message map.

The problem occurs when a function referred to in a message map has the incorrect number of parameters compared to the number of parameters expected by the message map handling macro. For example, a menu item selection would normally expect a function with no parameters such as:

	void OnColourRed();

and a context menu handler may expect a function with the following prototype:

	void OnContextMenu(CWnd	*wnd, CPoint point);

If the function specified in the message map has the wrong number of parameters (too many or not enough) the stack will not have the expected alignment. For various reasons, in debug code, this error gets fixed up. However in release mode the error does not get fixed. When the function gets called, assuming the function doesn’t crash because of the incorrect parameters, when the function returns, the wrong amount of data is popped from the stack, prior to calling the return instruction. When the return instruction executes, it pulls the return address from the stack at the wrong address and returns to the random address pulled off the stack. The application usually crashes at this point, with a corrupt stack.

Options

Memory Validator has settings that automatically detect when a function with the wrong signature (parameter description) is called in a message map. This functionality works in both debug and release mode. To enable the message map hooks, select Check MFC message maps for correct parameter passing on the MFC Message Map Checks tab.

Example

• Start the nativeExample.exe application.
• On the Memory Errors menu, choose Message Map Error.
• The application will beep (simulating work), then Memory Validator will identify that an broken message map function has been called. A dialog will be shown that explains the error, displays as much of a callstack as possible. You a given the opportunity to end the application, drop into the debugger or ignore the error. This dialog is shown below: