The Custom Hooks tab allows you to specify hooks for functions that Bug Validator does not know about. This allows you to monitor APIs in 3rd party products and in APIs that are released after Bug Validator was released.
This a very advanced topic, requiring you to know the parameter list and return type of each function to be hooked and the calling convention for each function. Failure to specify the information correctly may crash your application. If in doubt do not attempt to use this feature of Bug Validator. This topic is also discussed in the Bug Validator tutorial, which can be found on the Help menu.
The picture above shows some custom hook definitions in the testCustomDLL.dll DLL. Some of the definitions are for functions with __cdecl calling conventions and some are for functions with the __stdcall calling convention. To edit an existing definition, double click the entry you want to edit, or click the Edit... button.
•Add add a new custom hook. The custom hook dialog is displayed.
•Edit... edit the selected custom hook. The custom hook dialog is displayed.
•Remove delete the custom hook that is selected on the display. Alternatively you can press the Delete key on the keyboard to delete the currently selected entry.
•Remove All delete all custom hook definitions.
•Enable All enable all custom hook definitions.
•Disable All disable all custom hook definitions.
Typically, after setting the custom hook definitions, you will not want these reset when you choose to reset the other settings to the defaults by pressing the Reset button. However if you do wish to reset the custom hooks at this point you can select the Allow the Reset button to reset the custom hook definitions check box.
The custom hook dialog allows you to edit the definition of a custom hook.
A custom hook dialog for a function using the __cdecl calling convention is shown below. The function prototype for this function is:
extern "C" void *customAlloc1(int size); // input param
The function has been specified using the extern "C" specifier so that the function name has no C++ name mangling decoration. The function is the called customAlloc1 and is in the DLL testCustomDLL.dll, it uses the __cdecl calling convention, takes one input parameter. The custom hook is marked as an allocator by the Alloc definition. For datatype purposes it has been specified as returning the void *datatype. The custom hook will monitor the return parameter, marked as a pointer and the single input parameter, marked as a size specifier.
Type the name of the DLL without a file path in the DLL Name field. The name is case insensitive.
Type the name of the function in the Function Name field. The name should appear exactly as shown in the Export Address Table of the DLL (or the Import Address Table of any DLL that uses the function).
If the function is imported by ordinal (for example MAPIAllocateBuffer@8 in MAPI32.DLL), type the ordinal (as a decimal integer) in the Function Ordinal field. If the function is not imported by ordinal, type -1.
For the example MAPIAllocateBuffer@8 in MAPI32.DLL, the ordinal is 13. The ordinal can be found in the Export Address Table of the DLL.
Select the radio box that corresponds to the calling convention of the function. If you don't know the calling convention, DO NOT GUESS. Using the wrong calling convention will crash your application.
Number of Parameters
Type the number of parameters the function takes. This is the number of parameters the function has, not the number of parameters that you want to monitor. If you don't know the number of function parameters, DO NOT GUESS. Using the wrong number of function parameters will crash your application when used with the stdcall calling convention.
Select the check box to make the custom hook enabled. Deselect the check box to disable the custom hook. This setting only takes effect each time your application starts - changing this in the middle of a session has no effect.
The parameters section allows you to define the return value to be monitored.
The parameters section allows you to define the parameters to be monitored.
Click Add to add a parameter definition.
Click Remove to remove the selected parameter.
Click Remove All to remove all parameters.
The return types and each parameter definition can be edited by clicking on the field in the grid. A combo box will be displayed with appropriate choices. The various columns are:
This column is the name of the parameter.
This column indicates what datatype the parameter is. The following values are valid.
Reset All - Resets all global settings, not just those on the current page.
Reset - Resets the settings on the current page.