Please enable JavaScript to view this site.

Memory Validator Help

The Custom Hooks tab allows you to specify hooks for functions that Memory Validator does not initially know about.

 

For example, custom hooks might let you monitor APIs in third party products or in APIs that are released after Memory Validator was released.

 

noteThis a very advanced topic.

 

interactiveRead on, or click on a setting in the picture below to find out more

 

custom-hook-settings

 

 

Using custom hooks

 

Before adding custom hooks, you need to know the parameter list and return type of each function to be hooked and their calling conventionsexternalLink.

 

Add... instructionStep displays the custom hook dialog (below)

 

For other options (edit,remove, and enable), see modifying existing custom hooks further below.

 

seeAlsoThis topic is also discussed in the Memory Validator tutorial, which can be found on the Help menu.

 

 

Custom hook dialog

 

The custom hook dialog allows you to set up or edit the definition of a custom hook, including its parameters.

 

warningnote Take care! Failure to specify the information correctly may crash your application. If in doubt do not attempt to use this feature of Memory Validator.

 

At the end of this topic are some examples of how this dialog might be filled out for a few different functions.

 

interactive

custom-hook-dialog

 

 

Custom hook definition

 

DLL Name instructionStep Browse to navigate to a DLL instructionStep Open to enter the DLL name into the dialog  

 

Alternatively you can type the full path, or a relative path to the DLL into the DLL Name field.
 

Function Name instructionStep choose the name of the exported function from the drop-down list
 
       The list is automatically populated with the exported functions after choosing a DLL name above
 
       custom-hook-function-name

 

The function name must be exported from the DLL you specified in order to appear in the list. If the function is not exported it cannot be hooked.

 

Function Ordinal instructionStep type the ordinal (decimal integer) if the function is imported by ordinal, otherwise enter -1 to ignore it

 

For example MAPIAllocateBuffer@8 in MAPI32.DLL has the ordinal 13 as found in the Export Address Table of the DLL

 

hmtoggle_plus0 Finding the ordinal of an exported function in a DLL One way to find the ordinal of a function exported from a DLL is to use the dumpbin.exe utility provided with Microsoft Visual Studio. e.g. dumpbin filename.dll /EXPORTS

In Visual Studio 9.0, the dumpbin.exe utility is found in VC\bin and is dependent on link.exe in the same directory and mspdb80.dll in the  Common7\IDE directory.

 

Calling convention instructionStep select the option that corresponds to the calling conventionexternalLink of the function

 

calling-conventions

 

Do not guess this, if you don't know! Using the wrong calling convention will crash your application

 

seeAlsoSee also, the history of calling conventionsexternalLink

 

Number of Parameters instructionStep enter the number of parameters the function takes

 
This is the number of parameters the function has, not the number of parameters that you want to monitor (below).

 

Again - do not guess this! Using the wrong number of function parameters will crash your application when used with the stdcall calling convention.

 

Function Purpose instructionStep set the function purpose - Alloc, AllocMore, Realloc, or Free

 

Enabled instructionStep tick to enable hooking of the function

 

Only takes effect each time your application starts, so changing this in the middle of a session has no effect.

 

Datatype instructionStep enter the type of all allocations via this hook, or leave empty and Memory Validator will parse the source code to try and obtain the datatype

 

Do not include parentheses in the datatype. [] and <> are acceptable though.

 

 

Custom hook function parameters

 

The parameters section of the dialog allows you to define which parameters and return value are monitored for the function.

 

noteYou should specify all the other settings in the custom hook definition (see section above) prior to defining the parameters. This will ensure that the right menu choices are made available while setting up the parameters.

 

You don't need to monitor all the input parameters, or the return value  - just monitor what you need.

 

Generally for each function purpose (that you will have defined above), different parameter values are usually required:

 

For example:

 

Alloc        an input size, and an output pointer or return value pointer

AllocMore  an input size and an input pointer

Realloc        an input size, an input pointer, and an output pointer or return value pointer

Free        an input pointer

 

In some cases, an input parameter may also serve as an output.

 

If you want to monitor additional parameters, we recommend that you set their type to Miscellaneous or Pointer to Miscellaneous as appropriate.

 

Add instructionStep adds a row to the parameters table, firstly a return value, then parameters in order 0,1,2...

 

Remove instructionStep removes selected parameter definitions from the list

 

Remove All instructionStep clears the list of parameters
 

 

Defining custom hook function parameters

 

After adding a parameter definition, values can be edited by double clicking on the value in the table. A drop-down list will be displayed with appropriate choices in each column:

 

 

Parameter instructionStep choose the parameter index in the range 0 to N-1 where N is the number of parameters you specified to the function

 

You can only choose an index which is not already used by another parameter

 

You can't change the entry for a "return value"

 

Usage instructionStep set how the parameter is used:

 

custom-hook-function-parameter-usage

 

In is input only and Out is output only, while In/Out serves as both

Return value, this and any output parameters are usually pointers to something.

 

Type instructionStep choose what the parameter type is:

 

custom-hook-function-parameter-types

 

All Pointer to... options are parameters that need dereferencing to read the values.

 

Memory Validator x64 also supports the following values:

 

Size (QWORD)                                a size

Pointer to size (QWORD)                        a pointer to a size

Miscellaneous (QWORD)                        a miscellaneous value

Pointer to Miscellaneous (QWORD)        a pointer to a miscellaneous value

 

noteNote that when specifying parameters as Pointer to ..., it's OK if that parameter is occasionally NULL - this will be identified and the pointer will not be dereferenced. Many Microsoft APIs allow NULL values for optional data, for example.

 

 

Modifying existing custom hooks

 

Once you have some custom hooks set up you can edit, remove or en/disable them in the following ways:

 

Edit... or double click a hook in the list instructionStep opens the custom hook dialog to change the hook attributes

 

The usual controls apply for removing list items:

 

Remove instructionStep removes selected items from the list

 

Remove All instructionStep clears the list
 

Alternatively, press b_del to delete selected items, and b_ctrl + b_a to select all items in the list first.

 

Enable or disable a custom hook by ticking or unticking the yellow boxes or change them all at once

 

Enable All instructionStep ticks all the hooks, enabling them all
 

Disable All instructionStep unticks everything
 

Import... instructionStep Imports a custom hook XML file.
 

Export... instructionStep Exports the custom hooks as an XML file. This can be useful for moving a custom hook definitions from one computer to another without copying the whole settings file.

 

 

Resetting custom hooks

 

Normally, the custom hooks will not be reset when you press the reset button, but you can make that the case if you wish:

 

Allow the Reset button to reset the custom hook definitions instructionStep when ticked, custom hooks will be reset with all other global settings

 

 

Reset All - Resets most global settings including those on other pages, but not the settings on this page unless explicitly requested above.

 

 

Reset - Resets the settings on the current page.

 


 

These examples show how to use the custom hooks dialog

 

Custom hook dialog - example 1

 

A custom hook dialog for a function using the __cdecl calling convention is shown below.

 

The function prototype for customAlloc1 in the DLL testCustomDLL.dll (provided) is:

 

   extern "C" void *customAlloc1(DWORD   size);      // input param

 

It has been specified using the extern "C" specifier so that the function name has no C++ name mangling decoration.

 

It uses the __cdecl calling convention, takes one input parameter and is marked as an allocator by the Alloc definition.

 

For datatype purposes it can be specified as returning the BYTE datatype.

 

The custom hook will monitor:

 

the return parameter, marked as a pointer

the single input parameter, marked as a size specifier.

 

custom-hook-example1

 

 

Custom hook dialog - example 2

 

A custom hook dialog for a function using the __stdcall calling convention is shown below.

 

The function prototype for ?customStdCallAlloc3@@YGXKPAPAX@Z is in the DLL testCustomDLL.dll is:

 

   extern void  __stdcall customStdCallAlloc3(DWORD        size,         // input param
                                                   void        **ptr);       // output param

 

It has been compiled as C++, having C++ name mangling decoration.

 

It uses the __stdcall calling convention, and takes two input parameters.

 

The function purpose as an allocator is defined by the Alloc setting.

 

For datatype purposes it can be specified as returning the BYTE <STDCALL> datatype.

 

The custom hook will monitor:

 

the first input parameter, marked as a size specifier

the second output parameter, marked as a pointer to pointer.

 

 

custom-hook-example2

 

 

Custom hook dialog - example 3

 

Similar to example 2, except we have a different function name and its purpose is now as a Realloc function, with a pointer return type.

 

The function prototype for customStdCallReAlloc1 is:

 

   extern void*  __stdcall customStdCallReAlloc1(DWORD   size,         // input param
                                                       void    **ptr);       // output param

 

The custom hook will monitor:

 

the return parameter as a pointer

the first input parameter, marked as a size specifier

the second input parameter, marked as a pointer to pointer.

 

custom-hook-example3