Detecting memory leaks in ISAPI extensions

Three weeks ago I wrote about how to setup IIS for use with ISAPI extensions.

Today I’m going to use Memory Validator to show you how easy it is to monitor memory and handle allocations in IIS, and use that information to detect any memory leaks and handle leaks that may be present in your ISAPI extension. 

I’m going to provide step-by-step instructions, and also a short video for you to watch if step-by-step instructions aren’t the way you learn.

Secure services

IIS is a very secure Windows service. The account security for IIS means that it can’t access most parts of the Windows filesystem or even Windows objects. CGI and ISAPI extensions will only execute if you’ve configured IIS properly. You can’t launch IIS from a memory leak tool, and you can’t inject into IIS from a memory leak tool because of security constraints. That leaves you the option of using an API to interact with IIS from your tool of choice.

But I don’t want to use an API!

“But I don’t want to use an API” I hear you say.

You’re concerned about having to build multiple versions of your ISAPI. One for use with Memory Validator and one for production.

That’s a valid concern, but it’s not the case. You can just build one version of your ISAPI that uses Memory Validator and use that. If you haven’t done “Monitor ISAPI…” from the Launch menu prior to loading your ISAPI the Memory Validator dlls won’t be loaded.

There is no dependency on Software Verify DLLs, you can ship your DLL without any need to ship the Memory Validator DLLs.

But should you need to examine memory allocation behaviour you can just fire up Memory Validator and get to work without making a special build.

ISAPI API

ISAPI extensions provide 3 APIs for IIS to use:

BOOL WINAPI GetExtensionVersion(HSE_VERSION_INFO *pVer);

BOOL WINAPI TerminateExtension(DWORD	dwFlags);

DWORD WINAPI HttpExtensionProc(EXTENSION_CONTROL_BLOCK *pECB);

GetExtensionVersion() is called when the ISAPI is first loaded. We’ll use this to load Memory Validator into IIS.

TerminateExtension() is called when the ISAPI is unloaded. We’ll use this to tell Memory Validator that the work with IIS is done.

HttpExtensionProc() is used to process any requests made to the ISAPI extension.

Memory Validator API

To add the Memory Validator API to the ISAPI we do the following to the ISAPI source code:

1. Add two header files:

   #include "svlMVStubService.h"
   #include "svlServiceError.h"
   

2. We add the following code to GetExtensionVersion(). This includes logging success and failure so that we can identify if anything has gone wrong. Without the logging determining failure inside of IIS is very hard (a debugger and a special build of the Memory Validator DLL is required).

       // load Validator here
   
       svlMVStub_setLogFileName(L"C:\\testISAPIWebsite\\svl_MV_log.txt");
       svlMVStub_deleteLogFile();
   
       SVL_SERVICE_ERROR   errCode;
   #ifdef IS6432
       // x86 with x64 GUI
       errCode = svlMVStub_LoadMemoryValidator6432();
   #else   //#ifdef IS6432
       // x86 with x86 GUI
       // x64 with x64 GUI
       errCode = svlMVStub_LoadMemoryValidator();
   #endif   //#ifdef IS6432
       if (errCode != SVL_OK)
       {
           DWORD   lastError;
   
           lastError = GetLastError();
           svlMVStub_writeToLogFileW(L"C++ Memory Validator load failed. \r\n");
           svlMVStub_writeToLogFileLastError(lastError);
           svlMVStub_writeToLogFile(errCode);
   
           svlMVStub_dumpPathToLogFile();
       }
       else
       {
           svlMVStub_writeToLogFileW(L"C++ Memory Validator load success. \r\n");
   
           errCode = svlMVStub_StartMemoryValidatorForIIS();
           if (errCode != SVL_OK)
           {
                   DWORD   lastError;
   
                   lastError = GetLastError();
                   svlMVStub_writeToLogFileW(L"Starting C++ Memory Validator failed. \r\n");
                   svlMVStub_writeToLogFileLastError(lastError);
                   svlMVStub_writeToLogFile(errCode);
           }
   
           svlMVStub_writeToLogFileW(L"Finished starting C++ Memory Validator\r\n");
       }
   

   You’ll need to edit the location of the logfile to match the name of your website directory.

3. We add the following code to TerminateExtension().

       // unload Validator here
   
       svlMVStub_UnloadMemoryValidator();
   

Finding Memory Leaks in IIS

1. Start Memory Validator
2. From the Launch menu choose the IIS sub menu and then Monitor ISAPI….
3. The monitor ISAPI dialog is displayed. Any settings from a previous launch are displayed. Edit the settings appropriately.
   • Specify the ISAPI dll that is being checked for memory leaks. If working with multiple DLLs, choose the main ISAPI that uses the other DLLs.
   • Specify the website directory.
   • Specify the IIS process (choose Any IIS if you don’t know or don’t care).
   • Specify the web browser (the default is Microsoft Edge as it seems more tolerant of data format errors than Chrome)
   • Specify the URL you wish to use to test the ISAPI.
4. Click OK.

5. Memory Validator copies some DLLs to the directory containing the ISAPI, sets up various variables that will be used inside the IIS process, resets IIS, and launches the web browser to load the specified URL that will load the ISAPI.

6. Use your web application as usual to create memory leaks in your ISAPI.

7. When you’ve finished interacting with the ISAPI, you’ll need to stop IIS. Go to the Launch menu, choose the IIS sub menu then Stop IIS.

8. When IIS has stopped executing Memory Validator displays “Ready” on the status bar, the Memory tab will show any memory leaks and handle leaks that are present.

A short video of this process is shown below.

Conclusion

We’ve completely reworked our ISAPI support so that you have very little to do, just use the API as shown above and launch using the Monitor ISAPI command.

In comparison, here are some instructions for using Boundschecker with IIS. These are out of date and will no longer work. But look how much you have to do manually.