HeapCheck 1.2 Thanassis Tsiodras, Dr.-Ing HeapCheck is a Debugging Library for Win32 environments. What sets it apart from other heap-debugging libraries is its ability to pinpoint invalid read accesses to the heap, not just heap corruptions. It also detects memory leaks and reports memory usage statistics. Access checks are done through the paging hardware of your CPU, and don't introduce any CPU overhead. HeapCheck debug versions run at almost the same speed as normal debug versions - with performance hits noticeable only when allocating and deallocating. 1. Disclaimer I am NOT responsible for any damages this causes to your computer, company, fame, payroll, etc. I am releasing my humble efforts to the public so that some programmer's life gets a little easier. If it leads you on a wild goose chase and you waste two weeks debugging something, too bad. If you can't deal with the above, please don't use the software! I've written this in an attempt to help other people, not to get myself sued or prosecuted. 2. Features HeapCheck is a Win32 Debugging Allocator with the following features: 1. Detection of invalid read/write accesses to memory allocated with heap functions. Let me stress this out: READ as well as write accesses! 2. Detection of memory leaks 3. Detection of invalid read/write accesses to freed memory 4. Thread safe 5. File/Line info reporting, based on imagehlp.dll 6. A GNU license (which means that it's free - read GnuLicense) If you are eager to check it out, go to the Usage section. Just make sure you eventually read everything, or you'll be missing some goodies (like the ones in section ``Stack techniques''). 3. Introduction As you program with C or C++, you are bound to hit upon a heap-related bug sooner or later. What's a heap, you ask? Well, it's the system resource from which you remove chunks when you call malloc(), calloc(), new(), etc. Unfortunately, it is associated with a lot of really tough bugs. You see, in this code: main() { char *p = (char *) malloc(5); strcpy( p, "12345"); puts(p); } ...you probably get away clean in the debugging phase. You only allocated 5 bytes, you are writing 6 (5 for the string plus one trailing 0), but nothing seems to happen, right? Wrong. The extra byte you are writing moves your program into such things, others don't. Even those who seem to do, eventually don't. You see, the extra byte is written into memory not owned by the allocation. What is stored there can be anything from debugging info to application code. Then again, we don't free the memory we allocated. It seems we forgot to: free(p); Notice also that the call to 'puts' makes the system read 6 bytes, since puts is probably implemented in the C runtime library with some- thing like this: while(*p) putchar(*p++); ...which means that not only your code, but also system and RTL (run- time library) code accesses memory it shouldn't. This can make your program behave in a non-predictable way - crash sometimes, work OK as long as you don't run WinAmp... (you get the idea). If you want to quickly find heap-related bugs in your Win32 programming, you can use this library in addition to any others you have found/bought. It is a simple little library, which I have found useful in my programming efforts. Why do I use it instead of others (like the builtin debugging heap of Visual C++ 6.0)? It's small (and thus, easily configurable) and it supports C++. It also lacks any complicated interfaces, and provides you with a quick and easy way to detect invalid READ as well as write accesses (this last reason is probably the strongest point of the library). HeapCheck employs a technique invented in the Unix's Electric Fence, created by Bruce Perens. It uses the virtual memory hardware of your computer to place an inaccessible memory page immediately after (or before, at a #define option) each memory allocation. When software reads or writes this inaccessible page, the hardware issues a segmentation fault, stopping the program at the offending instruction. It is then trivial to find the erroneous statement since Visual C++ will point to the code doing the invalid access. In a similar manner, memory that has been released is made inaccessible, and any code that touches it will get a segmentation fault. HeapCheck also checks for any allocations you forgot to free/delete, and prints a report when your application exits to the Output window of Visual C++. This report tells you also the allocation requirements of your program (how much memory it needed in the previous run). 4. Usage You can easily use HeapCheck in your own programs in three easy steps: 1. In the source files where you want the checks to take place, include the swaps.h header file. Make sure that it is the first include file you are using, after the system and RTL include files, e.g. #include #include ... #include "swaps.h" #include "myfile1.h" #include "myfile2.h" ... This file #defines the calls to heap-related functions in your code to calls in the library. It also provides the prototypes for the new allocation and deletion operators. 2. If your code is in C only, add HeapCheck.c in the project, or If your code is in C++, add HeapCheck.c and CplusCheck.cpp in the project. 3. Add imagehlp.lib in your link libraries. That's it. Your code will be automatically checked for heap errors. To get correct file/line information, make sure you are using the correct versions of the debugging libraries (the Windows 2000 ones). I used dbghelp.dll version 5.0.2195.1, and imagehlp.dll version 5.0.2195.1. Maybe SymGetLineFromAddr() works with other combinations, but I used these DLLs successfully under both NT 4.0sp6 and Windows 2000. By default, range checks will be done for post-block accesses. After checking that everything works OK in this setup, modify your project so that PRE_CHECK is defined, or even edit swaps.h and define it on top: #define PRE_CHECK Recompile everything, and this way, pre-block checking will take place. HeapCheck messages appear on the Debugging Output of Visual C++. If you are running the program outside the IDE, use DebugView to see them (get it from www.sysinternals.com, an awesome site for programming utilities by the way). If you program with MFC, make sure you use it in the form of a dynamic DLL (Project - Settings - General - Microsoft Foundation Classes - Use MFC in a Shared DLL) . Otherwise, you 'll get link problems, since MFC defines a global operator new, just as HeapCheck. If you use it in the form of a DLL, MFC sticks to its own operator new, while your code calls HeapCheck's. When you have debugged your program, just recompile in Release mode. The debugging checks will melt away automatically. 5. Configuration In HeapCheck.c, you can change the way the library behaves by changing: 1. MAX_ALLOCATIONS: Total concurrent allocations possible. If your program needs more, you'll get an assertion at runtime telling you to increase this. 2. MAX_ALLOCATABLE_BLOCK: Total heap available to the application. If your program needs more, you'll get an assertion at runtime telling you to increase this. 3. NO_VC_HEAP_ERRS: It is possible to write code that bypasses the mechanisms of the library, and allocates from the standard VC-heap. An example is code inside the runtime library which is pre-compiled and uses different allocation mechanisms. Example: an enhanced new() operator in iostrini.cpp (line 21) and cerrinit.cpp (line 21) where _new_crt() is used. _new_crt() maps through a #define to a new() operator that takes extra parameters. HeapCheck can't catch this call, so when the time comes for the deletion of these blocks, the library's delete operator doesn't find them in it's tables. It is capable to understand that these are VC-heap blocks though, through the use of _CrtIsValidHeapPointer. If you #define NO_VC_HEAP_ERRS, the library won't complain for such situations. This is the default, but if your code issues direct calls to _malloc_dbg(), _calloc_dbg(), etc, you should disable this. If the previous text sounds like Chinese, leave it as it is. 6. Debugging - Other tools 1. Visual C++ 6.0 Runtime Heap Debugging: This only checks for heap corruption (i.e. destructive side-effects of writing outside the allocations) and even then, it only catches writings in the NO_MANS_LAND_SIZE (default:4 bytes) on either side of the blocks. The detection of these errors is done whenever the user (or the system) calls _CrtCheckMemory(). HeapCheck catches read accesses as well, at the EXACT place they are made, and in a much larger block (a page in i386 systems is 4096 bytes). 2. BoundsChecker: This is a very good debugging tool, capable of catching almost every bug. However, CPU cycles are used in order to perform all these checks, and especially in instrumentation mode and maximum memory checking, the program runs really slow (some programs, like protocol stacks, won't run correctly if they run too slowly). HeapCheck can only catch heap-related errors, but it uses the paging hardware to do the checking. The net result is that HeapCheck debug versions of programs run at almost the SAME speed as normal debug versions. In other words you can leave it in your project and forget about it (it will be automatically removed when you compile a Release version). 7. For UNIX gurus and fanatics only If you have a history using Electric Fence in UNIX, you'll remember that the only thing required was linking with libefence.a. Why can't we do something like this under Win32 for HeapCheck, too? 1. Win32 ain't UNIX. I guess you knew that, but there's this little fact about libraries that makes them almost useless: if a library uses any function of the C run-time library (like memchr, strcpy, etc) it has to link with one of the six versions of the run-time library (RTL): o Release, static, no multithreading o Release, static, multithreading o Release, dynamic, multithreading o Debug, static, no multithreading o Debug, static, multithreading o Debug, dynamic, multithreading Now suppose that your library is linked with one of these RTLs. When you use your library inside one of your own projects, your project MUST be linked with the same RTL! Amazing. I don't know about you, but I prefer to just add two source files in the project and be done with it, instead of building six HeapCheck libraries. I could replace calls to the C run-time library with my own versions of things, but that would lead to the next problem: 2. If HeapCheck was a library with implementations of standard functions (malloc, realloc, etc) then the projects using it would have to choose between using HeapCheck and using the C runtime library. Unless I am doing something wrong, the linker can't tolerate two libraries providing implementations of the same function. How is he to choose? The UNIX ld implements a first-serve attitude: Whichever library provides it first in the link line, is chosen. If anyone of you comes up with any ideas on these matters, do share. This is why HeapCheck is under the GNU license - I made it, it's your turn now to make it better! 8. Stack techniques If you are like me, you probably use code like this sometimes: void f() { char a[100]; ... } Unfortunately, you don't have the automatic HeapCheck's checks for buffers allocated this way. There is a way around this, if you code in C++. Place this template/macro in a commonly accessed header file: template class Array { public: Array(T *p):_p(p) {} ~Array() { delete [] _p; } operator T*() { return _p; } T& operator [](int offset) { return _p[offset]; } T *GetPtr() { return _p; } private: T *_p; }; #define ARRAY(x,y,z) Array y(new x[z]) and then, when you need a temporary array, use it like this: void f() { ARRAY(char,a,100); or, directly: void f() { Array a(new char[100]); As you see, this way heap space is used for your arrays, and it is automatically freed when the variable goes out of scope, just like a stack-based array. So, you get the best of both worlds: automatic freeing (through the template) and bounds checking (through HeapCheck). This technique also lowers your stack usage, which could mean something if you use recursive algorithms. 9. Known bugs HeapCheck uses the paging hardware of your machine to place an extra guard page before or after each allocation your program makes. A page is 4096 bytes for the vast majority of today's CPUs, which means that the memory requirements of your program can increase beyond usual. Of course these requirements vanish when you compile your Release versions, but still, make sure you have plenty of physical memory and/or swapfile for your Debug versions. For example, the default settings of HeapCheck create an 8MB heap. This consists of 8*1048576/4096 = 2048 pages. If you make 1024 allocations of 1 byte each, you'll exhaust this heap (each allocation reserves one data and one guard page). This is the worst case scenario, of course, but it shows how quickly memory fills up with HeapCheck. From version 1.2 onward, guard pages no longer occupy physical storage, which will lower your actual memory requirements by up to 50%. You'll probably have to increase MAX_ALLOCATABLE_BLOCK in most cases, though ...(you'll know when to do this, because HeapCheck will give you an assertion when the heap is exhausted). 10. Contacting the author Why? As you can see in the disclaimer, I am not responsible for any disasters HeapCheck causes to the universe. Seriously though, I am interested in any bug reports/comments. You can e-mail me at ttsiod@softlab.ntua.gr Good luck in your bughunts!