Normally, the break instruction exception can be triggered in following conditions:
1. Hardcode interrupt request, like: __asm int 3 (ASM), System.Diagnostics.Debugger.Break (C#), DebugBreak() (WinAPI).
2. OS enable memory runtime check, like Application Verifier can trigger after heap corruption, memory overrun.
3. Compiler can have some configuration to decide what should be filled to the uninitialized memory block and end of function(blank area, after retun..). For example, Microsoft VC complier can fill 0xCC if enable /GZ. 0xCC is actually a opcode of __asm int 3. So if some error cause the application run into such block, will trigger a break point.
A quick summary of what Microsoft's compilers use for various bits of unowned/uninitialized memory when compiled for debug mode (support may vary by compiler version):
Value Name Description
------ -------- -------------------------
0xCD Clean Memory Allocated memory via malloc or new but never
written by the application.
0xDD Dead Memory Memory that has been released with delete or free.
Used to detect writing through dangling pointers.
0xFD Fence Memory Also known as "no mans land." This is used to wrap
the allocated memory (surrounding it with a fence)
and is used to detect indexing arrays out of
bounds or other accesses (especially writes) past
the end (or start) of an allocated block.
0xCC When the code is compiled with the /GZ option,
uninitialized variables are automatically assigned
to this value (at byte level).
// the following magic values are done by the OS, not the C runtime:
0xAB (Allocated Block?) Memory allocated by LocalAlloc().
0xBAADF00D Bad Food Memory allocated by LocalAlloc() with LMEM_FIXED,but
not yet written to.
0xFEEEFEEE OS fill heap memory, which was marked for usage,
but wasn't allocated by HeapAlloc() or LocalAlloc().
Or that memory just has been freed by HeapFree().
Disclaimer: the table is from some notes I have lying around - they may not be 100% correct (or coherent).
As others have noted, one of the key properties of these values is that is a pointer variable with one of these values is dereferenced, it will result in an access violation, since on a standard 32-bit Windows configuration, user mode addresses will not go higher than 0x7fffffff.
For the related issue, we can use Application Verifier to enable heap page, which can break after memory overrun, heap corruption.