COMS W4181 · Security I · Fall 2026 · Lecture 7

Memory
Corruption

Columbia University

$ ./vuln $(python3 -c 'print("A"*144)') Segmentation fault (core dumped) # that crash is a control-flow primitive
[+] Where we are going
  • 01Basic memory corruption - the stack, and the stack smash
  • 02More corruption: integer overflows, format strings, use-after-free
  • 03Platform defenses: W^X/DEP, ASLR, canaries, CFI
  • 04Advanced attacks: ROP and heap spraying

Notice the shape of the lecture: attack, defense, better attack. That escalation is the actual history of the field.

[+] Memory corruption attacks
The attacker's goal Take over the target machine (e.g. a web server) - execute arbitrary code on the target by hijacking application control flow, leveraging memory corruption.

Examples

  • Buffer overflow attacks
  • Integer overflow attacks
  • Format string vulnerabilities

The bug corrupts data. The exploit turns that into control over what executes next.

[+] Example 1: buffer overflows
  • Extremely common bug in C/C++ programs First major exploit: the 1988 Internet Worm, via fingerd.
20%of all reported vulnerabilities, at its peak (source: NVD/CVE)
1988first major remote exploit
37years later, still shipping

The share has fallen as mitigations landed and as memory-safe languages spread - but the absolute count has not gone to zero, because the C and C++ codebase underneath everything has not gone away.

[+] What you need to know to do this
  • C functions, the stack, and the heap
  • How system calls are made - in particular the exec() system call
  • Which CPU and OS the target runs Our examples are for x86 running Linux or Windows. Details vary slightly between CPUs and OSes: little-endian vs big-endian (x86 vs Motorola), and stack frame structure (Unix vs Windows).

Exploitation is architecture-specific in a way that most of this course is not. The concepts transfer; the byte offsets do not.

[+] Linux process memory layout
  • Loaded from exec - the program text and initialized data, at a fixed low address.
  • The heap grows up from brk.
  • The stack grows down from the top, tracked by %esp.
  • Shared libraries sit in the middle - remember where libc lives; we will come back for it.

Two regions grow toward each other, and the attacker's input lands in both.

unused0xC0000000
user stack%esp, grows down
|
shared libraries0x40000000
|
run time heapbrk, grows up
loaded from exec0x08048000
unused0
[+] The stack frame

Every call pushes a frame. The frame contains both the function's data and the address it will return to.

The whole vulnerability, in one sentence A local buffer sits at a lower address than the saved return address, and the stack grows down - so writing past the end of the buffer walks upward, straight into the return address.
exception handlershigh
arguments
return address
stack frame pointer
local variablesSP, low

Stack growth is downward: toward low addresses. Writes into a buffer go upward.

[+] What are buffer overflows?

Suppose a web server contains this function:

void func(char *str) { char buf[128]; strcpy(buf, str); do_something(buf); }

When func() is called, the stack looks like the frame on the right.

argument: strhigh
return address
stack frame pointer
char buf[128]SP, low
[x] What if *str is 136 bytes long?
void func(char *str) { char buf[128]; strcpy(buf, str); // no length check do_something(buf); }
The problem There is no length checking in strcpy(). The extra 8 bytes run past the end of buf and overwrite the saved frame pointer and the return address.
argument: strhigh
return address*str
stack frame pointer*str
char buf[128]*str

after strcpy: the copy did not stop at the end of the buffer

[x] The basic stack exploit

Suppose *str is chosen so that, after strcpy, the stack looks like this.

  1. Program P is placed inside the buffer: exec("/bin/sh")
  2. The return address is overwritten to point back into the buffer
  3. When func() exits, the ret instruction jumps to P
  4. The user gets a shell.

Note: the attack code P runs on the stack. Hold on to that - it is what the first platform defense will attack.

...high
return address→ buf
Program P
char buf[128]low
control returns into attacker data
[x] The NOP slide

? How does the attacker determine the exact return address?

Solution: a NOP slide Guess the approximate stack state when func() is called, and insert many NOPs before program P. Land anywhere in the slide and execution slides forward into P.
nop xor eax, eax // also a no-op, harder to signature inc ax
...high
return address→ anywhere in the slide
Program P
NOP slide
char buf[128]low

A large target instead of a precise one.

[+] Details and examples

Some complications

  • Program P must not contain the '\0' character - because strcpy would stop copying there.
  • The overflow must not crash the program before func() exits

(In)famous remote stack-smashing overflows

  • 2007 - Windows animated cursors (ANI), LoadAniIcon() An animated cursor on a web page or in an email message resulted in arbitrary code execution. Used for rendering cursors, animated cursors and icons.
  • 2005 - Symantec Virus Detection test.GetPrivateProfileString "file", [long string] - the antivirus product was the attack surface.
[x] Many unsafe libc functions
strcpy (char *dest, const char *src) strcat (char *dest, const char *src) gets (char *s) scanf (const char *format, ...) // ... and many more
  • The "safe" versions are misleading strncpy() and strncat() take a length, but strncpy() may leave the string unterminated - which then breaks the next function that assumes a NUL.
  • Windows C run time (CRT): strcpy_s(*dest, DestSize, *src) ensures proper termination.

An API that is easy to misuse will be misused.

[+] Buffer overflow opportunities

The return address is the obvious target, but any code pointer will do.

  • Exception handlers (Windows SEH attacks) Overwrite the address of an exception handler in the stack frame, then cause an exception.
  • Function pointers (e.g. PHP 4.0.2, MS Media Player bitmaps) Overflowing buf overrides an adjacent function pointer - on the heap or the stack.
  • longjmp buffers (e.g. Perl 5.003) Overflowing a buffer next to pos overrides the value that longjmp(pos) will jump to.

Media Player skins were a heap overflow (Feb 2006). setjmp is used for exception handling - jumping to global error-handling code.

[x] Corrupting method pointers

Compiler-generated function pointers, e.g. in C++: an object holds a pointer to its vtable, and the vtable holds pointers to methods.

BEFORE ptr buf[256] data object T FP1 FP2 FP3 vtable method #2 AFTER OVERFLOW OF buf ptr → attacker's fake vtable NOP slide + shellcode next virtual call → attacker code
[+] Finding buffer overflows
  1. Run the web server on a local machine
  2. Issue malformed requests - for example, ones ending with "$$$$$" Many automated tools exist: fuzzers, symbolic and concolic execution.
  3. If the server crashes, search the core dump for "$$$$$" to find the overflow location
  4. Construct the exploit - not easy, given the latest defenses

The marker string is the trick: it lets you find where your input landed in a megabyte of memory, and therefore which offset controls the instruction pointer.

[ 02 ]

More memory
corruption

Integer overflows, double free, use-after-free, format strings.

[+] More corruption opportunities
  • Integer overflows (e.g. MS DirectX MIDI library)
  • Double free - free the same heap space twice Can cause the memory manager to write data to a specific location. Example: the CVS server.
  • Use after free - using memory after it has been freed The allocator has since handed that memory to someone else, so the two users disagree about what the bytes mean.
  • Format string vulnerabilities

Double free and use-after-free are the dominant classes in modern browsers - the stack has been hardened much more thoroughly than the heap.

[+] Integer overflows (see Phrack 60)

What happens when an integer exceeds its maximum value?

int m; // 32 bits short s; // 16 bits char c; // 8 bits c = 0x80 + 0x80 = 128 + 128 => c = 0 s = 0xff80 + 0x80 => s = 0 m = 0xffffff80 + 0x80 => m = 0

? Can this be exploited?

[x] An example
void func(char *buf1, *buf2, unsigned int len1, len2) { char temp[256]; if (len1 + len2 > 256) { return -1; } // the length check memcpy(temp, buf1, len1); // concatenate the buffers memcpy(temp + len1, buf2, len2); do_something(temp); }
What if len1 = 0x80 and len2 = 0xffffff80? len1 + len2 = 0, so the length check passes. The first memcpy copies 128 bytes. The second copies 4,294,967,168 bytes - and overflows the heap.

The check exists. It does not compute what you think.

[x] The format string problem
int func(char *user) { fprintf(stderr, user); // user data used AS the format string }

? What if *user = "%s%s%s%s%s%s%s"?

  • Most likely the program crashes - denial of service.
  • If not, the program prints memory contents - a privacy failure.
  • Full exploit using %n
// correct form: fprintf(stdout, "%s", user);
[+] Vulnerable functions

Any function that takes a format string.

Printing
printf, fprintf, sprintf, ... vprintf, vfprintf, vsprintf, ...
Logging
syslog, err, warn

The logging functions are the dangerous ones in practice - logging attacker-supplied data is exactly what a server does all day, and syslog(LOG_ERR, user_input) looks entirely reasonable.

[x] Format string exploitation
Dumping arbitrary memory Walk up the stack until the desired pointer is found:
printf("%08x.%08x.%08x.%08x|%s|")
Writing to arbitrary memory %n writes the number of bytes printed so far to the pointer argument:
printf("hello %n", &temp) writes 6 into temp.
printf("%08x.%08x.%08x.%08x.%n") writes to whatever the stack happens to hold - and the attacker controls both the value written and where.

A read primitive and a write primitive, from one printf.

[ 03 ]

Platform
defenses

If you cannot remove the bugs, remove their usefulness.

[+] Preventing hijacking attacks
  • Fix bugs Audit software - automated tools such as Coverity, Prefast/Prefix. Rewrite software in a type-safe language (Java, ML) - difficult for existing legacy code.
  • Concede the overflow, but prevent code execution If attacker bytes cannot execute, corrupting memory is no longer enough.
  • Add runtime code to detect overflow exploits Halt the process when an exploit is detected: StackGuard, LibSafe, and others.

Three different philosophies: eliminate the bug, neuter the bug, or detect the exploit. Real systems deploy all three, because none of them is complete.

[/] Marking memory non-executable (W^X) code pages executable, not writable stack + heap writable, not executable shellcode on the stack now faults instead of running
  • NX-bit on AMD Athlon 64, XD-bit on Intel P4 Prescott An NX bit in every Page Table Entry.
  • Deployment: Linux (via the PaX project), OpenBSD, Windows since XP SP2 (DEP). Visual Studio: /NXCompat[:NO].
  • Limitations: Some applications need an executable heap (e.g. JIT compilers). And it does not defend against Return-Oriented Programming.
[x] Attack: Return-Oriented Programming

Control hijacking without executing any custom code. If you cannot inject instructions, reuse the ones already there.

  1. Overflow as before, but instead of shellcode, write a chain of addresses
  2. Point the return address at existing code - e.g. exec() inside libc.so
  3. Place the argument - a pointer to the string "/bin/sh" - where the callee expects it
  4. The CPU executes only legitimate, executable library code. DEP is satisfied.

Chaining short "gadgets" that each end in ret makes this Turing-complete.

args→ "/bin/sh"
ret-addr→ exec()
sfp
local buflow
libc.so - mapped and executable printf() exec() "/bin/sh"
[/] Response: randomization
ASLR - Address Space Layout Randomization Map shared libraries to a random location in process memory, so the attacker cannot jump directly to the exec function.
  • Deployment (/DynamicBase) Windows 7: 8 bits of randomness for DLLs - aligned to a 64K page in a 16MB region, so 256 choices. Windows 8: 24 bits of randomness on 64-bit processors.
  • Other randomization methods System-call randomization (randomize syscall IDs); Instruction Set Randomization (ISR).
  • Everything in process memory must be randomized Stack, heap, shared libraries, base image. Windows 8's Force ASLR ensures all loaded modules use ASLR - one non-randomized module is enough to build a chain from.

NX and ASLR together are what raised the bar.

[+] ASLR example

Booting twice loads the libraries into different locations.

first boot
0x7c370000 kernel32.dll 0x77450000 user32.dll 0x10a80000 app.exe
second boot
0x7c8d0000 kernel32.dll 0x77c10000 user32.dll 0x0f620000 app.exe
Everything in process memory must be randomized Stack, heap, shared libraries, and the base image. One module left at a fixed address is enough to build a ROP chain from - which is why Windows 8 added Force ASLR to ensure every loaded module participates.
[/] Defenses: the stack canary
StackGuard Run-time tests for stack integrity: embed "canaries" in stack frames and verify their integrity prior to function return.

A sequential overflow that reaches the return address must pass through the canary. Check the canary before returning, and you detect the overwrite before the corrupted pointer is ever used.

There are many run-time checking techniques; we only discuss the ones relevant to overflow protection.

strframe 2, high
ret
sfp
canarychecked on return
local
strframe 1
ret
sfp
canary
localtop of stack, low
[+] Canary types
Random canary A random string chosen at program startup, inserted into every stack frame and verified before returning from a function. Exit the program if the canary changed - which turns a potential exploit into a denial of service.
To corrupt it, the attacker must first learn the current random string.
Terminator canary Canary = {0, newline, linefeed, EOF}. String functions will not copy beyond a terminator, so the attacker cannot use string functions to write past it.

Two different assumptions: the random canary assumes the attacker cannot read memory; the terminator canary assumes the attacker is limited to string functions. Both assumptions can fail.

[x] Canaries are not foolproof

Canaries are an important defense tool, but they do not prevent all control-hijacking attacks.

  • Heap-based attacks are still possible The canary protects a stack frame. It says nothing about a corrupted vtable pointer on the heap.
  • Integer overflow attacks are still possible
  • Exception-handling attacks may not be prevented - it depends on the implementation.

A canary protects one word, in one place, at one moment.

[+] More methods
StackShield At the function prologue, copy the return address RET and the saved frame pointer SFP to a "safe" location - the beginning of the data segment. On return, check that RET and SFP equal the copy. Implemented as an assembler file processor for GCC.
Control Flow Integrity (CFI) A combination of static and dynamic checking: statically determine the program's legal control flow, then dynamically enforce that every indirect transfer stays inside it.

CFI is the principled answer to ROP - it constrains where you may return to, not just what you may execute. Modern hardware support (Intel CET, ARM PAC/BTI) is CFI in silicon.

[ 04 ]

Advanced attacks:
heap spraying

A reliable method for exploiting heap overflows.

[x] Heap-based control hijacking

Suppose the vtable is on the heap, next to a string object.

BEFORE ptr buf[256] data object T FP1 FP2 FP3 vtable method #1 method #2 method #3 AFTER OVERFLOW OF buf ptr → shellcode shellcode written over the vtable region
[?] A reliable exploit?
<SCRIPT language="text/javascript"> shellcode = unescape("%u4343%u4343%..."); overflow_string = unescape("%u2332%u4276%..."); cause_overflow(overflow_string); // overflow buf[] </SCRIPT>
The problem The attacker does not know where the browser places the shellcode on the heap - so there is no address to put in the corrupted pointer.

The stack case had the NOP slide. The heap needs something bigger.

ptr→ ???
buf[256]
data
...
shellcodesomewhere
[x] Heap spraying [SkyLined 2004]
  1. Use JavaScript to spray the heap with shellcode (and NOP slides)
  2. Then point the vtable pointer anywhere in the spray area
HEAP NOP slideshellcode NOP slideshellcode NOP slideshellcode NOP slideshellcode NOP slideshellcode NOP slideshellcode heap spray area - hundreds of megabytes vtable ptr → almost any address
[x] JavaScript heap spraying
var nop = unescape("%u9090%u9090"); // 0x90 = NOP while (nop.length < 0x100000) nop += nop; // grow to 1 MB var shellcode = unescape("%u4343%u4343%..."); var x = new Array(); for (i = 0; i < 1000; i++) { x[i] = nop + shellcode; // 1000 x 1 MB }

Pointing the function pointer almost anywhere in the heap will now cause the shellcode to execute.

Note what the attacker is using: an ordinary, well-specified JavaScript feature. The only bug is the overflow that lets them redirect a pointer.

[x] Vulnerable buffer placement

Placing the vulnerable buf[256] right next to object O.

HEAP, AFTER GROOMING free block free block vulnerable buf[256] object O free block overflow walks straight into O
  • By a sequence of JavaScript allocations and frees, make the heap look like the picture
  • Allocate the vulnerable buffer in JavaScript and cause the overflow
  • Successfully used against a Safari PCRE overflow [DHM'08]
[x] Many heap spray exploits
Heap Feng Shui [Sotirov 2007] Reliable heap exploits on Internet Explorer without spraying - the technique gives the attacker full control of the IE heap from JavaScript.

The progression is worth naming: from hope the shellcode is somewhere, to make it everywhere, to place it exactly where I want it.

2004heap spraying - saturate the address space
2007Heap Feng Shui - deterministic layout control
2008engineered overflows against real browsers
[/] (Partial) defenses
  • Protect heap function pointers Encode or guard pointers so that a corrupted value is detected before use.
  • Better browser architecture Store JavaScript strings in a separate heap from the browser heap - so the attacker cannot place their data next to the browser's code pointers.
  • OpenBSD heap overflow protection Non-writable guard pages between allocations prevent cross-page overflows.
  • Nozzle [RLZ'08] Detect sprays by the prevalence of code-like content on the heap.

"Partial" is the operative word in the title. Each of these raises the cost of one technique; none removes the underlying memory-safety bug.

[+] Takeaways
  • Memory corruption becomes code execution through a code pointer. Return addresses, vtables, exception handlers, longjmp buffers - all equivalent targets.
  • Every mitigation is target-specific, so attacks migrate. Canaries pushed attackers to the heap; DEP pushed them to ROP; ASLR pushed them to info leaks.
  • Exploit reliability is an engineering problem, and it got solved. NOP slides, heap spraying, and grooming exist to remove the attacker's uncertainty.
  • The only complete fix is memory safety. Everything else raises cost.
$ ./memory --mitigations all --safe-language false # expensive. unreliable. still possible.
[+] References on heap spraying
  • Heap Feng Shui in JavaScript A. Sotirov. Blackhat Europe, 2007.
  • Engineering Heap Overflow Exploits with JavaScript M. Daniel, J. Honoroff and C. Miller. WOOT, 2008.
  • Nozzle: A Defense Against Heap-spraying Code Injection Attacks P. Ratanaworabhan, B. Livshits and B. Zorn.
  • Interpreter Exploitation: Pointer Inference and JIT Spraying Dion Blazakis.

Also worth reading: Aleph One, "Smashing the Stack for Fun and Profit," Phrack 49 (1996) - the paper that made this material public, and Phrack 60 on integer overflows.

[+] Acknowledgements

These lecture materials were provided by Suman Jana (Columbia University), and this deck is a re-typeset and extended version of his Security I slides.

The original slides were created by Prof. Dan Boneh.

Techniques and results: heap spraying [SkyLined, 2004]; Heap Feng Shui [A. Sotirov, 2007]; engineered heap overflows [Daniel, Honoroff, Miller, 2008]; Nozzle [Ratanaworabhan, Livshits, Zorn, 2008].

Errors introduced in the re-working are mine.

← all lectures