Instrumentation and debugging
Let's work with an actual C program that we'll use throughout this short guide. Source.
We'll compile this program with debugging symbols, disable optimization & add stack protection for security features.
gcc -g -O0 -fstack-protector-all debug_target.c -o debug_target
Invoke GDB with the program name to load the program into memory before execution.
gdb ./debug_target
To debug a program with command line arguments, specify them when starting GDB.
gdb --args ./debug_target "AAAAAAAAAAAAAAAAAAAA"
After loading the program, you need to actually start its execution. The program will run until it encounters a breakpoint, crashes or completes.
run
A breakpoint tells the debugger to pause execution when it reaches a specific location. We can then examine the program state.
break main
Set breakpoints at specific source code lines by referencing the file and line number.
break debug_target.c:58
Set up a function-specific breakpoint.
break auth
Set up a conditional breakpoint.
break create_list if i == 3
Once we hit a breakpoint, there are several options for controlling program execution. continue resumes normal execution until the next breakpoint or program termination.
continue
Step through the program one line at a time.
next
Follow execution into function calls.
step
Step through a function and continue until it returns to its caller.
finish
> Run till exit from #0 create_list (count=5) at debug_target.c:21
0x0000555555555507 in main (argc=2, argv=0x7fffffffde58) at debug_target.c:115
115 struct node *my_list = create_list(5);
Value returned is $1 = (struct node *) 0x5555555596b0
Execute one machine instruction at a time without entering function calls.
nexti
Execute a single machine instruction, following it into function calls if necessary.
stepi
Display the value of a variable or expression.
print my_list
> $2 = (struct node *) 0x5555555596b0
Examine pointers to structures.
print *my_list
> $3 = {value = 0, next = 0x555555559710}
Follow the entire linked list structure.
print *my_list->next
print *my_list->next->next
> $4 = {value = 10, next = 0x555555559770}
> $5 = {value = 20, next = 0x5555555597d0}
Set a watchpoint that triggers when a variable's value changes.
watch my_list->value
Inspect raw memory contents in various formats.
x/10xb my_list # examine 10 hex bytes at my_list address
> 0x5555555596b0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x5555555596b8: 0x10 0x97
x/4xw my_list # examine 4 hex words at my_list address
> 0x5555555596b0: 0x00000000 0x00000000 0x55559710 0x00005555
x/s secret_message # examine as string
> 0x555555556004: "secret"
Show the current execution path.
backtrace
> #0 main (argc=2, argv=0x7fffffffde58) at debug_target.c:117
Move between stack frames to inspect different levels of the call hierarchy.
frame 2 # switch to frame number 2
info locals # show local variables in current frame
> my_list = 0x5555555596b0
password = '\000' <repeats 29 times>
out = 0
bits = 0
Examine heap usage.
info proc mappings # view memory layout
set environment MALLOC_CHECK_ 3 # enable heap checking
A subsection of the mapped address spaces is revealed for proc mappings.
Start Addr End Addr Size Offset Perms objfile
0x555555554000 0x555555555000 0x1000 0x0 r--p /home/abdi/misc/debug_target
0x555555555000 0x555555556000 0x1000 0x1000 r-xp /home/abdi/misc/debug_target
0x555555556000 0x555555557000 0x1000 0x2000 r--p /home/abdi/misc/debug_target
0x555555557000 0x555555558000 0x1000 0x2000 r--p /home/abdi/misc/debug_target
0x555555558000 0x555555559000 0x1000 0x3000 rw-p /home/abdi/misc/debug_target
0x555555559000 0x55555557a000 0x21000 0x0 rw-p [heap]
0x7ffff7dc9000 0x7ffff7dcc000 0x3000 0x0 rw-p
0x7ffff7dcc000 0x7ffff7df2000 0x26000 0x0 r--p /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7df2000 0x7ffff7f47000 0x155000 0x26000 r-xp /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7f47000 0x7ffff7f9a000 0x53000 0x17b000 r--p /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7f9a000 0x7ffff7f9e000 0x4000 0x1ce000 r--p /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7f9e000 0x7ffff7fa0000 0x2000 0x1d2000 rw-p /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7fa0000 0x7ffff7fad000 0xd000 0x0 rw-p
0x7ffff7fc2000 0x7ffff7fc4000 0x2000 0x0 rw-p
0x7ffff7fc4000 0x7ffff7fc8000 0x4000 0x0 r--p [vvar]
0x7ffff7fc8000 0x7ffff7fca000 0x2000 0x0 r-xp [vdso]
0x7ffff7fca000 0x7ffff7
Disassemble the current function.
disassemble
Output for main (only relevant subsection) where 80 bytes is allocated for the stack frame. We can find where we store the stack canary value from the thread-local storage segment—FS register—since we compiled with stack protection.
0x00005555555554d0 <+0>: push %rbp
0x00005555555554d1 <+1>: mov %rsp,%rbp
0x00005555555554d4 <+4>: sub $0x50,%rsp <-- local stack space
0x00005555555554d8 <+8>: mov %edi,-0x44(%rbp) <-- argc
0x00005555555554db <+11>: mov %rsi,-0x50(%rbp) <-- argv
0x00005555555554df <+15>: mov %fs:0x28,%rax <-- stack canary value
0x00005555555554e8 <+24>: mov %rax,-0x8(%rbp)
0x00005555555554ec <+28>: xor %eax,%eax
# other instructions...
0x00005555555555ec <+284>: mov -0x8(%rbp),%rdx
0x00005555555555f0 <+288>: sub %fs:0x28,%rdx
0x00005555555555f9 <+297>: je 0x555555555600 <main+304>
0x00005555555555fb <+299>: call 0x555555555060 <__stack_chk_fail@plt>
0x0000555555555600 <+304>: leave
0x0000555555555601 <+305>: ret
In the epilogue, we jump to the frame cleanup if the stack isn't corrupted. Otherwise, __stack_chk_fail [1] terminates the program.
leaveis equivalent tomov %rbp, %rspfollowed bypop %rbp.
Mixing source code with assembly gives an improved context for the machine instructions.
disassemble /s count_bits
Change variable values to test different execution paths.
set password = "debug_me_if_you_can"
Or directly manipulate function return values.
break auth
commands
silent
set $eax = 1
continue
end
Intercept library function calls.
catch syscall write
break strcpy
> Catchpoint 5 (syscall 'write' [1])
> Breakpoint 6 at gnu-indirect-function resolver at 0x7ffff7e6a8d0
LLDB is an alternative debugger from the LLVM project with similar capabilities but a different syntax.
Start debugging the program with LLDB.
lldb ./debug_target
Set a breakpoint at a function.
breakpoint set --name main
Run the program with arguments.
run "AAAAAAAAAAAAAAAAAAAA"
Print variable values.
print my_list
print *my_list
Examine memory.
memory read --size 1 --format x --count 10 my_list
Show the call stack.
thread backtrace
Set a conditional breakpoint.
breakpoint set --name create_list --condition "i == 3"
Check leaks with valgrind.
valgrind --leak-check=full ./debug_target
There should be a discrepancy between 14 allocations and 8 frees with 6 missing frees which correspond to the leaked temp buffers. Heap and leak summary indicate this.
==15315== HEAP SUMMARY:
==15315== in use at exit: 300 bytes in 6 blocks
==15315== total heap usage: 14 allocs, 8 frees, 2,444 bytes allocated
==15315==
==15315== 300 bytes in 6 blocks are definitely lost in loss record 1 of 1
==15315== at 0x48417B4: malloc (vg_replace_malloc.c:381)
==15315== by 0x109253: create_list (debug_target.c:39)
==15315== by 0x109506: main (debug_target.c:115)
==15315==
==15315== LEAK SUMMARY:
==15315== definitely lost: 300 bytes in 6 blocks
==15315== indirectly lost: 0 bytes in 0 blocks
==15315== possibly lost: 0 bytes in 0 blocks
==15315== still reachable: 0 bytes in 0 blocks
==15315== suppressed: 0 bytes in 0 blocks
We can reveal how and if the program uses specific kernel mode functions.
strace ./debug_target
With timing information, pass -c. Optional features enabled are stack-trace=libunwind, stack-demangle, m32-mpers, mx32-mpers (check strace --version).
Monitor library calls.
ltrace ./debug_target
With -c, ltrace outputs the timing.
% time seconds usecs/call calls function
------ ----------- ----------- --------- --------------------
99.83 2.234673 2234673 1 __isoc99_scanf
0.05 0.001020 85 12 malloc
0.04 0.000852 94 9 printf
0.03 0.000725 120 6 sprintf
0.03 0.000626 208 3 puts
0.02 0.000390 65 6 free
0.01 0.000168 168 1 strcmp
------ ----------- ----------- --------- --------------------
100.00 2.238454 38 total
When working with stripped binaries where symbols have been removed, consider objdump for static analysis.
objdump -d debug_target
Identify strings in the binary.
strings debug_target
strings -a -t x debug_target # with addresses in hex
Analyze binary headers and sections.
readelf -a debug_target
Examine runtime process memory.
./debug_target & pid=$!
gcore -o memdump $(pidof debug_target)
Analyze the memory dump for patterns or specific content.
strings -a memdump | grep password
Instrumentation is mostly about recognizing common patterns.
In x64, function prologues set up the stack frame and save registers.
push rbp
mov rbp, rsp
sub rsp, 32
Function epilogues clean up the stack before returning.
mov rsp, rbp
pop rbp
ret
Loop constructs use a counter and conditional jump.
loop_start:
; loop body
dec ecx
jnz loop_start
Conditional branches compare values and jump based on the result.
cmp eax, 0
je equal_case
Binary patching modifies program behavior by changing the executable file. E.g. to bypass the authentication check in our program, we could patch auth to always return 1.
objdump -d debug_target | grep -A20 "<auth>:"
View the disassembled auth.
1331: 85 c0 test %eax,%eax
1333: 75 07 jne 133c <auth+0x3c>
1335: b8 01 00 00 00 mov $0x1,%eax
133a: eb 05 jmp 1341 <auth+0x41>
133c: b8 00 00 00 00 mov $0x0,%eax
We identified that the instruction at offset 0x1333 (jne 133c) was our target which jumps to return 0 if the password check fails.
We then replace the conditional jump 75 07 with two NOP instructions, 90 90 which would cause auth to always execute the return 1 code path instead—regardless of the password entered.
hexedit debug_target
Ctrl+G to open the "go to" dialog. Enter the hex offset 1333 and enter. Replace the chunks by just typing, Ctrl+X to exit and Y to save.
00001300 55 48 89 E5 48 83 EC 20 48 89 7D E8 64 48 8B 04 UH..H.. H.}.dH..
00001310 25 28 00 00 00 48 89 45 F8 31 C0 48 8B 45 E8 48 %(...H.E.1.H.E.H
00001320 8D 15 09 0D 00 00 48 89 D6 48 89 C7 E8 4F FD FF ......H..H...O..
00001330 FF 85 C0 90 90 B8 01 00 00 00 EB 05 B8 00 00 00 ................
--- debug_target --0x1333/0x4C70--25%-------------------------------
Try the patched binary.
chmod +x debug_target
./debug_target
PIN injects code into a running process without modifying the binary (i.e. dynamic instrumentation). It's a tool by Intel.
pin -t pintool.so -- ./debug_target
ASan (Address Sanitizer) is a ubiquitous extension for debugging memory corruption.
gcc -g -fsanitize=address debug_target.c -o debug_target_asan
./debug_target_asan "AAAAAAAAAAAAAAAAAAAA"
There are various anti-debugging techniques employed by malware and protected software.
They might check for debugger presence.
if (ptrace(PTRACE_TRACEME, 0, 1, 0) < 0) {
exit(1); /* debugger detected */
}
To counter such checks in GDB, catch ptrace itself.
catch syscall ptrace
commands
set $rax = 0
continue
end
Time-based anti-debugging checks compare execution duration.
time_t start = time(NULL);
/* op */
if (time(NULL) - start > 1) exit(1); /* debugger detected */
Bypass by setting the return value of time().
break time
commands
set $rax = 0x5FFFFFFF
continue
end
Identify key generation and transformation functions in cryptography programs. We might use watchpoints to see when certain memory regions are accessed.
watch -l *(char(*)[32])0x12345678
Core dumps provide debugging information after a crash.
ulimit -c unlimited
./debug_target "$(python -c 'print("A"*100)')"
gdb ./debug_target core
Now test the exploitability of the buffer overflow in vulnerable_function:
python -c 'print("A"*20 + "BBBB")' | ./debug_target
Check in GDB to see if we can control the instruction pointer.
run < <(python -c 'print("A"*20 + "BBBB")')
Dynamically trace decryption operations.
break *0x12345678 # address of decryption function
commands
silent
dump memory decrypted.bin $rdi $rdi+$rsi
continue
end
Analyze network traffic from the program.
tcpdump -i lo -w capture.pcap port 8080
Identify embedded structures.
binwalk debug_target
Reveal hidden data with memory dumps from specific regions.
gdb -p $(pidof debug_target)
dump memory heap.bin 0x08048000 0x08049000
- ← Previous
x86 lea instruction - Next →
Linking object files