Ctf Reverse

CTF Reverse Engineering

Quick reference for RE challenges. For detailed techniques, see supporting files.

Additional Resources

• [tools.md](tools.md) - Static analysis tools (GDB, Ghidra, radare2, IDA, Binary Ninja, dogbolt.org, RISC-V with Capstone, Unicorn emulation, Python bytecode, WASM, Android APK, .NET, packed binaries)
• [tools-dynamic.md](tools-dynamic.md) (includes Intel Pin instruction-counting side channel for movfuscated binaries) - Dynamic analysis tools: Frida (hooking, anti-debug bypass, memory scanning, Android/iOS), angr symbolic execution (path exploration, constraints, CFG), lldb (macOS/LLVM debugger), x64dbg (Windows), Qiling (cross-platform emulation with OS support), Triton (dynamic symbolic execution)
• [tools-advanced.md](tools-advanced.md) - Advanced tools: VMProtect/Themida analysis, binary diffing (BinDiff, Diaphora), deobfuscation frameworks (D-810, GOOMBA, Miasm), Rizin/Cutter, RetDec, advanced GDB (Python scripting, conditional breakpoints, watchpoints, reverse debugging with rr, pwndbg/GEF), advanced Ghidra scripting, patching (Binary Ninja API, LIEF)
• [anti-analysis.md](anti-analysis.md) - Comprehensive anti-analysis: Linux anti-debug (ptrace, /proc, timing, signals, direct syscalls), Windows anti-debug (PEB, NtQueryInformationProcess, heap flags, TLS callbacks, HW/SW breakpoint detection, exception-based, thread hiding), anti-VM/sandbox (CPUID, MAC, timing, artifacts, resources), anti-DBI (Frida detection/bypass), code integrity/self-hashing, anti-disassembly (opaque predicates, junk bytes), MBA identification/simplification, bypass strategies
• [patterns.md](patterns.md) - Foundational binary patterns: custom VMs, anti-debugging, nanomites, self-modifying code, XOR ciphers, mixed-mode stagers, LLVM obfuscation, S-box/keystream, SECCOMP/BPF, exception handlers, memory dumps, byte-wise transforms, x86-64 gotchas, signal-based exploration, malware anti-analysis, multi-stage shellcode, timing side-channel, multi-thread anti-debug with decoy + signal handler MBA
• [patterns-ctf.md](patterns-ctf.md) - Competition-specific patterns (Part 1): hidden emulator opcodes, LD_PRELOAD key extraction, SPN static extraction, image XOR smoothness, byte-at-a-time cipher, mathematical convergence bitmap, Windows PE XOR bitmap OCR, two-stage RC4+VM loaders, GBA ROM meet-in-the-middle, Sprague-Grundy game theory, kernel module maze solving, multi-threaded VM channels, backdoored shared library detection via string diffing
• [patterns-ctf-2.md](patterns-ctf-2.md) - Competition-specific patterns (Part 2): multi-layer self-decrypting brute-force, embedded ZIP+XOR license, stack string deobfuscation, prefix hash brute-force, CVP/LLL lattice for integer validation, decision tree function obfuscation, GLSL shader VM, GF(2^8) Gaussian elimination, Z3 single-line Python circuit, sliding window popcount, keyboard LED Morse code via ioctl
• [languages.md](languages.md) - Language/platform-specific: Python bytecode & opcode remapping, Python version-specific bytecode, Pyarmor static unpack, DOS stubs, Unity IL2CPP, HarmonyOS HAP/ABC, Brainfuck/esolangs, UEFI, transpilation to C, code coverage side-channel, OPAL functional reversing, non-bijective substitution, Roblox place file analysis, Godot game asset extraction, Rust serde_json schema recovery, Verilog/hardware RE, Android JNI RegisterNatives, Ruby/Perl polyglot, Electron ASAR extraction + native binary analysis, Node.js npm runtime introspection
• [languages-compiled.md](languages-compiled.md) - Go binary reversing (GoReSym, goroutines, memory layout, channel ops, embed.FS), Rust binary reversing (demangling, Option/Result, Vec, panic strings), Swift binary reversing (demangling, protocol witness tables), Kotlin/JVM (coroutine state machines), C++ (vtable reconstruction, RTTI, STL patterns)
• [platforms.md](platforms.md) - Platform-specific RE: macOS/iOS (Mach-O, code signing, Objective-C runtime, Swift, dyld, jailbreak bypass), embedded/IoT firmware (binwalk, UART/JTAG/SPI extraction, ARM/MIPS, RTOS), kernel drivers (Linux .ko, eBPF, Windows .sys), game engines (Unreal Engine, Unity, anti-cheat, Lua), automotive CAN bus, RISC-V advanced

---

Problem-Solving Workflow

1. Start with strings extraction - many easy challenges have plaintext flags
2. Try ltrace/strace - dynamic analysis often reveals flags without reversing
3. Try Frida hooking - hook strcmp/memcmp to capture expected values without reversing
4. Try angr - symbolic execution solves many flag-checkers automatically
5. Try Qiling - emulate foreign-arch binaries or bypass heavy anti-debug without artifacts
6. Map control flow before modifying execution
7. Automate manual processes via scripting (r2pipe, Frida, angr, Python)
8. Validate assumptions by comparing decompiler outputs (dogbolt.org for side-by-side)

Quick Wins (Try First!)

# Plaintext flag extraction
strings binary | grep -E "flag\{|CTF\{|pico"
strings binary | grep -iE "flag|secret|password"
rabin2 -z binary | grep -i "flag"

# Dynamic analysis - often captures flag directly
ltrace ./binary
strace -f -s 500 ./binary

# Hex dump search
xxd binary | grep -i flag

# Run with test inputs
./binary AAAA
echo "test" | ./binary

Initial Analysis

file binary           # Type, architecture
checksec --file=binary # Security features (for pwn)
chmod +x binary       # Make executable

Memory Dumping Strategy

Key insight: Let the program compute the answer, then dump it. Break at final comparison (b *main+OFFSET), enter any input of correct length, then x/s $rsi to dump computed flag.

Decoy Flag Detection

Pattern: Multiple fake targets before real check.

Identification:

1. Look for multiple comparison targets in sequence
2. Check for different success messages
3. Trace which comparison is checked LAST

Solution: Set breakpoint at FINAL comparison, not earlier ones.

GDB PIE Debugging

PIE binaries randomize base address. Use relative breakpoints:

gdb ./binary
start                    # Forces PIE base resolution
b *main+0xca            # Relative to main
run

Comparison Direction (Critical!)

Two patterns:

1. transform(flag) == stored_target - Reverse the transform
2. transform(stored_target) == flag - Flag IS the transformed data!

Pattern 2 solution: Don't reverse - just apply transform to stored target.

Common Encryption Patterns

• XOR with single byte - try all 256 values
• XOR with known plaintext (flag{, CTF{)
• RC4 with hardcoded key
• Custom permutation + XOR
• XOR with position index (^ i or ^ (i & 0xff)) layered with a repeating key

Quick Tool Reference

# Radare2
r2 -d ./binary     # Debug mode
aaa                # Analyze
afl                # List functions
pdf @ main         # Disassemble main

# Ghidra (headless)
analyzeHeadless project/ tmp -import binary -postScript script.py

# IDA
ida64 binary       # Open in IDA64

Binary Types

Python .pyc

Disassemble with marshal.load() + dis.dis(). Header: 8 bytes (2.x), 12 (3.0-3.6), 16 (3.7+). See [languages.md](languages.md#python-bytecode-reversing-disdis-output).

WASM

wasm2c checker.wasm -o checker.c
gcc -O3 checker.c wasm-rt-impl.c -o checker

# WASM patching (game challenges):
wasm2wat main.wasm -o main.wat    # Binary → text
# Edit WAT: flip comparisons, change constants
wat2wasm main.wat -o patched.wasm # Text → binary

WASM game patching (Tac Tic Toe, Pragyan 2026): If proof generation is independent of move quality, patch minimax (flip i64.lt_s → i64.gt_s, change bestScore sign) to make AI play badly while proofs remain valid. Invoke /ctf-misc for full game patching patterns (games-and-vms).

Android APK

apktool d app.apk -o decoded/ for resources; jadx app.apk for Java decompilation. Check decoded/res/values/strings.xml for flags. See [tools.md](tools.md#android-apk).

Flutter APK (Dart AOT)

If lib/arm64-v8a/libapp.so + libflutter.so present, use Blutter: python3 blutter.py path/to/app/lib/arm64-v8a out_dir. Outputs reconstructed Dart symbols + Frida script. See [tools.md](tools.md#flutter-apk-blutter).

.NET

• dnSpy - debugging + decompilation
• ILSpy - decompiler

Packed (UPX)

upx -d packed -o unpacked

If unpacking fails, inspect UPX metadata first: verify UPX section names, header fields, and version markers are intact. If metadata looks tampered or uncertain, review UPX source on GitHub to identify likely modification points.

Tauri Packed Desktop Apps

Tauri embeds Brotli-compressed frontend assets in the executable. Find index.html xrefs to locate asset index table, dump blobs, Brotli decompress. Reference: tauri-codegen/src/embedded_assets.rs.

Anti-Debugging Bypass

Common checks:

• IsDebuggerPresent() / PEB.BeingDebugged / NtQueryInformationProcess (Windows)
• ptrace(PTRACE_TRACEME) / /proc/self/status TracerPid (Linux)
• TLS callbacks (run before main — check PE TLS Directory)
• Timing checks (rdtsc, clock_gettime, GetTickCount)
• Hardware breakpoint detection (DR0-DR3 via GetThreadContext)
• INT3 scanning / code self-hashing (CRC over .text section)
• Signal-based: SIGTRAP handler, SIGALRM timeout, SIGSEGV for real logic
• Frida/DBI detection: /proc/self/maps scan, port 27042, inline hook checks

Bypass: Set breakpoint at check, modify register to bypass conditional. pwntools patch: elf.asm(elf.symbols.ptrace, 'ret') to replace function with immediate return. See [patterns.md](patterns.md#pwntools-binary-patching-crypto-cat).

For comprehensive anti-analysis techniques and bypasses (30+ methods with code), see [anti-analysis.md](anti-analysis.md).

S-Box / Keystream Patterns

Xorshift32: Shifts 13, 17, 5 Xorshift64: Shifts 12, 25, 27 Magic constants: 0x2545f4914f6cdd1d, 0x9e3779b97f4a7c15

Custom VM Analysis

1. Identify structure: registers, memory, IP
2. Reverse executeIns for opcode meanings
3. Write disassembler mapping opcodes to mnemonics
4. Often easier to bruteforce than fully reverse
5. Look for the bytecode file loaded via command-line arg

See [patterns.md](patterns.md#custom-vm-reversing) for VM workflow, opcode tables, and state machine BFS.

Sequential key-chain brute-force: When a VM validates input in small blocks (e.g., 3 bytes = 2^24 candidates) with each block's output key feeding the next, brute-force each block sequentially with OpenMP parallelization. Compile solver with gcc -O3 -march=native -fopenmp. See [patterns-ctf-2.md](patterns-ctf-2.md#vm-sequential-key-chain-brute-force-midnight-flag-2026).

Python Bytecode Reversing

XOR flag checkers with interleaved even/odd tables are common. See [languages.md](languages.md#python-bytecode-reversing-disdis-output) for bytecode analysis tips and reversing patterns.

Signal-Based Binary Exploration

Binary uses UNIX signals as binary tree navigation; hook sigaction via LD_PRELOAD, DFS by sending signals. See [patterns.md](patterns.md#signal-based-binary-exploration).

Malware Anti-Analysis Bypass via Patching

Flip JNZ/JZ (0x75/0x74), change sleep values, patch environment checks in Ghidra (Ctrl+Shift+G). See [patterns.md](patterns.md#malware-anti-analysis-bypass-via-patching).

Expected Values Tables

Locating:

objdump -s -j .rodata binary | less
# Look near comparison instructions
# Size matches flag length

x86-64 Gotchas

Sign extension and 32-bit truncation pitfalls. See [patterns.md](patterns.md#x86-64-gotchas) for details and code examples.

Iterative Solver Pattern

Try each byte (0-255) per position, match against expected output. Uniform transform shortcut: if one input byte only changes one output byte, build 0..255 mapping then invert. See [patterns.md](patterns.md) for full implementation.

Unicorn Emulation (Complex State)

from unicorn import * -- map segments, set up stack, hook to trace. Mixed-mode pitfall: 64-bit stub jumping to 32-bit via retf requires switching to UCMODE32 and copying GPRs + EFLAGS + XMM regs. See [tools.md](tools.md#unicorn-emulation).

Multi-Stage Shellcode Loaders

Nested shellcode with XOR decode loops; break at call rax, bypass ptrace with set $rax=0, extract flag from mov instructions. See [patterns.md](patterns.md#multi-stage-shellcode-loaders).

Timing Side-Channel Attack

Validation time varies per correct character; measure elapsed time per candidate to recover flag byte-by-byte. See [patterns.md](patterns.md#timing-side-channel-attack).

Godot Game Asset Extraction

Use KeyDot to extract encryption key from executable, then gdsdecomp to extract .pck package. See [languages.md](languages.md#godot-game-asset-extraction).

Roblox Place File Analysis

Query Asset Delivery API for version history; parse .rbxlbin chunks (INST/PROP/PRNT) to diff script sources across versions. See [languages.md](languages.md#roblox-place-file-analysis).

Unstripped Binary Information Leaks

Pattern (Bad Opsec): Debug info and file paths leak author identity.

Quick checks:

strings binary | grep "/home/"    # Home directory paths
strings binary | grep "/Users/"   # macOS paths
file binary                       # Check if stripped
readelf -S binary | grep debug    # Debug sections present?

Custom Mangle Function Reversing

Binary mangles input 2 bytes at a time with running state; extract target from .rodata, write inverse function. See [patterns.md](patterns.md#custom-mangle-function-reversing).

Rust serde_json Schema Recovery

Disassemble serde Visitor implementations to recover expected JSON schema; field names in order reveal flag. See [languages.md](languages.md#rust-serde_json-schema-recovery).

Position-Based Transformation Reversing

Binary adds/subtracts position index; reverse by undoing per-index offset. See [patterns.md](patterns.md#position-based-transformation-reversing).

Hex-Encoded String Comparison

Input converted to hex, compared against constant. Decode with xxd -r -p. See [patterns.md](patterns.md#hex-encoded-string-comparison).

Embedded ZIP + XOR License Decryption

Binary with named symbols (EMBEDDED_ZIP, ENCRYPTED_MESSAGE) in .rodata → extract ZIP containing license, XOR encrypted message with license bytes to recover flag. No execution needed. See [patterns-ctf-2.md](patterns-ctf-2.md#embedded-zip--xor-license-decryption-metactf-2026).

Stack String Deobfuscation (.rodata XOR Blob)

Binary mmaps .rodata blob, XOR-deobfuscates, uses it to validate input. Reimplement verification loop with pyelftools to extract blob. Look for 0x9E3779B9, 0x85EBCA6B constants and rol32(). See [patterns-ctf-2.md](patterns-ctf-2.md#stack-string-deobfuscation-from-rodata-xor-blob-nullcon-2026).

Prefix Hash Brute-Force

Binary hashes every prefix independently. Recover one character at a time by matching prefix hashes. See [patterns-ctf-2.md](patterns-ctf-2.md#prefix-hash-brute-force-nullcon-2026).

Mathematical Convergence Bitmap

Pattern: Binary classifies coordinate pairs by Newton's method convergence (e.g., z^3-1=0). Grid of pass/fail results renders ASCII art flag. Key: the binary is a classifier, not a checker — reverse the math and visualize. See [patterns-ctf.md](patterns-ctf.md#mathematical-convergence-bitmap-ehax-2026).

RISC-V Binary Analysis

Statically linked, stripped RISC-V ELF. Use Capstone with CS_MODE_RISCVC | CS_MODE_RISCV64 for mixed compressed instructions. Emulate with qemu-riscv64. Watch for fake flags and XOR decryption with incremental keys. See [tools.md](tools.md#risc-v-binary-analysis-ehax-2026).

Sprague-Grundy Game Theory Binary

Game binary plays bounded Nim with PRNG for losing-position moves. Identify game framework (Grundy values = pile % (k+1), XOR determines position), track PRNG state evolution through user input feedback. See

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: dailyyarn
• Source: dailyyarn/CTF-agent
• License: MIT

Install and usage instructions live in the source repository linked above.