Book summary “Reversing”: summary

Introduction

You often face situations where high-level source code is entirely missing, leaving you with only compiled binary files. Understanding how these programs function, identifying undocumented interfaces, or neutralizing malicious code requires you to master low-level analysis. Without access to original developers or documentation, you must rely on assembly language, operating system internals, and specialized debugging tools to reveal internal application logic.

Software reverse engineering enables you to decode binary applications, uncover undocumented interfaces, audit security vulnerabilities, and analyze malware using low-level code analysis and debugging tools.

This book provides a comprehensive guide to mastering binary analysis on the Windows platform. You will learn how compilers transform source code into machine instructions, how operating system structures govern execution, and how to operate professional disassemblers and debuggers. The text explores advanced techniques for auditing file formats, defeating copy protection mechanisms, and neutralizing sophisticated malware.

• How do compilers translate high-level control structures into assembly instructions?
• What mechanisms do debuggers use to monitor and modify running processes?
• How can you locate and exploit security vulnerabilities in compiled binaries?
• What techniques allow you to bypass code obfuscation and anti-debugging tricks?

1. Low-Level Software and Compilation

Compilers translate high-level language constructs into optimized machine code, transforming clear program boundaries into complex assembly sequences. You must understand how source code structures map to low-level instructions to successfully analyze binary files.

Compilers translate high-level language constructs into optimized machine code, transforming clear program boundaries into complex assembly sequences.

Assembly Language and Compiler Optimizations

The CMP instruction operates by subtracting the second operand from the first and discarding the result while setting processor flags. Reversers must understand that flags are read differently based on whether operands are signed or unsigned.

Compilers often reverse conditional checks in single-branch statements to structure binary code efficiently in memory. Two-way conditionals like if-else statements are identified in assembly by a reversed condition pointing to an else block and an unconditional jump ending the primary block.

Compound conditionals use logical AND and OR operators, which compilers implement using distinct patterns of consecutive conditional jumps. Compilers implement n-way conditionals or switch blocks using either pointer tables or binary tree search strategies.

Compilers optimize loops by converting them to posttested forms to eliminate jumps, and sometimes apply loop unrolling to reduce overhead. For instance, when loops are unrolled to execute 0x3e7 times, they will only be executed 0x14d times.

Optimizing compilers can eliminate conditional jumps entirely by using pure arithmetic sequences involving instructions like NEG and SBB. Working-set tuning reorganizes code layout in executables to reduce memory consumption and improve startup speed by grouping popular functions.

Data Structures and Memory Management

IA-32 processors use status flags stored in the EFLAGS register to reflect the results of arithmetic operations. Compilers handle signed and unsigned overflows by utilizing two separate flags instead of distinct instruction versions for addition and subtraction.

Adding 4390 and 29184 results in 0x8326, which sets the signed overflow flag OF while clearing the unsigned carry flag CF.

Compilers sometimes use the LEA instruction for quick additions and shifts because its execution unit is separate from standard arithmetic logic units. Compilers frequently replace multiplication instructions with combinations of binary shifts, additions, and LEA operations when the multiplier is a known constant. Multiplying by 32 can be compiled into one LEA and four ADD instructions instead of a single SHL because the combined low-latency instructions execute faster on certain processors.

Compilers implement division by constant divisors using reciprocal multiplication combined with fixed-point arithmetic and shifts to bypass slow division instructions. Multiplying a variable by 0xAAAAAAAB and shifting right by 2 effectively divides the number by 6.

A stack frame allocates specific memory regions for function parameters, return addresses, and local variables, managed through base and stack pointers. Calling conventions define how parameters and stack cleanup are handled during function calls, providing key signatures for reverse engineering.

Global variables reside in fixed addresses inside an executable module's data section, making them easy to identify via hard-coded memory references. Compilers manage local variables by either allocating them on the stack relative to EBP or optimizing them into processor registers for performance.

Compilers typically align data structure members to the processor's native word-size boundaries to prevent performance penalties from unaligned memory access. Arrays store sequential items without alignment spacing, making their low-level access distinct through register-multiplied base addresses.

Linked lists scatter items in memory and connect them using next and previous pointers, avoiding memory copying during insertions and deletions. Compilers implement polymorphism and virtual functions using virtual function tables placed in read-only memory sections.

2. Windows Fundamentals and Reversing Tools

Effective live and offline analysis requires mastery of operating system internals, executable formats, system calls, and professional disassemblers and debuggers.

Windows NT was designed as a pure 32-bit, virtual-memory capable, multithreaded and multiprocessor operating system utilizing a Hardware Abstraction Layer. The architecture separates physical hardware access through a special Hardware Abstraction Layer, supports preemptive multithreading without 16-bit GUI bottlenecks, and implements security via Access Control Lists on every system object and file. The Windows memory manager uses page tables to establish virtual memory, creating isolated address spaces that sandbox running processes. Processors divide memory into fixed-size pages and consult page tables to map virtual addresses to physical memory. This mechanism allows the operating system to enforce security flags, separating privileged kernel memory from nonprivileged user memory.

The 32-bit Windows memory address space is typically split into a 2GB user memory portion and a 2GB shared kernel memory portion. The upper 2GB contains all kernel-related memory and is shared among all address spaces, remaining protected from user-mode access. It includes paged and nonpaged pools, system cache, session space, and system page-table entries. Applications access kernel objects indirectly through process-specific numerical handles that index into private handle tables. The Windows kernel manages objects using a centralized object manager and generic headers. Applications cannot use direct pointers and instead use handles containing an access mask that determines allowed operations.

Operating System Architecture and Executable Formats

A process provides an isolated memory address space, while a thread acts as a primitive code execution unit possessing its own context and dual stacks. Threads alternate between user-mode and kernel-mode execution, requiring separate user-mode and kernel-mode stacks for security. The scheduler and dispatcher manage thread quantums and preemptive context switching to achieve concurrency.

When you analyze process creation, you encounter a precise initialization sequence:

• The creation of the process object and new address space is the first step: When a process calls the Win32 API CreateProcess, the API creates a process object and allocates a new memory address space for the process.
• CreateProcess maps NTDLL.DLL and the program executable into the newly created address space.
• CreateProcess creates the process's first thread and allocates stack space for it.
• The process's first thread is resumed and starts running in the LdrpInitialize function inside NTDLL.DLL.
• LdrpInitialize recursively traverses the primary executable's import tables and maps into memory every executable that is required for running the primary executable.
• At this point control is passed into LdrpRunInitializeRoutines, which is an internal NTDLL.DLL routine responsible for initializing all statically linked DLLs currently loaded into the address space. The initialization process consists of calling each DLL's entry point with the DLL_PROCESS_ATTACH constant.
• Once all DLLs are initialized, LdrpInitialize calls the thread's real initialization routine, which is the BaseProcessStart function from KERNEL32.DLL. This function in turn calls the executable's WinMain entry point, at which point the process has completed its initialization sequence.

Windows provides application programming interfaces through the high-level Win32 API and the direct, lower-level Native API. The Win32 API consists of Kernel, GDI, and USER components. The Native API serves as the direct interface to the NT kernel, exported from NTDLL.DLL and NTOSKRNL.EXE with Nt or Zw prefixes, bypassing user-mode validation when called from kernel mode. User-mode code transitions to kernel mode to execute operating system functions using specialized CPU instructions or interrupts. Older Windows systems invoked interrupt 2E to reach the KiSystemService dispatcher, while modern versions use the high-performance SYSENTER instruction via a SharedUserData stub to switch execution modes.

Windows executable files follow the Portable Executable format, utilizing relocatable sections, import/export tables, and relative virtual addresses. Executables are divided into sections with specific access attributes and aligned for files and memory. Dynamic linking is resolved at load time through the Import Address Table and DataDirectory structures. Windows provides structured exception handling to distribute hardware and software exceptions to registered thread handlers. Threads maintain an exception handler list referenced via the Thread Information Block through the FS segment register. When an exception occurs, the system traverses the registration records to locate a suitable handler.

Disassemblers, Debuggers, and Monitoring Utilities

Reversing methodologies are split into offline analysis, which translates binaries into human-readable code statically, and live analysis, which observes program execution dynamically in a debugger.

Offline analysis uses disassemblers or decompilers to convert binaries into readable text, providing a macro outline and making it easier to search for specific functions, but requiring guesswork regarding data flow. Live analysis runs the program in a debugger to monitor internal variables and how data affects code flow, making it superior for beginners and necessary for packed or encrypted code. Disassemblers decode binary machine code into readable assembly language, and they must employ recursive traversal algorithms rather than linear sweeps to accurately distinguish code from data. Compilers often insert data chunks like jump tables into code sections for performance reasons. A linear sweep goes instruction by instruction and can be confused by embedded data, whereas recursive traversal follows code flow so that addresses are disassembled only if reachable from previously disassembled instructions.

IDA Pro is an advanced, highly flexible commercial disassembler and debugger supporting multiple architectures and executable formats while generating complex flowcharts and call graphs. IDA Pro supports architectures like IA-32, IA-64, and AMD64, and file formats such as PE, ELF, and Xbox XBE files. It provides detailed disassemblies, operand highlighting, intrafunction flowcharts, and call graphs illustrating internal subroutines and external API usage. Reversing debuggers require robust disassemblers, software and hardware breakpoints, register and memory views, and detailed process information to analyze code behavior dynamically. Software breakpoints pause execution by inserting instructions into code at runtime, while hardware breakpoints use special CPU features to pause when a specific memory address is accessed, simplifying the mapping of data structures. Debuggers are divided into user-mode debuggers for single application control and kernel-mode debuggers for total system observation.

OllyDbg is a popular, free user-mode debugger built specifically for reversing, featuring a powerful code analyzer, assembly and patching engine, and extensive module views. Written by Oleh Yuschuk, OllyDbg excels in disassembler capabilities, identifying loops and switch blocks, showing API parameter names, and enabling cross-reference searches. It includes a built-in assembler and patching engine that allows users to overwrite instructions and commit changes or store patch lists. WinDbg is a free Microsoft debugger offering deep operating system integration and kernel-mode debugging capabilities via remote connection, though it features an inconvenient command-line interface. WinDbg surpasses user-mode alternatives in exposing internal system data structures such as user-mode heaps, security tokens, the Process Environment Block, and Thread Environment Block.

NuMega SoftICE is a local kernel-mode debugger that hooks the keyboard driver to instantly freeze the operating system and display a dedicated debugging window upon a hotkey press. SoftICE operates as a Windows kernel-mode driver that intercepts keystrokes, disables interrupts, and takes control of all processors. This allows users to examine system state locally, though it can occasionally destabilize the operating system and cause blue screens. Virtual machines emulate full PC hardware via software, providing isolated environments that safely host kernel debugging without risking the stability of the host computer. Virtual machines allow users to manage hard drives as single files, revert system states easily, and use nonpersistent drives to discard malware modifications upon shutdown.

System-monitoring tools observe I/O channels between applications and the operating system by hooking low-level components, enabling analysis without inspecting binary code. Created by Mark Russinovich and Bryce Cogswell, Sysinternals utilities monitor specific system aspects across processes.

Essential monitoring tools track distinct system operations:

• FileMon tracks file system traffic;
• TCPView and TDIMon monitor network connections and socket operations;
• RegMon tracks registry activity;
• PortMon monitors serial and parallel I/O;
• WinObj lists named objects;
• Process Explorer serves as an advanced replacement for Task Manager.

Executable-dumping tools analyze Portable Executable headers, import and export directories, relocation tables, and sections to help reverser understand binary structure and dependencies. Tools such as Microsoft's console-mode DUMPBIN, freeware GUI-based PEView, and PEBrowse Professional extract structural header information from PE files. Familiarity with PE header structures is necessary to interpret the output of these dumping utilities.

3. Applied Reversing, File Formats, and Auditing

Twenty years ago programs existed in isolation communicating directly with underlying hardware. Software engineers frequently need to interact with third-party libraries and operating systems without having access to source code. Vendor-supplied documentation is often deficient or omits critical parameters and features, making direct code analysis via disassemblers faster and more reliable than customer support.

Reverse engineering serves as an effective tool for achieving interoperability when vendor documentation is missing, incomplete, or unavailable.

The Windows operating system historically contained hundreds of undocumented APIs, such as those in NTDLL.DLL, which Microsoft sometimes kept undocumented to maintain compatibility or provide competitive advantages. You can discover undocumented APIs by dumping export directories of key executable modules and searching for relevant keywords. The author scanned the export directory of NTDLL.DLL and filtered for functions containing the term GenericTable and starting with the prefix Rtl.

Analyzing function endings and stack cleanup instructions reveals the calling convention and parameter count of disassembled functions. By inspecting the numeric operand of the RET instruction, you can determine how many bytes of stack the function unwinds, which confirms if it uses stdcall instead of cdecl. Dividing the total stack byte count by four in a 32-bit environment yields the total number of parameters. The instruction RET 14 at the end of RtlInitializeGenericTable indicates a stdcall convention with twenty decimal bytes of stack cleanup, translating to five parameters.

Disassembling initialization routines enables you to map out unknown internal data structures and their member offsets step by step. Analyzing RtlInitializeGenericTable revealed a forty-byte root table structure containing ten members, including element counts and callback pointers.

Deciphering File Formats

Data reverse engineering allows analysts to understand undocumented file formats or network protocols in order to achieve interoperability with third-party software. Many commercial products use proprietary, undocumented data formats or networking protocols that prevent communication with competing programs. Using data reverse engineering techniques, you can uncover missing format information and write code that reads or generates compatible data. Microsoft Word uses the undocumented .doc file format, which third-party programs had to reverse engineer to open and create compatible files.

Reversing an unknown file format begins by creating small files with known values, inspecting file headers via hex dumps, and listing imported functions to identify system and cryptographic services.

Cryptex is a command-line archive tool that encrypts multiple files into a .crx archive using the 3DES encryption algorithm derived from a textual password. Cryptex uses 3DES, which applies the original 56-bit DES algorithm three times using three different keys, effectively using a 168-bit key. The author created a command-line tool named Cryptex to demonstrate data reverse-engineering and security evaluation techniques. The author examined the import list of Cryptex.exe using DUMPBIN to identify KERNEL32, ADVAPI32, and MSVCR71 library calls.

When a user provides a password, Cryptex hashes it using the CALG_SHA algorithm to derive the 3DES encryption key. It then hashes that SHA result again using the CALG_MD5 algorithm and compares the 16-byte output with the checksum stored in the archive header. The author used OllyDbg to place breakpoints on printf and trace back from the invalid password error message to locate the header-verification function. Each cluster is 4,104 bytes long, with 4,092 bytes of actual content and a 4-byte header pointing to the next cluster index.

You can dump the file list inside a Cryptex archive by following precise execution steps:

• Initialize the Crypto API and open the archive file.
• Verify the 8-byte header signature.
• Calculate an SHA hash out of the typed password, and calculate an MD5 hash out of that.
• Verify that the calculated MD5 hash matches the stored MD5 hash from the Cryptex header.
• Produce a 3DES key using the SHA hash object.
• Read the first file list cluster in the same manner as it is read in Cryptex.
• Loop through those 152-bytes long entries and dump each entry's name if its offset +8 is nonzero.
• Proceed to read and decrypt additional file-list clusters if they are present and list any entries within those clusters.

Auditing Program Binaries

Computer security fundamentally means maintaining control over the flow of information on a system, ensuring that data stays secure and malicious code cannot enter. While encryption and user access rights matter, security in binary auditing primarily concerns ensuring programs do not inadvertently allow external data to execute malicious payloads or expose local data.

Stack overflows occur when programs copy external data into fixed-size stack buffers without performing proper bounds checking, which can overwrite local variables and the function return address.

Because stack variables have predetermined offsets, passing oversized input overwrites adjacent memory. Attackers exploit this by overwriting the function's return address with a pointer to their own shellcode. Consider a function using a local eight-character string buffer; copying unverified user data longer than eight characters overwrites adjacent variables and the return address.

Compiler-generated stack checking inserts a randomized cookie between local variables and the return address to detect and halt buffer overflows before they can execute malicious code. A pseudorandom security cookie is initialized when a module loads and verified before a function returns. If an overflow corrupts the cookie, the program terminates immediately via a security check failure handler. The Microsoft C/C++ compiler implements this feature using the /GS option, utilizing __security_cookie and __security_check_cookie.

Integer overflows happen when numerical bounds checks fail due to incorrect handling of signed types or arithmetic wrapping on user-supplied buffer lengths. Signed integer comparisons can be bypassed because negative values are treated as massive numbers. Furthermore, adding constants like header lengths to large user-supplied integers can wrap around, yielding a small allocation size while the copy routine still attempts to copy massive blocks.

The June 2001 Microsoft Internet Information Services vulnerability combined a length check confusing bytes with Unicode characters, enabling stack overflows exploited by the Code Red Worm. In idq.dll, the function CVariableSet::AddExtensionControlBlock compared URL query string lengths in bytes against a 400-byte limit, but DecodeURLEscapes decoded the input into a 200-character Unicode buffer. This mismatch allowed attackers to overwrite stack space and exploit exception-handling pointers. The Code Red Worm exploited this vulnerability in IIS versions 4 and 5, replacing web pages with the message HELLO! Welcome to http://www.worm.com ! Hacked By Chinese!

4. Malware, Piracy, Protection, and .NET

Analyzing advanced malware and copy protection schemes requires defeating code obfuscation, unpacking routines, active anti-debugging techniques, and managed code structures.

Reversing Malware

Malware is categorized into distinct types based on how it spreads and operates, including viruses, worms, Trojan horses, backdoors, mobile code, and adware or spyware. Sticky software employs aggressive self-preservation techniques to prevent users from manually uninstalling or removing it from the infected system.

A Trojan disguised as a screen saver file named Webcam Shots.scr uses an .scr extension to trick users into thinking it is a harmless image. It installed itself as ZoneLockup.exe in the SYSTEM32 folder.

Analyzing live malware requires a strictly isolated environment to prevent unintended network propagation or system infection. You must follow specific containment practices:

• Allocate a dedicated machine not connected to any local network or the internet.
• Alternatively, use a virtual machine detached entirely from the host and internet.
• Transfer executables using a recordable CD or DVD.
• Store the malicious program under a non-executable extension to prevent accidental launching.

Malware authors frequently use executable packers to compress binaries, hide imports, and reduce file sizes. You can unpack and examine these files using standard utilities and routines:

• Run the binary through DUMPBIN to inspect headers and import entries.
• Identify packer sections such as UPX0 and UPX1.
• Run UPX with the -d switch to permanently decompress the executable for analysis.

The Backdoor.Hacarmy.D sample was packed with UPX, reducing its file size from 27,680 bytes to 18,976 bytes. The backdoor communicates with its creator by connecting to an Internet Relay Chat server, authenticating with a password, and processing remote commands. You can interact with this communication channel using specific protocol steps:

• Join the designated IRC channel ##g##.
• Send the password string !morris tounge to register as an authorized controller.
• Issue remote commands starting with an exclamation mark, such as !info, !socks4, or !execute.

Piracy, Copy Protection, and Antireversing

Modern personal computers feature an open architecture where CPUs execute code without checking user rights, making it impossible to create a completely uncrackable software-based copy protection. A study by the Business Software Alliance and IDC estimated that over $30 billion worth of software was illegally installed worldwide in 2003, accounting for 36 percent of all installed software. Lowering piracy by 10 percentage points over four years adds more than 1 million new jobs and $400 billion in economic growth worldwide.

Crypto-processors integrate direct decryption hardware onto the motherboard or CPU to execute encrypted code safely without exposing plaintext to attackers. You can trace this secure hardware execution through a multi-step platform sequence:

1. Each individual processor is assigned a pair of encryption keys and a serial number as part of the manufacturing process.
2. When an end user purchases a program, the software developer requests the user's processor serial number, and then contacts the authority to obtain the public key for that serial number.
3. The program binaries are encrypted using the public key and shipped or transmitted to the end user.
4. The end user runs the encrypted program, and the crypto-processor decrypts the code using the internally stored decryption key and stores the decrypted code in a special memory region that is not software-accessible.
5. Code is executed directly from this theoretically inaccessible memory.

Programs can actively detect and interfere with live debuggers using system APIs, exception handling, or processor flags. You can implement checks against live debugging tools directly inside your code:

• Retrieve thread environment block offset +30 to point to process PEB.
• Read byte at offset +2 to check if a debugger is present.
• Conditionally terminate the program if a debugger is detected.

Simple copy protection schemes based on conditional jumps can be defeated by patching the binary to bypass the authorization check. You can crack these protections inside a debugger by executing precise workflow steps:

• Open the target executable in a debugger like OllyDbg.
• Locate imported APIs such as USER32.MessageBoxA and find references to code calls.
• Examine the conditional jump instructions controlling the authorization outcome.
• Use the Assemble dialog to replace unwanted checks or jumps with NOP instructions.
• Copy modifications to the executable and save the patched file.

Cracking the KeygenMe-3 program by Bengaly by replacing a JNZ instruction with NOPs so it always triggers the success message box.

Reversing .NET and Decompilation Theory

Reversing .NET applications is significantly easier than native code because binaries contain detailed intermediate language and full metadata. Managed code runs inside the Common Language Runtime, which validates operations and enforces a Common Type System separating values and references. Decompilers use flexible intermediate representations with tiny instruction sets and expression trees to model everything from low-level assembly to high-level expressions.

Native IA-32 decompilation is a developing technology that provides useful code approximations but requires human editing and realistic expectations.

10 Key Ideas

1. Compilers translate high-level language constructs into optimized machine code, transforming clear program boundaries into complex assembly sequences.
2. Effective live and offline analysis requires mastery of operating system internals, executable formats, system calls, and professional disassemblers and debuggers.
3. Reversing methodologies are split into offline analysis, which translates binaries into human-readable code statically, and live analysis, which observes program execution dynamically in a debugger.
4. Reverse engineering serves as an effective tool for achieving interoperability when vendor documentation is missing, incomplete, or unavailable.
5. Reversing an unknown file format begins by creating small files with known values, inspecting file headers via hex dumps, and listing imported functions to identify system and cryptographic services.
6. Stack overflows occur when programs copy external data into fixed-size stack buffers without performing proper bounds checking, which can overwrite local variables and the function return address.
7. Analyzing advanced malware and copy protection schemes requires defeating code obfuscation, unpacking routines, active anti-debugging techniques, and managed code structures.
8. Programs can actively detect and interfere with live debuggers using system APIs, exception handling, or processor flags, which reversers must neutralize to analyze binary behavior.
9. Reversing .NET applications is significantly easier than native code because binaries contain detailed intermediate language and full metadata.
10. Native IA-32 decompilation is a developing technology that provides useful code approximations but requires human editing and realistic expectations.