Implementing
Reverse
Engineering

The Real Practice of x86 Internals, Code Calling
Conventions, Ransomware Decryption, Application
Cracking, Assembly Language, and Proven
Cybersecurity Open Source Tools

Jitender Narula

www.bpbonline.com

FIRST EDITION 2022

Copyright © BPB Publications, India

ISBN: 978-93-91030-377

All Rights Reserved. No part of this publication may be reproduced, distributed or transmitted in any form or by any means or stored in a database or retrieval system, without the prior written permission of the publisher with the exception to the program listings which may be entered, stored and executed in a computer system, but they can not be reproduced by the means of publication, photocopy, recording, or by any electronic and mechanical means.

LIMITS OF LIABILITY AND DISCLAIMER OF WARRANTY

The information contained in this book is true to correct and the best of author’s and publisher’s knowledge. The author has made every effort to ensure the accuracy of these publications, but publisher cannot be held responsible for any loss or damage arising from any information in this book.

All trademarks referred to in the book are acknowledged as properties of their respective owners but BPB Publications cannot guarantee the accuracy of this information.

www.bpbonline.com

Dedicated to

My parents

Always seen God in them

About the Author

Jitender Narula is an experienced Cyber Security Specialist currently associated with International Institute of Cyber Security (www.iicybersecurity.com) having over 18 years of industry experience. He has many years of cyber security experience with governments and corporate world. In India, he has worked with government entities like Delhi Police, ICAI (Institute of Chartered Accountants of India), Delhi University and other private organizations.

He has worked on the projects of AT&T, Citrix, Google, Conexant, IPolicy Networks (Tech Mahindra now), Narus (a wholly owned subsidiary of The Boeing Company) and HFCL. Apart from this, he has conducted training programs for various corporate and government officials in India and Mexico.

He has published articles, research information and interviews in the area of cyber security for Information Security Newspaper (www.securitynewspaper.com), Noticias de seguridad informática (noticiasseguridad.com), Cibertip - Noticias de Hacking (www.cibertip.com), Exploit One (www.exploitone.com), iicybersecurity blog (https://www.iicybersecurity.com/blog-eng.html) and also contributed to the Vishvas News, which is part of Dainik Jagran (Indian language daily newspaper).

About the Reviewer

Sanil Nadkarni is CISO /CRO/DPO in an MNC. He has worked for organizations and clients such as Microsoft, Symantec, MphasiS, Capita, SLK and has supported International Banks and Financial Organizations.

He is an international speaker, author and trainer. Sanil has published over 100 articles online and also for various magazines.

His certifications include CFE, CISSP, CISA, CISM, Security+, ISO 27001 LA, ISO 22301 LA, A+, N+, MCP, MCSA, CCNA, SCSA, ISO 10012 (GDPR), RHCE, CEH,CBCP, ISO 31000 Risk and Microsoft Azure Architect.

Sanil has received honors and awards such as CISO TOP 100 Award, Dynamic CISO Award, Top 100 CIO award and many more.

He has published a book on Fundamentals on Information Security which is available on Amazon.

Acknowledgement

First and foremost, praises and thanks to my Dad, Mom and God for showering blessings throughout my work to complete the book successfully.

I would like to express my deep and sincere gratitude to Atul Narula, my colleague at International Institute of Cyber Security, Mexico for helping with the review process of this book. I am extremely grateful for what he has offered me. I would also like to thank him for his friendship, empathy and great sense of humor.

I am extremely grateful to my parents (Ramesh Narula and Mohini Narula) for their love, prayers, care and sacrifices for educating and preparing me for my future. I am very much thankful to my wife and son for their love, understanding, prayers and continuing support to complete this book. Also, I would like to express my thanks to Dr. Shilpi Sahi and Om Narula for their support and motivation throughout this process of writing. Once again, I would like to thank my family for putting up with me while I was spending many nights writing—I could have never completed this book without their support.

Finally, I would like to thanks BPB Publications for giving me the opportunity to write my first book for them.

Preface

Reverse Engineering (RE) is an art of understanding any program code when no source code is available. This book provides step-by-step explanation of the essential concepts and practical examples to understand and implement Reverse Engineering. It will enable the readers to understand the application code flow to identify vulnerabilities and bugs in the application.

This book is for the readers who want to start learning Reverse Engineering from basics in a step-by-step manner. The book is divided into three parts:

Exploring Reverse Engineering

Reverse Engineering Applications

Real World Examples with Solutions

The first part Exploring Reverse Engineering starts with the basic concepts of Computing System and Data Building Blocks of the Computing System. This part also enlists open-source tools required for RE applications and the programming instructions of RE. The second part Reverse Engineering Applications walks us through the different applications/programs to understand the implementation of RE. This part covers various practicals, which give the users a hands-on experience. All the applications/programs mentioned in this part are aligned in a systematic manner; from reverse engineering of basic C/C++ programs to complex C/C++ programs. In the third part Real World Examples and Solutions of this book, RE of well-known Windows application along with different exercises are demonstrated in a step-by-step manner. Over the 18 chapters in this book, you will learn the following:

PART 1: Exploring Reverse Engineering

In this part, the readers will understand the impact of RE on industry, building blocks of x86 computing system and the role of each in the overall functioning of x86 system.

Chapter 1 talks about the impact of RE on IT industry and how it originated as an area.

Chapter 2 talks about the building blocks of a computing system and the role of each building block in the overall functioning of the system. This chapter is important in order to understand the core concept behind the working of x86 computing systems.

Chapter 3 focuses on the open-source tools used in RE and how these tools are used for debugging and analysis. These tools will be used in all illustrations shown in this book.

Chapter 4 explains about the major assembly instructions used and also about how different instructions are segmented in various sections for easy understanding along with examples.

Chapter 5 helps us understand the different calling conventions along with practical illustrations.

PART 2: Reverse Engineering Applications

This is where the strategic way of learning RE applications/programs is explained with different illustrations. Every case is the outcome of research explained in a very simplified and step-by-step manner.

Chapter 6 gives a step-by-step understanding of the assembly code generated from basic C/C++ program.

Chapter 7 provides a step-by-step understanding of the assembly code generated from printf() function in C/C++ program.

Chapter 8 gives a step-by-step understanding of the assembly code generated from pointers in C/C++ program.

Chapter 9 provides a step-by-step understanding of the assembly code generated from decision control structure in C/C++ program.

Chapter 10 gives a step-by-step understanding of the assembly code generated from loop control structure in C/C++ program.

PART 3: Real World Examples and Solutions

In this part, understanding of whatever learned in the previous chapters is explained with real world exercises with solutions and also reversing of Windows well-known application is demonstrated.

Chapter 11 covers RE exercise of an array code along with the solution used in the RE process.

Chapter 12 covers RE exercise of a structure code along with the solution used in the RE process.

Chapter 13 explains RE exercise of a Scanf program along with the solution used in the RE process.

Chapter 14 explains RE exercise of a strcpy program along with the solution used in the RE process.

Chapter 15 covers RE exercise of a simple interest code along with the solution used in the RE process.

Chapter 16 explains RE exercise of breaking Wannacry ransomware with Ghidra.

Chapter 17 covers RE exercise using Cutter tool.

Chapter 18 demonstrates the process of RE of Windows Calculator in a step-by-step manner.

This book is to educate the learners on the topic of Reverse Engineering on x86 platform. This will be a good book for beginners and computer graduates in the area of RE. Professionals who want to switch their career to RE can also use this book. Other readers can be from schools, universities or those who are passionate to get into the area of cyber security.

Downloading the code
bundle and coloured images:

Please follow the link to download the
Code Bundle and the Coloured Images of the book:

https://rebrand.ly/b2fe9c

Errata

We take immense pride in our work at BPB Publications and follow best practices to ensure the accuracy of our content to provide with an indulging reading experience to our subscribers. Our readers are our mirrors, and we use their inputs to reflect and improve upon human errors, if any, that may have occurred during the publishing processes involved. To let us maintain the quality and help us reach out to any readers who might be having difficulties due to any unforeseen errors, please write to us at :

errata@bpbonline.com

Your support, suggestions and feedbacks are highly appreciated by the BPB Publications’ Family.

Did you know that BPB offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.bpbonline.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at business@bpbonline.com for more details.

At www.bpbonline.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on BPB books and eBooks.

BPB is searching for authors like you

If you're interested in becoming an author for BPB, please visit www.bpbonline.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

The code bundle for the book is also hosted on GitHub at https://github.com/bpbpublications/Implementing-Reverse-Engineering. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/bpbpublications. Check them out!

PIRACY

If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at business@bpbonline.com with a link to the material.

If you are interested in becoming an author

If there is a topic that you have expertise in, and you are interested in either writing or contributing to a book, please visit www.bpbonline.com.

REVIEWS

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at BPB can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about BPB, please visit www.bpbonline.com.

CHAPTER 1

Impact of Reverse Engineering

Before we start on the implementation of reverse engineering, it will be interesting to understand what reverse engineering really is, how it came into existence, and how it is beneficial in the modern era. Reverse engineering, as the name suggests, is a combination of two words: Reverse and Engineering. Engineering is the science of designing and building something beneficial for the human race. Engineering has provided us with both advantages and disadvantages. Engineering equipped us with the knowledge and means to build essential things for the human race, including roads, buildings, bridges, cars, airplanes, software, and more. However, gradually, we also started using engineering to produce weapons of mass destruction like missiles, malware, and other deadly products harmful for humans and nature itself. When anything is engineered, it goes through many phases of design, development, and testing. With reverse engineering, things have really changed.

The concept behind reverse engineering is to break something to understand its internal architecture to build a copy or for the purpose of improvement or modification. In this chapter, we will talk about some real-life examples to understand the importance of reverse engineering and how it is changing the way the software industry works.

Structure

In this chapter, we will cover the following topics:

Introduction to Reverse Engineering

Importance of Reverse Engineering

The Role of Reverse Engineering

Objective

After studying this chapter, you should be able to understand the importance of reverse engineering and its impact on the software industry. We will also talk about the opportunities associated with reverse engineering and how malware writers are using it to exploit the software systems of big companies.

Introduction to Reverse Engineering

In software terms, Reverse Engineering (RE) is the art of understanding any program code when no source code is available. All of this started in the late 1980s when the Disk Operating System (DOS) was in use. Most of us were not born at that time or might be in our childhood time. During that time, people used to play DOS-based video games. Most of the games were player-based video games, where the game player had a lifeline and is equipped with the weapons. This is where some group of computer geeks followed reverse engineering techniques to increase the lifeline of the game player and change the number of weapons a player could use. This was done by simply modifying the values at the memory location where the lifeline and the number of weapons of a player were stored. This might sound like cheating, but in reality, it was a way to breach the security of the video game.

To understand the importance of reverse engineering in the present times, we will take an example. Imagine that three people named Jitender, Shilpi, and Atul are working for a research and development organization, the International Institute of Cyber Security, having offices in India, Mexico, and the US. These three employees are working from three different geographical locations.

Figure 1.1: An example of reverse engineering

They are all working on some research and development project and so they share their research findings over the internet. They use some secure software to share the data among themselves. As the data is very critical for the organization, the security of the software used to share this data should also be very secure. Now, this software can be open-source software or closed-source software. If the software they are using is open-source, then they can check the security of the software using code review. But what if the software is closed-source? They will not have an access to the source code of the software.

In this case, reverse engineering plays a big role in checking the security of closed-source software. With the help of reverse engineering, software security can be evaluated even if you do not have the source code available. It will also help in finding vulnerabilities in the software or application if any.

The process of reverse engineering was initially applied to computer applications and hardware but now, reverse engineering is applied everywhere, from software and machinery to even human DNA. Reverse engineering is important especially when you have closed-source software or software with malicious content.

Let us study another famous example of reverse engineering. A company named Phoenix Technologies, based out in San Jose, wanted to develop a BIOS compatible with IBM PCs. Rather than developing a self-designed BIOS, they took the IBM proprietary BIOS, reverse engineered it using the "clean room" or "Chinese wall" approach. Under this approach, they took two teams of engineers. The first team reverse engineered the IBM proprietary BIOS to recreate the design of the IBM proprietary BIOS. Everything was documented by the first team of engineers for the second team to work on. Once this design was recreated, the second team followed the documentation of the design specifications along with the functional requirements created by the first team to code the BIOS compatible with IBM PCs. The second team was totally ignorant about the reverse engineering work of the first team. The final product developed by Phoenix Technologies was sold to other PC manufacturers. The product developed by Phoenix Technologies was operationally identical but with no copyright infringement.

Moreover, other companies like Advanced Micro Devices also reverse engineered Intel corporation microprocessors to make less expensive chips. Reverse engineering is not only used for unethical purposes but also ethical purposes. One among them is malware analysis. As malware’s are closed-source binaries, reverse engineering helps malware researchers decode malware functionality to break them.

To understand the real importance of reverse engineering, let’s talk about a famous ransomware known as Wannacry ransomware. Ransomwares are the kind of malwares that, when installed in a victim’s computer, encrypts the victim’s files and demands a ransom to decrypt those files. If the victim does not pay the ransom within time, the victim’s computer data may be deleted or the data may be left encrypted forever or there are chances that this data might be sold in the black market. Wannacry targeted Windows users by encrypting their data and then demanded a ransom to decrypt the data. To escape the law enforcement agencies, the ransom demanded in Bitcoin cryptocurrency. Bitcoin is a digital currency that is also known as cryptocurrency. It allows people to send and receive money on the internet without having to disclose the real identity of the sender or the receiver. With the efforts of a reverse engineer, Wannacry ransomware was made ineffective. We will study this in detail in Chapter 16, Breaking Wannacry Ransomware With Reverse Engineering.

Importance of Reverse Engineering

Studying an existing design

Before designing anything, it is always a good approach to study the existing products available in the market. A good understanding of what a product does and how it works is important for new insights, but identifying where it can be improved can lead to several advantages.

Redeveloping an outdated or lost product

Every product in the market today is the outcome of hard work in terms of time and money. Imagine a situation where a company’s product is in great demand in the market, but due to some unforeseen situation, the product is not getting any upgrades with time. This can be due to some internal reasons or the company that developed the product is no more in the market. With reverse engineering, such outdated products can be studied to recreate updated products.

Security auditing

Reverse engineering sometimes is a part of the security audit done for organizations. This is to check the security of software and the applications used within these organizations. It helps in finding unknown vulnerabilities running inside the organizations.

Finding sensitive data

Sensitive data encoded or encrypted in the software code can be extracted with the help of reverse engineering. This is done to validate the security posture of the software.

Military espionage

This is done to learn the strength of the opponent or enemy by capturing the high-level prototype of devices obtained by troops in the field and dismantling it to develop something new.

Finding product vulnerabilities

For the well-being and safety of the customers using a given product, reverse engineering is used to find defects or vulnerabilities in such a product. Every organization spends a substantial amount of time and money on efforts to find bugs or vulnerabilities in their products. But as it is well known, "nothing is secure". During the design, development, and testing, some bugs don’t get caught. This is where reverse engineering plays a vital role in aiding security researchers to uncover the issues that couldn’t be detected earlier.

Bounty for cyber enthusiasts

Earlier, product-based companies had an internal quality assurance team for security testing as well as functional testing for their products. But with time, everything changes. Cybersecurity requirements in the market changed drastically with an increase in cybersecurity attacks. Companies started offering security researchers a bounty to find vulnerabilities in their products. This helped both the security researchers in terms of money and the product companies in fixing uncaught bugs.

The Role of Reverse Engineering

Computer programs written in C/C++ are human-readable. When these programs are compiled using a compiler, an object file is created which is further passed through a linker to get a binary file or an executable file or, we can say, the ones and zeros of the machine language.

Figure 1.2: The role of reverse engineering

The ones and zeros are not human-readable. To convert the machine code back to a human-readable format, a tool called the decompiler is used. The role of a decompiler is to convert binary code into a human readable format and regenerate the code out of it. We will talk about such tools in Chapter 3, Up and Running with Reverse Engineering Tools

Conclusion

In this chapter, we learned how reverse engineering all began and how it is playing a big role in today’s era. We also studied the importance of reverse engineering and its impact on the software industry. We discussed opportunities associated with reverse engineering and how malware writers are using it to exploit the software of big companies. In the next chapter, we will study the internals of a computing system in terms of reverse engineering.

CHAPTER 2

Understanding Architecture of x86 Machines

In the future, every device or machine will become ‘smart’. The big difference between a normal device (or ‘the legacy device’, as we call it) and a smart device is the presence of the internet feature in a smart device. By smart, it means that the device is programmed to function in a smart fashion and it can be operated remotely using the internet feature. Today, most of the devices we use in our households are internet enabled or we can say, smart devices. Televisions are now smart televisions, washing machines are now smart washing machines, refrigerators we use are also now smart refrigerators, and many more. All this became possible with the introduction of a small computer in the legacy devices like televisions, washing machines, refrigerators, and others. Now a big question is, what’s inside these small computers and how do they work? These small computers are made up of small components, where every component plays an important role in the functioning of the overall system. Imagine that these small computers are a smaller version of your personal computer.

All these devices are addressed as modern computing devices. These computing devices are made up of several components for processing, data storage, data transfer, and more. Modern computing devices coupled with software are programmed to do many tasks. To understand Reverse Engineering (RE) on modern computing devices, we need to first understand what goes inside these computing devices and how they work.

Structure

In this chapter, we will cover the following topics:

Architecture of a Computing System

Building Blocks of a Computing System

History of the Different Types of Processors

Registers, Types of Registers and their Roles

Concept of Stack

Objective

In this chapter, we will talk about computing systems and their types. We will also talk about the components of modern computing systems. Then we will cover the topics of processors and the difference between processor variants along with their numbering scheme. We will also take a look at the role of stack in reverse engineering to understand the difference between caller and callee.

Architecture of a Computing System

Any computing system we see around is made up of some basic building blocks. When we say computing system, it can be your computer, laptop, mobile, IoT devices, and other devices which are capable of performing tasks. Basically, there are two types of computing systems:

Fixed Program Computing System: These systems are architected to perform a specific task. For example, a calculator.

Stored Program Computing System: On the other hand, these systems are architected in such a way that they can be programmed as per the requirements. They can run many tasks simultaneously and we can store and run applications on them. For example, a computer. The architecture of these systems was introduced by John von Neumann in 1945.

The von Neumann architecture is based on the stored program concept, where program data and instruction data are stored in the same memory. This design is used by modern computing systems, which are made up of the following building blocks:

Figure 2.1: Architecture of a Computing System

CPU

The Central Processing Unit controls the operations of our computing device or system. In our computing system, the CPU is also referred to as processor, which is the brain of our computing system. The job of the CPU is to fetch instructions from the memory, decode the instructions into a series of actions, and carry out these steps in a sequence. Inside the CPU, we have several components. Some of them are:

Control Unit: This is responsible for retrieving and decoding instructions from the memory or RAM.

Execution Unit: This unit is responsible for the execution of instructions with the help of registers.

Registers: To save time, the CPU does not access RAM every single time to fetch instructions. So CPU has in itself basic storage units called registers. There are many types of registers, which we will study in the following sections. One among them is Instruction Pointer register, which stores the memory address of the next instruction to be executed.

Flags: These are registers only, but they record the state of CPU after arithmetic calculations.

Memory

This can be Random Access Memory (RAM) or Read Only Memory (ROM). It can also be an external storage device such as Hard Disk (HDD), optical disk, and others. The primary purpose of memory is to store the sequence of instructions that our computer or computing system executes. This is also called program code. The second purpose of memory is to store data, on which our computer works.

Input/output Devices

All the devices which are interfaced with our computing system are called I/O (Input/Output) devices. This can be our keyboard, mouse, monitor, and others. These devices are interfaced using ports and there are two types of ports, Input & Output ports. Input ports are used for reading data from these peripheral devices into the computing system. Output ports are used to send data from the computing system to the peripheral devices such as video display, printer, and others.

System Bus

The System Bus can be imagined as a group of wires that carry information or data between different components in our computing system. Depending on the type of information carried between the components, buses are classified as Address Bus, Data Bus, and Control Bus.

Address Bus: These are parallel signal lines which are used to send out the address of the memory location that is to be read from or written to. The number of memory locations that a CPU can address is calculated by the number of signal lines or address lines. Suppose a CPU has N address lines, so the total number of memory locations the CPU can address is 2N. For example, a CPU has 8 address lines. This CPU can address 256 memory locations. If a CPU has 16 address lines, then the CPU can address 65,536 memory locations.

Data Bus: These are also parallel signal lines which are used to transfer data between the CPU and memory.

Control Bus: A Control Bus contains parallel signal lines carrying synchronizing signals to control various peripheral devices connected to the CPU. These are used to transfer information required to coordinate multiple tasks. This consists of 4-10 parallel signal lines to send out signals on the control bus. Typical control bus signals are I/O Read, I/O Write, Memory Read, and Memory Write. Suppose a CPU needs to read a byte of data from the memory location. In this process, the following activities will happen:

The CPU will send the memory address of the desired byte on the Address Bus.

The CPU will then send the Memory Read signal on the Control Bus.

The Memory Read signal will enable the addressed memory device to output data (or byte) on to the Data Bus.

The Data (or byte) travels from the desired memory address to the CPU using the Data Bus.

Building blocks of a Computing System

To understand reverse engineering, knowledge of the basic data building blocks is a must. These data building blocks include the meaning of Bit, Nibble, Byte, Word, and DWORD. All of these can be explained from the following figure:

Figure 2.2: Understanding Bit, Nibble, Byte

Humans can communicate with each other in different languages based on the countries they reside in. But when we talk about a computing system like computers, they can only understand binary, which is 0 or 1. Computers communicate with each other by sending or exchanging data. The smallest unit of data is called bit, which can be 0 or 1.

1 Nibble means 4 bits. Similarly, we can refer to BYTE, WORD, and DWORD as:

1 BYTE = 2 NIBBLES = 8 bits

1 WORD = 2 BYTES = 16 bits

1 DWORD = 4 BYTES = 32 bits

Microprocessor

As we know, the CPU is the brain of a computing system. The CPU is surrounded by circuitry which in its whole is referred to as the microprocessor. A microprocessor can have more than one CPU, like graphics processor. So, the CPU is actually a part of the microprocessor, but microprocessors can have more than one CPU. There are many types of microprocessors. You must have heard of companies like Intel, AMD, and many more. They are the top manufacturers of microprocessors. Some of the most popular models of the first generation microprocessors are:

8086

Intel 8086 came in the year 1978 and was the first Intel 16 bit microprocessor. This gave rise to the x86 architecture processor that became a successful segment of processors. The specification of the Intel 8086 processor were:

Data bus width: 16 bits

Address bus: 20 bits

80186

Intel 80186 was launched in the year 1982. It was the second model in the Intel family. The specification of the Intel 80186 processor were:

Data bus width: 16 bits

Address bus: 20 bits

80286

Intel 80286 was launched in the year 1982. It was the third model in the Intel family. The specification of the Intel 80286 processor were:

Data bus width: 16 bits

Address bus: 24 bits

80386

Intel 80386 was launched in the year 1985. It was the fourth model in the Intel family. The specification of the Intel 80386 processor were:

Data bus width: 32 bits

Address bus: 32 bits

80486

Intel 80486 came in the year 1989. It was the fifth model in the Intel family. The specifications of the Intel 80486 processor were:

Data bus width: 32 bits

Address bus: 32 bits

So collectively, all the processors are referred to as the x86 Intel family.

Generally, we refer to the Intel processor as follows:

x86-16: It means a 16-bit processor.

x86-32 (aka IA32): It means a 32-bit processor (IA means: Intel Architecture), also referred to as x86 only.

x86-64: It means a 64-bit processor, also referred to as x64.

Note: Throughout this book, we will focus on the Intel x86-32 processor.

Memory

The memory, which we call RAM, for a single process running on the x86-32 architecture is divided into the following sections:

Figure 2.3: Process Address Space

The memory address is ranged from 0x00000000 – 0xFFFFFFFF. The prefix 0x refers to hexadecimal numbers. Every hexadecimal number is 4 bit in size, so any memory address of x86-32 architecture is referred by a combination of 8 hexadecimal numbers, which make 4 x 8 = 32 bits in size. This is why, the memory address of x86-32 computer is 32 bits in size.

Kernel Space: 1GB is reserved for the Operating System kernel.

Stack: This is the space reserved for the function local variables and parameters. A stack grows up to a fixed memory size. It grows from a higher memory address to a lower memory address.

Libraries: This is where our Shared Libraries are loaded. The common dialog box like save dialog box is stored in library which is shared among many programs.

Heap: Heap grows down. When an image is loaded, depending on the size of the image, dynamic memory is required to load an image during the program execution. This memory is freed when the program finishes. This heap memory dynamically changes during program execution. It grows from a lower memory address to a higher memory address.

Data: This is the section of memory used to store static variables and global variables in the code.

Text: This section of memory holds the instruction or code to be executed to perform some action.

Registers

To save time, a small amount of temporary storage is available with the CPU called registers. In the x86 processor, registers are divided into the following categories:

Figure 2.4: Registers

General Purpose Register

X86 architecture has 8 general purpose registers:

EAX is used for arithmetic and logical operations. It is also used to store the function return value.

EBX is used as a pointer to data.

ECX is used for loop operations.

EDX is used for I/O and arithmetic operations.

ESI is used as a pointer to the source in string operations.

EDI is used as a pointer to the destination in string operations.

ESP is a pointer to the top of the stack.

EBP is a pointer to the base of the stack frame.

All general purpose registers are 32 bits in size and they can also be referred in the sizes of 16 bits and 8 bits. The smaller unit of any register can be referred as shown below.

The size of the EAX is 4 byte (32 bits). The “E” in EAX stands for Extended.

Figure 2.5: Smaller unit of register

AX is the lower half of the EAX register, which is of size 16 bits. AX is further divided into AH (A-High) and AL (A-Low), each of 8 bits in size. The same goes for other general purpose registers.

32-bit version of general purpose registers: EAX, EBX, ECX, EDX, ESI, EDI, EBP, ESP

16-bit version of general purpose registers: AX, BX, CX, DX, SI, DI, BP, SP

8-bit version of general purpose registers: AH, AL, BH, BL, CH, CL, DH, DL

Segment Registers

The segment registers CS, DS, SS, ES, FS and GS are 16 bits in size. A segment register points at the start of a segment in memory. Segments can be categorized based on the three types of storage: Code, Data, and Stack.

Code Segment: This segment contains all the instructions to be executed and the Code Segment (CS) register contains the pointer to the code.

Stack Segment: This segment holds the data, variables, and arguments of the functions. Stack Segment (SS) register holds the pointer to the stack.

Data Segment: For code efficiency and security, four separate Data Segments are created. These are:

One for data structure of current loaded module

Data exported from other third-party modules

Dynamically created data structures

Data which is shared among different programs

To access the different types of data structures and additional data segments, DS, ES, FS, GS registers are used.

Status Registers

The status register is EEFLAGS register. This register is also 4 byte in size. Each bit of this register represents some flag, which can be either zero (0) or one (1). The status of each bit represents some result of the CPU operation. Some of the common flags used while performing reverse engineering are:

Zero Flag (ZF): This flag is set to 1 when the result of the operation is zero. Otherwise, it is cleared.

Carry Flag (CF): This flag is set to 1 when the result of the operation is either too large or too small for the destination. Otherwise, it is cleared.

Sign Flag (SF): This flag is set to 1 when the result of the operation is negative. The flag is cleared, or we can say 0, when the result of the operation is a positive value.

Trap Flag (TF): This flag is set to 1 when the processor executes one instruction at a time and it is used for debugging.

Parity Flag (PF): It is set to 1 when Least Significant Bit (LSB) of the result contains an even number. Otherwise, it is cleared.

Overflow Flag (OF): This flag is set to 1 when the result of the operation is too large to fit.

Instruction Pointer Register

Instruction Pointer is also known as EIP. This register contains the memory address of the next instruction to be executed. EIP tells the processor what to do next. From the security point of view, EIP is very important. An attacker compromises any system by taking control over the instruction pointer. Once the EIP is in control of the attacker, a malware code can be executed to perform any task.

Concept of Stack

This is the most important concept in Reverse Engineering. Imagine a stack as a pile of lunch plates lying one above the other at your local restaurant or cafeteria. To pick up a plate, we take out the plate from the top and the same process is followed to add more plates to the pile. The plate added to the pile will be added on the top. If we have to take out the plate at the bottom of the pile, we will have to take out every plate above it, one by one. For the time being, assume that adding a plate to the pile is called PUSH and taking out a plate from the pile is called POP.

A stack works the same way, Last In First Out (LIFO) This means that the last thing added (pushed) will be the first thing to get pulled off (popped). To understand the working of a stack, we will take a pseudo code:

Figure 2.6: Pseudo code

In this pseudo code, there are two functions: one is the main function from where our code execution starts and another is the Foo function. Foo function has 3 arguments which are local to main function and 2 local variables. During the code execution, the Foo function is called from the main function. In this code, the main function is the Caller and the Foo function is the Callee.

We are assuming that the 3 arguments and 2 local variables mentioned in the code are of type int. The data type of int occupies 4 bytes in the memory, as sizeof(int) is 4 bytes.

When our execution reaches inside the Foo function, the typical stack state will look like something as follows:

Figure 2.7: Stack

Every function has a stack frame. As we have two functions main and Foo, we will have two stack frames. The highest address of the stack frame is the EBP of that stack frame.

Let’s go step by step to understand how a stack frame is set up and cleaned upon function return.

Caller Before Callee Call

In this section, we will look at the stack state when the main function is just about to call the Foo function. Consider the Foo function call as follows:

int main()

{

FooReturnValue = Foo(10, 20, 30);

}

Caller is our main function and is about to call the Foo function which is Callee. The main function has its own stack frame, where the ESP is pointing to the top of the stack and EBP is a base pointer of the main function’s stack frame.

Figure 2.8: Caller Before Callee Call

Before the Foo function is called, the main function pushes the EAX, ECX and EDX registers onto the stack, only if the content of these registers needs to be preserved. Next, the main function pushes the Foo function arguments one by one onto the stack.

FooReturnValue = Foo(10, 20, 30); //Foo function arguments are 10, 20, 30

Arguments are pushed from the right to left order, so first 30 then 20 and then 10 is pushed onto the stack. The assembly instructions for the same are:

PUSH 30

PUSH 20

PUSH 10

After pushing the Foo arguments on the stack, a call to Foo function is made using the CALL instruction in the assembly language:

CALL Foo

When the CALL instruction is executed, the content of the EIP register is pushed onto the stack as EIP points to the next instruction in the main after the CALL instruction. After pushing the EIP onto the stack, we will have the return address on the top of the stack. This return address will help the instruction pointer to resume execution in the main once the execution of the Foo function is over.

Callee After Function Call

When the Foo function gets the control, it will perform the following three tasks:

Set up the Foo function stack frame.

Allocate space for the Foo function local variables.

If required, preserve the contents of EBX, ESI and EDI.

The Foo function stack frame is set up using the following assembly instructions:

PUSH EBP

MOV EBP, ESP

To set up the stack frame for the Foo function, we will first preserve the main’s EBP (which is the base pointer of the main function) by pushing it onto the stack.

Figure 2.9: Callee after function call

After pushing the main’s EBP on the stack, ESP (which is pointing to the top of the stack) will become the new EBP (base pointer of the Foo function). This EBP, along with the offset, will be used to refer to the variables on the stack. As we can notice in the preceding figure, the first argument can be accessed using EBP plus 8 bytes (4 bytes for main’s EBP + 4 bytes for the Return Address).

Now we need to allocate space for the Foo function local variables onto the stack. This is done by subtracting 20 bytes from the stack pointer, where 20 bytes include 8 bytes (4 bytes + 4 bytes space for 2 variables) and 12 bytes for temporary storage. This will be done using:

SUB ESP, 20

Local variables and temporary variables can be accessed using the offset from EBP. After allocating room for the local variables on the stack, EBX, ESI and EDI registers are preserved by pushing on the stack.

The stack state after preserving the contents of EBX, ESI and EDI will be as follows:

Figure 2.10: Stack Frames of Foo and Main

During the Foo function execution, there can be many pushing and popping on the stack. In this case, ESP will move up and down, but the EBP will remain unchanged. With the help of EBP, variables can be accessed using the offset from EBP.

Callee Before Returning

Now suppose we are done with the Foo function execution. Before returning, the callee will make arrangements to save the Foo function return value in the EAX register. Secondly, the values of EBX, EDI and ESI are restored back. The Foo function stack frame will be taken down by the following assembly instructions:

MOV ESP, EBP

POP EBP

RET

These instructions will bring the stack to the following state by moving the stack pointer ESP back to the EBP (base pointer of the Foo function) and the old EBP (main’s EBP) is restored by popping EBP from the stack.

Figure 2.11: Callee before returning

RET, return instruction will pop return address from the stack and move it to EIP. This will point the Instruction pointer (EIP) back to the main function, to resume execution of main function.

Caller After Returning

As the instruction pointer is back in the main function, now we do not require arguments on the stack anymore.

Figure 2.12: Caller after returning

All the arguments on the stack are cleaned by adding ESP by 12 bytes (4 bytes for each argument and multiplied by the number of arguments, which is 3).

ADD ESP, 12

As the EAX register is holding the return value of the Foo function, the content of the EAX register could be moved to some other register. Finally, the preserved registers EAX, ECX, and EDX are popped from the stack and ESP is pointed back to the same location on the stack where it started.

Conclusion

We covered the architecture of modern computing devices and the different components that make up a computing device. We also covered the functioning of different components used in a computing device. This covered the basics about microprocessors, memory, and the different types of registers. The different processor variants were also discussed along with the relevance of the x86 numbering convention.

We covered an important aspect of stack. With the help of pseudo code, we understood the difference between caller and callee. Understanding of stack plays an important role in the reverse engineering of any application. So don’t skip the stack part. In the next chapter, we will talk about the different reverse engineering tools used by professionals in the industry.

CHAPTER 3

Up and Running with Reverse Engineering Tools

Tools play a vital role in every aspect of life. We often use mobile calculators to perform basic math in our day-to-day life. Before mobile calculators, we used hardware calculator to perform math calculations. This calculator is a simple example of a tool we use to perform certain tasks. Simple calculations can be done verbally but when it comes to complex ones, it becomes necessary to use a tool. Similarly, for reverse engineering there are plenty of tools available in the market. Some are commercial and some are free to use or open source. For selection of the correct tools, conceptual knowledge of the topic becomes essential.

If you search the internet for reverse engineering tools, you will find several. It is always important to have the right selection of tools based on your requirements. In this chapter, we will first learn about the concept of tools in reverse engineering and then understand the importance of these tools in the process of reverse engineering. We will talk about the tools that are easily and freely accessible.

Structure

In this chapter, we will cover the following topics:

Importance of tools in reverse engineering

Reverse engineering tools

Portable executable editors

Disassemblers

Debuggers

Objective

The objective of this chapter is to understand the basic concept behind tools and why they are required in the process of reverse engineering. Then we will talk about some tools required to read the binary format. We will also cover some concepts related to Portable Executable (PE) editors, disassemblers, and debuggers.

Importance of tools in reverse engineering

When we compile a program using a high-level language like C/C++, it gets converted to a series of bytes that a CPU can understand. This is a machine code, which is difficult to understand by humans. To make this code easier to understand, we use a tool called disassembler. This disassembler translates machine code to human-readable format. This format is called assembly code which uses the syntax of assembly language. The following figure will help you understand the concept throughout your life:

Figure 3.1: Importance of RE tools

Reverse engineering helps to regenerate the application logic without having the source code of an application or a program. Malware researchers follow this concept to perform reverse engineering tasks.

Now we will discuss a few tools required for reverse engineering. We will also use these tools throughout this book.

Reverse engineering tools

For easy understanding, reverse engineering tools are divided in different categories like PE editors, disassemblers, and debuggers. Within each category, we have free or open-source tools available, and all tools within a given category serve the same purpose with some slight feature differences. We will focus on free and open-source tools with graphical user interface. All these tools are used throughout this book.

Portable Executable Editors

PE stands for Portable Executable. Portable Executable is the standard Windows file format. Every Windows executable uses this file format. Dynamic Link Library (DLL), Component Object Model (COM) files, and .NET executable use the PE file format. The following figure shows the basic structure of PE:

Figure 3.2: PE structure

All the PE files start with the DOS header then PE header also called NT header and sections that are common in executable. These common sections are as follows:

.text: This section contains the actual binary executable code.

.bss: This section represents uninitialized data.

.rdata: This section contains read only data, such as constants, strings.

.rsrc: This section is the resource section where resource information is stored.

.edata: This is the export data section containing information about exported functions.

.idata: This is import data section which contains information about imported functions along with the import directory and import address table.

.debug: Initially, debug information was placed in this debug information section. PE files also support a separate debug file with .dbg extension.

To view and edit all these details, we can use the following tools.

CFF Explorer

This tool is designed to view and edit PE files without losing the internal structure of PE files. This tool is not only used by reverse engineers but also the developer of applications.

Figure 3.3: CFF Explorer

We will be using this CFF explorer to edit PE files in our subsequent chapters.

Disassembler

As we discussed in the earlier section that a machine code is not human readable, we need some tools to convert machine code into a human-readable format. This is where disassemblers come into picture. Disassemblers are used to convert a machine code into a human-readable assembly code. The following are some disassemblers used in this book.

Ghidra

This is an open-source tool developed by the National Security Agency (NSA). It is free and used for reverse engineering. Its source code was released on April 4, 2019. This tool is used by malware researchers and reverse engineers to analyze malwares and find vulnerabilities in applications.

Figure 3.4: Ghidra

Cutter

Cutter is an open-source interface to the Radare2 reverse engineering framework. Radare2 is the command line tool for reverse engineering and is used for static and dynamic analysis of binary formats on different platforms and architectures. Cutter is the graphical user interface of Radare2.

Figure 3.5: Cutter

Debuggers

When we run an application or a binary, we check the state of a running program. A debugger gives the dynamic state of a binary when it is executed in the memory. Some vulnerabilities are not captured by the developer while writing or executing a code. This is where we use debuggers to run the code and monitor the registers, memory locations, and other parameters. We will be heavily using the following debugger in this book.

x32dbg

A debugger comes in two flavors: 32 bit and 64 bit. x32dbg is used for x86 (32 bit) binaries. Another version is x64dbg, which is used for x64 (64 bit) binaries. As shown in the following figure, x32dbg is divided into 4 sections:

Figure 3.6: x32dbg

As shown in the preceding figure, these four sections are as follows:

Disassembly or CPU Instructions: This is where a machine code converted into an assembly code is shown. The first column is the memory address of the instruction. The second column is the opcode, also called the operation code. The third column is the assembly instructions. The forth column shows the comment about instructions.

Registers and Flags: This section shows the registers and their values during a dynamic analysis of the binary. It also shows the value of a flag to display the current state of the processor.

Stack: As discussed in the earlier chapter, a stack is Last In First Out (LIFO). This means the last thing added (pushed) will be the first thing to get pulled off (popped). A stack grows from a higher memory address to a lower memory address.

Memory dump: It is like hex editor which shows the hexadecimal code of a binary in the memory. It shows raw data in the hexadecimal and ASCII formats. The value on a particular memory address can be changed by double clicking on the respective memory location.

Conclusion

In this chapter, we covered different reverse engineering tools used throughout this book. We also discussed the difference between a disassembler and a debugger. The importance of tools in reverse engineering was also discussed and then we listed some PE editors, disassemblers, and debuggers. In the next chapter, we will walk through the assembly instructions that will help us read and understand a disassembled assembly code.

CHAPTER 4

Walk Through on Assembly Instructions

In the last chapter, we introduced some assembly language instructions. There are many types of instructions in the assembly language which can be grouped together to get a clear understanding and objective of a specific set of assembly instructions. To understand the relevance of walking over the basic assembly instructions in reverse engineering, we will take up a real-life example.

Have you ever opened a toy in your childhood? It was always fun to open a toy and check its internal components. Today, if you had to understand the internal working of a toy, you need to first understand the different components installed internally in the hardware design of the toy. Now, to uncover the internal working of the toy, we have to understand the working of each component installed in it. This whole process can be somewhat compared to the reverse engineering of the toy.

On similar lines, for reverse engineer of any software or application, we need to disassemble it into different instructions and the understanding of the disassembled assembly instructions becomes a must. So, to understand the working of any software or application, we need to have a clear understanding of the instructions and their execution path. This is where understanding of different instructions in assembly language is needed.

Structure

In this chapter, we will cover the following topics:

Different assembly language instructions

Syntax of the assembly instructions

Grouping of assembly instructions

Objective

After going through this chapter, you will be able to understand the important assembly instructions used in reverse engineering and how these instructions are grouped in various sections for easy understanding along with some examples. We will also learn the syntax of the instructions, their internal working, and the basic concept behind the different types of instructions.

Different assembly language instructions

Instructions are the basic building blocks of Assembly Code. Instructions are a combination of the operation code and zero or more operands.

Figure 4.1: Assembly instruction syntax

Operation code is often referred to as opcode. Operands can be of three types:

Intermediate operands: These are fixed hexadecimal value, like 0x0A.

Register operands: They can be any register, such as ECX, EAX, etc.

Memory address operands: These are memory addresses and are represented between square brackets, like [EAX].

The program code is saved in the memory in a sequence of operation code (or opcode) and operands. They are saved in consecutive memory locations. As shown in the following image, instruction 1 occupies two memory locations, instruction 2 occupies the same two memory locations, and instruction 3 occupies only one memory location.

Figure 4.2: Program code in memory

Look at the following example of an assembly instruction with register operands:

Instruction Format:

OPERATION_CODE DESTINATION_OPERAND SOURCE_OPERAND

Example

MOV EAX, ECX

OPERATION_CODE = MOV, stands for move

DESTINATION_OPERAND = EAX, is a destination register

SOURCE_OPERAND = ECX, is a source register

This instruction moves data from the ECX register to the EAX register. Now each instruction represented by the opcodes is also called the operation code. These opcodes tell the CPU what operation the program wants to perform. So, the opcodes for the preceding instruction is 89 C8, which is the hexadecimal representation of the instruction. The disassembler converts these opcodes into a human readable format. So, 89 C8 will be converted to MOV EAX, ECX.

Now that we are clear with the concept of assembly instructions, we will move on to the explanation of major assembly instructions that come on the way to understand reverse engineering. These assembly instructions can be segmented or grouped in various sections for easy understanding. Grouping of assembly instructions can be broadly classified into the following categories:

Stack Instructions

Data Transfer Instructions

Arithmetic Instructions

Program Execution Instructions

Branching Instructions

Bit Manipulation Instructions

Processor Control Instructions

String Instructions

Let’s walk through each section one by one and understand assembly instructions that fall under each category.

Stack Instructions

These are general purpose instructions used for transfer operations across the stack.

PUSH

Instruction Format:

PUSH SOURCE_OPERAND

Meaning:

Push copies the value from the source operand onto the top of the stack and decrements the ESP register.

Flags Affected: None

PUSHAD

Instruction Format:

PUSHAD

Meaning:

Pushes the values of all the registers onto the stack. Registers are pushed in the order of EAX, ECX, EDX, EBX, ESP, EBP, ESI, and EDI.

Flags Affected: None

PUSHFD

Instruction Format:

PUSHFD

Meaning:

This instruction pushes the flags register onto the stack.

Flags Affected: None

POP

Instruction Format:

POP DESTINATION_OPERAND

Meaning:

POP retrieves the value from the top of the stack and copies it to the destination operand. It increments the ESP register.

Flags Affected: None

POPAD

Instruction Format:

POPAD

Meaning:

This instruction pops the values of the stack and copies them to all the registers. Registers are popped in the order of EDI, ESI, EBP, ESP, EDX, ECX, and EAX. The ESP value is ignored in POPAD.

Flags Affected: None

POPFD

Instruction Format:

POPFD

Meaning:

This instruction pops the DWORD from the stack into the flags register.

Flags Affected: None

RET

Instruction Format:

RET [nBytes]

Meaning:

When a function calls another function, the function that calls another function is called ‘caller’ and the function that is called is named ‘callee’. RET transfers control from the callee to the caller, to the return address saved on the stack. nBytes are optional; when nBytes are mentioned, it means that n bytes are released to clean up the stack.

Data Transfer Instructions

These are general purpose instructions used for data transfer operations.

MOV

Instruction Format:

MOV DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

Moves data from the source operand to the destination operand and the result is stored in the destination operand.

Flags Affected: None

LEA

Instruction Format:

LEA REGISTER OPERAND

Meaning:

LEA stands for Load Effective Address and it loads the register with the memory address of the operand.

Flags Affected: None

XCHG

Instruction Format:

XCHG DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

XCHG instruction exchanges the values between the source operand and the destination operand.

Flags Affected: None

CMPXCHG

Instruction Format:

CMPXCHG DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

This instruction is used for comparing and exchanging. EAX is compared with the DESTINATION_OPERAND,

If EAX is equal to DESTINATION_OPERAND then DESTINATION_OPERAND is loaded with SOURCE_OPERAND.

If EAX is not equal to DESTINATION_OPERAND then EAX is loaded with DESTINATION_OPERAND.

Flags Affected: CF ZF SF AF PF OF

LAHF

Instruction Format:

LAHF

Meaning:

This instruction loads AH from the flags. It copies the status to the AH register and only 5 flags are copied to bits 7, 6, 4, 2, and 0 of the AH register; bits 5, 3 and 1 of the AH register will be unaffected.

AH register after running this instruction:

Figure 4.3: Load AH from Flags

Flags Affected: None

SAHF

Instruction Format:

SAHF

Meaning:

It stores the AH into the flags. This instruction copies the bits of the AH register (bits 7, 6, 4, 2, and 0) to SF, ZF, AF, PF, and CF flags.

Flags Affected: SF, ZF, AF, PF, and CF

LAR

Instruction Format:

LAR DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

Loads the effective right from the segment descriptor specified by the SOURCE_OPERAND into the DESTINATION_OPERAND.

Flags Affected: ZF

MOVSX

Instruction Format:

MOVSX DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

Move with Sign Extension. Suppose you have a smaller value and want to copy it to a big register. In that case, we will use MOVSX; it copies SOURCE_OPERAND to DESTINATION_OPERAND and fills the bits not provided by SOURCE_OPERAND with sign bits.

Flags Affected: None

MOVZX

Instruction Format:

MOVZX DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

Move with Zero Extension. Suppose you have a smaller value and want to copy it to a big register. In that case, we will use MOVSX; it copies SOURCE_OPERAND to DESTINATION_OPERAND and fills the bits not provided by SOURCE_OPERAND with zero.

Flags Affected: None

XLAT

Instruction Format:

XLAT

Meaning:

The XLAT instruction replaces the AL register from the table index to the table entry. It sets the AL to DS:[EBX + unsigned AL]. The value of AL is treated as an unsigned index, which is added to the EBX register to get the table entry. EBX contains the base address of the table.

MOVS

Instruction Format:

MOVS DESTINATION_STRING SOURCE_STRING

Meaning:

This instruction copies data (Byte, WORD, DWORD) from SOURCE_STRING to DESTINATION_STRING, where:

SOURCE_STRING is pointed by DS:SI (or ESI)

DESTINATION_STRING is pointed by ES:DI (or EDI)

SI is the offset address in Data Segment (DS)

DI is the offset address in Extra Segment (ES)

Flags Affected: None

Arithmetic Instructions

These instructions are used for arithmetic operations in the assembly language.

AAA

Instruction Format:

AAA

Meaning:

This instruction adjusts the ASCII after addition. The AAA instruction is used after the addition (ADD instruction) of two unpacked BCD numbers. Unpacked BCD numbers are ASCII single-digit numbers from 0 to 9 or 0x30 to 0x39. When the AAA instruction is used after the ADD instruction, it converts the result to a two-digit BCD number. For ASCII codes, refer to the Appendix.

When two ASCII coded numbers are added, the result is not ASCII. To convert this to ASCII, AAA is used after ADD.

Flags Affected: AF CF (SF,ZF,OF,PF undefined)

Example

XOR AH, AH        ;clear AH register

MOV AL, 32H      ;move ASCII 2 in AL

ADD AL, 39H      ;add ASCII 9, the result should be ASCII 11, but we get 6B in AL

AAA                     ;AH=0x01, AL=0x01 and AX=0x0101

AAS

Instruction Format:

AAS

Meaning:

The instruction adjusts the ASCII after subtraction. The AAS instruction is used after the subtraction (SUB instruction) of two unpacked BCD numbers. Unpacked BCD numbers are ASCII single-digit numbers from 0 to 9 or 0x30 to 0x39. When the AAS instruction is used after the SUB instruction, it converts the result to a two-digit BCD number.

Flags Affected: AF CF (SF,ZF,OF,PF undefined)

AAD

Instruction Format:

AAD

Meaning:

The ASCII is adjusted before division. This instruction is used before the division instruction to convert the unpacked BCD number in AH and AL to the binary equivalent. The quotient will be saved in AL and the remainder will be saved in AH; both are unpacked BCD.

Flags Affected: PF, SF, ZF

Example

Divide 68 by 8.

MOV AX, 0608H      ;AH=0x06, AL=0x08 and AX=0x0608

MOV CH, 08H          ;divide ASCII 8, the result should be 8 in quotient and 4 in remainder

AAD                          ;AX = 0044 = 44H = 68

DIV CH                     ;Divide AX by in CH, result: AL = 08, AH = 04 unpacked BCD

AAM

Instruction Format:

AAM

Meaning:

The ASCII is adjusted for multiplication. This instruction is used in the multiplication of two ASCII numbers. When the unpacked BCD digits are multiplied, the result stored in AX is converted to the unpacked BCD number in AH and AL.

Flags Affected: PF, SF, ZF

Example

MOV AL, 00000111      ;move 7 in AL, masking upper 4 bits

MOV BH, 00000101     ;move 5 in BH, masking upper 4 bits

MUL BH                        ;AX = 23H, 35 in decimal

AAM                              ;result: AX = 0203H, AH = 02H, AL = 03H unpacked BCD

ADC

Instruction Format:

ADC DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

Adds two operands, source operand and destination operand. The result is stored in the destination operand. If CF flag is set to 1, then 1 is added to the destination.

Flags Affected: AF CF OF PF SF ZF

ADD

Instruction Format:

ADD DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

The instruction adds two operands, source operand and destination operand. The result is stored in the destination operand.

Flags Affected: AF CF OF PF SF ZF

CMP

Instruction Format:

CMP DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

It subtracts two operands, source operand and destination operand. The result is not stored, but the flags are updated. The flags are subsequently checked in the instructions.

Flags Affected: AF CF OF PF SF ZF

DAA

Instruction Format:

DAA

Meaning:

The decimal is adjusted after addition. This instruction comes after the ADD or ADC instruction to adjust the final result in BCD. This only works with the AL registers.

Flags Affected: AF CF (OF, PF, SF, ZF undefined)

Example

MOV DX, 1122H     ;load 1122H BCD

MOV BX, 3088H     ;load 3088H BCD

MOV AL, BL           ;only AL register works for DAA

ADD AL, DL           ;

DAA                         ; BCD adjusted result, answer saved in AL

DAS

Instruction Format:

DAS

Meaning:

The decimal is adjusted after subtraction. This instruction comes after the SUB or SBB instruction to adjust the final result in BCD. This only works with the AL registers.

Flags Affected: AF CF (OF, PF, SF, ZF undefined)

Example:

MOV DX, 1122H     ;load 1122H BCD

MOV BX, 3088H     ;load 3088H BCD

MOV AL, BL           ;only AL register works for DAA

SUB AL, DL            ;

DAS                         ; BCD adjusted result, answer saved in AL

DEC

Instruction Format:

DEC DESTINATION_OPERAND

Meaning:

The instruction decrements one from the destination operand which can be register or a memory location. The result is saved back in the register or memory location.

Flags Affected: AF OF PF SF ZF

Example:

MOV EAX, 02H      ;load EAX with 02H

DEC EAX                ;EAX will be 01H

DIV

Instruction Format:

DIV SOURCE_OPERAND

Meaning:

The DIV instruction is used to divide the unsigned QWORD/DWORD/WORD by DWORD/WORD/BYTE.

When a WORD is divided by byte, the number to be divided, that is WORD, must be in the AX register and the divisor, which is source operand, can be in register or memory location. After division, the quotient will be saved in AL and the remainder will be in AH.

When a DWORD is divided by WORD, the number to be divided, that is DWORD, must be in DX:AX (most significant WORD in DX and least significant WORD in AX). The divisor, which is the source operand, can be in register or memory location. After division, the quotient will be saved in AX and the remainder will be saved in DX.

When a Double DWORD (QWORD) is divided by DWORD, the number to be divided, that is Double DWORD (QWORD), must be in EDX:EAX (higher order DWORD in EDX and lower order DWORD in EAX). The divisor, which is the source operand, can be in register or memory location. After division, the quotient will be saved in EAX and remainder in EDX.

Flags Affected: None

Example

MOX DX, 0                 ;load DX with 00H

MOV AX, 0x8003       ;load AX with 8003H

MOV CX, 0x100         ;load CX with 100H

DIV CX                       ;AX=80H, DX=03H

IDIV

Instruction Format:

IDIV SOURCE_OPERAND

Meaning:

The DIV instruction (Integer Divide) is used in division of signed data. The rest is the same with respect to the dividend, divisor, quotient, and remainder as in DIV instruction.

Flags Affected: None

MUL

Instruction Format:

MUL SOURCE_OPERAND

Meaning:

The MUL instruction is used to multiple the unsigned DWORD/WORD/BYTE by DWORD/WORD/BYTE.

When a BYTE is multiplied by BYTE, the multiplicand, that is BYTE, must be in the AL register and the multiplier, which is the source operand, can be in the register or memory location. After multiplication, the result will be saved in AX. Higher order 8 bits are stored in AH and lower order 8 bits are stored in AL.

When a WORD is multiplied by WORD, the multiplicand, that is WORD, must be in the AX register and the multiplier, which is the source operand, can be in the register or memory location. After multiplication, the result will be DWORD, which is saved in DX:AX. The higher order WORD is stored in DX and the lower order WORD in AX.

When a DWORD is multiplied by DWORD, the multiplicand, that is DWORD, must be in the EAX register and the multiplier, which is the source operand, can be in the register or memory location. After multiplication, the result will be QWORD, which is saved in EDX:EAX. The higher order DWORD is stored in EDX and the lower order DWORD in EAX.

Flags Affected: OF, CF

Example

MOV AX, 0x8003     ;load AX with 8003H

MOV CX, 0x100       ;load AX with 100H

MUL CX                   ;DX=80H, AX=0300H

IMUL

Instruction Format:

IMUL SOURCE_OPERAND

Meaning:

The Integer Multiple (IMUL) instruction is used to multiply the signed data. The rest is the same with respect to the multiplicand and multiplier as in MUL instruction.

Flags Affected: OF, CF

INC

Instruction Format:

INC DESTINATION_OPERAND

Meaning:

It increments or adds one to the destination operand, which can be the register or a memory location. Result is saved back in register or a memory location.

Flags Affected: AF OF PF SF ZF

Example

MOV EAX, 02H     ;load EAX with 02H

INC EAX                ;EAX will be 03H

NEG

Instruction Format:

NEG DESTINATION_OPERAND

Meaning:

This instruction changes the sign of the destination operand from a positive number to a negative number or from a negative number to a positive number. Basically, it subtracts 0 from the destination operand and performs the 2’s complement to save the result back in the destination operand. 2’s complement of an 8-bit number 00100101 -> 11011010 (Invert bits) -> then add 00000001 = 11011011

Flags Affected: AF OF PF SF ZF

Example

MOV EAX, 02H     ;load EAX with 02H, (2 in decimal)

NEG EAX               ;EAX will be FFFFFFFEH, (-2 in decimal)

SBB

Instruction Format:

SBB DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

It subtracts source operand from the destination operand. The result is stored in the destination operand. If the CF flag is set to 1, then 1 is subtracted from the destination.

Flags Affected: AF CF OF PF SF ZF

SUB

Instruction Format:

SUB DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

Subtract the source operand from destination operand. The result is stored in the destination operand.

Flags Affected: AF CF OF PF SF ZF

XADD

Instruction Format:

XADD DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

The Exchange and Add (XADD) instruction is the same as the ADD instruction where it adds the source operand and the destination operand, to store the result in the destination operand. The difference from ADD instruction is that after the XADD instruction, the original value of the destination operand is copied to the source operand.

Flags Affected: AF CF OF PF SF ZF

Example

MOV EAX, 0x00000001    ; load EAX with 01H, (1 in decimal)

MOV EBX, 0x00000002    ; load EBX with 02H, (2 in decimal)

XADD EAX,EBX              ; EAX= 03H (3 in decimal), EBX=01H

Program Execution Instructions

These instructions are used in controlling the program execution flow and flags.

CALL

Instruction Format:

CALL DESTINATION_OPERAND

Meaning:

During program execution, a procedure is called from another function; the CALL instruction is used. Using CALL instruction pointer jump to the code of a procedure called within a function. When the CALL instruction is executed, the memory address of the next instruction after the CALL instruction is pushed onto the stack. This memory address is popped back from the stack once the called procedure execution is over with the RET instruction.

Flags Affected: None

Example

CALL PROC_LABEL    ; Program flow is deviated to the procedure label memory location

ENTER

Instruction Format:

ENTER Storage, Level

Meaning:

In Intel Architecture, another method of performing the procedure call is supported with the ENTER and LEAVE instructions. These instructions create and release the stack frame for the procedure to store variables and pointers to return the execution from the procedure. In block-structured language like C and Pascal, these instructions provide machine language support for procedure calls.

The ENTER instruction has 2 operands:

Storage: This tells the number of bytes to be reserved on the stack for the procedure call.

Level: Also called lexical nesting level (from 0 to 31), it is the depth of procedure in a ladder of procedure calls.

Flags Affected: None

LEAVE

Instruction Format:

LEAVE

Meaning:

This instruction releases the stack frame created for the procedure to store variables and pointers to return the execution from the procedure by restoring (E)SP/(E)BP.

Flags Affected: None

INT

Instruction Format:

INT <type>

type is between 0 and 255.

Meaning:

Interrupt is a condition which halts the processor to temporarily work on some other task and then return to the main task. Interrupt is an event or signal that asks for the CPU’s attention. This is used by peripheral hardware devices to access the CPU. Whenever the interrupt occurs, the processor completes the current set of instructions and then starts the interrupt service routine (ISR) or interrupt handler. ISR is a routine which contains a set of instructions to handle specific interrupts. This ISR tells the processor what to do when an interrupt occurs.

Now, the INT instruction allows a program to explicitly call the interrupt handler. The following tasks are done when an INT instruction is called:

The FLAGS register is pushed on the stack.

The Code segment (CS) is pushed on the stack.

The offset of the next instruction after the INT instruction is pushed onto the stack.

The IP (Instruction Pointer) is loaded from an absolute memory address, which is a multiple of the interrupt type by 4. If INT is 8, the IP (Instruction Pointer) will be read from 8 * 4 = 32 decimal = 0x00020 memory location.

The code segment will be the next WORD location. The CS will be 0x00022 (0x00020 + 2).

Reset TF Trap flag (TF) and IF Interrupt flag (IF).

Flags Affected: IF, TF

INTO

Instruction Format:

INTO

Meaning:

Interrupt Overflow (INTO): If the Overflow flag (OF) is set, this instruction raises the overflow exception. If OF is not set, then the instruction execution continues without raising an exception. This helps to check the overflow condition. The following tasks are done when an INT instruction is called:

The FLAGS register is pushed on the stack.

The CS (code segment) is pushed on the stack.

The offset of the next instruction after the INT instruction is pushed onto the stack.

The IP (Instruction Pointer) is loaded from an absolute memory address, which is multiple of interrupt type by 4.

If the overflow condition is INT 4, the IP (Instruction Pointer) will be read from 4 * 4 = 16 decimal = 0x00010 memory location.

The code segment will be the next WORD location. The CS will be 0x00012 (0x00010 + 2).

Reset Interrupt flag and Trap flag to 0.

Flags Affected: IF, TF

IRET

Instruction Format:

IRET

Meaning:

Interrupt Return (IRET): This instruction is used to end the Interrupt Service Routine (ISR) or interrupt handler and return the execution to the interrupted program. When an Interrupt Service Routine (ISR) is called, the instruction pointer, code segment, and flags registers are pushed onto the stack. On return from the ISR instruction pointer, the code segment and flags are restored back from the stack to continue the execution of the interrupted program.

Flags Affected: AF, CF, DF, IF, ZF, SF, TF, PF

LOOP

Instruction Format:

LOOP DESTINATION

Meaning:

In LOOP instruction, the (E)CX register is decremented by 1. If the new value in the (E)CX register is non-zero, then a jump is taken to the destination mentioned in the instruction. If the (E)CX register is decremented and the ECX is equal to 0, then no action will be taken and the instruction next to the LOOPE instruction is executed.

Flags Affected: None

Example

LOOPE MEM_LOC     ; Program will jump to the memory location if ECX is non-zero

                                      ; after decrementing

LOOPE

Instruction Format:

LOOPE DESTINATION

Meaning:

In the LOOPE instruction, the (E)CX register is decremented by 1. If the new value in the (E)CX register is non-zero and the ZF flag is set to 1, then a jump is taken to the destination mentioned in the instruction. If the (E)CX register is decremented and ECX is equal to 0, then no action will be taken and the instruction next to the LOOPE instruction is executed.

Flags Affected: None

Example

LOOPE MEM_LOC     ; Program will jump to the memory location if ECX is non-zero

                                       ; after decrementing and ZF=1 due to the previous instruction

LOOPNE

Instruction Format:

LOOPNE DESTINATION

Meaning:

In the LOOPNE instruction, the (E)CX register is decremented by 1. If the new value in the (E)CX register is non-zero and the ZF flag is set to 0, then a jump is taken to the destination mentioned in the instruction. If the (E)CX register is decremented and ECX is equal to 0, then no action will be taken and the instruction next to the LOOPNE instruction is executed.

Flags Affected: None

Example

LOOPNE MEM_LOC    ; Program will jump to the memory location if ECX is non-zero

                                        ; after decrementing and ZF=0 due to the previous instruction

TEST

Instruction Format:

TEST DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

This instruction performs the logical AND between the source operand and the destination operand. Unlike the AND instruction, the TEST instruction does not update any of the operands. It updates the flags without saving the results. This instruction is used to check the registers for zero, without altering its value.

Flags Affected: SF, ZF, and PF

Example

TEST EAX, 01H    ; IF EAX=01H

                                ; 01H AND 01H = 01H (non –zero value)

                                ; Result is non-zero value, ZF is not set, ZF=0

TEST EAX, 02H    ; IF EAX=01H

                                ; 01H AND 02H = 00H (zero value)

                                ; Result is zero value, ZF is set, ZF=1

TEST EAX, EAX   ; if EAX is equal to 0, set ZF to 1

Branching Instructions

This includes instructions which help in controlling the code flow. There are two types of jumps in x86:

Unconditional jump: The instruction pointer jumps to the code path mentioned.

Conditional jump: The instruction pointer jumps after evaluating the condition. The condition is commonly evaluated using two instructions. Both the instructions do not store any result but change the flags in the EFLAGS register.

TEST instruction: It performs the logical AND.

CMP instruction: It performs subtraction.

Note: The DESTINATION_OPERAND used in branching instructions is also named as a LABEL or DESTINATION LABEL or DESTINATION ADDRESS. This is a displacement from the address of the unconditional/conditional jump instruction itself or the absolute address.

JMP

Instruction Format:

JMP DESTINATION_OPERAND

Meaning:

This is an unconditional jump instruction where the instruction pointer jumps to the destination operand during execution.

Flags Affected: None

Example

JMP PROC_LABEL   ;Program flow jumps to the procedure label memory location

JZ

Instruction Format:

JZ DESTINATION_OPERAND

Meaning:

This is a conditional jump instruction which jumps to the destination operand if the zero flag (ZF) is set to 1. If ZF is set to 0, then no action will be taken and the next instruction following it will be executed.

Flags Affected: None

Example

JZ MEM_LOC  ; Program will jump to the memory location if ZF=1

JNZ

Instruction Format:

JNZ DESTINATION_OPERAND

Meaning:

This is a conditional jump instruction which jumps to the destination operand if the zero flag (ZF) is set to 0. If ZF is set to 1, then no action will be taken and the next instruction following it will be executed.

Flags Affected: None

Example

JNZ MEM_LOC    ; The program will jump to the memory location if ZF=0

JE

Instruction Format:

JE DESTINATION_OPERAND

Meaning:

This instruction is the same as JZ. As Jump if Equal (JE) is a conditional jump instruction, it is commonly used after the CMP instruction. The jump will happen if the destination operand is equal to the source operand in the conditional jump.

Flags Affected: None

Example

CMP EAX, 01H    ; Compare the EAX value with 01H

JE MEM_LO        ; The program will jump to the memory location if EAX = 01H and ZF = 1

JNE

Instruction Format:

JNE DESTINATION_OPERAND

Meaning:

This instruction is the same as JNZ. As Jump if Not Equal (JNE) is a conditional jump instruction, it is commonly used after the CMP instruction. The jump will happen if the destination operand is not equal to the source operand.

Flags Affected: None

Example

CMP EAX, 01H    ; Compare the EAX value with 01H

JNE MEM_LO      ; The program will jump to the memory location if EAX != 01H and ZF = 0

JG

Instruction Format:

JG DESTINATION_OPERAND

Meaning:

As Jump if Greater (JG) is a conditional jump instruction, it is commonly used after the CMP instruction where it performs a signed comparison between the destination operand and the source operand. The jump will happen if the destination operand is greater than the source operand.

Flags Affected: None

Example

CMP EAX, 02H    ; Compare the EAX value with 02H

JG MEM_LO        ; The program will jump to the memory location if EAX > 02H

JGE

Instruction Format:

JGE DESTINATION_OPERAND

Meaning:

The Jump if Greater or Equal (JGE) is a conditional jump instruction, so it is commonly used after the CMP instruction where it performs a signed comparison between the destination operand and the source operand. The jump will happen if the destination operand is greater than or equal to the source operand.

Flags Affected: None

Example

CMP EAX, 02H     ; Compare the EAX value with 02H by subtracting 02H from EAX

JGE MEM_LO      ; The program will jump to the memory location if EAX ≥ 02H

JA

Instruction Format:

JA DESTINATION_OPERAND

Meaning:

This instruction is the same as JG. As Jump if above (JA) is a conditional jump instruction, it is commonly used after the CMP instruction where it perform an unsigned comparison between the destination operand and the source operand. The jump will happen if the destination operand is above the source operand.

Flags Affected: None

Example

CMP EAX, 04H    ; Compare the EAX value with 04H by subtracting 04H from EAX

JA MEM_LO        ; The program will jump to the memory location if EAX is above 04H

JAE

Instruction Format:

JAE DESTINATION_OPERAND

Meaning:

This instruction is the same as JGE. As Jump if above or equal (JAE) is a conditional jump instruction, it is commonly used after the CMP instruction where it performs an unsigned comparison between the destination operand and the source operand. The jump will happen if the destination operand is above or equal to the source operand.

Flags Affected: None

Example

CMP EAX, 04H    ; Compare the EAX value with 04H, by subtracting 04H from EAX

JAE MEM_LO     ; The program will jump to the memory location if EAX is above or equal to 04H

JL

Instruction Format:

JL DESTINATION_OPERAND

Meaning:

As Jump if Less (JL) is a conditional jump instruction, it is commonly used after the CMP instruction where it performs a signed comparison between the destination operand and the source operand. The jump will happen if the destination operand is less than the source operand.

Flags Affected: None

Example

CMP EAX, 02H    ; Compare the EAX value with 02H by subtracting 02H from EAX

JL MEM_LO        ; The program will jump to the memory location if EAX < 02H

JLE

Instruction Format:

JLE DESTINATION_OPERAND

Meaning:

The Jump if Less or equal (JLE) is a conditional jump instruction. It is commonly used after the CMP instruction where it performs a signed comparison between the destination operand and the source operand. The jump will happen if the destination operand is less than or equal to the source operand.

Flags Affected: None

Example

CMP EAX, 02H    ; Compare the EAX value with 02H by subtracting 02H from EAX

JLE MEM_LO      ; The program will jump to the memory location if EAX ≤ 02H

JB

Instruction Format:

JB DESTINATION_OPERAND

Meaning:

This instruction is the same as JL. As Jump if below (JB) is a conditional jump instruction, it is commonly used after the CMP instruction where it performs an unsigned comparison between the destination operand and the source operand. The jump will happen if the destination operand is below the source operand.

Flags Affected: None

Example

CMP EAX, 04H    ; Compare the EAX value with 04H by subtracting 04H from EAX

JB MEM_LO        ; The program will jump to the memory location if EAX is below 04H

JBE

Instruction Format:

JBE DESTINATION_OPERAND

Meaning:

This instruction is the same as JLE. As Jump if below or equal (JBE) is a conditional jump instruction, it is commonly used after the CMP instruction to perform an unsigned comparison between the destination operand and the source operand. The jump will happen if the destination operand is below or equal to the source operand.

Flags Affected: None

Example

CMP EAX, 04H    ; Compare the EAX value with 04H by subtracting 04H from EAX

JBE MEM_LO     ; The program will jump to the memory location if EAX is below or equal to ; 04H

JO

Instruction Format:

JO DESTINATION_OPERAND

Meaning:

This is a conditional jump instruction which jumps to the destination operand if the overflow flag (OF) is set to 1. If OF is set to 0, then no action will be taken and the next instruction following it will be executed.

Flags Affected: None

Example

ADD AL, BL       ; Add the signed bytes in AL and BL

JO MEM_LOC   ; The program will jump to the memory location if OF=1, due to addition ; above

JS

Instruction Format:

JS DESTINATION_OPERAND

Meaning:

This is a conditional jump instruction which jumps to the destination operand if the sign flag (SF) is set to 1. If SF is set to 0, then no action will be taken and the next instruction following it will be executed.

Flags Affected: None

Example

ADD AL, BL       ; Add signed bytes in AL and BL

JS MEM_LOC   ; The program will jump to the memory location if SF=1, due to addition ; above

JECXZ

Instruction Format:

JECXZ DESTINATION_OPERAND

Meaning:

This is a conditional jump instruction which jumps to the destination operand if the ECX register is equal to 0.

Flags Affected: None

Example

JECXZ MEM_LOC   ; The program will jump to the memory location if ECX = 0

Bit Manipulation Instructions

This includes instructions used in bit manipulation operations. Before we get started with the instructions, let’s walk through the truth table:

Figure 4.4: Truth table

BSWAP

Instruction Format:

BSWAP REGISTER32

Meaning:

This instruction changes the byte order of the register from Big-endian to Little-endian or from Little-endian to Big-endian.

Flags Affected: None

Example

MOV EAX, 87654321H    ; EAX is 87654321H

BSWAP EAX                     ; EAX will become 21436587H

AND

Instruction Format:

AND DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

This instruction performs the logical AND operation between the destination operand and the source operand. The result of the AND operation is saved in the destination operand.

Flags Affected: CF OF PF SF ZF

Example

MOV EAX, 87654321H    ; EAX is 87654321H

MOV EBX, 21436587H    ; EBX is 21436587H

AND EAX, EBX               ; EAX will become 0x01414101

NOT

Instruction Format:

NOT DESTINATION_OPERAND

Meaning:

This instruction inverts all the bits of the destination operand. All the 1s will become 0 and all the 0s will become 1 by taking one's complement. One’s complement is obtained by toggling all the bits.

Flags Affected: None

Example

MOV EAX, 12121212H    ; EAX is 12121212H

NOT EAX                          ; EAX will become EDEDEDEDH

OR

Instruction Format:

OR DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

This instruction performs the logical OR operation between the destination operand and the source operand. The result of the logical OR operation is saved in the destination operand.

Flags Affected: CF OF PF SF ZF

Example

MOV EAX, 87654321H    ; EAX is 87654321H

MOV EBX, 21436587H    ; EBX is 21436587H

OR EAX, EBX                  ; EAX will become 0xA76767A7

XOR

Instruction Format:

XOR DESTINATION_OPERAND SOURCE_OPERAND

Meaning:

This instruction performs the exclusive-OR operation between the destination operand and the source operand. The result of the XOR operation is saved in the destination operand.

Flags Affected: CF OF PF SF ZF

Example

MOV EAX, 87654321H    ; EAX is 87654321H

XOR EAX, EAX               ; EAX will become 0x00000000

RCL

Instruction Format:

RCL DESTINATION_OPERAND COUNT

Meaning:

Figure 4.5: RCL

Rotate through Carry Left (RCL) rotates the bits n (count) times in the destination operand from the right to left through the CF Flag. On every rotation, Most Significant Bit (MSB) is moved to the CF flag and the CF flag enters into Least significant bit (LSB).

Flags Affected: CF OF

Example

MOV EAX, 01H    ; EAX is 00000001H and CF=0

RCL EAX, 2          ; RCL EAX bits 2 times, EAX will become 0x00000004

RCR

Instruction Format:

RCR DESTINATION_OPERAND COUNT

Meaning:

Figure 4.6: RCR

Rotate through carry right (RCR) rotates the bits n (count) times in the destination operand from the left to right through the CF flag. On every rotation, LSB is moved to the CF flag and the CF flag enters into MSB.

Flags Affected: CF OF

Example

MOV EAX, 01H    ; EAX is 00000001H and CF=0

RCR EAX, 2          ; RCR EAX bits 2 times, EAX will become 0x80000000

ROL

Instruction Format:

ROL DESTINATION_OPERAND COUNT

Meaning:

Figure 4.7: ROL

Rotate Left (ROL) rotates the bits n (count) times in the destination operand from the right to left and the CF flag will have the value of the last bit rotated.

Flags Affected: CF OF

Example

MOV EAX, 40000001H      ; EAX is 40000001H and CF=0

ROL EAX, 2                    ; ROL EAX bits 2 times, EAX will become 0x00000005 and ; CF will set to 1

ROR

Instruction Format:

ROR DESTINATION_OPERAND COUNT

Meaning:

Figure 4.8: ROR

Rotate Right (ROR) rotates the bits n (count) times in the destination operand from the left to right and the CF flag will have the value of the last bit rotated.

Flags Affected: CF OF

Example

MOV EAX, 40000001H     ; EAX is 40000001H and CF=1

ROR EAX, 2                   ; ROR EAX bits 2 times, EAX will become 0x50000000 and ; CF will set to 0

SHR

Instruction Format:

SHR DESTINATION_OPERAND COUNT

Meaning:

To understand the bit shifting concept, refer to it in the Appendix:

Figure 4.9: SHR

Shift Logical Right (SHR) shifts the bits n (count) times in the destination operand from the left to right and the CF flag will have the value of the last bit shifted out. As it is logical shifting, the bit placeholder on every shift count is set to 0.

Flags Affected: CF OF PF SF ZF (AF undefined)

Example

MOV EAX, 40000001H     ; EAX is 40000001H and CF=0

SHR EAX, 2                    ; SHR EAX bits 2 times, EAX will become 0x10000000 and ; CF will set to 0

SHL

Instruction Format:

SHL DESTINATION_OPERAND COUNT

Meaning:

As said above, to understand bit shifting concept, refer bit shifting section in Appendix:

Figure 4.10: SHL

Shift Logical Left (SHL) shifts the bits n (count) times in the destination operand from the right to left and the CF flag will have the value of the last bit shifted out. As it is logical shifting, the bit placeholder on every shift count is set to 0.

Flags Affected: CF OF PF SF ZF (AF undefined)

Example

MOV EAX, 40000001H      ; EAX is 40000001H and CF=0

SHL EAX, 2                     ; SHL EAX bits 2 times, EAX will become 0x00000004 and ; CF will set to 1

SAR

Instruction Format:

SAR DESTINATION_OPERAND COUNT

Meaning:

Figure 4.11: SAR

Shift Arithmetic Right (SAR) shifts the bits n (count) times in the destination operand from the left to right and the CF flag will have the value of the last bit shifted out. As it is arithmetic shifting, the bit placeholder is determined by MSB.

Flags Affected: CF OF PF SF ZF (AF undefined)

Example

MOV EAX, 40000001H      ; EAX is 40000001H and CF=1

SAR EAX, 2                     ; SAR EAX bits 2 times, EAX will become 0x10000000 and ; CF will set to 0

SAL

Instruction Format:

SAL DESTINATION_OPERAND COUNT

Meaning:

Figure 4.12: SAL

Shift Arithmetic Left (SAL) shifts the bits n (count) times in the destination operand from the right to left and the CF flag will have the value of the last bit shifted out. As it is arithmetic shifting, the bit placeholder at LSB will always be 0.

Flags Affected: CF OF PF SF ZF (AF undefined)

Example

MOV EAX, 40000001H      ; EAX is 40000001H and CF=0

SAL EAX, 2                     ; SAL EAX bits 2 times, EAX will become 0x00000004 and ; CF will set to 1

SHLD

Instruction Format:

SHLD DESTINATION_OPERAND, SOURCE_OPERAND, COUNT

Meaning:

Figure 4.13: SHLD

Shift Left Double (SHLD) shifts the bits n (count) times in the destination operand from the right to left and the empty bit placeholders in the destination operand are filled by the bits shifted out of the source operand. The CF flag will have the value of the last bit shifted out of the destination operand. The source operand will not be modified.

Flags Affected: CF OF PF SF ZF (AF undefined)

Example

MOV EAX, 40000001H    ; EAX is 40000001H and CF=0

MOV EBX, 50000001H    ; EBX is 50000001H and CF=0

SHLD EAX, EBX, 2         ; EAX = 0x00000005 and EAX = 0x50000001

                                           ; CF will set to 1

SHRD

Instruction Format:

SHRD DESTINATION_OPERAND, SOURCE_OPERAND, COUNT

Meaning:

Figure 4.14: SHRD

Shift Right Double (SHRD) shifts the bits n (count) times in the destination operand from the left to right and the empty bit placeholders in the destination operand are filled by the bits shifted out of the source operand. The CF flag will have the value of the last bit shifted out of the destination operand. The source operand will not be modified.

Flags Affected: CF OF PF SF ZF (AF undefined)

Example

MOV EAX, 40000001H    ; EAX is 40000001H and CF=1

MOV EBX, 50000001H    ; EBX is 50000001H and CF=1

SHRD EAX, EBX, 2         ; EAX = 0x50000000 and EAX = 0x50000001

                                           ; CF will set to 0

Processor Control Instructions

This includes instructions used in controlling the processor operations.

CLC

Instruction Format:

CLC

Meaning:

Clear the Carry Flag (CLC): This instruction clears the Carry Flag.

Flags Affected: CF

Example

MOV EAX, 40000001H       ; EAX is 40000001H and CF=0

SAL EAX, 2                      ; SAL EAX bits 2 times, EAX will become 0x00000004 and ; CF will set to 1

CLC                                      ; Clear the CF flag, CF=0

CLD

Instruction Format:

CLD

Meaning:

Clear the Direction Flag (CLD): This instruction clears the Direction Flag to 0.

Flags Affected: CF

CLI

Instruction Format:

CLI

Meaning:

Clear the Interrupt Flag (CLI): This instruction clears the Interrupt Flag to 0. Once the IF flag is reset, the processor will not respond to the interrupt signal.

Flags Affected: CF

CMC

Instruction Format:

CMC

Meaning:

Complement Carry (CMC): This instruction inverts the Carry Flag.

Flags Affected: CF

Example

MOV EAX, 40000001H       ; EAX is 40000001H and CF=0

SAL EAX, 2                      ; SAL EAX bits 2 times, EAX will become 0x00000004 and ; CF will set to 1

CMC                                     ; Toggle CF flag from 1 to 0, CF=0

ESC

Instruction Format:

ESC OPCODE SOURCE_OPERAND

OPCODE = D8 to DF

SOURCE_OPERAND = REGISTER OR MEMORY

Meaning:

Escape to Floating Point Coprocessor (ESC): This instruction passes the instructions to the coprocessor, also called floating point or math coprocessor. The microprocessor fetches the instruction bytes and the coprocessor also fetches these instruction bytes from the data bus and queues them. All the normal microprocessor instructions are treated as NOP by the coprocessor but when the ESC instruction is fetched by the microprocessor, the coprocessor decodes the instruction to carry out the action. When the ESC instruction is executed, the microprocessor provides the memory address otherwise perform NOP.

Flags Affected: None

LOCK

Instruction Format:

LOCK: [INSTRUCTION]

Meaning:

LOCK is the not an instruction but an instruction prefix. When LOCK is used as a prefix in front of any instruction, the processor LOCK pin is activated or also called asserted. When the LOCK pin is activated by the LOCK instruction, the external bus master and other peripherals are disabled until the instruction after LOCK is executed. So, LOCK is used in front of critical instructions that are to be executed without any disturbance to bus master (Bus Master is a program that controls the bus on which the address and control signals flow) and system bus (System Bus is a common term for address bus and data bus).

Flags Affected: None

Example

LOCK: MOV EAX, EBX  ; Activate the LOCK pin of the processor for the MOV instruction ; execution

NOP

Instruction Format:

NOP

Meaning:

No Operation (NOP): This instruction does nothing. It is used to eat up the processor time and memory.

Flags Affected: None

Example

NOP    ; Do nothing

STC

Instruction Format:

STC

Meaning:

Set the Carry Flag (STC): This instruction sets the Carry Flag to 1.

Flags Affected: CF

Example

CLC   ; Clear the CF flag, CF=0

STC   ; Set the CF flag, CF=1

STD

Instruction Format:

STD

Meaning:

Set the Direction Flag (STD): This instruction sets the Direction Flag to 1.

Flags Affected: DF

STI

Instruction Format:

STI

Meaning:

Set the Interrupt Flag (STI): This instruction sets the Interrupt Flag to 1. As already explained in above instructions, when the IF flag is enabled, the interrupt event or signal will cause the processor to interrupt the program execution. Whenever the interrupt occurs, the processor completes the current set of instructions and then starts the interrupt service routine (ISR) or interrupt handler. ISR is a routine which contains a set of instructions to handle specific interrupts. This ISR tells the processor what to do when an interrupt occurs.

The IRET instruction after ISR returns the execution back to the interrupted program.

Flags Affected: IF

String Instructions

This includes instructions used in handling string operations.

CMPS/CMPSB/CMPSW

Instruction Format:

This instruction can be visualized as a combination of CMP for compare + S for String + B for Byte or W for WORD or D for DWORD

CMPSB SOURCE, DESTINATION

CMPSW SOURCE, DESTINATION

CMPSD SOURCE, DESTINATION

Meaning:

This instruction is used to compare two strings where the source string is pointed by ESI and the destination string is pointed by EDI. When CMPSB is used, the comparison is done between every byte. When CMPSW is used, the comparison is done between every word. When CMPSD is used, the comparison is done between every DWORD. By comparison, it means subtracting a byte/word/dword pointed by the ESI and EDI registers, just as we studied in the CMP instruction.

The CMPS instruction is used with the prefixes REPE/REPZ, which means repeat the comparison until ECX=0 or ZF=0. We will cover REPE/REPZ later.

If Direction flag, DF=0, then ESI and EDI are incremented by 1 for byte, 2 for word, and 4 for dword after each move.

If Direction flag, DF=1 then ESI and EDI are decremented by 1 for byte, 2 for word, and 4 for dword after each move.

Flags Affected: AF,CF,OF,PF,SF,ZF

Example

Figure 4.15: CMPSB example

MOV ESI, 0x41000001     ; ESI pointing to STRING1

MOV EDI, 0x82000001     ; ESI pointing to STRING2

MOV ECX, 0x08                ; ECX is initialized to 0x08

CLD                                    ; Clear direction flag, DF=0

REPE CMPSB                   ; Compare two strings until ECX=0 or ZF=0

IN/INSB/INSW/INSD

This instruction can be visualized as a combination of IN for Input + S for String + B for Byte or W for WORD or D for DWORD.

Meaning:

There are two types of instructions used to transfer data between the processor input/output ports and the peripheral devices. These two types of instruction are as follows:

Register input/output instructions: These instructions move the data between the processor I/O port and the register.

Block (or string) input/output instructions: These instructions move blocks of data between the processor I/O port and the memory.

Register input instruction format:

IN DESTINATION SOURCE

Where, SOURCE is the port address

And DESTNATION is the register, EAX (32-bit) register or AX (16-bit) register or AL (8-bit) register. The destination register length determines the amount of data to be read from the input port, which can be byte, word, or dword. The destination can be some other general purpose register.

IN EAX, 80    ; It will read dword (32-bits) from port 80 and store it in EAX.

IN AX, 80       ; It will read word (16-bits) from port 80 and store it in AX.

Block (or string) input instruction format:

INSB/INSW/INSD

Where the processor input port address is specified in the DX register and the destination is the memory location pointed by the EDI register.

The INSB instruction moves a byte (8-bits) data from the input port address specified in the DX register to the memory location pointed by the EDI register. To transfer blocks of data between the processor input port and the memory, the INSB instruction is used with the repeat prefix REP. After every byte transfer between the input port and the memory pointed by EDI, EDI is incremented (when DF=0) or decremented (when DF=1) by 1.

The INSW instruction moves a word (16 bits) data from the input port address specified in the DX register to the memory location pointed by the EDI register. To transfer blocks of data between the processor input port and the memory, the INSW instruction is used with the repeat prefix REP. After every word transfer between the input port and the memory pointed by EDI, EDI is incremented (when DF=0) or decremented (when DF=1) by 2.

The INSD instruction moves a dword (32 bits) data from the input port address specified in the DX register to the memory location pointed by the EDI register. To transfer blocks of data between the processor input port and the memory, the INSD instruction is used with the repeat prefix REP. After every dword transfer between the input port and the memory pointed by EDI, EDI is incremented (when DF=0) or decremented (when DF=1) by 4.

Flags Affected: None

Example

MOV DX, PORT_ADDRESS    ; Move the input port address to the DX register

MOV EDI, offset STR                ; Move the memory offset for string to the EDI register

INSW                                          ; Move a word (16 bits) from the input port address

                                                    ; specified in the DX register to the memory location pointed

                                                    ; by EDI register

OUT/OUTSB/OUTSW/OUTSD

This instruction can be visualized as a combination of OUT for Output + S for String + B for Byte or W for WORD or D for DWORD.

Meaning:

The same concept from input instruction is reiterated here. There are two types of instructions used to transfer data between the processor input/output ports and the peripheral devices. These two types of instruction are as follows:

Register input/output instructions: These instructions move the data between the processor I/O port and the register.

Block (or string) input/output instructions: These instruction move blocks of data between the processor I/O port and the memory.

Register output instruction format:

OUT DESTINATION SOURCE

SOURCE is the register, EAX (32 bit) register or AX (16 bit) register or AL (8 bit) register. The source register length determines the amount of data to be output from the source register to the port, which can be byte, word, or dword. The source can also be some other general purpose register.

And DESTNATION is the port address

OUT 80, EAX     ; It will output dword (32-bits) from EAX to port 80.

OUT 80, AX        ; It will output word (16-bits) from AX to port 80.

Block (or string) output instruction format:

OUTSB/OUTSW/OUTSD

Where the processor output port address is specified in the DX register and the source is the memory location pointed by the ESI register.

The OUTSB instruction outputs a byte (8-bits) data from the memory location pointed by the ESI register to the output port address specified in the DX register. To transfer blocks of data between the memory and the processor output port, the OUTSB instruction is used with the repeat prefix REP. After every byte (8-bits) output from the memory pointed by ESI to the output port address specified in the DX register, ESI is incremented (when DF=0) or decremented (when DF=1) by 1.

The OUTSW instruction outputs a word (16-bits) data from the memory location pointed by the ESI register to the output port address specified in the DX register. To transfer blocks of data between the memory and the processor output port, the OUTSW instruction is used with the repeat prefix REP. After every word (16-bits) output from the memory pointed by ESI to the output port address specified in the DX register, ESI is incremented (when DF=0) or decremented (when DF=1) by 2.

The OUTSD instruction outputs a dword (32-bits) data from the memory location pointed by the ESI register to the output port address specified in the DX register. To transfer blocks of data between the memory and the processor output port, the OUTSD instruction is used with the repeat prefix REP. After every dword (32-bits) output from the memory pointed by ESI to the output port address specified in the DX register, ESI is incremented (when DF=0) or decremented (when DF=1) by 4.

Flags Affected: None

Example

MOV ESI, offset STR                   ; Move memory offset for string to the ESI register

MOV DX, PORT_ADDRESS       ; Move Output Port address to the DX register

OUTSW                                         ; Output a word (16 bits) from memory location pointed

                                                       ; by ESI register to output Port address in DX register

LODS/LODSB/LODSW/LODSD

Instruction Format:

LODSB

LODSW

LODSD

Meaning:

Load String (LODS) loads the string pointed by the ESI register to the EAX register.

Load String Byte (LODSB) loads a byte of string pointed by the ESI register to the AL register.

Load String Word (LODSW) loads a word of string pointed by the ESI register to the AX register.

Load String DWORD (LODSD) loads a dword of string pointed by the ESI register to the EAX register.

ESI is incremented (when DF=0) or decremented (when DF=1) by 1 for Load String Byte (LODSB), 2 for Load String Word (LODSW), and 4 for Load String DWORD (LODSD).

Flags Affected: None

Example

CLD                                 ; Clear Direction flag, DF=0

MOV ESI, Offset STR     ; Move memory offset of string to ESI register

LODSB                            ; Loads a byte of the string pointed by ESI register to AL

                                         ; register

STOS/STOSB/STOSW

Instruction Format:

STOSB

STOSW

STOSD

Meaning:

Store String (STOS) stores the string from the EAX register to the memory location pointed by the EDI register.

Store String Byte (STOSB) stores a byte of string from the AL register to the memory location pointed by the EDI register.

Store String Word (STOSW) stores a word of string from the AX register to the memory location pointed by the EDI register.

Store String DWORD (STOSD) stores a dword of string from the EAX register to the memory location pointed by the EDI register.

EDI is incremented (when DF=0) or decremented (when DF=1) by 1 for Store String Byte (STOSB), 2 for Store String Word (STOSW), and 4 for Store String DWORD (STOSD).

Flags Affected: None

Example

CLD                               ; Clear Direction flag, DF=0

MOV ESI, Offset STR   ; Move memory offset of string to ESI register

STOSB                           ; Stores a byte of the string from AL register to memory

                                       ; pointed by EDI register

SCAS/SCASB/SCASW

Instruction Format:

SCASB

SCASW

SCASD

Meaning:

Scan String (SCAS) scans the string in the memory location pointed by the EDI register and compares (or subtracts) it with the contents of the EAX register. The result of the comparison or subtraction is discarded and the status flags are updated accordingly.

Scan String Byte (SCASB) scans a byte of string in the memory location pointed by the EDI register and compares it with the contents of the AL register.

Scan String Word (SCASW) scans a word of string in the memory location pointed by the EDI register and compares it with the contents of the AX register.

Scan String DWORD (SCASD) scans a dword of string in the memory location pointed by the EDI register and compares it with the contents of the EAX register.

EDI is incremented (when DF=0) or decremented (when DF=1) by 1 for Scan String Byte (SCASB), 2 for Scan String Word (SCASW), and 4 for Scan String DWORD (SCASD).

Flags Affected: None

Example

MOV ECX, 100             ; Scan a string of 100 characters

MOV EDI, offset STR   ; Move memory offset of string to ESI register

MOV AL, 0x20              ; Scanning string for space character, space=0x20

REPNE SCASB             ; Repeat until ECX=0 or ZF=1

MOVS/MOVSB/MOVSW

Instruction Format:

MOVSB

MOVSW

MOVSD

Meaning:

Move String (MOVS) moves the string in the memory location pointed by the ESI register to the memory location pointed by the EDI register.

Move String Byte (MOVSB) moves a byte (8-bits) of string in the memory location pointed by the ESI register to the memory location pointed by the EDI register.

Move String Word (MOVSW) moves a word (16-bits) of string in the memory location pointed by the ESI register to the memory location pointed by the EDI register.

Move String DWORD (MOVSD) moves a dword (32-bits) of string in the memory location pointed by the ESI register to the memory location pointed by the EDI register.

ESI and EDI are incremented (when DF=0) or decremented (when DF=1) by 1 for Move String Byte (MOVSB), 2 for Move String Word (MOVSW), and 4 for Move String DWORD (MOVSD).

Flags Affected: None

Example

MOV ESI, SRC_STR   ; Move source string location in ESI

MOV EDI, DST_STR   ; Move destination string location in EDI

MOV ECX, 05H           ; Initialize ECX to 0x05, which is string length

CLD                              ; Clear the direction flag, DF=0

REP MOVSB               ; Moves the string of length 5 bytes from src to dest

REP

Instruction Format:

REP <STRING-INSTRUCTION>

Meaning:

Repeat (REP) is not an instruction. It is a prefix which is used before string instructions. When REP is used before a string instruction, it will repeat the instruction until ECX counter becomes 0. On every execution of the instruction after REP, the ECX counter is decremented by 1 until ECX=0.

Flags Affected: Depends on the instruction used after REP.

Example:

MOV DWORD PTR DS:[0x011E8000], 0x41424344     ;write ABCD to 0x011E8000

                                                                                           ; memory location, ABCD will be

                                                                                           ; written in reverse order (little

                                                                                           ; endian)

MOV ESI, 0X11E8000                                                     ; Move source string location in ESI

MOV EDI, 0X11E8010                                                    ; Move source string location in EDI

MOV ECX, 0X05                                                           ; Initialize ECX to 0x05, to run iteration 5 times

CLD                                                                                 ; Clear the direction flag, DF=0

REP MOVSB                                                                  ; Moves the ABCD from src

                                                                                         ; to dest, until ECX=0

REPE/REPZ

Instruction Format:

REPE <STRING-INSTRUCTION>

REPZ <STRING-INSTRUCTION>

Meaning:

Repeat if Equal (REPE) and Repeat if Zero (REPZ) cause the preceding string instruction to repeat until ECX=0 or Zero flag (ZF) = 0. These prefixes are used with the CMPS and SCAS instructions.

Flags Affected: Depends on the instruction used after REPE/REPZ instruction.

REPNE/REPNZ

Instruction Format:

REPNE <STRING-INSTRUCTION>

REPNZ <STRING-INSTRUCTION>

Meaning:

Repeat if Not Equal (REPNE) and Repeat if Not Zero (REPNZ) cause the preceding string instruction to repeat until ECX=0 or ZF=1. These prefixes are used with the CMPS and SCAS instructions.

Flags Affected: Depends on instruction used after REPNE/REPNZ instruction.

Conclusion

In this chapter, we studied the explanation of major assembly instructions used in reverse engineering. We also covered the different types of instructions for stack, data transfer, arithmetic, program execution, branching, bit manipulation, processor control, and string. Examples with some instructions were also covered to elaborate the working of the instructions.

In the next chapter, we will talk about some stack based instructions in detail and understand the concept of code calling conventions. This concept is important from the reverse engineering point of view and you will come across this quite often in implementing reverse engineering.

CHAPTER 5

Types of Code Calling Conventions

In Chapter 2, Understanding Architecture of x86 Machines, we understood the concept of stack. Several things happen in the background when we call a function. Control is transferred to the new function in a way that the stack frame is allocated for local variables and parameters are passed for a callee to understand. On return, the return address is placed on the stack so that the caller can find it and a stack clean-up is performed as well. Imagine if the callee and the caller both clean up the stack, then it will create a disastrous situation where the stack is cleaned twice both by callee and caller. So, understanding the difference in code calling conventions becomes important. As there are many variants of CPUs, some CPUs have a strict protocol on how this is performed. But when we talk about the x86 architecture, it is all flexible where a programmer decides on how to call the methods. This is the reason that led to different calling conventions.

When you write C/C++ code wherein you use shared libraries, code calling convention becomes important, as the code you are interacting is beyond your control. If you are a programmer who writes C/C++ code, then as a programmer, you don’t have to worry as the compiler takes care of the calling convention for you. The compiler generally takes the default code convention automatically based on the language. In this chapter, we will understand the difference in code calling conventions.

Structure

In this chapter, we will cover the following topics:

Understand the types of calling conventions

Concept behind different calling conventions

Objective

After studying this chapter, you will be able to differentiate between different assembly codes with respect to the calling convention. If in case you receive some assembly code then, by going over the assembly listing, you will be able to evaluate the calling convention used. To understand the different code calling conventions, we will take up a pseudo code to run over the concept behind each calling convention in detail.

Understand types of calling conventions

To understand calling conventions, let’s take a basic pseudo code. In this code, function funcA calls another function (funcB). In this case, funcA is called the caller and funcB is called the callee.

FuncA()

{

Arg1;

Arg2;

FuncB(Arg1, Arg2);

}

Now when FuncB is called, it is called with 2 arguments (Arg1, Arg2). When this code is compiled with different compilers, it will generate different assembly codes. Calling convention is this set of rules that specify how C or C++ functions are converted into an assembly code. So, calling convention basically defines:

How arguments are passed to the function

How functions return values

How the caller calls the callee

How stack is managed when one function calls another

How stack is cleared

All these are defined by the calling convention method the compiler uses. In C/C++ language, there are three types of calling conventions majorly used: CDECL, STDCALL, and FASTCALL. We will walk through all these calling conventions one by one.

CDECL

CDECL can also be read as C Declaration. When CDECL calling convention is used:

Arguments are passed from the right to left order. In the same pseudo code, right to left order means Arg2 is first pushed on the stack and then Arg1 is pushed on the stack.

FuncA()

{

Arg1;

Arg2;

FuncB(Arg1, Arg2);

}

The function return value is passed into the EAX register.

Calling function, the caller cleans the stack.

STDCALL

STDCALL stands for Standard Call. This calling convention is defined by Microsoft as a standard calling convention for Win32 API. When STDCALL is used:

The first point is the same as CDECL. Arguments are passed from the right to left order.

The function return value is passed into the EAX register. This point is also the same as CDECL.

This point differs from CDECL where called function, callee cleans the stack.

FASTCALL

The main difference between CDECL/STDCALL and FASTCALL is that the initial arguments are not pushed on stack but rather passed in the registers. Keeping data in registers is faster than in memory, hence it is named FASTCALL. In FASTCALL, when the calling convention is used:

The first two or three parameters are passed in the registers EDX, ECX, or EAX. Additional parameters are passed on to the stack. Arguments are passed from the right to left order.

The function return value is passed into the EAX register.

Calling function, the caller cleans the stack if needed.

Concept behind different calling conventions

Now to understand the concept behind different calling conventions, we will write a simple code in C/C++. Then using cl.exe (VS compiler), we will compile the code with different calling conventions and understand the difference between the assembly codes generated.

The following image shows a simple C/C++ code that will add numbers:

Figure 5.1: AddNumber.cpp

The AddNumber.cpp code is a simple code of adding two numbers. A few points to notice in the preceding program are as follows:

The main function is the entry point of the program.

The main function is calling an addition function. So, the main function is the caller and the addition function is the callee.

The local variable defined in the main function and the addition function are of type integer.

Two parameters are passed to the callee, which are 4 and 5.

After writing the code, we will compile the code with cl.exe and use different switches that will force the compiler to change the calling convention.

/Gd: Use this switch for the CDCEL calling convention.

/Gz: Use this switch for the STDCALL calling convention.

/Gr: Use this switch for the FASTCALL calling convention.

We will take up each calling convention one by one and understand how they differ from each other.

CDECL

We will compile the AddNumber.cpp code with no optimization and with CDECL calling convention. We will use the /Gd switch for this. Run the commands given below on the Windows command prompt to set the environment for cl.exe (VS compiler) and then compile the code with the following switches.

/Gd: Use this switch for the CDCEL calling convention.

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\AddNumber\AddNumber

^

cl AddNumber.cpp /FaAddNumber-CDECL.asm /Gd /FeAddNumber-CDECL.exe

The following is the output of running the preceding commands:

Figure 5.2: CDECL

This will generate AddNumber-CDECL.asm. Now we will move on to analyze the assembly code generated in AddNumber-CDECL.asm: Our C++ code is divided into two functions, one is the main function (caller) and other is the addition function (callee). To understand CDECL calling convention, we will take up the main function code conversation from the C++ code to assembly.

Figure 5.3: Main function code in AddNumber.cpp

The assembly code of the main function becomes:

Figure 5.4: Main proc assembly code in AddNumber-CDECL.asm

Let’s analyze the assembly code of the main function in asm file:

Line 20-21 is function prolong. These are a sequence of instructions to start a function.

In line 22, the objective of PUSH ECX after the function prologue is not to save ECX on stack but to allocate 4 bytes on the stack for storing local variables, which is add.

Add variable can be accessed with the help of the _add$ macro, which is equal to -4. So, Add can be accessed at [EBP - 0x4].

From line 24, we will understand the concept of CDECL. Let’s recall the CDCEL calling convention points one by one to understand the concept practically:

Arguments are passed from the right to left order.

At line 12 of AddNumber.cpp, which is int add=addition(4,5);, we are passing 4,5 parameters to the addition function. Now, from right to left means 5 will be pushed on the stack first and then 4 will be pushed on the stack, as done in AddNumber-CDECL.asm code at line 24-25. After the parameters are pushed on the stack, the addition function is called at line 26 of AddNumber-CDECL.asm.

Calling function, caller cleans the stack.

After returning from the addition function, at line 27 of AddNumber-CDECL.asm instruction add esp, 8 shrinks the stack by 8. This is because the caller (which is main in our case) cleans up the stack as per CDECL's calling convention.

The function return values are passed into the EAX register.

The call to addition function at line 26 of AddNumber-CDECL.asm returns the addition function return value to the EAX register, which is then copied to add the variable location, [EBP – 0x4]. This copy operation is done by the instruction at line 28 of AddNumber-CDECL.asm:

mov DWORD PTR _add$[ebp], eax

Similarly, the main function returns 0, which is achieved by xor eax, eax instruction.

If you are not able to understand any instruction, please refer to Chapter 4, Walk Through On Assembly Instructions.

STDCALL

We will compile the code with no optimization and with STDCALL calling convention. We will use the /Gz switch for this. Run the commands given below on the Windows command prompt to set the environment for cl.exe (VS compiler) and then compile the code with the following switches:

/Gz: Use this switch for the STDCALL calling convention.

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\AddNumber\AddNumber

^

cl AddNumber.cpp /FaAddNumber-STDCALL.asm /Gz /FeAddNumber-STDCALL.exe

The following is the output of running the preceding commands:

Figure 5.5: STDCALL

This will generate AddNumber-STDCALL.asm. Now we will move on to analyze the assembly code in AddNumber-STDCALL.asm. To understand the STDCALL calling convention, we will again take up the main function code along with the addition function code to the see conversation from the C++ code to assembly.

Figure 5.6: Main & addition function code in AddNumber.cpp

In the STDCALL calling convention, the assembly code of the main function and the addition function becomes:

Figure 5.7: Main & addition proc assembly code in AddNumber-STDCALL.asm

Let’s analyze the assembly code with respect to STDCALL. Most of the points will be the same as that of CDCEL calling convention. We will discuss the few differences here:

In ASM code line 20-21 is function prolong. These are a sequence of instructions to start a function.

In ASM code line 22, the objective of PUSH ECX after the function prologue, is not to save ECX on stack but to allocate 4 bytes on the stack for storing local variable, which is add.

Add variable can be accessed with the help of the _add$ macro, which is equal to -4. So Add variable can be accessed at [EBP – 0x4].

From line 24 we will understand the concept of STDCALL. Let’s recall STDCALL points one by one:

Arguments are passed in from the right to left order, which is same as CDCEL

At line 12 of AddNumber.cpp, which is int add=addition(4,5);, we are passing 4,5 parameters to the addition function. Now from right to feft means 5 will be pushed on the stack first and then 4 will be pushed on the stack. This can be seen at line 26 of AddNumber-STDCALL.asm by PUSH 5 & PUSH 4 instructions before the addition function is called.

Called function, callee cleans the stack. This point differs from that in CDCEL.

This can be seen at line 47 of AddNumber-STDCALL.asm, by the RET 8 instruction. The callee cleans up the stack by using the RET nBytes instruction, where the RET instruction transfers control from the callee to the caller to the return address saved on the stack. nBytes, which in our case, is 8. So, 8 bytes are released on the stack to clean up the stack.

The function return values are passed into the EAX register.

The call to addition function at line 26 of AddNumber-STDCALL.asm returns the addition function return value to the EAX register, which is then copied to add the variable location, [EBP – 0x4]. This copy operation is done by the following instruction:

mov DWORD PTR _add$[ebp], eax

at line 27 of AddNumber-STDCALL.asm

Similarly, the main function returns 0, which is achieved by xor eax, eax instruction.

FASTCALL

As we read previously, the FASTCALL calling convention differs majorly in passing arguments. To understand this, we will compile the code with the optimization off and with the FASTCALL switch. We will use the /Gr switch for this. Run the commands given below on the Windows command prompt to set the environment for cl.exe (VS compiler) and then compile the code with the following switches:

/Gr: Use this switch for the FASTCALL calling convention.

/Fa: Name of output assembly listing file

/Fe: Name of output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\AddNumber\AddNumber

^

cl AddNumber.cpp /FaAddNumber-FASTCALL.asm /Gr /FeAddNumber-FASTCALL.exe

The following is the output of running the preceding commands:

Figure 5.8: FASTCALL

This will generate AddNumber-FASTCALL.asm. To analyze the assembly code in AddNumber-FASTCALL.asm, we will once again see our C/C++ code.

Figure 5.9: Main & addition function code in AddNumber.cpp

Let’s see AddNumber-FASTCALL.asm generated as follows:

Figure 5.10: Main & addition proc assembly code in AddNumber-FASTCALL.asm

Let’s analyze the assembly code in the same order we did for the other calling conventions.

The function prolong instruction on line 20-21 is a sequence of instructions to start a function.

In ASM code line 22, PUSH ECX is the same. It is not to save ECX on stack but to allocate 4 bytes on the stack for storing the local variable, which is add.

The add variable can be accessed with the help of the _add$ macro, which is equal to -4. So, add can be accessed at [EBP – 0x4].

From line 24, we will see a different approach to pass arguments to function. To understand, let’s recall the FASTCALL concept:

Initial arguments are not pushed on stack but rather passed in the registers. The first two or three arguments are passed in the registers EDX, ECX, or EAX. Additional arguments are passed on to the stack. Arguments are passed from the right to left order.

By now, we are clear with the right to left approach. The point to note here is that argument 5 is moved to EDX and argument 4 is moved to ECX, as done at line 24-25 of AddNumber-FASTCALL.asm. Once the arguments are moved to registers, the call to addition function is made at line 26 of AddNumber-FASTCALL.asm.

During the addition function execution, we can see that the arguments in EDX, ECX are passed to the stack for further processing between lines 45-49 of AddNumber-FASTCALL.asm.

Calling function, caller cleans the stack.

Arguments are passed to EDX, ECX so no arguments are passed to stack, no stack cleanup is needed.

Function return values are passed into EAX register this is same as other 2 calling convention we discussed. Restating same point about FASTCALL.

Call to addition function at line 26 of AddNumber-FASTCALL.asm returns the addition function return value to EAX register, which then is copied to add variable location, [EBP – 0x4]. This copy operation is done by this instruction:

mov DWORD PTR _add$[ebp], eax

at line 27 of AddNumber-FASTCALL.asm

Similarly, the main function returns 0, which is achieved by xor eax, eax instruction at line 29 of AddNumber-FASTCALL.asm.

Conclusion

In this chapter, we covered three types of code calling conventions majorly used: CDCEL, STDCALL, and FASTCALL. We learned that in CDCEL and STDCALL, arguments are passed from the right to left order and the function return value is passed into the EAX register. In the CDCEL calling function, the caller cleans the stack and in STDCALL, callee cleans the stack.

In FASTCALL, the initial arguments are not pushed on the stack but rather passed in the registers. The rest function return value is passed into the EAX register and in calling function, the caller cleans the stack if needed. In the next chapter, we will take C/C++ codes and compile them to understand assembly output.

CHAPTER 6

Reverse Engineering Pattern of Basic Code

In this chapter, we will write small pieces of code and compile them to understand assembly output. We will walk through step-by-step instructions in the assembly code and understand code flow from the assembly point of view.

Throughout this chapter of compiling small pieces of code, we will use Microsoft compiler on the 32-bit environment. All the programs are compiled on Microsoft Windows 32-bit environment. We will also use code optimization during our analysis.

Structure

In this chapter, we will cover the following topics:

What is code optimization

Understanding assembly pattern of the C/C++ program

Concept of code optimization

Tools used to generate the assembly pattern of C/C++ program

Objective

After studying this chapter, you should be able to:

Understand code optimization and its importance

Assembly code with and without optimization

What is Code Optimization?

Optimization means doing something at its best in order to effectively utilize resources. Code optimization means to transform the code to remove unnecessary lines, so as to consume fewer resources (Memory, CPU and others) during execution. When code is optimized by compilers, the following things are taken care of:

The meaning of the code should not be changed while optimizing the code.

An optimized code should consume fewer resources.

Optimization should not impact the compiling time of the program.

Let’s begin with a small C/C++ program and gradually move to complex programs. This process will help you understand the pattern of code in assembly language with respect to C/C++ applications.

Empty function

An empty function is something that does nothing. Let’s create an empty function in C/C++ code. Here, we are defining and declaring an empty function as EmptyFunction().

Figure 6.1: EmptyFunction.cpp

Empty Function without Optimization

Now, let’s compile it without optimization using the MSVC compiler (cl.exe). Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\EmptyFunction\EmptyFunction

^

cl EmptyFunction.cpp /FaEmptyFunction.asm /FeEmptyFunction.exe

The following is the output of running the preceding commands:

Figure 6.2: Empty Function without Optimization

The assembly code generated without optimization is as follows:

Figure 6.3: EmptyFunction.asm

We will walk through the assembly code line by line to understand the meaning and working of the code pattern. Line 1 says:

▼Line 1

; Listing generated by Microsoft (R) Optimizing Compiler Version 16.00.30319.01

This is a comment as it starts with a semicolon. This comment states the compiler that we are using to generate the assembly code, Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 16.00.30319.01 for 80x86. The compiler basically translates the program from one language to another. But when we talk about optimizing compiler, it improves the code to run faster.

▼Line 3

TITLE C:\JitenderN\REBook\EmptyFunction\EmptyFunction\EmptyFunction.cpp

Title defines the name of the absolute path of the C/C++ program.

▼Line 4

.686P

This enables all the instructions for the Pentium Pro processor (32-bit MASM only).

▼Line 5

.XMM

This means that the program requires a CPU with the Streaming SIMD Extensions instruction set.

▼Line 6

include listing.inc

This listing.inc file contains the assembler macros. Macros are used in assembly language for modular programming. Visual C++ does not embed the macro code in the assembly code to improve performance and align the code. You can find this file in the Visual C++ include folder.

Figure 6.4: listing.inc path

▼Line 7

.model flat

This is directive for enabling the flat memory model. To understand memory models, we have to understand that memory is accessed using 3 memory models:

Flat: This is a non-segmented memory model where the whole memory appears to a program as one massive array of bytes. Code, data, and stack all reside in the same address space.

Segmented: In this model, memory is divided into groups of spaces called segments.

Real: The real address memory model is a very old memory model used in the Intel 8086 processor.

▼Line 9-10

INCLUDELIB LIBCMT

INCLUDELIB OLDNAMES

With INCLUDELIB derivative, we link LIBCMT.LIB and OLDNAMES.LIB libraries located in the following path:

Figure 6.5: Libraries path

▼Line 12

PUBLIC _main

PUBLIC is the derivative which makes the _main procedure public. A derivative is an instruction which is used by the assembler to automate the assembly process; it also helps improve the code readability.

_main, all functions begin with an underscore. main function is labeled as a public function, which means it can be accessed by other modules.

▼Line 13

; Function compile flags: /Odtp

All the comments begin with semicolons. This line states that the code is compiled with the /Odtp switch.

▼Line 14

_TEXT SEGMENT

This is the start of the text segment or, we can say, the code segment.

▼Line 15

_main PROC

Procedures are defined with the PROC statement and they must be closed by the ENDP statement. We can see _main ENDP on line 25.

▼ Line 16

; File c:\jitendern\rebook\emptyfunction\emptyfunction\emptyfunction.cpp

This is a comment stating the file path of the C/C++ source code.

▼Line 17

; Line 9

This is a comment which states that line number 9 of the C/C++ source code file is mapped with the instruction following this comment.

▼Line 18-19

push ebp

mov ebp, esp

This is function prologue, which is the sequence of instructions at the start of a function.

▼Line 20-21

; Line 10

xor eax, eax

XOR is an exclusive OR. The EAX register is used for storing the return value of the function. The main function is returning 0 in the C/C++ code, so XOR EAX with EAX will set the EAX register to zero. Also, compilers can use MOV EAX, 0 in place of XOR EAX, EAX. But XOR is preferred over MOV as XOR occupies 2-byte opcode and an MOV instruction occupies 5 bytes.

▼Line 22

; Line 11

This comment states that line number 12 of the C/C++ source code file is mapped with the instruction following this comment.

▼Line 23

pop ebp

POP EBP is a function epilogue, which is the sequence of instructions to end a function.

▼Line 24

ret 0

RET is the return instruction. It returns the instruction pointer to the caller procedure. The syntax of the RET instruction is:

RET nBytes

The return instruction has an optional nBytes operand that specifies the number of bytes to be added to the value of the ESP register after the return.

▼Line 25-26

_main ENDP

_TEXT ENDS

This is the close of the main procedure and the end of the text segment or code segment.

▼Line 27

PUBLIC ?EmptyFunction@@YAXXZ    ; EmptyFunction

Internally, functions are represented by their decorated names, which are encoded string created during the compilation process. It appends the calling convention, function return type, function parameters, and other information with the function name. This process helps the linker find the correct function when linking an executable. This process of name decoration is also known as name mangling.

Like main, EmptyFunction is made public with the help of the PUBLIC derivative.

▼Line 28

; Function compile flags: /Odtp

It is the same as explained previously. It states that the code is compiled with the /Odtp switch.

▼Line 29

_TEXT SEGMENT

This is the start of the text segment or code segment.

▼Line 30

?EmptyFunction@@YAXXZ PROC    ; EmptyFunction

Starting of EmptyFunction procedure with PROC statement

▼Line 31

; Line 15

This comment states that line number 15 of the C/C++ source code file is mapped with the instruction following this comment.

▼Line 32-33

push ebp

mov ebp, esp

This is a function prologue for EmptyFunction.

▼Line 34

; Line 16

This comment states that line number 17 of the C/C++ source code file is mapped with the instruction following this comment.

▼Line 35

pop ebp

POP EBP is a function epilogue of EmptyFunction.

▼Line 36

ret 0

As EmptyFunction is doing nothing, it’s just an empty/blank function. So, it’s retuning the instruction pointer back to the caller. 0 means that the ESP will be unchanged.

▼Line 37-38

?EmptyFunction@@YAXXZ ENDP    ; EmptyFunction

_TEXT ENDS

This is the close of the EmptyFunction procedure and the end of the text segment or code segment.

▼Line 39

END

The END statement ends the source code.

Empty Function with Optimization

Now let’s compile the code with optimization using the /0x switch on the x86 platform. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Ox: Enable maximum optimization

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\EmptyFunction\EmptyFunction

^

cl EmptyFunction.cpp /FaEmptyFunction-Optimized.asm /Ox /FeEmptyFunction-Optimized.exe

The following is the output of running the preceding commands:

Figure 6.6: Empty Function with Optimization

The assembly code we get is as follows:

Figure 6.7: EmptyFunction-Optimized.asm

As we can see, when the code is optimized, it transforms the code to remove the unnecessary lines. During this optimization, the meaning of the code remains the same. Compilers nowadays are good at optimization. So as a reverse engineer, it is always a good practice to understand the concept or the logic behind the code rather than the original code. If we can understand the logic then we can write our prototype.

Coming back to the code, we can see most of the lines of the code are the same as explained in the earlier section. So, we will not restate all of them. We will take up instructions that are specific to the code optimization.

▼Line 18

xor eax, eax

main procedure is XOR’ing EAX to return zero, as the EAX register is used for storing the return value of the function.

▼Line 28

ret 0

EmptyFunction is returning the instruction pointer to the caller with just the RET instruction.

Returning Value

In this section, we will create a function in the C/C++ code to return a constant value. We will define and declare a ReturningValue().

Figure 6.8: ReturningValues.cpp

Returning Value without Optimization

Compile the code without optimization in the MSVC compiler on the same x86 Windows machine. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\ReturningValue\ReturningValue

^

cl ReturningValue.cpp /FaReturningValue.asm /FeReturningValue.exe

The following is the output of running the preceding commands:

Figure 6.9: Returning a Value without Optimization

Here is what we get after compiling:

Figure 6.10: ReturningValue.asm

We have discussed most of the lines in the code in earlier section. So, we will focus on the main instructions.

▼Line 20-24

; Line 10

xor eax, eax

; Line 11

pop ebp

ret 0

This is the part of instructions from the main function, where we are XOR’ing EAX to make EAX equal to zero. Once the EAX is zero, we are calling function epilogue using the POP instruction. The RET instruction returns the execution back to the caller, where the caller can take the return value from the EAX register.

▼Line 34-38

; Line 15

mov eax, 2020    ; 000007e4H

; Line 16

pop ebp

ret 0

Now, we move to the ReturningValue function, where EAX is filled with 2020, which is the return value of the ReturningValue function. The POP instruction is function epilogue and finally, RET passes the instruction pointer back to the caller, where the caller will take the result from the EAX register.

Returning Value with Optimization

Compile the code with optimization (with /0x switch) in the MSVC compiler on the x86 platform. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Ox: Enable maximum optimization

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\ReturningValue\ReturningValue

^

cl ReturningValue.cpp /FaReturningValue-Optimized.asm /Ox /FeReturningValue-Optimized.exe

The following is the output of running the preceding commands:

Figure 6.11: Returning Value with Optimization

Assembly code generated will be:

Figure 6.12: ReturningValue-Optimized.asm

We can observe in the optimized code that all the unnecessary code lines are removed.

The main function shows only 2 instructions:

▼Line 18,20

xor eax, eax

ret 0

EAX is XOR’ed to reset EAX to 0, and the return value is stored in EAX. The RET instruction passes the instruction pointer back to the caller.

The ReturningValue function also shows only 2 instructions:

▼Line 28, 30

mov eax, 2020    ; 000007e4H

ret 0

MOV fills EAX with the return value of 2020 and the RET instruction passes the instruction pointer back to the caller.

Basic “Hello, World” Program

In this simple C/C++ code, we are just printing “hello, world” on the console. We are printing it using the printf() function.

Figure 6.13: HelloWorld.cpp

Basic “Hello, World” Program without Optimization

Compile the code without optimization with the MSVC compiler on the x86 platform. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\HelloWorld\HelloWorld

^

cl HelloWorld.cpp /FaHelloWorld.asm /FeHelloWorld.exe

The following is the output of running the preceding commands:

Figure 6.14: Basic “Hello, World” program without Optimization

Following is the generated assembly code which we shall now analyze:

Figure 6.15: HelloWorld.asm

Let’s walk through the assembly code line by line:

▼Line 1-10

We have already discussed this in the EmptyFunction section.

▼Line 12-14

CONST SEGMENT

$SG4677 DB 'hello, world', 0aH, 00H

CONST ENDS

The string constant, which in our case is “hello, world”, is allocated in the constant segment. The CONST SEGMENT derivative is used to define the start of the constant segment in the memory. In our case linker renamed CONST SEGMENT to .rdata, which can be dumped using any debugger. In the following screenshot, you can see that we have opened the EXE file (generated after compilation) in x32dbg and in this EXE file, we have disabled the Address Space Layout Randomization (ASLR) using CFF Explorer. Follow the steps mentioned in the Appendix to disable ASLR on an EXE file. Disabling ASLR on an EXE file will help us load the EXE file on the same memory space every time the EXE file is opened in the debugger:

Figure 6.16: .rdata

$SG4677 is the internal name given by a compiler to handle the string constant. DB, Defines the byte, which is the data type. 'hello, world', 0aH, 00H is the string data, which is null terminated ASCII string.

By CONST END, the constant segment is ended.

▼Line 15

PUBLIC _main

PUBLIC is the derivative which makes the _main procedure public, to be accessed by other modules.

▼Line 16

EXTRN _printf:PROC

Note: A code is placed in the .code segment, a constant string is placed in the CONST (.rdata) segment and if it is not a constant, it is placed in the .data segment.

EXTRN derivative declares the extern function, which is printf in our case. All functions begin with an underscore.

▼Line 18

_TEXT SEGMENT

This starts the _TEXT segment or code segment, where our main function code resides.

▼Line 19

_main PROC

This is the start of the main procedure.

▼Line 20-23

; File c:\jitendern\rebook\helloworld\helloworld\helloworld.cpp

; Line 8

push ebp

mov ebp, esp

Here, it’s the same as we discussed earlier that all comments begin with semicolons. One comment is stating the C/C++ source code file path and the other comment is defining the line number in C/C++ source code is mapped with the instruction following this comment. With PUSH, the main function prologue code starts.

▼Line 25-27

push OFFSET $SG4677

call _printf

add esp, 4

Before calling the printf function, we push the pointer to our constant string onto stack with the help of the PUSH instruction. The CALL instruction is calling the printf function.

After the execution of the printf function, the control is transferred back to the caller, which is main function in our case. Throughout the execution of the printf function, the pointer to the string will be on the stack. So, when the execution is returned back to the main function, the stack needs to be cleaned as we don’t need the string pointer anymore.

Since we are following the CDECL calling convention, it is the caller’s responsibility to clean up stack, which in our case is done using the add esp, 4 instruction.

A 32-bit program uses 4 bytes for addressing. So, when we add 4 bytes to ESP, we increment ESP by 4 bytes to clean up the stack and remove the constant string pointer on the stack. An equivalent of ADD instruction can also be POP <some-register>, which is often used by other compilers.

▼Line 28-29

; Line 10

xor eax, eax

As main is returning 0 in C/C++ code and we know that the return value of the function is stored in the EAX register, EAX is XOR’ed to return 0.

▼Line 30-32

; Line 11

pop ebp

ret 0

It is calling the main function epilogue code and the RET instruction returns execution back to the caller, where the caller can take the return value from the EAX register.

▼Line 33

_main ENDP

With this, the _main function is closed.

▼Line 34-35

_TEXT ENDS

END

This is ending the _TEXT, code segment and source code.

Basic “Hello, World” Program with Optimization

Compile the code with the optimization flag, /Ox flag. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Ox: Enable maximum optimization

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\HelloWorld\HelloWorld

^

cl HelloWorld.cpp /FaHelloWorld-Optimized.asm /Ox /FeHelloWorld-Optimized.exe

The following is the output of running the preceding commands:

Figure 6.17: Basic “Hello, World” program with Optimization

The generate assembly code will be as follows:

Figure 6.18: HelloWorld-Optimized.asm

All the code lines are the same as we discussed in preceding section. The only difference is that the function prologue and the epilogue code are removed by the compilers to consume fewer resources (Memory, CPU, and so on).

All the rest of the instructions are the same as we discussed in the without optimization section. Point to note here is that the meaning of the code is the same as that in the without optimization code. We are pushing the pointer to the constant string on the stack to call the printf function.

The printf function upon execution returns the execution back to main (caller) to clean up stack and fill EAX with 0 to return what is in the EAX.

Conclusion

In this chapter we understood the concept of code optimization. We took examples of empty function, function returning value and printing “hello, world” programs to understand the assembly listing of optimized and non-optimized code. In the next chapter we will talk about the code optimization concept on the programs with printf function and also we will discuss on how Integer, Float and char variables are stored in the memory.

CHAPTER 7

Reverse Engineering Pattern of Printf Program

Every time we write a program, we use the printf function to print something or the other on the output screen. It can be something for the end user of the program or it can be something for the debugging purpose or a simple welcome message. The same logic is followed by malware or virus writers. Programs are coded to print something or the other using the printf function. So as a reverse engineer, we should understand the printf function pattern while reversing any program coded to behave as a virus or a malware. Most of the virus or malware writers while coding print something or the other for their own purpose or for the target to act upon.

So, it is important to understand the programs that use the printf function to print messages on the console. Along with this, we will also discuss the usage of printf with different variables. Different types of variables allocate different amounts of memory, which is quite interesting to know. In this chapter, we take C/C++ program that uses the printf function to print integer, float, and char on the console. Each program will be taken separately to understand the pattern of the printf program when reverse engineered.

Structure

In this chapter, we will cover the following topics:

Understanding the assembly pattern of printf with Integer

Understanding the assembly pattern of printf with Float

Understanding the assembly pattern of printf with Char

Objective

After studying this chapter, we will be able to understand the code optimization concept on programs with the printf function. We will understand how assembly code, with optimization, is different from without optimization. During this, we will also discuss how Integer, Float, and Char variables are stored in memory. A floating point variable takes a different approach in working as compared to integer or char. We will cover the approach with detailed examples.

Function printf with Integers

In this simple C/C++ code, we are printing 4 integers on the console. We are using the printf function to print integers.

Figure 7.1: printfWithIntegers.cpp

Function printf Printing Integers without Optimization

Compile the code without optimization with the MSVC compiler on the x86 platform. Run the following commands on the Windows Command Prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\printfWithIntegers\printfWithIntegers

^

cl printfWithIntegers.cpp /FaprintfWithIntegers.asm /FeprintfWithIntegers.exe

The following is the output of running the preceding commands:

Figure 7.2: Function printf printing Integers without optimization

The generated assembly code will be as follows:

Figure 7.3: printfWithIntegers.asm

As we have already discussed the initial instructions of assembly code generated in Chapter 6, Reverse Engineering Pattern of Basic Code, so we will start with line 12.

▼Line 12-14

CONST SEGMENT

$SG4677 DB 'integer1=%d; integer2=%d; integer3=%d, integer4=%d', 00H

CONST ENDS

It is the start and the end of CONST SEGMENT in memory, within which the string constant $SG4677 is defined. This can be viewed in the memory by dumping .rdata (linker renamed CONST SEGMENT to .rdata) using any debugger. You can view the string constant using the x32dbg debugger as follows:

Figure 7.4: .rdata

▼Line 15

PUBLIC _main

All functions begin with an underscore. The main function is labeled as public function, which means it can be accessed by other modules.

▼Line 16

EXTRN _printf:PROC

With the EXTRN derivative, it is defining the external symbol of the name _printf and the type procedure.

Tip: Syntax of EXTRN derivative is label:type, where label can be variable/function and Label Type can be as below:

Label Type

Meaning

BYTE

Variable of 8 bits

WORD

Variable of 16 bits

DWORD

Variable of 32 bits

QWORD

Variable of 64 bits

PROC

Procedure Name

Table 7.1

▼Line 18, 19

_TEXT SEGMENT

_main PROC

This starts the _TEXT segment where our main function code resides. This can also be visualized in the x32dbg debugger:

.text segment of printfWithIntegers.exe starts from the 0x00401000 address and we can see that the main function/procedure code starts from the same address.

Figure 7.5: .text

▼Line 20-23

; File c:\jitendern\rebook\helloworld\helloworld\helloworld.cpp

; Line 8

push ebp

mov ebp, esp

This is the same as earlier. This comment states the C/C++ source code file path and other defining line number in C/C++ source code that is mapped with the instruction following this comment. With PUSH instruction, the main function prologue code starts.

▼Line 25-30

push4

push3

push2

push1

push OFFSET $SG4677

call_printf

From here, something interesting begins. All the arguments to printf function are pushed onto the stack in a reverse order. Each argument is of type integer. In a 32-bit environment, each integer occupies 4 bytes in size. To understand the stack state during execution, we can put breakpoint on the call of the printf function in the x32dbg debugger. Once the breakpoint is set, we will run the code to see the stack state when breakpoint is hit. This will help us visualize how the arguments to printf are pushed onto the stack.

Figure 7.6: Breakpoint on the call of printf

We can see that the argument integer 4 is pushed first on the stack and then 3, 2, and 1 are pushed. The following is the explanation for the stack state when breakpoint is hit:

0012FF2C  00408140  "integer1=%d; integer2=%d; integer3=%d, integer4=%d"

0012FF30  00000001  Argument 1 of type int is pushed on stack

0012FF34  00000002  Argument 2 of type int is pushed on stack

0012FF38  00000003  Argument 3 of type int is pushed on stack

0012FF3C  00000004  Argument 4 of type int is pushed on stack

At the 0x0012FF2C location, the pointer to the constant string (which is 0x00408140) is pushed on the stack. We can dump the location in x32dbg to view the constant string, $SG4677.

Figure 7.7: String dumped

Once all the arguments are pushed on the stack, the call to the printf function is made by:

▼Line 30

call _printf

This will execute the printf function and after the execution of the printf function, the instruction pointer will return the execution pointer back to the caller.

▼Line 31

add esp, 20     ; 00000014H

As we are using the CDECL calling convention, the caller cleans up the stack. So, after returning from the printf function, add esp, 20 shrinks the stack by 0x20.

20 bytes is calculated by adding the size of 4 arguments, 4 bytes each plus one pointer argument to the constant string of 4 bytes size. This makes a total of 4x5 bytes, which is equal to 20 bytes (00000014H in Hex).

Now, we will understand something more interesting related to garbage on the stack. To understand the concept of garbage, we put breakpoint on the instruction next to add esp, 20. We see something like the following on the stack:

Figure 7.8: Garbage on stack

The caller, as per the CDECL calling convention, is responsible for cleaning the stack, which is done using the ADD instruction by moving ESP back by 20 bytes. As we can see, ESP is moved backed to 0x0012FF40, but all the arguments and pointer to the constant string are still on the stack. These values are not cleared or set to zeros. Everything above the ESP value is garbage with no meaning.

▼Line 34-39

; Line 9

xor eax, eax

; Line 10

pop ebp

ret 0

_main ENDP

_TEXT ENDS

END

All the remaining instructions are the same as we discussed in the chapters before. The main function is returning a zero by making EAX to zero.

With END, everything is closed and the program is ended.

Function printf Printing Integers with Optimization

Compile the code with optimization flag, /Ox flag. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Ox: Enable maximum optimization

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\printfWithIntegers\printfWithIntegers

^

cl printfWithIntegers.cpp /FaprintfWithIntegers-Optimized.asm /Ox /FeprintfWithIntegers-Optimized.exe

The following is the output of running the preceding commands:

Figure 7.9: Function printf printing integers with optimization

The generated assembly code will be as follows:

Figure 7.10: printfWithIntegers-Optimized.asm

In the optimized code, everything is the same except the main function prologue and the epilogue code is removed by compilers to consume fewer resources.

Function printf with Float

Most of the calculations are done using integers, but when it comes to accuracy, floating point plays an important role. The earlier x86 processor family has separate coprocessors for mathematical calculations that process floating point numbers. But later on, the capability of handling floating point numbers was integrated into the microprocessor itself. This unit which was integrated into the microprocessor to handle the floating point numbers is called the Floating Point Unit (FPU). Now to handle the floating point, two things are required:

There should be space to store the floating point numbers.

There must be instructions to handle and do operations on the floating point numbers.

Now, regarding the space to store floating point numbers, FPU has 8 registers that forms a stack, i.e., from ST0 to ST7. FPU is also referred to as the “x87” section or “FPU Register Stack” the "x87 Stack". Instructions to handle the floating point numbers are referred to as the "x87 instruction set".

The floating point numbers are generally 32-bit long for float type and 64-bit long for double type. So, to maintain maximum accuracy of the floating numbers, the FPU stack registers are 80-bit wide. To understand more about floating point, we will take a simple C/C++ code to print two floating numbers on the console. We are using printf to print the floating numbers.

Figure 7.11: printfWithFloat.cpp

Function printf Printing Float without Optimization

Compile the code without optimization in the MSVC compiler on the same x86 Windows machine. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\printfWithFloat\printfWithFloat

^

cl printfWithFloat.cpp /FaprintfWithFloat.asm /FeprintfWithFloat.exe

The following is the output of running the preceding commands:

Figure 7.12: Function printf printing Float without Optimization

The generated assembly code will be as follows:

Figure 7.13: printfWithFloat.asm

Let’s start understanding the assembly code generated line by line:

▼Line 1-10

; Listing generated by Microsoft (R) Optimizing Compiler Version 16.00.30319.01

TITLE C:\JitenderN\REBook\printfWithFloat\printfWithFloat\printfWithFloat.cpp

.686P

.XMM

include listing.inc

.model flat

INCLUDELIB LIBCMT

INCLUDELIB OLDNAMES

We have already discussed this in the earlier section, so we will move on to the next instruction.

▼Line 12-14

CONST SEGMENT

$SG4677 DB 'float1=%f, float2=%f', 00H

CONST ENDS

This is the start of CONST SEGMENT named by the linker as .rdata. The compiler is using the $SG4677 name to handle the string constant. DB is the data type to define byte and the string is terminated with null, 00H. The .rdata can be dumped to view the constant string.

Figure 7.14: .rdata

▼Line 15-16

PUBLIC __real@3ff0000000000000

PUBLIC __real@40011eb851eb851f

Floating point numbers can be represented in a binary number format and this binary numbering format is standardized. Real number format, short/single, and long/double real numbers are available in three sizes:

REAL4, 32-bit, short real or single precision

REAL8, 64-bit, long real or double precision

REAL10, 80-bit, temporary real or extended precision

REAL4, REAL8, REAL10 have different formats: The format for REAL4 is:

Figure 7.15: REAL4 format

Where:

S = sign bit (0=positive, 1=negative)

E = exponent bits

F = fraction bits of the significand

The format for REAL8 is:

Figure 7.16: REAL8 format

The format for REAL10 is:

Figure 7.17: REAL10 format

To keep it simple and easy to understand, we will not discuss in detail the individual formats. Keeping the real numbering format (REAL8, which is 64 bit for long real or double precision is used in our code) in mind, we will move to the same instructions:

PUBLIC __real@3ff0000000000000

PUBLIC __real@40011eb851eb851f

Public is a derivative that makes a variable public and make it available across modules. real is representing the real number format followed by the hexadecimal value of our function parameter. Let’s convert this hex values into floating number using an online line converter (https://gregstoll.com/~gregstoll/floattohex/):

Figure 7.18: Hex value into float1

This hexadecimal 3ff0000000000000 is equivalent to our float1(1.0) argument in C/C++ code.

Figure 7.19: Hex value into float2

This hexadecimal 40011eb851eb851f is equivalent to our float2(2.14) argument in C/C++ code.

▼Line 17

PUBLIC _main

The main function is labeled as a public function, which means it can be accessed by other modules.

▼Line 18-20

EXTRN _printf:PROC

EXTRN __fltused:DWORD

; COMDAT __real@3ff0000000000000

EXTRN derivative declares the external function, which is printf in our case. All functions begin with an underscore.

▼Line 22-29

CONST SEGMENT

__real@3ff0000000000000 DQ 03ff0000000000000r ; 1

CONST ENDS

; COMDAT __real@40011eb851eb851f

CONST SEGMENT

__real@40011eb851eb851f DQ 040011eb851eb851fr ; 2.14

; Function compile flags: /Odtp

CONST ENDS

This is the start/end of CONST SEGMENT. Within this segment, our function argument 2.14 represented by 40011eb851eb851f hexadecimal notation is stored in CONST SEGMENT or we can say .rdata. Let’s dump .rdata to view our float2 argument in C/C++ code using any debugger. As we deal with little-endian, the floating point argument’s hexadecimal representation will be stored in a reverse order.

40011eb851eb851f will be stored as 1F 85 EB 51 B8 1E 01 40 in the .rdata segment, as we can see in x32dbg at 0x0040C158 memory location. If you are thinking about other floating variables in .rdata segment, then wait. Things will be clear after a few instructions.

Figure 7.20: Floating point argument in .rdata

▼Line 30-34

_TEXT SEGMENT

_main PROC

; Line 7

push ebp

mov ebp, esp

In the TEXT segment, we have the main procedure which starts with a function prologue.

▼Line 36

sub esp, 8

Allocating 8 bytes on the stack for the main function local variable, which is third argument (2.14) to the printf function.

▼Line 37

fld QWORD PTR __real@40011eb851eb851f

FLD stands for Floating Point Load. This instruction pushes the floating point value on FPU stack, which is from ST0 to ST7. You can see the debugger output before and after running this instruction as follows:

Tip: You can insert breakpoint at the start of main. To insert breakpoint, scroll to the top of .text segment, you will find the same set of instructions as in our assembly file printfWithFloat.asm. Once breakpoint is set you can step into the instructions one by one.

x32dbg output before instruction execution of this instruction:

fld QWORD PTR __real@40011eb851eb851f

Figure 7.21: ST0 before

After instruction execution:

Figure 7.22: ST0 after

▼Line 38

fstp QWORD PTR [esp]

FSTP means Floating Point Store and POP. It moves the floating point value from ST0 to the top of the stack PTR [esp] and POP the value from ST0 completely.

Figure 7.23: Floating point Store and POP

▼Line 39

sub esp, 8

Again allocating 8 bytes on the stack for the main function local variable, which is the second parameter 1.0 to the printf function. Let’s see what it looks like in x32dbg.

Figure 7.24: Allocating space on stack

▼Line 40

fld1

This instruction loads the floating point value 1.0 on the FPU stack.

Figure 7.25: Loads floating point value 1.0

▼Line 41

fstp QWORD PTR [esp]

FSTP means Floating Point Store and POP, it moves the floating point value which is 1.0 from ST0 to the top of the stack PTR [esp] and POP the value from ST0 completely.

Figure 7.26: Floating Point Store and POP

▼Line 42

push OFFSET $SG4677

Now before calling the printf function, we have to pass the remaining string constant to the stack. This PUSH instruction is pushing the string constant $SG4677 onto the stack.

Figure 7.27: Push string

▼Line 43

call _printf

All the parameters to the printf function are pushed onto the stack before the CALL instruction to the printf function. After this instruction execution, both the variables will be printed on the console.

▼Line 44

add esp, 20     ; 00000014H

As we are following the CDECL calling convention, it’s the caller who cleans the stack. On returning from the printf function, the main cleans the stack by moving the stack point back by 20 bytes (we used 4 bytes for pushing the string constant, 8 bytes for pushing the 1.0 floating hex value and another 8 bytes were used for pushing the 2.14 floating hex value).

Before this instruction, the x32dbg screen looks as follows:

Tip: To move directly to add instruction, you can use step over to execute print function and stop at add instruction. Or you can insert breakpoint at the add instruction to stop execution at add instruction.

Figure 7.28: After call printf

After this instruction:

Figure 7.29: Stack cleaned

▼Line 45-52

; Line 9

xor eax, eax

; Line 10

pop ebp

ret 0

_main ENDP

_TEXT ENDS

END

The remaining instructions are making EAX zero as main is returning 0 in the C/C++ code. In the last it is calling function epilogue to end main, TEXT segment, and code.

Function printf Printing Float with Optimization

Compile the code with optimization flag, /Ox flag. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Ox: Enable maximum optimization

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\printfWithFloat\printfWithFloat

^

cl printfWithFloat.cpp /FaprintfWithFloat-Optimized.asm /Ox /FeprintfWithFloat-Optimized.exe

The following is the output of running the preceding commands:

Figure 7.30: Function printf printing Float with Optimization

The generated assembly code will be as follows:

Figure 7.31: printfWithFloat-Optimized.asm

All the instructions in the listing are the same except line 33. As the code is optimized, the function prologue and epilogue are removed, which we will discuss in this section.

▼Line 33

fld QWORD PTR __real@40011eb851eb851f

It is the same as before. This instruction pushes the floating point value 2.14 on the FPU stack, ST0.

▼Line 34

sub esp, 16     ; 00000010H

As we saw in the non-optimized code, SUB is called twice to allocate memory for the main function local variable. But in optimization, both instructions are combined to subtract 16 bytes from ESP to allocate 8+8 bytes for floating variables 1.0 and 2.14.

▼Line 35

fstp QWORD PTR [esp+8]

It moves the floating point value, which is 2.14 from ST0 to the top of the stack PTR [esp+8] and POP the value from ST0 completely.

▼Line 36

fld1

It is the same as earlier. This instruction loads the floating point value 1.0 on the FPU stack.

▼Line 37

fstp QWORD PTR [esp]

It moves the floating point value, which is 1.0 from ST0 to the top of the stack PTR [esp] and POP the value from ST0 completely.

▼Line 38-39

push OFFSET $SG4677

call _printf

Pushes the string constant offset on the stack to call the printf function.

▼Line 40-47

add esp, 20     ; 00000014H

; Line 9

xor eax, eax

; Line 10

ret 0

_main ENDP

_TEXT ENDS

END

The caller cleans the stack as per the CDECL calling convention. The remaining instructions return 0 by XORing the EAX register, and end the main procedure, text segment, and code in line 45-47.

Function printf with char

In this simple C/C++ code, we are printing 2 char on the console. We are using the printf function to print the char.

Figure 7.32: printfWithChar.cpp

Function printf Printing Char without Optimization

Compile the code without optimization in the MSVC compiler on the same x86 Windows machine. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\printfWithChar\printfWithChar

^

cl printfWithChar.cpp /FaprintfWithChar.asm /FeprintfWithChar.exe

The following is the output of running the preceding commands:

Figure 7.33: Function printf printing Char without Optimization

The generated assembly code will be as follows:

Figure 7.34: printfWithChar.asm

Most of the part of the assembly listing has already been discussed in the preceding section. Let’s move on to line 25.

▼Line 25-28

push 98     ; 00000062H

push 97     ; 00000061H

push OFFSET $SG4677

call _printf

Before calling the printf function, three parameters are pushed onto the stack. The first PUSH instruction is PUSH 98, where 98 is the ASCII value of char ‘b’ and the hex equivalent is 62H. Refer to the ASCII table in the appendix for a complete ASCII listing.

The second PUSH instruction is PUSH 97, where 97 is the ASCII value of ‘a’ and the hex equivalent is 61H.

And the third parameter to printf is the string constant $SG4677, defined in the CONST SEGMENT. This is how the stack looks in x32dbg before the CALL instruction. You can check the stack state by inserting breakpoint at the CALL to the printf function.

Figure 7.35: Stack state

1: [esp] 00408140 Memory location of $SG4677, "char1=%c, char2=%c"

2: [esp+4] 00000061 Hex value of char ‘a’ is PUSHed

3: [esp+8] 00000062 Hex value of char ‘b’ is PUSHed

4: [esp+C] 0012FF88 [EBP]

5: [esp+10] 00401209 return to printfwithchar.00401209 from printfwithchar.00401000

After returning from CALL, the caller cleans the stack with the ADD ESP, 12 instruction as we discussed in the CDECL calling convention. Further, the code is ended with 0 in EAX, which is achieved by XOR’ing EAX. As we discussed earlier, that function return value is stored in EAX.

Function printf printing Char with Optimization

Compile the code with the optimization flag, /Ox flag. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Ox: Enable maximum optimization

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\printfWithChar\printfWithChar

^

cl printfWithChar.cpp /FaprintfWithChar-Optimized.asm /Ox /FeprintfWithChar-Optimized.exe

The following is the output of running the preceding commands:

Figure 7.36: Function printf printing Char with Optimization

The generated assembly code will be as follows:

Figure 7.37: printfWithChar-Optimized.asm

Everything in the optimized code is the same except that the function prologue and epilogue are eliminated in the optimized code.

Conclusion

In this chapter, we learned to reverse engineer programs or applications with the printf function. We also spoke about the code optimization concept in programs with the printf function. We also discussed how integer, float, and char variables are stored in memory. The floating point variable takes a different approach in working as compared to integer or char. In the next chapter, we will talk about pointers and how they are handled in reverse engineering.

CHAPTER 8

Reverse Engineering Pattern of Pointer Program

Most of us find it difficult to understand pointers, but it is the one of the most interesting subjects in programming. In our real life, have you ever imagined that pointers are everywhere? When we watch the television with a cable connection, we have so many channels to watch. Each channel is associated with a number we often call the channel number. When this channel number is pressed on the remote of our cable modem, the respective channel broadcasting starts on the television. In the context of pointers, this channel number is the pointer to the channel. This is the number where the channel is stored and played when pressed on the remote control.

So now, to understand the concept of pointers, we will walk through its concept in programming. In C/C++, there are various types of variables to hold the different types of values. We have the Integer variables that store the Integer value, the Floating variables to store real numbers, Char to store characters, and others. Similarly, a pointer is a variable that stores the memory address of other variables. In this chapter, we will understand the pattern of pointers in assembly code.

Structure

In this chapter, we will cover the following topics:

Pointers

Pointers without Optimization

Pointers with Optimization

Objective

After studying this chapter, we should be able to understand how pointers are used in programming. We will study the basics of pointers from integer to float and char. This will help you understand the way pointers are handled with respect to memory allocation. After understanding pointers, we will write a simple C/C++ program to generate optimized and non-optimized ASM code and check pointer assembly pattern with and without optimization.

Pointers

To understand pointers, let’s take a simple declaration:

int iNumber = 3;

Here, we are defining an integer variable of the name, iNumber. We are using the Hungarian Notation as the variable naming convention. This declaration is used by the compiler to:

Reverse space for an integer variable in memory

Associate the name with the memory location

Store 3 on the reserved memory location

You can imagine this as:

0x0012FF3C is the memory location address which is holding 0x00000003 as a value.

So we saw how a variable is stored in memory. Now, we will take a C/C++ program to understand pointers and the concepts associated with them:

Figure 8.1: Pointers.cpp

Line 8 of C/C++ code defines a variable of type integer. In lines 9 and 10, we see two operators, * and &.

First, we will discuss the & operator. It means the address of the operator. So, &iNumber returns the memory address of the variable iNumber, which in the preceding case was 0x0012FF3C.

Second is the * operator. It is called the value at address operator. It gives the value stored at the particular address. So, *(&iNumber) returns the value at 0x0012FF3C memory location, which is 3.

On line 9, piNumber is declared as the pointer variable, which means it is capable of holding memory addresses. Declaring int *piNumber does not mean that piNumber contains an integer value. What it means is that piNumber will hold the memory address of an integer variable. Similarly, Float *pf means that pf will hold the address of a floating point variable. The output of the preceding C/C++ program is:

Address of iNumber = 0x0012FF3C

Address of iNumber = 0x0012FF3C

Address of piNumber = 0x0012FF38

Value of piNumber = 0012FF3C

Value of iNumber = 3

Value of iNumber = 3

Value of iNumber = 3

Figure 8.2: Pointers.exe output

You will get different output while executing the preceding code. Let us see what we get on compiling the code with and without optimization.

Pointer without Optimization

Compile the code without optimization in the MSVC compiler on the same x86 Windows machine. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\Pointers\Pointers

^

cl Pointers.cpp /FaPointers.asm /FePointers.exe

The following is the output of running the preceding commands:

Figure 8.3: Pointer without Optimization

The generated assembly code will be as follows:

Figure 8.4: Pointers.asm-Part 1

Figure 8.5: Pointers.asm-Part 2

Figure 8.6: Pointers.asm-Part 3

The code generated contains two segments: CONST (.rdata named by linker) and _TEXT segment (for code). Let’s walk through the segments in the assembly listing:

▼Line 12-24

CONST SEGMENT

$SG4679 DB 0aH, 'Address of iNumber = 0x%p', 00H

ORG $+1

$SG4680 DB 0aH, 'Address of iNumber = 0x%p', 00H

ORG $+1

$SG4681 DB 0aH, 'Address of piNumber = 0x%p', 00H

$SG4682 DB 0aH, 'Value of piNumber = %p', 00H

$SG4683 DB 0aH, 'Value of iNumber = %d', 00H

ORG $+1

$SG4684 DB 0aH, 'Value of iNumber = %d', 00H

ORG $+1

$SG4685 DB 0aH, 'Value of iNumber = %d', 00H

CONST ENDS

Constant strings in the code are terminated by zero byte and are allocated in the CONST segment, which can be seen by dumping .rdata using x32dbg.

Figure 8.7: .rdata

▼Line 28-30

_TEXT SEGMENT

_piNumber$ = -8      ; size = 4

_iNumber$ = -4      ; size = 4

To access the local variable on the stack frame, we have to add _<variable-name>$ to the EBP address. So, to access the piNumber variable on stack, we have to add -8 to the EBP address and add -4 to EBP to access the iNumber variable.

▼Line 31-38

_main PROC

; File c:\jitendern\rebook\pointers\pointers\pointers.cpp

; Line 7

push ebp

mov ebp, esp

sub esp, 8

; Line 8

mov DWORD PTR _iNumber$[ebp], 3

Let us understand the main procedure by placing the breakpoint at the start of the main call (main can be located by scrolling to the top of the disassembled code). Once the breakpoint is set, run the program and step into the instructions one by one. The main procedure starts with a usual function prologue and then with a SUB instruction. The SUB instruction is allocating 8 bytes on the stack for the main function local variables. Once the space is allocated on stack, the MOV instruction will push 3 on the main stack frame at [EBP-0x4] as shown in the following screenshot:

Figure 8.8: Stack state

Line 39-40

; Line 10

lea eax, DWORD PTR _iNumber$[ebp]

Load Effective Address (LEA) loads the address of [EBP-0x4] into EAX. As we know that iNumber = 3, which is stored on stack at [EBP-0x4] = 0x0012FF3C memory location. This instruction loads EAX with 0x0012FF3C, as shown in the following screenshot:

Figure 8.9: LEA output

▼Line 41

mov DWORD PTR _piNumber$[ebp], eax

This will move the memory location stored in EAX onto the stack at [EBP-0x8]. Now we have both values stored on the stack, integer value which is 0x00000003 and the pointer to the integer value, as shown in the following screenshot:

Figure 8.10: piNumber on stack

▼Line 42-45

; Line 12

lea ecx, DWORD PTR _iNumber$[ebp]

push ecx

Before calling the first printf function, we will have to push the arguments onto the stack. The first parameter that will be pushed on the stack will be the address of iNumber. LEA will load the address of iNumber into ECX, which is later pushed on the stack, as shown in the following screenshot:

Figure 8.11: Before calling first printf function

▼Line 45-46

push OFFSET $SG4679

call _printf

This push instruction is pushing another argument to printf, which is a string constant referred to by $SG4679. Once both the parameters are pushed, call to the printf function is made. While debugging in x32dbg, we are stepping over (using Debug > Step Over option) during the printf call. Return from the printf function call will print on the console the Address of iNumber, as shown in the following screenshot:

Figure 8.12: Pointer.exe output

▼Line 47

add esp, 8

On returning from the printf procedure, the stack is cleaned by adding 8 bytes to ESP, as shown in the following screenshot:

Figure 8.13: Stack clean

▼Line 48-50

; Line 13

mov edx, DWORD PTR _piNumber$[ebp]

push edx

Now this is preparing the stack for the second printf function call. It will first MOV the value stored at [EBP-0x8] to the EDX register and then push EDX onto the stack, as shown in the following screenshot:

Figure 8.14: EDX onto the stack

▼Line 51-53

push OFFSET $SG4680

call _printf

add esp, 8

This is pushing another argument to the printf function, which is a string constant onto the stack. Now again, both the arguments to printf are pushed on the stack. The call to the printf function is made. On return, printf will print Address of iNumber = piNumber and the stack will again be cleaned after the return.

▼Line 54-59

; Line 14

lea eax, DWORD PTR _piNumber$[ebp]

push eax

push OFFSET $SG4681

call _printf

add esp, 8

In third the printf call, we have to print the address of piNumber, which is &piNumber. For this, LEA is used to load the address of [EBP-0x8] to EAX and then pushed on the stack before the printf call. The string constant, represented by $SG4681, is also pushed on the stack before another printf call, as shown in the following screenshot:

Figure 8.15: Print address of piNumber

▼Line 60-62

; Line 15

mov ecx, DWORD PTR _piNumber$[ebp]

push ecx

Now we have to print, value of piNumber. So the MOV instruction will move the piNumber variable value at [EBP-0x8] into ECX. In the next instruction, it will push ECX onto the stack before the printf call.

▼Line 63-65

push OFFSET $SG4682

call _printf

add esp, 8

These are the same as we did in the earlier printf call. It then cleans up the stack with the ADD instruction.

▼Line 66-71

; Line 16

mov edx, DWORD PTR _iNumber$[ebp]

push edx

push OFFSET $SG4683

call _printf

add esp, 8

This instruction will move the iNumber variable value at [EBP-0x4] into EDX. Before the printf call, it pushes both the arguments (string constant and EDX value) on the stack. Whatever is pushed onto the stack before pushing the string constant will be printed on the output console, as shown in the following screenshot:

Figure 8.16: Printing iNumber

▼Line 72-77

; Line 17

mov eax, DWORD PTR _iNumber$[ebp]

push eax

push OFFSET $SG4684

call _printf

add esp, 8

These instructions are doing the same as the earlier ones, except that they are using EAX for pushing the iNumber variable value at [EBP-0x4] onto the stack.

▼Line 78-80

; Line 18

mov ecx, DWORD PTR _piNumber$[ebp]

mov edx, DWORD PTR [ecx]

The first MOV instruction will move the piNumber variable value at [EBP-0x8] into ECX. The second instruction will move the value stored at a memory location pointed by ECX into EDX, as shown in the following screenshot:

Figure 8.17: MOV [ECX] into EDX

▼Line 81-84

push edx

push OFFSET $SG4685

call _printf

add esp, 8

The push instruction is pushing the arguments on the stack before the printf call. On return from the printf function, the value of iNumber will be printed on the console. Stack cleaning is done with the ADD instruction, as shown in the following screenshot:

Figure 8.18: iNumber will be printed

▼Line 85-92

; Line 19

xor eax, eax

mov esp, ebp

pop ebp

ret 0

_main ENDP

_TEXT ENDS

END

The remaining instructions are the same as what we discussed in the earlier sections. The main function epilogue is called to end the segment and code.

Pointer with Optimization

Compile the code with optimization flag, /Ox flag. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Ox: Enable maximum optimization

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\Pointers\Pointers

^

cl Pointers.cpp /FaPointers-Optimized.asm /Ox /FePointers-Optimized.exe

The following is the output of running the preceding commands:

Figure 8.19: Pointer with Optimization

The generated assembly code will be as follows:

Figure 8.20: Pointers-Optimized.asm-Part 1

Figure 8.21: Pointers-Optimized.asm-Part 2

First, we will talk about the difference in the optimized and non-optimized code generated. In optimized code, the function prologue is removed, so all the places where EBP is referred is replaced with ESP. So, in the optimized assembly listing, we will observe the ESP reference in place of EBP as we observed in the non-optimized assembly listing. To explain this concept, we will check the stack state in detail for a better understanding. Let’s walk through the assembly listing by placing breakpoint at the start of the main procedure:

▼Line 33-34

; Line 7

sub esp, 8

This is the start of the main procedure, where ESP is subtracted by 8 bytes to create room for the main function local variables, which are integer variable (iNumber) and pointer to that integer variable (piNumber). Before the execution of this instruction, the stack will look as follows:

Figure 8.22: Start of the main procedure

▼Line 35-36

; Line 10

lea eax, DWORD PTR _iNumber$[esp+8]

The Load effective address (LEA) will load EAX with the address of _iNumber$[esp+8], which is evaluated to:

lea eax, DWORD PTR ss:[esp]

as _iNumber$ = -8

As we can see, EAX is filled with 0x0012FF3C, which is ESP itself. Now, EAX also points to the top of the stack as follows:

Figure 8.23: EAX also points to the top of stack

▼Line 37-39

; Line 12

mov ecx, eax

push ecx

Move EAX to ECX. Now, ECX is also filled with 0x0012FF3C, which is pushed on the stack with the push ecx instruction. With this, our one argument to the printf function is pushed on the stack.

▼Line 40

push OFFSET $SG4679

This will push the string constant on the stack, which is another argument to the printf function. As we can see, both the arguments to the printf function are pushed on the stack. The stack state is shown in the following screenshot and can be understood in the following manner:

[ESP]  0012FF34  00408140  “\nAddress of iNumber = 0x%p”, arg to printf()

[ESP+0x4] 0012FF38  0012FF3C  &iNumber is stored here, argument to printf()

[ESP+0x8] 0012FF3C  JUNKJUNK  This where we will store iNumber

[ESP+0xC] 0012FF40  JUNKJUNK  This where we will store &iNumber

[ESP+0x10] 0012FF44  00401272  return to 0x00401272 from 0x401000

Figure 8.24: Stack state

▼Line 41-42

mov DWORD PTR _iNumber$[esp+16], 3

mov DWORD PTR _piNumber$[esp+16], eax

Both instructions are evaluated to:

mov dword ptr ss:[esp+0x8], 0x3

mov dword ptr ss:[esp+0xC], eax

Earlier, in the stack state, we allocated room for local variables of the main function on the stack (shown as JUNKJUNK) in stack state). Now, using these instructions, we are storing the integer variable 0x00000003 at [ESP+0x8] and the pointer to this variable at [ESP+0xC]. Now, the stack will look like as follows:

[ESP]  0012FF34  00408140  "\nAddress of iNumber = 0x%p", argument to printf()

[ESP+0x4] 0012FF38  0012FF3C  &iNumber is stored here, argument to printf()

[ESP+0x8] 0012FF3C  00000003  iNumber is stored here  

[ESP+0xC] 0012FF40  0012FF3C  &iNumber is stored here  

[ESP+0x10] 0012FF44  00401272  return to 0x00401272 from 0x401000

Figure 8.25: After Line 41-42 execution

▼Line 43

call _printf

Once the arguments are pushed on the stack, call to printf is made. As per the C/C++ code at line 12:

printf ("\nAddress of iNumber = 0x%p", &iNumber) ;

So, on returning from the printf call, the following will be printed on the console:

Figure 8.26: After printf execution

▼Line 44-48

; Line 13

mov edx, DWORD PTR _piNumber$[esp+16]

push edx

push OFFSET $SG4680

call _printf

The C/C++ code on line 13 prints piNumber, which is the memory location of iNumber:

printf ("\nAddress of iNumber = 0x%p", piNumber) ;

In our assembly code, arguments to the printf function are pushed on the stack by first moving EDX with the pointer to the Integer variable. The same EDX is then pushed on to stack with another push of string constant. Once we have both arguments to printf function on stack, call to printf function is made. As shown in the stack state and the following screenshot:

[ESP]  0012FF2C  0040815C  "\nAddress of iNumber = 0x%p", argument to printf()

[ESP+0x04] 0012FF30  0012FF3C  piNumber is stored here, argument to printf()

[ESP+0x08] 0012FF34  00408140  "\nAddress of iNumber = 0x%p", argument to printf()

[ESP+0x0C] 0012FF38  0012FF3C  &iNumber is stored here, argument to printf()

[ESP+0x10] 0012FF3C  00000003  iNumber is stored here  

[ESP+0x14] 0012FF40  0012FF3C  &iNumber is stored here  

[ESP+0x18] 0012FF44  00401272  return to 0x00401272 from 0x401000

Figure 8.27: After the second printf execution

▼Line 49-53

; Line 14

lea eax, DWORD PTR _piNumber$[esp+24]

push eax

push OFFSET $SG4681

call _printf

The C/C++ code on line 14 prints &piNumber, which is the memory location of piNumber:

printf ("\nAddress of piNumber = 0x%p", &piNumber) ;

In our assembly code, the LEA instruction will be evaluated to:

lea eax, ss:[esp+0x14]

LEA will load EAX with the memory address of ss:[esp+0x14], which is the address of piNumber. Once EAX is loaded with the address, it is pushed onto the stack with another push of the string constant. Once we have both arguments to the printf function on the stack, call to the printf function is made. This is shown in the following screenshot:

[ESP]  0012FF24  00408178  "\nAddress of piNumber = 0x%p", argument to printf()

[ESP+0x04] 0012FF28  0012FF40  &piNumber is stored here, argument to printf()

[ESP+0x08] 0012FF2C  0040815C  "\nAddress of iNumber = 0x%p", argument to printf()

[ESP+0x0C] 0012FF30  0012FF3C  piNumber is stored here, argument to printf()

[ESP+0x10] 0012FF34  00408140  "\nAddress of iNumber = 0x%p", argument to printf()

[ESP+0x14] 0012FF38  0012FF3C  &iNumber is stored here, argument to printf()

[ESP+0x18] 0012FF3C  00000003  iNumber is stored here  

[ESP+0x1C] 0012FF40  0012FF3C  &iNumber is stored here  

[ESP+0x20] 0012FF44  00401272  return to 0x00401272 from 0x401000

Figure 8.28: After thrid printf execution

▼Line 54-58

; Line 15

mov ecx, DWORD PTR _piNumber$[esp+32]

push ecx

push OFFSET $SG4682

call _printf

The C/C++ code on line 15 prints piNumber, which is the memory location of iNumber:

printf ("\nValue of piNumber = %p", piNumber) ;

In our assembly code, the MOV instruction will be evaluated to:

mov ecx, dword ptr ss:[esp+0x1C]

MOV will move the value at ss:[esp+0x1C] to ECX, which points to the memory location of iNumber. Now, the arguments are again pushed to the stack for call to the printf function. This is shown in the following screenshot:

[ESP]  0012FF1C  00408194  "\nValue of piNumber = %p", argument to printf()

[ESP+0x04] 0012FF20  0012FF3C  piNumber is stored here, argument to printf()

[ESP+0x08] 0012FF24  00408178  "\nAddress of piNumber = 0x%p", argument to printf()

[ESP+0x0C] 0012FF28  0012FF40  &piNumber is stored here, argument to printf()

[ESP+0x10] 0012FF2C  0040815C  "\nAddress of iNumber = 0x%p", argument to printf()

[ESP+0x14] 0012FF30  0012FF3C  piNumber is stored here, argument to printf()

[ESP+0x18] 0012FF34  00408140  "\nAddress of iNumber = 0x%p", argument to printf()

[ESP+0x1C] 0012FF38  0012FF3C  &iNumber is stored here, argument to printf()

[ESP+0x20] 0012FF3C  00000003  iNumber is stored here  

[ESP+0x24] 0012FF40  0012FF3C  &iNumber is stored here  

[ESP+0x28] 0012FF44  00401272  return to 0x00401272 from 0x401000

Figure 8.29: After fourth printf execution

▼Line 59-63

; Line 16

mov edx, DWORD PTR _iNumber$[esp+40]

push edx

push OFFSET $SG4683

call _printf

The C/C++ code on line 16 prints iNumber, which is the value of iNumber:

printf ("\nValue of iNumber = %d", iNumber) ;

In our assembly code, the MOV instruction will be evaluated to:

mov edx, dword ptr ss:[esp+0x20]

MOV will move the value at ss:[esp+0x20] to EDX, which is the value of iNumber. Now, the arguments are again pushed to the stack for call to the printf function. This is shown in the following screenshot:

[ESP]  0012FF14  004081AC  "\nValue of iNumber = %d", argument to printf()

[ESP+0x04] 0012FF18  00000003  iNumber is stored here, argument to printf()

[ESP+0x08] 0012FF1C  00408194  "\nValue of piNumber = %p", argument to printf()

[ESP+0x0C] 0012FF20  0012FF3C  piNumber is stored here, argument to printf()

[ESP+0x10] 0012FF24  00408178  "\nAddress of piNumber = 0x%p", argument to printf()

[ESP+0x14] 0012FF28  0012FF40  &piNumber is stored here, argument to printf()

[ESP+0x18] 0012FF2C  0040815C  "\nAddress of iNumber = 0x%p", argument to printf()

[ESP+0x1C] 0012FF30  0012FF3C  piNumber is stored here, argument to printf()

[ESP+0x20] 0012FF34  00408140  "\nAddress of iNumber = 0x%p", argument to printf()

[ESP+0x24] 0012FF38  0012FF3C  &iNumber is stored here, argument to printf()

[ESP+0x28] 0012FF3C  00000003  iNumber is stored here  

[ESP+0x2C] 0012FF40  0012FF3C  &iNumber is stored here  

[ESP+0x30] 0012FF44  00401272  return to 0x00401272 from 0x401000

Figure 8.30: After fifth printf execution

▼Line 64-68

; Line 17

mov eax, DWORD PTR _iNumber$[esp+48]

push eax

push OFFSET $SG4684

call _printf

The C/C++ code on line 17 prints *(&iNumber), which itself is the value of iNumber:

printf ("\nValue of iNumber = %d", *(&iNumber)) ;

In our assembly code, the MOV instruction will be evaluated to:

mov eax, dword ptr ss:[esp+0x28]

MOV will move the value at ss:[esp+0x28] to EAX, which is the value of iNumber. Now, the arguments are again pushed to the stack for call to the printf function. As shown in the stack state and the following screenshot:

[ESP]  0012FF0C  004081C4  "\nValue of iNumber = %d", argument to printf()

[ESP+0x04] 0012FF10  00000003  *(&iNumber) is stored here, argument to printf()

[ESP+0x08] 0012FF14  004081AC  "\nValue of iNumber = %d", argument to printf()

[ESP+0x0C] 0012FF18  00000003  iNumber is stored here, argument to printf()

[ESP+0x10] 0012FF1C  00408194  "\nValue of piNumber = %p", argument to printf()

[ESP+0x14] 0012FF20  0012FF3C  piNumber is stored here, argument to printf()

[ESP+0x18] 0012FF24  00408178  "\nAddress of piNumber = 0x%p", argument to printf()

[ESP+0x1C] 0012FF28  0012FF40  &piNumber is stored here, argument to printf()

[ESP+0x20] 0012FF2C  0040815C  "\nAddress of iNumber = 0x%p", argument to printf()

[ESP+0x24] 0012FF30  0012FF3C  piNumber is stored here, argument to printf()

[ESP+0x28] 0012FF34  00408140  "\nAddress of iNumber = 0x%p", argument to printf()

[ESP+0x2C] 0012FF38  0012FF3C  &iNumber is stored here, argument to printf()

[ESP+0x30] 0012FF3C  00000003  iNumber is stored here  

[ESP+0x34] 0012FF40  0012FF3C  &iNumber is stored here  

[ESP+0x38] 0012FF44  00401272  return to 0x00401272 from 0x401000

Figure 8.31: After sixth printf execution

▼Line 69-74

; Line 18

mov ecx, DWORD PTR _piNumber$[esp+56]

mov edx, DWORD PTR [ecx]

push edx

push OFFSET $SG4685

call _printf

The C/C++ code on line 17 prints *piNumber, which itself is the value of iNumber:

printf ("\nValue of iNumber = %d", *piNumber) ;

In our assembly code, we see two MOV instructions, where one move instruction takes the memory location of iNumber and the other fetches the value stored at that memory address. The first MOV instruction is resolved to:

mov ecx, dword ptr ss:[esp+0x34]

This MOV will move the value at ss:[esp+0x34] to ECX, which is the memory location of iNumber. The second MOV fetches the value stored at the memory address in ECX and moves it to EDX. Now, the arguments are again pushed to the stack for call to the printf function. This is shown in the following screenshot:

[ESP]  0012FF04  004081DC  "\nValue of iNumber = %d", argument to printf()

[ESP+0x04] 0012FF08  00000003  *piNumber is stored here, argument to printf()

[ESP+0x08] 0012FF0C  004081C4  "\nValue of iNumber = %d", argument to printf()

[ESP+0x0C] 0012FF10  00000003  *(&iNumber) is stored here, argument to printf()

[ESP+0x10] 0012FF14  004081AC  "\nValue of iNumber = %d", argument to printf()

[ESP+0x14] 0012FF18  00000003  iNumber is stored here, argument to printf()

[ESP+0x18] 0012FF1C  00408194  "\nValue of piNumber = %p", argument to printf()

[ESP+0x1C] 0012FF20  0012FF3C  piNumber is stored here, argument to printf()

[ESP+0x20] 0012FF24  00408178  "\nAddress of piNumber = 0x%p", argument to printf()

[ESP+0x24] 0012FF28  0012FF40  &piNumber is stored here, argument to printf()

[ESP+0x28] 0012FF2C  0040815C  "\nAddress of iNumber = 0x%p", argument to printf()

[ESP+0x2C] 0012FF30  0012FF3C  piNumber is stored here, argument to printf()

[ESP+0x30] 0012FF34  00408140  "\nAddress of iNumber = 0x%p", argument to printf()

[ESP+0x34] 0012FF38  0012FF3C  &iNumber is stored here, argument to printf()

[ESP+0x38] 0012FF3C  00000003  iNumber is stored here  

[ESP+0x3C] 0012FF40  0012FF3C  &iNumber is stored here  

[ESP+0x40] 0012FF44  00401272  return to 0x00401272 from 0x401000

Figure 8.32: After seventh printf execution

▼Line 75-81

; Line 19

xor eax, eax

add esp, 64     ; 00000040H

ret 0

_main ENDP

_TEXT ENDS

END

XOR and ADD will clean up the EAX and stack, respectively and END the code by returning 0. The ADD instruction cleaned up the stack in one go, which is different from that in the non-optimized code, wherein we cleaned the stack after each printf call. This is shown in the stack state below and the following screenshot:

[ESP]  0012FF44  00401272  return to 0x00401272 from 0x401000

Figure 8.33: Cleaning stack

Conclusion

In this chapter, we learned the conceptual knowledge about pointers. We also learned how pointers are stored in memory with respect to integer, float, and char pointers. We also found out how non-optimized code is different from optimized assembly code. The non-optimized code uses EBP to refer to arguments on the stack and the stack cleaning is done after every printf call. But in case of optimized code, ESP is used to refer to the arguments and the stack cleaning is done in one go towards the END of the code. In the next chapter, we will talk about another interesting topic, which is decision. We will see how decisions statements are handled in assembly programming.

CHAPTER 9

Reverse Engineering Pattern of Decision Control Structure

In our day-to-day life, we take many decisions and every decision is related to an outcome. Let’s take an example. Before leaving home for office, we all prefer to check Google maps and if the traffic is high, we may ask our boss for permission to work from home. Another example is if I earn more, I will purchase a nice luxury car and a big house. All these decisions are linked with some condition and result in certain outcomes.

Similar to this, computers are also programmed to make decisions based on conditions. C/C++ programming also provides decision-making statements, which help programmers to write code that involves decisions based on the conditions. Different conditions are linked to the outcomes to behave in the natural manner. This chapter will help you understand the pattern of these conditions in assembling listing from the reverse engineering point of view.

Structure

In this chapter, we will cover the following topics:

If-else statement

If-else statement without Optimization

If-else statement with Optimization

Objective

The objective of this chapter is to understand the concept of assembly instructions used to make decisions in a program flow. We will learn about CMP and Jump instructions in the assembly code. As we tend to take several decisions for a single problem in our day-to-day life, similarly to handle problems or conditions in assembly we have CMP and jump instructions in assembly. We will also understand the differentiation between optimized and non-optimized assembly code of decision control structures.

If-else statement

In this section, we will cover simple C/C++ code with if-else statement, wherein we will ask the user to enter two integer numbers. If both the numbers are equal, it prints saying that both the numbers are equal on the console. But if both the entered numbers are not equal, the code will use else statement to print, the numbers are not equal. This code also includes scanf and printf in our C/C++ code. It is recommended to go through Chapter 7, Reverse Engineering Pattern of Printf Program, for a clear understanding.

Figure 9.1: ifelse.cpp

If-else statement without Optimization

Compile the code without optimization in the MSVC compiler on the same x86 Windows machine. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\ifelse\ifelse

^

cl ifelse.cpp /Faifelse.asm /Feifelse.exe

The following is the output of running the preceding commands:

Figure 9.2: If-else statement without Optimization

This will generate the assembly code and the EXE file. For analysis purpose, we will disable the Address Space Layout Randomization (ASLR). It is a security mechanism by which the base address of the PE file is randomized on every load of the Portable Executable (PE) file generated with our MSVC compiler. To disable ASLR, follow the same steps by using the CFF explorer and change the DllCharacteristics parameter to uncheck DLL can move. For more details, refer to the Appendix.

Now, let us move on to the generated assembly listing:

Figure 9.3: ifelse.asm-Part-1

Figure 9.4: ifelse.asm-Part-2

Let’s walk through the assembly code line by line. As we have discussed most of the initial assembly listing in the earlier chapters, we will move on to the instructions from the start of the main procedure.

▼Line 34-37

; Line 7

push ebp

mov ebp, esp

sub esp, 8

It starts with the main function prologue. With the SUB instruction, we are creating room for the local variables of main, which are iNumber1 and iNumber2. Each number will occupy 4 bytes of space, making it a total of 8 bytes.

To analyze the stack state, let us open the PE file generated in x32dbg and place a breakpoint at the start of the main procedure. We will step into the code to see the stack state after the SUB instruction.

[ESP]  0012FF38  004038DD  [EBP-8] now JUNK, Input placeholder for iNumber1

[ESP+0x4] 0012FF3C  0040445A  [EBP-4] now JUNK, Input placeholder for iNumber2

[ESP+0x8] 0012FF40  0012FF88  [EBP]

[ESP+0xC] 0012FF44  004012F7  return to ifelse.004012F7 from ifelse.00401000

iNumber1 can be accessed with the help of the _iNumber1$ macro, which is equal to -8. So, iNumber1 can be accessed by adding EBP with the _iNumber1$ macro, which is equal to [EBP-8].

Figure 9.5: Creating room for our local variable

▼Line 38-41

; Line 10

push OFFSET $SG4679

call _printf

add esp, 4

The C/C++ code on line 10 prints the string on the console:

printf("Input Number1 : ");

In our assembly code, the argument to the printf function is first pushed onto the stack. The argument of printf is a string constant stored in the .rdata segment and internally defined as $SG4679. A call to printf will print $SG4679 on the console and the ADD instruction after call to printf is cleaning the stack by 4 bytes.

[ESP-0x4] 0012FF34  0040A140  "Input Number1 : ", parameter to 1st printf()

[ESP]  0012FF38  004038DD  [EBP-8] now JUNK, Input placeholder for iNumber1

[ESP+0x4] 0012FF3C  0040445A  [EBP-4] now JUNK, Input placeholder for iNumber2

[ESP+0x8] 0012FF40  0012FF88  [EBP]

[ESP+0xC] 0012FF44  004012F7  return to ifelse.004012F7 from ifelse.00401000

Figure 9.6: Call to printf

▼Line 42-47

; Line 11

lea eax, DWORD PTR _iNumber1$[ebp]

push eax

push OFFSET $SG4680

call _scanf

add esp, 8

The C/C++ code on line 11 takes the input integer from the user and stores it at the &iNumber1 memory location:

scanf("%d", &iNumber1);

In our assembly code, LEA will effectively load the address of _iNumber1$[ebp], which is 0x0012FF38, into the EAX register. The purpose of loading EAX with the address of _iNumber1$[ebp] is to push the EAX register on the stack along with the string constant $SG4680. Both the address (0x0012FF38) and string $SG4680 are arguments to the scanf function.

To check the stack state after the scanf function call, put another breakpoint at ADD instruction after the scanf call. Now step into the instructions one by one and on call to scanf, do a step over. It will prompt you to enter the first number on the console. We will enter number 7 and press ENTER. After pressing ENTER, number 7 will be saved onto the stack at the input placeholder for the iNumber1 memory location, 0x0012FF38. With the scanf call, the number input by the user is saved onto the stack. After the call, it is time to again clean up the stack with the ADD instruction. ADD will move the stack pointer by 8 bytes for cleaning. The stack state is shown in the following screenshot:

[ESP-0x8] 0012FF30  0040A154  "%d", parameter to 1st scanf()

[ESP-0x4] 0012FF34  0012FF38  Memory location of placeholder for iNumber1

[ESP]  0012FF38  00000007  [EBP-8] 7 is stored, Input placeholder for iNumber1

[ESP+0xC] 0012FF3C  0040445A  [EBP-4] now JUNK, Input placeholder for iNumber2

[ESP+0x10] 0012FF40  0012FF88  [EBP]

[ESP+0x14] 0012FF44  004012F7  return to ifelse.004012F7 from ifelse.00401000

Figure 9.7: After scanf

▼Line 48-51

; Line 12

push OFFSET $SG4681

call _printf

add esp, 4

The C/C++ code on line 12 prints the string on the console:

printf("Input Number2 : ");

It is the same as the earlier. The argument to the printf function is first pushed onto the stack. The argument of printf is a string constant stored in the .rdata segment and internally defined as $SG4681. A call to printf will print $SG4681 on the console and the ADD instruction following the call to printf will clean the stack by 4 bytes. The stack state is shown in the following screenshot:

[ESP-0x4] 0012FF34  0040A158  "Input Number2 : ", parameter to 2nd printf()

[ESP]  0012FF38  00000007  [EBP-8] 7 is stored, Input placeholder for iNumber1

[ESP+0xC] 0012FF3C  0040445A  [EBP-4] now JUNK, Input placeholder for iNumber2

[ESP+0x10] 0012FF40  0012FF88  [EBP]

[ESP+0x14] 0012FF44  004012F7  return to ifelse.004012F7 from ifelse.00401000

Figure 9.8: Again call to printf

▼Line 52-57

; Line 13

lea ecx, DWORD PTR _iNumber2$[ebp]

push ecx

push OFFSET $SG4682

call _scanf

add esp, 8

The C/C++ code on line 13 takes the input integer from the user and stores it at the &iNumber2 memory location:

scanf("%d", &iNumber2);

It is same as the earlier scanf code. LEA will effectively load the address of _iNumber2$[ebp], which is 0x0012FF3C, into the ECX register. The purpose of loading ECX with the address of _iNumber2$[ebp] is to push the ECX register on the stack along with the string constant $SG4682. Both the address (0x0012FF3C) and string $SG4682 are arguments to the scanf function.

To check the stack state after the scanf function call, put another breakpoint at ADD instruction after the scanf call. Now step into the instructions one by one and on call to scanf, do a step over. It will prompt you to enter the second number on the console. We will enter number 8 and press ENTER. After pressing ENTER, number 8 will be saved onto the stack at the input placeholder for the iNumber2 memory location, 0x0012FF3C. With the scanf call, the number input by the user is saved onto the stack. After the call, it is time to again clean up the stack with the ADD instruction. ADD will move the stack pointer by 8 bytes for cleaning. The stack state is shown in the following screenshot:

[ESP-0x8] 0012FF30  0040A16C  "%d", parameter to 2nd scanf()

[ESP-0x4] 0012FF34  0012FF3C  Memory location of placeholder for iNumber2

[ESP]  0012FF38  00000007  [EBP-8] 7 is stored, Input placeholder for iNumber1

[ESP+0xC] 0012FF3C  00000008  [EBP-4] 8 is stored, Input placeholder for iNumber2

[ESP+0x10] 0012FF40  0012FF88  [EBP]

[ESP+0x14] 0012FF44  004012F7  return to ifelse.004012F7 from ifelse.00401000

Figure 9.9: Again after another scanf

▼Line 58-61

; Line 14

mov edx, DWORD PTR _iNumber1$[ebp]

cmp edx, DWORD PTR _iNumber2$[ebp]

jne SHORT $LN2@main

The C/C++ code on line 14 compares the two numbers:

if (iNumber1 == iNumber2)

Now we have both the numbers iNumber1 and iNumber2 stored at [EBP-8] and [EBP-4], respectively. To compare these two numbers, the first MOV instruction will move the value stored at [EBP-8] to the EDX register, which is number 0x00000007 and then use the CMP instruction to compare the value stored at [EBP-4] with that of the EDX register.

In our case, EDX = 0x00000007 is compared with [EBP-8] = 0x00000008.

The CMP instruction can be visualized as a SUB instruction and as it is a conditional JMP, it will work as follows:

[EDX] - [EBP-8], based on the subtraction result, the control flag is set

0x00000007 - 0x00000008 != 0, so it sets ZF (Zero Flag) = 0

Following the CMP is the JNE, which is Jump if Not Equal. As both numbers are not equal and ZF=0, the jump will occur to the location specified after the JNE instruction, which is $LN2@main. Let’s check the stack state and a screenshot of x32dbg before the JNE instruction:

[ESP-0x8] 0012FF30  0040A16C  "%d", parameter to 2nd scanf()

[ESP-0x4] 0012FF34  0012FF3C  Memory location of placeholder for iNumber2

[ESP]  0012FF38  00000007  [EBP-8] 7 is stored, Input placeholder for iNumber1

[ESP+0xC] 0012FF3C  00000008  [EBP-4] 8 is stored, Input placeholder for iNumber2

[ESP+0x10] 0012FF40  0012FF88  [EBP]

[ESP+0x14] 0012FF44  004012F7  return to ifelse.004012F7 from ifelse.00401000

Figure 9.10: Before JNE instruction

In our case, JNE will move the instruction pointer to the $LN2@main location. If in case the user enters the same numbers, the CMP instruction will result in ZF (Zero Flag) = 1. That will move the instruction pointer to print “Number1 and Number2 are equal” on the console, followed by the unconditional jump instruction to $LN3@main (JMP SHORT $LN3@main at line 67), where $LN3@main is the closing of the main function and the code. The stack state is shown in the following screenshot:

▼Line 68-72

$LN2@main:

; Line 17

push OFFSET $SG4686

call _printf

add esp, 4

The C/C++ equivalent is:

printf("Number1 and Number2 are not equal\n");

It prints ‘Number1 and Numver2 are not equal’. The ASM code pushes the string constant $SG4686 on the stack to be used by the printf function as an argument. This will print ‘Number1 and Number2 are not equal’ on the console. Following this is the ADD instruction which will clean the stack by 4 bytes, which was used by string constant $SG4686 in PUSH operation. The stack state after this will be as follows:

[ESP-0x4] 0012FF34  0040A190  "Number1 and Number2 are not equal\n", arg to printf

[ESP]  0012FF38  00000007  [EBP-8] 7 is stored, Input placeholder for iNumber1

[ESP+0xC] 0012FF3C  00000008  [EBP-4] 8 is stored, Input placeholder for iNumber2

[ESP+0x10] 0012FF40  0012FF88  [EBP]

[ESP+0x14] 0012FF44  004012F7  return to ifelse.004012F7 from ifelse.00401000

Figure 9.11: Print “Number1 and Number2 are not equal”

▼Line 73-81

$LN3@main:

; Line 18

xor eax, eax

mov esp, ebp

pop ebp

ret 0

_main ENDP

_TEXT ENDS

END

XOR and the function epilogue will clean up EAX and stack, respectively. END will end the code by returning 0. The stack state is shown in the following screenshot:

[ESP-0x10] 0012FF34  0040A190  JUNK

[ESP-0xC] 0012FF38  00000007  JUNK

[ESP-0x8] 0012FF3C  00000008  JUNK

[ESP-0x4] 0012FF40  0012FF88  [EBP] popped up

[ESP]  0012FF44  004012F7  return to ifelse.004012F7 from ifelse.00401000

Figure 9.12: Stack cleaned

If-else statement with Optimization

Compile the code with the optimization flag, /Ox flag. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Ox: Enable maximum optimization

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\ifelse\ifelse

^

cl ifelse.cpp /Faifelse-Optimized.asm /Ox /Feifelse-Optimized.exe

The following is the output of running the preceding commands:

Figure 9.13: If-else statement with Optimization

This will generate the assembly code and the EXE file. For analysis purpose, we will disable the ASLR. It is a security mechanism by which the base address of the PE file is randomized on every load of the PE file generated with our MSVC compiler. To disable ASLR, follow the same steps by using the CFF explorer and change the DllCharacteristics parameter to uncheck DLL can move. For step-by-step reference to disable ASLR, refer to the Appendix.

Now, let’s move on to the generated assembly listing:

Figure 9.14: ifelse-Optimized.asm-Part-1

Figure 9.15: ifelse-Optimized.asm-Part-2

This assembly code is optimized by removing unnecessary lines to consume fewer resources. In this code, we will not see the function prologue and epilogue. We can also observe that ESP is used as a reference rather than EBP; this is because of the absence of the function prologue and epilogue. Let us start the analysis and this time, we will enter the same number.

▼Line 34-35

; Line 7

sub esp, 8

The SUB instruction is creating room for the local variables of the main function on the stack, iNumber1 and iNumber2. To check the stack state, we will open the PE file in x32dbg and place our first breakpoint at the start of the main procedure. The stack state and the x32dbg screenshot after the SUB instruction is as follows:

[ESP]  0012FF3C  0040444A  right now is JUNK, Input placeholder for iNumber1

[ESP+0x4] 0012FF40  00000000  right now is JUNK, Input placeholder for iNumber2

[ESP+0x8] 0012FF44  004012F3  return to 0x004012F3 from 0x00401000

Figure 9.16: Creating room for our local variable

▼Line 36-38

; Line 10

push OFFSET $SG4679

call _printf

The C/C++ equivalent of the ASM is:

printf(“Input Number1 : “);

It will print the string constant $SG4679 on the console. $SG4679 was pushed on the stack before the printf function call. The stack state after the CALL instruction is as follows:

[ESP]  0012FF38  0040A140  “Input Number1 : “, parameter to printf()

[ESP+0x4] 0012FF3C  0040444A right now is JUNK, Input placeholder for iNumber1

[ESP+0x8] 0012FF40  00000000  right now is JUNK, Input placeholder for iNumber2

[ESP+0xC] 0012FF44  004012F3  return to 0x004012F3 from 0x00401000

▼Line 39-43

; Line 11

lea eax, DWORD PTR _iNumber1$[esp+12]

push eax

push OFFSET $SG4680

call _scanf

The C/C++ code on line 11 asks the user to input the first number, iNumber1. This number will be stored at the &iNumber1 location.

scanf("%d", &iNumber1);

In ASM, the LEA instruction is evaluated to:

lea eax, ss:[esp+0x4]

This will load the effective address of ESP+0x4 to EAX. This is the memory location on the stack where the iNumber1 input by the user will be stored. As the scanf function expects two arguments, so both the arguments to scanf are pushed onto the stack before the CALL instruction. Arguments to scanf are memory location, on which iNumber1 is stored, and the string constant $SG4680. When a call to scanf is done, it will ask the user to enter the number. We have inserted one more breakpoint after the scanf function to view the stack after the scanf call. Refer to the stack state and x32dbg in the following screenshot:

[ESP]  0012FF30  0040A154  “%d”, parameter to scanf()

[ESP+0x4] 0012FF34  0012FF3C  Memory location of Input placeholder for iNumber1  

[ESP+0x8] 0012FF38  0040A140  “Input Number1 : “, parameter to printf()

[ESP+0xC] 0012FF3C  00000007  7 is stored here, Input placeholder for iNumber1

[ESP+0x10] 0012FF40  00000000  right now is JUNK, Input placeholder for iNumber2

[ESP+0x14] 0012FF44  004012F3  return to 0x004012F3 from 0x00401000

Figure 9.17: Breakpoint after the first scanf

▼Line 44-46

; Line 12

push OFFSET $SG4681

call _printf

The C/C++ code on line 12 prints the following message on the console:

printf(“Input Number2 : “);

In ASM, it does the same by pushing the string constant $SG4681 on the stack and the call printf function.

[ESP]  0012FF2C  0040A158  “Input Number2 : “, parameter to printf()

[ESP+0x4] 0012FF30  0040A154  “%d”, parameter to scanf()

[ESP+0x8] 0012FF34  0012FF3C  Memory location of Input placeholder for iNumber1  

[ESP+0xC] 0012FF38  0040A140  “Input Number1 : “, parameter to printf()

[ESP+0x10] 0012FF3C  00000007  7 is stored here, Input placeholder for iNumber1

[ESP+0x14] 0012FF40  00000000  right now is JUNK, Input placeholder for iNumber2

[ESP+0x18] 0012FF44  004012F3  return to 0x004012F3 from 0x00401000

▼Line 47-51

; Line 13

lea ecx, DWORD PTR _iNumber2$[esp+24]

push ecx

push OFFSET $SG4682

call _scanf

The C/C++ code on line 13 asks the user to input the second number, iNumber2. This number will be stored at the &iNumber2 location.

scanf("%d", &iNumber2);

In ASM, the LEA instruction is evaluated to:

lea ecx, ss:[esp+0x14]

This will load the effective address of ESP+0x14 to ECX. This is memory location on the stack where the iNumber2 input by the user will be stored. As the scanf function expects two arguments, so both the arguments to scanf are pushed onto the stack before the CALL to the scanf function. Arguments to scanf are iNumber2 and the string constant $SG4682. When a call to scanf is done, it will ask the user to enter the number. This time, we have entered the same number, 7. We have inserted one more breakpoint after the scanf function to view the stack after the scanf call. Refer to the stack state and x32dbg in the following screenshot:.

[ESP]  0012FF24  0040A16C  "%d"

[ESP+0x4] 0012FF28  0012FF40

[ESP+0x8] 0012FF2C  0040A158  "Input Number2 : ", parameter to printf()

[ESP+0xC] 0012FF30  0040A154  "%d", parameter to scanf()

[ESP+0x10] 0012FF34  0012FF3C  Memory location of Input placeholder for iNumber1  

[ESP+0x14] 0012FF38  0040A140  "Input Number1 : ", parameter to printf()

[ESP+0x18] 0012FF3C  00000007  7 is stored here, Input placeholder for iNumber1

[ESP+0x1C] 0012FF40  00000007  7 is stored here, Input placeholder for iNumber2

[ESP+0x20] 0012FF44  004012F3  return to 0x004012F3 from 0x00401000

Figure 9.18: Breakpoint after the second scanf

▼Line 51-56

; Line 14

mov edx, DWORD PTR _iNumber1$[esp+32]

add esp, 24     ; 00000018H

cmp edx, DWORD PTR _iNumber2$[esp+8]

jne SHORT $LN2@main

The C/C++ code on line 14 compares the two numbers:

if (iNumber1 == iNumber2)

In ASM, the MOV instruction will move iNumber1 stored at ss:[esp+0x18] to EDX. The ADD instruction will clean the stack by adding 24 bytes to ESP. Once the stack is cleaned, the CMP instruction compares the value at EDX with ss:[esp+0x4]. As both the numbers are equal, it will set ZF=1. As Jump if Not Equal (JNE) condition is not met, the instruction pointer will move to the next instruction following the JNE. Refer to the stack state after the JNE instruction execution and x32dbg in the following screenshot:

[ESP-0x18] 0012FF24  0040A16C  JUNK

[ESP-0x14] 0012FF28  0012FF40  JUNK

[ESP-0x10] 0012FF2C  0040A158  JUNK

[ESP-0xC] 0012FF30  0040A154  JUNK

[ESP-0x8] 0012FF34  0012FF3C  JUNK

[ESP-0x4] 0012FF38  0040A140  JUNK

[ESP]  0012FF3C  00000007  7 is stored here, Input placeholder for iNumber1

[ESP+0x4] 0012FF40  00000007  7 is stored here, Input placeholder for iNumber2

[ESP+0x8] 0012FF44  004012F3  return to 0x004012F3 from 0x00401000

Figure 9.19: Stack state after JNE

▼Line 57-61

; Line 15

push OFFSET $SG4684

; Line 17

call _printf

add esp, 4

This will push the string constant $SG4684 on the stack for the printf function to print on the console. The ADD instruction will perform the stack cleaning. Refer to the stack state and x32dbg after the ADD instruction in the following screenshot:

[ESP-0x4] 0012FF38  0040A170  Now JUNK, "Number1 and Number2 are equal\n"

[ESP]  0012FF3C  00000007  7 is stored here, Input placeholder for iNumber1

[ESP+0x4] 0012FF40  00000007  7 is stored here, Input placeholder for iNumber2

[ESP+0x8] 0012FF44  004012F3  return to 0x004012F3 from 0x00401000

Figure 9.20: Stack state after the ADD instruction

▼Line 62-65

; Line 18

xor eax, eax

add esp, 8

ret 0

_main ENDP

_TEXT ENDS

END

XOR and the function epilogue will clean up the EAX and stack, respectively. END will end the code by returning 0. Refer to the stack state and x32dbg after the stack cleaning in the following screenshot:

[ESP-0xC] 0012FF38  0040A170  Now JUNK, "Number1 and Number2 are equal\n"

[ESP-0x8] 0012FF3C  00000007  Now JUNK, iNumber1 number 7 entered was stored here

[ESP-0x4] 0012FF40  00000007  Now JUNK, iNumber1 number 7 entered was stored here

[ESP]  0012FF44  004012F3  return to 0x004012F3 from 0x00401000

Figure 9.21: Stack cleaned

Conclusion

In this chapter, we learned the concept of assembly instructions used to make decisions in a program flow. We also learned about the usage of CMP and Jump instructions in assembly code. Along with this, we also understood the differentiating pattern between optimized and non-optimized assembly code of decision control structures. In the next chapter, we will learn about the disassembly of programs with looping control statements.

CHAPTER 10

Reverse Engineering Pattern of Loop Control Structure

In the last chapter, we discussed about the programs that are associated with some decision control statements. In our day-to-day life, we tend to do so many things over and over again. Like we get ready every morning and go to sleep in the night around the same time every day. This is where we see our routine happening in a loop every day.

Every programming language is equipped with a similar loop control structure, where the programs are coded in such a way that they run in a loop with some conditions. As a reverse engineer, this topic can be understood from a malware writer’s point of view, wherein malware’s are coded in such a way that they infect all the files on the target computer. When a malware gets executed, files in the computer are run through a loop condition so as to target the whole computer data. Understanding to identify loop patterns in assembly will help you decode programs using loop control structures. Most of the computer programs are coded with these loop control structures. It is important for us to should understand the pattern of these programs when reverse engineered.

Structure

In this chapter, we will cover the following topics:

Understanding about a loop control structure

How loops are handled in assembly

Objective

The objective of this chapter is to learn about different loop statements in C/C++ and understanding the code pattern in a disassembled code. We will also learn to put a conditional breakpoint in code execution. In an assembly listing generated from loop statement programs, we will check the CMP and JMP instruction patterns in assembly and note their differences in optimized and non-optimized code.

While Condition

In this C/C++ code, we will use the while condition to print numbers from 1 to 10 on the console/screen. This will also include the printf function in our C/C++ code.

Figure 10.1: while.cpp

While condition without Optimization

Compile the code without optimization in the MSVC compiler on the same x86 Windows machine. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\while\while

^

cl while.cpp /Fawhile.asm /Fewhile.exe

The following is the output of running the preceding commands:

Figure 10.2: While condition without optimization

This will generate the assembly code and the EXE file. For further analysis, we have disabled the ASLR manually. To disable ASLR, follow the same steps by using the CFF explorer and change the DllCharacteristics parameter to uncheck DLL can move. For step-by-step instructions, refer to the Appendix.

Now let’s move on to the generated assembly listing:

Figure 10.3: while.asm

Let’s walk through the assembly code line by line. As we have discussed most of the initial instructions of the assembly listing in the earlier chapters, we will move on to the instructions from the start of the main procedure.

▼Line 22-25

; Line 7

push ebp

mov ebp, esp

push ecx

The assembly code starts with the main function prologue first with the PUSH instruction and then with the MOV instruction. ECX is pushed onto the stack to create room for the iNumber integer variable of size 4 bytes.

▼Line 26-27

; Line 8

mov DWORD PTR _iNumber$[ebp], 1

The C/C++ code for this is as follows:

int iNumber = 1 ;

The iNumber integer variable is initialized to 1.

In the ASM code, we created room for the iNumber local variable to main function by pushing ECX onto the stack. 1 is moved to this memory location on stack [EBP-4] (We get this by adding iNumber macro, which is _iNumber$ = -4 with EBP). Let’s check the stack state by placing the breakpoint at the start of the main procedure and then stepping into the instructions. Refer to the following stack state and a screenshot of x32dbg:

[ESP]  0012FF3C  00000001  iNumber is stored here

[ESP+0x4] 0012FF40  0012FF88  [EBP]

[ESP+0x4] 0012FF44  00401224  return to while.00401224 from while.00401000

Figure 10.4: iNumber is stored on stack

▼Line 28-31

$LN2@main:

; Line 9

cmp DWORD PTR _iNumber$[ebp], 10  ; 0000000aH

jg SHORT $LN3@main

The C/C++ code for this is as follows:

while (iNumber <= 10)

As we see in the C/C++ code while condition checks iNumber value, for less than or equal to 10 (0x0A). In the ASM code, the value stored on the stack at [EBP-4] is compared with the value 10 with the CMP instruction. As both the operands are not equal in the current state, executing the CMP instruction will set ZF=0.

When the value at [EBP-4] will be greater than 10, then JG (Jump Greater) will jump to the label $LN3@main and the label $LN3@main will point towards the closure of the main function.

Considering the current case where iNumber is 1, the stack state and the screenshot of x32dbg will be as follows:

[ESP]  0012FF3C  00000001  iNumber is stored here

[ESP+0x4] 0012FF40  0012FF88  [EBP]

[ESP+0x4] 0012FF44  00401224  return to while.00401224 from while.00401000

Figure 10.5: After compare instruction when iNumber=1

▼Line 32-37

; Line 11

mov eax, DWORD PTR _iNumber$[ebp]

push eax

push OFFSET $SG4681

call _printf

add esp, 8

On line 11 of the C/C++ code is the printf function to print iNumber on the console.

The ASM code moves the value stored on the stack at [EBP-4] to the EAX register, which is later in the next instruction pushed onto the stack along with the other argument, string constant $SG4681. When both the arguments to the printf function are passed onto the stack, the call to printf is made. On return from the printf function, ADD is executed to clean the stack by adding 8 bytes to ESP.

The stack state and x32dbg screenshot before the ADD instruction will be as follows:

[ESP]  0012FF34  00408140  "%d\n", as a parameter to printf()

[ESP+0x4] 0012FF38  00000001  iNumber is pushed as a parameter to printf()

[ESP+0x8] 0012FF3C  00000001  iNumber is stored here

[ESP+0xC] 0012FF40  0012FF88  [EBP]

[ESP+0x10] 0012FF44  00401224  return to while.00401224 from while.00401000

Figure 10.6: After printf

▼Line 38-41

; Line 12

mov ecx, DWORD PTR _iNumber$[ebp]

add ecx, 1

mov DWORD PTR _iNumber$[ebp], ecx

Line 12 of the C/C++ code increments the iNumber value by 1.

iNumber = iNumber + 1 ;

In ASM, we are incrementing the iNumber value by first moving the iNumber value stored at [EBP-4] to ECX, and then incrementing ECX by 1. Once incremented, we will move ECX back to [EBP-4] memory location. This process or iteration is repeated every time until iNumber is less than or equal to 10 (0x0A).

The stack state and the screenshot of x32dbg when iNumber is incremented by 1 will be as follows:

[ESP]  0012FF3C  00000002  iNumber is stored here, now its incremented by 1

[ESP+0x4] 0012FF40  0012FF88  [EBP]

[ESP+0x4] 0012FF44  00401224  return to while.00401224 from while.00401000

Figure 10.7: When iNumber is incremented by 1

▼Line 42-43

; Line 13

jmp SHORT $LN2@main

This is an unconditional jump to the $LN2@main label, where the value stored at [EBP-4] is again compared with the value 10 (0x0A). If the value is less than or equal to 10, then the same instructions will print iNumber on screen, along with incrementing the iNumber value at [EBP-4] by 1.

The stack state will be the same as earlier after JMP.

[ESP]  0012FF3C  00000002  iNumber is stored here, now its incremented by 1

[ESP+0x4] 0012FF40  0012FF88  [EBP]

[ESP+0x4] 0012FF44  00401224  return to while.00401224 from while.00401000

Instruction pointer will be pointed to 0x0040100B as follows:

Figure 10.8: unconditional jump to $LN2@main

Now, we will see what happens when iNumber becomes 10 (0x0A). There are two ways to analyze the code:

Manually step into the code instruction by instruction. This is a time-consuming process.

Set the breakpoint on the condition when either the value stored at [EBP-4] becomes 0x0A or ECX == 0x0A. We will set the conditional breakpoint in x32dbg, when ECX == 0x0A.

Now, to set the conditional breakpoint, set the software breakpoint (press key F2) first at:

0x00401025   add ecx, 1

Then right-click on the instruction and select the Edit Breakpoint command from the context menu. Fill in the following conditional expression and then confirm and close the dialog box, as shown in the following screenshot:

Figure 10.9: Set the breakpoint on the condition

Run the code until you hit the conditional breakpoint. From there, we can manually step into the instructions. The stack state and the screenshot of x32dbg at this point will be as follows:

[ESP]  0012FF3C  0000000A  iNumber is stored here, now its incremented to 0xA

[ESP+0x4] 0012FF40  0012FF88  [EBP]

[ESP+0x4] 0012FF44  00401224  return to while.00401224 from while.00401000

Figure 10.10: When iNumber is incremented to 0x0A

As we step into, ECX and [EBP-4] will become 0x0B.

Figure 10.11: When iNumber is incremented to 0x0B

In the next iteration, the JG instruction will jump the instruction pointer to the $LN3@main label, which we will discuss next.

▼Line 44-52

$LN3@main:

; Line 14

xor eax, eax

mov esp, ebp

pop ebp

ret 0

_main ENDP

_TEXT ENDS

END

This will zero the EAX and call the main function epilogue. With this, the TEXT segment and code is ended.

While condition with Optimization

Compile the code with the optimization flag, /Ox flag. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Ox: Enable maximum optimization

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\while\while

^

cl while.cpp /Fawhile-Optimized.asm /Ox /Fewhile-Optimized.exe

The following is the output of running the preceding commands:

Figure 10.12: While condition with Optimization

It will generate the assembly code and the EXE file. Follow the same process to disable the ASLR manually. To disable ASLR, follow the same steps by using the CFF explorer and change the DllCharacteristics parameter to uncheck DLL can move. For a step-by-step process to disable ASLR, refer to the Appendix.

Now, let’s move on to the generated assembly listing:

Figure 10.13: while-Optimized.asm

We will walk through the generated assembly code instruction by instruction:

▼Line 21-22

; Line 7

push esi

The main procedure starts by pushing the ESI register. By pushing ESI onto the stack, we are preserving the ESI register value. ESI is restored back with POP ESI instruction at the end of the main procedure. So, it is basically persevering and restoring the register value at the start and end of the function with the use of the PUSH & POP instructions, respectively.

▼Line 23-25

; Line 8

mov esi, 1

npad 10

Line 8 of the C/C++ code initializes the iNumber variable:

int iNumber = 1 ;

In the ASM code, the ESI register will be used to hold the iNumber value and for subsequent operations. The MOV instruction initializes iNumber by moving 0x01 into the ESI register.

Next is the npad macro which inserts non-destructive and non-operational instructions. It means that it will insert an instruction that does nothing. For information on npad, please refer to the Appendix section. Our assembly listing is using npad 10, which is defined in LISTING.INC as jmp .+A; .npad 7; .npad 1, where

npad 10 is

Figure 10.14: npad 10

npad 7 is

Figure 10.15: npad 7

npad 1 is

Figure 10.16: npad 1

Together, npad 10 makes:

jmp .+A -> It will add 0x0A to instruction location (0x00401006 + 0x0A)

lea esp, [esp+00000000]

nop

Let’s see how the npad 10 macro is resolved in the x32dbg screenshot:

Figure 10.17: npad 10 in x32dbg

All the three instructions marked in RED have no effect on the code flow; they are just equivalent to a series of NOPs of size 10 bytes. The compiler pads 10 bytes between the mov esi,1 instruction and the label $LL2@main (defined next).

▼Line 26-30

$LL2@main:

; Line 11

push esi

push OFFSET $SG4681

call _printf

The $LL2@main label is defined followed by the comment as line 11. Line 11 of the C/C++ code prints the iNumber on the screen using the printf function.

printf("%d\n", iNumber);

In the ASM code, before calling the printf function, two arguments to printf are pushed onto stack. The first one is ESI holding the iNumber and the other is the string constant defined by $SG4681. The stack state after the printf call is shown as follows, along with the x32dbg screenshot for a better understanding:

[ESP]  0012FF38  00408140  "%d\n", parameter to printf()

[ESP+0x4] 0012FF3C  00000001  iNumber is pushed as a parameter to printf()

[ESP+0x8] 0012FF40  00000000  Old value of ESI is preserved

[ESP+0xC] 0012FF44  00401219  return to 0x00401219 from 0x00401000

Figure 10.18: After printf

▼Line 31-35

; Line 12

inc esi

add esp, 8

cmp esi, 10     ; 0000000aH

jle SHORT $LL2@main

The C/C++ code on line 12 increments the iNumber by 1.

iNumber = iNumber + 1 ;

In ASM, we are incrementing the iNumber stored in ESI by using the INC instruction. INC adds 1 to ESI and stores the result back in the ESI register. The ADD instruction cleans the stack by adding 8 bytes to ESP. Once the stack is cleaned, the value in ESI is compared with 10 (0x0A). It will perform a signed comparison jump after a CMP instruction, if the value in ESI is less than or equal to 10 (0x0A).

In our case, the current value of ESI = 0x02 after incrementing. So, the instruction pointer will be jumped to the $LL2@main label. The ESI is again printed on the console. This loop iterates until ESI becomes 0x0B. When ESI is incremented to 0x0B, a signed comparison jump after a CMP instruction will not take place. So, the instruction pointer will move to the end of the assembly code. The stack state at this point:

[ESP-0x8] 0012FF38  00408140  NOW JUNK

[ESP-0x4] 0012FF3C  0000000A  NOW JUNK

[ESP]  0012FF40  00000000  Old value of ESI is preserved

[ESP+0xC] 0012FF44  00401219  return to 0x00401219 from 0x00401000

Figure 10.19: ESI is incremented to 0x0B

▼Line 36-42

; Line 14

xor eax, eax

pop esi

ret 0

_main ENDP

_TEXT ENDS

END

It is the same as explained in the other sections. EAX is XOR’ed to return 0. A point to note here is that the value of ESI is restored back using the POP ESI instruction. With this, our main procedure, TEXT segment, and code ends. The stack state after popping ESI is as follows:

[ESP-0x8] 0012FF38  00408140  NOW JUNK

[ESP-0x4] 0012FF3C  0000000A  NOW JUNK

[ESP]  0012FF40  00000000  NOW JUNK

[ESP+0xC] 0012FF44  00401219  return to 0x00401219 from 0x00401000

Figure 10.20: Stack cleaned

For Loop

In this section, we will use for loop in the C/C++ code, which initializes an integer variable and increments it by 2 to print it on the screen or console.

Figure 10.21: for.cpp

For Loop without Optimization

Compile the code without optimization in the MSVC compiler on the same x86 Windows machine. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\for\for

^

cl for.cpp /Fafor.asm /Fefor.exe

The following is the output of running the preceding commands:

Figure 10.22: For Loop without Optimization

The compilation generates the assembly code and the EXE file. To manually disable ASLR, use the CFF explorer and change the DllCharacteristics parameter to uncheck DLL can move.

Now, let’s move on to the generated assembly listing:

Figure 10.23: for.asm

▼Line 22-25

; Line 7

push ebp

mov ebp, esp

push ecx

The ASM code starts with the main function prologue. The first PUSH ECX will create room for the iNumber integer variable on the stack.

▼Line 26-27

; Line 8

mov DWORD PTR _iNumber$[ebp], 1

The following line 8 of the C/C++ code initializes the iNumber variable to 1, which corresponds to the preceding ASM code:

int iNumber = 1;

In the ASM code, iNumber macro is defined as:

_iNumber$ = -4; size = 4

So, the preceding ASM code resolves to the following instruction in x32dbg:

mov dword ptr ss:[ebp-0x4], 0x1

MOV initializes the iNumber variable on the stack at ss:[ebp-0x4] to 0x01. This is the same place in memory where ECX was pushed for creating room for the iNumber variable. The stack state after this instruction execution is as follows:

[ESP]  0012FF3C  00000001  iNumber variable is stored here

[ESP+0x4] 0012FF40  0012FF88  [EBP]

[ESP+0x8] 0012FF44  0040122D  return to for.0040122D from for.00401000

Figure 10.24: iNumber on stack

▼Line 28-30

; Line 9

mov DWORD PTR _iNumber$[ebp], 1

jmp SHORT $LN3@main

This ASM code corresponds to the C/C++ code line 9, where iNumber is again initialized and the loop is started.

for (iNumber = 1 ; iNumber <= 10 ; iNumber = iNumber + 2)

In ASM, the MOV instruction again initializes the iNumber variable. As the code is not optimized, the compiler does not remove the unwanted or duplicate instructions. JMP is an unconditional jump to $LN3@main label, where the iNumber is compared to 10(0x0A) to get into the loop. We will see the $LN3@main label in the next ASM explanation. The stack state after this unconditional JMP will be the same:

[ESP]  0012FF3C  00000001  iNumber variable is stored here

[ESP+0x4] 0012FF40  0012FF88  [EBP]

[ESP+0x8] 0012FF44  0040122D  return to for.0040122D from for.00401000

▼Line 35-44

$LN3@main:

cmp DWORD PTR _iNumber$[ebp], 10  ; 0000000aH

jg SHORT $LN4@main

; Line 10

mov ecx, DWORD PTR _iNumber$[ebp]

push ecx

push OFFSET $SG4681

call _printf

add esp, 8

jmp SHORT $LN2@main

The preceding ASM code corresponds to the following C/C++ code lines 9-10, which compare iNumber with 10 (0x0A).

for (iNumber = 1 ; iNumber <= 10 ; iNumber = iNumber + 2)

printf ("%d\n", iNumber);

Let’s walk through the ASM code. The CMP instruction compares the iNumber value stored on the stack with 10 (0x0A). It will perform a signed comparison jump if the iNumber at [EBP-0x04] is greater than 10 (0x0A). Currently, the iNumber is 1 so the jump will not happen and the instruction pointer will move to the next section after the line 10 comment in the assembly listing. The CMP instruction also sets ZF=0.

Instructions after the line 10 comment move the iNumber value at [EBP-4] to the ECX register so that ECX can be pushed back to the stack along with the string constant $SG4681 as argument to printf. On call to the printf function, iNumber value = 1 will be printed on the screen.

On return from the printf call, the stack is cleaned with the ADD instruction by adding 8 bytes to ESP. The JMP, which is an unconditional jump, makes the instruction pointer move to the $LN2@main label, where the iNumber is incremented by 2. The stack state after the unconditional jump is as follows:

[ESP-0x8] 0012FF34  00408140  Now JUNK, "%d\n", parameter to print() was pushed

[ESP-0x4] 0012FF38  00000001  Now JUNK, iNumber parameter to print() was pushed

[ESP]  0012FF3C  00000001  iNumber variable is stored here

[ESP+0x4] 0012FF40  0012FF88  [EBP]

[ESP+0x8] 0012FF44  0040122D  return to for.0040122D from for.00401000

Figure 10.25: Stack state after unconditional jump

▼Line 31-34

$LN2@main:

mov eax, DWORD PTR _iNumber$[ebp]

add eax, 2

mov DWORD PTR _iNumber$[ebp], eax

This ASM code corresponds to the following C/C++ code line 9, where the iNumber is incremented by 2.

for (iNumber = 1 ; iNumber <= 10 ; iNumber = iNumber + 2)

In ASM, the MOV instruction takes the iNumber value from [EBP-0x4] to the EAX register, where EAX is incremented by 2 using the ADD instruction and moved back to [EBP-0x4], the memory placeholder for iNumber. Following this instruction is the same instruction when iNumber is compared with 10 (0x0A), as discussed previously. The stack state after the increment is as follows:

[ESP-0x8] 0012FF34  00408140  Now JUNK, "%d\n", parameter to print() was pushed

[ESP-0x4] 0012FF38  00000001  Now JUNK, iNumber parameter to print() was pushed

[ESP]  0012FF3C  00000003  iNumber variable is stored here

[ESP+0x4] 0012FF40  0012FF88  [EBP]

[ESP+0x8] 0012FF44  0040122D  return to for.0040122D from for.00401000

Figure 10.26: iNumber on stack is incremented

▼Line 35-37

$LN3@main:

cmp DWORD PTR _iNumber$[ebp], 10  ; 0000000aH

jg SHORT $LN4@main

This ASM code corresponds to the following C/C++ code line 9, which compares the iNumber with the 10 (0x0A).

for (iNumber = 1 ; iNumber <= 10 ; iNumber = iNumber + 2)

Now, we will take that iteration wherein the iNumber is incremented to 11 (0xB). At this point, it will perform a signed comparison jump as the iNumber at [EBP-0x04] is greater than 10 (0x0A). The iNumber is 11 (0xB), so the jump will happen to the $LN4@main label. The stack state after the JG (JUMP if Greater) is as follows:

[ESP-0x8] 0012FF34  00408140  Now JUNK, "%d\n", parameter to print() was pushed

[ESP-0x4] 0012FF38  00000009  Now JUNK, last iNumber value pushed as arg to print

[ESP]  0012FF3C  0000000B  iNumber variable is stored here

[ESP+0x4] 0012FF40  0012FF88  [EBP]

[ESP+0x8] 0012FF44  0040122D  return to for.0040122D from for.00401000

Figure 10.27: iNumber is incremented to 11 (0x0B)

▼Line 45-53

$LN4@main:

; Line 11

xor eax, eax

mov esp, ebp

pop ebp

ret 0

_main ENDP

_TEXT ENDS

END

In this $LN4@main label, EAX is zeroed to return 0. The function epilogue is called to END the main function, TEXT segment, and code. The stack state will be as follows:

[ESP-0x10] 0012FF34  00408140  Now JUNK, "%d\n", parameter to print() was pushed

[ESP-0xC] 0012FF38  00000009  Now JUNK, last iNumber value pushed as arg to print

[ESP-0x8] 0012FF3C  0000000B Now JUNK,  iNumber variable is stored here

[ESP-0x4] 0012FF40  0012FF88  Now JUNK, EBP popped

[ESP]  0012FF44  0040122D  return to for.0040122D from for.00401000

Figure 10.28: Stack cleaned

For Loop with Optimization

Compile the code with the optimization flag, /Ox flag. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Ox: Enable maximum optimization

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\for\for

^

cl for.cpp /Fafor-Optimized.asm /Ox /Fefor-Optimized.exe

The following is the output of running the preceding commands:

Figure 10.29: For Loop with Optimization

To manually disable the ASLR, use the CFF explorer and change the DllCharacteristics parameter to uncheck DLL can move.

Now, let’s move on to the generated assembly listing:

Figure 10.30: For-Optimized.asm

Let’s walk through the generated ASM code using the optimization flag.

▼Line 21-22

; Line 7

push esi

In the function prologue, ESI is pushed onto the stack and in the function epilogue, ESI is popped up. ESI will be used as the iNumber placeholder. So before using it as a placeholder for the iNumber, the old value ESI is preserved onto the stack and in the main procedure epilogue, the ESI value is restored to the old value using POP ESI.

▼Line 23-25

; Line 9

mov esi, 1

npad 10

Line 9 of the C/C++ code initializes the iNumber variable:

for (iNumber = 1 ; iNumber <= 10 ; iNumber = iNumber + 2)

In the ASM code, the ESI register will be used as a placeholder for the iNumber value. So, 0x01 is moved to the ESI register to initialize the iNumber.

npad macro inserts non-destructive non-operational instructions. For information on npad, please refer to the Appendix section.

The compiler inserts three instructions that will have no effect on the code flow; they are just equivalent to a series of NOPs of size 10 bytes. In x32dbg, npad macro is resolved as marked within the RED box:

Figure 10.31: npad 10

The compiler pads 10 bytes between the mov esi,1 instruction and the label $LL3@main (defined next).

▼Line 26-34

$LL3@main:

; Line 10

push esi

push OFFSET $SG4681

call _printf

add esi, 2

add esp, 8

cmp esi, 10     ; 0000000aH

jle SHORT $LL3@main

Line 9-10 of the C/C++ code initializes the iNumber value to enter into the loop to print iNumber and increment every iNumber iteration by 2.

for (iNumber = 1 ; iNumber <= 10 ; iNumber = iNumber + 2)

printf ("%d\n", iNumber);

In ASM, it is done by pushing ESI and the string constant $SG4681 onto the stack. Once we have both the arguments to the printf function on the stack, the call to the printf function is made. This will print the current value of iNumber stored in the ESI register on the screen.

Now, ESI is incremented by 2 by adding 0x02 to the ESI register. This is to increment the iNumber value by 2.

Next, the ADD instruction cleans up the stack by adding 8 bytes to the ESP register.

CMP will compare the value in ESI with 0x10. It will perform a signed comparison jump after a CMP instruction if the value in ESI is less than or equal to 10 (0x0A).

In our case, the current value of ESI = 0x03 after incrementing, which is less than 10 (0x0A). So, the instruction pointer will jump to the $LL3@main label. The stack state and x32dbg screenshot at this point will be as follows:

[ESP-0x8] 0012FF38  00408140  "%d\n", parameter to printf()

[ESP-0x4] 0012FF3C  00000001  ESI holding iNumber is pushed, as arg to printf

[ESP]  0012FF40  00000000  Old value of ESI is preserved on stack

[ESP+0x4] 0012FF44  0040121B  return to 0x0040121B from 0x00401000

Figure 10.32: After JMP

This loop iterates until ESI becomes 0x0B. When ESI is incremented to 0x0B, a signed comparison jump after a CMP instruction will not take place. So, the instruction pointer will move to the end of the assembly code. The stack state at this point is as follows:

[ESP-0x8] 0012FF38  00408140  "%d\n", parameter to printf()

[ESP-0x4] 0012FF3C  00000009  ESI holding iNumber is pushed, as arg to printf

[ESP]  0012FF40  00000000  Old value of ESI is preserved on stack

[ESP+0x4] 0012FF44  0040121B  return to 0x0040121B from 0x00401000

Figure 10.33: ESI is incremented to 0x0B

▼Line 35-41

; Line 11

xor eax, eax

pop esi

ret 0

_main ENDP

_TEXT ENDS

END

With this, the main function, TEXT segment, and code is ended by XOR’ing EAX to return 0 and the old value of ESI is restored from the stack by the POP ESI instruction. The stack state at this point will be as follows:

[ESP-0xC] 0012FF38  00408140  Now JUNK

[ESP-0x8] 0012FF3C  00000009  Now JUNK

[ESP-0x4] 0012FF40  00000000  Now JUNK

[ESP]  0012FF44  0040121B  return to 0x0040121B from 0x00401000

Conclusion

In this chapter, we learned about the different loop statements in C/C++ and how the code pattern of different loop statements can be identified in a disassembled code. We also found out how to putt a conditional breakpoint to check the stack state and the major flag bits. In the assembly listing generated from loop statement programs, we checked CMP and Jump instruction patterns in assembly and learned the difference between optimized and non-optimized code. In the subsequent chapters, we will talk about real-world problems and their solutions.

CHAPTER 11

Array Code Pattern in Reverse Engineering

Imagine you are a programmer working in some company to develop software to mark the attendance of 50 students in a class. Now, as a software developer, you have an option to define separate variables for students, which is quite difficult to manage and not appropriate in case the student count increases in the future. To manage these situations where you have similar data type variables, every programming language is equipped with the concept of arrays. Arrays are sets of similar elements stored in contiguous memory locations. So, in developing the attendance software for a class, an array will be the most appropriate.

We have already studied the pattern of pointers in the earlier chapter. Pointers and array are correlated to each other. The importance of array in reverse engineering is very important. Sometimes, malware writers code malware in such a way that they infect the list of files with a specific extension. In most of these cases, all the specific file extensions are defined in an array. So, adding any other extension in the future to the malware becomes easy for malware coders. In this chapter, we will work on this real-world problem by coding the same program using array and then reversing it to understand its pattern.

Structure

In this chapter, we will cover the following topics:

Understanding an array

Array loop without Optimization

Array loop with Optimization

Objective

The objective of this chapter is to understand the working of an array with respect to reverse engineering. We will talk about the array code pattern in a disassembled code and how arrays are stored in memory. Arrays are stored in contiguous memory locations and depending on the data types of the array, the storage is allocated in memory. We will also learn to put a conditional breakpoint in code execution. We will also cover an array program pattern when the code is optimized and not optimized.

Understanding an array

In this example, we have declared and defined an array named iArray. We will iterate through the array up to its length and print each element along with the index on the screen.

Figure 11.1: Array.cpp

Array Loop without Optimization

Compile the code without optimization in the MSVC compiler on the same x86 Windows machine. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\Array\Array

^

cl Array.cpp /FaArray.asm /FeArray.exe

The following is the output of running the preceding commands:

Figure 11.2: Array Loop without Optimization

Compilation generates the EXE file and assembly code. Disable ASLR manually. To disable ASLR, use the CFF explorer and change the DllCharacteristics parameter to uncheck DLL can move. For detailed steps to understand how to disable ASLR, please refer to the Appendix.

Now, let’s move on to the generated assembly listing:

Figure 11.3: Array.asm-Part-1

Figure 11.4: Array.asm-Part-2

▼Line 12-14

CONST SEGMENT

$SG4682 DB 'iArray[%d]=%d', 0aH, 00H

CONST ENDS

The code defines two segments, one being CONST SEGMENT. This derivative is used to define the start of the constant segment in the memory. The linker renames CONST SEGMENT to .rdata the code is placed in the .code segment, the constant string is placed in the CONST(.rdata) segment and if not constant, it is placed in the .data segment], which can be dumped using any debugger. Below screenshot shows $SG4682 in the memory dump:

Figure 11.5: .rdata

The $SG4682 is the internal name given by the compiler to handle the string constant. DB, defines the byte which is the data type. 'iArray[%d]=%d', 0aH, 00H is the string data, which is null terminated ASCII string.

By CONST END, the constant segment is ended.

▼Line 15-16

PUBLIC __$ArrayPad$

PUBLIC _main

PUBLIC is the derivative which makes the _main procedure and __$ArrayPad$ macro public, which makes it accessible to other modules.

▼Line 17-19

EXTRN _printf:PROC

EXTRN ___security_cookie:DWORD

EXTRN @__security_check_cookie@4:PROC

The EXTRN derivative declares the extern function, which is printf in our case. All functions begin with an underscore.

▼Line 21-24

_TEXT SEGMENT

_iArray$ = -48      ; size = 40

__$ArrayPad$ = -8     ; size = 4

_iLoop$ = -4      ; size = 4

This is the start of the _TEXT segment, where our main function code resides. After _TEXT, we have the different variable macro defined. To access the local variable on the stack frame, we have to add _<variable-name>$ to the EBP address.

▼Line 25

_main PROC

This is the start of the main procedure.

▼Line 27-30

; Line 7

push ebp

mov ebp, esp

sub esp, 48; 00000030H

Line 7 of the C/C++ code starts the main function.

int main() {

The ASM code starts with the main function prologue of PUSH and MOV. SUB is creating room for variables on the stack by subtracting 48 (0x30) from ESP.

▼Line 31-33

mov eax, DWORD PTR ___security_cookie

xor eax, ebp

mov DWORD PTR __$ArrayPad$[ebp], eax

To understand all these three instructions, we will first have to understand the concept of stack cookie. A stack cookie is the protection mechanism against buffer overflow attacks. To understand buffer overflow in layman terms, imagine somebody pours more coffee in a cup; the coffee poured in excess will spill on the table. Similarly, when a buffer stored in memory is filled with more data than its size, the excess data will be spilled causing the critical memory locations to be overwritten. A stack cookie is a random value generated at each execution. This value is XOR’ed with EBP and then placed on the stack. This value stored on the stack is placed between local variables (buffer or array in our case) and EBP.

Figure 11.6: Stack cookie

Once this value is stored on the stack after the function prologue, to prevent against buffer overflow attacks, this value is checked before the function epilogue. If this value is not the same before the function epilogue, then it means that a buffer overflow exploit has occurred.

Now coming back to the first MOV instruction, it moves the value stored at the stack cookie location to EAX. EAX is XOR’ed with EBP and the result of XOR is stored in EAX. This XOR value stored in EAX is moved to the stack at ss:[ebp-0x8] location. This is where our stack cookie is stored on the stack. Let’s see the stack state and the stack cookie value in the memory dump.

Stack Cookie stored at 0x0040B000 location is = 0xED4B8BCF

EBP = 0x0012FF40

Stack Cookie XOR EBP = 0xED4B8BCF XOR 0x0012FF40 = 0xED59748F

[ESP]  0012FF10  0012FEFC [EBP-0x30] JUNK Right Now

[ESP+0x4] 0012FF14  00000004 [EBP-0x2C] JUNK Right Now

[ESP+0x8] 0012FF18  0012FF78 [EBP-0x28] JUNK Right Now

[ESP+0xC] 0012FF1C  004024E0 [EBP-0x24] JUNK Right Now

[ESP+0x10] 0012FF20  ED0B1017 [EBP-0x20] JUNK Right Now

[ESP+0x14] 0012FF24  FFFFFFFE [EBP-0x1C] JUNK Right Now  

[ESP+0x18] 0012FF28  0040548C [EBP-0x18] JUNK Right Now

[ESP+0x1C] 0012FF2C  004054A0 [EBP-0x14]  JUNK Right Now

[ESP+0x20] 0012FF30  0040343B [EBP-0x10] JUNK Right Now

[ESP+0x24] 0012FF34  0012FF48 [EBP-0x0C] JUNK Right Now

[ESP+0x28] 0012FF38  ED59748F [EBP-0x08] Stack Cookie xor EBP value placed here

[ESP+0x2C] 0012FF3C  0040343B [EBP-0x04] JUNK Right Now

[ESP+0x30] 0012FF40  0012FF88 [EBP]

[ESP+0x34] 0012FF44  00401299 return to array.00401299 from array.00401000

Figure 11.7: Stack cookie on stack

▼Line 34-44

; Line 8

mov DWORD PTR _iArray$[ebp], 0

mov DWORD PTR _iArray$[ebp+4], 1

mov DWORD PTR _iArray$[ebp+8], 2

mov DWORD PTR _iArray$[ebp+12], 3

mov DWORD PTR _iArray$[ebp+16], 4

mov DWORD PTR _iArray$[ebp+20], 5

mov DWORD PTR _iArray$[ebp+24], 6

mov DWORD PTR _iArray$[ebp+28], 7

mov DWORD PTR _iArray$[ebp+32], 8

mov DWORD PTR _iArray$[ebp+36], 9

Line 8 of the C/C++ code defines iArray of type int.

int iArray[10] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};

In ASM, iArray is of type integer, so each element of the array will occupy 4 bytes of space as required for that integer. All these move instructions will place the elements of iArray on the stack. iArray macro is defined as:

_iArray$ = -48 (-0x30)

So _iArray$[ebp] will correspond to ss:[ebp-0x30]. Similarly, other iArray elements can be accessed on the stack by adding _<variable-name>$ to the EBP address. The stack state after the execution of the preceding instructions is as follows:

[ESP]  0012FF10  00000000  [EBP-0x30] iArray first element

[ESP+0x4] 0012FF14  00000001  [EBP-0x2C] iArray second element

[ESP+0x8] 0012FF18  00000002  [EBP-0x28] iArray third element

[ESP+0xC] 0012FF1C  00000003  [EBP-0x24] iArray forth element

[ESP+0x10] 0012FF20  00000004  [EBP-0x20] iArray fifth element

[ESP+0x14] 0012FF24  00000005  [EBP-0x1C] iArray sixth element

[ESP+0x18] 0012FF28  00000006  [EBP-0x18] iArray seventh element

[ESP+0x1C] 0012FF2C  00000007  [EBP-0x14]  iArray eighth element

[ESP+0x20] 0012FF30  00000008  [EBP-0x10] iArray ninth element

[ESP+0x24] 0012FF34  00000009  [EBP-0x0C] iArray tenth element

[ESP+0x28] 0012FF38  ED59748F  [EBP-0x08] Stack Cookie xor EBP value placed

[ESP+0x2C] 0012FF3C  0040343B  [EBP-0x04] JUNK Right Now

[ESP+0x30] 0012FF40  0012FF88  [EBP]

[ESP+0x34] 0012FF44  00401299  return to array.00401299 from array.00401000

Figure 11.8: iArray on stack

▼Line 35-47

; Line 11

mov DWORD PTR _iLoop$[ebp], 0

jmp SHORT $LN3@main

Line 11 of the C/C++ code initializes the iLoop variable to iterate over the iArray elements. The MOV instruction is placing 0x00 on the iLoop variable placeholder on the stack at ss:[ebp-0x4]. The JMP instruction makes an unconditional jump to the $LN3@main label.

▼Line 52-54

$LN3@main:

cmp DWORD PTR _iLoop$[ebp], 10  ; 0000000aH

jge SHORT $LN1@main

The CMP instruction compares the iLoop value stored on the stack with 10(0x0A). It will perform a signed comparison jump if the iLoop at [EBP-0x04] is greater than or equal to 10(0x0A). Currently, iLoop is 0, so the jump will not happen and the instruction pointer will move to the next instruction. The stack state will be as follows:

[ESP]  0012FF10  00000000  [EBP-0x30] iArray first element

[ESP+0x4] 0012FF14  00000001  [EBP-0x2C] iArray second element

[ESP+0x8] 0012FF18  00000002  [EBP-0x28] iArray third element

[ESP+0xC] 0012FF1C  00000003  [EBP-0x24] iArray forth element

[ESP+0x10] 0012FF20  00000004  [EBP-0x20] iArray fifth element

[ESP+0x14] 0012FF24  00000005  [EBP-0x1C] iArray sixth element

[ESP+0x18] 0012FF28  00000006  [EBP-0x18] iArray seventh element

[ESP+0x1C] 0012FF2C  00000007  [EBP-0x14]  iArray eighth element

[ESP+0x20] 0012FF30  00000008  [EBP-0x10] iArray ninth element

[ESP+0x24] 0012FF34  00000009  [EBP-0x0C] iArray tenth element

[ESP+0x28] 0012FF38  ED59748F  [EBP-0x08] Stack Cookie xor EBP value placed

[ESP+0x2C] 0012FF3C  00000000  [EBP-0x04] iLoop placeholder here

[ESP+0x30] 0012FF40  0012FF88  [EBP]

[ESP+0x34] 0012FF44  00401299  return to array.00401299 from array.00401000

Figure 11.9: iLoop is 0

▼Line 55-64

; Line 12

mov ecx, DWORD PTR _iLoop$[ebp]

mov edx, DWORD PTR _iArray$[ebp+ecx*4]

push edx

mov eax, DWORD PTR _iLoop$[ebp]

push eax

push OFFSET $SG4682

call _printf

add esp, 12     ; 0000000cH

jmp SHORT $LN2@main

Line 12 of the C/C++ code prints the element of iArray array along with the index on the screen.

printf ("iArray[%d]=%d\n", iLoop, iArray[iLoop]);

As the printf function takes three arguments to print on the screen, all these three arguments need to be pushed onto the stack before the call to the printf function. In the ASM code, the first argument to be pushed on the stack will be iArray[iLoop]. To push this value on the stack, the iLoop value stored at [EBP-0x04] is moved to ECX. ECX is the current value of array index. It is multiplied by 4 (the size of the integer) and then added to the iArray macro to calculate the memory location of the element stored for the particular array index. This will give us the value of the first array element in the EDX register. So, by pushing EDX onto the stack, we pushed the first argument to printf onto the stack.

Next argument, iLoop is pushed onto the stack by first pushing the iLoop value stored at [EBP-0x04] to EAX, and the push EAX which is second argument to printf.

Last argument, the string constant is pushed by push OFFSET $SG4682.

Now, the three arguments are on the stack. So a call to printf is made. The stack state at this point in time is as follows:

[ESP]  0012FF04  00408140  [EBP-0x40] "iArray[%d]=%d\n", argument to printf

[ESP+0x4] 0012FF08  00000000  [EBP-0x38] iLoop value is pushed as printf argument

[ESP+0x8] 0012FF0C  00000000  [EBP-0x34] iArray first element pushed as printf arg

ESP+0xC] 0012FF10  00000000  [EBP-0x30] iArray first element

[ESP+0x10] 0012FF14  00000001  [EBP-0x2C] iArray second element

[ESP+0x14] 0012FF18  00000002  [EBP-0x28] iArray third element

[ESP+0x18] 0012FF1C  00000003  [EBP-0x24] iArray forth element

[ESP+0x1C] 0012FF20  00000004  [EBP-0x20] iArray fifth element

[ESP+0x20] 0012FF24  00000005  [EBP-0x1C] iArray sixth element

[ESP+0x24] 0012FF28  00000006  [EBP-0x18] iArray seventh element

[ESP+0x28] 0012FF2C  00000007  [EBP-0x14]  iArray eighth element

[ESP+0x2C] 0012FF30  00000008  [EBP-0x10] iArray ninth element

[ESP+0x30] 0012FF34  00000009  [EBP-0x0C] iArray tenth element

[ESP+0x34] 0012FF38  ED59748F  [EBP-0x08] Stack Cookie xor EBP value placed

[ESP+0x38] 0012FF3C  00000000  [EBP-0x04] iLoop placeholder here

[ESP+0x3C] 0012FF40  0012FF88  [EBP]

[ESP+0x40] 0012FF44  00401299  return to array.00401299 from array.00401000

Figure 11.10: Printf arg on stack

On return from call to the printf function, the stack is cleaned by 12 bytes by the ADD instruction. Next, an unconditional jump to $LN2@main label is made. The stack state after the unconditional jump is as follows:

[ESP-0x0C] 0012FF04  00408140  [EBP-0x40] NOW JUNK

[ESP-0x8] 0012FF08  00000000  [EBP-0x38] NOW JUNK

[ESP-0x4] 0012FF0C  00000000  [EBP-0x34]  NOW JUNK

[ESP]  0012FF10  00000000  [EBP-0x30] iArray first element

[ESP+0x4] 0012FF14  00000001  [EBP-0x2C] iArray second element

[ESP+0x8] 0012FF18  00000002  [EBP-0x28] iArray third element

[ESP+0xC] 0012FF1C  00000003  [EBP-0x24] iArray forth element

[ESP+0x10] 0012FF20  00000004  [EBP-0x20] iArray fifth element

[ESP+0x14] 0012FF24  00000005  [EBP-0x1C] iArray sixth element

[ESP+0x18] 0012FF28  00000006  [EBP-0x18] iArray seventh element

[ESP+0x1C] 0012FF2C  00000007  [EBP-0x14]  iArray eighth element

[ESP+0x20] 0012FF30  00000008  [EBP-0x10] iArray ninth element

[ESP+0x24] 0012FF34  00000009  [EBP-0x0C] iArray tenth element

[ESP+0x28] 0012FF38  ED59748F  [EBP-0x08] Stack Cookie xor EBP value placed

[ESP+0x2C] 0012FF3C  00000000  [EBP-0x04] iLoop placeholder here

[ESP+0x30] 0012FF40  0012FF88  [EBP]

[ESP+0x34] 0012FF44  00401299  return to array.00401299 from array.00401000

Figure 11.11: Stack cleaned

▼Line 48-51

$LN2@main:

mov eax, DWORD PTR _iLoop$[ebp]

add eax, 1

mov DWORD PTR _iLoop$[ebp], eax

At this label, the iLoop value from [EBP-0x04] is moved to EAX and incremented by 1. This increment is placed back to [EBP-0x4]. The instruction pointer will again iterate over the label $LN3@main to compare the iLoop value with 10(0x0A).

▼Line 52-54

$LN3@main:

cmp DWORD PTR _iLoop$[ebp], 10  ; 0000000aH

jge SHORT $LN1@main

Now, imagine that we iterated 9 times over the instructions and the iLoop value has become 10 (0x0A). At this point, the CMP instruction will perform a signed comparison jump as the iLoop at [EBP-0x04] is equal to 10(0x0A) and it will also set ZF=1. As iLoop is 10(0xA), so the jump will happen to the $LN1@main label. The stack state after JUMP if greater equal (JGE) is as follows:

[ESP-0x0C] 0012FF04  00408140  [EBP-0x40] NOW JUNK

[ESP-0x8] 0012FF08  00000009  [EBP-0x38] NOW JUNK

[ESP-0x4] 0012FF0C  00000009  [EBP-0x34] NOW JUNK

[ESP]  0012FF10  00000000  [EBP-0x30] iArray first element

[ESP+0x4] 0012FF14  00000001  [EBP-0x2C] iArray second element

[ESP+0x8] 0012FF18  00000002  [EBP-0x28] iArray third element

[ESP+0xC] 0012FF1C  00000003  [EBP-0x24] iArray forth element

[ESP+0x10] 0012FF20  00000004  [EBP-0x20] iArray fifth element

[ESP+0x14] 0012FF24  00000005  [EBP-0x1C] iArray sixth element

[ESP+0x18] 0012FF28  00000006  [EBP-0x18] iArray seventh element

[ESP+0x1C] 0012FF2C  00000007  [EBP-0x14] iArray eighth element

[ESP+0x20] 0012FF30  00000008  [EBP-0x10] iArray ninth element

[ESP+0x24] 0012FF34  00000009  [EBP-0x0C] iArray tenth element

[ESP+0x28] 0012FF38  ED59748F  [EBP-0x08] Stack Cookie xor EBP value placed

[ESP+0x2C] 0012FF3C  0000000A  [EBP-0x04] iLoop placeholder here

[ESP+0x30] 0012FF40  0012FF88  [EBP]

[ESP+0x34] 0012FF44  00401299  return to array.00401299 from array.00401000

Figure 11.12: The stack state after JGE

▼Line 65-77

$LN1@main:

; Line 14

xor eax, eax

; Line 15

mov ecx, DWORD PTR __$ArrayPad$[ebp]

xor ecx, ebp

call @__security_check_cookie@4

mov esp, ebp

pop ebp

ret 0

_main ENDP

_TEXT ENDS

END

At this label, EAX is zeroed using XOR to return 0 as per line 14 of the C/C++ code.

Security Cookie is checked before the function epilogue by first moving the Security Cookie stored at [EBP-0x08] to ECX and then XOR ECX with EBP. A call to the security_check_cookie procedure is made to prevent the buffer overflow attack.

Security Cookie stored at [EBP-0x08] = 0xED59748F

EBP = 0x0012FF40

ECX XOR EBP = 0xED59748F XOR 0x0012FF40 = 0xED4B8BCF

The result will be saved in ECX.

As we can see, this value is the same as the value generated and stored at the 0x0040B000 memory location. As we are dealing with little-endian, the value is stored in the reversed order. The stack state at this point is the same as earlier:

[ESP-0x0C] 0012FF04  00408140  [EBP-0x40] NOW JUNK

[ESP-0x8] 0012FF08  00000009  [EBP-0x38] NOW JUNK

[ESP-0x4] 0012FF0C  00000009  [EBP-0x34] NOW JUNK

[ESP]  0012FF10  00000000  [EBP-0x30] iArray first element

[ESP+0x4] 0012FF14  00000001  [EBP-0x2C] iArray second element

[ESP+0x8] 0012FF18  00000002  [EBP-0x28] iArray third element

[ESP+0xC] 0012FF1C  00000003  [EBP-0x24] iArray forth element

[ESP+0x10] 0012FF20  00000004  [EBP-0x20] iArray fifth element

[ESP+0x14] 0012FF24  00000005  [EBP-0x1C] iArray sixth element

[ESP+0x18] 0012FF28  00000006  [EBP-0x18] iArray seventh element

[ESP+0x1C] 0012FF2C  00000007  [EBP-0x14]  iArray eighth element

[ESP+0x20] 0012FF30  00000008  [EBP-0x10] iArray ninth element

[ESP+0x24] 0012FF34  00000009  [EBP-0x0C] iArray tenth element

[ESP+0x28] 0012FF38  ED59748F  [EBP-0x08] Stack Cookie xor EBP value placed

[ESP+0x2C] 0012FF3C  0000000A  [EBP-0x04] iLoop placeholder here

[ESP+0x30] 0012FF40  0012FF88  [EBP]

[ESP+0x34] 0012FF44  00401299  return to array.00401299 from array.00401000

Figure 11.13: Call to security_check_cookie

If we step into the call @__security_check_cookie@4, we can see that our XOR result saved in ECX is compared with the security cookie stored at 0x0040B000.

Figure 11.14: Security_check_cookie function

As our security cookie stored at 0x0040B000 is the same as that in ECX, ZF=1, the instruction pointer will return back to the main function. On return, the function epilogue is called to end the main function, TEXT segment, and code. Below is the stack state explained before return instruction:

[ESP-0x34] 0012FF10  00000000  NOW JUNK

[ESP-0x30] 0012FF14  00000001  NOW JUNK

[ESP-0x2C] 0012FF18  00000002  NOW JUNK

[ESP-0x28] 0012FF1C  00000003  NOW JUNK

[ESP-0x24] 0012FF20  00000004  NOW JUNK

[ESP-0x20] 0012FF24  00000005  NOW JUNK

[ESP-0x1C] 0012FF28  00000006  NOW JUNK

[ESP-0x18] 0012FF2C  00000007  NOW JUNK

[ESP-0x14] 0012FF30  00000008  NOW JUNK

[ESP-0x10] 0012FF34  00000009  NOW JUNK

[ESP-0x0C] 0012FF38  ED59748F  NOW JUNK

[ESP-0x08] 0012FF3C  0000000A  NOW JUNK

[ESP-0x04] 0012FF40  0012FF88  NOW JUNK

[ESP]  0012FF44  00401299  return to array.00401299 from array.00401000

Figure 11.15: Stack cleaned

Array Loop with Optimization

Compile the code with the optimization flag, /Ox flag. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Ox: Enable maximum optimization

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\Array\Array

^

cl Array.cpp /FaArray-Optimized.asm /Ox /FeArray-Optimized.exe

The following is the output of running the preceding commands:

Figure 11.16: Array Loop with Optimization

The compilation generates the EXE file and assembly code. Disable ASLR manually. To disable ASLR, use the CFF explorer and change the DllCharacteristics parameter to uncheck DLL can move. For detailed steps to disable ASLR, please refer to the Appendix.

Now, let us move on to generated assembly listing:

Figure 11.17: Array-Optimized.asm-Part-1

Figure 11.18: Array-Optimized.asm-Part-2

With optimization enabled, all of the unwanted code is removed in the ASM listing. As we walk through the ASM code, we see that the standard function prologue and the epilogue are not in the optimized code. So, the EBP reference is replaced with ESP in all the instructions.

We will directly jump to the instructions as we have covered some of the common ASM listings in the earlier section.

▼Line 26-31

; Line 7

sub esp, 44     ; 0000002cH

mov eax, DWORD PTR ___security_cookie

xor eax, esp

mov DWORD PTR __$ArrayPad$[esp+44], eax

push esi

The SUB instruction creates room for the local variables on the stack by adding 44 (0x2C) bytes to ESP. 44 bytes are coming from the 10 elements of iArray wherein each element is of 4 bytes each and 4 bytes are used for storing the stack cookie on the stack to prevent buffer overflow as explained earlier.

The MOV instruction moves the stack cookie stored in the .data segment to the EAX register, where it is XOR’ed with ESP. The XOR’ed result is placed on the stack to prevent a buffer overflow exploit to happen. This XOR value will be checked before the main function epilogue.

ESI will be used for the placeholder of the iLoop variable; the old value of ESI is preserved on the stack using the PUSH instruction. The stack state after pushing the ESI register is as follows:

[ESP]  0012FF14  00000000  old ESI value is preserved here

[ESP+0x04] 0012FF18  0012FF78  JUNK

[ESP+0x08] 0012FF1C  004024D0  JUNK

[ESP+0x0C] 0012FF20  132AFDF8  JUNK

[ESP+0x010] 0012FF24  FFFFFFFE  JUNK

[ESP+0x014] 0012FF28  0040547C  JUNK

[ESP+0x18] 0012FF2C  00405490  JUNK

[ESP+0x1C] 0012FF30  0040342B  JUNK

[ESP+0x20] 0012FF34  0012FF48  JUNK

[ESP+0x24] 0012FF38  004028AE  JUNK

[ESP+0x28] 0012FF3C  0040342B  JUNK

[ESP+0x2C] 0012FF40  13789938  XOR of stack cookie and ESP is stored here

[ESP+0x30] 0012FF44  0040128B  ESP before ‘sub esp, 44’ instruction

Figure 11.19: Old ESI value is preserved

▼Line 32-44

; Line 8

xor esi, esi

mov DWORD PTR _iArray$[esp+48], esi

mov DWORD PTR _iArray$[esp+52], 1

mov DWORD PTR _iArray$[esp+56], 2

mov DWORD PTR _iArray$[esp+60], 3

mov DWORD PTR _iArray$[esp+64], 4

mov DWORD PTR _iArray$[esp+68], 5

mov DWORD PTR _iArray$[esp+72], 6

mov DWORD PTR _iArray$[esp+76], 7

mov DWORD PTR _iArray$[esp+80], 8

mov DWORD PTR _iArray$[esp+84], 9

npad 3

Line 8 of the C/C++ code initializes iArray:

int iArray[10] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};

In the ASM code, the XOR instruction resets the ESI register to 0x00 and then moves ESI to the [ESP+0x04] location as the first element of iArray. All the other MOV instructions push the remaining elements of iArray on the stack.

We see the npad macro, which inserts the non-destructive and non-operational instructions rather than a series of NOP instructions. Npad 3 corresponds to lea ecx, [ecx+00]. For more details related to npad, refer to the Appendix. The stack state immediately after the npad macro is as follows:

[ESP]  0012FF14  00000000   old ESI value is preserved here

[ESP+0x04] 0012FF18  00000000   iArray first element pushed here

[ESP+0x08] 0012FF1C  00000001   iArray second element pushed here

[ESP+0x0C] 0012FF20  00000002   iArray third element pushed here

[ESP+0x010] 0012FF24  00000003   iArray forth element pushed here

[ESP+0x014] 0012FF28  00000004   iArray fifth element pushed here

[ESP+0x18] 0012FF2C  00000005   iArray sixth element pushed here

[ESP+0x1C] 0012FF30  00000006   iArray seventh element pushed here

[ESP+0x20] 0012FF34  00000007   iArray eighth element pushed here

[ESP+0x24] 0012FF38  00000008   iArray ninth element pushed here

[ESP+0x28] 0012FF3C  00000009   iArray tenth element pushed here

[ESP+0x2C] 0012FF40  13789938   XOR of stack cookie and ESP is stored here

[ESP+0x30] 0012FF44  0040128B   ESP before ‘sub esp, 44’ instruction

Figure 11.20: The stack state after npad

▼Line 45-55

$LL3@main:

; Line 12

mov eax, DWORD PTR _iArray$[esp+esi*4+48]

push eax

push esi

push OFFSET $SG4682

call _printf

inc esi

add esp, 12     ; 0000000cH

cmp esi, 10     ; 0000000aH

jl SHORT $LL3@main

Line 12 of the C/C++ code prints the elements of iArray along with the index on the screen.

printf ("iArray[%d]=%d\n", iLoop, iArray[iLoop]);

This same concept was discussed in the previous section without optimization. The printf function takes three arguments to the print iArray elements and indexes on the screen. So, all these three arguments need to be pushed onto the stack before the call to the printf function.

In the ASM code, the first argument to be pushed on the stack will be iArray[iLoop]. To push this value on the stack, the iLoop value stored in the ESI register is multiplied by 4 (the size of the integer) and then added to the iArray macro and offset of 48 bytes to calculate the memory location of the element stored for the particular iLoop value. So, during the first iteration when the iLoop stored at ESI = 0, the first MOV instruction will result in:

mov eax, DWORD PTR _iArray$[esp+esi*4+48]

mov eax, dword ptr ss:[esp+esi*4+0x4], as ESI = 0

mov eax, dword ptr ss:[esp+0x4]

This will give us the value of the first array element in the EAX register. So, by pushing EAX onto the stack, we pushed the first argument to printf onto the stack.

The next argument to printf is the iLoop value. It is pushed onto the stack by pushing the ESI register.

Last argument, which is string constant is pushed by push OFFSET $SG4682.

Now that the three arguments are on the stack, a call to printf is made. Stack at this point in time is as follows:

[ESP]  0012FF08  00408140   "iArray[%d]=%d\n", string argument to printf

[ESP+0x04] 0012FF0C  00000000   iLoop value, ESI is pushed as printf argument

[ESP+0x08] 0012FF10  00000000   iArray first element is pushed as printf argument

[ESP+0x0C] 0012FF14  00000000   old ESI value is preserved here

[ESP+0x10] 0012FF18  00000000   iArray first element pushed here

[ESP+0x14] 0012FF1C  00000001   iArray second element pushed here

[ESP+0x18] 0012FF20  00000002   iArray third element pushed here

[ESP+0x1C] 0012FF24  00000003   iArray forth element pushed here

[ESP+0x20] 0012FF28  00000004   iArray fifth element pushed here

[ESP+0x24] 0012FF2C  00000005   iArray sixth element pushed here

[ESP+0x2C] 0012FF30  00000006   iArray seventh element pushed here

[ESP+0x30] 0012FF34  00000007   iArray eighth element pushed here

[ESP+0x34] 0012FF38  00000008   iArray ninth element pushed here

Figure 11.21: Arg to printf on stack

The INC instruction will increment the ESI register. This means that it increments the iLoop value stored in ESI.

The ADD instruction cleans the stack by 12 bytes. Next, ESI is compared with 10 (0x0A). At this point, the CMP instruction will perform a signed comparison jump to the label $LL3@main as the iLoop at ESI less than 10(0x0A). The stack state after JUMP if Less than (JL) is as follows:

[ESP-0x0C] 0012FF08  00408140   Now JUNK

[ESP-0x08] 0012FF0C  00000000   Now JUNK

[ESP-0x04] 0012FF10  00000000   Now JUNK

[ESP]  0012FF14  00000000   old ESI value is preserved here

[ESP+0x04] 0012FF18  00000000   iArray first element pushed here

[ESP+0x08] 0012FF1C  00000001   iArray second element pushed here

[ESP+0x0C] 0012FF20  00000002   iArray third element pushed here

[ESP+0x010] 0012FF24  00000003   iArray forth element pushed here

[ESP+0x014] 0012FF28  00000004   iArray fifth element pushed here

[ESP+0x18] 0012FF2C  00000005   iArray sixth element pushed here

[ESP+0x1C] 0012FF30  00000006   iArray seventh element pushed here

[ESP+0x20] 0012FF34  00000007   iArray eighth element pushed here

[ESP+0x24] 0012FF38  00000008   iArray ninth element pushed here

[ESP+0x28] 0012FF3C  00000009   iArray tenth element pushed here

[ESP+0x2C] 0012FF40  13789938   XOR of stack cookie and ESP is stored here

[ESP+0x30] 0012FF44  0040128B   ESP before ‘sub esp, 44’ instruction

Figure 11.22: The stack state after JL

Now, imagine that we iterated 10 times over the instructions and the iLoop value at ESI has become 10 (0x0A). At this point, the CMP instruction will result in ZF=1 and the jump will not happen to the $LL3@main label. The instruction pointer will move to the next instruction. The stack state at this stage will be:

[ESP-0x0C] 0012FF08  00408140   Now JUNK

[ESP-0x08] 0012FF0C  00000009   Now JUNK

[ESP-0x04] 0012FF10  00000009   Now JUNK

[ESP]  0012FF14  00000000   old ESI value is preserved here

[ESP+0x04] 0012FF18  00000000   iArray first element pushed here

[ESP+0x08] 0012FF1C  00000001   iArray second element pushed here

[ESP+0x0C] 0012FF20  00000002   iArray third element pushed here

[ESP+0x010] 0012FF24  00000003   iArray forth element pushed here

[ESP+0x014] 0012FF28  00000004   iArray fifth element pushed here

[ESP+0x18] 0012FF2C  00000005   iArray sixth element pushed here

[ESP+0x1C] 0012FF30  00000006   iArray seventh element pushed here

[ESP+0x20] 0012FF34  00000007   iArray eighth element pushed here

[ESP+0x24] 0012FF38  00000008   iArray ninth element pushed here

[ESP+0x28] 0012FF3C  00000009   iArray tenth element pushed here

[ESP+0x2C] 0012FF40  13789938   XOR of stack cookie and ESP is stored here

[ESP+0x30] 0012FF44  0040128B   ESP before ‘sub esp, 44’ instruction

Figure 11.23: The stack state after ESI is 0x0A

▼Line 56-66

; Line 15

mov ecx, DWORD PTR __$ArrayPad$[esp+48]

pop esi

xor ecx, esp

xor eax, eax

call @__security_check_cookie@4

add esp, 44     ; 0000002cH

ret 0

_main ENDP

_TEXT ENDS

END

The MOV instruction will move the stack cookie stored at [ESP+0x2C] to ECX.

The POP instruction restores the value of ESI from the stack.

The XOR instruction will XOR the stack cookie moved to ECX with ESP, and the result of XOR will be stored back in the ECX register. On the call to the security_check_cookie procedure, this ECX value is compared with the stack cookie stored in the .data section.

Figure 11.24: Call to security_check_cookie

As the stack cookie value is unchanged, the instruction pointer will return back to main. On return, EAX is XOR with EAX to return 0 and the ADD instruction will clean up the stack to end the main procedure, TEXT segment, and code. The stack state in the end will be as follows:

[ESP-0x3C] 0012FF08  00408140   Now JUNK

[ESP-0x38] 0012FF0C  00000009   Now JUNK

[ESP-0x34] 0012FF10  00000009   Now JUNK

ESP-0x30] 0012FF14  00401087   Now JUNK

[ESP-0x2C] 0012FF18  00000000   Now JUNK

[ESP-0x28] 0012FF1C  00000001   Now JUNK

[ESP-0x24] 0012FF20  00000002   Now JUNK

[ESP-0x20] 0012FF24  00000003   Now JUNK

[ESP-0x1C] 0012FF28  00000004   Now JUNK

[ESP-0x18] 0012FF2C  00000005   Now JUNK

[ESP-0x14] 0012FF30  00000006   Now JUNK

[ESP-0x10] 0012FF34  00000007   Now JUNK

[ESP-0x0C] 0012FF38  00000008   Now JUNK

[ESP-0x08] 0012FF3C  00000009   Now JUNK

[ESP-0x04] 0012FF40  13789938   Now JUNK

[ESP]  0012FF44  0040128B   ESP at the start of main function

Figure 11.25: The stack state in the end

Conclusion

In this chapter, we discussed about the working of an array with respect to reverse engineering. We saw the array code pattern in a disassembled code and understood how arrays are stored in contiguous memory locations. As an integer occupies 4 bytes of memory, so the integer array occupies 4 bytes multiplied by the number of elements in an array. We saw how contiguous memory locations are allocated in stack. We also covered the array program pattern when a code is optimized and not optimized. In the next chapter, we will talk about reversing structures that can handle dissimilar data types.

CHAPTER 12

Structure Code Pattern in Reverse Engineering

In the real world, we can describe an individual using several attributes. These attributes, or we can say parameters or characteristics, help us identify an individual uniquely. The attributes using which we can uniquely identify an individual can be their name, age, sex, height, weight, nationality, and many more. All these attributes correlate to different types of data. It means that the age is an integer, the sex is a string, the weight can be float, and the nationality is a string.

Now, as a computer programmer, if we have to code an application to record the details of all the individuals present in a geographical location, then we have to write an application in such a way that it can handle the data of the individuals in a well-managed and easy manner. In the earlier chapter, we have already seen ordinary variables that can hold a single piece of information. We have also seen how an array can hold data of a similar data type. But in the case of recording individual data, we have to use structures, which are used to record data of dissimilar data types. In this chapter, we will be reversing a structure which is used in many applications.

Structure

In this chapter, we will cover the following topics:

Understanding of structures

Structure without Optimization

Structure with Optimization

Objective

In this chapter, we will study about pointers to structures with respect to reverse engineering. We will talk about structures code pattern in disassembled code and how structures are stored in memory. We will also cover structures program with optimized and not optimized code.

Understanding of structures

In this example, let’s demonstrate the structure pointer. As we have a pointer to an integer or a pointer to a char, similarly, we have a pointer to structures. In C, we have an arrow operator that refers to the elements of a structure.

Figure 12.1: Structures.cpp

Structure without Optimization

Compile the code without optimization in the MSVC compiler on the same x86 Windows machine. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\Structures\Structures

^

cl Structures.cpp /FaStructures.asm /FeStructures.exe

The following is the output of running the preceding commands:

Figure 12.2: Structure without Optimization

The compilation generates the EXE file and assembly code. Disable ASLR manually. To disable ASLR, use the CFF explorer and change the DllCharacteristics parameter to uncheck DLL can move.

Now, let us move onto the generated assembly listing:

Figure 12.3: Structures.asm-Part-1

Figure 12.4: Structures.asm-Part-2

Figure 12.5: Structures.asm-Part-3

Let’s move onto the explanation of the ASM code:

▼Line 30-33

; Line 9

push ebp

mov ebp, esp

sub esp, 64     ; 00000040H

It starts with the main function prologue, wherein the old EBP is pushed onto the stack and the current ESP is moved to a new EBP.

The SUB instruction creates room for the security cookie and the local variable by subtracting 64 bytes from the ESP register. Space allocated for variables on stack can be segregated as follows:

char rgName[40]; = 40 bytes

int  iAge;  = 4 bytes

unsigned long long uMobile; = 8 bytes

struct SSubscriber *puser; = 4 bytes

Stack Cookie = 4 bytes

▼Line 34-36

mov eax, DWORD PTR ___security_cookie

xor eax, ebp

mov DWORD PTR __$ArrayPad$[ebp], eax

As shown in the following screenshot, the stack cookie stored at 0x0040B000 is moved to EAX where it is XOR’ed with EBP. The result of XOR from EAX is moved back to __$ArrayPad$[ebp], which is [EBP-0x08]. The stack at this point in time is as follows:

[ESP]  0012FF00  00000000  Right now JUNK, space for local variables

[ESP+0x04] 0012FF04  00000000  Right now JUNK, space for local variables

[ESP+0x08] 0012FF08  7FFDE000  Right now JUNK, space for local variables

[ESP+0x0C] 0012FF0C  0040345B  Right now JUNK, space for local variables

[ESP+0x10] 0012FF10  0012FEFC  Right now JUNK, space for local variables

[ESP+0x14] 0012FF14  00000004  Right now JUNK, space for local variables

[ESP+0x18] 0012FF18  0012FF78  Right now JUNK, space for local variables

[ESP+0x1C] 0012FF1C  00402500  Right now JUNK, space for local variables

[ESP+0x20] 0012FF20  0185B22A  Right now JUNK, space for local variables

[ESP+0x24] 0012FF24  FFFFFFFE  Right now JUNK, space for local variables

[ESP+0x28] 0012FF28  004054AC  Right now JUNK, space for local variables

[ESP+0x2C] 0012FF2C  004054C0  Right now JUNK, space for local variables

[ESP+0x30] 0012FF30  0040345B  Right now JUNK, space for local variables

[ESP+0x34] 0012FF34  0012FF48  Right now JUNK, space for local variables

[ESP+0x38] 0012FF38  01D7D682  XOR of Stack Cookie and EAX is stored here

[ESP+0x3C] 0012FF3C  0040345B  Right now JUNK, space for local variables

[ESP+0x40] 0012FF40  0012FF88  [EBP]

[ESP+0x44] 0012FF44  004012B3  return to structures.004012B3

Figure 12.6: Stack cookie

▼Line 37-56

; Line 18

mov eax, DWORD PTR $SG5660

mov DWORD PTR _user$[ebp], eax

mov ecx, DWORD PTR $SG5660+4

mov DWORD PTR _user$[ebp+4], ecx

mov dl, BYTE PTR $SG5660+8

mov BYTE PTR _user$[ebp+8], dl

xor eax, eax

mov DWORD PTR _user$[ebp+9], eax

mov DWORD PTR _user$[ebp+13], eax

mov DWORD PTR _user$[ebp+17], eax

mov DWORD PTR _user$[ebp+21], eax

mov DWORD PTR _user$[ebp+25], eax

mov DWORD PTR _user$[ebp+29], eax

mov DWORD PTR _user$[ebp+33], eax

mov WORD PTR _user$[ebp+37], ax

mov BYTE PTR _user$[ebp+39], al

mov DWORD PTR _user$[ebp+40], 30  ; 0000001eH

mov DWORD PTR _user$[ebp+48], -691168947 ; d6cd994dH

mov DWORD PTR _user$[ebp+52], 1

; Line 18

This is a comment which states that the ASM instructions preceding it will represent line 18 of the C/C++ code, which is:

struct SSubscriber user = {"Jitender", 30, 7898765645};    // Structure Variable

In the ASM code, we see several MOV instructions. To understand these instructions, we will have to understand that the elements of structure are always stored in contiguous memory locations. All the MOV instructions will store the elements of structure onto the stack. To understand the stack state, let’s first take each element of structure in a hex representation as follows:

char rgName[40]; = 40 bytes = Jitender (0x4A6974656E646572)

4A

69

74

65

6e

64

65

72

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

int iAge; = 4 bytes = 30 (0x0000001E)

00

00

00

1E

unsigned long long uMobile; = 8 bytes = 7898765645 (0x00000001D6CD994D)

00

00

00

01

D6

CD

99

4D

The stack state after all the MOV instructions will be as follows:

[ESP]  0012FF00  6574694A  4 bytes of array, rgName in little-endian etiJ

[ESP+0x04] 0012FF04  7265646E  4 bytes of array, rgName in little-endian redn

[ESP+0x08] 0012FF08  00000000  String terminator with NULL in remaining array

[ESP+0x0C] 0012FF0C  00000000  NULL values in char array of size 40 bytes

[ESP+0x10] 0012FF10  00000000  NULL values in char array of size 40 bytes

[ESP+0x14] 0012FF14  00000000  NULL values in char array of size 40 bytes

[ESP+0x18] 0012FF18  00000000  NULL values in char array of size 40 bytes

[ESP+0x1C] 0012FF1C  00000000  NULL values in char array of size 40 bytes

[ESP+0x20] 0012FF20  00000000  NULL values in char array of size 40 bytes

[ESP+0x24] 0012FF24  00000000  array from 0x0012FF00 to 0x0012FF28, total 40 bytes

[ESP+0x28] 0012FF28  0000001E  Second element of Structure, iAge is stored here

[ESP+0x2C] 0012FF2C  004054C0  return to 0x004054C0 from 0x00405477

[ESP+0x30] 0012FF30  D6CD994D  4 bytes of uMobile stored here

[ESP+0x34] 0012FF34  00000001  remaining 4 bytes of uMobile stored here

[ESP+0x38] 0012FF38  01D7D682  XOR of Stack Cookie and EAX is stored here

[ESP+0x3C] 0012FF3C  0040345B  structures.0040345B

[ESP+0x40] 0012FF40  0012FF88  [EBP]

[ESP+0x44] 0012FF44  004012B3  return to 0x004012B3 from 0x00401000

Figure 12.7: Stack after all the MOV instructions

▼Line 57-59

; Line 21

lea ecx, DWORD PTR _user$[ebp]

mov DWORD PTR _puser$[ebp], ecx

This is a comment which states that the ASM instructions preceding it will represent line 21 of the C/C++ code, which is:

puser = &user;

In the ASM code, the pointer to structure is pushed onto the ECX register using the Load Effective Address instruction. The pointer to the structure points to the first element of the structure, as structures are always stored in contiguous memory locations.

The MOV instruction will push the pointer to structure on the stack at [EBP-0x04]. The stack state after this instruction will be as follows:

[ESP]  0012FF00  6574694A  4 bytes of array, rgName in little-endian etiJ

[ESP+0x04] 0012FF04  7265646E  4 bytes of array, rgName in little-endian redn

[ESP+0x08] 0012FF08  00000000  String terminator with NULL in remaining array

[ESP+0x0C] 0012FF0C  00000000  NULL values in char array of size 40 bytes

[ESP+0x10] 0012FF10  00000000  NULL values in char array of size 40 bytes

[ESP+0x14] 0012FF14  00000000  NULL values in char array of size 40 bytes

[ESP+0x18] 0012FF18  00000000  NULL values in char array of size 40 bytes

[ESP+0x1C] 0012FF1C  00000000  NULL values in char array of size 40 bytes

[ESP+0x20] 0012FF20  00000000  NULL values in char array of size 40 bytes

[ESP+0x24] 0012FF24  00000000  rgName from 0x0012FF00 to 0x0012FF28, total 40 bytes    

[ESP+0x28] 0012FF28  0000001E  Second element of Structure, iAge is stored here

[ESP+0x2C] 0012FF2C  004054C0  return to 0x004054C0 from 0x00405477

[ESP+0x30] 0012FF30  D6CD994D  4 bytes of uMobile stored here

[ESP+0x34] 0012FF34  00000001  remaining 4 bytes of uMobile stored here

[ESP+0x38] 0012FF38  01D7D682  XOR of Stack Cookie and EAX is stored here

[ESP+0x3C] 0012FF3C  0012FF00  Pointer to structure is stored here, puser

[ESP+0x40] 0012FF40  0012FF88  [EBP]

[ESP+0x44] 0012FF44  004012B3   return to 0x004012B3 from 0x00401000

Figure 12.8: Pointer to structure on stack

▼Line 60-71

; Line 22

mov edx, DWORD PTR _user$[ebp+52]

push edx

mov eax, DWORD PTR _user$[ebp+48]

push eax

mov ecx, DWORD PTR _user$[ebp+40]

push ecx

lea edx, DWORD PTR _user$[ebp]

push edx

push OFFSET $SG5662

call _printf

add esp, 20     ; 00000014H

This is a comment which states that the ASM instructions preceding it will represent line 22 of the C/C++ code, which is:

printf("\n%s %d %llu", user.rgName, user.iAge, user.uMobile);

In the ASM code, we are pushing all the arguments to the stack one by one and then a call to the printf function is made. The stack state just before the call to the printf function is as follows:

[ESP]  0012FEEC  0040814C  "\n%s %d %llu", argument to printf() is pushed here

[ESP+0x04] 0012FEF0  0012FF00  pointer to rgName array is pushed here

[ESP+0x08] 0012FEF4  0000001E  iAge value is pushed here

[ESP+0x0C] 0012FEF8  D6CD994D  uMobile value is pushed here

[ESP+0x10] 0012FEFC  00000001  uMobile value is pushed here

[ESP+0x14] 0012FF00  6574694A  4 bytes of array, rgName in little-endian etiJ

[ESP+0x18] 0012FF04  7265646E  4 bytes of array, rgName in little-endian redn

[ESP+0x1C] 0012FF08  00000000  String terminator with NULL in remaining array

[ESP+0x20] 0012FF0C  00000000  NULL values in char array of size 40 bytes

[ESP+0x24] 0012FF10  00000000  NULL values in char array of size 40 bytes

[ESP+0x28] 0012FF14  00000000  NULL values in char array of size 40 bytes

[ESP+0x2C] 0012FF18  00000000  NULL values in char array of size 40 bytes

[ESP+0x30] 0012FF1C  00000000  NULL values in char array of size 40 bytes

[ESP+0x34] 0012FF20  00000000  NULL values in char array of size 40 bytes

[ESP+0x38] 0012FF24  00000000  rgName from 0x0012FF00 to 0x0012FF28, total 40 bytes    

[ESP+0x3C] 0012FF28  0000001E  Second element of Structure, iAge is stored here

[ESP+0x40] 0012FF2C  004054C0  return to 0x004054C0 from 0x00405477

[ESP+0x44] 0012FF30  D6CD994D  4 bytes of uMobile stored here

[ESP+0x48] 0012FF34  00000001  remaining 4 bytes of uMobile stored here

[ESP+0x50] 0012FF38  01D7D682  XOR of Stack Cookie and EAX is stored here

[ESP+0x54] 0012FF3C  0012FF00  Pointer to structure is stored here, puser

[ESP+0x58] 0012FF40  0012FF88  [EBP]

[ESP+0x5C] 0012FF44  004012B3   return to 0x004012B3 from 0x00401000

Figure 12.9: Before the call to printf

After the call to the printf function, the stack is cleaned using the ADD instruction.

▼Line 72-85

; Line 23

mov eax, DWORD PTR _puser$[ebp]

mov ecx, DWORD PTR [eax+52]

push ecx

mov edx, DWORD PTR [eax+48]

push edx

mov eax, DWORD PTR _puser$[ebp]

mov ecx, DWORD PTR [eax+40]

push ecx

mov edx, DWORD PTR _puser$[ebp]

push edx

push OFFSET $SG5663

call _printf

add esp, 20     ; 00000014H

This is a comment which states that the ASM instructions preceding it will represent line 23 of the C/C++ code, which is:

printf("\n%s %d %llu", puser->rgName, puser->iAge, puser->uMobile);

In the C/C++ code, we are accessing all the elements of structure using pointer to the structure and using -> operator. So, in ASM, we can observe that in the first MOV instruction, pointer to structure is pushed onto the EAX register and then all the subsequent MOV instructions are referring to the elements using the EAX register, as EAX points to the start of the structure memory. In all the subsequent instructions, we are pushing the structure elements onto the stack using a combination of MOV and PUSH instructions.

All the MOV instructions are accessing variables using EAX+offset (where offset is the number of bytes. When added to EAX, it gives the location of the element value stored on the stack) and moving it to an available register. Next, the PUSH instruction pushes the structure element value onto the stack. Once all the elements are pushed onto the stack, a call to the printf function is made. The stack state before the call to the second printf function is as follows:

[ESP]  0012FEEC  00408158  "\n%s %d %llu", argument to printf() is pushed here

[ESP+0x04] 0012FEF0  0012FF00  pointer to rgName array is pushed here

[ESP+0x08] 0012FEF4  0000001E  iAge value is pushed here

[ESP+0x0C] 0012FEF8  D6CD994D  uMobile value is pushed here

[ESP+0x10] 0012FEFC  00000001  uMobile value is pushed here

[ESP+0x14] 0012FF00  6574694A  4 bytes of array, rgName in little-endian etiJ

[ESP+0x18] 0012FF04  7265646E  4 bytes of array, rgName in little-endian redn

[ESP+0x1C] 0012FF08  00000000  String terminator with NULL in remaining array

[ESP+0x20] 0012FF0C  00000000  NULL values in char array of size 40 bytes

[ESP+0x24] 0012FF10  00000000  NULL values in char array of size 40 bytes

[ESP+0x28] 0012FF14  00000000  NULL values in char array of size 40 bytes

[ESP+0x2C] 0012FF18  00000000  NULL values in char array of size 40 bytes

[ESP+0x30] 0012FF1C  00000000  NULL values in char array of size 40 bytes

[ESP+0x34] 0012FF20  00000000  NULL values in char array of size 40 bytes

[ESP+0x38] 0012FF24  00000000  rgName from 0x0012FF00 to 0x0012FF28, total 40 bytes    

[ESP+0x3C] 0012FF28  0000001E  Second element of Structure, iAge is stored here

[ESP+0x40] 0012FF2C  004054C0  return to 0x004054C0 from 0x00405477

[ESP+0x44] 0012FF30  D6CD994D  4 bytes of uMobile stored here

[ESP+0x48] 0012FF34  00000001  remaining 4 bytes of uMobile stored here

[ESP+0x50] 0012FF38  01D7D682  XOR of Stack Cookie and EAX is stored here

[ESP+0x54] 0012FF3C  0012FF00  Pointer to structure is stored here, puser

[ESP+0x58] 0012FF40  0012FF88  [EBP]

[ESP+0x5C] 0012FF44  004012B3    return to 0x004012B3 from 0x00401000

Figure 12.10: Before call to second printf

After the call to the printf function, the stack is cleaned using the ADD instruction.

▼Line 86-87

; Line 25

xor eax, eax

This is a comment which states that the ASM instructions preceding it will represent line 25 of the C/C++ code, which is:

return 0;

In the ASM code, EAX is zeroed to return 0, as we discussed earlier that the return value of a function is stored in the EAX register.

▼Line 88-97

; Line 26

mov ecx, DWORD PTR __$ArrayPad$[ebp]

xor ecx, ebp

call @__security_check_cookie@4

mov esp, ebp

pop ebp

ret 0

_main ENDP

_TEXT ENDS

END

This ASM code that retrieves the stack cookie value from [EBP-0x08] and XOR with EBP to match the stack cookie value stored at 0x0040B000. On return from the security_check_cookie procedure, the function epilogue is called to return 0 and end the main procedure, TEXT segment, and code. The stack state before the return is as follows:

[ESP-0x44] 0012FF00  6574694A   Now JUNK

[ESP-0x40] 0012FF04  7265646E   Now JUNK

[ESP-0x3C] 0012FF08  00000000   Now JUNK

[ESP-0x38] 0012FF0C  00000000   Now JUNK

[ESP-0x34] 0012FF10  00000000   Now JUNK

[ESP-0x30] 0012FF14  00000000   Now JUNK

[ESP-0x2C] 0012FF18  00000000   Now JUNK

[ESP-0x28] 0012FF1C  00000000   Now JUNK

[ESP-0x24] 0012FF20  00000000   Now JUNK

[ESP-0x20] 0012FF24  00000000   Now JUNK

[ESP-0x1C] 0012FF28  0000001E   Now JUNK

[ESP-0x18] 0012FF2C  004054C0   return to 0x004054C0 from 0x00405477

[ESP-0x14] 0012FF30  D6CD994D   Now JUNK

[ESP-0x10] 0012FF34  00000001   Now JUNK

[ESP-0x0C] 0012FF38  01D7D682   Now JUNK

[ESP-0x08] 0012FF3C  0012FF00   Now JUNK

[ESP-0x04] 0012FF40  0012FF88   Now JUNK

[ESP]  0012FF44  004012B3   return to 0x004012B3 from 0x00401000

Figure 12.11: Stack cleaned

Structure with Optimization

Compile the code with the optimization flag, /Ox flag. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Ox: Enable maximum optimization

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\Structures\Structures

^

cl Structures.cpp /FaStructures-Optimized.asm /Ox /FeStructures-Optimized.exe

The following is the output of running the preceding commands:

Figure 12.12: Structure with Optimization

The compilation generates the EXE file and assembly code. Disable ASLR manually. To disable ASLR, use the CFF explorer and change the DllCharacteristics parameter to uncheck DLL can move.

Now, let’s move onto the generated assembly listing:

Figure 12.13: Structures-Optimized.asm-Part-1

Figure 12.14: Structures-Optimized.asm-Part-2

Let’s analyze the ASM code:

▼Line 29-33

; Line 9

sub esp, 60     ; 0000003cH

mov eax, DWORD PTR ___security_cookie

xor eax, esp

mov DWORD PTR __$ArrayPad$[esp+60], eax

In the ASM code, 60 bytes are subtracted from ESP to create room for the variables on the stack. The MOV instruction moves the stack cookie stored at 0x0040B000 to EAX, where it is XOR’ed with ESP and the result is moved to the stack at [ESP+0x38]. As instruction:

mov DWORD PTR __$ArrayPad$[esp+60], eax

When this instruction is viewed in x32dbg it will be shown as below:

mov dword ptr ss:[esp+0x38], eax

The stack state after this instruction is as follows:

[ESP]  0012FF08  7FFDF000  JUNK

[ESP+0x4] 0012FF0C  0040345B  JUNK

[ESP+0x8] 0012FF10  0012FEFC  JUNK

[ESP+0xC] 0012FF14  00000004  JUNK

[ESP+0x10] 0012FF18  0012FF78  JUNK

[ESP+0x14] 0012FF1C  00402500  JUNK

[ESP+0x18] 0012FF20  858BFA6B  JUNK

[ESP+0x1C] 0012FF24  FFFFFFFE  JUNK

[ESP+0x20] 0012FF28  004054AC  JUNK

[ESP+0x24] 0012FF2C  004054C0  JUNK

[ESP+0x28] 0012FF30  0040345B  JUNK

[ESP+0x2C] 0012FF34  0012FF48  JUNK

[ESP+0x30] 0012FF38  004028DE  JUNK

[ESP+0x34] 0012FF3C  0040345B  JUNK

[ESP+0x38] 0012FF40  85D99E8B  XOR of Stack Cookie and ESP is stored here

[ESP+0x3C] 0012FF44  004012B4  return to structures-optimized.004012B4

Figure 12.15: Stack cookie

▼Line 34-37

; Line 18

mov eax, DWORD PTR $SG5660

mov ecx, DWORD PTR $SG5660+4

mov dl, BYTE PTR $SG5660+8

These instructions move the first 4 bytes of the rgName array into the EAX register in the little-endian format and the next 4 bytes in the ECX register. The null terminated character is moved to the DL register.

▼Line 38-61

; Line 22

push 1

mov DWORD PTR _user$[esp+64], eax

xor eax, eax

push -691168947    ; d6cd994dH

mov DWORD PTR _user$[esp+77], eax

mov DWORD PTR _user$[esp+81], eax

mov DWORD PTR _user$[esp+85], eax

mov DWORD PTR _user$[esp+89], eax

mov DWORD PTR _user$[esp+93], eax

mov DWORD PTR _user$[esp+97], eax

mov DWORD PTR _user$[esp+101], eax

mov WORD PTR _user$[esp+105], ax

mov BYTE PTR _user$[esp+107], al

push 30     ; 0000001eH

lea eax, DWORD PTR _user$[esp+72]

push eax

push OFFSET $SG5662

mov DWORD PTR _user$[esp+84], ecx

mov BYTE PTR _user$[esp+88], dl

mov DWORD PTR _user$[esp+120], 30  ; 0000001eH

mov DWORD PTR _user$[esp+128], -691168947 ; d6cd994dH

mov DWORD PTR _user$[esp+132], 1

call _printf

In the non-optimized section, we saw how EBP is referred to access the structure elements on the stack. In the preceding ASM code, the stack is similarly filled with the elements of structure by referring to the stack memory with ESP+offset. Also, for the first printf function call, the arguments are pushed onto the stack. The stack state before the call to printf will be:

[ESP]  0012FEF4  0040814C  "\n%s %d %llu", argument to printf() is pushed here

[ESP+0x4] 0012FEF8  0012FF08  pointer to rgName array is pushed here

[ESP+0x8] 0012FEFC  0000001E  iAge value is pushed here

[ESP+0xC] 0012FF00  D6CD994D  uMobile value is pushed here

[ESP+0x10] 0012FF04  00000001  uMobile value is pushed here

[ESP+0x14] 0012FF08  6574694A  4 bytes of array, rgName in little-endian etiJ

[ESP+0x18] 0012FF0C  7265646E  4 bytes of array, rgName in little-endian redn

[ESP+0x1C] 0012FF10  00000000  String terminator with NULL values in array

[ESP+0x20] 0012FF14  00000000  NULL values in char array of size 40 bytes

[ESP+0x24] 0012FF18  00000000  NULL values in char array of size 40 bytes

[ESP+0x28] 0012FF1C  00000000  NULL values in char array of size 40 bytes

[ESP+0x2C] 0012FF20  00000000  NULL values in char array of size 40 bytes

[ESP+0x30] 0012FF24  00000000  NULL values in char array of size 40 bytes

[ESP+0x34] 0012FF28  00000000  NULL values in char array of size 40 bytes

[ESP+0x38] 0012FF2C  00000000  rgName from 0x0012FF08 to 0x0012FF30, total 40 bytes

[ESP+0x3C] 0012FF30  0000001E  Second element of Structure, iAge is stored here

[ESP+0x40] 0012FF34  0012FF48  Pointer to structure is stored here, puser

[ESP+0x44] 0012FF38  D6CD994D  4 bytes of uMobile stored here

[ESP+0x48] 0012FF3C  00000001  remaining 4 bytes of uMobile stored here

[ESP+0x4C] 0012FF40  85D99E8B  XOR of Stack Cookie and EAX is stored here

[ESP+0x50] 0012FF44  004012B4  return to structures-optimized.004012B4

Figure 12.16: The stack state before the call to printf

▼Line 62-72

; Line 23

mov ecx, DWORD PTR _user$[esp+132]

mov edx, DWORD PTR _user$[esp+128]

mov eax, DWORD PTR _user$[esp+120]

push ecx

push edx

push eax

lea ecx, DWORD PTR _user$[esp+92]

push ecx

push OFFSET $SG5663

call _printf

The first three MOV and PUSH instructions push the uMobile and iAge values on the stack. Load Effective Address (LEA) moves the pointer to structure on the ECX register and then to the stack. Now, we have all three arguments to printf on the stack. We can call the printf function. The stack state before the call to the second printf will be:

[ESP]  0012FEE0  00408158  "\n%s %d %llu", arg to 2nd printf() pushed here

[ESP+0x4] 0012FEE4  0012FF08  pointer to rgName array is pushed here

[ESP+0x8] 0012FEE8  0000001E  iAge value is pushed here

[ESP+0x0C] 0012FEEC  D6CD994D  uMobile value is pushed here

[ESP+0x10] 0012FEF0  00000001  uMobile value is pushed here

[ESP+0x14] 0012FEF4  0040814C  "\n%s %d %llu", arg to 1st printf() pushed here

[ESP+0x18] 0012FEF8  0012FF08  pointer to rgName array is pushed here

[ESP+0x1C] 0012FEFC  0000001E  iAge value is pushed here

[ESP+0x20] 0012FF00  D6CD994D  uMobile value is pushed here

[ESP+0x24] 0012FF04  00000001  uMobile value is pushed here

[ESP+0x28] 0012FF08  6574694A  4 bytes of array, rgName in little-endian etiJ

[ESP+0x2C] 0012FF0C  7265646E  4 bytes of array, rgName in little-endian redn

[ESP+0x30] 0012FF10  00000000  String terminator with NULL in remaining array

[ESP+0x34] 0012FF14  00000000  NULL values in char array of size 40 bytes

[ESP+0x38] 0012FF18  00000000  NULL values in char array of size 40 bytes

[ESP+0x3C] 0012FF1C  00000000  NULL values in char array of size 40 bytes

[ESP+0x40] 0012FF20  00000000  NULL values in char array of size 40 bytes

[ESP+0x44] 0012FF24  00000000  NULL values in char array of size 40 bytes

[ESP+0x48] 0012FF28  00000000  NULL values in char array of size 40 bytes

[ESP+0x4C] 0012FF2C  00000000  rgName from 0x0012FF08 to 0x0012FF30, total 40 bytes

[ESP+0x50] 0012FF30  0000001E  Second element of Structure, iAge is stored here

[ESP+0x54] 0012FF34  0012FF48  Pointer to structure is stored here, puser

[ESP+0x58] 0012FF38  D6CD994D  4 bytes of uMobile stored here

[ESP+0x6C] 0012FF3C  00000001  remaining 4 bytes of uMobile stored here

[ESP+0x64] 0012FF40  85D99E8B  XOR of Stack Cookie and EAX is stored here

[ESP+0x68] 0012FF44  004012B4  return to structures-optimized.004012B4

Figure 12.17: Stack state before call to second printf

▼Line 73-83

; Line 26

mov ecx, DWORD PTR __$ArrayPad$[esp+100]

add esp, 40     ; 00000028H

xor ecx, esp

xor eax, eax

call @__security_check_cookie@4

add esp, 60     ; 0000003cH

ret 0

_main ENDP

_TEXT ENDS

END

The rest of the ASM code is the same, where the stack is cleaned using the ADD instruction to match the stack cookie for the buffer overflow. Finally, after a call to the security_check_cookie procedure, the stack is again cleaned to return 0 to end the main function, TEXT segment, and code. The stack state x32dbg will be as follows:

[ESP-0x40] 0012FF08  6574694A   Now JUNK

[ESP-0x38] 0012FF0C  7265646E   Now JUNK

[ESP-0x34] 0012FF10  00000000   Now JUNK

[ESP-0x30] 0012FF14  00000000   Now JUNK

[ESP-0x2C] 0012FF18  00000000   Now JUNK

[ESP-0x28] 0012FF1C  00000000   Now JUNK

[ESP-0x24] 0012FF20  00000000   Now JUNK

[ESP-0x20] 0012FF24  00000000   Now JUNK

[ESP-0x1C] 0012FF28  00000000   Now JUNK

[ESP-0x18] 0012FF2C  00000000   Now JUNK

[ESP-0x14] 0012FF30  0000001E   Now JUNK

[ESP-0x10] 0012FF34  0012FF48   Now JUNK

[ESP-0x0C] 0012FF38  D6CD994D   Now JUNK

[ESP-0x08] 0012FF3C  00000001   Now JUNK

[ESP-0x04] 0012FF40  85D99E8B   Now JUNK

[ESP]  0012FF44  004012B4   return to structures-optimized.004012B4

Figure 12.18: The stack is cleaned

Conclusion

In this chapter, we learned pointers to structures with respect to reverse engineering. We discussed about structures code pattern in disassembled code and how structures are stored in memory. The main point to understand is that, pointer to the structure points to the first element of structure as structures are always stored in contiguous memory locations. We also covered structures program with optimized and non-optimized code.

CHAPTER 13

Scanf Program Pattern in Reverse Engineering

Imagine you downloaded some hacking software from the internet. On running this software, it asks you to enter the password. The password is not mentioned anywhere on the site from where you downloaded this software. Using reverse engineering, we can break the password. Breaking a password is unethical and against laws until and unless we are permitted to do so. Now you must be thinking how one can break a password. Breaking a password is dependent on the quality of the software code. If the code is written with security in mind, chances are that a stronger mechanism must have been used.

In this chapter, we will discuss the reverse engineering part of the mechanism used by a software developer to capture user input. We will be talking about scanf function, which is used to capture the user input from the console.

Structure

In this chapter, we will cover the following topics:

Function scanf with Integers

Function scanf without Optimization

Function scanf with Optimization

Objective

In this chapter, we will understand the scanf function with respect to reverse engineering. We will talk about the scanf code pattern in the disassembled code and understand how scanf inputs are stored in memory. We will also cover the scanf program with both optimized and non-optimized code.

Function scanf with Integers

In this simple C/C++ code, we ask the user to input a number of type integer and print it on the console. We use scanf to capture user input and printf to print the number entered by the user in our C/C++ code:

Figure 13.1: scanfWithIntegers.cpp

Function scanf without Optimization

Compile the code without optimization in the MSVC compiler on the same x86 Windows machine. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\scanfWithIntegers\scanfWithIntegers

^

cl scanfWithIntegers.cpp /FascanfWithIntegers.asm /FescanfWithIntegers.exe

The following is the output of running the preceding commands:

Figure 13.2: Function scanf without Optimization

This will generate the assembly code and the EXE file. This time, before analyzing, we will disable the Address Space Layout Randomization (ASLR). It’s a security mechanism by which the base address of the PE file is randomized on every load of the Portable Executable (PE) file generated with our MSVC compiler. This will help us reload the PE file without randomizing its base address. To disable ASLR, refer to the Appendix.

We will now use the PE file with the disabled ASLR for further analysis using x32dbg. Let’s move to the generated assembly listing:

Figure 13.3: scanfWithIntegers.asm-Part-1

Figure 13.4: scanfWithIntegers.asm-Part-2

Let’s walk through the code:

▼Line 27-30

; Line 7

push ebp

mov ebp, esp

push ecx

The code starts with a simple main function prologue.

▼Line 31-34

; Line 9

push OFFSET $SG4678

call _printf

add esp, 4

The C/C++ code on line 9 prints the string constant on the console:

printf ("Enter a Number: ");

In our assembly code, before calling the printf function, the string constant $SG4678 stored in the .rdata segment is pushed onto the stack. Once the arguments to the printf functions are pushed onto the stack, the call to the printf function is made. On return, 4 bytes are added to ESP for stack cleaning. For analyzing the assembly instruction, we will load the PE file in x32dbg, and then put two breakpoints. The first breakpoint is at the start of the .TEXT segment and the other breakpoint is just after the scanf function return. Now, after loading the PE file in x32dbg, run the code. The execution will stop at the first breakpoint. From the first breakpoint step into the code using x32dbg, until you hit the preceding add esp,4 instruction. The ADD instruction cleans the stack and the following is how the stack looks after executing the ADD instruction:

[ESP-0x4] 0012FF38  0040A140  "Enter a Number: ", arg to 1st printf

[ESP]  0012FF3C  00000001  ECX pushed, later to store Integer input passed

[ESP+0x4] 0012FF40  0012FF88  [EBP]

[ESP+0x8] 0012FF44  004012C4  return to 0x004012C4 from 0x00401000

Figure 13.5: Stack after ADD insruction

▼Line 35-40

; Line 10

lea eax, DWORD PTR _iInput$[ebp]

push eax

push OFFSET $SG4679

call _scanf

add esp, 8

The C/C++ code on line 10 calls the scanf function, which accepts the following two arguments:

scanf ("%d", &iInput);

In the assembly code, the LEA instruction is evaluated to:

lea eax, ss:[ebp-0x4]

LEA will store the input placeholder memory location in EAX. By 'input placeholder memory location’, we mean the memory location which will be used to store the integer input by the user. Once this input placeholder memory location is stored in EAX, then it is pushed onto the stack as a parameter to the scanf function. Then another push instruction pushes the string constant $SG4679 on the stack.

The call to scanf is made after pushing both the arguments. On return, 8 bytes are added to ESP for stack cleaning. The breakpoint is added at the 0x0040101F memory location. The stack state at breakpoint is as follows:

[ESP]  0012FF34  0040A154  "%d", parameter to scanf ()

[ESP+0x4] 0012FF38  0012FF3C  Input placeholder memory location

[ESP+0x8] 0012FF3C  00000007  Input placeholder, Number 7 is entered by user

[ESP+0xC] 0012FF40  0012FF88  [EBP]

[ESP+0x10] 0012FF44  004012C4  return to 0x004012C4 from 0x00401000

Figure 13.6: User entered number on stack

▼Line 41-46

; Line 11

mov ecx, DWORD PTR _iInput$[ebp]

push ecx

push OFFSET $SG4680

call _printf

add esp, 8

The C/C++ code on line 11 calls the printf function. This will print the number entered by the user:

printf ("Number you entered is %d\n", iInput);

In the assembly code, the MOV instruction is evaluated to:

mov ecx, dword ptr ss:[ebp-0x4]

The MOV instruction will move the user input stored at ss:[ebp-0x4] to ECX. Now, both the arguments the first is the string constant and the second is the number that the user entered to the printf function are pushed on the stack:

[ESP-0x8] 0012FF34  0040A158  "Number you entered is %d\n", arg to 2nd printf()

[ESP-0x4] 0012FF38  00000007  Number 7 is entered by user, arg to 2nd printf()

[ESP]  0012FF3C  00000007  Input placeholder, Number 7 is entered by user  

[ESP+0x4] 0012FF40  0012FF88  [EBP]

[ESP+0x8] 0012FF44  004012C4  return to 0x004012C4 from 0x00401000

Figure 13.7: After printf function

▼Line 47-55

; Line 12

xor eax, eax

; Line 13

mov esp, ebp

pop ebp

ret 0

_main ENDP

_TEXT ENDS

END

XOR and function epilogue will clean up EAX and stack, respectively. EAX is XOR’ed to return 0. ENDP will end the main procedure and ENDS will end the text segment. The stack will be as follows:

[ESP+0x10] 0012FF34  0040A158  JUNK

[ESP+0xC] 0012FF38  00000007  JUNK

[ESP+0x8] 0012FF3C  00000007  JUNK

[ESP+0x4] 0012FF40  0012FF88  [EBP] popped up

[ESP]  0012FF44  004012C4  return to 0x004012C4 from 0x00401000

Figure 13.8: Stack cleaned

Function scanf with Optimization

Compile the code with the optimization flag, /Ox flag. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Ox: Enable maximum optimization

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\scanfWithIntegers\scanfWithIntegers

^

cl scanfWithIntegers.cpp /FascanfWithIntegers-Optimized.asm /Ox /FescanfWithIntegers-Optimized.exe

The following is the output of running the preceding commands:

Figure 13.9: Function scanf with optimization

Let’s move to the generated assembly listing:

Figure 13.10: scanfWithIntegers-Optimized.asm-Part-1

Figure 13.11: scanfWithIntegers-Optimized.asm-Part-2

The main difference between optimized and non-optimized code is that in optimized code, the function prologue and epilogue are eliminated. Secondly, the stack cleaning is not done after each and every function call but towards the end of the main function. We will walk through the assembly instruction in the same way we did in the non-optimization section by putting breakpoints in x32dbg:

▼Line 27-31

; Line 7

push ecx

; Line 9

push OFFSET $SG4678

call _printf

The C/C++ code on line 9 prints the string constant on the console:

printf ("Enter a Number: ");

The string constant $SG4678 stored in the .rdata segment is pushed onto the stack before calling the printf function. Note that the function prologue as well as the stack cleaning after the printf function call are eliminated. We will also see that the stack is cleaned towards the end. The following is the stack state after executing the preceding instruction:

[ESP]  0012FF3C  0040A140  "Enter a Number: ", parameter to 1st printf()

[ESP+0x4] 0012FF40  00000001  ECX is pushed, used later to store Integer input

[ESP+0x8] 0012FF44  004012BA  return to 0x004012BA from 0x00401000

Figure 13.12: After printf

▼Line 32-36

; Line 10

lea eax, DWORD PTR _iInput$[esp+8]

push eax

push OFFSET $SG4679

call _scanf

The C/C++ code on line 10 calls the scanf function, which accepts the following two arguments:

scanf ("%d", &iInput);

In the assembly code, the LEA instruction is evaluated to:

lea eax, ss:[esp+0x4]

LEA will load the effective address of the input placeholder memory location in EAX. As already stated in non-optimized section, input placeholder memory location means, the memory location which will be used to store the Integer input by user. The Scanf function takes two arguments, one is the memory location where the user’s input will be stored and the other is the string constant $SG4679.

Once both the parameters are pushed onto the stack, the call to scanf is made. During the call, the user is asked to enter the number. The integer number supplied by the user will be stored at the input placeholder memory location. We can also notice that the stack cleaning is not done after the function call. The breakpoint is added at the 0x0040101A memory location. The stack state at breakpoint is as follows:

[ESP]  0012FF34  0040A154  "%d", parameter to scanf ()

[ESP+0x4] 0012FF38  0012FF40  Input placeholder memory location

[ESP+0x8] 0012FF3C  0040A140  "Enter a Number: ", parameter to 1st printf()

[ESP+0xC] 0012FF40  00000007  Input placeholder, Number 7 is entered by user

[ESP+0x10] 0012FF44  004012BA  return to 0x004012BA from 0x00401000

Figure 13.13: After scanf

▼Line 37-41

; Line 11

mov ecx, DWORD PTR _iInput$[esp+16]

push ecx

push OFFSET $SG4680

call _printf

The C/C++ code on line 11 calls the printf function. This will print the number that the user entered:

printf ("Number you entered is %d\n", iInput);

In the assembly code, the MOV instruction is evaluated to:

mov ecx, dword ptr ss:[esp+0xC]

The MOV instruction will move the user integer input stored at ss:[esp+0xC] to ECX. Now, both the arguments (first is the string constant and the second is the integer number that the user entered) of the printf function are pushed on the stack before the printf call. The stack state after the CALL instruction will be:

[ESP]  0012FF2C  0040A158  "Number you entered is %d\n", arg to 2nd printf()

[ESP+0x4] 0012FF30  00000007  Number 7 is entered by user, arg to 2nd printf()

[ESP+0x8] 0012FF34  0040A154  "%d", parameter to scanf ()

[ESP+0xC] 0012FF38  0012FF40  Input placeholder memory location

[ESP+0x10] 0012FF3C  0040A140  "Enter a Number: ", parameter to 1st printf()

[ESP+0x14] 0012FF40  00000007  Input placeholder, Number 7 is entered by user

[ESP+0x18] 0012FF44  004012BA  return to 0x004012BA from 0x00401000

Figure 13.14: After second printf

▼Line 42-49

; Line 12

xor eax, eax

; Line 13

add esp, 24     ; 00000018H

ret 0

_main ENDP

_TEXT ENDS

END

XOR will clean up EAX to return 0. You can notice that the stack is cleaned towards the end of the main function with the ADD instruction. The ADD instruction cleans the stack by adding 24 bytes to ESP. The END derivative ends the code. The stack state will be as follows:

[ESP-0x18] 0012FF2C  0040A158  JUNK

[ESP-0x14] 0012FF30  00000007  JUNK

[ESP-0x10] 0012FF34  0040A154  JUNK

[ESP-0xC] 0012FF38  0012FF40  JUNK

[ESP-0x8] 0012FF3C  0040A140  JUNK

[ESP-0x4] 0012FF40  00000007  JUNK

[ESP]  0012FF44  004012BA  return to 0x004012BA from 0x00401000

Figure 13.15: Stack cleaned

Conclusion

In this chapter, we understood how the input captured from a user is stored in memory using the scanf function. We saw the scanf code pattern in a disassembled code for both optimized and non-optimized code. In the next chapter, we will study the strcpy function and how its pattern looks while reverse engineering.

CHAPTER 14

Strcpy Program Pattern in Reverse Engineering

We have understood the patterns of arrays and pointers in the earlier chapters. Now we will talk about some real-world examples of code that use all these as a part of a single program. There are times when you want to copy data from one place to another place. To do so, we use some standard predefined functions. But in this chapter, we will talk about the implementation of strcpy, which is used to copy data from the source to the destination.

The implementation of strcpy will help you learn the combination of pointers and arrays in a single program and understand the code pattern in assembly. These patterns can be seen in many applications or software and you will find it interesting to know that this will allow you to find vulnerabilities in them.

Structure

In this chapter, we will cover the following topics:

Understand strcpy function

Strcpy without Optimization

Strcpy with Optimization

Objective

In this chapter, we will talk about the strcpy function implementation with respect to reverse engineering. This function is very popular in software implementation to perform the string copy operation from the source location to the destination. We will also talk about byte-by-byte operations that happen during the strcpy execution. We will also cover strcpy program assembly pattern with optimized and non-optimized code.

Strcpy

In this example, we will take up the strcpy function with the name of xstrcpy to copy a string.

Figure 14.1: strcpy.cpp

Strcpy without Optimization

Compile the code without optimization in the MSVC compiler on the same x86 Windows machine. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\strcpy\strcpy

^

cl strcpy.cpp /Fastrcpy.asm /Festrcpy.exe

The following is the output of running the preceding commands:

Figure 14.2: Strcpy without Optimization

The compilation generates the EXE file and assembly code. Disable ASLR manually. To disable ASLR, use the CFF explorer and change the DllCharacteristics parameter to uncheck DLL can move.

Now, let’s move on to the generated assembly listing:

Figure 14.3: strcpy.asm-Part-1

Figure 14.4: strcpy.asm-Part-2

Figure 14.5: strcpy.asm-Part-3

Figure 14.6: strcpy.asm-Part-4

▼Line 12-15

CONST SEGMENT

$SG4688 DB 'ReverseEngg', 00H

$SG4690 DB '%s', 0aH, 00H

CONST ENDS

Constant Segment defines two string constants, $SG4688 and $SG4690. The linker renames CONST SEGMENT to .rdata (the code is placed in the .code segment, the constant strings are placed in CONST (.rdata) segment and if not, the constant is placed in the .data segment), which can be viewed in the memory dump using x32dbg, shown as follows:

Figure 14.7: .rdata

$SG4688 and $SG4690 are internal names given by the compiler to handle the string constant. DB defines the byte, which is the data type.

'ReverseEngg', 00H is the string data, which is null terminated ASCII string.

'%s', 0aH, 00H is also a string data.

By CONST END, the constant segment is ended.

First, we will take the main code starting with:

▼Line 72-73

PUBLIC __$ArrayPad$

PUBLIC _main

PUBLIC is the derivative which makes the _main procedure and the __$ArrayPad$ macro public, which can be accessed by other modules:

▼Line 74-76

EXTRN _printf:PROC

EXTRN ___security_cookie:DWORD

EXTRN @__security_check_cookie@4:PROC

The EXTRN derivative declares the extern function, which is printf in our case. All functions begin with an underscore.

▼Line 78-81

_TEXT SEGMENT

_src$ = -44      ; size = 12

_dest$ = -32      ; size = 25

__$ArrayPad$ = -4     ; size = 4

This is the start of the _TEXT segment, where our main function code resides. After _TEXT, we have the different variable macro defined. The local variable on the stack frame can be accessed by adding _<variable-name>$ to the EBP address.

▼Line 82

_main PROC

This is the start of the main procedure.

▼Line 83-89

; Line 35

push ebp

mov ebp, esp

sub esp, 44     ; 0000002cH

mov eax, DWORD PTR ___security_cookie

xor eax, ebp

mov DWORD PTR __$ArrayPad$[ebp], eax

Line 35 of the C/C++ code starts the main function:

int main() {

The ASM code starts with the main function prologue of PUSH and MOV instruction. The SUB instruction creates room for variables on the stack by subtracting 44 (0x2C) bytes from ESP. 44 bytes can be visualized as 4 bytes for the security_cookie, 12 bytes for the src array, and 28 bytes for the dest array. The src and dest array sizes have been round off to the multiple of 4 bytes.

▼Line 87-89

mov eax, DWORD PTR ___security_cookie

xor eax, ebp

mov DWORD PTR __$ArrayPad$[ebp], eax

We have already discussed the concept of a stack cookie. In the preceding instruction, the stack cookie is moved from EAX to XOR with EBP and the resultant XOR value is placed on the stack. This value is stored on the stack right above the old EBP. This value will be validated before the main function epilogue to check the buffer overflow condition. Now, let’s move on to the next instructions:

▼Line 90-96

; Line 36

mov eax, DWORD PTR $SG4688

mov DWORD PTR _src$[ebp], eax

mov ecx, DWORD PTR $SG4688+4

mov DWORD PTR _src$[ebp+4], ecx

mov edx, DWORD PTR $SG4688+8

mov DWORD PTR _src$[ebp+8], edx

Line 36 of the C/C++ code is:

char src[] = "ReverseEngg";

In the ASM code, we can see that the string $SG4688 is moved to the stack in three steps and with six MOV instructions. The first two MOV instructions move the first 4 bytes of string (“Reve”) to EAX and then from EAX, it is pushed to the stack at the [EBP - 0x2C] location. Similarly, the next 4 bytes of string (“rseE”) are moved to ECX and then from ECX, it is pushed to the stack at the [EBP - 0x28] location. Then in the remaining bytes, padded with NULL (“ngg” + “0x00”) are moved to EDX and then from EDX to the stack at the [EBP - 0x24] location. The stack state after the execution of the preceding instruction in x32dbg will be as follows:

[ESP]  0012FF14  65766552   [EBP-0x2C]  “eveR” is pushed here, in little endian

[ESP+0x04] 0012FF18  45657372   [EBP-0x28]  “Eesr” is pushed here, little endian

[ESP+0x08] 0012FF1C  0067676E   [EBP-0x24]  “ ggn” is pushed here, little endian

[ESP+0x0C] 0012FF20  AC9802CF   [EBP-0x20]  JUNK HERE

[ESP+0x10] 0012FF24  FFFFFFFE   [EBP-0x1C]  JUNK HERE

[ESP+0x14] 0012FF28  0040549C   [EBP-0x18]  JUNK HERE

[ESP+0x18] 0012FF2C  004054B0   [EBP-0x14]  JUNK HERE

[ESP+0x1C] 0012FF30  0040344B   [EBP-0x10]  JUNK HERE

[ESP+0x20] 0012FF34  0012FF48   [EBP-0x0C]  JUNK HERE

[ESP+0x24] 0012FF38  004028CE   [EBP-0x08]  JUNK HERE

[ESP+0x28] 0012FF3C  ACCA6657   [EBP-0x04]  XOR of stack cookie and EBP stored here

[ESP+0x2C] 0012FF40  0012FF88   [EBP]       EBP of earlier stack frame stored here

[ESP+0x30] 0012FF44  004012A8   [EBP+0x04]  return to 0x004012A8 from 0x00401050

Figure 14.8: $SG4688 is moved to the stack

▼Line 97-102

; Line 39

lea eax, DWORD PTR _src$[ebp]

push eax

lea ecx, DWORD PTR _dest$[ebp]

push ecx

call ?xstrcpy@@YAPADPADPBD@Z   ; xstrcpy

Line 39 of the C/C++ code is:

printf("%s\n", xstrcpy(dest, src));

In the ASM code, before making the printf function call, we need to push the return value of the xstrcpy function and the string $SG4690 onto the stack. To evaluate the return value of the xstrcpy function, arguments to the function are pushed onto the stack. So, the first LEA (Load Effective Address) loads the address on the stack (which is [EBP - 0x2C], 0x0012FF14) pointing to the source array in the EAX register. From EAX, it is further pushed onto the stack. Similarly, the second LEA instruction loads the address on the stack (which is [EBP - 0x20], 0x0012FF20) pointing to the destination array in the ECX register. From ECX, it is further pushed onto the stack. As the destination array is not initialized in the C/C++ code, it is pointing to the uninitialized memory location (which is [EBP - 0x20], 0x0012FF20) the on stack. Once both the arguments to the xstrcpy function are pushed onto the stack, a call to the xstrcpy function is made. The stack state before the call to the xstrcpy function is as follows:

[ESP]  0012FF0C  0012FF20   [EBP-0x34]  2nd arg to xstrcpy, ptr to dest array

[ESP+0x04] 0012FF10  0012FF14   [EBP-0x30]  1st arg to xstrcpy(),ptr to src array

[ESP+0x08] 0012FF14  65766552   [EBP-0x2C] “eveR” is pushed here, in little endian

[ESP+0x0C] 0012FF18  45657372   [EBP-0x28]  “Eesr” is pushed here, little endian

[ESP+0x10] 0012FF1C  0067676E   [EBP-0x24]  “ ggn” is pushed here, little endian

[ESP+0x14] 0012FF20  AC9802CF   [EBP-0x20]  dest array start, uninitialized JUNK

[ESP+0x18] 0012FF24  FFFFFFFE   [EBP-0x1C]  JUNK HERE

[ESP+0x1C] 0012FF28  0040549C   [EBP-0x18]  JUNK HERE

[ESP+0x20] 0012FF2C  004054B0   [EBP-0x14]  JUNK HERE

[ESP+0x24] 0012FF30  0040344B   [EBP-0x10]  JUNK HERE

[ESP+0x28] 0012FF34  0012FF48   [EBP-0x0C]  JUNK HERE

[ESP+0x2C] 0012FF38  004028CE   [EBP-0x08]  JUNK HERE

[ESP+0x30] 0012FF3C  ACCA6657   [EBP-0x04]  XOR of stack cookie and EBP stored here

[ESP+0x34] 0012FF40  0012FF88   [EBP]       EBP of earlier stack frame stored here

[ESP+0x38] 0012FF44  004012A8   [EBP+0x04]   return to 0x004012A8 from 0x00401050

Figure 14.9: Before call to xstrcpy

The following is the start of the xstrcpy function:

▼Line 24-27

; Line 9

push ebp

mov ebp, esp

push ecx

The xstrcpy function prologue is called. The ECX register has the pointer to the dest array, so it is saved onto the stack by pushing the ECX register. The stack state after pushing the ECX register is as follows. From this point onwards, the two stack frames have been differentiated with EBP marked with superscript, that is, EBP for main and xstrcpy.

[ESP]  0012FF00  0012FF20    [EBPmain-0x40][EBPxstrcpy-0x04]  Pointer to dest

[ESP+0x04] 0012FF04  0012FF40    [EBPmain-0x3C][EBPxstrcpy] EBP of main() stack frame

[ESP+0x08] 0012FF08  00401087    [EBPmain-0x38] [EBPxstrcpy+0x04] return to main()

[ESP+0x0C] 0012FF0C  0012FF20    [EBPmain-0x34] [EBPxstrcpy+0x08]  2nd arg to xstrcpy

[ESP+0x10] 0012FF10  0012FF14    [EBPmain-0x30] [EBPxstrcpy+0x0C]  1st arg to xstrcpy

[ESP+0x14] 0012FF14  65766552    [EBPmain-0x2C] “eveR” is pushed here

[ESP+0x18] 0012FF18  45657372    [EBPmain-0x28] “Eesr” is pushed here

[ESP+0x1C] 0012FF1C  0067676E    [EBPmain-0x24] “ ggn” is pushed here

[ESP+0x20] 0012FF20  AC9802CF    [EBPmain-0x20] uninitialized dest array

[ESP+0x24] 0012FF24  FFFFFFFE    [EBPmain-0x1C] JUNK HERE

[ESP+0x28] 0012FF28  0040549C    [EBPmain-0x18] JUNK HERE

[ESP+0x2C] 0012FF2C  004054B0    [EBPmain-0x14] JUNK HERE

[ESP+0x30] 0012FF30  0040344B    [EBPmain-0x10] JUNK HERE

[ESP+0x34] 0012FF34  0012FF48    [EBPmain-0x0C] JUNK HERE

[ESP+0x38] 0012FF38  004028CE    [EBPmain-0x08] JUNK HERE

[ESP+0x3C] 0012FF3C  ACCA6657    [EBPmain-0x04] XOR of stack cookie and EBP

[ESP+0x40] 0012FF40  0012FF88    [EBPmain]      EBP of earlier stack frame

[ESP+0x44] 0012FF44  004012A8    [EBPmain+0x04] return to 0x004012A8 from 0x00401050

Figure 14.10: Stack state after pushing ECX

▼Line 28-30

; Line 11

cmp DWORD PTR _dest$[ebp], 0

jne SHORT $LN3@xstrcpy

Line 11 of the C/C++ code is:

if (dest == NULL)

The ASM code compares the value at [EBPxstrcpy+0x08] with NULL. A jump will take place when the value at [EBPxstrcpy+0x08] is not equal to NULL. As memory is already allocated to the dest array at [EBPxstrcpy+0x08] which is not equal to NULL, so the jump will take place to the $LN3@xstrcpy label. The stack state after the jump instruction will be the same as earlier.

▼Line 34-37

$LN3@xstrcpy:

; Line 15

mov eax, DWORD PTR _dest$[ebp]

mov DWORD PTR _ptr$[ebp], eax

Line 11 of the C/C++ code is:

char *ptr = dest;

In the ASM code it is just taking the pointer to the dest array [EBPxstrcpy+0x08] into EAX and then pushing it on to the stack at [EBPxstrcpy-0x04]. If you remember, we saved ECX at the start of the xstrcpy function on-to the stack at the same location [EBPxstrcpy-0x04]. ECX was having the same pointer to the dest array. So basically in these two instructions we are overwriting [EBPxstrcpy-0x04] with the same value. So the stack state will be the same as earlier:

Figure 14.11: Stack state after line 37

▼Line 38-43

$LN2@xstrcpy:

; Line 19

mov ecx, DWORD PTR _src$[ebp]

movsx edx, BYTE PTR [ecx]

test edx, edx

je SHORT $LN1@xstrcpy

Line 19 of the C/C++ code is:

while (*src != '\0')

In the ASM code, the MOV instruction is moving the pointer to the src array [EBPxstrcpy+0x0C] in the ECX register. The next MOV instruction moves the first byte of the src array into EDX, making EDX=0x00000052 (ascii “R”).

The TEST instruction will perform an AND operation of EDX with itself, resulting in a non-zero value in EDX and ZF=0. So, a jump to the label $LN1@xstrcpy will not take place. The instruction pointer will move to the next instruction. The stack state will be the same as earlier:

Figure 14.12: Stack state after je

▼Line 44-48

; Line 21

mov eax, DWORD PTR _dest$[ebp]

mov ecx, DWORD PTR _src$[ebp]

mov dl, BYTE PTR [ecx]

mov BYTE PTR [eax], dl

Line 21 of the C/C++ code is:

*dest = *src;

As this ASM code is non-optimized, we will come across many instructions which are duplicating operations. We can see that the MOV instruction is again moving the pointer from the dest array [EBPxstrcpy+0x08] to EAX register and pointer to src array [EBPxstrcpy+0x0C] is moved to the ECX register. Now, in the remaining two MOV instructions, we are copying the first byte from the src array to the dest array using the DL register. This is done by copying a byte from the memory pointed by the ECX register (which is pointing to the src array) to the DL register and then from the DL register to the memory pointed by the EAX register (which is pointing to the dest array). The stack state after these instructions will be:

[ESP]  0012FF00  0012FF20  [EBPmain-0x40][EBPxstrcpy-0x04]  Pointer to dest

[ESP+0x04] 0012FF04  0012FF40  [EBPmain-0x3C][EBPxstrcpy] EBP of main() stack frame

[ESP+0x08] 0012FF08  00401087  [EBPmain-0x38] [EBPxstrcpy+0x04] return to main()

[ESP+0x0C] 0012FF0C  0012FF20  [EBPmain-0x34] [EBPxstrcpy+0x08]  2nd arg to xstrcpy

[ESP+0x10] 0012FF10  0012FF14  [EBPmain-0x30] [EBPxstrcpy+0x0C]  1st arg to xstrcpy

[ESP+0x14] 0012FF14  65766552  [EBPmain-0x2C] “eveR” is pushed here

[ESP+0x18] 0012FF18  45657372  [EBPmain-0x28] “Eesr” is pushed here

[ESP+0x1C] 0012FF1C  0067676E  [EBPmain-0x24] “ ggn” is pushed here

[ESP+0x20] 0012FF20  AC980252  [EBPmain-0x20] only “R” is copied here at dest array

[ESP+0x24] 0012FF24  FFFFFFFE  [EBPmain-0x1C] JUNK HERE

[ESP+0x28] 0012FF28  0040549C  [EBPmain-0x18] JUNK HERE

[ESP+0x2C] 0012FF2C  004054B0  [EBPmain-0x14] JUNK HERE

[ESP+0x30] 0012FF30  0040344B  [EBPmain-0x10] JUNK HERE

[ESP+0x34] 0012FF34  0012FF48  [EBPmain-0x0C] JUNK HERE

[ESP+0x38] 0012FF38  004028CE  [EBPmain-0x08] JUNK HERE

[ESP+0x3C] 0012FF3C  ACCA6657  [EBPmain-0x04] XOR of stack cookie and EBP

[ESP+0x40] 0012FF40  0012FF88  [EBPmain]      EBP of earlier stack frame stored here

[ESP+0x44] 0012FF44  004012A8  [EBPmain+0x04] return to 0x004012A8 from 0x00401050

Figure 14.13: Stack state after line 48

▼Line 49-52

; Line 22

mov eax, DWORD PTR _dest$[ebp]

add eax, 1

mov DWORD PTR _dest$[ebp], eax

Line 22 of the C/C++ code is:

dest++;

In this ASM section, the pointer to the dest array [EBPxstrcpy+0x08] is again moved to the EAX register so that it can be incremented to 1 by the ADD instruction. This incremented pointer value to the dest array is moved back on the stack at the [EBPxstrcpy+0x08] location. The stack state after these instructions will be as follows:

[ESP]  0012FF00  0012FF20  [EBPmain-0x40][EBPxstrcpy-0x04]  Pointer to dest  

[ESP+0x04] 0012FF04  0012FF40  [EBPmain-0x3C][EBPxstrcpy] EBP of main() stack frame

[ESP+0x08] 0012FF08  00401087  [EBPmain-0x38] [EBPxstrcpy+0x04] return to main()

[ESP+0x0C] 0012FF0C  0012FF21  [EBPmain-0x34] [EBPxstrcpy+0x08] dest array incremented

[ESP+0x10] 0012FF10  0012FF14  [EBPmain-0x30] [EBPxstrcpy+0x0C] First arg to xstrcpy

[ESP+0x14] 0012FF14  65766552  [EBPmain-0x2C] “eveR” is pushed here

[ESP+0x18] 0012FF18  45657372  [EBPmain-0x28] “Eesr” is pushed here

[ESP+0x1C] 0012FF1C  0067676E  [EBPmain-0x24] “ ggn” is pushed here

[ESP+0x20] 0012FF20  AC980252  [EBPmain-0x20] only “R” is copied here at dest array

[ESP+0x24] 0012FF24  FFFFFFFE  [EBPmain-0x1C] JUNK HERE

[ESP+0x28] 0012FF28  0040549C  [EBPmain-0x18] JUNK HERE

[ESP+0x2C] 0012FF2C  004054B0  [EBPmain-0x14] JUNK HERE

[ESP+0x30] 0012FF30  0040344B  [EBPmain-0x10] JUNK HERE

[ESP+0x34] 0012FF34  0012FF48  [EBPmain-0x0C] JUNK HERE

[ESP+0x38] 0012FF38  004028CE  [EBPmain-0x08] JUNK HERE

[ESP+0x3C] 0012FF3C  ACCA6657  [EBPmain-0x04] XOR of stack cookie and EBP

[ESP+0x40] 0012FF40  0012FF88  [EBPmain]      EBP of earlier stack frame stored here

[ESP+0x44] 0012FF44  004012A8  [EBPmain+0x04] return to 0x004012A8 from 0x00401050

Figure 14.14: Dest array incremented

▼Line 53-56

; Line 23

mov ecx, DWORD PTR _src$[ebp]

add ecx, 1

mov DWORD PTR _src$[ebp], ecx

Line 23 of the C/C++ code is:

src++;

In this ASM section, the same steps as the preceding ones are done with the src array. The src array [EBPxstrcpy+0x0C] is moved to the ECX register so that it can be incremented to 1 by the ADD instruction. This incremented pointer value to the src array is moved back on the stack at the [EBPxstrcpy+0x0C] location. The stack state after these instructions will be as follows:

[ESP]  0012FF00  0012FF20  [EBPmain-0x40] [EBPxstrcpy-0x04]  Pointer to dest array

[ESP+0x04] 0012FF04  0012FF40  [EBPmain-0x3C] [EBPxstrcpy] EBP of main() stack frame

[ESP+0x08] 0012FF08  00401087  [EBPmain-0x38] [EBPxstrcpy+0x04] return to main()

[ESP+0x0C] 0012FF0C  0012FF21  [EBPmain-0x34] [EBPxstrcpy+0x08] dest array incremented

[ESP+0x10] 0012FF10  0012FF15  [EBPmain-0x30] [EBPxstrcpy+0x0C] src array incremented

[ESP+0x14] 0012FF14  65766552  [EBPmain-0x2C] “eveR” is pushed here

[ESP+0x18] 0012FF18  45657372  [EBPmain-0x28] “Eesr” is pushed here

[ESP+0x1C] 0012FF1C  0067676E  [EBPmain-0x24] “ ggn” is pushed here

[ESP+0x20] 0012FF20  AC980252  [EBPmain-0x20] only “R” is copied here at dest array

[ESP+0x24] 0012FF24  FFFFFFFE  [EBPmain-0x1C] JUNK HERE

[ESP+0x28] 0012FF28  0040549C  [EBPmain-0x18] JUNK HERE

[ESP+0x2C] 0012FF2C  004054B0  [EBPmain-0x14] JUNK HERE

[ESP+0x30] 0012FF30  0040344B  [EBPmain-0x10] JUNK HERE

[ESP+0x34] 0012FF34  0012FF48  [EBPmain-0x0C] JUNK HERE

[ESP+0x38] 0012FF38  004028CE  [EBPmain-0x08] JUNK HERE

[ESP+0x3C] 0012FF3C  ACCA6657  [EBPmain-0x04] XOR of stack cookie and EBP

[ESP+0x40] 0012FF40  0012FF88  [EBPmain]      EBP of earlier stack frame stored here

[ESP+0x44] 0012FF44  004012A8  [EBPmain+0x04] return to 0x004012A8 from 0x00401050

Figure 14.15: Src array incremented

▼Line 57-58

; Line 24

jmp SHORT $LN2@xstrcpy

Line 24 of the C/C++ code is:

} //while loop closing

This ASM instruction will perform an unconditional jump to the label $LN2@xstrcpy. This unconditional jump will copy the remaining bytes from the src array (stored at [0012FF14]) to the dest array (stored at [0012FF20]). Now we will consider the iteration where all the bytes of the src array are copied to the dest array. The stack state after copying all the bytes will be as follows:

[ESP]  0012FF00  0012FF20  [EBPmain-0x40][EBPxstrcpy-0x04]  Pointer to dest array  

[ESP+0x04] 0012FF04  0012FF40  [EBPmain-0x3C][EBPxstrcpy] EBP of main() stack frame

[ESP+0x08] 0012FF08  00401087  [EBPmain-0x38][EBPxstrcpy+0x04] return to main()

[ESP+0x0C] 0012FF0C  0012FF2B  [EBPmain-0x34][EBPxstrcpy+0x08]  dest array incremented

[ESP+0x10] 0012FF10  0012FF1F  [EBPmain-0x30][EBPxstrcpy+0x0C]  src array incremented

[ESP+0x14] 0012FF14  65766552  [EBPmain-0x2C] “eveR” is pushed here

[ESP+0x18] 0012FF18  45657372  [EBPmain-0x28] “Eesr” is pushed here

[ESP+0x1C] 0012FF1C  0067676E  [EBPmain-0x24] “ ggn” is pushed here

[ESP+0x20] 0012FF20  65766552  [EBPmain-0x20] “eveR” is copied here at dest array

[ESP+0x24] 0012FF24  45657372  [EBPmain-0x1C] “Eesr” is copied here at dest array

[ESP+0x28] 0012FF28 0067676E  [EBPmain-0x18] “ggn” is copied here at dest array

[ESP+0x2C] 0012FF2C  004054B0  [EBPmain-0x14] JUNK HERE

[ESP+0x30] 0012FF30  0040344B  [EBPmain-0x10] JUNK HERE

[ESP+0x34] 0012FF34  0012FF48  [EBPmain-0x0C] JUNK HERE

[ESP+0x38] 0012FF38  004028CE  [EBPmain-0x08] JUNK HERE

[ESP+0x3C] 0012FF3C  ACCA6657  [EBPmain-0x04] XOR of stack cookie and EBP

[ESP+0x40] 0012FF40  0012FF88  [EBPmain]      EBP of earlier stack frame stored here

[ESP+0x44] 0012FF44  004012A8  [EBPmain+0x04]  return to 0x004012A8 from 0x00401050

Figure 14.16: Src array are copied to dest array

Now, we will take the iteration (where an unconditional jump to the label $LN2@xstrcpy was made) after all the bytes are copied from the src array to the dest array.

▼Line 38-43

$LN2@xstrcpy:

; Line 19

mov ecx, DWORD PTR _src$[ebp]

movsx edx, BYTE PTR [ecx]

test edx, edx

je SHORT $LN1@xstrcpy

Line 19 of the C/C++ code is:

while (*src != '\0')

In the ASM code, the MOV instruction is moving the pointer from the src array byte (0x0012FF1F at [EBPxstrcpy+0x0C]) to the ECX register. The next MOV instruction moves byte stored at 0x0012FF1F to EDX, making EDX=0x00000000.

The TEST instruction will perform an AND operation of EDX with itself, resulting in a zero value in EDX and ZF=1. So, a jump to the label $LN1@xstrcpy will take place. The stack state will be the same as earlier:

Figure 14.17: EDX is 0x00

▼Line 59-62

$LN1@xstrcpy:

; Line 27

mov edx, DWORD PTR _dest$[ebp]

mov BYTE PTR [edx], 0

Line 27 of the C/C++ code is:

*dest = '\0';

In ASM code is pushing pointer to dest array (0x0012FF2B at [EBPxstrcpy+0x08]) to EDX register, where it is filled with the null character byte. The stack state will be the same as earlier:

▼Line 63-69

; Line 30

mov eax, DWORD PTR _ptr$[ebp]

$LN4@xstrcpy:

; Line 31

mov esp, ebp

pop ebp

ret 0

Line 30 of the C/C++ code is:

return ptr;

In the ASM code, the pointer to the dest array stored at [EBPxstrcpy-0x04]) is pushed to the EAX register, as the function return value will be stored in the EAX register. Once this value is moved to EAX, the xstrcpy function epilogue is called with the RET instruction. The RET instruction will move the instruction pointer back to the main function. The stack state before the RET instruction is as follows:

[ESP-0x08] 0012FF00  0012FF20   [EBPmain-0x40] Now JUNK  

[ESP-0x04] 0012FF04  0012FF40   [EBPmain-0x3C] Now JUNK  

[ESP]  0012FF08  00401087   [EBPmain-0x38] return address to main()

[ESP+0x04] 0012FF0C  0012FF2B   [EBPmain-0x34] Pointer to dest array incremented

[ESP+0x08] 0012FF10  0012FF1F   [EBPmain-0x30] Pointer to src array incremented

[ESP+0x0C] 0012FF14  65766552   [EBPmain-0x2C] “eveR” is pushed here

[ESP+0x10] 0012FF18  45657372   [EBPmain-0x28] “Eesr” is pushed here

[ESP+0x14] 0012FF1C  0067676E   [EBPmain-0x24] “ ggn” is pushed here

[ESP+0x18] 0012FF20  65766552   [EBPmain-0x20] “eveR” is copied here at dest array

[ESP+0x1C] 0012FF24  45657372   [EBPmain-0x1C] “Eesr” is copied here at dest array

[ESP+0x20] 0012FF28  0067676E   [EBPmain-0x18] “ggn” is copied here at dest array

[ESP+0x24] 0012FF2C  004054B0   [EBPmain-0x14] JUNK HERE

[ESP+0x28] 0012FF30  0040344B   [EBPmain-0x10] JUNK HERE

[ESP+0x2C] 0012FF34  0012FF48   [EBPmain-0x0C] JUNK HERE

[ESP+0x30] 0012FF38  004028CE   [EBPmain-0x08] JUNK HERE

[ESP+0x34] 0012FF3C  ACCA6657   [EBPmain-0x04] XOR of stack cookie and EBP

[ESP+0x38] 0012FF40  0012FF88   [EBPmain]      EBP of earlier stack frame

[ESP+0x3C] 0012FF44  004012A8   [EBPmain+0x04]   return to 0x004012A8

Figure 14.18: Stack state before RET

▼Line 103-107

add esp, 8

push eax

push OFFSET $SG4690

call _printf

add esp, 8

Line 30 of the C/C++ code is:

printf("%s\n", xstrcpy(dest, src));

On return, the ADD instruction will clean up the stack by 8 bytes.

The printf function is called by pushing the return value of xstrcpy onto the stack and the string constant $SG4690 (which is $SG4690 DB '%s', 0aH, 00H) to call the printf function.

This will print the copied string on the screen.

The ADD instruction after printf is called to clean up the stack by 8 bytes. The stack state before the ADD instruction is as follows:

[ESP]  0012FF0C  0040814C  [EBPmain-0x34] "%s\n", argument to printf()

[ESP+0x04] 0012FF10  0012FF20  [EBPmain-0x30] "ReverseEngg", argument to printf()

[ESP+0x08] 0012FF14  65766552  [EBPmain-0x2C] “eveR” is pushed here

[ESP+0x0C] 0012FF18  45657372  [EBPmain-0x28] “Eesr” is pushed here

[ESP+0x10] 0012FF1C  0067676E  [EBPmain-0x24] “ ggn” is pushed here

[ESP+0x14] 0012FF20  65766552  [EBPmain-0x20] “eveR” is copied here at dest array

[ESP+0x18] 0012FF24  45657372  [EBPmain-0x1C] “Eesr” is copied here at dest array

[ESP+0x1C] 0012FF28  0067676E  [EBPmain-0x18] “ggn” is copied here at dest array

[ESP+0x20] 0012FF2C  004054B0  [EBPmain-0x14] JUNK HERE

[ESP+0x24] 0012FF30  0040344B  [EBPmain-0x10] JUNK HERE

[ESP+0x28] 0012FF34  0012FF48  [EBPmain-0x0C] JUNK HERE

[ESP+0x2C] 0012FF38  004028CE  [EBPmain-0x08] JUNK HERE

[ESP+0x30] 0012FF3C  ACCA6657  [EBPmain-0x04] XOR of stack cookie and EBP

[ESP+0x34] 0012FF40  0012FF88  [EBPmain]      EBP of earlier stack frame stored here

[ESP+0x38] 0012FF44  004012A8  [EBPmain+0x04] return to 0x004012A8

Figure 14.19: Stack state before ADD instruction

▼Line 108-119

; Line 41

xor eax, eax

; Line 42

mov ecx, DWORD PTR __$ArrayPad$[ebp]

xor ecx, ebp

call @__security_check_cookie@4

mov esp, ebp

pop ebp

ret 0

_main ENDP

_TEXT ENDS

END

Line 30 of the C/C++ code is:

return 0;

}

In the ASM code, EAX is cleared by XOR and then the stack cookie is checked for any buffer overflow attack by calling the security_check_cookie procedure. On return, the main function epilogue is called to return 0 and end the main function, TEXT segment, and code. The stack state before the RET instruction is as follows:

[ESP-0x0C] 0012FF38  004028CE   Now JUNK HERE

[ESP-0x08] 0012FF3C  ACCA6657   Now JUNK HERE

[ESP-0x04] 0012FF40  0012FF88   Now JUNK HERE

[ESP]  0012FF44  004012A8   return to strcpy.004012A8 from strcpy.00401050

Figure 14.20: Stack state before RET instruction

Strcpy with Optimization

Compile the code with the optimization flag, /Ox flag. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Ox: Enable maximum optimization

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\strcpy\strcpy

^

cl strcpy.cpp /Fastrcpy-Optimized.asm /Ox /Festrcpy-Optimized.exe

The following is the output of running the preceding commands:

Figure 14.21: Strcpy with Optimization

The compilation generates the EXE file and the assembly code. We will again manually disable ASLR. To disable ASLR, use the CFF explorer and change the DllCharacteristics parameter to uncheck DLL can move.

Now, let’s move on to the generated assembly listing:

Figure 14.22: strcpy-Optimized.asm-Part-1

Figure 14.23: strcpy-Optimized.asm-Part-2

Figure 14.24: strcpy-Optimized.asm-Part-3

Let’s directly jump to the start of the TEXT segment of the _main function.

▼Line 63-67

_TEXT SEGMENT

_src$ = -44      ; size = 12

_dest$ = -32      ; size = 25

__$ArrayPad$ = -4     ; size = 4

_main PROC

The local variable on the stack frame can be accessed by adding _<variable-name>$ to the EBP address.

▼Line 67

_main PROC

Start of the main procedure.

▼Line 68-72

; Line 35

sub esp, 44     ; 0000002cH

mov eax, DWORD PTR ___security_cookie

xor eax, esp

mov DWORD PTR __$ArrayPad$[esp+44], eax

Line 35 of the C/C++ code starts the main function.

int main() {

This ASM code is optimized and starts by subtracting ESP by 44(0x2C) bytes to create room for the local variables. 44 bytes can be visualized as 4 bytes for security_cookie plus 12 bytes for the src array, and 28 bytes for the dest array. The sizes of src and dest arrays have been round off to a multiple of 4 bytes.

The next MOV instruction moves the stack cookie from the EAX register to XOR with ESP and stores the resultant on the stack. In a non-optimized code, this XOR procedure happens with EBP. Let’s see the stack state after moving the stack cookie on the stack:

[ESP]  0012FF18  0012FF78  JUNK HERE

[ESP+0x04] 0012FF1C  004024F0  JUNK HERE

[ESP+0x08] 0012FF20  8E50CA61  JUNK HERE

[ESP+0x0C] 0012FF24  FFFFFFFE  JUNK HERE

[ESP+0x10] 0012FF28  0040549C  JUNK HERE

[ESP+0x14] 0012FF2C  004054B0  JUNK HERE

[ESP+0x18] 0012FF30  0040344B  JUNK HERE

[ESP+0x1C] 0012FF34  0012FF48  JUNK HERE

[ESP+0x20] 0012FF38  004028CE  JUNK HERE

[ESP+0x24] 0012FF3C  0040344B  JUNK HERE

[ESP+0x28] 0012FF40  8E02AEA1  XOR of stack cookie and EBP stored here

[ESP+0x2C] 0012FF44  004012A0  ESP at the start of main procedure

Figure 14.25: Stack cookie on the stack

▼Line 73-79

; Line 36

mov eax, DWORD PTR $SG4688

mov edx, DWORD PTR $SG4688+8

mov ecx, DWORD PTR $SG4688+4

mov DWORD PTR _src$[esp+44], eax

push esi

mov DWORD PTR _src$[esp+56], edx

Line 36 of the C/C++ code starts the src array initialization:

char src[] = "ReverseEngg";

In the ASM code, we see three MOV instructions to move the string $SG4688 in three parts (4 bytes + 4 bytes + 4 bytes) to different registers. The first MOV instruction moves the first 4 bytes of the string (“Reve”) to EAX, another 4 bytes of the string (“ngg” + “0x00”) are moved to EDX, and then the remaining bytes (“rseE”) are moved to the ECX register.

In the next MOV instruction, the first part of the string $SG4688 is moved to the top of the stack.

The PUSH instruction stores the ESI value on the stack, so that it can be restored later.

The last MOV instruction moves the third part of the string $SG4688 stored in EDX to the stack at [ESP+0x0C].

The stack state after the execution of the preceding instruction in x32dbg will be:

[ESP]  0012FF14  00000000   ESI is saved here

[ESP+0x04] 0012FF18  65766552  “eveR” is pushed here, in little endian

[ESP+0x08] 0012FF1C  004024F0  “Eesr” will be pushed here, in little endian

[ESP+0x0C] 0012FF20  0067676E  “ ggn” is pushed here, little endian

[ESP+0x10] 0012FF24  FFFFFFFE  JUNK HERE

[ESP+0x14] 0012FF28  0040549C  JUNK HERE

[ESP+0x18] 0012FF2C  004054B0  JUNK HERE

[ESP+0x1C] 0012FF30  0040344B  JUNK HERE

[ESP+0x20] 0012FF34  0012FF48  JUNK HERE

[ESP+0x24] 0012FF38  004028CE  JUNK HERE

[ESP+0x28] 0012FF3C  0040344B  JUNK HERE

[ESP+0x2C] 0012FF40  8E02AEA1  XOR of stack cookie and EBP stored here

[ESP+0x30] 0012FF44  004012A0  ESP at the start of main procedure

Figure 14.26: String $SG4688 on stack

▼Line 80-88

; Line 39

lea eax, DWORD PTR _dest$[esp+48]

mov esi, eax

lea edx, DWORD PTR _src$[esp+48]

mov DWORD PTR _src$[esp+52], ecx

sub edx, esi

mov cl, 82     ; 00000052H

pop esi

npad 6

Line 39 of the C/C++ code starts the printf function.

printf("%s\n", xstrcpy(dest, src));

In the ASM section, the Load Effective Address (LEA) instruction loads the address [ESP+0x10] form the dest array to the EAX register. This allocates space for the dest array on the stack. EAX is then moved to the ESI register using the MOV instruction. This is to store the address of the dest array in the ESI register and at this address is where the src array will be copied. ESI will become 0x0012FF24.

The next LEA instruction does the same for the src array. It loads the address of the src array [ESP+0x04] to the EDX register. EDX will become 0x0012FF18.

The MOV instruction moves the second part of the string $SG4688 stored in ECX to the stack at [ESP+0x08].

The SUB instruction will calculate the offset between the dest array and the src array by subtracting EDX from ESI. The resultant offset will be saved in EDX for later calculations. EDX=0x0012FF18

ESI=0x0012FF24

SUB= EDX-ESI = 0xFFFFFFF4

The CL register is loaded with the first byte 52(“R”) of the src array using the MOV instruction.

Using POP ESI, ESI is restored back to the old value.

Next is the npad macro, which inserts the non-destructive, non-operational instructions. For information on npad, please refer to the Appendix section. Our assembly listing uses the npad 6, which is defined in LISTING.INC as lea ebx, [ebx+00000000]. The stack state after the npad macro is as follows:

[ESP-0x04] 0012FF14  00000000  JUNK HERE

[ESP]  0012FF18  65766552  “eveR” is pushed here, in little endian

[ESP+0x04] 0012FF1C  45657372  “Eesr” will be pushed here, in little endian

[ESP+0x08] 0012FF20  0067676E  “ ggn” is pushed here, in little endian

[ESP+0x0C] 0012FF24  FFFFFFFE  Space for dest array

[ESP+0x10] 0012FF28  0040549C  JUNK HERE

[ESP+0x14] 0012FF2C  004054B0  JUNK HERE

[ESP+0x18] 0012FF30  0040344B  JUNK HERE

[ESP+0x1C] 0012FF34  0012FF48  JUNK HERE

[ESP+0x20] 0012FF38  004028CE  JUNK HERE

[ESP+0x24] 0012FF3C  0040344B  JUNK HERE

[ESP+0x28] 0012FF40  8E02AEA1  XOR of stack cookie and EBP stored here

[ESP+0x2C] 0012FF44  004012A0  ESP at the start of main procedure

Figure 14.27: Stack state after npad

▼Line 89-94

$LL4@main:

mov BYTE PTR [eax], cl

mov cl, BYTE PTR [edx+eax+1]

inc eax

test cl, cl

jne SHORT $LL4@main

This piece of ASM code is where the src array is copied to the dest array memory location byte-by-byte. The first MOV instruction copies the “R” (first char or byte) in the src array to the start of the dest array memory location, 0x0012FF24. The stack with the src and dest array can be visualized as follows:

65 “e”

0x0012FF1B

76 “v”

0x0012FF1A

65 “e”

0x0012FF19

52 “R”

0x0012FF18

45 “E”

0x0012FF1F

65 “e”

0x0012FF1E

73 “s”

0x0012FF1D

72 “r”

0x0012FF1C

00

0x0012FF23

67 “g”

0x0012FF22

67 “g”

0x0012FF21

6E “n”

0x0012FF20

0x0012FF27

0x0012FF26

0x0012FF25

52 “R”

0x0012FF24

0x0012FF2B

0x0012FF2A

0x0012FF29

0x0012FF28

0x0012FF2F

0x0012FF2E

0x0012FF2D

0x0012FF2C

Table 14.1: Stack with src and dest array - Part-1

Once the first byte is copied from the src array to the dest array, the CL register is filled with the next byte of the src array using the second MOV instruction with calculations explained as follows:

mov cl, BYTE PTR [edx+eax+1]

EDX = 0XFFFFFFF4

EAX = 0x0012FF24 (this is the memory address of the start of dest array)

EDX+EAX+1 =   0x0012FF19

Now, the byte will be copied from 0x0012FF19 (which is “e” or 0x65) to the CL register.

As the byte copied to the CL register needs to be moved to the next memory location, EAX is incremented by 1 using the INC instruction. EAX will become 0x0012FF25.

The TEST instruction performs the AND of CL with itself, resulting in a non-zero output in CL and ZF=0. So, a jump to the label $LL4@main will take place. The next iteration of this same ASM code results in copying the next byte to the dest array. The stack state will become as follows:

65 “e”

0x0012FF1B

76 “v”

0x0012FF1A

65 “e”

0x0012FF19

52 “R”

0x0012FF18

45 “E”

0x0012FF1F

65 “e”

0x0012FF1E

73 “s”

0x0012FF1D

72 “r”

0x0012FF1C

00

0x0012FF23

67 “g”

0x0012FF22

67 “g”

0x0012FF21

6E “n”

0x0012FF20

0x0012FF27

0x0012FF26

65 “e”

0x0012FF25

52 “R”

0x0012FF24

0x0012FF2B

0x0012FF2A

0x0012FF29

0x0012FF28

0x0012FF2F

0x0012FF2E

0x0012FF2D

0x0012FF2C

Table 14.2: Stack with src and dest array - Part-2

Now, imagine the iteration where we reach the following stack state where we copy all bytes from the src array to the dest array till 0x67 or “g”.

65 “e”

0x0012FF1B

76 “v”

0x0012FF1A

65 “e”

0x0012FF19

52 “R”

0x0012FF18

45 “E”

0x0012FF1F

65 “e”

0x0012FF1E

73 “s”

0x0012FF1D

72 “r”

0x0012FF1C

00

0x0012FF23

67 “g”

0x0012FF22

67 “g”

0x0012FF21

6E “n”

0x0012FF20

65 “e”

0x0012FF27

76 “v”

0x0012FF26

65 “e”

0x0012FF25

52 “R”

0x0012FF24

45 “E”

0x0012FF2B

65 “e”

0x0012FF2A

73 “s”

0x0012FF29

72 “r”

0x0012FF28

0x0012FF2F

67 “g”

0x0012FF2E

67 “g”

0x0012FF2D

6E “n”

0x0012FF2C

Table 14.3: Stack with src and dest array - Part-3

In this iteration, 0x67 or “g” is copied at the 0x0012FF2E memory location with the first MOV instruction. At this point:

EAX=0x0012FF2E

EDX=0xFFFFFFF4

So, the second MOV instruction will be evaluated to:

mov cl, byte ptr ds:[edx+eax*1+0x1]

mov cl, byte ptr ds:[0x0012FF23]

This will copy 0x00 to the CL register. The next INC instruction will increment EAX to 0x0012FF2F.

The TEST instruction this time will set ZF=1 as CL=0x00, stopping the jump instruction and breaking the loop to move on to the next instruction. The stack state after this part of the ASM code will be:

[ESP-0x04] 0012FF14  00000000  JUNK HERE

[ESP]  0012FF18  65766552  “eveR” is pushed here, in little endian

[ESP+0x04] 0012FF1C  45657372  “Eesr” will be pushed here, in little endian

[ESP+0x08] 0012FF20  0067676E  “ ggn” is pushed here, in little endian

[ESP+0x0C] 0012FF24  65766552  “eveR” is pushed here in dest array

[ESP+0x10] 0012FF28  45657372  “Eesr” will be pushed here in dest array

[ESP+0x14] 0012FF2C  0067676E  “ggn” is pushed here in dest array

[ESP+0x18] 0012FF30  0040344B  JUNK HERE

[ESP+0x1C] 0012FF34  0012FF48  JUNK HERE

[ESP+0x20] 0012FF38  004028CE  JUNK HERE

[ESP+0x24] 0012FF3C  0040344B  JUNK HERE

[ESP+0x28] 0012FF40  8E02AEA1  XOR of stack cookie and EBP stored here

[ESP+0x2C] 0012FF44  004012A0  ESP at the start of main procedure

Figure 14.28: Stack with src and dest array

As we saw in the preceding instructions, the dest array is not NULL terminated as we move out of the loop before NULL is copied to 0x0012FF2F. The answer lies in the next instruction. Let’s move on to the next instructions:

▼Line 95-99

mov BYTE PTR [eax], cl

lea eax, DWORD PTR _dest$[esp+44]

push eax

push OFFSET $SG4690

call _printf

In the first preceding MOV instruction, the ASM code moves 0x00 byte from the CL register to 0x0012FF2F. This will NULL terminate the dest array shown as follows:

65 “e”

0x0012FF1B

76 “v”

0x0012FF1A

65 “e”

0x0012FF19

52 “R”

0x0012FF18

45 “E”

0x0012FF1F

65 “e”

0x0012FF1E

73 “s”

0x0012FF1D

72 “r”

0x0012FF1C

00

0x0012FF23

67 “g”

0x0012FF22

67 “g”

0x0012FF21

6E “n”

0x0012FF20

65 “e”

0x0012FF27

76 “v”

0x0012FF26

65 “e”

0x0012FF25

52 “R”

0x0012FF24

45 “E”

0x0012FF2B

65 “e”

0x0012FF2A

73 “s”

0x0012FF29

72 “r”

0x0012FF28

00 “\0”

0x0012FF2F

67 “g”

0x0012FF2E

67 “g”

0x0012FF2D

6E “n”

0x0012FF2C

Table 14.4: Stack with the src and dest array - Part-4

Now, we have a pointer to the src array (0x0012FF18) and a pointer to the dest array (0x0012FF24) on the stack. Next, we have to call the printf function for which the arguments to printf need to be pushed onto the stack.

The LEA instruction moves the pointer from the dest array to EAX. The PUSH instruction pushes the first argument onto the stack by PUSH EAX.

The second PUSH instruction pushes the string constant onto the stack, which is the second argument to printf. Once both arguments are pushed, a call to printf is made. The stack state before the execution of the CALL instruction is as follows:

[ESP]  0012FF10  0040814C  "%s\n", second argument to printf is pushed here

[ESP+0x04] 0012FF14  0012FF24  "ReverseEngg", second argument to printf()is pushed

[ESP+0x08] 0012FF18  65766552  “eveR” is pushed here, in little endian

[ESP+0x0C] 0012FF1C  45657372  “Eesr” will be pushed here, in little endian

[ESP+0x10] 0012FF20  0067676E  “ ggn” is pushed here, in little endian

[ESP+0x14] 0012FF24  65766552  “eveR” is pushed here in dest array

[ESP+0x18] 0012FF28  45657372  “Eesr” will be pushed here in dest array

[ESP+0x1C] 0012FF2C  0067676E  “ggn” is pushed here in dest array

[ESP+0x20] 0012FF30  0040344B  JUNK HERE

[ESP+0x24] 0012FF34  0012FF48  JUNK HERE

[ESP+0x28] 0012FF38  004028CE  JUNK HERE

[ESP+0x2C] 0012FF3C  0040344B  JUNK HERE

[ESP+0x30] 0012FF40  8E02AEA1  XOR of stack cookie and EBP stored here

[ESP+0x34] 0012FF44  004012A0  ESP at the start of main procedure

Figure 14.29: Stack before CALL instruction

▼Line 100-110

; Line 42

mov ecx, DWORD PTR __$ArrayPad$[esp+52]

add esp, 8

xor ecx, esp

xor eax, eax

call @__security_check_cookie@4

add esp, 44     ; 0000002cH

ret 0

_main ENDP

_TEXT ENDS

END

In the preceding ASM code, the MOV instruction moves the stack cookie stored at [ESP+0x30] to ECX, where it is XOR’ed with ECX to check the buffer overflow condition by calling the security_check_cookie procedure. The rest of the ADD instructions clean up the stack to end the main procedure, TEXT segment, and code. The stack state before the RET instruction is as follows:

[ESP-0x34] 0012FF10  0040814C  Now JUNK HERE

[ESP-0x30] 0012FF14  0012FF24  Now JUNK HERE

[ESP-0x2C] 0012FF18  65766552  Now JUNK HERE

[ESP-0x28] 0012FF1C  45657372  Now JUNK HERE

[ESP-0x24] 0012FF20  0067676E  Now JUNK HERE

[ESP-0x20] 0012FF24  65766552  Now JUNK HERE

[ESP-0x1C] 0012FF28  45657372  Now JUNK HERE

[ESP-0x18] 0012FF2C  0067676E  Now JUNK HERE

[ESP-0x14] 0012FF30  0040344B  Now JUNK HERE

[ESP-0x10] 0012FF34  0012FF48  Now JUNK HERE

[ESP-0x0C] 0012FF38  004028CE  Now JUNK HERE

[ESP-0x08] 0012FF3C  0040344B  Now JUNK HERE

[ESP-0x04] 0012FF40  8E02AEA1  Now JUNK HERE

[ESP]  0012FF44  004012A0  ESP at the start of main procedure

Figure 14.30: Stack cleaned

Conclusion

In this chapter, we covered the strcpy function implementation by copying data from one memory location to another with respect to reverse engineering. We also learned about byte-by-byte operations that happen during the strcpy execution. In the optimized assembly listing, during the stack cookie operation, the XOR procedure happens with ESP and in a non-optimized assembly listing, it happens with EBP. We also covered the difference between optimized and non-optimized codes of strcpy program assembly pattern. In the next chapter, we will talk about another real-world example, wherein we will write a program to calculate simple interest code.

CHAPTER 15

Simple Interest Code Pattern in Reverse Engineering

We all dream about buying a beautiful house and a nice car. Now to buy any of these, we need money which can either be earned with our skills or it can be borrowed from a bank. This is where terms like interest come into picture. Imaging a person named Atul Narula who works for International Institute of Cyber Security and wants to buy a car. Now, Atul has savings of 10,000 dollars and decides to take the remaining 10,000 dollars from a bank. A bank offers him a loan on an annual simple interest rate of 2 percent for a term of 5 years. Atul goes to the bank and asks the bank official about the interest that he has to pay over the term of the loan. The bank official uses a software which uses a mathematical formula to calculate interest. The mathematical formula that goes inside the software will be $10,000 times 2 percent times 5 years, or 10,000 x .02 x 5. This is the interest amount Atul has to pay over the entire term of the loan.

Now imagine, as a reverse engineer, you have to extract the mathematical formula of this banking software without any access to the software code. In this chapter, we will use this real-life software example to understand the assembly pattern of such kinds of software. This type of reverse engineering will help you understand some patterns used by malware writers.

Structure

In this chapter, we will cover the following topics:

Program to calculate simple interest

Calculate simple interest without Optimization

Objective

In this chapter, we will talk about a real-life software that uses an internal mathematical formula to calculate simple interest. We will find out how an integer and float are together handled in memory. Reverse engineering a software with mathematical calculation will help us understand the assembly pattern of the software along with the mathematical calculations.

Program to Calculate Simple Interest

Let’s write down a C/C++ program to calculate simple interest from a set of values represented by Principal Amount, Interest Amount, and Number of Years.

Figure 15.1: SimpleInterestCalculation.cpp

Calculate Simple Interest Without Optimization

Compile the code without optimization in the MSVC compiler on the same x86 Windows machine. Run the following commands on the Windows command prompt to set the environment for cl.exe (VS compiler), and then compile the code with the following switches:

/Fa: Name of the output assembly listing file

/Fe: Name of the output executable file

cd C:\Program Files\Microsoft Visual Studio 10.0\VC

vcvarsall.bat

^

cd C:\JitenderN\REBook\SimpleInterestCalculation\SimpleInterestCalculation

^

cl SimpleInterestCalculation.cpp /FaSimpleInterestCalculation.asm /FeSimpleInterestCalculation.exe

The following is the output of running the preceding commands:

Figure 15.2: Simple Interest without Optimization

The generated assembly code will be as follows:

Figure 15.3: SimpleInterestCalculation.asm-Part-1

Figure 15.4: SimpleInterestCalculation.asm-Part-2

Figure 15.5: SimpleInterestCalculation.asm-Part-3

We have already discussed most of the parts in the listing in the preceding chapters. Let’s move on to line 12.

▼Line 12-14

CONST SEGMENT

$SG4681 DB '%f', 00H

CONST ENDS

The string constant is allocated in the constant segment. In our case, the linker is renamed from CONST SEGMENT to .rdata (the code is placed in the .code segment, the constant strings are placed in the CONST (.rdata) segment, and if not a constant, it is placed in the .data segment), which can be dumped using any debugger. Following screenshot demonstrates string $SG4681 in the memory dump:

Figure 15.6: .rdata

▼Line 15-21

PUBLIC __real@4059000000000000

PUBLIC __real@40f00000

PUBLIC _main

EXTRN _printf:PROC

EXTRN __fltused:DWORD

; COMDAT __real@4059000000000000

; File c:\jitendern\rebook\simpleinterestcalculation\simpleinterestcalculation\simpleinterestcalculation.cpp

The public derivative is to make the real variable public to make it available across modules. The EXTRN derivative declares the extern function which is printf and _fltused. The rest are all comments.

▼Line 22-29

CONST SEGMENT

__real@4059000000000000 DQ 04059000000000000r ; 100

CONST ENDS

; COMDAT __real@40f00000

CONST SEGMENT

__real@40f00000 DD 040f00000r   ; 7.5

; Function compile flags: /Odtp

CONST ENDS

Here, the real variables/numbers are allocated in the .rdata segment. Both can be viewed by dumping the .rdata segment:

Figure 15.7: Float 7.5 to hex

The floating point argument’s hexadecimal representation will be stored in a reverse order, as we are dealing with the little-endian.

Figure 15.8: .rdata with float hex

▼Line 30-35

_TEXT SEGMENT

tv76 = -20      ; size = 4

_fInterestAmt$ = -16     ; size = 4

_iPrincipalAmt$ = -12     ; size = 4

_iNoOfYrs$ = -8      ; size = 4

_fSimpleInterest$ = -4     ; size = 4

From here, our text segment starts. To access the local variable on the stack frame, we have to add _<variable-name>$ to the EBP address. The following figure will help us understand the concept behind accessing arguments on the stack:

Figure 15.9: Stack view

To access the local variable fInterestAmt on the stack frame, we have to add _fInterestAmt$ to the EBP address. So, to access the fInterestAmt variable on the stack, we have to add -16 to the EBP address. The same applies to other variables, each of 4 bytes in size. The same preceding stack can be visualized as follows:

Figure 15.10: Variables on stack

▼Line 36-40

_main PROC

; Line 7

push ebp

mov ebp, esp

sub esp, 20     ; 00000014H

The main procedure starts here and the real code begins with the function prologue. The SUB instruction is creating room for the local variables by subtracting 20 bytes (14H in Hex), equivalent to the space for 5 local variables as shown in the following screenshot.

Figure 15.11: Creating room for variables on the stack

As we have only 4 local variables, you may be wondering why a space for 5 local variables is required. To understand this point, we will walk through the code instruction-by-instruction by putting a breakpoint point on the main procedure call. We will use x32dbg to step into the next instruction:

▼Line 41-42

; Line 11

mov DWORD PTR _iPrincipalAmt$[ebp], 10000 ; 00002710H

This instruction will move the first variable, that is iPrincipalAmt, on the stack at [EBP-12] as shown in the following screenshot.

Figure 15.12: iPrincipalAmt on the stack

▼Line 43-44

; Line 12

mov DWORD PTR _iNoOfYrs$[ebp], 5

This instruction will move the second variable, that is iNoOfYrs, on the stack at [EBP-8].

▼Line 45-46

; Line 13

fld DWORD PTR __real@40f00000.

The Floating Point Load (FLD) will push the floating point value on the FPU stack. In x32dbg, we can see the same instruction as follows:

fld st0, dword ptr ds:[0x0100C150]

This means pushing the floating point value stored at ds:[0x0100C150] to the ST(0) register as shown below.

Figure 15.13: Floating point value on the FPU stack

▼Line 47

fstp DWORD PTR _fInterestAmt$[ebp]

This will move the fInterestAmt floating point value stored at ST(0) to the stack [EBP-0x10] and POPs the variable from the FPU stack. The same instruction is viewed as follows in x32dbg:

fstp dword ptr ss:[ebp-0x10], st0

Figure 15.14: fInterestAmt on stack

▼Line 49

mov eax, DWORD PTR _iPrincipalAmt$[ebp]

It will move the iPrincipalAmt variable value to EAX for further calculation.

▼Line 50

imul eax, DWORD PTR _iNoOfYrs$[ebp]

This multiplies the iNoOfYrs variable stored on the stack [EBP-8] with the iPrincipalAmt variable stored in EAX. The result of the multiplication is stored in the EAX register.

▼Line 51

mov DWORD PTR tv76[ebp], eax

The same can be viewed as follows in x32dbg:

mov dword ptr ss:[ebp-0x14], eax

MOV will move the multiplication value of iPrincipalAmt and iNoOfYrs onto the stack at [EBP-0x14] as a temporary location. We can imagine this multiplication result as a variable local to the main procedure. This is where we our 5th variable comes in picture as shown below in the screenshot.

Figure 15.15: Multiplication value on the stack

▼Line 52

fild DWORD PTR tv76[ebp] ;In x32dbg, fild st0, dword ptr ss:[ebp-0x14]

The multiplication result of iPrincipalAmt and iNoOfYrs stored at [EBP-0x14] is signed-integer. If it has to be moved to the FPU stack, it has to be converted to the floating point format. The FILD instruction converts the signed-integer into a floating point format and then pushes the value onto the FPU register stack.

Figure 15.16: FILD instruction output

▼Line 53

fmul DWORD PTR _fInterestAmt$[ebp]    ;in x32dbg, fmul st0, dword ptr ss:[ebp-0x10]

It will multiple the fInterestAmt variable stored on the stack with the value stored at ST(0). So, this basically multiples the multiplication result of iPrincipalAmt and iNoOfYrs (stored at ST(0)) with fInterestAmt (stored on the stack [EBP-0x10]).

Figure 15.17: FMUL instruction output

▼Line 54

fdiv QWORD PTR __real@4059000000000000

; in x32dbg, fdiv st0, qword ptr ds:[0x0100C148]

Up until now, we have multiplied all three, iPrincipalAmt * iNoOfYrs * fInterestAmt, and the result of this is stored in the FPU stack at ST(0). Now, this will divide the resultant with 100 as per our C/C++ code. FDIV will divide QWORD (1 QWORD = 8 bytes) stored at ds:[0x0100C148] with the value on the FPU stack.

▼Line 55

fstp DWORD PTR _fSimpleInterest$[ebp]

;In x32dbg, fstp dword ptr ss:[ebp-0x4], st0

This will move the fSimpleInterest that we got from ST(0) to the stack [EBP-0x4] and POP the variable from FPU stack as shown in the following screenshot.

Figure 15.18: FSTP instruction output

▼Line 57

fld DWORD PTR _fSimpleInterest$[ebp]

;In x32dbg, fld st0, dword ptr ss:[ebp-0x4]

This pushes the fSimpleInterest floating point value onto the FPU stack.

▼Line 58

sub esp, 8

Before calling the printf function, it will create room for arguments on the stack by 8 bytes.

▼Line 59

fstp QWORD PTR [esp]

;In x32dbg, fstp qword ptr ss:[esp], st0

This will move fSimpleInterest from ST(0) to the stack [ESP] in the QWORD format and then POP the variable from FPU stack. In the following screenshot, variables are marked on stack for proper understanding.

Figure 15.19: fSimpleInterest from ST(0) to stack

▼Line 60

push OFFSET $SG4681

; in x32dbg, push 0x100C140

This will push the string constant on the stack.

Figure 15.20: Before call to printf

▼Line 61-62

call _printf

add esp, 12     ; 0000000cH

xor eax, eax

mov esp, ebp

pop ebp

ret 0

_main ENDP

_TEXT ENDS

END

Now, the printf function arguments are pushed onto the stack; CALL instruction will call the printf function. On return as per the CDECL calling convention, the caller cleans the stack. With this, the listing ends by calling the function epilogue and returning 0.

Conclusion

In this chapter, we learned about a real-life software that uses internal mathematical formula to calculate simple interest. We saw how an integer and float are together handled in memory. We studied the assembly pattern of a software with mathematical calculations. In the next chapter, we will reverse engineer a popular ransomware to crack.

CHAPTER 16

Breaking Wannacry Ransomware With Reverse Engineering

Ransomware is a kind of malware that encrypts the victim’s file and demands a ransom to decrypt those files. If the ransom is not given within time, the victim’s computer data is deleted or left encrypted forever or, sometimes, is sold in the black market. Wannacry was such ransomware which targeted the Windows computers by encrypting data and then demanding a ransom to decrypt the data. Ransom was demanded in the form of Bitcoin cryptocurrency. The impact of Wannacry was so big that it infected millions of computers worldwide and, moreover, it also infected Apple & other servers’ OS. Information Security Newspaper reported that many big companies’ manufacturing plants, like Honda’s, was shut down after some of their computers got infected with Wannacry. Check the following link for reference:

https://www.securitynewspaper.com/2017/06/21/one-month-later-wannacry-ransomware-still-shutting-factories/

As a reverse engineer, we will analyze and break the Wannacry ransomware. When we say ‘break it’, it means that we will try to dig into the code flow of Wannacry and find something that can change the operation to make it ineffective. We will use the reverse engineering framework called Ghidra to analyze and break the Wannacry ransomware.

Structure

In this chapter, we will cover the following topics:

Installation of reverse engineering framework called Ghidra

How to analyze malware using reverse engineering

How to kill Wannacry malware

Objective

The objective of this chapter is to understand the steps involved in installing the reverse engineering framework called Ghidra. After installing it, we will analyze the Wannacry malware and find the kill switch of Wannacry. The attack of Wannacry was stopped after a few days of the kill switch discovery. It affected thousands of computers in 150 countries with a loss of billions of dollars.

Installation

As we have covered the basic introduction of Ghidra in Chapter 3, Up and Running with Reverse Engineering Tools, in this chapter we will walk over the installation of the reverse engineering framework. Ghidra installation is very simple and we will use the Windows 10 64-bit version to carry out our Ghidra installation. Follow the given procedure step by step to install Ghidra:

Download and install JDK 11 from the official website of Oracle –

https://www.oracle.com/java/technologies/javase-jdk11-downloads.html

Select Windows x64 Installer for download.

While downloading JDK, you will be redirected to create an account on the Oracle website as follows:

https://profile.oracle.com/myprofile/account/create-account.jspx

Figure 16.1: Registration for JDK download

After creating an account, login to download and install the JDK on your machine.

Add the JDK bin directory to your windows machine PATH variable. To do that, follow:

Right-click on the Windows Start button and click on System.

Click on Advanced system settings.

Under Advance TAB in the System Properties window, click on Environment Variables.

Under System variables, edit the PATH variable. At the end of the variable, add a semicolon followed by <installation path of JDK dir>\bin as follows:

Figure 16.2: Add JDK to PATH

Now the JDK path is set. It is time to download Ghidra. Do so from the following link:

https://ghidra-sre.org/

Extract the downloaded folder and run ghidraRun.bat as follows:

Figure 16.3: Start Ghidra

It will prompt you to Agree the Ghidra User Agreement. On accepting, you will get the NO ACTIVE PROJECT screen as we have no active projects on Ghidra.

Figure 16.4: Ghidra first screen

With this, we have successfully installed and started Ghidra.

Analyzing and Breaking Wannacry

Before analyzing the Wannacry ransomware, we need to get the copy of the ransomware from the following website. This is 32-bit version of the ransomware binary.

https://www.ghidra.ninja/samples/wannacry.zip

Note: Do not run executable file compressed in the Wannacry zip file on your machine. If you want to run and check how Wannacry malware works, then do it on a virtual machine with no important data on it. Once this binary is executed on machine, you will not be able to access files on that virtual machine.

If you run the Wannacry executable or binary on your virtual machine, disable Defender to get the following screen:

Figure 16.5: Wannacry

Let’s load Wannacry into Ghidra by downloading the Wannacry zip (password: ghidra.ninja) from the preceding link and extract it. Open Ghidra to create a new project from File > New Project. Name it wannacry.

Figure 16.6: Ghidra new project screen

Figure 16.7: Project name

On clicking Finish, your Wannacry project will come under Active Project. Drag and drop the Wannacry executable under Active Project and it will import the binary and flash the following screen:

Figure 16.8: Drag drop wannacry binary

On clicking OK, the next screen will show the Import Result Summary for the Wannacry binary. This will list the Executable format, Compiler ID, Processor, Endianess, and many other components as follows:

Figure 16.9: Import result summary

Press OK and double click on the wannacry project to open CodeBrowser as follows. In the CodeBrowser, it will ask you to analyze the executable.

Figure 16.10: Wannacry analyze now screen

Click Yes to continue. You will have to enable a few analysis options like WindowsPE x86 Propagate External Parameter and Decompiler Parameter ID.

Figure 16.11: Enable analysis options

Once enabled, click on Analyze and ignore the warnings if any.

Figure 16.12: Disassembly view

On the right, we see Program Trees which displays the PE section details. In the Symbol Tree, it displays the symbols currently defined in the program. The middle screen is the disassembled view of the binary and on the left, we see Decompiler. It shows No Function as we have not selected any function from the Functions TAB on the bottom of the Decompiler window. To view the entry point of this executable, type in entry in the functions tab and double click on the entry function to view its disassembled and decompiled code.

Figure 16.13: Entry function

The assembly code we see in the listing section of the preceding screenshot is the entry code of the binary. This entry function is a main() or WinMain() function. For breaking Wannacry, we will have to analyze and understand the code flow. In this process of understanding the code flow, we will first see all the function calls made from the entry disassembled function. This is done by going to the menu Windows > Function Call Graph. A window with all the calls made from the entry function will be shown in a graphical way:

Figure 16.14: Entry function call graph

Most of the calls are made to the internal libraries functions. Let’s analyze the functions of our interest.

Figure 16.15: FUN_00409ba1

FUN_00409ba1 is just a RET, so it does not revealing anything. Move to the next function as follows:

Figure 16.16: FUN_00409b8c

FUN_00409b8c also does not revealing anything. Let’s move to the next function, FUN_00408140, as follows:

Figure 16.17: FUN_00408140-Part-1

Figure 16.18: FUN_00408140-Part-2

This function is something of our interest. We will analyze the disassembled code from whatever we have learned from the earlier chapters. Ghidra also decompiles the binary, but we will first go over the disassembled code step by step to review our learning. While analyzing the disassembled code line by line, we will visualize the stack state to get an overview of the stack state during code flow.

Note: We will be doing a static analysis of the disassembled code. We will not execute this binary during our analysis. Static analysis will help us understand the code flow as well as the working of the ransomware.

Before we start the analysis, let’s visualize a stack where ESP is pointing on the top of the stack. In the following screenshot, you can see the visualization of the stack in a sequence of 4 cells, one above the other. Each cell denotes a byte and the memory addressing is done on the left side from a higher memory location to a lower memory location as we move up. The memory location marked with green is the location of ESP, pointing to the top of the stack having some data marked with XX bytes. The initial stack state before starting the FUN_00408140 function will be as follows. Right now, all the cells are filled with some data which is marked as JUNK. With the instructions flow, bytes will be pushed and popped of the stack.

Figure 16.19: Initial stack state

▼Line 01-18

undefined4 __stdcall FUN_00408140(void)

undefined4        EAX:4          <RETURN>

undefined1        Stack[-0x1]:1  local_1    XREF[1]:     00408177(W)  

undefined2        Stack[-0x3]:2  local_3    XREF[1]:     0040816c(W)  

undefined4        Stack[-0x7]:4  local_7    XREF[1]:     00408168(W)  

undefined4        Stack[-0xb]:4  local_b    XREF[1]:     00408164(W)  

undefined4        Stack[-0xf]:4  local_f    XREF[1]:     00408160(W)  

undefined4        Stack[-0x13]:4 local_13   XREF[1]:     0040815c(W)  

undefined4        Stack[-0x17]:4 local_17   XREF[1]:     00408158(W)  

undefined1        Stack[-0x50]:1 local_50   XREF[2]:     0040814f(*),

SUB ESP,0x50

PUSH ESI

PUSH EDI

At the start of FUN_00408140, we see that different variable macros are defined. SUB is creating room for variables on the stack by subtracting 0x50 from ESP. After creating some room, the ESI and EDI values are preserved on the stack by pushing them on the stack. The stack state after these instructions will be as follows:

Figure 16.20: After creating room for the local variables

▼Line 19-32

MOV ECX,0xe

MOV ESI,s_http://www.iuqerfsodp9ifjaposdfj_004313d0

LEA EDI=>local_50,[ESP + 0x8]

XOR EAX,EAX

MOVSD.REP ES:EDI,ESI=>s_http://www.iuqerfsodp9ifjaposdfj

MOVSB ES:EDI,ESI=>s_http://www.iuqerfsodp9ifjaposdfj

MOV dword ptr [ESP + local_17],EAX

MOV dword ptr [ESP + local_13],EAX

MOV dword ptr [ESP + local_f],EAX

MOV dword ptr [ESP + local_b],EAX

MOV dword ptr [ESP + local_7],EAX

MOV word ptr [ESP + local_3],AX

With the MOV instruction, ECX is filled with 0x0E which will act as a counter to the loop instruction afterwards.

ESI is pointing to offset 'http://www.iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com',0

The length of this URL string is 57 bytes, where one byte is for null termination.

The Load effective address (LEA) instruction is loading the address of ESP+0x08 in EDI.

In these instructions, you will see the REP and MOVS instruction. The REP instruction repeats the string operation ECX times, where ECX is initialized to 0x0E, which is 14 in decimal. As the MOVSD instruction is copying the bytes in the chunk of 4 bytes that is, DWORD from ESI to EDI. So, the total number of bytes that are copied using the MOVSD.REP operation is 14 multiplied by 4 bytes (1 DWORD has 4 bytes) = 56 bytes. An additional null byte for termination is copied using the MOVSB operation, resulting in a total of 57 bytes that are copied from ESI to EDI. This is the same as the length of this URL string, which is 57 bytes.

During the start of the function, we created room for the local variables by subtracting 0x50 (80 bytes in decimal) from ESP. After copying the URL string in this 80-bytes room, the remaining 23 bytes (80 bytes minus 57 bytes) of the memory location on the stack will be cleared using XOR and several MOV instructions. EAX is cleared using the XOR operation and with the remaining MOV instructions, the 22 bytes of the memory location will be cleared as shown in the following stack state screenshot. The yellow-marked cells are the URL string data copied from ESI to EDI on the stack. The blank cells marked with 0x00 show the result of the XOR and MOV XOR operations. The remaining 1 byte will be cleared in the subsequent MOV instruction.

Figure 16.21: URL copied from ESI to EDI

▼Line 34-40

PUSH    EAX  ; dwFlags

PUSH    EAX  ; lpszProxyBypass

PUSH    EAX  ; lpszProxy

PUSH    0x1  ; dwAccessType

PUSH    EAX  ; lpszAgent

MOV     byte ptr [ESP + local_1],AL

CALL    dword ptr [->WININET.DLL::InternetOpenA]

The InternetOpenA function is called, as we can see in this set of assembly listing. As per the Microsoft documentation, the InternetOpenA syntax is defined as follows:

void InternetOpenA(

LPCSTR lpszAgent,

DWORD dwAccessType,

LPCSTR lpszProxy,

LPCSTR lpszProxyBypass,

DWORD dwFlags

);

All the parameters to InternetOpenA are pushed on the stack using the PUSH instructions one by one. This is marked in the following stack state for a better understanding. The MOV instruction is clearing the remaining 1 byte location on the stack as shown in the following stack state:

Figure 16.22: InternetOpenA call

▼Line 42-51

PUSH 0x0       ; dwContext

PUSH 0x84000000      ; dwFlags

PUSH 0x0       ; dwHeadersLength

LEA ECX=>local_50,[ESP + 0x14]    

MOV ESI,EAX

PUSH 0x0       ; lpszHeaders

PUSH ECX       ; lpszUrl

PUSH ESI       ; hInternet

CALL dword ptr [->WININET.DLL::InternetOpenUrlA]

The InternetOpenA function in wininet.dll (32-bit) returns with the RETN 0x14 instructions, which clears off the pushed parameters of the InternetOpenA function on the stack upon RET. Now, with the preceding instructions, the stack is again populated with parameters to the InternetOpenUrlA function. According to the Microsoft documentation, the syntax of InternetOpenUrlA function is:

void InternetOpenUrlA(

HINTERNET hInternet,

LPCSTR    lpszUrl,

LPCSTR    lpszHeaders,

DWORD     dwHeadersLength,

DWORD     dwFlags,

DWORD_PTR dwContext

);

The return value of the InternetOpenA function is stored in the EAX register and is pushed onto the stack as a parameter to the InternetOpenUrlA function with MOV ESI, EAX, and PUSH ESI instructions. The LEA instruction loads the address of the URL string at [ESP + 0x14] into ECX, which is later pushed onto the stack as a parameter to the InternetOpenUrlA function. The stack state before the call to the InternetOpenUrlA function will be as follows:

Figure 16.23: Before call to InternetOpenUrlA

▼Line 53-77

MOV    EDI,EAX

PUSH   ESI

MOV    ESI,dword ptr [->WININET.DLL::InternetCloseHan   = 0000a7b2

TEST   EDI,EDI

JNZ    LAB_004081bc

CALL   ESI=>WININET.DLL::InternetCloseHandle

PUSH   0x0

CALL   ESI=>WININET.DLL::InternetCloseHandle

CALL   FUN_00408090  

POP    EDI

XOR    EAX,EAX

POP    ESI

ADD    ESP,0x50

RET    0x10

LAB_004081bc   XREF[1]:     004081a5(j)

CALL   ESI=>WININET.DLL::InternetCloseHandle

PUSH   EDI

CALL   ESI=>WININET.DLL::InternetCloseHandle

POP    EDI

XOR    EAX,EAX

POP    ESI

ADD    ESP,0x50

RET    0x10

The InternetOpenUrlA function in wininet.dll (32-bit) returns with the RETN 0x18 instruction, which clears off the pushed parameters of the InternetOpenUrlA function on the stack. In this set of assembly instructions, we see something really interesting. The return value of the InternetOpenUrlA function in EAX is moved to EDI. As per the documentation of Microsoft for InternetOpenUrlA function, if the return value is NULL, it means that the connection to the URL failed. If the return value is a valid handle to the URL, it means that the connection to the URL is successfully established. This condition is checked with the TEST instruction in the preceding assembly listing. This brings us to some interesting conclusions from the Function Graph.

Figure 16.24: TEST condition

TEST   EDI,EDI

JNZ    LAB_004081bc

The TEST instruction performs the logical AND between EDI and EDI. This instruction is used to check the registers for zero without altering its value. If EDI is equal to 0, set ZF to 1. If the ZF is set to 1, then no action will be taken and the next instruction following it will be executed. This is a conditional jump instruction which jumps to the LAB_004081bc location if the zero flag (ZF) is set to 0.

The TEST instruction checks the return value of the InternetOpenUrlA function. If the request to http://www.iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com fails, the InternetOpenUrlA function returns the null handle which then closes the handle to calls FUN_00408090 function. This is where the ransomware does all its working.

But if the request to http://www.iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com passes, it simply closes the handle and then quits the Wannacry program to make the ransomware ineffective.

According to Information Security Newspaper, link as mentioned below:

(https://www.securitynewspaper.com/2018/06/09/marcus-hutchins-engineer-who-stopped-wannacry-virus-imputed-in-new-accusations/), Marcus Hutchins was the hero behind the switch of Wannacry. So once he registered this domain, Wannacry was shut down.

So we saw by understanding the assembly listing, we are able to decode the functioning of the malware and find a way to break the Wannacry ransomware. Now, we can validate our understanding of the code flow with the decompiled code generated by Ghidra as follows:

Figure 16.25: Decompiled code

Whatever we have analyzed from the step-by-step assembly instructions is in line with the decompiled code generated from Ghidra. If the InternetOpenUrlA function returns the null handle, the ransomware does all its working with the FUN_00408090 function. Or else, it simply closes the handle and then quits the Wannacry program.

Conclusion

In this chapter, we covered the steps to install a reverse engineering framework called Ghidra. Using Ghidra, we analyzed the Wannacry malware to disassemble the malware. With what we learned from the earlier chapters, we analyzed the code flow with the help of visualizing the stack state during the instruction flow. This helped us understood the code flow and find the kill switch of Wannacry.

CHAPTER 17

Generate Pseudo Code From Binary File

In the earlier chapter, we covered the different patterns of assembly code for various C/C++ applications and some real-life examples. The job of a reverse engineer is to get the copy of the binary for reverse engineering and understanding the code flow. This binary can be of any software or application or it can be malware. All the modern malwares are coded with some URL to communicate with the server to upload data or to send feeds of the malware activities. So, these URL are not hard coded in the plain text format but are encrypted or encoded inside the code.

In this chapter, we will generate the pseudo code from the binary file to crack such encrypted URL. The encryption logic used in such cases can be standard ones or custom made, all depending on the malware writer. We will use Cutter, which is an open-source interface to the Radare2 reverse engineering framework. Radare2 is used for static and dynamic analysis of binary formats on different platforms and architectures. Radare2 is for those who love to work on command line interface. Cutter is the graphical user interface of Radare2.

Structure

In this chapter, we will cover the following topics:

Installation of the reverse engineering framework called Cutter

Binary analysis using Cutter

Decrypting a hidden URL

Objective

The objective of this chapter is to understand the steps involved in installing the reverse engineering framework called Cutter. After installing, we will analyze the binary to generate the pseudo code to extract the encryption key. This encryption key is used to encrypt the URL in the binary. We will also see how this encryption of URL is used to escape from reverse engineering.

Cutter Installation

Cutter can be downloaded from the GitHub repository. It can be compiled from the source code or can be downloaded as a binary. It is available for different platforms (Windows, Linux, macOS). We will use the Windows 10 64-bit version to carry out our Cutter installation. Following is the step-by-step procedure to install Cutter:

Download and extract the zip file from the GitHub repository:

https://github.com/rizinorg/cutter/releases/download/v1.12.0/Cutter-v1.12.0-x64.Windows.zip

In the extracted folder, run cutter.exe to open the first screen of Cutter. This is the screen where we have to select and open the binary file to analyze.

Figure 17.1: Cutter first screen

Download this CrackMe.exe from the following link:

https://github.com/bpbpublications/Implementing-Reverse-Engineering

Once the binary file to be analyzed is selected, click Open to get the cutter Load Options. We are keeping everything default. If you have a symbol file, we can select the Load PDB option.

Figure 17.2: Cutter Load Options

On clicking Ok, you will be presented with the disassembly view of the binary. Along with this, we get to see several tabs at the bottom of window.

Figure 17.3: Binary disassemble view

With this, we have successfully started Cutter for further analysis. But before we begin our analysis, we will walk through the different functionalities and terminologies in the Cutter graphical user interface.

Binary Analysis Using Cutter

The Cutter user interface has many tabs. We will walk through each tab and explain its relevance in analyzing any binary file.

Dashboard

The Dashboard tab shows the binary path, architecture, endianness, and many other details. Our binary file is a portable executable (PE) format file and compiled for 32-bit architecture as follows. The values present in the Hashes section is used to ensure that the file is not corrupted or altered by unauthorized users. Some antivirus companies use these values to determine if a file is malicious or not. They maintain the hash database of known malwares and upon scanning, they evaluate the file hash to compare with that in the database. If the hash values match the values in the database, the file infection is triggered.

Figure 17.4: Dashboard view

Strings

The Strings tab shows the text string found in the binary. It is the first level of analysis for any binary as sometimes it gives a lot of clues about the binary internals. Imagine if some IP address or URL is used in a plain text format in a malware code, then that IP or URL will be reflected in the Strings tab.

Figure 17.5: Strings view

Imports

This tab shows the libraries imported by the binary. If the binary is using the internet to connect to some service, then it can be figured out with the use of the relevant functions used to connect to the internet.

Figure 17.6: Imports view

Disassembly

This tab shows the disassembled view of the binary. Whatever concepts and instructions we have learned in the earlier chapters will help us understand this flow. It might be overwhelming for you in the first view but don’t worry, we will understand this in a step-by-step manner.

Figure 17.7: Disassemble view

Graph

As seen in the earlier figure, this tab is named as Graph (Empty). Empty is suffixed as we have not selected any function. On this tab selection, we get an empty screen with the same message to select a function.

Hexdump

This shows the hex dump of the binary file and is represented by 3 columns. The first column shows the offset, the second column shows the hexadecimal output, and the third column is the representation of data in the ASCII format.

Figure 17.8: Hexdump view

Decompiler

To populate this tab, we will first go to the Windows menu and select Functions to get the list of functions in the binary. In the filter, search for the main function as it is the starting point of the binary. Once the main function is double-clicked, the decompiler will analyze the binary to display the high-level representation of the assembly code in the Decompiler tab. Cutter supports plugin for multiple decompilers such as RetDec and Ghidra.

Figure 17.9: Decompiler view

Decrypting the Hidden URL

Now we will move back to the Graph section which displays the visual process flow of the main function. This graph view can be zoomed in or out using the Ctrl++ or Ctrl— shortcuts. We will understand the execution path in a step-by-step manner to understand what the binary is actually doing. The following is the screenshot of what we got in the graph view for the main function. In the graph view, we see blocks connected with arrows. Arrows are representations of different jumps such as JMP, JGE, JE. The green arrow shows what happens when a jump takes place. The red arrow shows if a jump does not take place. The blue arrow shows the loop.

Figure 17.10: Exported graph of main

Let’s walk over the disassembled code block one by one to understand the code flow:

Figure 17.11: Exported graph of main-Block-1

After the function prologue, the SUB instruction is creating room for the local variable by subtracting ESP by 0x3C (60 bytes in decimal). At [EBP-0x08], the stack cookie is stored by XOR’ing EAX and EBP. In the subsequent MOV instructions, the text string “yyy0kke{dgtugewtkv{0eqo” with string length 24 (in decimal) is copied on the stack. This text string seems to be the encrypted version of something. We will call it the encrypted text string. Out of 60 bytes (in decimal) on the stack, we have consumed 24 bytes (in decimal) for encrypted text string and 4 bytes for the stack cookie. The remaining memory locations on the stack are cleared using the MOV instructions after XOR’ing the EAX instruction. This ends the blocks with a JMP instruction. Let’s move to the remaining instructions in the blocks.

Figure 17.12: Exported Graph of main remaining blocks

We see the looping arrow represented by the blue arrow. It indicates that ECX is initialized to 0x32 (50 in decimal) to loop over with the CMP and JGE instructions. The green arrow on the right indicates that ECX reaches a count of 50 in decimal. This green arrow moves to the end of the code block. The red arrow indicates that we are inside the loop.

The MOVSX instruction is copying every char from the encrypted text string on the stack to EAX. Check if it is NULL or not. If it is NULL, then move to the end of the code block at the bottom right. Or else, again copy the first byte (char) from the encrypted text string on the stack to EDX using the MOVSX instruction. The SUB instruction subtracts 2 from the HEX value of the char copied in EDX. From EDX, it is moved to overwrite the first char of the encrypted text string on the stack in the first loop. This loop is carried out until all the HEX values of the characters of the encrypted text string are subtracted by 2. Thus, 2 is the encrypting key used to encrypt some text string.

With this, we can generate a high-level pseudo code for the binary as follows:

EncryptedText = “yyy0kke{dgtugewtkv{0eqo”

for(iteration=50, EncryptedText !=’\0’, iteration++)

{

EncryptedText[iteration] = EncryptedText[iteration] -2;

}

return 0;

So, we saw how to generate the pseudo code by stepping over the assembly instructions and extracting the meaning out of it. Your pseudo code might not be the same as the original binary code, but it’s a glimpse of what’s happening inside the binary working. With this, we are now clear on how to extract the hidden text behind the encrypted text string.

To extract the original text string, we can either follow a manual or automated process. Manually, it can be done as follows. Refer to the ASCII table in the Appendix for the char to hex conversion.

Figure 17.13: Decrypted text

So, the decrypted text is a URL, which is www.iicybersecurity.com. This was a simple custom encryption to hide the URL in the code. If a plain text URL is used in the code, then it would be visible in the Strings tab of the Cutter graphical interface.

The automated approach to extract the URL from an encrypted URL is to write the Python code as follows:

Figure 17.14: Python code to get the decrypted text

The output of the preceding Python program is as follows:

www.iicybersecurity.com

In the preceding Python code, we are iterating over every char of encstr and subtracting the encryption key 2 from the ASCII value to get the decrypted char using the chr function.

It is time to check the original C++ code from which the CrackMe binary is generated.

Figure 17.15: Binary CPP code

With this, we are able to crack the simple encryption used in the binary.

Conclusion

We covered the steps involved in the installation of the reverse engineering framework called Cutter. With what we have learned in the previous chapters, we were able to analyze the binary to generate a pseudo code to extract the encryption key. We also covered the manual as well as the automated way to extract a hidden URL from the encrypted text. In the next chapter, we will learn some new things about the well-known Windows application.

CHAPTER 18

Fun With Windows Calculator Using Reverse Engineering

In this chapter, we will take up an example to understand how we can use reverse engineering to modify applications or software behaviour without having access to the source code. We will take a well-known Windows application used by everyone. Even those who know the basics of computer use it. We are talking about the Windows Calculator. It is used by computer learners, intermediates, and experts. Everyone uses it for basic and advance calculations. So, what are we going to do with this calculator? This will be an interesting real-life example where, as a reverse engineer, we will change the working of an application with having its source code.

If you are reverse engineering a malware, then this type of real-life scenario will help you change the execution flow of any malware. We will also talk in detail about many concepts involved in this process. This will be a fun exercise to understand.

Structure

In this chapter, we will cover the following topics:

Reverse engineering a calculator

Understanding the code flow with breakpoints

Finding a placeholder to call our code

Writing our code in the Code Cave

Patching the binary

Objective

In this chapter, using reverse engineering, we will change the working of a calculator by modifying its behaviour to output our defined string for any calculation that we perform. It means that rather than getting 8 as an output to 2+6 or any other calculation, the calculator will display our defined message on the press of equal to button. For this, we will use the Win32 Calculator available in the old Windows XP.

Reverse Engineering Calculator

If we want to change the behaviour of any application, we can do so in the application source code and recompile it to get the desired result on execution. In this case, we only have the calculator binary or Portable Executable (PE) file. We will use engineering to modify the calculator binary by writing our code to work as desired. To get the desired result, we will follow a 4-step process to modify the calculator binary:

Understanding the code flow with breakpoints

Finding a placeholder to call our code

Writing our code in the Code Cave

Patching the binary

Understanding the code flow with breakpoints

This step involves understanding the code flow of an application, the calculator in our case, using x32dbg or Ollydbg. Our objective is to modify the calculator is such a way that if somebody presses the equal to button, the user will be presented with our desired string rather than the calculated output. To start, take a copy of calc.exe from the WinXP system32 folder.

Figure 18.1: Calculator binary path

CFF Explorer shows that the binary is Portable Executable 32. Open this binary in x32dbg and go to Debug | Run to start the calculator. Or you can also press F9 to run.

Figure 18.2: Open the calculator binary in x32dbg

First, we will check the list of Win32 function calls in our binary by going to the CPU tab, right-clicking to get the context menu, and then going to Search for -> All Modules -> Intermodular calls:

Figure 18.3: Intermodular calls

This will bring up the All Modules (Calls) tab, where we can see a bunch of function calls and in the bottom, we have a filter option to find any specific function call reference. Now, Win32 offers the SetWindowText function to change the text of control.

BOOL SetWindowTextW(

HWND    hWnd,

LPCWSTR lpString

);

It takes 2 parameters:

hWnd is the handle to the window or control whose text is to be changed.

lpString is the control text

Let’s find this function in the Search filter to find all the references to the function call in our code and set the breakpoint.

Figure 18.4: Search SetWindowTextW and set the breakpoint

Now, we are all set to understand the code flow of the application. We will perform a calculation of 2+6 to see if our breakpoint is hit or not. Let’s begin by first pressing the button for the digit 2. As soon as 2 is pressed on the calculator, our breakpoint is hit.

Figure 18.5: Breakpoint hit as 2 is pressed

The breakpoint is set at the SetWindowTextW function call and we can see that before the CALL to SetWindowTextW, the arguments to the function are pushed onto the stack.

01002621

FF7424 10

push dword ptr ss:[esp+10]

LPCTSTR lpString, second argument

01002625

8BF8

mov edi,eax

01002627

56

push esi

HWND hWnd, first argument

01002628

FF15 10110001

call dword ptr ds:[<&SetWindowTextW>]

SetWindowTextW

The second argument of SetWindowTextW is LPCWSTR lpString, which is pushed first on the stack by the instruction:

push dword ptr ss:[esp+0x10]

This pushes [ESP+0x10] on the stack. While running this push instruction, [ESP+0x10] was pointing to 0x0006F7E0, which is the memory location of the digit we pressed on the calculator, 2. So that means the memory location of the digit 2 (which we pressed on the calculator) was pushed on the stack.

Let’s see how the digit 2 is stored in the memory by dumping 0x0006F7E0 in the memory dump. We can observe that the digit is stored in the Unicode format. For understanding ASCII and Unicode, refer to the Appendix.

Figure 18.6: Memory location of the digit 2

The first argument of SetWindowTextW is HWND hWnd, which is the handle to the control whose text is to be changed. This is pushed on the stack by the PUSH ESI instruction.

We now understand how a digit that we press on the calculator is stored in the memory. Now we will press Run in x32dbg to return the control back to the calculator. Once the calculator has the control, press plus (+) on the calculator. After pressing plus (+), the breakpoint is hit again.

Figure 18.7: Breakpoint hit when + is pressed

All the registers are unchanged at this point and we see that there is no identifier to differentiate when the plus (+) is pressed on the calculator. Press Run in x32dbg again to return the control back to the calculator.

Now we will press the next digit, which is 6, on the calculator. We can again see that our breakpoint is hit and the stack is pushed with the memory location of the digit 6. The same behaviour was observed earlier, wherein the parameters to the SetWindowTextW function were pushed onto the stack.

Figure 18.8: Breakpoint hit on pressing 6

Next, to evaluate 2+6, we will press the equal to button on the calculator. After pressing equal to, our breakpoint is hit but nothing special is observed on the stack or in the register value to help us differentiate when the equal to button is pressed on the calculator.

Figure 18.9: Breakpoint hit when = is pressed

So, we will pass the execution by pressing Run again. This time when the breakpoint is hit, we can see that the result of 2+6, which is 8, is pushed on the stack.

Figure 18.10: Sum 8 is pushed on the stack

Again, 0x0006F7E0 is the memory location of the evaluated result, which is 8. By dumping 0x0006F7E0 in the memory dump, we can observe that the result is stored in the Unicode format.

On the next Run in x32dbg, our evaluated value 8 is displayed on the calculator.

Figure 18.11: Sum 8 is displayed on the calculator

We can observe that every time we press any digit on the calculator, the SetWindowTextW function is called, which pushes the memory location of the digit on the stack. But we were not able to find any identifier or differential flow till that point to help us identify when the user pressed the equal to button on the calculator.

Finding a placeholder to call our code

The objective of this step is to identify a condition where we can differentiate between the press of the equal to button on the calculator and the press of another digit or a button on the calculator. We have to be able to find that differential flow or some register value to help us identify the press of the equal to button. Using that as a condition or trigger, we can jump to our written code to flash our defined string on the calculator when the equal to button is pressed.

To find that differentiating parameter, we will follow a simple process, wherein we will note down the value of all the registers and the stack at the press of any button on the calculator. As we have the x32dbg debugger attached to our calculator, we will follow the given steps:

First press 2 on the calculator to hit the breakpoint. When the execution is paused at the breakpoint, note the register value and the stack.

Now press Run in x32dbg to return the control to the calculator. Press the plus (+) button to hit breakpoint again and note the register value and the stack.

Press Run in x32dbg again and then press 6 to hit breakpoint to note the register value and the stack.

Press Run again in x32dbg to return the control to the calculator. Then press equal to (=) to hit the breakpoint. At this point, you will see that the calculated value of 8 is not pushed on the stack. Press Run in x32dbg again and note the registers’ values and the stack.

Now if you press Run in x32dbg again, it will update the calculator with the calculated value, which is 2+6 = 8.

For your better understanding, we have placed the output of the preceding steps in a table format. This will also help us analyze all the registers and the stack to find any differentiating parameter to help us identify the press of the equal to button on the calculator.

Table 18.1: Comparison of registers and the stack when BP at CALL

As we can see from the table, the register values are the same when the different keys are pressed on the calculator. This is not leading us to find any differentiating parameter to determine whether the user pressed the equal to button or some other digit. Let’s move our breakpoint position to the start of the procedure and see if we can find something there to differentiate between the keys pressed on the calculator. Set the breakpoint at the start of procedure and follow the same debugging process to calculate 2+6 on the calculator.

Table 18.2: Comparison of the registers and the stack when BP at the PROC start

In this exercise, we can observe two points:

First, EAX is holding the memory location of the evaluated result (that is 2 + 6 = 8) and we know that EAX holds the return value of the caller function.

Second, the ESI register can be of our interest. When 2 or plus (+) or 6 is pressed on the calculator, the value of ESI is always lower than let’s say about 1000 (in decimal) (0x3E8 in hex). But when the equal to button is pressed on the calculator, the ESI value is greater than 1000 (in decimal) (0x3E8 in hex). This condition will help us differentiate between the other buttons that are pressed and when the equal to button is pressed on the calculator.

So, we can use this as a triggering condition to jump to our code and to automate this whole process of checking when the equal to button is pressed on the calculator. The pseudo code of this can be as follows:

If (ESI ≤ 0x3E8)  // 2 or plus (+) or 6 button is pressed on the calculator

{

Continue execution normally

} else    // equal to button is pressed on the calculator

{

Run our code to print text on calculator screen

}

Now, we will move on to write our code. But the big question is, where are we going to write our code?

The answer to the question is that we will write our code in the Code Cave. But first, we will understand what code cave is in the following section.

Writing our code in the Code Cave

To modify any application or add some functionality to any application, we need to have the source code. Having a source code and modifying it is the simplest solution to add some functionality to the application. But what if we don’t have the source code?

In that case, we will have to modify the application binary and add our code to it. For adding our code to the compiled application or binary, we will use the code cave.

For adding the code, we will have to find an unused area in the compiled application or exe file. This unused space will be our cave. We will insert our custom code in this cave. Finally, somewhere in the original binary, we will add jump to our code cave so that it is executed along with the original binary execution. At the end, to return the execution from our code cave to the original binary, we will add return to our code cave. This process is used by many malware writers to add custom code to the complied application.

Figure 18.12: Code cave concept

Sometimes, an empty space is not enough to write a big code. In such cases, a new section can be added to the binary with executable privileges. Then we can jump to that section and return back. This technique is not covered here, so let’s move on to code cave.

While writing a code in the cave, we always have to remember that every byte counts. The size of the code should be as small as possible. To find the space available for us, let’s walk through the compiled binary of the calculator and based on that, we will take a decision.

Compiled application, exe, portable exe, binary – All these mean the same.

As we move towards the end of the calculator executable, we see enough space to write our code (shown in the following screenshot). We will select some sections of the space and fill it with the NOP instruction. This section can be referred to as the cave.

Figure 18.13: Code cave with NOPs

After filling the cave with the NOPs, we can start writing our code. The objective of this code is to print JITENDER NARULA IS WATCHING YOU on the calculator when somebody performs some calculation and presses the equal to button for the result. This output text can be customized to anything of your choice. The code that goes into the cave is:

Figure 18.14: Code cave

The code has been explained as follows:

010136BB

cmp esi,3E8

On pressing the equal to button, the ESI value is differentiating the parameter. When ESI > 1000d (0x3E8), the user pressed the equal to button.

010136C1

jle calc.1013739

If user the pressed anything other than equal to, jump and execute normally. Else, jump to print the text ‘JITENDER NARULA IS WATCHING YOU’.

010136C3

mov dword ptr ds:[eax],49004A

As EAX=0x0006F7E0, which is the memory location of the evaluated result - 8.

Here, we are modifying the content of the 0x0006F7E0 memory location and incrementing the memory location until we print our custom text message, that is, JITENDER NARULA IS WATCHING YOU.

The first MOV instruction is writing JI in the Unicode (UTF-16) and little-endian formats at 0x0006F7E0, which is the memory location.

Refer to the ASCII & Unicode sections in the Appendix.

Similarly, at EAX+4==0x0006F7E4, the next two characters (TE) of our text are overwritten using the Unicode and little-endian formats.

The same is done for the remaining chars using the MOV instruction.

Adding the string delimiter in the end.

010136C9

mov dword ptr ds:[eax+4],450054

010136D0

mov dword ptr ds:[eax+8],44004E

010136D7

mov dword ptr ds:[eax+C],520045

010136DE

mov dword ptr ds:[eax+10],4E0020

010136E5

mov dword ptr ds:[eax+14],520041

010136EC

mov dword ptr ds:[eax+18],4C0055

010136F3

mov dword ptr ds:[eax+1C],200041

010136FA

mov dword ptr ds:[eax+20],530049

01013701

mov dword ptr ds:[eax+24],570020

01013708

mov dword ptr ds:[eax+28],540041

0101370F

mov dword ptr ds:[eax+2C],480043

01013716

mov dword ptr ds:[eax+30],4E0049

0101371D

mov dword ptr ds:[eax+34],200047

01013724

mov dword ptr ds:[eax+38],4F0059

0101372B

mov dword ptr ds:[eax+3C],55

01013732

mov dword ptr ds:[eax+40],0

01013739

push esi

If the user pressed anything other than equal to, the instruction pointer will jump here. These instructions are a part of the original compiled binary where we have added our jump to the code cave.

0101373A

push edi

0101373B

push 193

01013740

jmp calc.100260B

Jump to return back to the executable code.

Table 18.3: Code cave explained instruction by instruction

Now we have written our code in the cave. To call this code during the calculator execution, we have to modify the execution flow of the executable as explained in figure 18.12. To do this, we can either use the CALL instruction or the JMP instruction. We can use the CALL instruction to call our code but since we don’t have any space constraints, we will use the basic JMP instruction for easy understanding. Follow below screens to modify executable code flow at the start of the procedure where we inserted our breakpoint for the comparison of registers and the stack to get the triggering condition in table 18.2.

Figure 18.15: Original executable

Figure 18.16: After adding jump to Code Cave

The JMP instruction is pointing to the code cave and the instructions occupied by NOP are called in the code cave. Now we are done with our code insertion in the compiled binary. We will now patch this binary to save the changes in the calculator exe.

Patching the binary

Patching will help us in writing our code in the compiled calculator application on the disk. To patch, go to File -> Patch File, Select All to Patch File. Specify the path & filename to save the file on the disk.

Now, by performing any calculation on our patched calculator binary, the following result will be produced on pressing the equal to button. For example, if we press 2+6 and then the equal to button, we will get the following result:

Figure 18.17: Patched calculator

Placing the breakpoint in our code cave shows that we have overwritten the calculator result in the memory with our custom text JITENDER NARULA IS WATCHING YOU.

Figure 18.18: Patched calculator in x32dbg

Conclusion

In this chapter, we learned to reverse engineer the calculator by modifying its behaviour to output our defined string for any calculation. This means that rather than getting 8 as an output to 2+6 or any other calculation, the calculator will display our defined message on pressing the equal to button. We also learned the concept of finding a cave in the closed binary and writing a code cave to change the execution path of the binary. Further, we learned how to patch the binary to permanently write the changes on the disk.

Appendix

Macro

Macros in the assemble language are used to write modular programs. Macros are the sequence of instruction, assigned by name. They can be used anywhere in the code. The macros make programs shorter and readable.

Procedure

A procedure or subroutine is a sequence of instructions that perform certain tasks. They are very important in assembly language. They are identified by name and the end of the procedure is identified by RET statement.

npad

The npad is a macro defined in the listing.inc which resides in Root_Folder\VC\include folder of MSVC. npad macro inserts non-destructive and non-operational instructions, rather than a series of NOP instruction. While doing optimization the compiler insert non-operational instruction for enforcing alignment of the data. A series of NOP’s can also be used, but single instructions are always good for better CPU performance.

Figure A.1: Listing.inc file path

npad macro is accompanied by a number (X). Where number X can be from 1 to 15, this number defines the amount of memory alignment or padding required by the compiler during optimization. Compiler pads the extra space between the previous instruction/data and the one after npad macro.

If we look into LISTING.INC:

npad 1 defines padding of 1 byte NOP

npad 2 defines padding of 2 bytes with mov edi, edi instruction.

npad 2 defines padding of 3 bytes with lea ecx, [ecx+00] instruction, so on

All these are basically different variations of NOP, which have no impact on the code flow.

LISTING.INC for reference:

;; LISTING.INC

;;

;; This file contains assembler macros and is included by the files created

;; with the -FA compiler switch to be assembled by MASM (Microsoft Macro

;; Assembler).

;;

;; Copyright (c) 1993-2003, Microsoft Corporation. All rights reserved.

;; non-destructive nops

npad macro size

if size eq 1

nop

else

if size eq 2

mov edi, edi

else

if size eq 3

; lea ecx, [ecx+00]

DB 8DH, 49H, 00H

else

if size eq 4

; lea esp, [esp+00]

DB 8DH, 64H, 24H, 00H

else

if size eq 5

add eax, DWORD PTR 0

else

if size eq 6

; lea ebx, [ebx+00000000]

DB 8DH, 9BH, 00H, 00H, 00H, 00H

else

if size eq 7

; lea esp, [esp+00000000]

DB 8DH, 0A4H, 24H, 00H, 00H, 00H, 00H

else

if size eq 8

; jmp .+8; .npad 6

DB 0EBH, 06H, 8DH, 9BH, 00H, 00H, 00H, 00H

else

if size eq 9

; jmp .+9; .npad 7

DB 0EBH, 07H, 8DH, 0A4H, 24H, 00H, 00H, 00H, 00H

else

if size eq 10

; jmp .+A; .npad 7; .npad 1

DB 0EBH, 08H, 8DH, 0A4H, 24H, 00H, 00H, 00H, 00H, 90H

else

if size eq 11

; jmp .+B; .npad 7; .npad 2

DB 0EBH, 09H, 8DH, 0A4H, 24H, 00H, 00H, 00H, 00H, 8BH, 0FFH

else

if size eq 12

; jmp .+C; .npad 7; .npad 3

DB 0EBH, 0AH, 8DH, 0A4H, 24H, 00H, 00H, 00H, 00H, 8DH, 49H, 00H

else

if size eq 13

; jmp .+D; .npad 7; .npad 4

DB 0EBH, 0BH, 8DH, 0A4H, 24H, 00H, 00H, 00H, 00H, 8DH, 64H, 24H, 00H

else

if size eq 14

; jmp .+E; .npad 7; .npad 5

DB 0EBH, 0CH, 8DH, 0A4H, 24H, 00H, 00H, 00H, 00H, 05H, 00H, 00H, 00H, 00H

else

if size eq 15

; jmp .+F; .npad 7; .npad 6

DB 0EBH, 0DH, 8DH, 0A4H, 24H, 00H, 00H, 00H, 00H, 8DH, 9BH, 00H, 00H, 00H, 00H

else

%out error: unsupported npad size

.err

endif

endif

endif

endif

endif

endif

endif

endif

endif

endif

endif

endif

endif

endif

endif

endm

;; destructive nops

dpad macro size, reg

if size eq 1

inc reg

else

%out error: unsupported dpad size

.err

endif

endm

LSB and MSB

To understand least significant bit (LSB) and most significant bit (MSB), we will take an example of byte:

0000 0001

In the above byte example, bit in the right end most is set to 1. This is what is referred as the LSB. Now take another example:

1000 0000

In the preceding byte example, bit in the left end most is set to 1. This is what is referred as the MSB.

This same concept goes with the WORD, DWORD and so on.

WORD

(MSB) 1000 0000 0000 0001 (LSB)

DWORD

(MSB) 1000 0000 0000 0000 0000 0000 0000 0001 (LSB)

Signed and Unsigned

In mathematics we have positive (1, 2, 3, 4, so on) and negative numbers (-1, -2, -3, -4, so on). Similarly to represent positive and negative number concept in programming, signed and unsigned terms are used.

Unsigned

Data is represented by byte of data, where 00 is the lowest number in byte and FF is the highest number in byte. The positive decimal numbers are represented in bytes as shown below and similarly numbers can also be represented in WORD, DWORD:

Figure A.2: Positive decimal numbers in bytes

For WORD this range is from 0000 to FFFF and for DWORD it is from 00000000 to FFFFFFFF. These all are referred to as unsigned numbers.

Signed

When data is represented as a signed, then 0x00 to 0x7F is treated as positive and 0x80 to 0xFF is treated as negative numbers:

Figure A.3: Signed numbers in bytes

As we can observe MSB of all negative number is set to 1 and for positive numbers MSB is set to 0. Consider -4 and + 4 in decimal, its byte and binary representation is:

+4 = 0x04 = 0000 0100, MSB = 0

-4 = 0xFC = 1111 1100, MSB = 1

But when the same decimal is represented in WORD, -4 (decimal) will not become 0x00FC. As 0x00FC is not negative, it represents positive number that falls between 0000 and 7FFF. To represent -4 in WORD, we need to extend it as 0xFFFC and in DWORD it is 0xFFFFFFFC.

Now we understood that simply changing the MSB bit will not change the polarity of number. To convert positive number to negative and vice versa, can be done by performing 2’s complement of the number. To calculate 2’s complement of number following process is followed:

Invert all bits in byte or WORD or DWORD

After flapping all bits, add 1 to the number

Bit Shifting

To understand bit shifting concept, take 0x44 which in binary is:

0100 0100

In bit shifting, bits are shifted either right or left. When shifted to right 0x44 will become:

?0100 010

When shifted to left, 0x44 will become:

100 0100?

When we shifted to right or left a bit placeholder position is created, denoted by question mark (?) above. Now this question mark can be 0 or 1, which is decided by the type of shifting done .i.e. Logical and Arithmetic shifting.

Logical bit shifting

In logical bit shifting, when bits are shifted to the right, bit placeholder which is question mark is always set to 0. Example:

0100 0100

Logical shift right of 0x44 becomes,

00100 010

When bit shifting is done to left, LSB is always set to 0:

0100 0100

Logical shift left of 0x44 becomes,

100 01000

Arithmetic bit shifting

In Arithmetic bit shifting, when bits are shifted to the right, bit placeholder which is question mark is decided by most significant bit (MSB). Example:

0100 0100

Arithmetic shift right of 0x44 becomes,

00100 010

Take another example:

100 01000

Arithmetic shift right of 0x44 becomes,

1100 0100

When bit shifting is done to left, LSB is always set to 0:

0100 0100

Arithmetic Shift left of 0x44 becomes,

100 01000

ASCII

We all are familiar with the ASCII, where 7 bits are used to represent 128 characters and storing them in 8 bits. Every character occupies 1 byte. Below is the ASCII table for reference:

Dec

Hex

symbol

0

00

NULL

(Null character)

1

01

SOH

(Start of Header)

2

02

STX

(Start of Text)

3

03

ETX

(End of Text)

4

04

EOT

(End of Transmission)

5

05

ENQ

(Enquiry)

6

06

ACK

(Acknowledgement)

7

07

BEL

(Bell)

8

08

BS

(Backspace)

9

09

HT

(Horizontal Tab)

10

0A

LF

(Line feed)

11

0B

VT

(Vertical Tab)

12

0C

FF

(Form feed)

13

0D

CR

(Carriage return)

14

0E

SO

(Shift Out)

15

0F

SI

(Shift In)

16

10

DLE

(Data link escape)

17

11

DC1

(Device control 1)

18

12

DC2

(Device control 2)

19

13

DC3

(Device control 3)

20

14

DC4

(Device control 4)

21

15

NAK

(Negative acknowledgement)

22

16

SYN

(Synchronous idle)

23

17

ETB

(End of transmission block)

24

18

CAN

(Cancel)

25

19

EM

(End of medium)

26

1A

SUB

(Substitute)

27

1B

ESC

(Escape)

28

1C

FS

(File separator)

29

1D

GS

(Group separator)

30

1E

RS

(Record separator)

31

1F

US

(Unit separator)

32

20

(Space)

33

21

!

(Exclamation mark)

34

22

"

(Quotation mark ; quotes)

35

23

#

(Number sign)

36

24

$

(Dollar sign)

37

25

%

(Percent sign)

38

26

&

(Ampersand)

39

27

'

(Apostrophe)

40

28

(

(round brackets or parentheses)

41

29

)

(round brackets or parentheses)

42

2A

*

(Asterisk)

43

2B

+

(Plus sign)

44

2C

,

(Comma)

45

2D

-

(Hyphen)

46

2E

.

(Dot, full stop)

47

2F

/

(Slash)

48

30

0

(number zero)

49

31

1

(number one)

50

32

2

(number two)

51

33

3

(number three)

52

34

4

(number four)

53

35

5

(number five)

54

36

6

(number six)

55

37

7

(number seven)

56

38

8

(number eight)

57

39

9

(number nine)

58

3A

:

(Colon)

59

3B

;

(Semicolon)

60

3C

<

(Less-than sign)

61

3D

=

(Equals sign)

62

3E

>

(Greater-than sign ; Inequality)

63

3F

?

(Question mark)

64

40

@

(At sign)

65

41

A

(Capital A)

66

42

B

(Capital B)

67

43

C

(Capital C)

68

44

D

(Capital D)

69

45

E

(Capital E)

70

46

F

(Capital F)

71

47

G

(Capital G)

72

48

H

(Capital H)

73

49

I

(Capital I)

74

4A

J

(Capital J)

75

4B

K

(Capital K)

76

4C

L

(Capital L)

77

4D

M

(Capital M)

78

4E

N

(Capital N)

79

4F

O

(Capital O)

80

50

P

(Capital P)

81

51

Q

(Capital Q)

82

52

R

(Capital R)

83

53

S

(Capital S)

84

54

T

(Capital T)

85

55

U

(Capital U)

86

56

V

(Capital V)

87

57

W

(Capital W)

88

58

X

(Capital X)

89

59

Y

(Capital Y)

90

5A

Z

(Capital Z)

91

5B

[

(square brackets or box brackets)

92

5C

\

(Backslash)

93

5D

]

(square brackets or box brackets)

94

5E

^

(Caret or circumflex accent)

95

5F

_

(underscore, understrike, underbar or low line)

96

60

`

(Grave accent)

97

61

a

(Lowercase a)

98

62

b

(Lowercase b)

99

63

c

(Lowercase c)

100

64

d

(Lowercase d)

101

65

e

(Lowercase e)

102

66

f

(Lowercase f)

103

67

g

(Lowercase g)

104

68

h

(Lowercase h)

105

69

i

(Lowercase i)

106

6A

j

(Lowercase j)

107

6B

k

(Lowercase k)

108

6C

l

(Lowercase l)

109

6D

m

(Lowercase m)

110

6E

n

(Lowercase n)

111

6F

o

(Lowercase o)

112

70

p

(Lowercase p)

113

71

q

(Lowercase q)

114

72

r

(Lowercase r)

115

73

s

(Lowercase s)

116

74

t

(Lowercase t)

117

75

u

(Lowercase u)

118

76

v

(Lowercase v)

119

77

w

(Lowercase w)

120

78

x

(Lowercase x)

121

79

y

(Lowercase y)

122

7A

z

(Lowercase z)

123

7B

{

(curly brackets or braces)

124

7C

|

(vertical-bar, vbar, vertical line or vertical slash)

125

7D

}

(curly brackets or braces)

126

7E

~

(Tilde ; swung dash)

127

7F

DEL

(Delete)

Table A.1: ASCII table

Unicode

There are many versions of Unicode, UTF-16 is the most popular one. It is represented by 16 bits, which is needed to satisfy any language efficiently. It ranges from 0x0000 to 0xFFFF.

In ASCII it was not possible to store characters of different languages, as it just 7 bits. But Unicode versions are expanded to 16, 32 bits

Example: ASCII of ‘A’ = 0x41 and Unicode (UTF=16) representation is 0041.

Disable Address Space Layout Randomization

Address Space Layout Randomization (ASLR) is a security mechanism by which base address of PE file is randomized on every load. To disable the ASLR on the Portable Executable (PE) file generated with our MSVC compiler we will following steps. This will help us reload PE file without randomizing base address of PE file. To disable ASLR we will use a CFF Explorer (it can be download from https://ntcore.com/?page_id=388). Open the PE file generated in CFF Explorer and follow steps:

Select from the left panel, Optional Header

In the Optional Header, find DllCharacteristics.

Figure A.4: PE file optional header using CFF Explorer

Click on Click here on the right bottom.

Uncheck the DLL can move and click OK.

Figure A.5: Disable ASLR on PE file

Now go to File menu and Save changes.

Now using x32dbg, we can load PE file to do analysis without randomizing base address on every load.

Index

A

AddNumber.cpp code 86

Address Space Layout Randomization (ASLR)

about 187, 460

disabling 460, 461

Advanced Micro Devices 3

architecture, computing system

about 9

CPU 9

input/output devices 10

memory 10

System Bus 10

arithmetic bit shifting 456

arithmetic instructions

AAA 41

AAD 42

AAM 43

AAS 42

ADC 43

ADD 43

CMP 43, 44

DAA 44

DAS 44

DEC 45

DIV 45

IDIV 46

IMUL 47

INC 47

MUL 46, 47

NEG 47, 48

SBB 48

SUB 48

XADD 48, 49

array

about 247

defining 248

array code

reverse engineering pattern 247

array loop

with optimization 267-277

without optimization 248-266

ASCII 456, 460

assembly instructions

about 20, 23

arithmetic instructions 41

bit manipulation instructions 61

branching instructions 54

categories 35

data transfer instructions 37

processor control instructions 70

program execution instructions 49

stack instructions 35

string instructions 73

assembly instruction, with register operands

example 35

assembly language instructions 34

B

basic code

reverse engineering pattern 97

binary analysis, with Cutter

dashboard tab 418, 419

decompiler tab 422, 423

disassembly tab 421

graph tab 421

hexdump tab 422

imports tab 420

strings tab 419, 420

bit manipulation instructions

about 61

AND 61, 62

BSWAP 61

NOT 62

OR 62

RCL 63

XOR 63

bit shifting 455

branching instructions

about 54

JA 57

JAE 57, 58

JB 59

JBE 59

JE 55

JECXZ 60, 61

JG 56

JGE 57

JL 58

JLE 58

JMP 54

JNE 56

JNZ 55

JO 60

JS 60

JZ 55

RCR 64

ROL 64

ROR 65

SAL 68

SAR 67, 68

SHL 66, 67

SHLD 69

SHR 65, 66

SHRD 69

building blocks, of computing system

Bit 11

Byte 11

DWORD 11

Nibble 11

Word 11

C

calling conventions

about 84

CDECL 85

FASTCALL 85

STDCALL 85

types 84

CDECL calling convention 85

CFF Explorer, PE 28

Clear the Carry Flag (CLC) instruction 70

Clear the Direction Flag (CLD) instruction 70

Clear the Interrupt Flag (CLI) instruction 71

code cave concept 442

code optimization

about 98

empty function 98

empty function with optimization 106, 107

empty function without optimization 99-105

common sections, PE

.bss 27

.debug 28

.edata 28

.idata 28

.rdata 28

.rsrc 28

.text 27

Complement Carry (CMC) instruction 71

Component Object Model (COM) 27

computing system

about 8

architecture 9

building blocks 11

fixed program computing system 8

stored program computing system 8

concept, behind different calling conventions

about 86

CDECL 87-89

FASTCALL 93-95

STDCALL 89-92

conditional jump 54

CPU, computing system

control unit 9

execution unit 9

flags 10

registers 9

cryptocurrency 4

Cutter

about 30, 416

downloading 416

installing 416-418

using, for binary analysis 418

D

data transfer instructions

CMPXCHG 38

LAHF 39

LAR 39

LEA 38

MOV 37

MOVS 41

MOVSX 40

MOVZX 40

SAHF 39

XCHG 38

XLAT 40

debuggers

about 30

x32dbg 31

decision control structure

reverse engineering pattern 185, 186

disassemblers

about 29

Cutter 30

Ghidra 29

Disk Operating System (DOS) 2

DLL can move 187

DllCharacteristics parameter 187

Dynamic Link Library (DLL) 27

E

EEFLAGS register 17

EIP. See Instruction Pointer register

Escape to Floating Point Coprocessor (ESC) instruction 71

Exchange and Add (XADD) instruction 48

F

FASTCALL 85

first generation microprocessors

Intel 8086 12

Intel 80186 12

Intel 80286 13

Intel 80386 13

Intel 80486 13

fixed program computing system 8

Floating Point Unit (FPU) 131

Foo function 19

for loop

about 230

with optimization 240-245

without optimization 230-239

function printf with char

about 147

function printf printing char with optimization 149-151

function printf printing char without optimization 147-149

function printf with float

about 131

function printf printing float with optimization 143-146

function printf printing float without optimization 131-143

function printf with integers

about 122

function printf printing integers with optimization 129, 130

function printf printing integers without optimization 122-129

function scanf

with integers 312

with optimization 320-326

without optimization 312-319

G

general purpose registers, x86 architecture

EAX 16

EBP 16

EBX 16

ECX 16

EDI 16

EDX 16

ESI 16

ESP 16

Ghidra

about 29

installing 394-397

H

Hard Disk (HDD) 10

Hashes section 418

“hello, world” program

about 113

with optimization 117-119

without optimization 113-117

hidden URL

decrypting 423-428

Hungarian Notation 154

I

iArray 248

if-else statement

about 186

with optimization 199-210

without optimization 186-199

installation

Ghidra 394

Instruction Pointer register 18

Integer Multiple (IMUL) instruction 47

Intel 8086 12

Intel 80186 12

Intel 80286 13

Intel 80386 13

Intel 80486 13

Intel processor

x86-16 14

x86-32 (aka IA32) 14

x86-64 14

Intel x86-32 processor

memory 14

registers 15

Interrupt Overflow (INTO) 51

Interrupt Return (IRET) instruction 52

Interrupt Service Routine (ISR) 52

iNumber 154

I/O (input/output) devices, computing device 10

J

JDK 11

downloading 394

installing 394

Jump if above (JA) instruction 57

Jump if above or equal (JAE) instruction 57

Jump if below (JB) instruction 59

Jump if below or equal (JBE) instruction 59

Jump if Equal (JE) instruction 55

Jump if Greater (JG) instruction 56

Jump if Greater or Equal (JGE) instruction 57

Jump if Less (JL) instruction 58

Jump if Less or equal (JLE) instruction 58

Jump if Not Equal (JNE) instruction 56

jumps, in x86

conditional jump 54

unconditional jump 54

L

Last In First Out (LIFO) 18

least significant bit (LSB) 453

Load Effective Address (LEA) 162

Load String Byte (LODSB) 78

Load String DWORD (LODSD) 78

Load String (LODS) 78

Load String Word (LODSW) 78

logical bit shifting 455

loop control structure

reverse engineering pattern 211

M

macro 449

main function 19

memory, computing system

Random Access Memory (RAM) 10

Read Only Memory (ROM) 10

memory, Intel x86-32 processor

data 15

heap 15

kernel space 15

libraries 15

stack 15

text 15

microprocessor

about 12

first generation microprocessor models 12

most significant bit (MSB) 453

Move String Byte (MOVSB) 80

Move String DWORD (MOVSD) 80

Move String (MOVS) 80

Move String Word (MOVSW) 80

N

National Security Agency (NSA) 29

No Operation (NOP) instruction 72

npad 449

npad macro 450

O

operands

intermediate operands 34

memory address operands 34

register operands 34

operation code 31, 34

optimization 98

OUTSB instruction 77

OUTSD instruction 77

OUTSW instruction 77

Overflow flag (OF) 51

P

patched calculator 446

pointer program

reverse engineering pattern 153

pointers

about 154-156

with optimization 169-183

without optimization 156-169

POP 18

Portable Executable Editors

about 27

basic structure 27

CFF Explorer 28

common sections 27

printf program

reverse engineering pattern 121

procedure 449

processor control instructions

CLC 70

CLD 70

CLI 71

CMC 71

ESC 71

LOCK 72

NOP 72

STC 72, 73

STD 73

STI 73

program code 34

program execution instructions

about 49

CALL 49

ENTER 49, 50

INT 50, 51

INTO 51

IRET 51, 52

LEAVE 50

LOOP 52

LOOPE 52

LOOPNE 53

TEST 53

pseudo code

about 18

Foo function 19

main function 19

PUSH 18

R

Radare2 30

Random Access Memory (RAM) 10

ransomwares 3

Read Only Memory (ROM) 10

registers, Intel x86-32 processor

about 15, 17

general purpose registers 16

Instruction Pointer register 18

segment registers 17

Repeat if Equal (REPE) 81

Repeat if Not Equal (REPNE) 82

Repeat if Not Zero (REPNZ) 82

Repeat if Zero (REPZ) 81

Repeat (REP) 81

returning value

about 108

returning value with optimization 110-112

returning value without optimization 108-110

Reverse Engineering (RE)

about 1, 2

bounty for cyber enthusiasts 5

example 2, 3

existing design, studying 4

importance 3, 4

military espionage 4

outdated or lost product, redeveloping 4

product vulnerabilities, finding 5

role 5, 6

security audit 4

sensitive data, finding 4

reverse engineering tools

categories 27

debuggers 30

disassemblers 29

importance 26

Portable Executable Editors 27

Rotate Left (ROL) 64

Rotate Right (ROR) 65

Rotate through Carry Left (RCL) instruction 63

Rotate through carry right (RCR) 64

S

scanf program pattern

in reverse engineering 311

Scan String Byte (SCASB) 79

Scan String DWORD (SCASD) 79

Scan String (SCAS) 79

Scan String Word (SCASW) 79

segment registers, x86 architecture

Code Segment (CS) 17

Data Segment (DS) 17

ES 17

FS 17

GS 17

Stack Segment (SS) 17

Set the Carry Flag (STC) instruction 73

Set the Direction Flag (STD) instruction 73

Set the Interrupt Flag (STI) instruction 73

Shift Arithmetic Left (SAL) 68

Shift Arithmetic Right (SAR) 68

Shift Left Double (SHLD) 69

Shift Logical Left (SHL) 67

Shift Logical Right (SHR) 66

Shift Right Double (SHRD) 69

signed terms 454, 455

simple interest calculation

program, writing for 376

without optimization 376-392

simple interest code pattern

in reverse engineering 375

stack

about 18, 19

pseudo code 18

stack frame

about 19

Callee After Function Call 21, 22

Callee Before Returning 23

Caller After Returning 24

Caller Before Callee Call 20, 21

stack instructions

about 35

POP 36

POPAD 37

POPFD 37

PUSH 36

PUSHAD 36

PUSHFD 36

RET 37

status register, x86 architecture

Carry Flag (CF) 17

Overflow Flag (OF) 18

Parity Flag (PF) 18

Sign Flag (SF) 17

Trap Flag (TF) 18

Zero Flag (ZF) 17

STDCALL 85

stored program computing system 8

Store String Byte (STOSB) 78

Store String DWORD (STOSD) 79

Store String (STOS) 78

Store String Word (STOSW) 79

strcpy

about 328

with optimization 357-374

without optimization 329-356

strcpy program pattern

in reverse engineering 327

string instructions

CMPS/CMPSB/CMPSW 73, 74

IN/INSB/INSW/INSD 75, 76

LODS/LODSB/LODSW/LODSD 78

MOVS/MOVSB/MOVSW 80

OUT/OUTSB/OUTSW/OUTSD 76, 77

REP 81

REPE/REPZ 81

REPNE/REPNZ 82

SCAS/SCASB/SCASW 79

STOS/STOSB/STOSW 78

structure

about 282

with optimization 299-310

without optimization 282-298

structure code

reverse engineering pattern 281

subroutine 449

System Bus, computing system

Address Bus 10

Control Bus 10

Data Bus 10

T

tools, reverse engineering. See reverse engineering tools

U

unconditional jump 54

Unicode 460

unsigned terms 454

UTF-16 460

V

von Neumann architecture 9

W

Wannacry ransomware

about 3

analyzing 397-401

breaking 393

disassembling 402-414

while condition

about 212

with optimization 222-229

without optimization 212-222

Windows Calculator

reverse engineering 429

Windows Calculator, reverse engineering

about 430

binary, patching 445, 446

code flow, with breakpoints 430-438

code, writing in code cave 441-445

placeholder, finding to call code 439-441

X

x32dbg

about 31

disassembly or CPU instructions 31

memory dump 31

registers and flags 31

stack 31

x86 Intel family 14