CSCI 2330
Introduction to Systems

Bowdoin College
Spring 2016
Instructor: Sean Barker

Project 3 - Stack Smasher

This project should be completed individually.

This project will teach you about different ways that common classes of bugs can lead to real-world security vulnerabilities. In exploring these vulnerabilities, you will gain a deeper understanding of stack mechanisms and instruction encodings as well as experience with debugging tools. You will also gain a better understanding of some of the features provided by compilers and operating systems to make programs more secure.

Note: We do not condone the use of any form of attack to gain unauthorized access to any system resources. Nor does the government, for that matter - please act responsibly!

Start by reading through the entire project description!

Project Overview

Most of the bitbombs on our machines have been safely defused, but a few unusual ones remain. While it appears at first glance that these bitbombs are duds, we believe that they are actually prototypes of much more powerful implosive bytebombs! The bytebombs appear to be detonated via activation strings that exploit security vulernabilities hidden in the code. Luckily, we've managed to obtain partial source code to the bytebombs that should prove useful in figuring out how they work.

We've placed the bytebombs in a secure, binary-shielded area -- we need you to find working activation strings so that we can safely defend against these bytebombs in the future. Get out your stack hammer and start smashing!

ByteBomb Files

Each of you have a set of files pertaining to your bytebomb that have been checked into your SVN directory under proj3. First, run svn update within your directory to download these files. Within the proj3 directory, there are several key files:

ByteBomb Input

Each bytebomb reads an activation string when run using the following getbuf function:

unsigned getbuf() {
    char buf[BUFFER_SIZE];
    return 1;

The getbuf function is called from the test function, which is as follows:

void test() {
  int val;
  val = getbuf(); 
  printf("Nothing happens. (0x%x)\n", val);

The function Gets is similar to the standard library function gets -- it reads a string from standard input (terminated by '\n' or end-of-file) and stores it (along with a null terminator) at the specified destination. In this code, you can see that the destination is the array buf, declared as having BUFFER_SIZE bytes. At the time your targets were generated, BUFFER_SIZE was a compile-time constant specific to your bytebombs.

You hopefully notice that this function presents an entry point through which you can smash the stack! Experiment with your bytebomb to confirm that this is the case (you can defer examining bigbytebomb until later, though that works similarly). You should be able to very easily make the program segfault!

Note that unlike in the previous assignment, there is no penalty associated with detonating (or not detonating, as the case may be) the bytebombs. Feel free to fire away with any input strings you wish!

Also note that each bomb prints its unique ID when run. As with the BUFFER_SIZE constant, the value of this ID is unique to your bomb.

While crashing the bomb is quite easily done, your goal is to write more clever input strings that will cause the bomb to activate in a variety of ways. These input strings are called exploit strings.

Formatting Exploit Strings

Your exploit strings will typically contain byte values that do not correspond to the ASCII values used for printing characters, and therefore you cannot easily type your exploit strings directly. Instead, you'll need to generate raw (i.e., binary) strings consisting of arbitrary byte values. The included program hextoraw will allow you easily generate raw strings. The hextoraw program expects input on stdin unless given the -f file option, in which case it will read input from file.

In a hex-formatted string, each byte value is represented by two hex digits. Byte values are separated by spaces. For example, the string "012345" could be entered in hex format as 30 31 32 33 34 35. Remember that the ASCII code for decimal digit N is 0x3N. Run man ascii for a full table of ASCII values. Non-hex digit characters are ignored, including the blanks in the example shown.

You should use the existing text files 1-hiss.txt through 5-implodebig.txt to store your (human-readable) exploit strings for each phase. If your exploit string is contained in a file called exploit.txt, you can easily pass it to your bytebomb via hextoraw using Unix pipes (a standard way of passing output from one program to the input of another) like so:

$ cat exploit.txt | ./hextoraw | ./bytebomb

The above command outputs the contents of exploit.txt using the cat command, then passes that output to the program hextoraw as standard input, then takes the (raw) output of hextoraw and passes it to bytebomb.

You can also store the raw string bytes in a file using I/O redirection, then redirect them back to the bomb, as follows:

$ ./hextoraw -f exploit.txt > exploit.bytes
$ ./bytebomb < exploit.bytes

The above will create a new file called exploit.bytes containing the raw (i.e., not human-readable) exploit string, and then give that as input to the bytebomb. Using this method, you can also easily pass the raw bytes to the program from within gdb:

$ gdb ./bytebomb
(gdb) run < exploit.bytes

One important note is that your exploit string must not contain byte value 0x0A at any intermediate position, since this is the ASCII value for a newline ('\n'). When Gets() encounters this byte, it will assume that you intended to terminate the string input. However, hextoraw will warn you if it encounters this byte value.

Generating Byte Codes

You may wish to return to this section after reading the exploit section below.

Your exploit strings will often want to include the actual encodings of assembly instructions (i.e., byte codes). You can find these actual encodings by hand-writing assembly instructions, using gcc as an assembler, and objdump as a disassembler. For example, suppose you write a file example.s containing the following assembly code:

# Example of hand-generated assembly code
pushq   $0xabcdef     # Push value onto stack
addq    $17,%rax      # Add 17 to %rax
movl    %eax,%edx     # Copy lower 32 bits to %edx

You can now assemble and then disassemble this file:

$ gcc -c example.s
$ objdump -d example.o > example.d

The generated file example.d now contains the following lines of interest:

0: 68 ef cd ab 00     pushq $0xabcdef
5: 48 83 c0 11        add $0x11,%rax
9: 89 c2              %eax,%edx

Each line shows a single instruction generated from an assembly language instruction. The number of the left indicates the starting address (starting with 0) while the hex digits after the colon indicate the byte codes for the instruction. Thus, we can see that the instruction push $0xABCDEF has hex-formatted byte code 68 ef cd ab 00.

Remember that endinaness matters! For example, note that the value 0xABCDEF is specified in reverse byte order starting at byte address 1 above, since we're running on a little-endian machine.

From the above, we can read off the entire byte sequence for the code:

68 ef cd ab 00 48 83 c0 11 89 c2

This hex-formatted exploit string can then be passed through hextoraw to generate a raw input string for the bytebomb. Alternately (and perhaps preferably) you can simply edit the example.d file to omit extraneous characters and to contain C-style commends for readability, yielding:

   68 ef cd ab 00   /* pushq  $0xabcdef  */
   48 83 c0 11      /* add    $0x11,%rax */
   89 c2            /* mov    %eax,%edx  */

However, remember that you should store your human-readable exploit strings in the existing .txt files located in your directory for that purpose. The hextoraw program will ignore C-style strings like those in the example above, and thus you should feel free to (and should) comment your exploit string files.


There are five exploits to tackle to fully unlock the capabilities of your bytebombs. The first three exploits involve attacking the bytebomb program, while the last two exploits involve attacking the bigbytebomb program. The goal of each exploit is to 'repurpose' the bomb to execute a function that it should not normally execute.

However, the methods you will use to accomplish each exploit will vary. In particular, the first three exploits require you to use code-injection attacks, while the last two exploits require you to use return-oriented programming. Each of the five exploits are described in more detail below.

Part I: Code Injection Attacks

The first three exploits will have you exploiting the bytebomb executable using code injection attacks. Recall that the basic idea in a code injection attack is to use a vulnerable function to inject a series of commands onto the stack, then overwrite the function's return address with the address of the injected commands. Then, when the function executes the ret instruction, rather than returning to the calling function, the program will jump to the injected commands.

Exploit 1: Hiss

The first exploit simply requires you to get the bytebomb to execute the hiss function, which is defined as follows:

void hiss() {
  vlevel = 1;       /* Part of validation protocol */
  printf("The bytebomb hisses loudly!\n");

To do so, you will need to smash the stack and change the return address of getbuf. All the information you need to devise your exploit string for this stage can be determined by examining a disassembled version of bytebomb using objdump. However, you may want to use gdb to step through the last few instructions of getbuf to make sure it is doing the right thing. Remember to be careful about byte ordering (i.e., endianness) as well as the placement of buf within the stack frame.

If you are having trouble starting, review the material and slides from class on buffer overflows.

Exploit 2: Glow

Exploit 2 involves injecting code as part of your exploit string. Here, the objective is to make the bytebomb glow via the glow function, defined as follows:

void glow(unsigned val) {
  vlevel = 2;       /* Part of validation protocol */
  if (val == bombcode) {
    printf("The bytebomb glows brightly!\n");
  } else {
    printf("The bytebomb flickers faintly. (0x%.8x)\n", val);

You will note that this is not quite as simple as hiss, since now you need the argument to appear as if you have passed a particular value. Hmm...I wonder what that id.txt file is for...

Some specific advice for this stage:

Exploit 3: Implode

Exploit 3 is another code injection attack to the implode function, but here the argument is a string and there is a helper function involved:

/* Compare string to hex represention of unsigned value */
int hexmatch(unsigned val, char* sval) {
  char cbuf[110];
  /* Make position of check string unpredictable */
  char *s = cbuf + random() % 100;
  sprintf(s, "%.8x", val);
  return strncmp(sval, s, 9) == 0;

void implode(char* sval) {
  vlevel = 3;       /* Part of validation protocol */
  if (hexmatch(bombcode, sval)) {
    printf("BOOM!!! The bytebomb implodes!\n");
  } else {
    printf("The bytebomb smokes slightly. (\"%s\")\n", sval);

While the idea is the same as in exploit 2, this one is quite a bit trickier. For one, you will need to include a string representation of your bomb ID in your exploit string. The string should consist of the eight hex digits (ordered from most to least significant) without a leading "0x". Remember that a string in C is terminated by a null character (i.e., byte value 0).

Most significant, however, is that hexmatch and strncmp will push data onto the stack when they are called, potentially overwriting portions of memory that held the buffer used by getbuf. As a result, you will need to be careful where you place the string representation of your bomb ID.

Part II: Return-Oriented Programming

Exploits 4 and 5 require you to attack the bigbytebomb executable. While the source code of the bigbytebomb is nearly identical to that of the bytebomb, code-injection attacks against the bigbytebomb are more difficult due to the usage of two techniques to thwart such attacks:

However, despite the above protections, the bigbytebomb is still vulnerable to return-oriented programming attacks (ROP). Recall that the key idea in ROP is to identify byte sequences within the existing program consisting of one or more instructions followed by the ret instruction. Such sections of code are called gadgets. By smashing the stack using gadget addresses (review the ROP slides) and possibly other data, you can construct a chain of gadgets that implements an exploit.

Locating series of bytes in the program that encode useful instructions (i.e., locating useful gadgets) is tricky. Luckily, we've located the source code to a series of functions within the bigbytebomb that might contain useful gadgets. This set of functions is called the gadget farm and is contained within the farm.c source file. For exploits 4 and 5, you will need to identify useful gadgets from within the gadget farm to perform attacks similar to those in exploits 2 and 3.

To help you in locating gadgets, we have compiled a handout detailing the encodings of useful instructions. More details are provided below.

Exploit 4: GlowBig

Exploit 4 requires you to repeat the Glow exploit against the bigbytebomb (note that if you try your working Exploit 2 against the bigbytebomb, it will fail due to the security measures mentioned above). You can construct your ROP exploit using gadgets only touching the first eight x86-64 registers (%rax-%rdi) and including only the following instruction types:

Note that the nop instruction (pronounced "no op", short for "no operation") is an instruction whose only effect is to increment the program counter (%rip) by 1. In effect, this instruction can be used to provide 'padding' in an instruction byte sequence.

Some specific advice for this stage:

Exploit 5: ImplodeBig

Before you tackle Exploit 5, consider what you have accomplished so far. In Exploits 2 and 3, you caused a program to execute machine code of your own design. If bytebomb had been a network server, you could have injected your own code into a remote machine. In Exploit 4, you circumvented two of the primary ways modern systems use to thwart buffer overflow attacks. Although you did not inject your own code, you were able to hijack the operation of the program using pieces of existing code.

Exploit 5 requires you to perform the Implode attack (Exploit 3) on the bigbytebomb. This is substantially more difficult than the Glow attack of the previous stage. In light of this difficulty, Exploit 5 is only worth 5% of your project grade. Think of it more like an extra credit problem for those of you looking for a challenge rather than a necessary component of the project in order to get a good score.

Some specific advice for this stage:


You are responsible for two tasks:

As usual, your final submission will consist of your committed files at the time of the due date.


You will be evaluated both on determining working exploits for each stage as well as clearly documenting your methods and insights in notebook.txt (even if an exploit does not work). Total points for each exploit are listed below:

Note that the points for the last exploit (ImplodeBig) do not accurately reflect its difficulty - think of the last exploit as something extra if you are looking for a challenge rather than something that you have to complete in order to get a good score.