ReFormat: Automatic Reverse Engineering of Encrypted Messages


Zhi Wang , Xuxian Jiang, Weidong Cui, Xinyuan Wang , and Mike Grace

North Carolina State University {zhi wang,xjiang4,mcgrace}
Microsoft Research
George Mason University

Protocol reverse engineering is described as time-consuming and error-prone by the authors. Tools for automating this process existed before ReFormat, but are limited by their inability to handle encrypted messages. ReFormat is a tool for automating protocol reverse engineering that can analyze encrypted messages.

ReFormat relies on the observation that the two phases of normal protocol processing are significantly different in what kinds of instructions are executed. With this knowledge, it undergoes four steps:

1. Collect execution trace of message.

– Read system calls, track addresses and state of call stack

2.  Identify the two phases from this trace

– One phase requires much more arithmetic operations

3. Locate buffers containing plain-text

– These are passed between the two phases, they are identified by what buffers are still active between phases

4. Analyze these buffers

The tool was tested on and succeeded in analyzing messages sent over known and unknown protocols. Included in the asserted limitations of this implementation, though, is that not all programs will have two easily distinguishable phases. The idea presented may break down if the two phases cannot be easily located.


About Sebastian Chande

Sebastian is a fourth year student at Florida State University. He will be graduating in Fall 2012 with a B.S. in computer science, and plans on staying at FSU to pursue a master's degree. He joined the mobile lab at FSU in Spring 2012 to work on innovative projects that draw from multiple areas of computer science.