We all love strace, but have you ever wondered how it works? How does it interject itself between the kernel and the userspace program? This post will walk through a minimal implementation of strace in about 70 lines of C. It won’t be nearly as functional as the real thing, but in the process you’ll learn most of what you need to know about the core interfaces it uses.

On Linux (and probably some other UNIXes) strace uses a somewhat arcane interface known as ptrace, the process-tracing interface. ptrace allows one proccess to monitor the status of another process, and to inspect (or even manipulate) its internal state.

ptrace is a complex system call, taking a magic “request” first parameter, and doing completely different things depending on its value. Its general prototype looks like:

long ptrace(enum __ptrace_request request, pid_t pid, void *addr, void *data);

However, because different values of request use anywhere from zero to three of the remaining parameters, glibc prototypes it as a varargs function, allowing a a developer to only list as many parameters as a given call needs.

In order for one process to trace another, it attaches to that process, and temporarily becomes that process’s parent. When a process is ptraced, the tracer can ask for the child to stop whenever various events happen, such as the child making a system call. When this happens, the kernel will stop the child with SIGTRAP. Since the tracer is now the child’s parent, it can thus watch for this using the standard UNIX waitpid system call.

Our miniature strace will only support the strace COMMAND form of strace (as opposed to strace -p), and we’ll only print syscall numbers and return values — no decoding of names or arguments or anything. So a sample run might look like:

$ ./ministrace ls
…
syscall(6) = 0
syscall(54) = 0
syscall(54) = 0
syscall(5) = 3
syscall(221) = 1
syscall(220) = 272
syscall(220) = 0
syscall(6) = 0
syscall(197) = 0
syscall(192) = -1219706880
…

Not the most useful thing in the world, but it shows off the core tracing tools. So, let’s see the code:

#include <sys/ptrace.h>
#include <sys/reg.h>
#include <sys/wait.h>
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>

We start with the necessary headers. sys/ptrace.h defines ptrace and the __ptrace_request constants, and we’ll need sys/reg.h to help decode system calls. More about that later. Everything else you should probably recognize.

int do_child(int argc, char **argv);
int do_trace(pid_t child);

int main(int argc, char **argv) {
    if (argc < 2) {
        fprintf(stderr, "Usage: %s prog args\n", argv[0]);
        exit(1);
    }

    pid_t child = fork();
    if (child == 0) {
        return do_child(argc-1, argv+1);
    } else {
        return do_trace(child);
    }
}

We’ll start with the entry point. We check that we were passed a command, and then we fork() to create two processes — one to execute the program to be traced, and the other to trace it.

int do_child(int argc, char **argv) {
    char *args [argc+1];
    memcpy(args, argv, argc * sizeof(char*));
    args[argc] = NULL;

The child starts with some trivial marshalling of arguments, since execvp wants a NULL-terminated argument array.

    ptrace(PTRACE_TRACEME);
    kill(getpid(), SIGSTOP);
    return execvp(args[0], args);
}

Next, we just execute the provided argument list, but first, we need to start the tracing process, so that the parent can start tracing the newly-executed program from the very start.

If a child knows that it wants to be traced, it can make the PTRACE_TRACEME ptrace request, which starts tracing. In addition, it means that the next signal sent to this process wil stop it and notify the parent (via wait), so that the parent knows to start tracing. So, after doing a TRACEME, we SIGSTOP ourselves, so that the parent can continue our execution with the exec call.

(You might have noticed that strace COMMAND output always starts with an execve call. Now you should understand why — we’re actually going to start tracing immediately after the kill returns, so we see the execve call that starts the new program).

int wait_for_syscall(pid_t child);

int do_trace(pid_t child) {
    int status, syscall, retval;
    waitpid(child, &status, 0);

In the parent, meanwhile, we prototype a function we’ll need later, and start tracing. We immediately waitpid on the child, which will return once the child has sent itself the SIGSTOP above, and is ready to be traced.

    ptrace(PTRACE_SETOPTIONS, child, 0, PTRACE_O_TRACESYSGOOD);

I mentioned earliar that ptrace turns basically all events into a SIGTRAP on the child. This is inconvenient because it means that when you see the child has stopped due to SIGTRAP, there’s no good way to know which of several possible reasons it stopped for.

PTRACE_SETOPTIONS lets us set a number of options for how we want to trace the child. We use it here to set PTRACE_O_TRACESYSGOOD, which means that when the child stops for a syscall-related reason, we’ll actually see it stopped with signal number SIGTRAP | 0x80, so we can easily distinguish syscall stops from other stops. Since (for the purposes of this demo), we only care about syscalls, this is very convenient.

    while(1) {
        if (wait_for_syscall(child) != 0) break;

Now we enter the tracing loop. wait_for_syscall, defined below, will run the child until either entry to or exit from a system call. If it returns non-zero, the child has exited and we end the loop.

        syscall = ptrace(PTRACE_PEEKUSER, child, sizeof(long)*ORIG_EAX);
        fprintf(stderr, "syscall(%d) = ", syscall);

Otherwise, though, we know that the child is entering a system call, and so we need to decode the system call number (and potentially arguments, if this were a less toy example). The PTRACE_PEEKUSER ptrace request reads a word of data from the child’s “user area”, which is a logical area that holds all of its registers and other internal non-memory state. On i386, the syscall number lives in %eax. For various technical reasons, however, the kernel has already clobbered the child’s %eax at this point, but it saves the original value at a different offset, ORIG_EAX, which comes from `sys/regs.h’.

        if (wait_for_syscall(child) != 0) break;

Once we have the syscall number, we wait_for_syscall again, which should leave us stopped at the syscall return.

        retval = ptrace(PTRACE_PEEKUSER, child, sizeof(long)*EAX);
        fprintf(stderr, "%d\n", retval);

Return values on i386 are also passed in %eax, so this time we can read it directly and print the return value, and then return to the top of the loop to wait for the next syscall.

     }
    return 0;
}

And once the child exits, we just return.

int wait_for_syscall(pid_t child) {
    int status;
    while (1) {
        ptrace(PTRACE_SYSCALL, child, 0, 0);

wait_for_syscall is a simple helper function. We continue the child using PTRACE_SYSCALL, which allows a stopped child to continue executing until the next entry to or exit from a system call.

        waitpid(child, &status, 0);

We then waitpid to wait for something interesting to happen in the child.

        if (WIFSTOPPED(status) && WSTOPSIG(status) & 0x80)
            return 0;

Because of the PTRACE_O_SYSGOOD we set above, we can detect a syscall stop by checking if the child was stopped by a signal with the high bit set. If so, we return.

        if (WIFEXITED(status))
            return 1;
    }
}

If the child exited, we’re done here; Otherwise, it stopped for some reason we don’t care about (like an execve, for instance), and so we loop to start it again until it hits a syscall.

And that’s all there is to it. You can find the version I just posted on github if you want to download and try it out.

Making it more useful

While it works, the previous version isn’t exactly what I’d call particularly helpful. You have to decode the syscall numbers by hand, and you don’t get any syscall arguments.

It’s a little long to include in the post, but I’ve pushed a slightly more functional version to master in the same github repository. It includes a Python script to scan the Linux source to pick up syscall numbers and argument counts and types, and it knows how to decode string arguments, so that you can see filenames and read and write data.

Reading arguments is easy — on i386, they’re passed in registers, so it’s just another PTRACE_GETUSER for each argument. Perhaps the most interesting piece is the read_string function, which is used to read a NULL-terminated string from the child process. (Of course, NULL-terminated isn’t quite right — the real strace knows about the count arguments to read() and write(), for instance. But it’s close enough for a demo):

char *read_string(pid_t child, unsigned long addr) {

read_string takes a child to read from, and the address of the string it’s going to read.

    char *val = malloc(4096);
    int allocated = 4096, read;
    unsigned long tmp;

We need some variables. A buffer to copy the string into, counters of how much data we’ve copied and allocated, and a temporary variable for reading memory.

    while (1) {
        if (read + sizeof tmp > allocated) {
            allocated *= 2;
            val = realloc(val, allocated);
        }

We grow the buffer if necessary. We read data one word at a time.

        tmp = ptrace(PTRACE_PEEKDATA, child, addr + read);
        if(errno != 0) {
            val[read] = 0;
            break;
        }

PTRACE_PEEKDATA returns a work of data from the child at the specified offset. Because it uses its return for the value, we need to check errno to tell if it failed. If it did (perhaps because the child passed an invalid pointer), we just return the string we’ve got so far, making sure to add our own NULL at the end.

        memcpy(val + read, &tmp, sizeof tmp);
        if (memchr(&tmp, 0, sizeof tmp) != NULL)
            break;
        read += sizeof tmp;

Then it’s a simple matter of appending the data we read, and breaking out if we found a terminating NULL, or else looping to read another word.

    }
    return val;
}

The Problem

We want to write some toplevel declarations that look like:

#define SUITE_NAME example
BEGIN_SUITE("Example test suite");

#define TEST_CASE core
BEGIN_TEST_CASE("Core tests");
…

and so on, and somehow translate them into code that does the equivalent of:

Suite *s = suite_create ("Example test suite");
TCase *tc_core = tcase_create ("Core tests");
…

First steps

Instead of having our macros somehow lay down code directly, we’ll use them to define some global data structures, which we will then loop across at runtime, and make the appropriate Check library calls. We’ll define some structs that we’re going to use for this purpose:

typedef void (*test_func)(int);
typedef void (*test_hook)(void);

struct test_case {
    test_func  *funcs, *end_funcs;
    const char *test_name;
    test_hook *setup_hooks, *end_setup_hooks;
    test_hook *teardown_hooks, *end_teardown_hooks;
};

struct test_suite {
    struct test_case *cases, *end_cases;
    const char *suite_name;
};

Since tests and test fixtures are code, we’re going to need to keep function pointers to them. We use typedefs to keep thos readable, and then we define structs to represent a test suite, and a test case. We’re going to represent arrays (of test cases, fixture hooks, or test functions) using a pair of pointers, to the start and end of the array. There’s a specific reason for this representation, which I’ll get to shortly.

ELF Sections

We can easily define global instances of these structs, but the problem is how to get the pointers to the child arrays correctly. These macros are going to be interspersed with other code, so we can’t just use a literal array, like

struct test_suite s = {
 .suite_name = "Example test suite",
 .cases = {
   …
 }
};

because that “…” is going to have arbitrary code shoved in it.

The solution is to take advantage of a feature of the ELF object file format, and some GCC extensions that let us take advantage of it.

ELF object files can contain an arbitary set of named sections, each of which can contain code or data (or both, but that’s not typical). GCC allows us to specify that a given variable should live in a given section using __attribute__((section("section_name")). So, by placing all our struct test_cases for a given test_suite into a separate section, we can cause them to get placed contiguously, which means we just need to get pointers to the start and end. That we can accomplish by declaring a zero-length array in the appropriate section, which (being of size zero) won’t generate any data, but will have the appropriate address.

So, the code we’re going to generate for the above will look something like:

struct test_case _begin_example_cases[];
struct test_case _end_example_cases[];
struct test_suite _suite_example = {
  .cases = _begin_example_cases,
  .end_cases = _end_example_cases,
  .suite_name = "Example test suite"
};
struct test_case _begin_example_cases[0]
  __attribute__((section(".data.example.cases")));

struct test_case _test_example_core
  __attribute__((section(".data.example.cases"))) = {
  .test_name = "Core tests",
  …
};

…

struct test_case _end_example_cases[0]
  __attribute__((section(".data.example.cases")));

Preprocessor tricks

So far, I haven’t actually talked about the preprocessor at all. I’ve just shown the code we want our macros to generate. So now let’s get to the business of actually generating this code. I’ll walk through the definition of `BEGIN_SUITE’, which expands to the code above. The other definitions are all essentially analagous.

My first version of this library didn’t use the SUITE_NAME and TEST_CASE macros, instead requiring those arguments to be passed to every macro. I like the decreased repetition that this method gives, but we’ll start with the explicit version, since it requires less trickery to understand.

From the above example, we can see that BEGIN_SUITE(example, "Example")' is given theexample’ token, and needs to construct both symbols (_suite_example) and strings (".data.example.cases") containing that symbol. To do the first, we need to the processor operator ##, which concatenates two tokens into one. For example, given

#define PASTE(x,y) x##_##y

`PASTE(foo, bar)’ is equivalent to just having written “foo_bar” literally in your code.

Secondly, to construct strings, we need the #x' preprocessor operator. This operator, when applied to an argument to a macro, returns a string literal containing the tokens passed to the macro. A common usage of this is for anassert` macro, that can print the failed expression:

#define assert(x) if (!(x)) { die("Assertion failed: %s\n", #x); }

And finally, we’re going to need the implicit concatenation feature of the C language, which lets you type "foo" "bar" with the same meaning as “foobar”.

Putting it all together, and we get:

#define BEGIN_SUITE(sym, name)                                          \
    struct test_case _begin_ ## sym ## _cases[];                        \
    struct test_case _end_ ## sym ## _cases[];                          \
    struct test_suite _suite_ ## sym = {                                \
        .cases       = (struct test_case*)_begin_ ## sym ## _cases,     \
        .end_cases   = (struct test_case*)_end_ ## sym ## _cases,       \
        .suite_name  = name,                                            \
    };                                                                  \
    struct test_case _begin_ ## sym ## _cases[0]                        \
    __attribute__((section(".data." #sym ".cases")))

Two last things to note: we deliberately leave off a trailing semicolon, requiring the user to provide one, and we need to end every line with a \, to include the entire body in the macro definition.

Another layer of indirection

The one final bit is to get rid of that explicit sym argument everywhere, in favor of the SUITE_NAME define. Our first try might look something like:

#define BEGIN_SUITE(name)                                               \
    struct test_case _begin_ ## SUITE_NAME ## _cases[];                 \
    …

You’ll quickly realize, however, that since ## is applied at the same time as function-like macro expansion, this will always give you the symbol _begin_SUITE_NAME_cases. The solution is to force the preprocessor to perform multiple phases of macro expansion, to cause the ## to be applied only after SUITE_NAME is expanded. In this case, we require three layers of macros to get the order right:

#define BEGIN_SUITE(name) _BEGIN_SUITE(SUITE_NAME, name)
#define _BEGIN_SUITE(sym, name) __BEGIN_SUITE(sym, name)
#define __BEGIN_SUITE(sym, name)                                        \
    struct test_case _begin_ ## sym ## _cases[];                        \

This causes the following sequence of expansions for BEGIN_SUITE("Example");:

BEGIN_SUITE("Example");                     // (a)
_BEGIN_SUITE(SUITE_NAME, "Example");        // (b)
__BEGIN_SUITE(sym, "Example");              // (c)

The key here is that when expanding a function-like macro, the preprocessor performs the following steps, in order:

1. Apply any # or ## operators.
2. Macro-expand any arguments to the call
3. Perform the substitution
4. Macro-expand the result of the substitution

In going from (a) to (b), only step (3) applies. When going from (b) to (c), rule (2) means that SUITE_NAME gets expanded first, leaving (c). Rule (1) means that the _BEGIN_SUITE macro is necessary, since a call to __BEGIN_SUITE(SUITE_NAME, "Example") will get the # and ## applied to SUITE_NAME before it can be expanded.

Conclusions

So, that’s basically all there is to the Check-Plus implementation. If you’re comfortable with that, you can try your hand at unravelling the implementation of the Linux kernel’s tracing macros, next (sample usage)!

Here’s my advice for keeping a lab notebook as a computer scientist:

Use a text file

There’s no need to use anything fancier. And if you’re writing code all day, you can just keep your log file open in your editor of choice, so that you don’t need to leave your hacking environment to make a note of something. I’m an emacs junkie and use org-mode to keep all my notes, but use whatever you’re comfortable with.

You could use a physical notebook, but unless you expect to want documentation of what you were working on for some legal reason, I don’t see the point. And most programmers I know hate writing things out by hand.

Treat your file as append-only

One of the downsides of just using a text file is that you need to enforce some discipline on yourself. It’s tempting to go back and fix up things that have changed, or to keep editing a desgin description as it evolved. But creating a perfect finished piece of documentation is a different task. The point here is to have a log of everything you do, not a single finished document. You can work on one side-by-side with the (dated or even timestampped) append-only log.

Keep automatic backups.

Do this any way you want. I keep my log files in git, and have a cron job that commits and pushes to a server every five minutes. This provides at least two benefits. First, it applies an automatic timestamp (to within five minutes) on anything I type, and second, it serves to make sure that a recent version is available on my server, in case I switch machines and want to keep working on a project.

Train yourself to make regular entries.

Keeping this nice text file, regularly backed up, is useless if you don’t record relevant things into it. Getting yourself into the habit of writing enough down is tricky. One strategy I’ve found helpful is to write down a list of “triggers” for situations that you promise to always record. With an explict list of these that you’ve decided to always record, you can get good at making a note whenever you perform one of these tasks, which gets you into the habit of writing things down. Here’s a partial list of some of the sorts of things I make a point of writing down:

• Whenever I take a measurement. That is, any time I run time, or look at top to figure out how much memory a program under development is using, or similar, I’ll write down the number and the circumstances under which it was recorded. I’ve found that I almost always would like to have a better record of such things later.
• Whenever I do a non-trivial search and find a result. This can be anything from googling for a solution to some problem where it takes me several tries to find the right keywords, or grepping a large codebase searching for the function that performs some task, or tracking down some API function in documentation. If I wanted it once, I’ll often want it later, and if it was hard to find, I’ll feel stupid if I didn’t write it down.
• Whenever I make an explicit design decision. If I’m thinking about how to design some function, system, process, or whatever, I’ll stop and take a moment to write down the options I was considering, and why I made the choice I did. This ensures I can go back and remember why I built a system the way it is.
• Whenever I wish I had written something down last time. This should be obvious, but if you find yourself thinking “I know I did this before; Why didn’t I take notes?”, you should always make a point of writing it down this time, no matter what excuse you come up with for why you really won’t need that information a third time.

These categories are deliberately broad, but at the same time well-defined. It’s fairly easy to train yourself to notice whenever any of these points triggers, and remember to stop and write something down. And that’s the first step to getting in the habit of writing down everything that you’ll later wish you had written down.

Most of the projects I’ve been working on today have fairly strict code review policies. My work requires code review on most of our code, and as we bring on an army of interns for the summer, I’ve been responsible for reviewing lots of code. Additionally, about five months ago BarnOwl, the console-based IM client I develop, adopted an official pre-commit review policy. And I have a confession to make: I hate mandatory code review.

I should be clear that I think I still believe in code review as a useful tool for developers working together on a project. A reviewer will flag as bad style or inefficient or ugly things that you let slide working for yourself. A reviewer comes at code without the assumptions of how it’s supposed to work that you made, and can often spot errors you made because mixed up a mental model of your intent with the code you actually wrote.

In addition, code review is a great way to ensure that at least two people are familiar with each piece of your infrastructure. There is often a natural tendency for different developers to take ownership of specific pieces of code or infrastructure, and be the only ones who touch or maintain it. But then it breaks while they’re on vacation, and everyone else is left trying to figure out how this code was ever supposed to work. A mandatory code review policy is often a great way to mitigate that class of problem.

But, theoretical and observed benefits of code review aside, speaking personally, as both a developer and as a reviewer, I’ve been finding it a really frustrating process.

Software engineers, as a rule, suck at writing things down. Part of this is training – unlike chemists and biologists who are trailed to obsessively document everything they do in their lab notebooks, computer scientists are taught to document the end results of their work, but aren’t, in general, taught to take notes as they go, and document the steps they take in building a system. 6.005, MIT’s new introductory software engineering class, attempted to require its students to keep lab notebooks for a few semesters, and was met with near-universal complaints and ridicule from the students (“Lab notebooks? For a software engineering class? What the hell?”). (To be fair, I suspect they did a horrible job of it, but I’m not sure that students would have been any less confused at the idea).

Part of the reason is probably also the nature of software that makes it very easy to record certain things as part of our tools, and that makes experiments cheaply reproducible. Version control lets us document the process of developing a piece of code, so it feels superfluous to be taking additional notes on the side. Computers are mostly deterministic, and cycles are cheap, so why bother meticuously recording the results of a test run somewhere when you can just run it again later, any time you want? Computers feel much neater and simpler than messy bio or chem labs, and software is much simpler than complicated biology experiments or chemical syntheses, and so no one feels the need to be nearly as careful.

However, I am increasingly of the opinion that most software engineers’ total inability to work in a lab-notebook style, where you meticulously document your work, is unfortunate and often seriously detrimental to their work. While it’s true that things like commit logs do a good job of documenting certain processes, here are some types of situations where I’ve found working with meticulous notes at every step can be invaluable:

Debugging subtle problems

Debugging is very much a problem of gathering data and making and testing hypotheses. For subtle bugs in large programs, the amount of state you need to keep track of can rapidly get out of hand. And good luck when a bug is tricky enough that your debugging gets spread across multiple days, or even across a lunch break.

If you’ve ever found yourself wondering “Wait – did I see the bug after I made $CHANGE to my code or test environment?”, you should have been writing more things down.

This is especially important for non-deterministic bugs, such as rare race conditions. If it takes you half a hour on average to reproduce a bug, and you are experimenting with a dozen different variables in your test environment that might affect the bug, you can’t afford to forget the results of a single test, or to forget a single detail of what you did to test. This is the point at which you should be writing down every single command you type in any relevant prompts, and every single code change (or, since we have technology, obsessively saving the output of `history`, making commits to test branches, and recording the correlation between them).

Profiling and optimization

This is a process similar in many ways to debugging, but even more data-driven. When you’re done with a session of optimizing a piece of code or a system, if you can’t show documented evidence of exactly how much faster you’ve made it, where that speedup came from, and all the things you tried and how much they helped, perhaps you should be writing more things down. And if you (or ideally, even someone else) can’t go back and reproduce the experiments you did, with approximately the same results, you probably haven’t been documenting your work well enough.

Even if you’re happy with the performance improvements you’ve made, you may need to come back and wring even more performance out of the system, and it’d be nice not to start from scratch. Or maybe future testing will reveal that or more of your optimizations was invalid, and you need to go back and consider alternate options.

This is critically important when you’re optimizing not just a piece of code, but some kind of system with lots of configuration and setup, that you’ll later have to duplicate somewhere else, instead of just checking the result into source control.

Understanding a new project’s code or documentation

Whenever I’m first diving into a large code-base or first playing with a large new API, I find it invaluable to take notes as I go about what I look for and where I find it. I’ll often need to look up half a dozen different API calls or pieces of code to understand something, often too many to keep in my head as I go and dive through more and more pieces of code or docs.

And when the documentation is ambiguous, I’ll often drop into a REPL or build test programs to make various calls and understand what happens. Again, after more than two or three of these, it’s vital that I’ve been writing down my findings.

This is one example where a chronological style documenting exactly in what order I found things is less critical, but that detailed notes as I go are still vital.

Designing things

Whenever you’re designing something – be it an API, a protocol, an interface, some kind of system, or something else – it’s worth taking notes on the process you took to get to your final decisions, and the choices you considered and rejected, and why.

You’ll presumably end up a producing a piece of code or a design document that indicates what you ended up deciding, but understanding why you made the decisions you made is often important to understanding how your system is supposed to work, and how to best use or extend it in the future. Hopefully, when you’re done, you’d do this writeup in brief somewhere anyways, but the best way to make sure you don’t forget is to take good notes as your thought process happens.

And nothing in software is ever complete. If you have to revise the design for some reason, because someone points out problems or new requirements come up, you’ll probably want to remember the other possibilities you came up with – maybe one of them is now more right.

So, if you’re a software engineer, I strongly encourage you to try to get better at writing things down. In a future post, I’ll hopefully write up the techniques I’ve started using to take notes as I code, debug, and design, but in the meanwhile, I encourage you to just grab a text editor or a physical lab notebook, whichever is more comfortable for you, and start taking more notes on what you’re doing.