The main special feature of reptyr is that it actually changes the controlling terminal of the target process. The “controlling terminal” is a concept maintained by UNIX operating systems that is independent of a process’s file descriptors. The controlling terminal governs details like where ^C gets delivered, and how applications are notified of changes in window size.

Processes are grouped into two levels of hierarchical groups: sessions, and process groups. Each group is named by an ID, which is the PID of the initial leader (either “session leader” or “process group leader”). Even if the leader exits, that number is still the ID for the group. Sessions are used for terminal management — Every process in a session has the same controlling terminal, and each terminal belongs to at most one session. Process groups are a sub-division within sessions, and are used primarily for job control within the shell. For a more in-depth explanation, see part 3 of my earlier series on termios.

If you check out tty_ioctl(4), you’ll find that Linux has an ioctl, TIOCSCTTY, that can be used to set the controlling terminal of a process, and you could be forgiven for thinking that all we need is to make the target call that ioctl, and we’re done.

However, if we read closer, we find that it has several restrictions. In particular:

The calling process must be a session leader and not have a controlling terminal already. If this terminal is already the controlling terminal of a different session group then the ioctl fails with EPERM […]

In the typical case, where I’m trying to attach a (say) mutt that you spawned from your shell, mutt won’t be a session leader — your shell will be the session leader, and mutt will be the process group leader for a process group containing only itself.

So, we need to make the target a session leader. Conveniently, there’s a system call for that: setsid(2).

However, reading that man page, we find a new caveat: setsid(2) fails with EPERM if

The process group ID of any process equals the PID of the calling process. Thus, in particular, setsid() fails if the calling process is already a process group leader.

The shell creates a new process group for every job you launch, and so our target mutt will be a process group leader, and unable to setsid(). The usual solution for programs that want to setsid is to fork(), so that the child is still in the parent’s session and process group, and then setsid() in the child. However, fork()ing our mutt and killing off the parent seems potentially disruptive, so let’s see if we can avoid that.

So, we’re going to need to change mutt‘s process group ID, so that there are no processes with process group IDs equal to its PID. Following some trusty SEE ALSO links, we get to setpgid(2). There’s a bunch of text in that man page, but the key bit is:

If setpgid() is used to move a process from one process group to another, both process groups must be part of the same session (see setsid(2) and credentials(7)). In this case, the pgid specifies an existing process group to be joined and the session ID of that group must match the session ID of the joining process.

We need to find a process group in the same session as mutt to move our mutt into, and then we’ll be able to setsid. We could try to find one — the shell is a plausible candidate, for instance — but there’s an alternate, more direct route: Create one.

While we have mutt captured with ptrace, we can make it fork(2) a dummy child, and start tracing that child, too. We’ll make the child setpgid to make it into its own process group, and then get mutt to setpgid itself into the child’s process group. mutt can then setsid, moving into a new session, and now, as a session leader, we can finally ioctl(TIOCSCTTY) on the new terminal, and we win.

It turns out I didn’t invent this technique — injcode and neercs work the same way. But I did discover it independently of them, and it was a fun little hunt through unix arcana.

You can grab the source, or read on for some more details.

There’s a shell script called screenify that’s been going around the internet for nigh on 10 years now that is supposed to use gdb to accomplish the same thing. There’s also a project called retty that tries to do the same thing, in C using ptrace() directly.

The difference between those programs and reptyr is that reptyr works much, much, better.

If you attach a less using screenify or retty, it will still take input from the old terminal. If you attach an ncurses program, and resize the window, the program probably won’t resize correctly. ^C and ^Z will still be processed on the old terminal — typing them in the new terminal won’t do anything useful.

reptyr fixes all of these problems and more, and is the only such tool I know of that does so. I’ve never seen a program that doesn’t behave noticeably incorrectly after attaching with retty or screenify, whereas with reptyr most programs I have tried work flawlessly.

How does it work?

reptyr works in the same basic way as screenify and retty — it attaches to the target process using the ptrace API, opens the new terminal, and dup2s it over the old file descriptors. It also copies the termios settings from the old terminal to the new terminal.

The main thing that reptyr does that no one else does is that it actually changes the controlling terminal of the process you are attaching. This is the detail that makes many things Just Work, including ^C and ^Z and window resizing.

Switching the target’s controlling terminal is not easy and involves a fair bit of trickery with ptrace and Linux’s terminal APIs. I will probably do another blog post some time about the dirty details of how I make this work, but for now you can check out attach.c if you really want to know.

reptyr still has a number of limitations — it doesn’t generally work, for example, if the target process has any children. I know how to fix most of these problems, though, so expect it to get better with time. Please let me know if you find it useful!

Appendix

(Edited to add:) Nothing is really new. A commenter on reddit pointed out that injcode and neercs both accomplish the same thing, even using the same trick to change the CTTY. Ah well, I had run writing it anyways, and apparently I wasn’t the only one who didn’t know about the existing alternatives. neercs is a full screen replacement, though, and I think that reptyr should be more robust than injcode — I use a different techique for ptrace-hijacking, for example — and so hopefully this tool still has a niche as a more robust standalone utility. Certainly, judging from the amount of enthusiasm I’ve seen for this tool, this still isn’t a problem that is solved to the average user’s satisfaction.

For my final entry on termios, I will be looking at job control in the shell (i.e. backgrounding and foreground jobs) and the very closely related topic of signal generation by termios, in response to INTR and friends.

Sessions and Process Groups

For the purposes of termios, processes are organized into two hierarchical groups, process groups and sessions. Every process belongs to exactly one process group and one session, and each process group is contained entirely within a session.

Process groups and sessions are both named by the process ID of the process to create the group. This process is known as the process group leader or session leader. A process creates a new session using setsid(2), or a new process group using setpgid(2).

On Linux, you can inspect the process group and session of a process using the stat field in /proc/$PID. The first several fields in that file are:

    pid (name) state ppid pgid sid …

or, in more words:

    [process id] ([name]) [state] [parent process id] [process group id] [session id] …

Sessions

Sessions are the fundamental group of terminal management. Every session may have an associated controlling terminal, which is treated specially. A process may open and talk to any number of terminals, but the special behaviors related to access control and job control only apply to a process’s controlling terminal. Each terminal may be the controlling terminal of at most one session. It follows that calling setsid(2) to create a new session causes a process to lose its previous controlling terminal. Acquiring a controlling terminal is OS-specific, but can usually be accomplished by opening a terminal device without the O_NOCTTY flag, while not already having a controlling terminal.

Generally, all your processes within a single login session, or within a single instance of your terminal emulator, are within the same session, with the pty allocated by your terminal emulator or ssh or whatever as their controlling terminal.

Process Groups

Process groups are the unit of control for signal generation by a terminal. A terminal never sends a signal to a specific process, but always to all processes within a process group.

Access to a terminal is also mediated in terms of process groups. In addition to having an associated session, every terminal has exactly one foreground process group. Every other process in that terminal’s session is a background process group.

The foreground process group is awarded special access to its controlling terminal. It is allowed unrestricted access to read from and write to the terminal, as well as to call various control functions, such as tcsetattr on it.

In addition, if any signal-generating character is read by a terminal, it generates the appropriate signal to the foreground process group.

Background process groups are restricted in their access to their controlling terminal. If any process in a background process group attempts to read from its controlling terminal, it will result in SIGTTIN being sent to its process group. Background processes may write to their controlling terminal, unless TOSTOP is set in c_lflag, in which case doing so will generate SIGTTOU to its process group. Calling terminal control functions such as tcsetattr is treated like a write operation with TOSTOP set (i.e. SIGTTOU is sent unless the process is blocking or ignoring it).

The foreground process group for a terminal may be set by the tcsetpgrp(3) function, which may be called by any process in the terminal’s session, but is treated in the same way as tcsetattr in the previous paragraph.

Job control

We’ve now got most of what we need to understand job control in your shell.

When processing each command line, the shell uses setpgid to place all of the programs executed by the line into the same process group, and then calls tcsetpgrp to make that job the foreground job, and does a waitpid to wait on that process.

Thus, when you run a shell pipeline (foo | bar | grep baz), all the programs in the pipeline are in the same process group, and in the foreground, which is why a ^C kills all of them.

When you ^Z a jobs, all the processes in the process group are stopped, and the shell’s waitpid returns, informing it of the status change. The shell restores itself to the foreground process group, and marks the job as backgrounded.

When you use bg to background a stopped job, the shell just uses killpg(2) to SIGCONT the group. If a job in the background tries to read from the terminal, the SIGTTIN stops it and the shell’s wait detects the state change and adjusts the job’s state appropriately.

If you launch a job in the background, the shell simply doesn’t tcsetpgrp it into the foreground, nor wait on it.

In conclusion

That’s probably all I want to say about termios. I could talk more about terminal emulation, ncurses, $TERM and friends, but it’s less interesting to me — I think I’m a kernel hacker at heart, and that stuff is just userspace programs talking to each other at this point. I hope you found this series interesting and/or informative, and I’m always happy to answer questions.

In this entry, we’ll look at the interfaces that are used to control the behavior of the “termios” box sitting between the master and slave pty. The behaviors I described last time are fine if you have a completely dumb program talking to the terminal, but if the program over on the right is using curses (like emacs or vim), or even just readline (like bash), it will want to disable or customize some of the behaviors.

The primary programmatic interface to termios is the struct termios and two functions:

   int tcgetattr(int fd, struct termios *termios_p);
   int tcsetattr(int fd, int optional_actions,
                 const struct termios *termios_p);

which retrieve and set the struct termios associated with a given terminal device. They are all documented in termios(3) (If you’re unfamiliar with the convention, that means document termios in section 3 of the unix man pages — man 3 termios on a command-line will get it for you).

So what’s inside struct termios? POSIX specifies that this structure contains at least the following fields:

       tcflag_t c_iflag;      /* input modes */
       tcflag_t c_oflag;      /* output modes */
       tcflag_t c_cflag;      /* control modes */
       tcflag_t c_lflag;      /* local modes */
       cc_t     c_cc[NCCS];   /* control chars */

Each “flag” field contains a number of flags (implemented as a bitmask) that can be individually enabled or disabled. c_iflag and c_oflag contain flags that affect the processing of input and output, respectively. c_cflag we will mostly ignore, as it contains settings that relate to the control of modems and serial lines that are mostly irrelevant these days. c_lflag is perhaps the most interesting of the flag values. It contains flags that control the broad-scale behavior of the tty. I’ll look at just a few of the interesting bits in each:

local modes

• ICANON – Perhaps the most important bit in c_lflag is the ICANON bit. Enabling it enables “canonical” mode — also known as “line editing” mode. When ICANON is set, the terminal buffers a line at a time, and enables line editing. Without ICANON, input is made available to programs immediately (this is also known as “cbreak” mode).

• ECHO in c_lflag controls whether input is immediately re-echoed as output. It is independent of ICANON, although they are often turned on and off together. When passwd prompts for your password, your terminal is in canonical mode, but ECHO is disabled.

• ISIG in c_lflag controls whether ^C and ^Z (and friends) generate signals or not. When unset, they are passed directly through as characters, without generating signals to the application.

input and output modes

There are also a few flags in c_iflag and c_oflag worth mentioning.

• IXON in c_iflag enables the “flow control” mediated by ^S and ^Q (by default). With IXON, once ^S has been received by the master pty, the slave will not accept any output (writes to it will hang) until ^Q is received by the master pty.

• IUTF8 in c_iflag is an interesting hack. In canonical mode, backspace needs to delete the previous character in the input buffer. In non-ASCII encodings, a single character may be several bytes long, but the terminal still only sees a byte stream, and has no explicit information about the encoding or character boundaries on either end. IUTF8 tells termios that the input stream is utf-8 encoded, which permits the correct handling of backspace. If IUTF8 is unset, and you enter a multibyte character and then press backspace, only the final byte will be deleted, leaving you with a corrupt utf-8 stream.

• OLCUC in c_oflag “Map[s] lowercase characters to uppercase on output.” Just in CASE YOU NEED YOUR TERMINAL TO LOOK MORE LIKE SHOUTING.

There are many more flags, controlling such details as newline translation and how character erase works. The full list is documented in termios(3).

c_cc

Next up is c_cc. This field sets the various control characters used to interact with the terminal. Characters like ^C and ^Z and delete that have special meanings to termios are not hard coded anywhere, but rather defined via the c_cc array.

c_cc is indexed by various constants for the various control characters, and the value at any index is the character that should have that effect. Some of the notable ones are:

• VINTR — Generate a SIGINT (^C by default).

• VSUSP — Generate a SIGTSTP (stop the program) (^Z by default).

• VERASE — Erase the previous character. This tends to be one of ^H and ^? (ASCII 0x7f) by default — if you’ve ever pressed “backspace” and been greeted by a ^H, your terminal and your struct termios disagree on the value of VERASE.

• VEOF — End of file. Sends the current line to the program without waiting for end-of-line, or, as the first character on the line, causes the next read call by the slave to return return end-of-file. (^D by default)

• VSTOP and VSTART — ^S and ^Q by default, stop and start output.

Setting any of these to NULL (0) disables that special control character. Many of the c_cc elements are only relevant when certain modes are active — VINTR and VSUSP, for instance, only matter if ISIG is enabled in c_lflag, and VSTOP and VSTART are ignored unless IXON is set.

(A brief note on the representation of control characters — The characters ^A through ^Z, pronounced “Ctrl-FOO”, are represented by the bytes with values 1 through 26. So when I say that c_cc[VINTR] is equal to ^C by default, that’s actually just the number 3 — your terminal took the keypresses and just translated them into the byte 3 on the wire.)

stty

While termios(3) is the standard programmatic interface to control termios, a much more convenient interface for experimentation is the stty program, which is just a thin wrapper around tcgetattr and tcsetattr designed to be usable from shell scripts or directly from the shell.

stty gets or sets options on a terminal device. By default, it operates on the one connected to its standard out, but you can pass it an arbitrary device using the -F option.

Without aguments, stty prints in what way its terminal’s settings differ from an internal set of “sane” defaults. stty -a causes it to print the value of every flag in the struct termios in a human-readable format.

You can toggle flags using stty flag to enable a flag, and stty -flag to disable it. So for instance, stty -isig will disable signal generation — run a program after doing this, and you’ll find yourself unable to ^C it. In general it uses the same names as the C constants, except in lowercase, but check the man page if in doubt.

stty can also change the value of the control characters in c_cc, using stty symbolic-name character. If you wanted ^G to be the interrupt character, instead of ^C, a simple stty intr ^G would suffice. You can spell 0 as undef to disable a given control character. So, if you hate flow control and want to totally disable it, you could try stty -ixon stop undef — disable IXON, and then also disable the VSTOP character for good measure. (You might still be foiled by screen or some other layer doing its own flow control, unfortunately).

stty‘s -F option can be great for peeking at what some other program is doing to its terminal. If you run tty in a shell, it will print the path to that shell’s terminal device (usually of the form /dev/pts/N, at least under Linux). Now, from a different shell, you can run stty -a -F /dev/pts/N to see how the first shell’s terminal is configured. You can then run programs in the first shell, and repeat the stty incant in shell two to see what settings are getting set. For example, if I run stty -F /dev/pts/10 right now (while I have a bash talking to a gnome-terminal via that pty), I see:

$ stty -F /dev/pts/10
speed 38400 baud; line = 0;
eol = M-^?; eol2 = M-^?; swtch = M-^?; lnext = <undef>; min = 1; time = 0;
-icrnl iutf8
-icanon -echo

So we can see that bash/readline has disabled CR→LF translation on input (icrnl), disabled canonical mode and echo, but turned on UTF-8 mode (because bash detected a utf-8 locale). Note that if I run stty directly in that shell, I see something slightly different:

$ stty
speed 38400 baud; line = 0;
eol = M-^?; eol2 = M-^?; swtch = M-^?;
iutf8

This is because bash maintains its own set of termios settings (for readline), and saves and restores settings around running programs, so that settings in place while running a program are different from those in place while you’re typing at the bash prompt.

In the next post, we’ll look at signal generation from ISIG and how it interacts with job control in your shell.

Addendum: ioctl(2)

This last section is a brief aside, which has very little to do with termios specifically, so you should feel free to skip it. But read on if you’re curious about some of the low-level details of how the APIs work.

If you’re familiar with man page conventions, you may have noticed that the termios functions are in man page section 3, which means that they’re provided by system libraries, and are not system calls. But at the same time, I told you last time that termios is implemented inside the kernel — so how are the libraries talking to the kernel, if not through syscalls?

The answer is a single odd little catch-all system call, known as ioctl. Historically, one of the “big ideas” of UNIX was that “everything is a file” — you could communicate with devices just like you could files, by opening files in /dev/. But a file on UNIX is also just a stream of bytes, without any OS-imposed structure. And for a device, you may often need to send out-of-band control data — e.g. to set the baud and parity bit settings on a serial port. And adding new system calls for every new device type would be untenable for a number of reasons.

So the answer was one new new system call, ioctl (pronounced as any of “I-O-cuddle”, “I-octal” or “I-O-C-T-L”). ioctl is prototyped as:

int ioctl(int fd, int request, ...);

It takes a file descriptor, a numeric “request” code, and an unspecified number of other arguments. ioctl looks up whatever device (or file system, network protcol, or whatever) is backing that file descriptor, and hands it the “request” and the arguments to do with as they will.

So any device that needs extra control channels can define some ioctl numbers and parameters and document them somewhere, and they become the interface to control that device. So, for instance (at least on Linux), an ioctl on a tty device with a “request” of TCGETS (defined in termios.h) takes a parameter that is a pointer to a struct termios, and copies the in-kernel settings for that tty to the provided struct. So somewhere in libc, tcgetattr(fd, p) is just defined to do an ioctl(fd, TCGETS, p). Similar ioctls are defined for tcsetattr and all the functions in termios(3). On Linux, at least, the morbidly curious can find out all the gory details in tty_ioctl(4).

The answer to all these questions involves the unix terminal emulation layer, and the termios interface for interacting with asynchronous communication devices (such as serial ports connected to terminals, or their virtual equivalents). This blog post will be part 1 of what will probably be a three-part introduction to tty emulation and termios in unix (with a focus on Linux, since that’s what I know and use) and to the APIs for interacting with it, as a springboard to understanding how these layers all work and where these behaviors come from. Along the way, you’ll learn a bit more about customizing the behavior of these layers, which can come in handy both as a user, and if you write terminal-based applications for unix.

The terminal device

The primary abstraction that governs any interaction with a terminal in Unix is a “terminal” device, or tty for short. In most cases these days, since you’re not interacting with a physical terminal, you’ll be dealing with a “pseudo-terminal” (“pty” for short — see pty(7)), which is a purely virtual construct. A pseudo-terminal, simply, is a pair of endpoints (implemented as character devices in /dev/) that provide a bidirectional communication channel. Whatever is written into one can be read out the other, and vice versa. These two ends are known as the “master” and “slave” ends. Unlike pipes or sockets, however, they don’t always pass data directly through. What makes terminal devices special is the layer that sits between the master and slave, which can filter, transform, and act upon the streams in both directions.

In standard usage, the “master” pty is connected to your terminal emulator (e.g. xterm), and the “slave” pty is connected to the program being run (e.g. your shell). “Input” flows from the user into the master pty and out the slave, and “output” flows from your program into the slave and then out the master. The basic diagram for what’s going on, then is:

This diagram is a simplification in a few ways, but it still captures most of what’s going on.

What happens in the middle?

That block I’ve labelled “termios” in the middle is responsible for essentially all of the behavior I listed in the first paragraph of this post. By default, it implements behavior including:

• Line buffering — when characters come in the left, it holds on to them until it receives a newline, at which point they are all emitted out the right at once.

• Echo — when a character comes in the left, in addition to being buffered or sent out the right, it is also sent back out the left. This is why you can see what you’re typing.

• Line editing — When the ERASE character (^?, ASCII 0x7f by default) comes in the left, assuming that there is anything in the input buffer, the last character in the input buffer is deleted, and the sequence "\b \b" is sent out the left. "\b" (ASCII 0x08) tells your terminal to move the cursor one column to the left, so this moves the cursor one column backwards, overwrites the last character with a blank space, and then moves the cursor back.

• Newline translation — if a newline ("\n", ASCII 0x0A) comes in the right, a carriage-return/line-feed combination (CRLF, "\r\n", ASCII 0x0D 0x0A) is sent out the left. While most programs on UNIX accept a bare "\n" as a newline, your terminal needs both – "\r" tells it to move the cursor to the beginning of the line, and "\n" tells it to move to the next line.

• Signal generation — if a INTR (^C, ASCII 0x3 by default) comes in the left, it is discarded and a SIGINT is sent to the program on the far right. Similarly, if a SUSP (^Z, ASCII 0x1A) comes in the left, it is discarded and a SIGTSTP is sent to the program on the far right. (SIGTSTP, by default, stops a process. It differs from SIGSTOP primarily in that it can be caught and handled by a program, whereas SIGSTOP‘s effect is unconditional)

  This is the source of at least two of the additional complications I mentioned to the basic diagram above:

  • The “termios” block does not strictly sit in the middle of the diagram. It also knows about the program(s) all the way over on the right, and can interact with them in (certain) ways that aren’t just sending characters to the slave pty.
  • There may be multiple programs connected to the slave pty. Which one(s) should be allowed to actually read from it? Which one(s) should SIGINT or SIGTSTP go to? The answers are fairly complicated, and I don’t claim to fully understand them, although I know some of the basic rules. I may write a future blog post once I understand them better.

There is even more that happens in that box, but these are the most noticeable ones, and selected to give you a feel of how the system works. Programs using the terminal can selectively enable or disable each of these features individually, as well as all the others I haven’t mentioned. In a future post, I will look at the interfaces we can use to control what processing does and does not happen, as well as how new pty pairs are created and used.

If this all seems somewhat arbitrary or weirdly constructed to you — well, it pretty much is. It makes sense, however, in the context of when this interface was first invented. When the tty layer was first coined, your “terminal” was a physical terminal — perhaps a VT100 or its ilk. C was the state of the art for programming languages, code size was expensive and libraries barely existed. Putting logic into the left side of the picture meant upgrading a physical piece of hardware. Putting logic into the right side of the picture meant putting it into every program you might want to talk to on the right side — libreadline and shared libraries didn’t really exist yet. And so, the logic went into the only place it could — the kernel, which could sit between a dumb terminal and a dumb program and mediate just enough logic to let programmer correct their typos as they worked. The kernel already had to mediate the connection, since terminals were physical devices, and there already had to be interfaces to control this layer from software, since software needed to be able to change such details as the baud rate and presence of parity or stop bits to communicate with the terminals. And so this layer that was already sitting between user programs and the console grew additional intelligence a bit at a time. And once we had terminal emulators and virtual terminals, the whole layer moved along with them, as legacy interfaces are wont to do.

Next time: The termios(3) APIs.