Unless we’re using a richly-typed language like Haskell, where we can use newtype, the best solutions tend to just rely on convention. The much-maligned Hungarian Notiation evolved in part to try to combat this sort of problem — If you decide on a convention that variables representing physical addresses start with pa and virtual addresses start with va, then anyone who encounters a uintptr_t pa_bootstack; can immediately decode the intent.

It turns out, though, that we can get something very much like newtype in familiar old C. Suppose we’re writing some of the paging code for a toy x86 architecture. We’re going to be passing around a lot of physical and virtual addresses, as well as indexes of pages in RAM, and it’s going to be easy to confuse them all. The traditional solution is to use some typedefs, and then promise to be very careful to mix them up:

typedef uint32_t physaddr_t;
typedef uint32_t virtaddr_t;
typedef uint32_t ppn_t;

We have to promise not to mess up, though — the compiler isn’t going to notice if I pass a ppn_t to a function that wanted a physaddr_t.

This example was inspired by JOS, a toy operating system used by MIT’s Operating Systems Engineering class. JOS remaps all of physical memory starting at a specific virtual memory address (KERNBASE), and so provides the following macros:

/* This macro takes a kernel virtual address -- an address that points above
 * KERNBASE, where the machine's maximum 256MB of physical memory is mapped --
 * and returns the corresponding physical address.  It panics if you pass it a
 * non-kernel virtual address.
 */
#define PADDR(kva)                                          \
({                                                          \
        physaddr_t __m_kva = (physaddr_t) (kva);            \
        if (__m_kva < KERNBASE)                                     \
                panic("PADDR called with invalid kva %08lx", __m_kva);\
        __m_kva - KERNBASE;                                 \
})

/* This macro takes a physical address and returns the corresponding kernel
 * virtual address.  It panics if you pass an invalid physical address. */
#define KADDR(pa)                                           \
({                                                          \
        physaddr_t __m_pa = (pa);                           \
        uint32_t __m_ppn = PPN(__m_pa);                             \
        if (__m_ppn >= npage)                                       \
                panic("KADDR called with invalid pa %08lx", __m_pa);\
        (void*) (__m_pa + KERNBASE);                                \
})

Because the typedefs are unchecked by the compiler, though, it is a common mistake to use a physical address where a virtual address is meant, and nothing will catch it until your kernel triple-faults, and a long, painful debugging session ensues.

Inspired by Haskell’s newtype, though, it turns out we can get the compiler to check it for us, with a little more work, by using a singleton struct instead of a typedef:

typedef struct { uint32_t val; } physaddr_t;

If we wanted to be overly cute, we could even use a macro to mimic Haskell’s newtype:

#define NEWTYPE(tag, repr)                  \
    typedef struct { repr val; } tag;       \
    static inline tag make_##tag(repr v) {  \
            return (tag){.val = v};         \
    }                                       \
    static inline repr tag##_val(tag v) {   \
            return v.val;                   \
    }

NEWTYPE(physaddr, uint32_t);
NEWTYPE(virtaddr, uint32_t);
NEWTYPE(ppn,  uint32_t);

Given those definitions, PADDR and KADDR become:

#define PADDR(kva)                                          \
({                                                          \
    if (virtaddr_val(kva) < KERNBASE)                       \
            panic("PADDR called with invalid kva %08lx", virtaddr_val(kva)); \
    make_physaddr(virtaddr_val(kva) - KERNBASE);            \
})

#define KADDR(pa)                                           \
({                                                          \
    uint32_t __m_ppn = physaddr_val(pa) >> PTXSHIFT;        \
    if (__m_ppn >= npage)                                   \
            panic("KADDR called with invalid pa %08lx", physaddr_val(pa)); \
    make_virtaddr(physaddr_val(pa) + KERNBASE);             \
})

We have to use some accessor and constructor functions, but in exchange, we get strong type-checking: If you pass PADDR a physical address (or anything other than a virtual address), the compiler will catch it.

The wrapping and unwrapping is slightly annoying, but we can for the most part avoid having to do it everywhere, by pushing the wrapping and unwrapping down into some utility functions. For instance, a relatively common operation at this point in JOS is creating a page-table entry, given a physical address. If you want to construct the PTE by hand, you need to use physaddr_val every time. But a better plan is a simple utility function:

 static inline pte_t make_pte(physaddr addr, int perms) {
     return physaddr_val(addr) | perms;
 }

In addition to losing the need to unwrap the physaddr everywhere, we gain a measure of clarity and typechecking — if you remember to use make_pte, you’ll never accidentally try to insert a virtual address into a page table.

We can add similar functions for converting between types, as well a a struct Page, used to track metadata for a physical page. As an experiment, I went and reimplemented JOS’s memory management primitives using these definitions, and only needed to use FOO_val or make_FOO a very few times outside of the header files that defined KADDR and friends.

Performance

While the typechecking is nice, any C programmer implementing a memory-management system is probably going to want to know: How much does it cost me? You’re creating and unpacking these singleton structs everywhere – does that have a cost?

The answer, though, in almost all cases is “no” — A half-decent compiler will optimize the resulting code to be completely identical to the code without the structs, in almost all cases.

Also, the in-memory representation of the struct is going to be exactly the same as the bare value — it’s even guaranteed to have the same alignment and padding constraints, so if you need to embed a physaddr inside another struct, or into an array, the representation is identical to the physaddr_t typedef.

On i386, parameters are passed on the stack, so that means that passing the struct is identical to passing the uint32_t. On amd64, as described last week, small structures are passed in registers, and so, again, the calling convention is identical.

Unfortunately, the i386 ABI specifies that returned structs always go on the stack (while integers go in %eax), so you do pay slightly if you want to return one of these typedef’d objects. amd64 will also break it down into a register, though, so on a 64-bit machine it’s again identical.

If you’re worried, though, you can always use the preprocessor to make the checks vanish for a production build:

#ifdef NDEBUG
#define NEWTYPE(tag, repr)                  \
    typedef repr tag;                       \
    static inline tag make_##tag(repr v) {  \
            return v;                       \
    }                                       \
    static inline repr tag##_val(tag v) {   \
            return v;                       \
    }
#else
/* Same definition as above */
#endif

Because the types have identical representations, you can safely serialize your structs and exchange them between code compiled with either version. On amd64, you can probably even call between compilation units defined either way.

The next time you’re writing some subtle C code that has to deal with multiple types with the same representation, I encourage you to consider using this trick.

Addendum

I didn’t invent this trick, although as far as I know the NEWTYPE macro is my own invention (Edited to add: A commenter points out that I’m not the first to use the newtype name in C, although I think I prefer my implementation).

. I learned this trick from the Linux kernel, which uses it for a very similar application — distinguishing entries in different levels of the x86 page tables. page.h on amd64 includes following definitions [Taken from an old version, but the current version has equivalent ones):

/*
 * These are used to make use of C type-checking..
 */
typedef struct { unsigned long pte; } pte_t;
typedef struct { unsigned long pmd; } pmd_t;
typedef struct { unsigned long pud; } pud_t;
typedef struct { unsigned long pgd; } pgd_t;

I claimed above that the struct and the bare type will have the same alignment and padding. I don’t believe this is guaranteed by C99, but the SysV amd64 and i386 ABI specifications both require:

Structures and unions assume the alignment of their most strictly aligned component. Each member is assigned to the lowest available offset with the appropriate alignment. The size of any object is always a multiple of the object’s alignment.

(text quoted from the amd64 document, but the i386 one is almost identical).

And C99 requires (§6.7.2.1 para 13):

… A pointer to a structure object, suitably converted, points to its initial member (or if that member is a bit-field, then to the unit in which it resides), and vice versa. There may be unnamed padding within a structure object, but not at its beginning.

I believe these requirements, taken together, should be enough to ensure that the struct and the bare type will have the same representation.

On the one hand, ORMs are vastly more pleasant to work with than code that constructs raw SQL, even, generally, from a tool that gives you an object model to construct SQL, instead of requiring (Cthulhu help us all) string concatenation or interpolation.

On the other hand, there tends to be an enormous mismatch between the objects the ORM is modelling on one end, and the relational database on the back end. Basically, the object model encourages dealing with one object at a time, whereas the relational database wants you to push your queries, JOINs, and even your entire transactions back into it, so that it can better optimize them for you. And so, you tend to have a choice between writing shockingly inefficient code, and jumping through weird hoops and strange syntax to give your ORM the tools to generate efficient queries.

The most fundamental problem, perhaps, is the question of how much data the ORM should request from the database. When I ask for a collection of objects (using some syntax to generate the SQL WHERE clause automatically), the ORM has to decide:

• Which fields in the object should the ORM SELECT from the database, and
• Which associated models (with foreign keys to or from the object in question) the ORM should JOIN in and pull out at the same time.

(You can think of the second as a more involved version of the first, but involves more complexity, and is interesting for being potentially recursive).

SELECTing too much data in either case makes the database do too much work, touch too many blocks on disk, read too many indexes, and so on, and makes your queries slow. SELECTing not enough data means your ORM needs to go back and make additional queries when you ask for fields or models that were not pulled in initially, killing your performance even more.

I’d like to muse about two possible solutions, neither of which I’m aware of implementations of, although to be honest I haven’t looked too hard. I’d love it if anyone could point me at work on either sort.

The static solution

Given a language with a sufficiently awesome type system (Haskell comes to mind :)), you could imagine encoding information about which fields and models are or are not SELECTed into the type of a record object, making it statically an error to write a query that under-selects. Given type inference, we can probably even go on to statically infer (probably with a bit of programmer help in the form of type annotations) which fields and models we need to SELECT, based precisely on which fields of the result get accessed.

This is the kind of solution that occurs immediately to a certain type of person (of which I know a lot) when they think about this type of problem enough. It’s fun to think about and there are probably some papers in it (if no one’s beaten you to it), but I suspect it’s the kind of thing that’s unlikely to gain much traction among mainstream developers, if only because it would require you write in Haskell or equivalent.

The dynamic solution

Another solution occurred to me recently, which I don’t think I’ve heard people talking about. I like it because I’m pretty sure it could be grafted onto an existing ORM with little or no API change, which means there’s potential for incrementel adoption and improvement.

The key insight here is that, while the decision of what to SELECT is critical for performance, it’s (mostly)¹ irrelevant for correctness whether you have to make additional queries later, or pull in too much data. A good ORM, if it notices you ask for a field it doesn’t have cached, will just go ahead and make the SELECT for you.

This opens the door for heuristic solutions. A good ORM will already have hooks for the programmer to specify which fields or models to include or exclude — Django, to pick a popular example, has the select_related and defer methods on its QuerySets.

So, my proposal is this: Let’s develop tools to profile an application, and automatically generate those annotations in a profile-driven manner. Since all accesses to your data are through the ORM, we can instrument the ORM to determine which fields are selected after a query, and remember that information for next time it sees the same query.

In a webapp environment, the possibilities for optimization get potentially even more exciting. A webapp framework like Django knows an awful lot of information at any given moment, like what user is logged into your app, and what URL is currently being generated. We could imagine associating that data with the profile results, so that the ORM could automatically discover relations between the application structure and the ORM queries. It could “learn”, for example, that certain users are administrators, and so will access more fields on certain models than other users. It could learn that even though /places/list/ and /places/details/ pull up the exact same set of models (maybe even through the same helper function!), the former needs only their names and IDs, while the latter needs to pull in full details, as well as rows from the associated reviews table.

All without a single programmer annotation. Of course, in cases where the behavior is too complex for the tool’s heuristics to pick up, or where the programmer absolutely needs predictable behavior or needs to micro-optimize, you could turn off the heuristics and fall back the old behavior (with the same old annotations available to the programmer).

Is anyone aware of any work in this space? It sounds like a potentially extremely exciting project to me, but I probably don’t have the time to attempt to throw together an implementation.

1.Ignoring, for example, race conditions where you need to select a model and related models atomically, and don’t want or can’t afford a transaction around the entire lifetime of the objects in the app code.

My main complaint about Check is that (unsurprisingly for a framework written in C), it’s not very declarative. After you define all your tests as separate functions, you need to write code to manually collect them into “test cases”, which you then collect into “test suites”, which you can then run. While this is an understandable interface for a C library to provide, as a user I found that it rapidly annoyed me, and so I wrote Check Plus.

Check Plus is a simple library I wrote that uses some preprocessor tricks and GNU C extensions to provide a declarative EDSL within C for declaring and collecting your test cases. Check-Plus is a single self-contained header file distributed under the LGPL (like Check), so it should be easy to pull into your project.

Example

Check-Plus lets you declare your Check suites and test cases in a declarative manner. A simple example, shipped with Check-Plus, would look like:

#include <string.h>
#include <stdlib.h>

#include "check-plus.h"

#define SUITE_NAME example
BEGIN_SUITE("Example test suite");

#define TEST_CASE core
BEGIN_TEST_CASE("Core tests");

TEST(strcmp)
{
    fail_unless(strcmp("hello", "HELLO") != 0);
    fail_unless(strcasecmp("hello", "HELLO") == 0);
}
END_TEST

TEST(strcat)
{
    char buf[1024] = {0};
    strcat(buf, "Hello, ");
    strcat(buf, "World.");
    fail_unless(strcmp(buf, "Hello, World.") == 0);
}
END_TEST

END_TEST_CASE;
#undef TEST_CASE
END_SUITE;
#undef SUITE_NAME

Having done that, you can extract the declared test suite using construct_test_suite(example), and run it using the standard Check test runner tools.

You can, of course, define multiple test cases within a single suite, and you can even define test fixtures using

DEFINE_FIXTURE(setup, teardown);

within the BEGIN_TEST_CASE … END_TEST_CASE block.

The equivalent code, without Check-Plus, would look similar, but instead of the BEGIN_ and END_ blocks to define the suite and test case, you’d have to write something like:

Suite *
example_suite (void)
{
  Suite *s = suite_create ("Example test suite");

  /* Core test case */
  TCase *tc_core = tcase_create ("Core tests");
  tcase_add_test (tc_core, test_strcmp);
  tcase_add_test (tc_core, test_strcat);
  suite_add_tcase (s, tc_core);

  return s;
}

It’s not a huge difference, but it’s big enough that I really appreciate it once the test suite gets large. For more documentation on Check, check out its manual.

Let me know if you find this useful for anything. Next week, I’ll do a writeup on the tricks behind the implementation.

Most of the projects I’ve been working on today have fairly strict code review policies. My work requires code review on most of our code, and as we bring on an army of interns for the summer, I’ve been responsible for reviewing lots of code. Additionally, about five months ago BarnOwl, the console-based IM client I develop, adopted an official pre-commit review policy. And I have a confession to make: I hate mandatory code review.

I should be clear that I think I still believe in code review as a useful tool for developers working together on a project. A reviewer will flag as bad style or inefficient or ugly things that you let slide working for yourself. A reviewer comes at code without the assumptions of how it’s supposed to work that you made, and can often spot errors you made because mixed up a mental model of your intent with the code you actually wrote.

In addition, code review is a great way to ensure that at least two people are familiar with each piece of your infrastructure. There is often a natural tendency for different developers to take ownership of specific pieces of code or infrastructure, and be the only ones who touch or maintain it. But then it breaks while they’re on vacation, and everyone else is left trying to figure out how this code was ever supposed to work. A mandatory code review policy is often a great way to mitigate that class of problem.

But, theoretical and observed benefits of code review aside, speaking personally, as both a developer and as a reviewer, I’ve been finding it a really frustrating process.