Design retrospective · Ouroboros

Userland maturation: /bin, pipelines, and where a command belongs

One long day that turned the shell's compiled-in builtins into a real /bin of standalone programs with genuine multi-stage pipelines, then began abstracting the filesystem. The through-line: where does this command actually belong — in the shell, in a program, or nowhere yet?

🧩 The ninth retrospective, and a companion to the capability & hardening and network stack retrospectives whose mechanisms it builds on. Not a bug hunt: a design day whose most interesting decisions were subtractions.

The starting point

Every command was a shell builtin: the shell parsed a line and called a cmd_* function compiled into its own binary. exec /path could launch a program, but passed it no arguments. The capability model, per-task isolation, grant/safecopy IPC, and runtime capability delegation all already existed. What didn't exist was any reason for a program other than the shell to run — no argv, no /bin, no environment, no way to compose programs. The plan was a staged arc: an argv ABI, a /bin + PATH lookup, a shell environment, then externalize the commands over a shared ulib crate. It mostly went as planned; the lessons are where it didn't.

Lesson 1: not every command wants to be a program

The externalization went smoothly — echo, ls, cat, mkdir, cp, mv, and the network commands all became clean /bin programs. So the next batch seemed obvious: ps, kill, wait. They were built, and mostly worked — then a test exposed the flaw:

$ ps                 # shows a long-lived task at slot 5
$ kill 5             # correctly kills it
$ wait 5             # "that task is protected"

wait 5 was waiting on itself. An externalized command runs in a spawnable slot, and slots are reused immediately, so by the time wait ran it had become task 5. The whole point of these commands is to act on the shell's view of the task table; putting them in a spawned task makes them appear in their own output and makes any task number they're handed racy. This is not a bug — it's a category error, and exactly why kill/wait/jobs/fg/bg are shell builtins in bash. (Unix ps is external only because PIDs are stable; Ouroboros reuses slots the instant they free.) The work was reverted and the finding recorded.

The lesson: "externalize the commands" is not a uniform goal. A command that takes a path and does I/O externalizes cleanly; one that manipulates the shell's own control state does not. Testing, not review, drew the line — the code compiled and ran; only exercising it revealed the abstraction was wrong. Knowing when to stop an arc is part of executing it.

Lesson 2: a packed bitfield has a ceiling you can walk into

Raising the spawnable-slot count from two to five was meant to be a mechanical NUM_TASKS bump. It nearly wasn't. Per-slot capabilities pack into one u32: the low NUM_TASKS bits are the IPC send-mask, and the resource caps (CAP_BLOCK/CAP_CON/CAP_NET) sat at bits 8/9/10. At NUM_TASKS = 7 the send-mask stopped at bit 6 — a gap. At 10 it reaches bit 9: a spawnable slot's send-mask bit would have aliased CAP_CON, granting or denying a device capability as a side effect of the slot count. Caught by reading the layout before flipping the constant. A packed representation encodes an invariant nothing enforces; the fix moved the resource caps to bit 16+, clear of any plausible slot count, with a comment so the next person doesn't rediscover it.

Lesson 3: build the mechanism for a consumer that exists

Runtime capability delegation shipped during the network arc, and the capability postmortem flagged a trap: don't build the general (transitive, multi-target) version until a consumer needs it. This day produced two consumers, and both needed less than feared.

The network commands reach netd, which a spawnable slot's static mask doesn't include. The fix wasn't to widen the policy for every program — it was for the shell (which holds TO_NET) to DELEGATE it to the specific child at spawn. One subtlety: a tick can let the child call netd before the delegation lands, so it gets MSG_ERR_DENIED; rather than add synchronization, ulib::net_call retries briefly on that error — the race resolves itself.

Multi-stage pipelines looked like the case that would finally require general delegation — relay-free program chains were the roadmap's own example for it. But a linear chain doesn't: task a delegates to b, b to c, and each task has exactly one downstream target — which the existing one-target-per-task delegation already models. The general version still has no consumer. The specific mechanism was enough; building the general one first would have been waste.

Lesson 4: a heuristic that doesn't generalize is technical debt

The two-stage pipe had grown a patchwork: it split on the first | only, the right side had to be a single argument-less token with an explicit path, and it decided "program or builtin?" by a /-prefix heuristic. Each was a reasonable shortcut for one builtin | program pipe; none survived N stages, arguments, or bare names. The fix replaced them rather than extending them: split on every |, resolve each stage like any command ($PATH or path), tokenize each into its own argv, spawn the chain right-to-left delegating each link. The rewrite was larger than an extension, but left one code path instead of three, and the "no arguments" and "one | per line" limits simply evaporated. When a shortcut needs a special case to handle the next case, that's the signal to generalize.

Lesson 5: to test a format, you have to be able to make one

The filesystem arc opened with a pure refactor (extract fsd's hardcoded FAT32 into a Filesystem enum, prove byte-identical) and then GPT + multi-partition support. The code was straightforward; testing it was the problem. The GPT parser only matters if fed a real GPT disk, and macOS ships no GPT tooling (hdiutil builds MBR). Worse, the disk has to be bootable — the parser runs after UEFI loads the kernel from it — and UEFI validates a GPT's CRC32s before booting. So testing meant building the artifact by hand: scripts/mkgpt.py wraps the FAT32 partition in a protective MBR, primary and backup GPT headers with correct CRC32s, and an ESP entry. The test is conclusive precisely because the GPT disk has no real MBR table (only the protective 0xEE entry): fsd mounting it means the GPT parser found it. For a format or protocol, the harness that produces valid inputs is part of the feature — a parser you can't feed a valid input isn't tested.

The shape of the day

Two arcs closed (standalone binaries, multi-stage pipelines) and a third begun (filesystems), and the discipline underneath was the one the earlier retrospectives keep finding: scope it down before you build it, and let testing tell you where the real boundaries are. Three of the day's decisions were subtractions — job control stays builtin, general delegation isn't needed, the pipeline heuristics go away — and each made the system simpler, not poorer. Everything was QEMU-verified with zero exception-trace aborts. The one honest gap: none of it has run on real Parallels hardware yet — a real-hardware regression pass is the debt this day leaves behind.