From a bare shell prompt to a real, disk-backed OS

This is the second piece of a four-part debugging history for Ouroboros, a from-scratch ARM64 microkernel. Part one covers getting from "UEFI hands us control" to a live, working console on real hardware. Part three covers getting real keyboard input into that console. This one covers everything in between: turning a bare shell prompt into a real, disk-loaded userland process with real commands, a real filesystem underneath it, and real write support — and the bugs found doing it, several of which are genuinely reusable lessons for anyone else writing a small freestanding kernel.

One theme dominates this stretch more than any single bug: a whole family of crashes that all trace back to the same root cause, found three separate times before the underlying pattern was fully understood. That's worth reading as its own throughline, not just unrelated war stories.

The recurring bug: pointers baked into your binary don't know where they'll actually load

Ouroboros's userland programs are loaded as flat, position-dependent binaries — no ELF, no relocation processing, linked with a fixed base address of zero, then copied byte-for-byte to wherever the loader actually placed them in memory. Direct function calls are fine: AArch64 compiles a call as a PC-relative branch, correct no matter where the code ends up. The problem is anything that isn't a direct call — any pointer stored as data, computed once at compile time and baked into the binary's read-only section. That pointer was computed against the link-time base of zero. Nothing ever adjusts it for the real load address, because there's no relocation step at all. This bit twice, in two completely different-looking ways, before the general shape of the problem was understood.

First bite: formatting macros crash any loaded program

Symptom: a new command's first implementation used an ordinary formatting macro to print a number — and crashed immediately, the program counter landing on a tiny, near-zero address instead of any real code.

Root cause: Rust's formatting machinery doesn't dispatch argument formatting through direct calls — it builds an array of function pointers, one per formatted argument, as data, baked into the binary's read-only section at compile time. For a normal, properly relocated executable this is invisible. For a binary linked at base zero and loaded somewhere else with zero relocation processing, those pointer values are simply wrong wherever the code actually runs.

Fix: hand-roll the one piece of formatting actually needed instead of going through the standard machinery at all. Not elegant, but the only viable fix short of writing a real relocating loader.

Second bite: even a plain comparison against a literal can crash

Symptom, much later, in completely unrelated code: a path-resolution routine needed to check whether the current directory was already root — a plain comparison of a byte slice against a string literal. It crashed with the exact same signature: the program counter landed inside the program's own code at a small, garbage, code-layout-dependent address that shifted between builds.

Root cause: the identical underlying problem, in a form much easier to miss than a formatting macro. Comparing a slice against a literal needs a reference to that literal's bytes somewhere in memory — exactly the kind of thing computed once, at compile time, relative to the link-time base.

// crashes: needs a reference to literal data in .rodata
if cwd_bytes != b"/" { ... }

// fine: only scalar comparisons, no reference to anything
if !(cwd_bytes.len() == 1 && cwd_bytes[0] == b'/') { ... }

How this one was actually found: by bisection, not inspection — binary-searching temporary print statements through the path-resolution function until the exact crashing line was isolated. The symptom looked nothing like its cause until that isolation was done.

Turning the shell into a real userland process

Before any of the above, the shell was just kernel code running at a low privilege level, compiled directly into the kernel image. A deliberate architecture change made it a genuine, separate program: loaded from disk, selected by a small config file, replaceable without rebuilding the kernel at all — using UEFI's own filesystem driver during boot services, rather than building a real runtime block-device driver just for this, an explicit scope decision rather than an oversight.

A genuinely nice side effect: this killed an earlier, hard-won compiler alignment ceiling entirely (see part one) — a disk-loaded program doesn't need a compile-time-aligned static at all, since the loader can just ask for however many runtime-determined pages a program needs. A new alignment problem immediately took its place — the page allocator only guarantees a coarser granularity than the identity-map builder's block-splitting logic implicitly relied on — solved by deliberately over-allocating and freeing the excess, recovering the same guarantee at runtime instead of compile time.

A measurement trap: why the "actual" tick rate was a lie for ten seconds

Making the shell feel responsive surfaced a real bug: typed characters could take up to a full second to echo, because the kernel's round-robin task switch fires unconditionally on every timer tick, whether or not either task has anything to do. Lowering the tick interval fixed the practical symptom.

The interesting part is how the diagnosis nearly went sideways. The first attempt to measure the real tick rate used a whole-run exception count from the emulator's own internal trace — and concluded the tick was already firing far faster than expected, even before the fix, which would have meant the entire latency theory was wrong. That number was an artifact: firmware's own boot-time code is still resident and executing for the first several seconds of every run, taking real interrupt exceptions of its own at a much higher rate than the kernel ever will. A targeted debug print placed directly inside the timer re-arm function — immune to this contamination — gave the true number, matching the configured interval exactly.

The disk that lied about its own format

Bringing up runtime disk I/O needed a virtio block driver and a hand-rolled FAT32 reader, and surfaced two more genuinely confusing mistakes.

A misleading diagnostic tool, not a misleading device. Figuring out which transport a block device had attached over needed inspecting the emulator's own internal device tree — and the first attempt appeared to show no block device at all, obviously wrong since the kernel demonstrably boots from it every run. The real bug was in the diagnostic script: reading the monitor's response with one fixed sleep before switching to non-blocking reads cut the response off partway through, consistently at the same byte count — exactly what makes a truncation bug look like real, complete data.

The fast development disk format isn't what production images use. The emulator's own convenience disk-passthrough driver, used for the fast day-to-day development loop, turned out to produce FAT16, not FAT32 — confirmed by decoding the on-disk header fields by hand rather than trusting the tool's name. The real production disk image, the same path real testing hardware ultimately boots from, is genuinely FAT32. A separate boot target using the real image was added specifically because the fast loop's disk can never satisfy a genuine FAT32 reader.

The project's own directory name didn't fit the filesystem's own naming limit. FAT's short-name convention caps a path component at eight characters plus a three-character extension; this project's own boot-configuration directory name was nine. Rather than implement long-filename parsing (a real feature, on a hand-rolled, no-allocator reader) to accommodate one avoidable name the project itself controlled, the directory was simply renamed to fit.

The hang with zero output

This is the single most instructive bug in this whole stretch, because of how it failed, not just why.

Symptom: navigating up two directory levels in a row hung the entire system. Not a crash, not an error message, not a reported exception. Nothing. The whole machine simply stopped responding, forever.

Root cause: a real, on-disk FAT32 convention this reader hadn't accounted for. A subdirectory's own "parent directory" entry conventionally stores cluster number zero to mean "the root directory," rather than root's own actual cluster number. That zero flowed directly into a sector-address calculation that subtracts a small constant from the cluster number — zero minus that constant, in unsigned arithmetic, wraps around to an enormous, nonsensical sector number instead of a negative one, and the resulting disk read against that garbage address simply never completed.

Why this produced silence, not a crash: the read happens inside a system call, and taking any exception — including the timer interrupt that drives this kernel's entire preemptive scheduler — is masked from the moment a system call is entered until it returns. A runaway operation that would ordinarily eventually get preempted had nothing that could ever interrupt it. The one mechanism that could have surfaced the problem was, by design, exactly the thing turned off for the duration.

Fix: substitute the real root cluster number wherever a resolved directory entry's cluster number comes back as zero. How the trigger was isolated: direct, deterministic testing — a single "up one level" from a two-level-deep path worked fine every time; a second one immediately after it hung every time, before the fix, with no variance.

Adding real write support, one narrow case at a time

With reads working, the project crossed into real filesystem writes, each added as the narrowest useful case, in a consistent discipline worth naming explicitly: claim a resource before using it, and check every precondition before writing anything. Creating a directory claims its cluster in the file table before anything else touches disk, so a failure partway through leaves it correctly claimed rather than silently reusable; removing one checks it's a directory, isn't root, and is genuinely empty before freeing anything, so a rejected request never partially applies.

Two more real bugs surfaced here — both caught by reasoning through the design before writing any test, a genuinely different discipline from the hardware-confirmed debugging that dominates the rest of this project's history. The cluster-zero-means-root substitution from the hang bug above had been applied to every resolved path component, not just directories — harmless until empty-file support was about to make "cluster zero, and it's a real file, not root" a common case; without a fix, deleting a fresh empty file would have tried to free root's own cluster, corrupting the entire filesystem on first use. And moving a directory to a different parent needed the identical convention applied on a write path this time — caught the same way, by tracing through what the operation needed before implementing it.

Techniques that generalized well

• When two bugs share an exact crash signature in unrelated code, suspect a shared root cause before debugging either one from scratch.
• Bisect with temporary print statements when a crash's location and its cause are far apart.
• A tool that reads a live process's own diagnostic interface can lie just as easily as the process itself. The truncation bug was in the test harness, not the thing under test.
• Decode a foreign on-disk format's header fields by hand before trusting what a tool calls it.
• A bug inside a no-preemption code path can hang with zero signal. That changes how suspicious you should be of unsigned arithmetic and externally-sourced values in exactly that code.
• Not every bug needs a crash to be found. Tracing through a design's own logic before writing a test caught two real, serious bugs before they ever had a chance to corrupt anything.

Where this ended up

A real, disk-loaded userland shell — not kernel code — with real tokenized commands, a real hand-rolled FAT32 filesystem underneath it, and a full read/write file-management surface: listing, reading, creating and removing both directories and files, writing real content, copying, and moving. Everything in part three that follows this was built on top of a shell that, by the end of this arc, was already a genuinely real program running on genuinely real storage — just still waiting for anything to reach it besides a piped test harness.