One long day: adding exFAT and ext2 to fsd — each
read-only then read-write — validating every step against foreign checkers,
and then taking the whole body of work to real Parallels hardware, which
found a bug the emulator can never show.
fsd, the userland filesystem server, was FAT32: one hardcoded engine, one mounted filesystem. A prior refactor had wrapped it in a Filesystem enum with a single arm and a mount() that probed partitions — a "prove it's byte-identical first" step that set up for this. The goal: a second filesystem (exFAT), then a third and structurally different one (ext2), each read-only first and read-write second, all behind the unchanged FSOP_* IPC protocol clients already spoke.
exFAT went in easily, and that was almost a trap. Structurally it's the same shape as FAT — a partition, a FAT, a heap of clusters, a directory of entries — so the read machinery mirrored fat32.rs and the enum barely stretched. If the arc had stopped there, the "abstraction" would have been validated only against a near-twin of what it already held. That proves very little.
ext2 is what actually tested it. A genuinely different model: inodes own metadata and a directory entry is just name -> inode number, so path resolution bounces between directory blocks and a separate inode table; files are reached through block-group descriptors and direct/indirect block pointers; names are case-sensitive. Making ext2 work through the unchanged protocol — nothing above fsd touched — is the real proof the abstraction was an abstraction and not FAT with extra coats of paint. If every implementation behind an interface is similar, the interface is untested.
The most valuable habit of the arc was never trusting the driver's own reads to prove its own writes. Every write stage was checked with a tool written by someone else:
fsck_exfat pronounced the bitmap and hierarchy clean after a create/write/delete/rename churn; and a binary copied by the OS read back byte-identical when dumped on the host.e2fsck -fn ran all five passes clean after each stage, and debugfs dumped a copied binary matching the original byte-for-byte.When a different vendor's fsck signs off on your on-disk structures — checking invariants your own code never looks at (free-count agreement across three places, link counts, orphan lists) — you have evidence your own reader can't give you, because your reader shares your bugs.
The best bug of the arc. After rm/rmdir, ext2 files were freed correctly by every measure I could see: link count zero, inode bitmap cleared, a deletion time set, and s_last_orphan was 0. Yet e2fsck reported a "corrupted orphan linked list" naming exactly the deleted inodes.
The cause: e2fsck treats a links-0 inode whose i_dtime is less than the inode count as an entry on the orphan list, using i_dtime itself as the next-orphan pointer. The driver had written a sentinel deletion time of 1 — read by e2fsck as "next orphan = inode 1", fabricating a bogus chain. Fixed by writing a plausible timestamp (a fixed constant; no RTC). The lesson generalizes: a field can be overloaded to mean different things depending on the record's state, and a value obviously harmless in one reading can be actively wrong in another — exactly what a foreign checker exists to puncture.
The corruption risk in a filesystem lives entirely in the writes, so each filesystem landed read-only as one milestone, then read-write as a sequence of small, independently-fsck-validated stages: allocation + touch; then data writes; then directories; then mv. exFAT's two checksums came out right on the first real test precisely because entry-set construction was the only new thing being checked that stage — a narrow surface is a debuggable surface.
With all of it QEMU-green, the arc ended on a real Parallels VM. Boot, shell, servers, task table, relocation self-test — all clean, confirming the churn hadn't regressed the hardware path. Then the filesystem-on-real-hardware test found something QEMU structurally cannot show. Parallels exposes no disk to the kernel except USB mass storage, so it needed a real USB stick — and two boot configs produced an inverse correlation:
Neither is a filesystem bug; the FS drivers read and wrote correctly whenever the block layer served them. It's the xHCI/USB stack failing to keep a HID keyboard and a mass-storage device concurrently live. QEMU never hinted at it because there the keyboard is a synthetic device and storage is virtio-blk — they never share an xHCI controller, so the contention that defines the bug doesn't exist to be found. Some bugs only exist where real hardware forces a sharing the emulator abstracts away. An emulator is a hypothesis about the hardware; a real-hardware pass is how you find where the hypothesis is wrong.
Two filesystems, each read then write, all behind one unchanged protocol; every stage checked by a tool that didn't share the driver's assumptions; and a real-hardware pass that both confirmed the work and surfaced the one thing the emulator could never reveal. The recurring disciplines were the old ones — prove the abstraction with a different implementation, stage the risky writes, verify against something outside your own code — plus a new one worth keeping: run on real hardware not to confirm success but to find the failures your emulator is structurally incapable of producing.