Giving Parallels a disk

This is a write-up of one day's work taking Ouroboros, a from-scratch ARM64 microkernel, from "our primary hardware target has no disk and we don't know if it ever can" to reading a real USB stick, loading programs from it at runtime, and passing messages between them — on real Parallels Desktop on Apple Silicon. If you're writing a bare-metal OS and wondering how to get any disk on a hypervisor that doesn't offer you one, or building a USB mass-storage driver over your own xHCI stack, some of this may save you real time. Everything below was confirmed by direct evidence — decoded descriptors, on-screen device inventories, byte-level round trips — not guessed.

The starting problem: Parallels on Apple Silicon boots this kernel fine (UEFI reads it off a virtual hard disk), and by the start of this day the kernel had a working framebuffer console, a USB keyboard, preemptive multitasking, and a full shell — but no way to touch a disk after boot. Firmware reads the boot volume during UEFI boot services; the moment boot services exit, that access is gone, and every disk command in the shell printed "no filesystem mounted this boot." The obvious question: what storage interface does this hypervisor actually expose to the guest, and can we drive it?

The answer is "none" — and proving a negative takes real experiments

A PCI device inventory (a diagnostic the kernel already carried) showed five devices: HD Audio, an EHCI USB 2.0 controller, an xHCI USB 3.0 controller, virtio-net, and one vendor-specific Parallels device with no public documentation. No storage controller of any kind — despite the VM demonstrably booting from its "SATA" hard disk every single run. "SATA" in the hypervisor's configuration does not mean an AHCI controller on the guest bus.

That could still have meant "wrong attachment type," so the alternatives were tested directly, from the host, using the hypervisor's own CLI. It told us an ARM64 EFI VM accepts exactly two disk interfaces (IDE and SCSI), and SCSI accepts three controller subtypes. Attaching a scratch disk as each of the two LSI Fusion-MPT subtypes — real, documented hardware that would have been an implementable spec — produced a byte-identical PCI inventory both times. The emulated controllers simply don't exist on Apple Silicon, even when the configuration accepts them. The third subtype was refused by the hypervisor itself as incompatible with EFI firmware.

Conclusion, with evidence rather than resignation: all storage on this platform flows through a proprietary, non-PCI path. "Implement a documented spec" is not available for any attached-image disk. The one documented thing on that bus with a path to storage was the xHCI controller the kernel already drove for its keyboard — which pointed at USB mass storage via a passed-through physical stick.

A tooling problem worth recording from the same session: the kernel's PCI inventory prints during UEFI boot services, and this platform's framebuffer console clears the screen when it installs — about two seconds after power-on. A capture loop screenshotting the booting VM at 0.4-second intervals never caught anything but the finished shell. The fix was to make the diagnostic return its data so the kernel could re-print it through the console that survives.

USB passthrough routes by speed — USB 2.0 goes somewhere you can't follow

The first enumeration attempt used the USB 2.0 stick on hand. Passed through and confirmed connected to the VM by the host's own tooling — and it never appeared on the xHCI controller. A temporary diagnostic (dump every connected xHCI port at scan time, wait six wall-clock seconds, dump again) showed the same two ports — virtual mouse, keyboard — before and after the wait, ruling out slow attachment.

The VM has both an EHCI (USB 2.0) and an xHCI (USB 3.0) controller, and the hypervisor routes passthrough devices by their speed: high-speed devices land on EHCI, SuperSpeed devices on xHCI. With no EHCI driver and no appetite for writing a second host-controller driver for one stick, the answer was: get a USB 3.x stick. When one arrived, it appeared on an xHCI root port at SuperSpeed, enumerated through the kernel's own multi-device scan, and presented exactly the textbook mass-storage interface: SCSI-transparent command set over Bulk-Only Transport.

Passthrough attaches late — your boot-time scan will miss it

The same six-second diagnostic produced a second, subtler finding: on this hypervisor, a passed-through USB device attaches several seconds after the VM starts — after the kernel's one-shot boot-time port scan has already run (boot to shell takes about two seconds; the attach lands around six to eight). The very first bare run missed the stick entirely; the diagnostic build only caught it because its deliberate six-second delay happened to push the scan late enough.

This turned "no hot-plug support" from a documented nicety into a hard design input: the storage driver got a runtime port rescan, triggered explicitly by a mount shell command — boot, wait a moment, type mount. QEMU's monitor turned out to be a perfect stand-in for testing this: genuine hot-plug of a storage device into a running VM, fully scripted, no hardware required.

What single-device assumptions quietly hide

Before storage, the xHCI driver was deliberately one-device: the port scan tried each port until it found the keyboard and abandoned everything else. Preparing for keyboard and stick to coexist exposed two pieces of per-device state that were single only because of that abandonment:

• The Output Device Context — the memory the controller itself writes a device's state into, through a per-slot pointer array. One shared context with two live slots pointing at it is real memory corruption, not a bookkeeping wrinkle.
• The default-endpoint transfer ring — each device's Address Device command declares its own ring position to hardware. The old shared ring needed an explicit software rewind per candidate device; two live devices can't share the memory at all.

Both became small per-device pools. Two ordering and routing rules came out of the same work, and both generalize: activate the endpoint that generates unsolicited traffic last (the keyboard's interrupt endpoint is armed only after the whole scan finishes, so keystroke events can't interleave with a later device's setup), and route completion events by slot and endpoint ID — with more than one live device, "the next event" and "my event" are different things.

The mid-transfer keystroke that would have killed the keyboard permanently

The sharpest correctness point in the storage driver was found by reading the event loop before it ever failed. During a bulk transfer, the driver waits synchronously for its completion event on the shared event ring. A keystroke arriving in that window posts its event to the same ring. The naive wait skips and drops foreign events — which here would not just lose the keystroke: the keyboard's single interrupt buffer is only re-armed when its event is processed, so a dropped event means the keyboard never receives anything again. Typing during disk I/O would brick input for the rest of the boot.

The fix is routing, not dropping: the wait hands keyboard events to the exact same report-processing code the normal poll uses (factored so both can call it), which re-arms the buffer and queues the keystrokes. Verified by firing keystrokes interleaved with disk commands: zero lost characters.

SCSI Unit Attention — found by the hot-plug test, not the spec

The QEMU hot-plug test produced a beautifully diagnosable failure on first contact with a freshly attached device: INQUIRY succeeded, then the first READ CAPACITY failed with an error status, and a manual retry succeeded. That pattern is the explanation: a freshly attached-or-reset SCSI device reports a Unit Attention condition, failing the first command with CHECK CONDITION until the sense data is fetched — and INQUIRY is spec-exempt from Unit Attention, which is exactly why it sailed through. The standard bring-up sequence (a short TEST UNIT READY / REQUEST SENSE loop before the first real command) cleared it. Real sticks set Unit Attention too, so the QEMU-found fix mattered on hardware.

Two error codes that collapse into one will eventually lie to you

Two same-day findings from outside the storage driver, same underlying moral. First: the kernel bounds every userland syscall buffer at 512 bytes, and rejects longer ones with the same generic error code used for "no such file." The shell's new append operator (>>) used a 1024-byte buffer for its read-back — the rejection was indistinguishable from "file doesn't exist," whose legitimate meaning in an append is "create it." Result: append silently behaved as overwrite, discarding the existing content, caught only because the very first append test compared actual file contents rather than trusting the exit status.

Second: the day the collapsed filesystem error code was split into specific ones, the new messages instantly exposed a real mis-mapping that had been hiding for weeks — rmdir / reported "invalid name" instead of "can't remove the root directory," because a path-splitting helper failed before the root check ever ran. Invisible for as long as both causes collapsed into the same vague message.

A "can't happen yet" is a bug with a start date

Directory extension (letting mkdir grow a full directory by a cluster) was a small feature — the real find was in rmdir, which freed exactly one cluster. That was correct for as long as every directory was single-cluster by construction, and became a silent cluster leak the moment directories could grow — exactly the sequence (fill a directory, empty it, remove it) the new feature made routine. The same shape appeared earlier in this project's history and will appear again: an invariant maintained by a limitation elsewhere is a bug scheduled to trigger when that limitation is lifted. When you lift one, grep for everyone who was leaning on it.

The toolchain can poison you even in release mode

Ouroboros userland is position-independent and links against Rust's prebuilt core library, which contains some non-position-independent object code. The known rule was "userland must build in release" (debug builds pull the poisoned objects in through panic machinery). This day added a sharper corollary: even in release, an ordinary string slice by range pulled in the slice-bounds panic path — whose can't-actually-happen error branch formats the offending string with enough of the formatting machinery to drag a non-PIC object into the link and fail the build outright.

// fails to link: the slice's panic path pulls non-PIC
// formatting code out of the prebuilt core library
let head = &line[..offset];

// links fine: non-panicking, no panic path to pull in
// (the offset is a known-good char boundary anyway)
let head = line.get(..offset)?;

The anomaly we recorded and did not explain

Honesty section. The very first real-hardware run after assigning the USB stick to the VM showed no synthetic keystrokes arriving at all — the boot completed normally and the screen simply never echoed a single typed character. It never reproduced: a control run without the stick was clean, a six-command timeline probe typed straight through the attach window was clean, and every subsequent full run was clean. The best (unproven) explanation is that the scripted keystroke stream was lost on the host side during first-attach USB turbulence. It's recorded as exactly that — one unreproduced run, bounded by control experiments — rather than silently forgotten or hand-wavingly "fixed."

Techniques that generalized well

• Prove a negative with experiments, not absence. "No storage controller" was established by attaching every controller type the hypervisor offers and showing the bus never changed — not by failing to find one once.
• A headless hardware lab pays for itself in a day. The hypervisor's own CLI drove every experiment here unattended: boot, per-scancode typing (including held-Shift and held-Ctrl chords), screenshots, USB assignment — no human watching a screen.
• One variable per hardware round trip. The USB 2.0 question, the late-attach question, and the keystroke anomaly each got their own dedicated boot with one thing changed.
• Organic tests beat synthetic ones where the system allows them. Tiny 512-byte clusters made directory growth reachable with fifteen real files; QEMU monitor hot-plug made the rescan path testable without hardware; a deliberate two-second delay in a test program made a blocking wait testable by a human-speed typist.
• Read the event loop before it fails. The keyboard-death trap was designed against, then verified — not discovered by a dead keyboard.

Where this ended up

The same screen that started the day saying "no filesystem mounted this boot" ended it showing: mount finding a real Lexar stick, its real INQUIRY vendor strings, a 116 GiB capacity, a mounted FAT32, a listing of the stick's actual files — then exec loading a program off that stick at runtime (a first for this kernel on real hardware), a complete message-passing round trip through it, and a blocking wait collecting a second program's exit status, with the two tasks' output visibly interleaved on the console. Every userland facility the kernel has now runs on the platform it was built for.