This is a write-up of one day's work on Ouroboros, a from-scratch ARM64 microkernel, that took it from no networking at all to a concurrent HTTP server you can point a browser at — plus ping, resolve (DNS), and fetch (client HTTP) at the shell. Everything is hand-rolled, no_std, no crates: a virtio-net driver, ARP, IPv4, ICMP, UDP, DNS, and a TCP that does flow control, fast retransmit, a retransmit timeout, and multiple simultaneous connections.
Like the others, it's kept out of the project's own record (CLAUDE.md, CHANGELOG.md) because most of it isn't Ouroboros-specific. If you've ever written a TCP from scratch, or tried to test loss recovery on a network that never loses a packet, some of this may save you time.
The shape of the day
Networking landed as one long staged arc, each stage built and tested before the next depended on it:
- Stage 1 — the virtio-net driver: raw Ethernet frames in and out, proven by a hand-built ARP round-trip.
- Stage 2 — the driver/protocol split: the stack moved to a userland server (
netd) reached through gated syscalls; ARP + IPv4 + ICMP and apingcommand. - Stage 3 — UDP, proven by real DNS resolution (
resolve). - Stage 4a — a client TCP (
fetch— a real HTTP GET to a real site). - Stage 4b — the async-receive primitive, and
netdbecomes a server. - Stage 4c–4g — a real static-file HTTP server: files from the filesystem server, a directory listing, correct
Content-Type/Content-Length, andHEAD. - Stage 4h–4i — TCP loss recovery: fast retransmit, then a timer-based retransmit timeout (RTO).
- Stage 4j — concurrent connections.
Three of those had a bug or a design turn worth writing down. The rest were "read the spec, hand-roll it, cross-check the bytes."
Thread 1: the driver/protocol split is forced, not chosen
The first decision was where the code lives. This is a microkernel — it had already moved its filesystem and console out of the kernel into supervised, MMU-isolated userland servers. Networking wants the same, but a hard constraint decides which half stays in the kernel.
There is no IOMMU. A device programmed to DMA can read or write anywhere in physical memory. So the code that owns the device's DMA rings and buffers cannot be an untrusted EL0 task — a bug there is a write primitive over all of RAM. The DMA owner has to stay in the trusted kernel; it's the same rule that already kept the virtio-blk driver there.
So the split writes itself: the DMA-owning virtio-net driver stays in the kernel, exposed through three gated syscalls (NET_SEND/NET_RECV/NET_MAC) that move opaque frames and are accepted from one task only. The entire protocol stack lives in netd, a userland server holding the only capability to call those syscalls. The kernel never learns what a TCP segment is — so a whole new transport (UDP, then TCP) is pure userland code, no kernel change at all.
The lesson generalizes past "no IOMMU": let the trust boundary, not convenience, decide the split. The narrowest thing that must be trusted is the only thing that should be privileged; everything above it is a normal program.
Thread 2: a server that can only initiate isn't a server
Stages 1–4a all initiate: send a request, wait for the reply. The server's loop is trivial. A server is the opposite — it reacts to input that arrives unsolicited — and that exposed a gap. netd's loop could block on an IPC message or poll the NIC for a frame, but a task can only block on one thing. Block on IPC and you're deaf to the wire; poll the wire and you spin (and the supervisor's heartbeat restarts you as wedged).
This is the classic poll/select problem, and it needed a kernel primitive: a task can wait for "a frame arrives or a message lands," whichever comes first (a NET_WAIT syscall, backed by a WaitReason::NetInput the per-tick wake-check evaluates over both sources). That one primitive turned netd into an event loop, and the guest started answering the network — replying to ARP for its own address, and serving TCP.
It grew once more for RTO: a retransmit timeout must fire when nothing is arriving (a silent peer sends no frames and no messages). So NET_WAIT gained an optional timeout — wake on a frame, a message, or a deadline. It's the only WaitReason with a timer, and it's general. The broader point: a blocking primitive that can't also time out can't drive a protocol timer. Build the timeout in from the start.
Thread 3: testing loss recovery on a wire that can't lose a packet
The two hardest stages — fast retransmit and RTO — recover from lost segments. But the dev environment (QEMU's SLIRP user-mode networking) is a reliable local path: it never drops a packet. The exact condition these features exist for cannot occur in testing.
The answer was to inject the loss in the sender itself: a temporary hook that skips putting one segment on the wire but still advances the send cursor, so the peer sees a genuine gap. Reverted after; the acceptance test was the file arriving byte-identical despite the drop.
Isolating which mechanism fired took one more step. A mid-stream drop makes the peer send duplicate ACKs, which triggers fast retransmit — so to test RTO, fast retransmit had to be temporarily disabled, leaving the timer as the only path back. That's the general recipe for testing a fallback: you can't test the fallback until you disable the thing that would rescue you first.
Two real bugs came out of this that no reading would have found:
The go-back-N wrap-to-huge stall. After a fast retransmit rewinds the send cursor to the gap and resends, a buffering receiver (real Linux/macOS TCP buffers out-of-order segments) can acknowledge past the rewound point in one jump once the gap fills. That left snd_una (highest acked) ahead of snd_nxt (next to send); the in-flight calc snd_nxt - snd_una, unsigned, wrapped to ~4 billion, so "is there room in the window?" always answered no, and the transfer stalled forever at ~86 KB. The fix is a one-line invariant — never let snd_nxt fall below snd_una — but the symptom (a clean stall at a specific offset, no crash) is one you only diagnose by watching sequence numbers in a packet trace.
The supervisor restarting the server mid-transfer. A userland server that stops answering a periodic health-ping (~160 ms) is judged wedged and restarted. Streaming a large file is slow here — every chunk is an IPC round-trip plus an emulated disk read — so an early version that sent a whole window in one uninterrupted burst missed the ping and got restarted mid-transfer, losing the connection state. The fix wasn't "go faster"; it was yield often enough to stay responsive: send in small bursts, acking the health-ping between each. Supervision and long-running work are in tension, and the long work has to cooperate.
Thread 4: the guard page as a running stack budget
An earlier milestone gave every program a stack guard page — an unmapped page below the stack, so an overflow faults cleanly instead of silently corrupting the code below. Across this arc it became a budget check that reports when you exceed it. netd overflowed its stack twice more as it grew, each caught as a clean fault:
- The client-op path (
fetch→ TCP client → DNS) nests several 1600/2048-byte buffers down one chain, tipping over the 16 KB stack → grown to 24 KB. - Concurrent connections put an array of connection structs (each ~2 KB) permanently on the server's frame, and a client op nesting on top of that tipped over 24 KB → grown to 32 KB.
The second has a diagnostic detail worth keeping: which half overflowed told us where to look. The concurrent streaming path was fine — four simultaneous transfers, zero faults — because pumping doesn't nest deeply. It was the client path, nesting on the now-larger permanent frame, that overflowed. A guard-page fault plus which operation triggered it points straight at the deep call chain. netd is now the most stack-hungry program in the system — learned by measurement, not guessing.
Thread 5: verify against something that isn't your own code
A recurring discipline, sharpest here: never confirm a network feature from the kernel's own log alone. Every stage was cross-checked against a tcpdump decode of a packet capture, and the server side against a real host curl through a forwarded port. When the kernel prints "sent an ARP reply" and the pcap shows it on the wire with checksums tcpdump accepts, those are two independent witnesses. The wrap-to-huge stall was found in the trace: the kernel's output showed nothing wrong; the sequence numbers showed the transfer jumping backward and flat-lining. A from-scratch stack lies to you in its own logs by construction — it only knows what you told it to print. The wire, and a mature peer's reaction to it, don't.
What it adds up to, and what's honestly still missing
By the end of the day the guest resolves DNS, fetches web pages, and serves its own filesystem over HTTP to a browser opening several concurrent connections — over a TCP that paces itself to the peer's window and recovers lost segments two ways. All hand-rolled, all in a userland server the kernel can restart if it crashes.
What's deliberately not there: IRQ-driven receive (the NIC is still polled — the biggest remaining item); congestion control and RTT estimation (only the peer's window is honored; the RTO is a fixed 1 s); SACK (loss recovery is go-back-N); and it's QEMU-only (the Parallels target exposes virtio-net over PCI, which this project's virtio path doesn't drive). None are hard to see the shape of; they're just the next milestones.
The meta-lesson is the staging. A network stack sounds like one enormous thing. Split into "driver, then one protocol at a time, each proven end-to-end before the next," it became ten small milestones — each of which either worked on the first real test or failed in a way a packet trace explained in minutes. The hard parts announced themselves exactly when the milestone that needed them arrived, and not before.