Ouroboros manual

What Ouroboros is, in one line

An ARM64 microkernel OS whose filesystem, console, network and accounts are userland servers reached over a uniform file protocol — which is exactly what lets several machines share their disks, processes, consoles, and even run programs on each other over the network. That last part is the cluster section, and it is what the project is ultimately about.

Prerequisites

• Rust — pinned by rust-toolchain.toml (stable channel; the aarch64-unknown-uefi and aarch64-unknown-none targets and the llvm-tools component install automatically on first build). No nightly, no -Z build-std.
• QEMU — brew install qemu, which also provides the aarch64 OVMF firmware the run targets point at.
• macOS's hdiutil — used by make image to build the FAT32 disk image.
• Parallels Desktop (optional, for real-hardware runs) — provides prl_disk_tool (used by make parallels-hdd) and prlctl (used by make test-parallels).

Building

make build                  # kernel only (debug profile)
make build PROFILE=release  # kernel, release profile
make shell-bin              # the default shell (userland)
make esp                    # stage the full ESP directory layout
make test                   # host unit tests for the pure crates + clippy

The workspace has 62 crates: kernel (a UEFI application, aarch64-unknown-uefi); every userland program under programs/ — the shell, the four servers, and the /bin commands (aarch64-unknown-none); and the shared libraries both sides import, of which syscall-abi matters most, since it is why syscall numbers can never silently drift. Userland programs always build --release regardless of PROFILE — a hard toolchain constraint of the position-independent build, not a preference (a debug build fails to link; see processes.md's "Binary format").

The staged ESP layout:

\EFI\BOOT\BOOTAA64.EFI   the kernel (UEFI auto-boots this path)
\EFI\ORBS\SH.BIN         the default shell
\EFI\ORBS\FSD.BIN        the filesystem server (task slot 2)
\EFI\ORBS\COND.BIN       the console server (slot 3)
\EFI\ORBS\NETD.BIN       the network server (slot 4)
\EFI\ORBS\ACCOUNTD.BIN   the account server (slot 5)
\EFI\ORBS\INIT.CFG       one line: the path of the program to boot into
\bin\                    the standalone commands
\man\                    their manual pages
\etc\                    passwd, group, and the cluster identity

Swap the boot program by editing INIT.CFG — no kernel rebuild needed. (ORBS, not the full project name: FAT's 8.3 short-name limit.)

Running on QEMU

make run             # fastest loop - vvfat FAT16 disk (no real filesystem)
make run-image       # boots build/esp.img - real FAT32; disk commands work
make run-image-net   # real FAT32 + a NIC - the fullest single-machine run

The one everyday gotcha: make run's disk is FAT16 (a QEMU vvfat artifact), and FAT16 is the one format the filesystem server does not implement — it reads FAT32, exFAT and ext2 — so every disk command prints "no filesystem mounted" there. Use make run-image whenever you want ls/cat/write/exec to actually work. Exit QEMU with Ctrl+a x.

That's enough to get a working shell. The full run/test guide — every make run-* target, the exFAT/ext2/GPT test disks, the network runs, the 9P host peers, and the two-node cluster setup — is testing-qemu.md. You will want it for the cluster section below.

Running on Parallels (real hardware)

make image           # build/esp.img - raw MBR+FAT32 disk image
make parallels-hdd   # wraps it into build/esp.hdd via prl_disk_tool

Attach build/esp.hdd as the VM's Hard Disk device. Two confirmed traps: Parallels rejects raw .img files on the Hard Disk device, and attaching the image to the CD/DVD device looks like it works but doesn't (the optical driver wants ISO9660). Also: esp.hdd stores a pointer to esp.dmg's absolute path, so the two files must stay together.

What works on real Parallels hardware today: the GOP framebuffer console, a real USB keyboard (including Shift and Ctrl chords), preemptive multitasking, the whole shell — and real disk access via a passed-through USB stick (Parallels exposes no storage controller this kernel can drive, but USB mass storage over the xHCI driver works: pass a USB 3.x stick through, boot, wait a few seconds for it to attach, and type mount; USB 2.0 sticks land on the EHCI controller instead and can't be used). There is no networking on Parallels — its virtio-net is PCI, which this project's virtio path doesn't drive — so the cluster section is QEMU-only.

Scripted real-hardware testing

make test-parallels CMDS="help;ls;uptime" [BOOT_WAIT=12] [VM_NAME=Ouroboros]

Rebuilds esp.hdd, boots the registered VM headlessly, types each ;-separated command via prlctl send-key-event, and saves a screenshot after each into a parallels-test-<timestamp>/ directory (gitignored — delete it when done). Supports > (a held-Shift chord) and a CTRL-C pseudo-command. Requires the VM registered in Parallels with its Hard Disk pointed at this repo's esp.hdd.

Using the shell

Full builtin reference: shell-commands.md. The short tour (boot make run-image so the disk is mountable):

$ help                          # list the builtins
$ ls /                          # BIN/  EFI/
$ ls /bin                       # the standalone commands (see below)
$ write notes.txt hello world   # create/replace a file's contents
$ cat notes.txt
$ echo backup >> log.txt        # output redirection: > replace, >> append
$ cat log.txt | grep back | wc  # multi-stage pipelines
$ mkdir docs ; mv notes.txt docs
$ cd docs ; pwd
$ ps                            # one line per task slot: state + name
$ set PATH=/bin ; env           # a real environment; $VAR expansion

Commands come from two places

Builtins run inside the shell itself, and the set is deliberately minimal — a command stays builtin only if it mutates shell state, needs job control, or must run with no disk mounted: help cd bind mount unmount erase partition format exec exit/logout shutdown halt ps kill fg wait env set unset su cpu. Everything else lives in /bin.

The shell gates each session on a login (username + password against /etc/passwd; the dev images ship root/root and user/user), and the prompt shows # for root, $ for a normal user. logout ends the session and returns to the login prompt; su <user> lets root drop to a named user — a logged-in user cannot escalate. A user lands in their home under /Users, and ~ expands to it.

Account management lives in /bin and is root-only — useradd, groupadd, usermod — with one deliberate exception: passwd. Any user may change their own password (supplying the current one); only root may set someone else's. That works because the check lives in a server (accountd) which asks the kernel who sent the request, rather than in a setuid bit on a file — a setuid binary cannot work here, since the shell reads the binary, so "this is setuid" would be a claim made by a task the capability model distrusts.

On ext2, file permissions are enforced — owner→group→other, matching a primary or any supplementary group, root bypasses, and every ancestor directory's search bit is checked. So a normal user can write in ~ but not in /, and cannot read /etc/shadow at all. On FAT32 and exFAT there are no modes to enforce, and login says so at the prompt.

/bin programs are real standalone binaries loaded from disk, found on $PATH, spawned with arguments, and reaped:

files      ls tree cat cp mv mkdir rmdir touch rm write writeat chmod chown more
basics     echo pwd uptime clear args send recv selftest bootid man printenv id
filters    grep wc head tail nl rev uniq sort upper
network    ping resolve fetch dial serve
admin      passwd useradd groupadd usermod clusterkey

You type them the same way you would a builtin (ls, cat x); the shell finds /bin/LS on PATH. A /bin command resolves relative paths against the shell's working directory, delivered to it at spawn. cp and mv refuse an existing destination without -f — deliberate, on a system with no undo.

Usage help: append -? to any command that takes arguments, builtin or /bin, for a one-line reminder. For a full manual page, man <command>; pipe a long one to the pager, man grep | more. Pages are plain-text files under /man on disk.

Wildcards & completion: * and ? expand against the filesystem, and a pattern matching nothing is left literal. Press Tab to complete the last word as a filename — one match fills it in, several extend to the common prefix and then list.

Redirection & pipelines work across both kinds of command: echo hi > f.txt, >> f.txt, and a | b | c with each stage a separate program. A single builtin may sit at any pipeline position, and a | b > file redirects the pipeline's output.

Spawning, job control, and IPC demos

$ exec /bin/LS /             # exec spawns a program explicitly (like typing it)
$ exec /EFI/ORBS/SH.BIN      # spawn a second shell...
$ fg 6                       # ...and hand it the keyboard (a nested session)
$ exit                       # (in the nested shell) hand it back
$ kill 6                     # destroy a task
$ exec /EFI/ORBS/PONG.BIN    # the IPC echo-server demo...
$ send 6 hello ; recv        # ...send it a message and read its echo

Job-control notes: one task owns the keyboard at a time (fg moves it; when the owner dies it returns to the task that held it when fg handed it over, so a nested shell gets it back from its own commands, and to the boot shell if that task is gone; Ctrl+C terminates a foreground command the owner is blocked waiting on, but merely detaches a nested shell handed the keyboard with fg (the keyboard reverts to its parent, the shell survives with its login and cwd, and fg resumes it), and the boot shell ignores it). Ctrl+\ does the same in every case, and is the way out of a program that reads Ctrl+C as a key (raw keyboard mode, kbd_mode, syscall 70: an editor that binds ^C). An exited task holds its slot as a zombie until waited (ps shows it); kill reaps immediately. Ctrl+C (or Ctrl+\) also interrupts a stuck wait in the boot shell; in a nested shell it detaches that shell to its parent (the wait keeps running; kill it from the parent, or fg back).

Disks and mounts

$ mount                      # show what's mounted (format, partition, capacity)
$ mount -a                   # mount the kernel's block device (or rescan USB)
$ mount 1 /mnt/f             # mount the disk's 2nd partition at /mnt/f
$ mount -p /proc             # the process table as files
$ mount -n /net              # this machine's network identity (ip, mac)
$ mount -c /dev/cons         # the console as a writable file
$ bind /work /EFI/ORBS       # /work now resolves to /EFI/ORBS, for this shell
$ unmount                    # drop the mounted filesystem

Every mount and bind changes only this shell's namespace, and the commands it spawns inherit it — a per-task view, never a global one. Preparing a blank disk from inside the guest: erase disk, then partition fat32 (or exfat/ext2), then format fat32, then mount -a.

Cluster: sharing resources across machines

This is what Ouroboros is ultimately about: several machines sharing their resources as one system — one machine reads another's disk, inspects its processes, writes its screen, or runs a program on it — over a from-scratch 9P-style protocol carried on the network. It is the Plan 9 model: "remote" is just the same file protocol over TCP instead of local IPC, so the commands you already know work across machines with no new syntax. Only where a path points changes.

Trust, stated plainly

Every machine has its own Ed25519 keypair and authorizes its peers by public key — the known_hosts model, not a shared password. The private half lives in /etc/cluster/id (mode 0600); peers are listed one per line in /etc/cluster/authorized. Every request is signed, and so is every reply, so the exchange is mutually authenticated: a client checks the reply against the key it expects for the address it dialled.

A held session is keyed (since v0.21.0): its first request carries an X25519 key inside the signature, and every later message on that connection carries an HMAC tag and a sequence number instead, about a seventeenth of the crypto cost and not replayable. An older node answers without a key and the session stays signed, so a mixed cluster works.

It is integrity, not secrecy — bytes cross the wire in cleartext, so a passive sniffer reads your files and can replay a one-shot request. Encryption, replay protection for one-shot requests and per-user keys are gated behind a "leaving a trusted network" trigger. Do not expose an Ouroboros export to a hostile network yet.

This replaced a shared secret. The difference that matters is revocation: with one secret every member was interchangeable and removing one meant re-keying all of them; now you delete a line. It fails closed in both directions — a machine with no /etc/cluster/id cannot sign, so its export is closed and it can make no remote requests; a machine with no authorized peers serves nobody. A \NOEXEC flag file is the one other knob: share the disk, refuse remote execution.

Who is asking, not just which machine. Each request also carries the name of the user making it, inside the signature so it cannot be altered without the machine's private key. The far side resolves that name through its own /etc/passwd and applies its own permissions, refusing a name it does not know. Two consequences: both machines need an /etc/passwd and an account is matched by name (uid numbers need not match, which is why the name travels rather than the number); and supplementary groups do not cross: a remote caller is judged on uid and primary group only. This is still auth at the machine level: a machine the cluster authorizes can claim any user who has an account on the exporting machine, so it protects you from the users of a machine you trust, not from a machine you don't. Root is the exception (root squash): a claim resolving to uid 0 or gid 0 is refused unless the exporter's authorized line for that peer ends with root, so a compromised machine is root only where it was trusted with root. Per-user keys, which close the rest, are planned.

The built-in HTTP server is anonymous and unprivileged. netd also serves the disk over plain HTTP on port 80, which has no user model at all — so it reads files as nobody, never as root: world-readable files are served, anything else gets 403. On FAT32 and exFAT, which record no mode, everything remains readable. It requires no key and no account; do not treat it as private.

The QEMU images stage a dev identity automatically, so the two-VM targets authenticate out of the box. Their authorized also lists intruder, a test identity no machine holds, which the impersonation gate signs as. On a real machine: run clusterkey new on each, clusterkey line <name> <ip> on one to print the line the other should hold, and paste it in. clusterkey peers lists what a machine currently accepts and flags a line that cannot work. Key generation refuses without real entropy rather than producing a guessable identity.

Setting up two machines

The dev setup is two QEMU guests on a shared virtual link — make run-image-2vm-a in one terminal, make run-image-2vm-b in another; see testing-qemu.md for the exact commands. Machine A is 10.0.2.10 and B is 10.0.2.11; each netd runs a 9P export on TCP port 564 automatically, so every machine is ready to share the moment it boots. Everything below is typed at machine B, reaching machine A.

Sharing a disk — mount -r

$ mount -r 10.0.2.10:564 /mnt/a     # bind A's export at /mnt/a
$ ls /mnt/a                         # A's disk root
$ cat /mnt/a/EFI/ORBS/INIT.CFG      # read a file on A
$ mkdir /mnt/a/reports              # create on A's disk
$ write /mnt/a/reports/note.txt hi  # write to A's disk
$ cp /BIN/LS /mnt/a/ls.copy         # stream a local file onto A (chunked)

The mount is per-shell and inherited by spawned commands. Read and write both work (single-writer by convention — there is no distributed lock yet). If A disconnects, the next operation on /mnt/a fails cleanly rather than hanging.

Another machine's processes, screen, and address

/proc, /dev/cons and /net are synthetic filesystems — the task table as files, the console as a writable file, and the machine's IPv4/MAC as read-only files. Because netd's export serves them too, reading another machine's is just a longer path:

$ mount -p /proc                    # your own process table
$ cat /proc/cpus                    # how many cores take tasks
$ cat /proc/2/state                 # runnable | blocked | zombie | unused
$ cat /proc/6/core                  # the core executing task 6 right now, or none
$ ls /mnt/a/proc                    # MACHINE A's task slots
$ cat /mnt/a/proc/2/state           # A's filesystem-server state

$ mount -c /dev/cons                # your console as a file
$ echo hello > /dev/cons            # ...prints on this screen
$ write /mnt/a/dev/cons "ping from B"   # prints on MACHINE A's screen

$ mount -n /net ; cat /net/ip       # your own address (10.0.2.11)
$ cat /mnt/a/net/ip                 # MACHINE A's address -> 10.0.2.10

Running a program on another machine — cpu

cpu <host:port> <command> runs the command on the remote machine's CPU and streams its output back — and, crucially, the command reads your files, through your namespace imported at /host. That is the Plan 9 cpu model: ship the computation to another machine while its data stays yours.

$ cpu 10.0.2.10:564 ls /            # run `ls` ON machine A; see A's disk root
$ cpu 10.0.2.10:564 uptime          # run `uptime` on A; A's uptime
$ cpu 10.0.2.10:564 ls /host        # /host is THIS machine - YOUR root
$ write local.txt hi
$ cpu 10.0.2.10:564 cat /host/local.txt   # cat runs on A, reads YOUR local.txt

So within one remote command, / is the machine it runs on and /host is the machine you launched it from. The command's program is loaded from the remote's disk, so both machines should have the same /bin. Output up to about 2 KB comes back whole; larger output is still bounded, with truly unbounded streaming a documented later refinement.

Using another machine's network — /net/tcp

A machine can open a TCP connection out of another machine's NIC, and equally accept connections on it. Both are Plan 9's /net/tcp connection files — read clone for a connection number, write connect or announce to its ctl, then read and write its data. The dial and serve commands drive the whole sequence:

$ mount -n /net                              # bind THIS machine's /net
$ dial /net 93.184.216.34 80 GET / HTTP/1.0  # dial out of OUR nic
$ mount -r 10.0.2.10:564 /mnt/a              # mount machine A's export
$ dial /mnt/a/net 93.184.216.34 80 GET / HTTP/1.0   # dial out of A's nic

$ serve /net 9000 hi there         # answer clients that connect to US
$ serve /mnt/a/net 9000 hi there   # answer clients that connect to A

The only thing that changes is the base path: /net is your network, /mnt/a/net is A's. So dial /mnt/a/net reaches the web through A's connection, and serve /mnt/a/net makes a program here answer clients who connect to A's address — A lends its ingress, this machine owns the service. Both are scoped to small transactions: a TCP client, and a listener with a couple of concurrent connections.

Syscall ABI

Full, current per-syscall detail: architecture.md's syscall table; the authoritative constants live in syscall-abi/src/lib.rs and ninep-abi/src/lib.rs, imported by both the kernel and every userland program. The table below covers the core syscalls only — the filesystem protocol has since moved to the uniform ninep-abi verb set (NP_READDIR/NP_READ/NP_WRITE/…) and the network and cluster operations to the NETOP_* protocol, both documented in those crates.

Calling convention (Linux-shaped, not Linux-compatible): syscall number in x8, up to four arguments in x0–x3, return value in x0, via svc #0. Pointer/length arguments are bounded per buffer, and longer buffers are rejected rather than truncated. 512 (MAX_USER_LEN) is the default bound, not a universal one: the message syscalls take 768 (MSG_MAX_LEN), and each staging call is bounded by its own published maximum instead (argv 512, cwd 128, namespace 256, environment 2048). An out buffer for reading one of those stores back is the exception: no capacity is too large, since the copy is at most one entry. Every buffer must also lie inside the caller’s own loaded region (for such a read, the bytes it copies), which is the check that carries the safety.

Error convention: all failure codes live in a reserved top band of u64: any return value >= FS_ERR_MIN (u64::MAX - 43, and it moves down each time a code is reserved - check syscall-abi, never this sentence) is an error; everything below is a real result (byte counts, sizes, exit statuses). NO_FS (MAX-1) means no filesystem is mounted this boot.

Writing your own program

The full guide is processes.md; programs/demos/hello is the working template. The essentials: #![no_std]/#![no_main], _start placed first in .text via .text.start, syscalls via inline svc #0, release builds only, and no static mutable state (the linker script asserts .data/.bss empty — keep state in main's stack frame). Stage the stripped ELF on the ESP and either point INIT.CFG at it or run it from the shell.