From "it doesn't boot" to a real console: an ARM64 kernel bring-up postmortem

This is a write-up of the earlier stretch of work getting Ouroboros, a from-scratch ARM64 microkernel, from "UEFI hands us control" to a genuinely stable, interactive foundation: exception handling that survives a real fault, an MMU running on the kernel's own page tables instead of firmware's, a timer-driven preemption tick, a real EL0/syscall boundary, and — the part that took the most real-hardware round trips — an actual console on real Parallels-on-Apple-Silicon hardware, after three separate discovery mechanisms and one whole driver turned out to be dead ends.

Like the companion pieces above, this is kept separate from the project's own internal history because most of what's here isn't really Ouroboros-specific. If you're bringing up your own bare-metal ARM64 kernel and something works fine under an emulator but not on real hardware, or your exception vector table faults the instant you jump to it, or your MMU switch hangs forever right after you flip the translation table base register — there's a real chance one of these is your bug too.

The recurring shape of these bugs

Almost everything below falls into one of two buckets:

• "It compiles and it should work" wasn't good enough. Several of these were verified correct by hand, cross-checked against Linux's own kernel headers, and still didn't work — because the actual bug was one level below where the reasoning was happening: a linker section attribute, a compiler's alignment ceiling, a hypervisor's device emulation policy.
• An emulator is a permissive host; real hardware is not. A hardcoded address that's merely wrong on an emulator quietly does nothing. The same wrong address on real hardware can take an entire VM down. More than one fix below exists specifically because "confirmed on the emulator" turned out not to mean "confirmed."

Exception vectors: a linker section name that silently wasn't executable

Symptom: a minimal AArch64 exception vector table, installed right after leaving UEFI boot services, faulted the instant any exception tried to reach it — including the deliberate test fault used to verify it worked at all.

Everything looked right: the table linked without error, the vector base register read back exactly the address it was set to, the assembly itself was correct. And yet:

ESR_EL1 decoded: EC 0x21 (Instruction Abort, same EL)
                 IFSC 0x0F (Permission Fault, level 3)

The page the vector table lived on existed and was mapped — just not executable, which is a permissions fault a page-not-present bug would never produce, and easy to misread as "my MMU setup is wrong" rather than "my linker script is wrong."

Root cause: the vector table's assembly lived in a custom section name, on the reasonable assumption that a name starting with the standard code-section prefix would inherit the usual executable treatment. It doesn't, at least not reliably on this project's PE/COFF-targeting toolchain: the backend infers executable-section characteristics from exact, recognized section names, not prefixes. An unrecognized custom name gets emitted as a plain, non-executable data section — readable, present at the right address, and unusable as code.

Fix: use the literal standard section name directly, not a custom one.

How this was actually verified, not just "believed fixed": a deliberate fault at a known-unmapped address, cross-referenced against the emulator's own internal interrupt trace. Before the fix, the trace showed the correct initial dispatch into the vector table, immediately followed by an endless identical fault at the vector table's own address — a fault loop, since jumping into a non-executable handler immediately re-faults. That's the same failure shape separately observed taking down real hardware entirely (see below) — a genuinely useful pattern: an infinite tight fault loop, rather than one reported fault, often means the handler itself is unreachable.

The MMU switch: matching firmware's own translation-table starting level

Symptom: replacing firmware's translation tables with the kernel's own — same technique either way, just swapping which tables the base register points at while the MMU stays continuously enabled — hard-faulted immediately after the switch: a permission fault at translation level 2 on the very next instruction, then an identical fault on the exception vector table itself, looping forever.

The first version used a single top-level table sized for a 39-bit input address space — architecturally legal, and every table entry and every relevant attribute bit was hand-verified against Linux's own page-table headers, independently re-derived from the real runtime register values with a throwaway decode script. Removing execute-permission bits entirely changed nothing. All the reasoning checked out; the kernel still faulted the instant it ran under its own tables.

Root cause, confirmed by direct A/B test with everything else held constant: firmware had configured its own tables for a larger, 44-bit input address space, which requires an extra table level before the one this kernel had built. Switching to a table hierarchy of a different depth than firmware's own — while the MMU stayed continuously enabled the whole time, no disable/re-enable step in between — appears not to be tolerated, at least on this CPU model, even when every individual attribute bit is correct.

Fix: match firmware's starting level exactly, rather than simplifying to a smaller, "sufficient" hierarchy. The rest of the original reasoning had been right all along.

An honestly unresolved mystery: EL0 couldn't share a page with running EL1 code

Not every bug in this arc got a clean root cause — and that's worth recording too, since routing around an unexplained failure is sometimes the right call, not a failure of diagnosis.

The goal: let user-mode code execute out of the same RAM block the kernel's own privileged code was actively running from, just with the page permissions widened to allow user-mode access too.

What happened: an immediate, identical-looking hard fault — a permission fault on the very first instruction fetch after the table switch, in exactly the same shape as the bug above.

What was ruled out, each by a direct, repeatable test, not assumption: the permission bit's exact position (cross-checked three separate ways); an execute-never bit (removing it entirely changed nothing); a newer architectural protection feature (this CPU is too old to implement it at all — confirmed directly by attempting the raw register access and getting an "unknown instruction" trap, not a permission effect); shareability attributes and instruction-cache invalidation; and page granularity (rebuilding the same region at a finer block size changed nothing). A read-only control case correctly produced a data abort instead of an instruction abort — solid evidence the permission bits were being interpreted correctly in general; only this one specific combination faulted, which architecturally shouldn't be possible at all.

Resolution, not explanation: user-mode code was moved into a genuinely separate, dedicated memory region, isolated from any actively-executing privileged code, with its own page-table entries. That worked immediately and has been reliable ever since. The underlying "why" was never found.

GIC and the timer tick: confirmed addresses, not textbook ones

Bringing up a periodic preemption tick needed real hardware addresses for the interrupt controller. Rather than use commonly-quoted addresses from memory or documentation, the actual values were pinned down by dumping the specific emulator build's own internal hardware description and decoding it directly — the same discipline that turned out to matter enormously for the console-discovery work below. An address "everyone knows" for one specific emulator build is still a guess, not a fact, the moment you're not on that exact build or that exact hardware.

The interrupt path itself needed a real architectural change, not just new addresses: every exception vector before this one only ever needed to report and halt — diverge, never return. A periodic tick has to resume whatever it interrupted, which meant a full register-save/restore trampoline and a normal function call into the handler instead of a diverging jump. Verified as genuinely sustained, not just "printed once": a 20-plus-second run producing correctly-spaced ticks with no drift or corruption, cross-checked against the emulator's own internal exception trace showing zero aborts across the entire run.

A compiler/toolchain ceiling, found by bisection: 8KB, not 2MB

Giving user-mode code its own isolated region meant that region needed to be a compile-time-aligned static — and the original plan was a comfortable, round 2MB alignment. That triggered a genuine compiler crash on this specific PE/COFF-targeting toolchain. Rather than accept "2MB doesn't work" as the final answer, the actual limit was found by bisection: section alignment tops out at exactly 8KB — not a round number, not documented anywhere obvious, just the real ceiling. The isolated region was sized to fit under it.

(This ceiling later disappeared on its own, as a side effect of an unrelated design change — once user programs were loaded from disk at runtime instead of compiled into the kernel image, they no longer needed a compile-time-aligned static at all. A small reminder that a hard-won constraint can become moot for reasons that have nothing to do with solving it directly.)

A quieter bug: idle-wait traps back to the kernel by default

Once user-mode code could actually run, its idle loop immediately trapped back into privileged mode instead of actually idling. Root cause: a control register default that makes a lower-privilege idle-wait instruction trap up, rather than execute directly — a control completely unrelated to page tables or the isolation work happening at the same time, easy to miss if you assume every user-mode fault must be a permissions issue. Diagnosed directly from the trapped exception's own cause code, and fixed by clearing the relevant control bits before ever dropping to user mode.

Console discovery: three real mechanisms, three confirmed dead ends, in order

With the kernel itself stable, the next real problem was output: how do you find a serial console on hardware you don't control the description of? Three standard discovery mechanisms were tried, in order, each logging exactly why it failed before the next was attempted — genuinely useful discipline, since "no console found" alone tells you nothing about which of several possible causes is real.

1. A devicetree blob, looked up via the standard UEFI configuration table. Confirmed dead on both the emulator and real hardware — both report none at all, because both platforms describe their hardware via ACPI instead.
2. ACPI's serial port console table (a hand-rolled parser — not a full ACPI library, since this only needs a handful of fixed-offset struct reads). Confirmed working on the emulator, first try — real discovery, resolving the exact address a hardcoded fallback used to assume. Confirmed dead on real hardware, and not from a parsing bug: the higher-level ACPI tables both parsed successfully there too (so ACPI itself is present and well-formed), there is simply no console table entry among the platform's own tables at all — tested with and without a serial port explicitly configured in the VM's own settings, no difference either way.
3. PCI enumeration for a classic UART-family serial controller — a completely different piece of hardware from what the other two mechanisms look for. Confirmed dead on both platforms too.

A real, hard-earned lesson living inside this discipline: there used to be a hardcoded fallback address, used whenever discovery failed, on the theory that a wrong guess beats no console at all. It was removed after direct confirmation that it hard-crashes real hardware — nothing is mapped at that emulator-shaped address on the real platform's virtual chipset, so the write faults, and with no exception vectors installed yet at the point this ran, that fault had nowhere to go and took the entire VM down.

virtio-console: a real, working driver — confirmed not to be the answer here

Research suggested this class of Apple Silicon virtualization exposes its devices via the same convention a common open-source emulator uses, and that a guest should expect a virtio-based console rather than a classic UART. A full driver was built against this lead and confirmed working end-to-end on the emulator — every boot message reaching a real host-side output, confirmed byte-for-byte.

Confirmed, directly, not to be what real hardware uses. A permanent bus-enumeration diagnostic was built specifically to answer this on real hardware, and it found real virtio devices present — but only for networking, never anything matching a virtio-console device over either transport this project can drive. The one remaining unexplained device on the bus carries the platform vendor's own PCI vendor ID with an unclassified device class — almost certainly the real serial mechanism, and almost certainly proprietary and undocumented. Reverse-engineering an undocumented device was considered and explicitly declined as open-ended with no guaranteed payoff — a real, deliberate scope decision, not an oversight.

The framebuffer console: the real answer, and four more real hardware bugs

With every byte-stream mechanism confirmed dead, the fallback that actually worked was a standard, fully-specified graphics protocol needing no address guessing at all — query the current display mode and write text directly into the framebuffer. Getting this working on real hardware took five real-hardware test rounds and found four more real, independently-confirmed bugs.

Bug: opening the display protocol exclusively silently disconnects firmware's own console

Symptom: on the first real-hardware test, boot progressed normally right up through device enumeration — and then simply stopped. No crash, no further output, not even this module's own unconditional success/failure log line.

Root cause: the graphics protocol was being opened in exclusive mode, which — per the protocol's own documented semantics — forcibly disconnects any other driver currently holding it. Firmware's own boot-time text console was still holding that exact protocol to render the very screen the test was being watched on, so the instant this code ran, it silently disconnected the visible console before a single further line could print. An emulator-based test never caught this, because that test ran fully headless — there was no console driver attached in the first place, so nothing to disconnect.

Fix: open the protocol read-only, non-exclusive — this code only ever needs to query the current mode and framebuffer address once.

Bug: a later discovery step froze the boot before any console existed to report through

Once the bug above was fixed, discovery and mapping worked — but the boot froze solid before reaching anything visibly working. Reordering the fallback chain (try the now-proven framebuffer path before a still-unproven one) didn't just work around this — it got a console installed before the freeze, which meant the next attempt could finally see what was actually happening.

Bug: a confirmed real bus fault reading a fixed, unconfirmed device address

With a console now live to report through, the exact failure became visible directly:

ESR_EL1 decoded: EC   0x25  (Data Abort, same EL)
                 DFSC 0x10  (Synchronous External abort —
                            a real bus fault, not permission/mapping)
FAR_EL1: matched a fixed device-discovery scan address exactly

Root cause: a device-discovery scan reads a fixed set of addresses looking for a "magic value" marking a populated device slot — a convention confirmed correct on the emulator by directly peeking those addresses through its own monitor interface before any driver code was written. What had never been verified was what an unpopulated slot does on hardware that isn't the emulator: the working assumption — never actually tested — was "reads back as zero." On real hardware, the very first probe address instead produced a genuine external bus abort. Real hardware objects loudly to a read targeting nothing, where the emulator's software model just returns a pattern as a courtesy.

Fix: gate every caller of this scan behind a heuristic — only attempt it on platforms where a byte-stream console was already found through one of the confirmed-safe discovery mechanisms above, the only platform shape this scan has ever actually been proven safe on.

Bug: the identical fault, at a different address — a structural finding, not a second instance

Once the scan above was gated off, the boot progressed further and hit a second, differently-addressed instance of the exact same fault signature — this time matching the interrupt controller's own fixed, "well-known for this emulator" base address.

The real significance: this wasn't just a second bug to patch — it was proof that the entire convention of trusting a fixed, low-address, emulator-shaped location for any device is unsafe on real hardware, not just the one device that happened to be tried first. The timer itself turned out to be architecturally exempt — it's accessed purely through CPU registers, never memory-mapped I/O at all — but the interrupt controller depended on the same unconfirmed convention.

Fix: broadened the same safety gate to cover interrupt-controller and timer setup as well, accepting no preemptive multitasking on this platform as a real, known, documented consequence.

The payoff: the very next real-hardware test reached the actual goal of this entire arc — a live, rendering, prompt-displaying shell on real hardware, for the first time in the project's history.

Closing the preemption gap: real ACPI interrupt-controller discovery, and WFE strikes again

The paragraph just above accepted "no preemptive multitasking on this platform" as a real, documented consequence of gating the interrupt controller off entirely. That gap got closed in a later session — and closing it needed one more real driver, plus a second, independent encounter with the exact same instruction (wfe) that already caused trouble once in this document, in a completely different way the second time.

Why the old GIC address could never be the real fix. The interrupt controller addresses this project had been using were pinned down by dumping one specific emulator's own internal hardware description — the same QEMU-shaped convention a real bus fault, decoded above, already proved unsafe on real hardware. Guessing better addresses wasn't the fix; discovering them genuinely was. ACPI has a standard table for exactly this — the MADT (the historical name for a table whose actual on-disk signature is "APIC") — the same portable-discovery role this document's ACPI SPCR parser already plays for the console. A new parser walks the MADT's own interrupt-controller structures and reads the platform's real interrupt-controller version and addresses directly, no emulator-specific dump required.

A second real complication: the two platforms don't even run the same interrupt controller generation. The GIC version already in use (GICv2) is memory-mapped throughout — a distributor, a CPU interface, done. Real Apple Silicon virtualization almost certainly runs GICv3 instead, architecturally different enough to need a genuinely separate driver: the CPU interface moves entirely into CPU system registers, per-interrupt enable for the timer's own line moves from the shared distributor to a per-CPU redistributor region that has to be located and explicitly woken up first, and even finding which redistributor belongs to this CPU needs a real discovery walk rather than assuming the first one found is the right one — the same "discover, don't assume" discipline this project had already learned the hard way once, in the keyboard postmortem's own "wrong physical device" bug.

Two real bugs, both found on the safe, fast, iterable platform before ever spending a real-hardware round trip on them. Forcing the emulator itself into the newer interrupt-controller generation (a one-line configuration flag it already supports) let both get found and fixed cheaply: the newer controller's global enable register needs more than the one bit its predecessor used (enabling only that bit silently routes interrupts to the wrong internal delivery group — no fault, no crash, the timer just silently never reached the handler, isolated only by noticing a supposedly-live tick counter had stayed frozen at zero); and each interrupt line also needs an explicit per-line assignment to that same group, at the newly-relevant per-CPU redistributor level this generation introduces.

On real hardware, the interrupt-controller discovery itself worked cleanly, first try — a genuinely different, platform-specific address from the emulator's, no crash, no hang. Timer interrupts reaching the CPU and being correctly acknowledged was separately, conclusively confirmed too: a real, live, continuously increasing tick count, verified by deliberately disabling just the next piece and confirming the count kept climbing on its own.

Then the second WFE bug showed up — not the same one as before, a new instance, one level deeper. This document's own earlier section already covered wfe trapping to the wrong exception level by default — a config-register fix inside this kernel's own control. This time, enabling the actual task-switch (not just the interrupt delivery that drives it) hung the system completely, on real hardware only, the very first time that exact code path had ever run there — no exception, no crash report, just silence. A single-variable check (leaving interrupt delivery fully active while skipping only the switch itself) proved the interrupt path was completely fine on its own; the hang was specifically in resuming a second, previously-idle task, whose own loop was, again, wfe — and this time the fix wasn't a register bit this kernel controls at all. The leading explanation, still not fully proven: a real hypervisor sitting above this guest kernel may intercept or emulate wfe in a way that never faithfully reproduces on the software emulator this project develops against day to day — a decision this kernel has no visibility into, one exception level higher than the register this document's earlier wfe fix ever had to touch. Swapping that one idle loop for a plain busy-wait made the hang disappear completely, confirmed by a sustained, multi-command interactive session with a correctly climbing tick count throughout.

Techniques that generalized well

• When a fault is a permission fault, not a translation fault, look one level up the stack — a linker section attribute, a page permission bit, something about the mapping rather than whether it exists at all.
• A fault loop usually means the handler itself is unreachable, not that the underlying condition is unusual — worth recognizing as its own distinct failure shape.
• Re-derive register values independently before trusting them, even ones already checked once — a second, differently-sourced derivation catches transcription mistakes a single careful read-through won't.
• Don't trust an address "everyone knows" for one specific emulator build. Dumping and decoding that exact build's own hardware description, every time, is what actually made the addresses reliable.
• A permissive host and a strict one will disagree specifically about invalid accesses. Treat "confirmed on the emulator" as real but incomplete, not final.
• When you genuinely exhaust the documented explanations, changing the shape of the problem is a legitimate resolution — and worth writing down as an unresolved mystery rather than quietly implying it was understood.

Where this ended up

A kernel that survives a real fault and reports it usefully instead of just disappearing; an MMU running on its own tables, matching real discovered RAM rather than a hardcoded range; a genuine preemption tick; a real user-mode/syscall boundary; and, after three dead-end discovery mechanisms and one dead-end driver, a real, live, interactive console on real Parallels-on-Apple-Silicon hardware — the actual foundation everything in the later work was built on top of: turning that bare prompt into a real, disk-backed userland shell, real keyboard input reaching it, and — the last major piece, closed in a later session covered in this same document above — real, preemptive multitasking on that same real hardware, not just the emulator.