Developer Guide

This guide captures the current build and run workflow for KoraOS. It focuses on the LLVM/Clang toolchain, the CMake-based build system, and how to exercise the kernel under QEMU and on real Raspberry Pi hardware.

Toolchain Overview

KoraOS is built with the LLVM toolchain while targeting aarch64-none-elf. The CMake build system generates compile_commands.json, which keeps clangd and other tooling in sync with the cross-compilation flags.

Required Packages

• LLVM/Clang (including clang, ld.lld, and llvm-objcopy)
• CMake 3.20 or newer
• Python 3 (shared-build coordination and regression tooling)
• mtools (mformat, mcopy), used to build the embedded FAT32 filesystem image; the build fails without it
• QEMU (only for virtualization workflows)
• Optional: GNU Make (the legacy Makefile is still functional)

Installing the Toolchain

macOS (Homebrew):

brew install llvm cmake qemu mtools
echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zprofile

Ubuntu/Debian:

sudo apt update
sudo apt install clang lld llvm llvm-objcopy cmake qemu-system-aarch64 mtools

After installation, confirm the tools are available:

clang --version
ld.lld --version
llvm-objcopy --version

Generating compile_commands.json

Run ./build.sh at least once. CMake writes build/compile_commands.json, which clangd automatically discovers via the repository’s .clangd configuration.

Building and selecting targets

./build.sh --target qemu_raspi3b
./build.sh --target qemu_raspi4b
./build.sh --target qemu_virt
./build.sh --target hw_raspi3b --release
./build.sh --target hw_raspi4b --release
./build.sh --target hw_raspi3b --target hw_raspi4b --release
./build.sh --target all

No target argument means all five. Repeated selections are deduplicated in first-seen order. --debug (default) or --release applies to selected kernel configurations. Each has its own cache and kernel.elf, kernel.img, and kernel.map in build/<debug|release>/<target>/; --build-dir sets the root. The shared userfs producer owns build/userfs/aarch64/user/*.elf and build/userfs/aarch64/koraos.img, independent of the selected board and kernel configuration. It keeps the current AArch64/O2/PIE userland flags.

--clean rebuilds selected configurations and the owned shared userfs, preserving unselected configurations and explicitly supplied userfs. Invalid arguments are rejected before cleanup. clean removes selected generated configurations without rebuilding. The shared producer is serialized against another build invocation and publishes its image atomically, keeping the previous valid image if generation fails. Avoid running kernel consumers while replacing their image through an unrelated manual command.

# Build the shared artifact alone; consume it explicitly (as CI does)
./build.sh --userfs-only
./build.sh --target qemu_virt --userfs-dir build/userfs/aarch64

A supplied --userfs-dir is a prepared artifact: the caller controls its contents and freshness. Its koraos.img must exist and be nonempty. CI produces it from the same checkout and downloads it into each consumer job.

The first selected kernel’s compilation database is copied to build/compile_commands.json, matching .clangd. A custom build root needs a matching clangd setting. Select the development target first for multi-target builds.

Running QEMU

./run-qemu.sh --target qemu_raspi3b
./run-qemu.sh --target qemu_raspi4b
./run-qemu.sh --target qemu_virt
KORA_QEMU_FB=1 ./run-qemu.sh --target qemu_virt
./run-qemu.sh --target qemu_virt --release --build-dir my-build -s -S
./run-qemu.sh --target qemu_virt --smp 4

--smp N (or KORA_QEMU_SMP) gives virt 1 to 4 cores (default 1); QEMU raspi3b and raspi4b always have four.

qemu_raspi4b is the Pi 4 kernel built for QEMU’s raspi4b machine (QEMU 9.0 or newer): the GIC-400, the PL011 console, the framebuffer and the SD card on EMMC2, with USB left out (QEMU has no Pi 4 PCIe/xHCI). It covers the Pi 4 code paths that hw_raspi4b otherwise only meets on real hardware. KORA_QEMU_SD=IMAGE puts an SD card in either raspi machine’s slot, through a QEMU snapshot unless KORA_QEMU_SD_WRITABLE=1; on raspi3b, KORA_QEMU_USB=IMAGE plugs in a USB drive the same way (KORA_QEMU_USB_WRITABLE=1). See smp.md. --gic 3 selects a GICv3 for virt (default 2).

On an Apple Silicon Mac, --accel hvf runs virt on the host’s own cores through Hypervisor.framework (-cpu host; hvf needs a GICv3, so it implies --gic 3). The cores really run in parallel, with the CPU’s weak memory ordering, which TCG emulation does not reproduce: use it for anything touching SMP and locking.

./run-qemu.sh --target qemu_virt --accel hvf --smp 4
tests/run-qemu-smoke.py --target qemu_virt --gic 3 --accel hvf --smp 4

The launcher uses the matching configuration’s kernel.img and forwards extra QEMU arguments. Ctrl-A X quits. KORA_QEMU_DISPLAY overrides the native display choice (cocoa on macOS, gtk on Linux). The build and launcher share target names, build roots and configuration flags.

Raspberry Pi deployment

./build.sh --target hw_raspi3b --target hw_raspi4b --release --install-to /Volumes/BOOT

All selected builds must succeed before installation starts. Only selected hardware targets are installed, with firmware, DTBs, config and licence. Existing boot filenames remain kernel8-rpi3.img and kernel8-rpi4.img. Building alone does not copy anything to an external volume. Low-level hardware configurations retain install_hw, using their cached BOOTMNT destination.

# Package already-built hardware targets without mounting or root privileges
./create-sd-image.sh --target all --release --output build/koraos-boot.img

Verifying the Image

List the files on the boot volume to ensure kernel8-hw.img is present, eject the volume safely, and boot the Raspberry Pi. The kernel prints K on the serial console, followed by its boot log.

On hardware the serial console and the user console are separate. The UART carries the kernel’s diagnostic log (printf, console_log); the shell runs on the HDMI screen and reads the USB keyboard: through the DWC2 controller on the Pi 3, and through the VL805 USB 3 (xHCI) controller behind PCIe on the Pi 4, on any of its four type-A ports (USB 2 or USB 3), directly or through a hub.

USB limitations: devices are enumerated once at boot, so plug the keyboard and any USB drives (or a hub with them) in before powering on; there is no hot-plug yet. On the Pi 4 only the four type-A ports are supported (the VL805 xHCI controller); the USB-C port is a different controller (DWC2) that the Pi 4 build does not include, and it normally powers the board anyway. On the Pi 3 a keyboard behind an extra hub does not work yet: low/full-speed devices behind a High Speed hub need split transactions, which the DWC2 has to schedule in software (the Pi 4’s xHCI does this in hardware). Keyboard auto-repeat is not enabled yet.

Typing on the serial line talks to the kernel debug console (koraos> , src/console.c), which runs alongside the shell: it is fed from the UART interrupt and from the shell’s idle loop, not a task of its own, so its commands must not block. Press Ctrl-T to hand the serial line to the screen terminal as a fallback keyboard (what you type is then echoed on the screen, not on the serial line), and Ctrl-T again to return. Besides help and version, irqs shows the interrupt controller in use (the legacy BCM controller on the Pi 3, the GIC-400 on the Pi 4), the system tick count next to the uptime (about 100 ticks per second when interrupts work), and how often each connected IRQ has fired.

Under QEMU the serial line starts on the screen terminal and the shell is mirrored to the UART as before; Ctrl-T reaches the debug console. QEMU’s raspi3b models the Pi 3’s DWC2 USB controller, so the raspi3b kernel enumerates there too (QEMU’s usb-kbd and usb-storage devices work); raspi4b has no PCIe, so no Pi 4 USB. The keyboard layout defaults to US; pick another of Circle’s keymaps at configure time with -DKORAOS_KEYMAP=IT (or UK, DE, FR, ES, DV).

The screen terminal understands the xterm escape sequences that the xterm-256color terminfo entry uses: cursor movement and addressing, erase/insert/delete, scroll regions, SGR attributes with 16, 256 and 24-bit colour, the alternate screen, save/restore cursor, tab stops, DEC line drawing (and UTF-8 box characters), and the status/size reports. termdemo shows most of it. Shift+PgUp / Shift+PgDn scroll back through the history (TERM_SCROLLBACK_LINES, default 1000); new output returns to the live screen. The cursor is a block by default; set TERM_DEFAULT_CURSOR in include/video/term.h to TERM_CURSOR_UNDERLINE or TERM_CURSOR_BAR (programs can also change it with ESC [ n SP q).

Tests

Pure-logic kernel code has host-side unit tests that run on your machine under AddressSanitizer/UndefinedBehaviorSanitizer, with no QEMU or hardware:

tests/run-host-tests.sh

A QEMU smoke test boots the kernel on raspi3b and drives the shell over serial, checking the output and framebuffer screenshots:

./build.sh --target qemu_raspi3b
tests/run-qemu-smoke.py --target qemu_raspi3b

See tests/README.md for what each covers and how to add tests.

Further Documentation

• filesystem.md – FAT32 volumes, reads/writes, the ramdisk and disposable write testing.
• how-userland-works.md – what happens when a program is loaded, where it lives in memory, how processes coexist, and the (deliberate) lack of memory protection.
• syscalls.md – the full system-call ABI.
• writing-userland-programs.md – how to write, build, and run a userland program (no compiler or libc on the device yet).

QEMU virt development profile

./build.sh --target qemu_virt
KORA_QEMU_FB=1 ./run-qemu.sh --target qemu_virt
KORA_QEMU_DISK=build/userfs/aarch64/koraos.img ./run-qemu.sh --target qemu_virt

The supported profile is one cortex-a72 CPU, GICv2 and TCG, with 256 MiB RAM below 4 GiB. KORA_QEMU_RAM=128M changes RAM. The kernel discovers RAM/device locations from QEMU’s DTB; use raw kernel.img for boot and kernel.elf for symbols. Pi GPIO, mailbox and Circle USB are excluded from virt.

ramfb provides a 1024x768 XRGB8888 screen and fb_info; KORA_QEMU_RAMFB=0 omits it for serial fallback. The VirtIO keyboard uses a US keymap with Shift/Ctrl/Caps, editing/navigation and Shift+PgUp/PgDn scrollback. KORA_QEMU_KEYBOARD=0 omits it. UART mirrors shell output and Ctrl-T switches to the debug console. Pi USB layouts still use the Circle keymap setting.

An external disk uses modern VirtIO MMIO and defaults to read-only media. KORA_QEMU_DISK_READONLY=off enables FAT32 file writes on a supplied disposable image; the embedded ramdisk stays read-only. Never enable writes on the shared build/userfs/aarch64/koraos.img artifact or a card containing Pi firmware. A missing disk uses embedded userfs; a configured broken disk does not silently fall back. See filesystem.md.

For an end-to-end write test, install dosfstools (fsck.fat) alongside mtools:

./build.sh --target qemu_virt
tests/run-qemu-write.py

This creates a fresh 64 MiB scratch root, runs EL0 writes and quoted file tools, then verifies it with fsck and exact mtools readback after the VM closes. The immutable shared image hash is checked before and after the run. cp creates new destinations only; mv does not overwrite or move across volumes. All mutating tools sync before reporting success. Use the shell’s sync command to flush changes made by other programs. Names use UTF-8, while interactive console input currently accepts ASCII; quotes handle spaces in ASCII paths.

SD card (Pi 3, Pi 4, QEMU raspi3b and raspi4b)

The Pi kernels read the SD card through the SoC’s SDHCI controller (the Arasan controller on the Pi 3, EMMC2 on the Pi 4): the card’s FAT32 partitions show up in volumes next to the ramdisk, which stays the boot volume. On hardware that is the boot card, so its KORAOS partition becomes browsable. The card is read-only unless the kernel is built with --storage-writable (alias --sd-writable); keep it that way for a card with Pi firmware on it.

In QEMU, tests/create-sd-card-image.py makes a disposable card (64 MiB is presented as a standard-capacity card, 4096 MiB as SDHC) and the smoke test attaches it with --sd, read-only through a QEMU snapshot (on raspi4b to EMMC2, where the Pi 4’s slot is):

tests/create-sd-card-image.py build/userfs/aarch64/koraos.img --output build/sd-card.img --size-mib 4096
tests/run-qemu-smoke.py --target qemu_raspi3b --sd build/sd-card.img

For writes, build a separate --storage-writable kernel and let the wrapper check the card with fsck.fat and mtools after QEMU exits (add --target qemu_raspi4b to both for the Pi 4):

./build.sh --target qemu_raspi3b --storage-writable --build-dir build/storage-writable --userfs-dir build/userfs/aarch64
tests/run-qemu-storage-write.py

USB drives (Pi 3, Pi 4, QEMU raspi3b)

USB sticks and drives plugged in at boot work through Circle’s mass storage (bulk-only) driver, on any port the keyboard works on. Each drive’s FAT32 partitions show up in volumes next to the ramdisk and the SD card. Drives are read-only unless the kernel is built with --storage-writable, the same flag as the SD card. Drives must use 512-byte sectors and be at most 2 TB; GPT and exFAT are not supported (as for any volume). The FAT32 layer transfers one sector at a time and each USB command takes a few USB frames, so the driver reads ahead 16 KB per command. Writes stay one sector per command: in QEMU (full speed) copying a 300 KB file takes about a minute.

In QEMU raspi3b the smoke test plugs a disposable drive into QEMU’s USB, next to a QEMU USB keyboard, and reads it from core 0 and from the last core:

tests/create-sd-card-image.py build/userfs/aarch64/koraos.img --output build/usb-stick.img --label USBSTICK
tests/run-qemu-smoke.py --target qemu_raspi3b --usb-stick build/usb-stick.img
tests/run-qemu-storage-write.py --media usb    # or both, with a --storage-writable kernel as above

CI builds shared userfs once and passes its artifact to kernel jobs. It builds all four targets, both hardware release targets, a complete SD payload, and retains host sanitizer/CLI tests and raspi3b smoke. Independent virt profiles cover embedded graphics, external disk/keyboard at 128 MiB, serial-only 64 MiB, EL2 entry, writable scratch-root integrity, and repeated EL0 page/malloc allocation and nested-process lifetime. A raspi3b storage job reads standard and high-capacity scratch cards and a USB drive, and writes a card alone and a card and a USB drive together with a --storage-writable kernel. A raspi4b job (in a Debian container, for QEMU 10) runs the smoke test with stress and the EMMC2 card reads and writes. Logs and screenshots are retained on failure. QEMU does not validate real Pi USB, HDMI, cache behavior or firmware. The current virt profile has no SMP, GICv3, PCI, networking, audio or VirtIO GPU; scheduling remains cooperative.

Migration and direct CMake

--qemu, --virt, --hw and --variant remain deprecated aliases. Hardware aliases use RPI_VERSION, while canonical names select the board explicitly. Aliases produce the new directory layout too. BOOTMNT does not implicitly install; use --install-to. Direct legacy CMake variant options remain available, but normal development should use the orchestrating script.

Modern kernel CMake configurations receive KORAOS_TARGET and KORAOS_USERFS_DIR; they only consume the prepared image. The standalone producer is cmake/userfs. Register user programs in cmake/userfs.cmake. Cross-target static libraries use LLVM ar/ranlib. Use a fresh build directory when changing archive tools in an existing CMake cache.