Mettā

decentralized internets

Event kernels

This post came to be as part of my LLM-assisted research, text here is wholly generated by Fable 5.1 model. All such posts are tagged with llm-gen so you can easily spot them. The summary is quite educational however, so I decided to publish it.

Distinction between process and event kernels

  • Process kernel: every user thread has its own kernel stack. When a thread traps into the kernel (syscall, page fault, interrupt), the kernel runs "on behalf of" that thread, on that thread's kernel stack. If the kernel needs to wait for something (disk I/O, a lock), it can just block mid-syscall β€” the thread's kernel state lives on its stack, and the scheduler switches to another thread's kernel stack. Linux, Windows, the BSDs, and Fuchsia all work this way.
  • Event (interrupt) kernel: there's one kernel stack per CPU, period. Every trap is treated as an event: the kernel runs a handler from start to finish, updates data structures, picks a thread to resume, and unwinds the stack back to empty. The kernel never blocks. If a thread needs to wait, the kernel marks it as waiting in a thread control block and returns β€” nothing about the operation is "kept on a stack". seL4, early L4 variants, and many RTOS kernels work this way.
Process kernelEvent (interrupt) kernel
MemoryA kernel stack per thread
(Linux: 16 KB each)
One stack per CPU β€” tiny
Kernel codeNatural, sequential; can block anywhereEvery wait must be a explicit state machine
Preemption in kernelEasy β€” just switch stacksHard; typically kernel runs to completion with interrupts off
LatencyLong syscalls can be preemptedKernel paths must be short and bounded
Verification / reasoningMany interleaved kernel stacks = hard to reason aboutKernel is atomic, run-to-completion: much easier to prove correct
(this is why seL4 chose it)
Cache/TLB behaviourStack switching pollutes cachesVery cache-friendly
  • a process kernel makes the kernel code easy to write at the cost of more memory and more complex concurrency inside the kernel;
  • an event kernel makes the kernel small, fast, and formally tractable at the cost of forcing every long operation to be written as an explicit continuation β€” which is why event kernels are almost always microkernels that push the "long" work (drivers, filesystems) out to user space.

Event kernels

The constraint that shapes everything: the kernel can never block, so any operation that can't finish "now" has to be representable as a small, fixed-size piece of state in a kernel object. Every good event-kernel API decision is some consequence of that. Here are the considerations, each paired with a real design that gets it right (and one that got it wrong).

1. Every wait point must be a first-class kernel object

In a process kernel, "waiting" is implicit β€” the stack just sits there. In an event kernel the API must expose what you're waiting on as an object the kernel can record a pointer to in your TCB.

seL4 does this well: the only things a thread can block on are an Endpoint (synchronous IPC) or a Notification (asynchronous signal). A TCB in state BlockedOnReceive holds exactly one endpoint pointer plus a queue link. That's the entire continuation.

TCB
β”œβ”€ state: BlockedOnReceive
β”œβ”€ waiting_on: -> Endpoint #7
└─ queue_link ──> (next waiter)

What makes it good: the kernel never has to reconstruct what a thread was doing. It just looks at the object and the state enum. Compare Linux, where a thread's kernel stack may be 12 frames deep inside a filesystem when it blocks.

2. Payloads must be bounded and page-fault-free

If the kernel copies a message and the copy page-faults, the kernel now has to block mid-operation β€” which an event kernel can't do. So message size and location must be controlled.

L4 got this wrong first, then right. Original L4 had "long IPC": arbitrary-length copies between address spaces. Page faults during the copy forced the kernel to nest a fault handler inside IPC β€” the single most complex code path in the kernel. seL4 removed it entirely. Now IPC is:

  • up to ~4 words in registers (zero copy, fastest path)
  • up to 120 words in a per-thread IPC buffer that the kernel has already pinned β€” guaranteed no fault
  • anything bigger: share a page and pass a capability to it

What makes it good: the kernel's worst-case IPC time is a compile-time constant. Bulk data moves by changing who can see a page, not by copying.

3. Long operations get preemption points, not blocking

Some kernel work is unavoidably long: zeroing a 1 GiB region, revoking a capability that has thousands of children. You can't block, but you also can't hold interrupts off for milliseconds.

seL4's answer: the operation is written so its progress lives in the object being modified, and the kernel periodically checks for pending interrupts. If one is pending, it returns to user space with "restart this syscall". The user thread re-issues it, and it resumes from the recorded position.

// shape of seL4's revoke loop
fn cnode_revoke(cte: &mut Cte) -> SyscallResult {
  while let Some(child) = cte.first_child() {
    delete(child);                   // progress persists in the CDT itself
    if irq_pending() {
      return SyscallResult::Restart; // unwind to empty stack, come back later
    }
  }
  SyscallResult::Done
}

What makes it good: interrupt latency is bounded and the kernel stack is empty at every preemption point. The API-visible cost is that user code must tolerate a syscall returning "not done yet" β€” which is exactly what a kernel with one stack has to ask.

4. Fuse the common send+wait pairs into one syscall

In an event kernel every syscall is a full trap β†’ handle β†’ return cycle. A client that does send(request) then recv(reply) pays two traps and, worse, between them the server might run before the client is queued as a receiver β€” a race the kernel has to handle.

L4's Call and ReplyRecv solve both. The client's Call atomically sends and enters BlockedOnReply. The server's ReplyRecv atomically replies to the previous client and waits for the next one. A full RPC round trip is two traps total, and the server thread is never runnable-but-idle.

client                    kernel                    server
  |                         |                         |
  |── Call(ep, msg) ───────>|  (switch directly) ────>| running handler
  |   BlockedOnReply        |                         |
  |                         |<──── ReplyRecv(ep) ─────|
  |<── reply, runnable ─────|                       server now BlockedOnReceive

What makes it good: it also enables direct process switch β€” the kernel hands the CPU straight from client to server without touching the scheduler's run queue, because the fused syscall tells the kernel exactly who should run next.

5. Identify the sender without a second round trip

A server handling many clients needs to know who called. Asking "who are you?" costs a syscall, and trusting a field in the message is a security hole.

seL4 badges: a capability to an endpoint can be minted with an immutable integer badge. When a message arrives through that capability, the kernel delivers the badge alongside it. The server hands each client a differently-badged cap to the same endpoint.

What makes it good: identity is authenticated by the kernel for free, and the server's dispatch is match badge { ... } β€” the whole event loop is one ReplyRecv in a loop with a switch.

6. Separate "signal" from "message"

Interrupts and other asynchronous events don't need a payload and must never block the signaler (a device can't wait for a driver).

seL4 Notifications are a word-sized bitmask. Signal ORs bits in and never blocks. Wait returns the accumulated mask and clears it. Multiple signals coalesce. Interrupts are delivered as notification signals, so a driver's inner loop is just Wait(notification) β€” identical to handling any other async event.

What makes it good: it's the "async" half of the API, it can't fail, can't block, and can't queue unboundedly (bits coalesce). Contrast Mach ports, which buffer full messages for async sends and therefore needed queue limits, back-pressure, and blocking senders β€” all things an event kernel can't afford.

7. Make the reply continuation an object too (MCS seL4)

Classic L4 stored "who to reply to" as a hidden per-thread reply capability in the TCB β€” implicit state, awkward for servers that want to defer replies or hand them to another thread.

seL4 MCS made it explicit: Reply objects. A server can receive a call, stash the reply object, go handle other clients, and reply later β€” or pass the reply cap to a helper thread. A SchedulingContext (a time budget) travels with the call so the server runs on the client's time, which prevents a server from being starved or used to steal CPU.

What makes it good: nothing about an in-flight RPC lives anywhere except in kernel objects the user holds capabilities to. Deferred replies, forwarding, and priority inheritance all fall out without new kernel code paths.

8. Push policy and long work out of the kernel

Because the kernel can't have long paths, anything long β€” page-table walks for mapping, driver logic, filesystem operations β€” must be a sequence of short kernel calls driven by user space.

seL4's memory API is the extreme version: there is no mmap. User space allocates page tables from Untyped memory (Retype), maps each page with an explicit Page_Map call, and handles page faults as IPC messages delivered to a handler thread. The kernel has no allocator and no paging policy at all.

What makes it good: every kernel call touches O(1) objects. The price is a verbose API, which is why real systems put a library layer on top (seL4's CAmkES, or Rust sel4-sys wrappers).

The pattern

ConsiderationGood exampleAnti-pattern
Waits are objectsseL4 Endpoint/NotificationBlocking read() inside a driver
Bounded, pinned payloadsL4 register IPC + pinned IPC bufferL4 long IPC (removed)
Restartable long opsseL4 preemption points β†’ RestartHolding IRQs off during revoke
Fused syscallsCall / ReplyRecvSeparate send + recv
Kernel-authenticated identityBadged capabilitiesSender ID in message body
Signal β‰  messageNotification bitmaskMach async ports with queues
Explicit continuationsMCS Reply objectsHidden reply cap in TCB
Policy in user spaceUntyped + Page_MapIn-kernel allocator / mmap

If you want a Rust mental model: a good event-kernel API is one where the entire kernel could be written as fn handle(event: Event, objects: &mut Objects) -> NextThread with no await, no recursion into user memory, and every function bounded by a constant β€” and every API decision above is just what you have to expose to users to make that signature possible.

home Premature generalization