Host API¶

Status: implemented for host-fed runs on one Blackhole card.

import komorebi as km

with km.Session(card=0) as session:
    run = session.load(program)
    handles = [run.submit("map", group=(job * 8, 8), args=[...]) for job in range(15)]
    run.wait(handles, timeout=5.0)
    run.stop()
    trace = run.trace()

import komorebi needs no tt-metal tree. km.Session loads the package's native module on first use; neither ttnn nor torch is imported. Data goes to and from the card as numpy arrays (see Tensors).

`Program`¶

km.Program(name, kernels, buffers=[], args=None, compile_args={}, compute={}, defines={}, depth=4, handlers={}, opt={}) describes what the persistent kernel runs. It opens nothing.

Field	Meaning
`kernels`	a directory with `reader.cpp`, `compute.cpp` and `writer.cpp`, or a dict of the three paths
`buffers`	`km.Buffer(index, format, page_bytes, pages)`: circular buffers on every core; `format` is a data type name (`"bfloat16"`, `"float32"`, `"int32"`, `"uint32"`, `"uint16"`, `"uint8"`, `"bfloat8_b"`, `"bfloat4_b"`)
`args`	`args(core) -> (reader, writer, compute)`: the program's own runtime arguments of each core, after the runtime's start-up arguments
`compile_args`	`{"reader": [...], "writer": [...], "compute": [...]}`
`compute`	compute kernel options, the fields of tt-metal's `ComputeConfigDescriptor` (v0.81.1): `math_fidelity` (`"LoFi"`, `"HiFi2"`, `"HiFi3"`, `"HiFi4"`), `unpack_to_dest_mode` (`"UnpackToDestFp32"` or `"Default"` for every circular buffer, or `{cb: name}` with the others `Default`), and the flags `fp32_dest_acc_en`, `math_approx_mode`, `dst_full_sync_en`, `bfp8_pack_precise`, `enable_trisc2_rvv`; any other name raises `ProgramError`
`defines`	`{name: value}` for all three kernels
`depth`	mailbox slots the cores scan and the dispatcher uses, 1 to 16
`handlers`	`{name: number}`, or a header whose `constexpr uint32_t kHandlerName = n;` lines give them (`kHandlerBigStep` becomes `"big_step"`); numbers 1 to 239
`opt`	`{"reader": "Os", "writer": "Os"}`: an optimisation level per kernel, one of the runtime's `O0`, `O1`, `O2`, `O3`, `Os`, `Ofast`, `Oz` (anything else raises `ProgramError`); a kernel without one keeps the runtime's default, `O2` for the reader and writer and `O3` for the compute kernel (its three threads together). A set level also defines `KM_OPT_LEVEL` for that kernel, so that it is cached apart

program.thread(cluster=1, sources, regions, words, host_fallback, deadline, prefetch="documented") adds an X280 thread on a work cluster (one per cluster; clusters=(1, 2, 3) declares it on each, program.region(name, value) a region shared by threads on one bank); see X280 threads.

A program's three kernels share the session's kernel buffer, 70,656 bytes on Blackhole unless the session sets kernel_bytes. When a program is loaded its build is laid out against that buffer: program.sizes (a km.KernelSizes) holds the packed text per thread, the total and the buffer, and the session log has them in one line. A program that does not fit raises km.KernelSizeError before anything is loaded, naming the total, the buffer and the largest thread (in place of the runtime's "Program size ... too large for kernel config buffer"). Give the session more with kernel_bytes, or lower the reader and writer to "Os" first and the compute kernel only when that is not enough, since its hot loops are what O3 is for.

A program's own header (its handlers, buffers and argument indices, shared by its kernels and its driver) is read with km.abi.constants(path, known=km.abi.headers().constants): every constexpr uint32_t k*, #define KM*_* and enum value, as a dict; it may use the package's constants by their C++ names (km::kBroadcastArgs). A constant the reader cannot evaluate raises abi.HeaderError.

The runtime owns circular buffers 29 and 30 (the command's token for compute and writer) and 31 (the trace); a program uses the others.

`Session`¶

km.Session(card=None, device=None, pll_mhz=1000, tree=None, log=None, grid_timeout=10.0, kernel_bytes=None).

pll_mhz is the X280 clock (the runtime's steps: 200, 800, 1000, 1750). One PLL clocks all four X280 clusters, so the work clusters' threads run at it too. 1750 is an opt-in: it loses loads when the Tensix grid is under full matrix load (the card lowers its core voltage; the dispatcher trapped in 9 of 18 such runs, and in 0 of 23 at 1000 MHz); safe use is the caller's claim. A host-fed command at depth 16 costs 1.228 microseconds at 1000 MHz and 0.726 at 1750, one after the other on one card (experiments/dispatcher/measure.py bench --n 100000 --depth 16 --group 1 --batch 64 --host-depth 64 --pll 1000, then --pll 1750). Before the session sets the clock, the runtime that opens the cluster leaves it at its safe 200 MHz; 1750 was never the card's own choice. The session's log names the clock it set.
card selects the card through TT_VISIBLE_DEVICES (an error if that variable already names another card); device uses a device (the native module's Device) the caller opened and keeps. The session opens the card in slow dispatch, as a 1 x 1 mesh through tt-metal's own API, with the program cache off.
tree is the tt-metal tree for builds (default TT_METAL_HOME). log(msg) receives one line per step (launch, kernel started, runtime closed, read back).
As a context manager it stops a loaded run at the exit: with run.stop() on a normal exit, through the X280 runtime (no EXIT) on an exception, then it closes the device it opened. session.close() does the same without with.
The X280 runtime must be closed while the device is still open: a process that ends with it open, or closes the device first, can leave the card needing a reset. So the session also closes itself when the process exits without closing it (atexit) and on SIGTERM (a handler is installed if the process has none; it closes the runtime, then exits with status 143 through the with blocks). A clean-up that raises leaves the run loaded and the device open, so the next of these paths tries again; each try has a timeout (5 s in the runtime). SIGKILL cannot be caught: after one, reset the card. With device=, close the session before the device.
What an abort (an exception inside with, session.close() of a loaded run, the exit hooks) does: it writes the stop flag into every core and waits up to 2 s for the grid, then asks the X280 runtime to stop the dispatcher, which finishes whatever is in flight (5 s timeout); run.result() then holds its result block, with abandoned the commands left without their KM_EVENT_DONE. The card needs no reset after it. An exception in the kernel needs none either: the kernel finishes on its own and the run raises KernelFailed with error trap, naming the X280 core, mcause, mepc and mtval. After an abort or a kernel failure the session log also holds the first 512 bytes of the kernel's state as DRAM holds them (kernel state +0x... lines). A reset is still needed when the kernel's core 0 has stopped running its loop (a WaitTimeout says so): the runtime cannot restore cluster 0, and KernelFailed carries the runtime's error message for each core.
grid_timeout: the timeout, in seconds, of every wait for the grid's kernels. When it passes, km.GridTimeout is raised and the session does not close the device: the runtime closes it at the process's exit, waits for the cores up to TT_METAL_OPERATION_TIMEOUT_SECONDS, then puts them in reset. A thread that is only stuck (a loop, a blocked wait) is recovered by the next open; a kernel that left a compute unit running is not, and when the next open fails the card needs a reset. The variable is read once, at the process's first device open, and is the timeout of every wait the runtime makes for the cores in that process, not only the exit's; unset, it means no timeout, and only then would the close wait without one. A session that opens the device therefore sets it to 30 s when it is unset (with a log line) and keeps a value of the user's.
kernel_bytes is how much of each core's L1 holds the program's kernels (the runtime's kernel config buffer: the five threads' text, their arguments and the buffer configs), a multiple of 64 from 64 to 1,531,712; None keeps the runtime's 70,656 bytes. Every byte added is taken from the L1 left for circular buffers and L1 tensors (1,461,120 bytes by default; 1,400,704 with 128 KiB of kernels): the session opens the device with the runtime's worker_l1_size that gives it, so it is set only when the session opens the device. session.l1() returns (kernel buffer, L1 left) of the open device, from the runtime's layout.

session.load(program, depth=None, trace=0, deadline=1.0, wait=2.0, idle=120.0, kernel_idle=0.0, start=10.0, host_depth=64, event_depth=128, batch=64, settle_microseconds=1.0, image=None, thread="card") returns a Run:

finds or builds the dispatcher image and the native module (see below), and checks that the image is there and X280 cluster 0 is idle (else km.StartError);
opens the X280 runtime and lays out the kernel's memory;
launches the three kernels on every core. Kernels compile here, before any X280 image exists, so a compile error raises km.ProgramError and leaves nothing running;
starts the dispatcher and waits (with start seconds as its timeout) for every column of the grid to report READY; otherwise km.StartError, which carries the kernel's result block.

From the launch on, every failure path reads the grid back. When the dispatcher is not there to stop the grid (it never started, or it ended before it could reach the cores), the host writes the stop flag into every core's control block itself, through the device API, and then waits for the grid within grid_timeout.

Argument	Meaning
`trace`	sequence numbers traced per core (0: no trace); 16 bytes of L1 each
`deadline`	seconds a command may take from submit to its group's completion, waiting behind earlier commands included
`wait`, `idle`	the cores' timeout on a wait for another core, and on a wait for the next command (the longest the host may stay silent)
`kernel_idle`	the dispatcher's own idle timeout; 0 turns it off
`host_depth`, `event_depth`	the command ring in the X280's memory and the event ring in host memory; powers of two, at most 64 and 128
`batch`	commands stored per batch: every line, then one fence
`settle_microseconds`	the host's pause between a fence and its next store into the card
`thread`	for a program with an X280 thread: `"card"` builds and starts it after READY, `"host"` runs its `host_fallback` instead

Event room. The dispatcher takes a command that ends in an event only while the unread events, one per busy slot and one more fit the event ring; it counts as read what the host last wrote into its read-count line. The host keeps a tighter rule: it stores a command only while the events it has not read plus the commands it stored without an event fit the event ring. It writes its read count with every batch, and while it waits whenever the kernel could be holding a command for it, so the kernel waits for room only while that count is on its way.

Exclusive commands. The host stores a command with KM_FLAG_EXCLUSIVE like any other; the dispatcher holds it at the head of its command ring while a busy command's group shares a core with it, and submits it once that command's leader has acknowledged. Host commands behind a held one wait with it, whatever their groups. Its deadline counts from the kernel's submit, not from the store. So back-to-back exclusive commands on one group are the kernel's boundary apart, while one submitted after idle still pays the host's store reaching the X280 (a round trip of about 14 us). Measured on 2026-11-13, one p150a, X280 at 1,000 MHz, host load 3 to 9, before (the host held each exclusive command until it had read the event of the overlapping one) and after, alternating in one session:

Commands back to back, exclusive	Before (us)	After (us)
NOP on 120 cores, period	16.25, 16.26	2.27, 2.27
NOP on 12 cores, period	4.28, 16.21	2.28, 2.28
NOP on 1 core, period	15.95, 15.96	1.90, 1.91
`examples/03_broadcast`, 120 cores, per job	61.1, 60.4	47.9, 46.9

The period is the first core's take of one command to that of the next (the cores' trace, median of 99, after 10 commands of warm-up). The broadcast row is broadcast.py --rect 0,0,12,10 --blocks 16 --pages 4 --window 2 --jobs 40 --no-check (host time per job). The 4.28 is a run in which the host's stores reached the X280 fast, a state seen now and then.

Cores' NoC coordinates. session.nocs() returns a layout.Nocs for the open device, before any load: grid; by core (x, y) the translated NoC address noc_xy (the one the dispatcher's core table holds) and the NoC0 coordinates noc0; words, the translated addresses as x | y << 16 in core order; and the grid's panels. These are what layout.broadcast_args and program arguments that name other cores take.

Tensors¶

A program's inputs and outputs are tensors the session puts on the card:

Call	Returns
`session.tensor(data, dtype, layout="row", shard_grid=None)`	a `Tensor` holding `data` (anything `np.asarray` takes) as `dtype` (`"bfloat16"`, `"float32"`, `"int32"`, `"uint32"`, `"uint16"`, `"uint8"`), written before it returns. `layout="row"`: a page a row (the last dimension), interleaved over the DRAM banks; `"tile"`: a page a 32 x 32 tile of four 16 x 16 faces, the last two dimensions zero-padded to multiples of 32. `shard_grid=(x, y)`: rows `[cores, words]`, one in the L1 of each core of the grid at one address
`session.empty(shape, dtype, layout="row", shard_grid=None)`	the same, not written
`session.grid`	the Tensix grid `(x, y)` of the open device

session.device.mesh_capsule() hands the device to a native module that takes it: a capsule named komorebi.mesh_device holding a std::shared_ptr<MeshDevice>.

A Tensor has address (its address in every bank, or in every core's L1), shape (on the card, padded), dtype, layout, page_bytes, aligned_page_bytes, pages; accessor_args() (the compile-time arguments of a TensorAccessor over it, for Program(compile_args=...)); write(data); read() (the whole tensor as a numpy array, bfloat16 as float32) and read_bytes(); free(). Its write and read wait for the cores of a launched program to return, which the persistent grid does only at the stop: before load and after run.stop() they are the plain way. bfloat16 values are rounded to nearest, ties to even; komorebi.formats has the conversions (bf16_bits, bf16_values, tilize, untilize, to_bytes, from_bytes) on their own.

Reading tensors while a run is loaded. session.read(*tensors) returns the raw pages of interleaved DRAM tensors ([pages, page_bytes] uint8 each) and session.read_bf16(*tensors) bfloat16 tile tensors as float32 arrays. Page p of such a tensor lies in DRAM bank p % banks, so the native module reads each bank's pages in one go, after one DRAM barrier for all the tensors of the call. These reads do not wait for the cores to be idle, so they work while a run is loaded; read then only what no kernel writes any more (an output whose command's event has arrived). The control blocks live in L1 and are read with Tensor.read() after the stop.

A part of one tensor is read the same way:

Call	Returns
`session.read(tensor, first=0, pages=None)`	`[pages]`: pages `first` .. `first + pages - 1` as one `[pages, page_bytes]` uint8 array (`pages=None`: to the end); a range outside the tensor raises `km.KomorebiError` naming its page count
`session.read_chunks(tensor, chunk_bytes=2 << 20)`	a generator of `(first, pages)` over the whole tensor, each chunk at most `chunk_bytes`
`session.read_tiles_bf16(tensor, first, pages)`	tiles `first` .. `first + pages - 1` of a bfloat16 tile tensor (one tile a page, row-major tile order) as float32 `[pages, 32, 32]`

Reading back is slow: the card test reads a 300-page tensor (614,400 bytes) whole in 0.29 s and in ten chunks in 0.30 s, about 2 MB per second (host load about 7). A long read is silent, and a job watcher that kills on silence cannot tell it from a hang. So read a large tensor with read_chunks and log a line per chunk, and check a large result by sample: the pages of a few rows or tiles, from read or read_tiles_bf16.

run.thread.read(name) reads only the region's pages of the thread's bank (over 4 MB, in 2 MB chunks with a log line each): a 1 MB region in 0.48 s, 5 MB in 3.8 s (experiments/x280_threads/region_read.py, host load about 4). The load still writes the regions' whole tensor, 8 MB per MB of regions: no call the host uses writes a page range.

`Run`¶

Call	What it does
`submit(handler, group=None, args=(), exclusive=False, result=False, queued=False)`	queues one command and returns its handle (1, 2, ...). `handler` is a name or a number (`0` is NOP); `group` is `(first, count)`, default every core; `args` up to ten 32-bit words. Groups and handlers are checked here. With `exclusive` (`KM_FLAG_EXCLUSIVE`), the dispatcher submits the command only once every queued command sent before it has left the kernel's job queue and no command in flight has a group that overlaps it; the host stores it as soon as the ring has room, like any other command (below, "Exclusive commands"). With `result` (`KM_FLAG_RESULT`), its done event carries the group leader's 32-bit result word. With `queued` (`KM_FLAG_QUEUE`), `group` is a count: the kernel gives the command the lowest first core whose group overlaps no busy one, and waits until there is one (nothing is rejected for a full grid); `group(handle)` is then `(None, count)` until its done event names the first core (8 bits: at most 256 cores; `KM_CORES_MAX` is 128). The kernel's job queue holds 64 commands waiting for room; one more is rejected (`CommandRejected`, reason 9), so with at most 64 queued commands unfinished none is rejected. Queued commands share the program's `depth` completion slots with every other command: at most `depth` (16 at most) run at once over the whole grid. A queued command cannot be exclusive
`submit_many(handler, firsts, counts, args=None, exclusive=False)`	queues n commands at once and returns their handles as a `range`. `handler`, `firsts` and `counts` are one value or n each; `args` is an `[n, m]` integer matrix (m at most 10) or None. Checked as `submit` checks, all before any is queued
`run_hostfed(graph, bindings=None, timeout=10.0)`	runs a graph host-fed: its executor on the host, its commands through this run; returns `HostfedResult(status, error, registers, runs, commands, events)`
`wait(handles, timeout=10.0)`	stores queued commands up to the handles in batches, reads events until each handle has one; returns the `Event` (or a list) and forgets those handles. The stores and the event reads run in the native module with the GIL released
`wait_any(handles, timeout=10.0)`	stores what is queued and waits until one of the handles has its event; returns that `Event` (the lowest handle if several arrived) and forgets only that handle. Raises as `wait` does
`drain(timeout=10.0)`	the same for everything submitted; returns how many events it waited for
`event(handle)`	`Event(handle, kind, auxiliary, ticks, result)` or None; `kind` is `done`, `timeout`, `pong` or `rejected`; `auxiliary` is the first core of the group that ran the command (done; for a queued command, the one the kernel chose), the slot (timeout) or the reason (rejected): the low 8 bits of the event's auxiliary, whose next 16 bits, the low bits of the command's sequence number, `sequence(handle)` reads (layout: the kernel README, "Sources"); `ticks` is submit to completion in the X280's 50 MHz timer, `microseconds` the same in microseconds; for a command submitted with `result=True`, `result` is the leader's word (the kernel sends it in place of the time, so `ticks` and `microseconds` then hold it too), else None
`sequence(handle)`, `group(handle)`	the sequence number the kernel gave the command (the trace's index), from its done or timeout event (None before it, and for a rejected command); its group, whose first core for a queued command comes from its done event (None before it)
`forget(handles)`	releases what the run keeps for handles whose events arrived; for callers that read events with `event()` after `drain()`. A traced run keeps groups and sequences for the trace
`stop(timeout=10.0)`	`timeout` is one deadline for all of it: cancels every engine still running (or `unknown`) and waits for their ends, drains, sends EXIT to every core and waits for it, sends the kernel STOP, waits for its result, closes the X280 runtime, waits for every core's kernels to return, then reads the state words and the trace. Returns the result block; raises `KernelFailed` unless the result is ready, cluster 0 restored, the state DONE and the error 0. Any error on the way (an `EngineTimeout` of a cancel not answered, a `WaitTimeout`) still closes the runtime first
`thread_result(timeout=10.0)`	a program with an X280 thread: waits for it (or stops it), returns its `ThreadResult`; after `stop()`, `run.thread.read(name)` returns a region's bytes
`abort()`	the failure path: stops the grid from the host (with a timeout), then the kernel through the runtime, closes it, reads back; `result()` then has `abandoned`; raises `KernelFailed` if the runtime could not restore cluster 0, with the runtime's account of each core (phase, stage, trap cause and address) in its message and the session log
`result()`, `state()`, `state_words()`, `trace()`	after the stop: the kernel's result block (decoded), each core's state words as a dict (with `ports`: count and wanted count of every port slot, and `broadcasts`: each broadcast's VALID word, first give-up, blocks, wait cycles and DONE-BY words; a reader or writer that gave up waiting on a port or an emit names its `port`), the raw state words, the trace

run.submit_s adds up the host's time in submit and submit_many; run.wait_s the time in wait and drain, of which run.native_s was in the native loop.

trace() returns a komorebi.state.Trace: command(handle) gives each core's wall-clock words (taken, outputs written, completion sent, ring passes); phases(handle) gives microseconds after the first core took it; lead(earlier, later) is how long before handle a's completion a core of b took its command (positive: both were in flight at once).

Engines: uploads, `start`, events¶

The dispatcher runs streams and graphs from an upload in its upload region, with no host step between the start and the end event. These Run calls write the upload, start it, and turn its events into values.

from komorebi.engines.graph import Graph

step_handler, left_handler, right_handler, norm_handler = (
    program.handler(name) for name in ("step", "left", "right", "norm")
)
graph = Graph(deadline_microseconds=200_000, max_runs=50, req=6, keep=True)
state, iteration, limit = graph.bindings("state", "iteration", "limit")
error = graph.register()
step = graph.command(step_handler, group=(0, 60), args=[state, iteration])
left = graph.command(left_handler, group=(0, 30), args=[state], after=[step])
right = graph.command(right_handler, group=(30, 30), args=[state], after=[step])
norm = graph.command(
    norm_handler, group=(0, 60), args=[state, 60], after=[left, right], result=error
)
cpu = graph.cpu(after=[norm])
cpu.addi(iteration, iteration, 1)
cpu.lt(graph.again, error, limit)  # go round again while error < limit

upload = run.upload(graph)
first = {"state": x, "iteration": 0, "limit": 10**6}
event = run.wait_engine(run.start(upload, first), 10.0)
second = {"state": y, "iteration": 3, "limit": 10**7}
event = run.wait_engine(run.restart(6, second), 10.0)
iteration_end, error_end = event.registers  # registers 0 (its low 24 bits) and 1
run.release(upload)  # the kept graph is dropped, its lines are free again

This is the graph of examples/06_graph (the same handlers; x and y are the addresses of two state tensors): a kept graph, run, then run again with other bindings and no new upload.

Call	What it does
`upload(stream_or_graph, kind=None)`	encodes a `Stream` or a `Graph` (or takes its encoded bytes, with `kind` `"stream"` or `"graph"`) and writes it into the upload region as whole lines, at the first free place; returns an `Upload` (`offset`, `bytes`, `kind`, `req`, `keep`, `state`). Several uploads live side by side. An upload that does not fit raises `ValueError` before anything is written. The host checks nothing else: a bad upload is the kernel's to reject, with its error event
`release(upload, timeout=10.0)`	frees an upload's lines. A running one is rejected; a kept graph is first dropped (`CANCEL`, then its cancelled event, both within `timeout`)
`start(upload, bindings=(), kept=None, timeout=10.0)`	sends `KM_HANDLER_RUN` and returns an `EngineHandle`. A graph takes `bindings`: a dict by name for a graph uploaded from a `Graph` (its constants are added, as `binding_values` does), or a list of up to 16 words; they go to one line of the region, freed at the end. `kept=True` or `False` rewrites the header's keep bit first; None leaves it. Another running or kept engine with the same `req` raises `ValueError`. When the kernel does not answer the RUN within `timeout` it raises `EngineTimeout` with the handle marked `unknown`: the engine may have started, and `cancel` (or `stop`) settles it
`restart(req, bindings=(), timeout=10.0)`	runs the kept graph with `req` again with new bindings (`KM_HANDLER_RUN`, `a2 = 2`)
`cancel(handle, timeout=10.0)`	sends `KM_HANDLER_CANCEL`; the engine then ends with its error event, reason `cancelled`. Nothing is sent for an engine that has ended. For an `unknown` handle it also learns the RUN's answer: raises `EngineRejected` if the RUN was rejected (it never ran); otherwise the engine runs or has ended as usual. `EngineTimeout` if the CANCEL is not answered
`wait_engine(handle, timeout=10.0)`	waits for the engine's end and returns its `EngineEvent`; raises `EngineError` for an error event, `EngineTimeout` when `timeout` passes (the engine may still run), `KernelFailed` if the kernel finished
`run_hostfed(graph, bindings=None, timeout=10.0)`	runs a graph or a stream on the host instead, its commands through the host ring: the same commands and registers as in the kernel (graphs); for a stream, the simulator's interpreter, `commands[i].node` the program counter of the `SUBMIT` or `ENQUEUE`, `runs` 1

start, restart and cancel send the command and a PING behind it and return when the PING is answered: the kernel answers a RUN or CANCEL with an event only when it rejects it, and that event comes first. A rejection raises EngineRejected (a CommandRejected: reason, reason_name); the host does not retry.

An EngineHandle has kind, req, upload, events (every event so far) and end. An EngineEvent(handle, kind, value, registers, reason, reason_name, pc, node) is one of:

`kind`	From	Fields
`event`	a stream's `EVENT a`	`value`: register `a`, 32 bits
`round`	a graph's round ended and another starts	`value`: rounds done
`end`	`HALT`, or a graph's last node	a stream's `value`: the halt register, 32 bits; a graph's `registers`: register 0's low 24 bits, register 1
`error`	any error	`reason`, `reason_name` (the `KM_STREAM_ERROR_*` name in lower case, read from the headers), the stream's `pc` or the graph's `node` (None when the error has no node)

The engines' events carry the upload's req, not a command number: they go to their handles, so wait and wait_any of the host's own commands see none of them, and wait_engine none of the commands' events. The commands an engine submits are answered to the engine, never to the host. While it waits, wait_engine also writes the host's read count into the kernel, since an engine's posts wait for event room.

The region is the one the kernel's ready block names: what is left of X280 cluster 0's local memory after the kernel's layout (707,840 bytes on a p150a), none for a program with an X280 thread, whose ring lies there. The host never writes into the range of a running stream or graph, or of a kept graph. The kernel's EXIT waits until no engine runs, so stop cancels the engines still running first.

The kernel numbers the commands of every source (the host, an engine, a work-cluster thread) in one sequence. A command's done or timeout event carries the low 16 bits of its number, and the host takes the rest from its previous command's: sequence(handle) and the trace's command(handle) are exact while fewer than 32,768 commands of other sources are submitted between two host commands. When the host sees a longer gap it raises KomorebiError for that handle instead of giving another command's number; it cannot tell a gap of 65,536 or more. A queued command's number may fall up to 2,048 behind the host's order (the queue lets later jobs go first); a lag beyond that is read as a gap and raises too.

Errors¶

All derive from km.KomorebiError.

Error	When	Facts it carries
`CardBusy`	`Session.open`: another live process holds the card (below)	`pid` of the holder
`ProgramError`	a kernel did not compile or the launch failed	the build message
`KernelSizeError`	the program's kernels do not fit the session's kernel buffer; a `ProgramError`, raised before anything is loaded	`sizes`: text per thread, total, buffer
`KernelInstructionError`	a kernel's ELF holds an instruction its core lacks (below); a `ProgramError`, raised before anything is loaded	`findings`: thread, instruction, function, source line
`StartError`	the image is missing, cluster 0 is not idle, a base handed to the kernels (control block, X280 local memory, its upload region) is not a multiple of 16 bytes, or the dispatcher did not report READY	`result` (the kernel's result block, when it ran)
`WaitTimeout`	a host-side wait ran out; the run is still loaded. The message ends with what the kernel was doing: whether core 0 still runs its loop (its pass marker, read twice 20 ms apart), its state and where, and the runtime's account of each core	`pending` handles
`CommandRejected`	the kernel consumed the command and submitted nothing (a reserved handler or flag, a group outside the grid, a command after EXIT); the run goes on	`handle`, `reason` (`KMD_REJECT_*`), `reason_name`
`CommandTimeout`	a group missed its deadline; the kernel stopped the grid, the run is stopped and read back	`handle`, `slot`, `sequence`, `group`, `cores`: each core of the group that did not report, with where its reader and writer gave up
`KernelFailed`	the dispatcher finished with any other error, did not finish cleanly, or sent an event for a command not in flight	`result`
`EngineRejected`	the kernel rejected a `RUN` or `CANCEL` (four streams or two graphs running, no such req, a command after EXIT)	as `CommandRejected`
`EngineError`	a stream or graph ended with an error event	`event`, `handle`, `reason`, `reason_name`, `pc`, `node`
`EngineTimeout`	`wait_engine` ran out, or a RUN or CANCEL was not answered; a `WaitTimeout`	`handle`
`GridTimeout`	the grid's kernels did not return within `grid_timeout`; the runtime closes the device at the process's exit

A card stays held by a process from its first device open until that process exits, even after the device is closed: tt-metal's driver takes the chip's lock at the first open and gives it back only when the process ends. A second process that opens the card meanwhile would wait without a timeout, so Session.open first reads the lock's owner (from the driver's lock file, without taking the lock) and raises km.CardBusy naming that process when it still maps the file: a process that died holding the lock, a zombie or a PID reused since is not taken for the holder. A test run that opens the card in its own process therefore cannot start a card script as a child (see tests/hardware/README.md).

The kernel's reasons for a rejection (CommandRejected.reason, from KMD_REJECT_* in the kernel's header; reason_name is the lower-case suffix):

`reason`	`reason_name`	The command
1	`handler`	has a handler from 240 to 251
2	`flags`	sets the reserved flag bit 0
3	`group`	has a group of 0 cores, or one that does not lie inside the cores
4	`exited`	came after EXIT
9	`queue`	is queued (`queued=True`) while the kernel's job queue holds 64 commands

submit and submit_many check handlers and groups before anything is queued, the host sets no reserved flag, and a stopped run submits nothing, so through this API a rejection other than 9 means that the host and the kernel were built from different headers.

Builds¶

The dispatcher image (km-dispatch.bin) and the native module (_km_host.abi3.so) are used from beside the package if they are there. Otherwise they are built from the tt-metal tree into a cache directory, KMR_CACHE or build/komorebi/ in a checkout, keyed by a hash of their sources and the tree: python -m komorebi.native builds both and prints their paths.

Instructions the cores lack¶

The compute threads are built for a target that includes a float unit (the pack thread's vector flags imply it), so the compiler may emit instructions the Tensix cores do not implement. A core that meets one stops without a report and the program ends in a timeout that names no cause. After the kernels are built and before anything is loaded, Session.load therefore disassembles each thread's linked ELF with the toolchain's own objdump and raises km.KernelInstructionError naming the thread, the instruction, the function and, from the debug info, the source line. Rejected are the groups the core's instruction set documents as absent: float divide and square root (fdiv, fsqrt), vector integer divide and remainder (vdiv, vrem), and vector float divide, square root and their estimates (vfdiv, vfrdiv, vfsqrt, vfrsqrt7, vfrec7). The result is kept with the build, so a cached build is not checked again. The check itself runs objdump once per thread ELF the first time that build is seen, about 11 ms each (some 0.06 s for a program's five); the runtime's compile of a new kernel, which now runs just before it, is most of a first load (about 0.9 s for example 01). Without the toolchain's objdump, or when the build left no ELF, the load raises ProgramError; without its addr2line, a finding is still rejected, without its source line. A / on floats is the usual cause: divide by an integer (div/divu are implemented), multiply by a reciprocal computed on the host, or, on the vector unit, use a reciprocal with Newton steps as the projects do (recip, exact division and square root in divsqrt.hpp). KMR_INSN_CHECK=0 turns the check off, for a kernel that never executes the code found.

Rules the host keeps¶

The host never reads or writes a core's memory while the program runs, with one exception: the stop flag, when the dispatcher cannot send it. State words and the trace are read after the stop.
While the dispatcher runs, every host wait has a timeout. The X280 runtime is closed before any readback, on every failure path, at the process's exit and on SIGTERM, always before the device.
Host stores into the card go in batches: every line, one fence, then a pause of settle_microseconds before the next store.
Unread events plus commands in flight never exceed the event ring.
EXIT goes to every core, after everything in flight has completed.