Skip to content

Part 1. One core · lesson 10 of 42

10. When it goes wrong

A command that does not finish is stopped by one of two time limits, and they are not the same limit: the host's timeout ends only your waiting, and the card's deadline ends the command. This lesson relies on lesson 8, Handlers and lesson 9, Many commands.

Why two limits

A program that waits for a core can wait for ever if the kernel has a bug. Komorebi has two answers, at two places.

  • The timeout is in Python: the timeout= of wait. When it runs out, wait raises km.WaitTimeout and the program decides what to do. The command is not touched. It may be slow, not broken, and you can call wait again.
  • The deadline is on the card: how long a command may take from submit until every core of its group has reported. The card measures it itself. When it runs out, the card stops the cores and the next wait raises km.CommandTimeout. The deadline is a setting of load, and its default is one second.

The general form

run = session.load(program, deadline=SECONDS)

try:
    event = run.wait(handle, timeout=SECONDS)
except km.WaitTimeout:
    ...  # the host gave up waiting; the command is still on the card
except km.CommandTimeout as error:
    ...  # the card gave up on the command; the run is over
  • deadline= is the card's limit for every command of this run.
  • timeout= is the host's limit for this one wait.
  • After WaitTimeout the run goes on: wait again, or send something else.
  • After CommandTimeout the run has been stopped by the card, and so you do not call run.stop(). To go on, load again.

A kernel that takes a while

To have something that is slow, the kernel of this lesson has a handler that counts up to the number it is given, which takes about six milliseconds per million:

writer.run([&](const km_command_t& command) {
    if (km::handler(command) == kHandlerCount) {
        const uint32_t limit = command.a0;
        volatile uint32_t number = 0;
        while (number < limit) {
            number = number + 1;
        }
        writer.result(number);
    }
    return true;
});

volatile makes the compiler count for real instead of replacing the loop by its result.

One limit at a time

handle = run.submit("count", group=(0, 1), args=[50_000_000], result=True)
try:
    run.wait(handle, timeout=0.05)
except km.WaitTimeout:
    print("the host stopped waiting after 0.05 seconds; the command goes on")
event = run.wait(handle, timeout=10.0)
print(f"a second wait: {event.kind}, the core counted to {event.result}")

The command counts to 50 million, about 0.3 seconds. The host waits 0.05 seconds, and wait raises WaitTimeout. The except prints a line, and the command goes on: the second wait waits up to ten seconds and gets the event.

handle = run.submit("count", group=(0, 1), args=[400_000_000], result=True)
try:
    run.wait(handle, timeout=10.0)
except km.CommandTimeout as error:
    print(f"the card stopped the command: {error}")

Now 400 million, more than two seconds, with a host that would wait ten. The card's deadline of one second passes first. The card stops the cores, and wait raises CommandTimeout, whose message is what the card reports.

Two clocks start at the same moment. The first to run out decides what you see.

The whole program

"""When it goes wrong: the host stops waiting, and the card stops a late command."""

from pathlib import Path

import komorebi as km

HERE = Path(__file__).resolve().parent

program = km.Program(
    "when_it_goes_wrong",
    kernels=HERE / "kernels",
    handlers=HERE / "kernels" / "handlers.h",
)

with km.Session() as session:
    run = session.load(program, deadline=1.0)

    handle = run.submit("count", group=(0, 1), args=[50_000_000], result=True)
    try:
        run.wait(handle, timeout=0.05)
    except km.WaitTimeout:
        print("the host stopped waiting after 0.05 seconds; the command goes on")
    event = run.wait(handle, timeout=10.0)
    print(f"a second wait: {event.kind}, the core counted to {event.result}")

    handle = run.submit("count", group=(0, 1), args=[400_000_000], result=True)
    try:
        run.wait(handle, timeout=10.0)
    except km.CommandTimeout as error:
        print(f"the card stopped the command: {error}")

// The handlers of this program: the numbers that the kernels and Python agree on.
#pragma once

#include <stdint.h>

constexpr uint32_t kHandlerCount = 1;

// The writer kernel: it counts up to the first argument, which takes some time.
#include "komorebi/device/km_worker.hpp"
#include "handlers.h"

void kernel_main() {
    km::Writer writer;
    writer.run([&](const km_command_t& command) {
        if (km::handler(command) == kHandlerCount) {
            const uint32_t limit = command.a0;
            volatile uint32_t number = 0;
            while (number < limit) {
                number = number + 1;
            }
            writer.result(number);
        }
        return true;
    });
}

the host stopped waiting after 0.05 seconds; the command goes on
a second wait: done, the core counted to 50000000
the card stopped the command: command 2 timed out: slot 0, sequence 2, group first 0 count 1; not reported: core 0 reader gave up (stop) in next writer gave up (stop) in complete

Reading the last line, CommandTimeout's message:

  • command 2: the second command of this run, the one with handle 2.
  • group first 0 count 1: the cores it was sent to.
  • not reported: core 0: the core of the group that had not reported when the deadline passed.
  • what follows says where reader.cpp and writer of that core were when the card told them to stop. They are Komorebi's own step names, and you do not need them yet; lesson 39, Finding a bug uses them.

The card is fine afterwards. The counting kernel did end by itself, and a new program can open the card at once.

Common mistakes

  • Reading WaitTimeout as "the command failed". It says that the host stopped waiting.
  • A timeout that is longer than the deadline. The card's deadline passes first and the error is CommandTimeout, not WaitTimeout.
  • A deadline shorter than the honest running time of a command. The card then stops a command that was working. Set the deadline for the longest command of the program.
  • A writer that does not return true (lesson 4, Hello, core). The core never reports, and the card stops the command at its deadline with CommandTimeout.

That is the end of Part 1. You can now send work to a core, with numbers, get a number back, choose among several pieces of work, send many commands, and read what the card says when one does not finish. Part 2 does the same on many cores at once.

Next: lesson 11, A core group.