Skip to content

Part 5. X280 threads · lesson 36 of 42

36. Twelve cores

When the same thread runs on all three clusters, kmx_parallel_all runs one function on all twelve X280 cores, and one cluster, the lead, tells the others what to do. This lesson relies on lesson 34, Four cores and lesson 35, Sharing memory.

Why three clusters

Work that is arithmetic, such as the sum of lesson 34, Four cores, gets faster with every core you add. A cluster has four. The card has three clusters that you can use, and so twelve cores in all. The price is that the clusters are three separate computers: each has its own copy of the thread's regions, in its own memory bank, and a core reads only the bank of its own cluster.

The general form

In Python, where thread was:

program.thread(
    clusters=(1, 2, 3),
    sources=[FILES],
    regions={NAME: SIZE},
    words=[NUMBERS],
)
results = run.thread_results(timeout=SECONDS)
part = run.threads[CLUSTER].read(NAME)

In the C file, one image for all three clusters:

static void piece(int cluster, int core, uint64_t argument) {
    BODY
}

int kmx_main(int core) {
    if (kmx_cluster() != kmx_lead()) {
        return kmx_serve() == KMX_OK ? 0 : 2;
    }
    BODY
    int status = kmx_parallel_all(piece, ARGUMENT);
    ...
}
  • clusters=(1, 2, 3) declares the same thread on each of these clusters. Each gets its own copy of every region, with the same names and the same ids.
  • kmx_cluster() is the number of the cluster this code runs on, and kmx_lead() is the lowest of the clusters that run this thread: the lead. With all three, the lead is cluster 1.
  • The lead runs the program. The other clusters call kmx_serve(), which waits for the lead's calls and carries them out, until the lead's kmx_main returns. kmx_serve returns KMX_OK when it ends well.
  • kmx_parallel_all(piece, ARGUMENT) is called on core 0 of the lead only. It runs piece(cluster, core, ARGUMENT) on every core of every cluster, twelve in all, and returns when all twelve have finished. ARGUMENT is one 64-bit number, the same on all of them. The first call waits until the other clusters are serving.
  • piece is yours. It gets the cluster, the core in that cluster (0 to 3), and the argument. The number of the core in the whole, 0 to 11, is (cluster - 1) * 4 + core.
  • run.thread_results(timeout=SECONDS) is thread_result for all the threads: a dictionary from cluster to result.
  • run.threads[CLUSTER].read(NAME) reads a region of one cluster's thread, after run.stop(), as in lesson 35, Sharing memory. Data from one cluster to another goes by a call kmx_push; this lesson does not need it, because Python reads each cluster's piece.

One line at a time

program.thread(
    clusters=(1, 2, 3),
    sources=[HERE / "thread" / "twelve.c"],
    regions={"pieces": 256},
    words=[N],
)

The same declaration as before, with clusters=(1, 2, 3) in the place of cluster=1. The region pieces is 256 bytes, four lines of 64: one line for each core of a cluster.

if (kmx_cluster() != kmx_lead()) {
    return kmx_serve() == KMX_OK ? 0 : 2;
}

The first lines of kmx_main. Clusters 2 and 3 never leave this if: they serve until the lead is done. Only cluster 1 goes on.

static void piece(int cluster, int core, uint64_t argument) {
    const uint32_t share = (uint32_t)argument / 12;
    const uint32_t rank = (cluster - 1) * CORES_PER_CLUSTER + core;
    uint64_t bytes;
    uint32_t* pieces = (uint32_t*)kmx_region(0, &bytes);
    pieces[core * LINE_WORDS] = scrambled_sum(rank * share, (rank + 1) * share);
    kmx_flush_at(0, core * 64, 64);
}

Each of the twelve cores takes a twelfth of the numbers, adds them as in lesson 34, Four cores, and stores the sum in its line of its own cluster's region. kmx_flush_at(0, core * 64, 64) flushes that one line, from this core alone.

uint32_t microseconds[2];
for (int call = 0; call < 2; call++) {
    const uint64_t start = kmx_time();
    if (kmx_parallel_all(piece, n) != KMX_OK) {
        return 1;
    }
    microseconds[call] = (uint32_t)((kmx_time() - start) / 50);
}

The lead calls kmx_parallel_all twice, and times each call. The sum is the same both times; the point of the second call is to see what the first one costs.

pieces = []
for thread in run.threads.values():
    lines = thread.read("pieces").view(np.uint32).reshape(4, 16)
    pieces += lines[:, 0].tolist()

After the run, Python reads the pieces region of each cluster's thread and keeps the first number of each of the four lines. Twelve pieces in all. Python adds them and compares with numpy.

One lead, two clusters that serve, twelve cores that work.

The whole program

"""Twelve cores: one thread on three clusters adds up numbers on 12 X280 cores."""

from pathlib import Path

import numpy as np

import komorebi as km

HERE = Path(__file__).resolve().parent

N = 12_000_000

program = km.Program("twelve_cores", kernels=HERE / "kernels")
program.thread(
    clusters=(1, 2, 3),
    sources=[HERE / "thread" / "twelve.c"],
    regions={"pieces": 256},
    words=[N],
)

with km.Session() as session:
    run = session.load(program)
    results = run.thread_results(timeout=20.0)
    run.stop()
    pieces = []
    for thread in run.threads.values():
        lines = thread.read("pieces").view(np.uint32).reshape(4, 16)
        pieces += lines[:, 0].tolist()

total = sum(pieces) % 2**32
numbers = np.arange(N, dtype=np.uint32)
numbers ^= numbers >> 15
numbers *= np.uint32(2246822519)
numbers ^= numbers >> 13
expected = int(numbers.sum(dtype=np.uint64)) % 2**32
print(f"twelve pieces: {pieces}")
print(f"their sum, modulo 2**32: {total}")
print(f"the scrambled numbers 0 to {N - 1}:    {expected}")
first, second = results[1].words
print(f"the first call took {first} microseconds, the second {second}")

assert all(result.code == 0 for result in results.values())
assert total == expected

/* An X280 thread on three clusters: twelve cores add up scrambled numbers 0 to n - 1,
 * each a twelfth of them, and store their piece in a line of their cluster's region. */
#include "kmx.h"

#define LINE_WORDS 16 /* a 64-byte cache line holds 16 words */
#define CORES_PER_CLUSTER 4

// A number made from i, with no formula for the sum of them.
static uint32_t scramble(uint32_t i) {
    i ^= i >> 15;
    i *= 2246822519u;
    i ^= i >> 13;
    return i;
}

static uint32_t scrambled_sum(uint32_t first, uint32_t last) {
    uint32_t sum = 0;
    for (uint32_t i = first; i < last; i++) {
        sum += scramble(i);
    }
    return sum;
}

static void piece(int cluster, int core, uint64_t argument) {
    const uint32_t share = (uint32_t)argument / 12;
    const uint32_t rank = (cluster - 1) * CORES_PER_CLUSTER + core;
    uint64_t bytes;
    uint32_t* pieces = (uint32_t*)kmx_region(0, &bytes);
    pieces[core * LINE_WORDS] = scrambled_sum(rank * share, (rank + 1) * share);
    kmx_flush_at(0, core * 64, 64);
}


int kmx_main(int core) {
    (void)core;
    if (kmx_cluster() != kmx_lead()) {
        return kmx_serve() == KMX_OK ? 0 : 2;
    }
    const uint32_t n = kmx_args_get()->words[0];

    uint32_t microseconds[2];
    for (int call = 0; call < 2; call++) {
        const uint64_t start = kmx_time();
        if (kmx_parallel_all(piece, n) != KMX_OK) {
            return 1;
        }
        microseconds[call] = (uint32_t)((kmx_time() - start) / 50);
    }
    kmx_result(0, microseconds, 2);
    return 0;
}

The kernels are the empty ones. Run it from the folder tutorial/36_twelve_cores:

python twelve_cores.py

twelve pieces: [3536795701, 2585887106, 491538612, 1823753602, 1767080444, 812221010, 3242642678, 283345683, 1337308, 3339338716, 1264411118, 2583323059]
their sum, modulo 2**32: 256838557
the scrambled numbers 0 to 11999999:    256838557
the first call took 67265 microseconds, the second 6003

The twelve pieces add up to the sum numpy computes. The second call took about six milliseconds for 12 million numbers. By the rate of lesson 34, Four cores, one core adds four million of them in 24 milliseconds, so one core would take 72 for these twelve million: the twelve cores were 12 times as fast. The first call took far longer, tens of milliseconds, and its time changed from run to run, as the other clusters were still starting. A thread that calls kmx_parallel_all once pays that wait once.

Work that mostly moves memory does not gain as much as this sum does. Each cluster reads its bank at a fixed speed, so more cores do not make memory move faster. Arithmetic scales; copying does not.

Common mistakes

  • Writing the thread for one cluster and declaring it on three. Every cluster runs the same kmx_main. The if that sends the others to kmx_serve is what makes them serve. Without it, the lead's first kmx_parallel_all waits for clusters that are not serving.
  • A piece that reads a region of another cluster. A core reaches only the bank of its own cluster. Put the data in each cluster's own region, or push it with kmx_push.
  • A number of items that twelve does not divide. As in lesson 34, Four cores, the last items belong to nobody's piece, and the sum is wrong without a message.
  • Counting the first call. Time the second, or time the whole program.

Next: lesson 37, A core asks the X280 turns round: a Tensix core asks the X280 for something.