Using sk-check

sk-check validates generated CUDA kernels in KernelBench format. When contributing to Standard Kernels, use it against a small validation candidate that imports the public standardkernels API. That proves the integrated dispatch path, rather than a separate copy of the generated kernel.

KernelBench input format

Each check takes two Python files. reference.py defines the trusted PyTorch model and its input generators:

import torch


class Model(torch.nn.Module):
    def __init__(self, scale):
        super().__init__()
        self.scale = scale

    def forward(self, x):
        return x * self.scale


def get_init_inputs():
    return [2.0]


def get_inputs():
    return [torch.randn(1024, 768, dtype=torch.bfloat16)]

candidate.py defines ModelNew. It must accept the same init inputs and forward inputs and return the same tensor structure:

import torch
from standardkernels.ops import your_public_op


class ModelNew(torch.nn.Module):
    def __init__(self, scale):
        super().__init__()
        self.scale = scale

    def forward(self, x):
        return your_public_op(x, self.scale)

Configure the checks

Next, configure the checks in sk-check. sk-check has many checks, for this tutorial we'll use the timing-matrix and compute-sanitizer checks.

For now, we'll set up the The Timing Matrix check with the following config:

timing-matrix.json

{
  "schema_version": 1,
  "seed": 42,
  "timing_warmups": 10,
  "timing_trials": 100,
  "atol": 0.01,
  "rtol": 0.01,
  "min_percent_within_tolerance": 99.0,
  "cuda_device_index": 0
}

And we'll set up the Compute Sanitizer check with the following config:

compute-sanitizer.json

{
  "schema_version": 1,
  "seed": 42,
  "graph_replays": 3,
  "graph_warmups": 1,
  "tool_timeout_seconds": 600.0,
  "print_limit": 20,
  "cuda_device_index": 0
}

Run the checks

Now you can run the checks like so:

STANDARDKERNELS_STRICT=1 sk-check deterministic-test timing-matrix \
  --reference path/to/reference.py \
  --candidate path/to/library_candidate.py \
  --check-config path/to/timing-matrix.json \
  --run-root artifacts/sk-check/timing-matrix

STANDARDKERNELS_STRICT=1 sk-check deterministic-test compute-sanitizer \
  --reference path/to/reference.py \
  --candidate path/to/library_candidate.py \
  --check-config path/to/compute-sanitizer.json \
  --run-root artifacts/sk-check/compute-sanitizer

Timing Matrix compares the outputs and timings in eager, CUDA Graph, torch.compile, and combined graph/compile configurations. Meanwhile, Compute Sanitizer runs memcheck, initcheck, racecheck, and synccheck.

CUDA Graph support for Standard Kernels

Support for CUDA Graph capture of ops from sk-lib is being added to sk-check. Currently, you can capture and replay CUDA graphs for sk-lib ops, but sk-check cannot yet run its graph-based checks on them.

Until then, please use the non-graph Timing Matrix results for eager execution and torch.compile with CUDA Graphs disabled.