Kernel Coverage
Legend
Inference
✗ Not supported✓ Supported
Training
✗ Not supported✓ Supported✓ Supported + faster backward
✓ means training works with our faster forward kernel, but uses PyTorch's default backward pass.
✓ means we additionally provide a custom backward kernel to speed up gradient computation as well.
No shipped variants match your search and filters.
+ Request a kernelComposite kernels
Kernels produced from fusing multiple mathematical operations together.
GEGLU (go to API)
This fused pre-norm GEGLU feed-forward block computes x + down(gelu(gate(norm(x))) * up(norm(x))). It was introduced in the paper GLU Variants Improve Transformer as a gated alternative to standard Transformer feed-forward activations.
Variants
| Precision | Hardware | Shape | Provenance | Training | Inference |
|---|---|---|---|---|---|
| bf16 | SM90a (H100) | (1024, 768, 1448), static | In-house (SKC) | ✓ | ✓ |