Field manual · 7 November 2026
The DGX Spark field manual: what I learned controlling GB10 without the CUDA stack
I drive the GPU in my DGX Spark myself: my own driver calls, my own command streams, my own machine code, written in Omega. No CUDA runtime, no nvcc. These are the rules the hardware taught me along the way. Each one shows what the docs say, what I saw, the fix, and where the code lives so you can check it.
Labels: DOCUMENTED means a vendor source says it. OBSERVED means I saw it. CHIP_TESTED means it ran on the GB10 itself; HOST_TESTED means it was checked on the CPU only. INFERENCE means a conclusion I drew, not a measurement.
Get the code
Everything cited below is published on Radicle from my own node. Clone a repository with its ID, then open the path given:
rad clone rad:z4RxD8vXigmWg61nM8miQ8DVPAXEP # CUDApp
rad clone rad:z49She8eifZpLd2aiSvk2pph3DMUo # physics-mv
rad clone rad:z6do9WDBhuSnvopDkgFdFr1yWA62 # omega-mvCommits cited: CUDApp b774723f, physics-mv 56d02ae5, omega-mv 943d9555. The full list of repositories is on the switched-off page.
The cuda-crack repository: rad clone rad:z631r2JJiVT8it4tST7RK24hpowi.
Replicate it with any AI agent: AGENT-PROMPT.md
The paper behind this work: Reverse Engineering the CUDA Software Stack, version 2, DOI 10.5281/zenodo.23677465. It is a preprint, not peer reviewed.
1. A flag the CPU polls must be uncached for the GPU
Symptom. The GPU finishes, but the host does not see the result. Or the host rewrites a buffer and the next launch still reads the old bytes.
What the docs say. NVIDIA's open kernel modules, src/common/sdk/nvidia/inc/nvos.h lines 1114 to 1118 (580.173.02), on the GPU_CACHEABLE attribute:
"YES - Enable gpu caching if supported on this surface type. For system memory this will not be coherent with direct CPU mappings."
The GB10 whitepaper says the chip-to-chip interface "allows full cache coherency between CPU and GPU" (GB10 whitepaper, PDF page 6). That sentence is about the link between the dies. It does not make the GPU's own L2 cache coherent with a CPU mapping of a cached buffer, and the driver header above says so.
What I measured. OBSERVED, CHIP_TESTED. A test uploads one weight tensor, runs a launch, rewrites the same buffer from the CPU, and launches again. With the buffer GPU-cached, 4096 of 4096 cache lines came back stale on every launch, 5 runs out of 5. With the buffer uncached, every run read fresh data, 5 out of 5. Record: CUDApp tests/gpu_l2_stale_test.c lines 2 to 19.
The fix. Allocate every word that both sides poll with GPU caching off. My driver layer has a separate call for it, nvrm_alloc_gpu_uncached, next to the default cached nvrm_alloc. Use it for flags and queues, not for bulk data.
Reproduce. physics-mv nvrm/nvrm.c lines 545 to 555 (the two allocators and the reason in the comment); CUDApp tests/gpu_l2_stale_test.c (the test and its negative control, --cached 0).
2. Uncached is not enough: flush L2, then write a second marker
Symptom. The completion flag arrives, but the output the kernel wrote is not all in memory yet. Moving only the flag to uncached memory just moves the stall somewhere else.
What the docs say. The channel class defines the flush as a memory operation, src/common/sdk/nvidia/inc/class/clc96f.h line 72: NVC96F_MEM_OP_D_OPERATION_L2_FLUSH_DIRTY 0x00000010. The public headers do not say when a GB10 compute result becomes visible to the CPU. The docs are silent on that ordering.
What I measured. OBSERVED, CHIP_TESTED. This release block has run on the chip: physics-mv m16/m16_native.h line 35 records it as "the block m16/m16_concurrent.c proved on the chip". The Omega launch in finding 5 ends with the same completion markers and final cache flush, and it passed on the chip (CUDApp docs/speculation/NATIVE-KERNEL.md lines 8 to 11). My native Omega driver also passed 2,052 distinct completion payloads across two contexts, including ring wrap; a deliberately wrong expected payload timed out and latched a fault, which is the negative control (CUDApp docs/speculation/NATIVE-DEVICE.md line 57).
The fix. End every submission with this sequence: wait for idle and release the first marker; issue the L2_FLUSH_DIRTY memory operation (methods 0x28 to 0x34, operation 0x10 in bits 31 to 27); then release a second marker, in uncached memory. The host reads output only after it has seen both markers. INFERENCE: on the CUDA side the closest equivalent is __threadfence_system() before the flag store.
Reproduce. CUDApp m16/m16_concurrent.c lines 60 to 85 (build_release_flushed); physics-mv m16/m16_native.h lines 34 to 40; the Omega version is CUDApp native/blackwell/launch.om line 219.
3. DRAM is the ceiling, not the chip-to-chip link
Symptom. Bandwidth plans built on the link speed promise more than the machine can deliver.
What the docs say. The DGX Spark hardware overview: "128 GB LPDDR5x unified system memory, 256-bit interface, 4266 MHz, 273 GB/s bandwidth" and "16 channels (256 bit) LPDDR5X 8533". The Hot Chips 2025 GB10 talk, slide 9: "~600GB/s of aggregate bandwidth". NVIDIA's own documents also disagree on the coherency protocol: the whitepaper says "Arm AMBA CHI-G" (PDF page 7), the Hot Chips slides say "CHI-E". Neither gives the link speed per direction.
What I measured. OBSERVED, CHIP_TESTED as a driver report, not a throughput test. On my DGX Spark, nvidia-smi c2c -s reports 8 links at 43.125 GB/s each. The arithmetic, checked in Wolfram: 8 × 43.125 = 345 GB/s per direction for the link, and 8533 MT/s × 256 bit ÷ 8 = 273.056 GB/s for DRAM. So the link has headroom and the memory behind it sets the limit. My own bandwidth test so far ran on the CPU side only; the GPU-side measurement is still pending. No performance counter for the link is exposed.
The fix. INFERENCE: plan against 273 GB/s. Every byte the GPU reads comes from the same LPDDR5x the CPU uses, so a kernel that is limited by memory is limited by DRAM, and the CPU and GPU share that ceiling.
Reproduce. On any DGX Spark: nvidia-smi c2c -s and nvidia-smi -q. This is a reading from the vendor tool, so no repository is needed.
4. sm_121 is compute capability 12.1, not datacenter Blackwell
Symptom. Code, kernels and tuning advice written for B200 or GB200 do not apply, or do not build, for GB10.
What the docs say. NVIDIA's CUDA GPU compute capability table lists "NVIDIA GB10 (DGX Spark)" under 12.1, in the workstation column. "NVIDIA GB200 NVIDIA B200" are 10.0. Same brand name, different instruction target.
What I measured. DOCUMENTED for the number. OBSERVED, HOST_TESTED for my encoder: it writes 128-bit sm_121 instruction words, and a gate reads each instruction form back with NVIDIA's disassembler, run only on my bytes, and compares the name it prints with the name my code generator gave it. Record: omega-mv seed/sm121/gate6.sh step 6 and seed/sm121/README.md; CUDApp om/encoder/bw_enc_sm121.om lines 1 to 20.
The fix. Target sm_121 by name. Treat anything written for sm_100 as a different chip until you have checked it on GB10.
Reproduce. CUDApp om/encoder/bw_enc_sm121.om; omega-mv seed/sm121/sm121.tbl (the instruction table, one quarter-word per line).
5. sm_121 machine code straight from Omega, no nvcc
Symptom. None. This is the result: on a GB10 the CUDA toolkit is not required to put working code on the GPU.
What the docs say. I found no NVIDIA document of the sm_121 instruction encoding. Each line of my instruction table names the line of my own code generator it was read from, and the disassembler gate in finding 4 checks the forms.
What I measured. OBSERVED, CHIP_TESTED, 27 October 2026. My instruction encoder, written in Omega, wrote the machine code for a vector add on the GB10. The log reads "kernel built in 340 us: 32 instructions, 12 registers, no compiler run, no cache" and "checked 1048576 of 1048576 elements: 0 wrong, 0 never written". The planted-bug control in the same run reads "1 wrong, 1 never written" and FAIL, as it should. The machine code of all 70 kernels in that run is byte-identical to my C encoder's, and the run's doctor log says "CUDA toolkit: not needed and not used". Record: CUDApp evidence/CUDAPP-OM-ENCODER/chip-20261027T080240Z/ (om-hello.log, om-hello-control.log, kernels-om.sha256, kernels-c.sha256, om-doctor.log).
Limits, said plainly: in that run, register allocation, the kernel builders, the runtime and the driver client were still my C code, not Omega. A driver and launch path written entirely in Omega ran a vector add once (CUDApp evidence/speculation/native-kernel/receipt.json, source commit 0cf8cd7d), but its sources have changed since, so the current code is NOT_RUN on the chip. On 7 November I found two encoder defects by reading and diffing: a dropped load offset, whose fix is HOST_TESTED, and ignored predicates on most operations, whose fix is not written yet. This runs on the GB10 only, and it is not a drop-in CUDA replacement.
Reproduce. CUDApp om/encoder/bw_enc_sm121.om (the encoder) and the evidence directory above; for the all-Omega path, CUDApp native/blackwell/launch.om, native/nvrm/context.om and docs/speculation/NATIVE-KERNEL.md; omega-mv seed/sm121/README.md.
6. Unified memory rules
Symptom. nvidia-smi shows no memory use, and cudaMemGetInfo reports less free memory than you can actually allocate.
What the docs say. The DGX Spark known issues: nvidia-smi "will display “Memory-Usage: Not Supported” even though per-process GPU memory is listed. This is expected because iGPUs do not have dedicated framebuffer memory." And: "the memory size reported by cudaMemGetInfo may be smaller than the actual allocatable memory". The driver header adds the cache rule from finding 1: the default policy is "gpu uncached for system memory" (nvos.h line 1116).
What I measured. DOCUMENTED. My own rules follow from findings 1 to 3, which are CHIP_TESTED.
The fix. One pool of 128 GB serves CPU and GPU. Read free memory from /proc/meminfo (MemAvailable plus SwapFree), as NVIDIA's note shows. Keep polled words uncached and bulk data cached, and flush before the host reads.
Reproduce. physics-mv nvrm/nvrm.c lines 545 to 555; CUDApp tests/gpu_l2_stale_test.c.
The full findings list
Every finding I am ready to publish, one line each, with its label and where to check it. MEASURED-TOOL means a system tool on this machine reported it, not my own kernel. Hypothesis and interpretation rows are marked as such. A few findings wait for their receipt to be attached and are not listed yet.
| Finding | Label | Where to check |
|---|---|---|
The GB10 is a Blackwell GPU with compute capability 12.1 (sm_121) plus a 20-core Arm CPU: 10 Cortex-X925 and 10 Cortex-A725. | DOCUMENTED | GB10 whitepaper |
| 128 GB of LPDDR5x is one pool shared by CPU and GPU, at 273 GB/s. There is no separate GPU memory. | DOCUMENTED | DGX Spark hardware overview |
| The memory controllers sit on the CPU die. The GPU die holds compute and the L2. Every byte the GPU moves to or from DRAM crosses the chip-to-chip link. | DOCUMENTED | GB10 whitepaper, figures 3 and 6 |
| The link reports 345 GB/s per direction, above the 273 GB/s of DRAM, so DRAM sets the limit for bulk traffic. The GPU-side measurement is still pending. | MEASURED-TOOL | Finding 3 above |
| NVIDIA's own documents disagree on the coherency protocol and on the link bandwidth. Trust what the machine reports. | DOCUMENTED | Finding 3 above |
No performance counter for the link is exposed: /sys/bus/event_source lists only smmuv3_pmcg, armv8_pmuv3 and arm_spe. | MEASURED-TOOL | ls /sys/bus/event_source/devices on any DGX Spark |
| The GPU's own PCIe link shows Gen1 x1. It carries control only; data goes over the chip-to-chip link. | MEASURED-TOOL | nvidia-smi -q on any DGX Spark |
| First-touch latency on one CPU core: L2 about 8 ns, L3 about 40 to 57 ns, DRAM about 130 to 155 ns. | HOST_TESTED | My measurement notes; not in a public repository yet |
cudaMemGetInfo may report less free memory than you can allocate, because the CPU can swap pages. | DOCUMENTED | DGX Spark known issues |
The GPU identifies as PCI 10de:2e12, "NVIDIA GB10", sm_121. | CHIP_TESTED | CUDApp src/omega_gpu_chip.c line 15; the doctor log in finding 5 |
Everything libcuda does to the hardware goes through ioctls on three device files: /dev/nvidiactl, /dev/nvidia<minor> and /dev/nvidia-uvm (opened twice). The open kernel module defines every call in public MIT headers. | CHIP_TESTED | CUDApp docs/DRIVER.md lines 7 to 13 |
| Opening the GPU takes 11 steps, from the root client to registering the GPU with the unified-memory driver. | CHIP_TESTED | CUDApp docs/DRIVER.md lines 15 to 30; nvrm/nvrm.c nvrm_open() |
Three classes are required: BLACKWELL_COMPUTE_B, BLACKWELL_CHANNEL_GPFIFO_A and HOPPER_USERMODE_A. The doorbell is one user-space store into the mapped 64 KiB usermode page. | CHIP_TESTED | CUDApp docs/DRIVER.md lines 23 to 24; nvrm/nvrm.c lines 45 and 66 |
The driver interface is pinned to open module 580.173.02. My make doctor refuses other versions. | CHIP_TESTED | CUDApp README.md; the doctor log in finding 5 |
| My driver client is about 900 lines of C, written against 67 unchanged MIT headers. | CHIP_TESTED | CUDApp docs/DRIVER.md line 3 |
| The launch order (code pointer, launch descriptor, cache invalidates, fire) came from Mesa's NVK driver and the open class headers. | CHIP_TESTED | CUDApp docs/HOW_I_DID_IT.md line 45; src/omega_gpu_cmdlist.c |
| An extra dependent launch with a wait-for-idle barrier cost 25.09 to 31.97 µs in one passing run; without the barrier, 2.68 to 9.68 µs. These figures are for that workload only. | CHIP_TESTED | CUDApp evidence/OWN30B-LAUNCH-COST-20261014T0120Z/r2/receipt.json |
| GPU-cacheable system memory is not coherent with CPU mappings, by NVIDIA's own header. | DOCUMENTED | Finding 1 above |
| Hypothesis, not proven: when the CPU rewrites a buffer the chip may still read, the chip can read its own stale L2 copy. "Wrong only under load or on the second dispatch" points here first. | Hypothesis | Open test idea |
| Interpretation: "full cache coherency between CPU and GPU" describes the link between the dies, not the GPU's L2, so the flush in finding 2 is still required. | INFERENCE | Findings 1 and 2 above |
My runtime refuses a cached host-polled buffer outright (OMEGA_GPU_PLAT_REFUSED_CACHE_POLICY). | HOST_TESTED | CUDApp docs/runtime/PORTABILITY.md lines 125 to 126 |
| After one uncertain completion, the session refuses further submissions by design. Fix the first stall; do not retry. | IMPLEMENTED | CUDApp src/omega_gpu_session.c |
| NVIDIA does not publish the machine-code encodings. I recovered them three ways: my own probe kernels compiled offline and read back one detail at a time; the MIT-licensed Mesa NAK sources, with no code copied; and acceptance only when the chip runs the encoding correctly. | CHIP_TESTED | CUDApp docs/HOW_I_DID_IT.md lines 31 to 41; THIRD_PARTY.md |
| Every instruction is 128 bits. The vector add is 32 instructions, 512 bytes. The Omega encoder handles IR operations up to 63 and refuses programs longer than 1024 instructions. | CHIP_TESTED | CUDApp om/encoder/bw_enc_sm121.om lines 8, 17 and 471; the paper |
| Nine FP32 transcendental functions match a fixed CPU sequence on all 4,294,967,296 inputs, with zero mismatches. This is parity, not correct rounding, and sine and cosine have a restricted domain. | CHIP_TESTED | omega-mv evidence/E1-TRANSC-GB10/ at 3364a1df |
| My Omega encoder and my C encoder are byte-identical on the 70 kernels the chip ran and on 243 kernels on the host. | CHIP_TESTED, HOST_TESTED | CUDApp evidence/CUDAPP-OM-ENCODER/chip-20261027T080240Z/ |
| The GSP firmware is signature-checked by the chip and boots nothing else, so it stays NVIDIA's. | DOCUMENTED | CUDApp docs/HOW_I_DID_IT.md line 60 |
| My own kernel driver, meant to replace NVIDIA's, has run on the host, in QEMU and in a simulator only. Nothing of it has run on the GB10. | HOST_TESTED | CUDApp README.md, driver row |
Other DGX Spark owners report Docker images without sm_121a kernels, a missing environment variable that silently breaks NVFP4 MoE, and hard hangs on deep-context speculative decoding. | Third-party reports | CUDApp docs/PAIN_POINTS.md section 2.12, with numbered sources |
What is not here
NVIDIA software still in my path: the open kernel modules, version 580.173.02, and NVIDIA's signed GSP firmware; the GPU checks its signature and boots nothing else. NVIDIA's compiler and disassembler were used offline as references only, to learn encodings and to read my own bytes back. I built all of this by directing AI coding agents. Every result is from one machine family, the GB10.
No NVIDIA binary, firmware, key or document copy is published on this page; vendor sources are linked and quoted briefly. Nothing on this page changes thermal, power or safety protection. If a finding is wrong, tell me: drake@aienos.com or the contact page.