Research question
Can you make useful GPU optimization decisions before you have access to the GPU?
Attention is a good test case because it exercises global memory, shared memory, reductions, and matrix multiplication. Its usual formulation computes scores from QKᵀ, applies softmax, and combines those weights with V. A direct implementation writes an attention matrix whose size grows quadratically with sequence length.
Phantom A100 tests whether simulation can identify bottlenecks and rank changes to this workload. It also asks a harder question: when a simulator reports a win, is that win caused by the mechanism the engineer intended?
Reporting rule. Simulation-derived speedups are not hardware benchmarks. The project reports simulated values separately from measurements taken later on an A100-SXM4-80GB.
System design
The repository separates CUDA kernel code, Python bindings, simulator infrastructure, tests, and experiment results. Kernel headers under csrc/kernels/ are shared by the PyTorch interface and the simulator. That reduces the chance that two different implementations are being compared.
The simulator supports architectural investigation; the hardware test checks whether its predictions transfer.
Why attention changes with memory strategy
The baseline materializes attention scores in global memory and launches separate stages. Tiling reuses Q, K, and V through shared memory. Fusion avoids storing the full score matrix. Vectorization reduces the number of memory instructions. Tensor Cores move the matrix multiplications away from scalar FP32 CUDA-core execution.
Kernel progression
The six versions isolate different architectural choices. The sequence matters because an optimization that removes memory traffic can still lose performance if it increases register pressure enough to reduce occupancy.
| Version | Approach | What changes |
|---|---|---|
| v1 · naive | Three-stage attention | Score matrix, softmax, and output each computed separately |
| v2 · tiled | Shared-memory tiles | 32×32 tiles and warp-per-row softmax reduce redundant reads |
| v3 · fused | Online softmax | One kernel streams K/V instead of writing the full score matrix |
| v3.1 · fused_vec | 128-bit vector loads | Lower instruction count and smaller FP16 shared-memory footprint |
| v3.2 · fused_db | cp.async buffering | Overlaps K/V tile transfers with work, where supported |
| v4 · fused_tc | Tensor Core WMMA | Moves QKᵀ and PV to matrix instructions with FP32 accumulation |
CUDA 13.2 compilation checks cover 84 kernel instantiations across sm_75, sm_80, sm_89, and sm_90. The README reports no register spills for those original instantiations. Register-capped experiments in the later tuning study are separate and did produce spills on sm_80.
Fusion has a cost
At the simulated N = 128 workload, v3 used roughly 236 registers per thread and reached about 10% occupancy. Even though it avoided the attention matrix, it ran slower than the tiled v2 kernel. Version v3.1 brought register use down to roughly 165 and improved occupancy, turning the fused approach into a faster option.
Simulated performance
The first experiments ran the CUDA kernels inside a patched GPGPU-Sim A100 model with a 14-SM configuration. Results below use BH = 42 and N = 128. These speedups describe the simulator, not an actual A100.
v3.2 is omitted because the simulator does not model cp.async timing; it was evaluated for correctness, not speed.
Making the simulator trustworthy
Five simulator issues had to be fixed before the runs could be interpreted: three WMMA emulation problems, a warp-shuffle bug that returned zero, and an uninitialized shuffle latency. Small primitive-level tests helped distinguish a broken kernel from a broken simulator. This validation work is part of the result, not just setup.
Counter-guided optimization
The second part of the project turns human bottleneck analysis into a search policy. It reads simulator counters, applies explicit diagnostic rules, ranks possible code transformations, and checks whether a proposed change produced the expected counter signature.
The guided search rediscovered the route v1 → v2 → v3 → v3.1 → v4 using 7 of 15 simulations, ending at the best trusted simulated point. It also permitted a temporary regression when a necessary structural change did not pay off immediately. In the small reported search space, guided search needed six simulations to get within 5% of the best point, versus 8.5 on average for random search.
Trust check example. Simulator runs suggested register caps could improve some kernels. The compiler, however, reported 188 bytes of register spills per thread on sm_80 in one case. The optimizer flagged that apparent win rather than accepting it.
What happened on a real A100
The project later measured 23 kernel variants on an A100-SXM4-80GB. The optimizer was frozen, and predictions were published before the hardware session. This pre-registration makes the comparison more useful than tuning a simulator after seeing measured results.
| Question | Outcome on A100 | Meaning |
|---|---|---|
| Which kernel is fastest? | v4 led all 23 | Final selection transferred |
| Do rankings transfer? | Spearman 0.800 | Matched the stated threshold |
| Does fusion lose to tiling? | No; v3 was 1.25× faster than v2 | Simulator reversed this relationship |
| Do register caps help? | Worse than simulation in 6 of 6 cases | Spill warning was useful |
| How large is the Tensor Core gain? | 3.46× measured versus 10.75× simulated | Magnitude did not transfer |
The mismatch around v3 matters. Simulation suggested that a fused version needed further changes before it became worthwhile. On real hardware, basic fusion already beat tiling. That means the search's non-greedy step was compensating for a simulator artifact, not a real hardware valley.
Direction is easier than magnitude
In a post-hoc breakdown, the simulator got the direction of 18 out of 19 structural transformations right. For tile shapes and register caps, it often predicted ties where the A100 showed meaningful differences. This suggests simulated counters can narrow design choices while still missing the details needed for final tuning.
Limits and next steps
What this study establishes
- CUDA kernels can be checked for correctness without the target GPU.
- Simulation can expose occupancy and register bottlenecks.
- Counter-guided search can select a strong endpoint under a fixed budget.
- Pre-registration makes the hardware comparison auditable.
What remains open
- Counter-by-counter fidelity on the A100 was not measured because profiling permissions were blocked.
- The guided search has not been rerun using hardware counters.
cp.asyncperformance is not modeled by the simulator.- Broader kernel families and larger search spaces are still needed.
The implementation also leaves room for register-level softmax and accumulator handling, combined asynchronous loading with Tensor Core compute, and expanded support for backward attention, head dimension 128, and BF16. The repository discusses FlashAttention-2 as a design comparison, not a verified simulated head-to-head benchmark.
Implementation and reproducibility
The source includes CUDA headers and PyTorch bindings, a simulator harness and patches, an autotune/ package, correctness tests, benchmarking scripts, and saved simulated and measured outputs. The project documents a single-command hardware run as well as CPU-only algorithm tests.
GPU commands require a compatible CUDA environment and dependencies. Consult the repository setup instructions before running the full benchmark.
Benchmark methodology described in the repository includes correctness gating before timing, CUDA-event measurements after warm-up, and reporting medians plus percentile spreads. The benchmark also records out-of-memory or incorrect-result cases instead of treating those as usable speed measurements.
Conclusion
Phantom A100 shows how far careful CUDA performance work can proceed before direct access to the target GPU. The simulator supported correctness checks, bottleneck diagnosis, and a search policy that found the strongest tested endpoint. The later A100 experiment confirmed that choice while exposing real limits in intermediate rankings and performance predictions.
The useful lesson is not that simulation replaces hardware. It is that a simulator can make the early optimization loop more disciplined, provided its failures are tested, its assumptions are visible, and its predictions are checked when hardware becomes available.
References & source material
[1] Sharma, D. Phantom A100, repository, README, CUDA implementations, simulation and autotuning results. github.com/b4batunde/Phantom-A100.
[2] Phantom A100, Autotune measured findings and pre-registration artifacts. Available under results/autotune/ in [1].
[3] Dao et al. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022). arxiv.org/abs/2205.14135.
[4] Dao. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023). arxiv.org/abs/2307.08691.
[5] Milakov and Gimelshein. Online normalizer calculation for softmax (2018). arxiv.org/abs/1805.02867.