Phantom A100 / Technical paper
PA.
Engineering research / Technical report
2026 · Version 1.0
GPU systems · CUDA optimization · Performance research

Phantom A100:
Optimizing attention without direct GPU access

A CUDA attention study that uses architectural simulation to guide kernel design, then checks which predictions hold on an NVIDIA A100.

AuthorDev Sharma
ImplementationCUDA C++ · PyTorch · GPGPU-Sim · A100
ProjectSource code ↗

Abstract

GPU kernels are hard to optimize when the target hardware is unavailable. Phantom A100 studies that problem using transformer attention as a workload. The project implements six kernel variants, starting with a naive attention pipeline and progressing through shared-memory tiling, online-softmax fusion, vectorized memory operations, asynchronous double buffering, and Tensor Core execution. A patched GPGPU-Sim model provides correctness checks, performance counters, and simulated rankings. The project then adds a counter-guided optimizer that proposes and verifies transformations. Its decisions were frozen before a later A100 hardware test. Simulation predicted a 57.9× speedup for the Tensor Core variant over the naive kernel. On actual hardware, that variant remained the fastest of 23 tested kernels, but several intermediate rankings and predicted magnitudes differed. The main result is a repeatable workflow for separating useful simulator guidance from claims that require hardware measurement.

CUDA kernelsGPU architectureAttentionGPGPU-SimTensor CoresCounter-guided tuning

Research question

Can you make useful GPU optimization decisions before you have access to the GPU?

Attention is a good test case because it exercises global memory, shared memory, reductions, and matrix multiplication. Its usual formulation computes scores from QKᵀ, applies softmax, and combines those weights with V. A direct implementation writes an attention matrix whose size grows quadratically with sequence length.

Phantom A100 tests whether simulation can identify bottlenecks and rank changes to this workload. It also asks a harder question: when a simulator reports a win, is that win caused by the mechanism the engineer intended?

Reporting rule. Simulation-derived speedups are not hardware benchmarks. The project reports simulated values separately from measurements taken later on an A100-SXM4-80GB.

System design

The repository separates CUDA kernel code, Python bindings, simulator infrastructure, tests, and experiment results. Kernel headers under csrc/kernels/ are shared by the PyTorch interface and the simulator. That reduces the chance that two different implementations are being compared.

Research workflowFrom kernel code to checked claims
BUILD
CUDA variantsSame attention operation, different memory and compute strategies
VERIFY
Correctness testsFloat64 / FP32 reference checks, causal and non-causal cases
SIMULATE
GPGPU-SimCycle model, latency, registers, occupancy, throughput
DIAGNOSE
Counter rulesIdentify bottlenecks and propose transformations
PREDICT
Pre-registerFreeze optimizer, hypotheses, and expected rankings
MEASURE
Real A100Time 23 variants and compare with frozen predictions

The simulator supports architectural investigation; the hardware test checks whether its predictions transfer.

Why attention changes with memory strategy

The baseline materializes attention scores in global memory and launches separate stages. Tiling reuses Q, K, and V through shared memory. Fusion avoids storing the full score matrix. Vectorization reduces the number of memory instructions. Tensor Cores move the matrix multiplications away from scalar FP32 CUDA-core execution.

Attention(Q, K, V) = softmax(QKᵀ / √d) V

Kernel progression

The six versions isolate different architectural choices. The sequence matters because an optimization that removes memory traffic can still lose performance if it increases register pressure enough to reduce occupancy.

Implemented variantsFrom repository README
VersionApproachWhat changes
v1 · naiveThree-stage attentionScore matrix, softmax, and output each computed separately
v2 · tiledShared-memory tiles32×32 tiles and warp-per-row softmax reduce redundant reads
v3 · fusedOnline softmaxOne kernel streams K/V instead of writing the full score matrix
v3.1 · fused_vec128-bit vector loadsLower instruction count and smaller FP16 shared-memory footprint
v3.2 · fused_dbcp.async bufferingOverlaps K/V tile transfers with work, where supported
v4 · fused_tcTensor Core WMMAMoves QKᵀ and PV to matrix instructions with FP32 accumulation

CUDA 13.2 compilation checks cover 84 kernel instantiations across sm_75, sm_80, sm_89, and sm_90. The README reports no register spills for those original instantiations. Register-capped experiments in the later tuning study are separate and did produce spills on sm_80.

Fusion has a cost

At the simulated N = 128 workload, v3 used roughly 236 registers per thread and reached about 10% occupancy. Even though it avoided the attention matrix, it ran slower than the tiled v2 kernel. Version v3.1 brought register use down to roughly 165 and improved occupancy, turning the fused approach into a faster option.

Simulated performance

The first experiments ran the CUDA kernels inside a patched GPGPU-Sim A100 model with a 14-SM configuration. Results below use BH = 42 and N = 128. These speedups describe the simulator, not an actual A100.

57.9×
v4 compared with v1 at the stated workload
24 / 24
All six versions across tested causal and shape cases
9.5e−4
Against a float64 reference in simulator tests
Speedup over naive kernelSimulated · GPGPU-Sim
v1 naive
1.0×
v2 tiled
3.6×
v3 fused
2.2×
v3.1 vec
5.4×
v4 tensor
57.9×

v3.2 is omitted because the simulator does not model cp.async timing; it was evaluated for correctness, not speed.

Making the simulator trustworthy

Five simulator issues had to be fixed before the runs could be interpreted: three WMMA emulation problems, a warp-shuffle bug that returned zero, and an uninitialized shuffle latency. Small primitive-level tests helped distinguish a broken kernel from a broken simulator. This validation work is part of the result, not just setup.

Counter-guided optimization

The second part of the project turns human bottleneck analysis into a search policy. It reads simulator counters, applies explicit diagnostic rules, ranks possible code transformations, and checks whether a proposed change produced the expected counter signature.

Decision loopInterpretable, predict-then-verify search
OBSERVE
Read countersOccupancy, instruction counts, memory and compute limits
HYPOTHESIZE
Choose a changeSpecify what metric should improve and why
TEST
Run candidateSimulate and verify the output remains correct
COMPARE
Check signatureDid the expected mechanism occur?
TRUST
Reject weak winsFlag register spills or unsupported simulator features
CONTINUE
Update searchTry the next candidate under the run budget

The guided search rediscovered the route v1 → v2 → v3 → v3.1 → v4 using 7 of 15 simulations, ending at the best trusted simulated point. It also permitted a temporary regression when a necessary structural change did not pay off immediately. In the small reported search space, guided search needed six simulations to get within 5% of the best point, versus 8.5 on average for random search.

Trust check example. Simulator runs suggested register caps could improve some kernels. The compiler, however, reported 188 bytes of register spills per thread on sm_80 in one case. The optimizer flagged that apparent win rather than accepting it.

What happened on a real A100

The project later measured 23 kernel variants on an A100-SXM4-80GB. The optimizer was frozen, and predictions were published before the hardware session. This pre-registration makes the comparison more useful than tuning a simulator after seeing measured results.

23
A100-SXM4-80GB
0.800
Spearman, simulated versus measured
3.46×
Versus 10.75× predicted by simulation for that comparison
Prediction auditPre-registered claims and later measurement
QuestionOutcome on A100Meaning
Which kernel is fastest?v4 led all 23Final selection transferred
Do rankings transfer?Spearman 0.800Matched the stated threshold
Does fusion lose to tiling?No; v3 was 1.25× faster than v2Simulator reversed this relationship
Do register caps help?Worse than simulation in 6 of 6 casesSpill warning was useful
How large is the Tensor Core gain?3.46× measured versus 10.75× simulatedMagnitude did not transfer

The mismatch around v3 matters. Simulation suggested that a fused version needed further changes before it became worthwhile. On real hardware, basic fusion already beat tiling. That means the search's non-greedy step was compensating for a simulator artifact, not a real hardware valley.

Direction is easier than magnitude

In a post-hoc breakdown, the simulator got the direction of 18 out of 19 structural transformations right. For tile shapes and register caps, it often predicted ties where the A100 showed meaningful differences. This suggests simulated counters can narrow design choices while still missing the details needed for final tuning.

Limits and next steps

What this study establishes

  • CUDA kernels can be checked for correctness without the target GPU.
  • Simulation can expose occupancy and register bottlenecks.
  • Counter-guided search can select a strong endpoint under a fixed budget.
  • Pre-registration makes the hardware comparison auditable.

What remains open

  • Counter-by-counter fidelity on the A100 was not measured because profiling permissions were blocked.
  • The guided search has not been rerun using hardware counters.
  • cp.async performance is not modeled by the simulator.
  • Broader kernel families and larger search spaces are still needed.

The implementation also leaves room for register-level softmax and accumulator handling, combined asynchronous loading with Tensor Core compute, and expanded support for backward attention, head dimension 128, and BF16. The repository discusses FlashAttention-2 as a design comparison, not a verified simulated head-to-head benchmark.

Implementation and reproducibility

The source includes CUDA headers and PyTorch bindings, a simulator harness and patches, an autotune/ package, correctness tests, benchmarking scripts, and saved simulated and measured outputs. The project documents a single-command hardware run as well as CPU-only algorithm tests.

# Clone and inspect the project git clone https://github.com/b4batunde/Phantom-A100.git cd Phantom-A100 # CPU-only check of online-softmax mathematics python -m pytest tests/test_algorithm.py # On a supported GPU, after environment setup python -m pytest -q tests python bench/benchmark.py # Project's scripted full GPU experiment bash scripts/run_all.sh

GPU commands require a compatible CUDA environment and dependencies. Consult the repository setup instructions before running the full benchmark.

Benchmark methodology described in the repository includes correctness gating before timing, CUDA-event measurements after warm-up, and reporting medians plus percentile spreads. The benchmark also records out-of-memory or incorrect-result cases instead of treating those as usable speed measurements.

Conclusion

Phantom A100 shows how far careful CUDA performance work can proceed before direct access to the target GPU. The simulator supported correctness checks, bottleneck diagnosis, and a search policy that found the strongest tested endpoint. The later A100 experiment confirmed that choice while exposing real limits in intermediate rankings and performance predictions.

The useful lesson is not that simulation replaces hardware. It is that a simulator can make the early optimization loop more disciplined, provided its failures are tested, its assumptions are visible, and its predictions are checked when hardware becomes available.

References & source material

[1] Sharma, D. Phantom A100, repository, README, CUDA implementations, simulation and autotuning results. github.com/b4batunde/Phantom-A100.

[2] Phantom A100, Autotune measured findings and pre-registration artifacts. Available under results/autotune/ in [1].

[3] Dao et al. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022). arxiv.org/abs/2205.14135.

[4] Dao. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023). arxiv.org/abs/2307.08691.

[5] Milakov and Gimelshein. Online normalizer calculation for softmax (2018). arxiv.org/abs/1805.02867.