Skip to content

Add benchmark harness and tensor GEMM benchmark#2

Open
mmshad wants to merge 1 commit into
mainfrom
add-benchmark-harness
Open

Add benchmark harness and tensor GEMM benchmark#2
mmshad wants to merge 1 commit into
mainfrom
add-benchmark-harness

Conversation

@mmshad

@mmshad mmshad commented Apr 8, 2026

Copy link
Copy Markdown

Summary

Transforms the repo from a collection of one-off SLURM scripts into a parameterized, architecture-aware benchmarking harness and adds a new tensor-core saturation benchmark (FP16/FP8).

Changes

Shared Infrastructure (common/)

File Purpose
config.sh Centralized SLURM/container defaults (partition, account, SIF path, modules). All values are env-var-overridable.
detect_arch.sh GPU architecture detection via nvidia-smi compute capability. Gates precision support (FP8 on Hopper, FP4 blocked until Blackwell).
parse_results.py Auto-detects benchmark type from .out files and produces Markdown summary tables for all 5 benchmark types.

Parameterized Launchers

Each benchmark (p1-p5) now has a single run_*.sh launcher that:

  • Accepts configuration via environment variables (NODES, GPUS_PER_NODE, N, SLOPPY_TYPE, etc.)
  • Auto-resubmits with proper --partition, --account, --constraint via exec sbatch --export=ALL if not already inside a SLURM job
  • Replaces the need to maintain separate scripts per GPU/precision configuration

Benchmark-specific features:

  • HPL -- Generates HPL.dat inline; supports sweep mode (HPL_SWEEP="92160:1:1 190464:2:2")
  • HPL-MxP -- Maps friendly precision names (SLOPPY_TYPE=FP8) to numeric flags; auto-generates GPU affinity; gates FP4 by arch + container version
  • HPCG -- Parameterized problem size and runtime
  • STREAM -- Parameterized dtype (FP32/FP64) and array size

New: Tensor-Core GEMM Saturation Benchmark (p5_tensor_gemm/)

A cuBLASLt-based microbenchmark that measures raw FP16 and FP8 tensor-core throughput via dense GEMM. This answers: "How close can my GPU get to its theoretical peak tensor-core TFLOPS?"

  • tensor_gemm_bench.cu -- ~440 lines, supports FP16 and FP8 (E4M3) on Hopper, FP4 gated to Blackwell
  • run_tensor_gemm.sh -- SLURM launcher with smoke, saturation, transformer, custom modes
  • 4 example scripts in examples/
  • Auto-detects H100 vs H200, reports % of theoretical peak

Measured results (single GPU, 1980 MHz):

GPU FP16 TFLOPS % Peak FP8 TFLOPS % Peak
H100 SXM 928.4 93.8% 1654.8 83.6%
H200 SXM 885.5 89.5% 1595.7 80.6%

README Updates

  • Added p5 section with theoretical peaks and measured results
  • Added benchmark taxonomy table (Linpack-style vs tensor-core saturation vs memory/system)
  • Added common infrastructure documentation
  • Fixed GPU count errors in p1 and p2 READMEs ("2N x 4GPUs" = 8 GPUs total, not 4)
  • Added sloppy-type precision table to p2

What's NOT Changed

  • All original scripts in runs/ directories are untouched
  • Existing benchmark output files preserved as-is

Testing

All benchmarks verified on H200 node (holygpu8a10302, 4x NVIDIA H200 SXM, 1980 MHz):

Test Result
common/config.sh -- config loads, container path resolves PASS
common/detect_arch.sh -- detects SM 90 / Hopper PASS
p1 HPL -- N=8192, residual PASSED PASS
p2 HPL-MxP FP16 -- N=8192, residual PASSED PASS
p3 HPCG -- VALID result PASS
p4 STREAM -- FP32 and FP64 bandwidth measured PASS
p5 Tensor GEMM -- FP16 and FP8 saturation sweeps PASS
FP4 gating -- correctly refused on Hopper PASS

- Add shared infrastructure (common/config.sh, detect_arch.sh, parse_results.py)
  for centralized SLURM/container configuration, GPU arch detection with
  precision gating, and unified result parsing across all benchmark types.

- Add parameterized launchers (run_*.sh) for HPL, HPL-MxP, HPCG, and STREAM
  that accept env-var configuration and auto-resubmit with proper SLURM flags.

- Add tensor-core GEMM saturation benchmark (p5_tensor_gemm/) with cuBLASLt-based
  FP16/FP8 microbenchmark, SLURM launcher with smoke/saturation/transformer/custom
  modes, and example scripts.

- Update READMEs with benchmark taxonomy, common infrastructure docs, p5 results,
  and fix GPU count errors in p1/p2 multi-node descriptions.
@mmshad
mmshad requested a review from Naeemkh April 8, 2026 04:01
@mmshad mmshad self-assigned this Apr 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant