GPU Hardware and Architecture

TL;DR

• The problem: an H100 has 3.2× the BF16 compute of an A100, yet batch-1 decode of a 7B model is only 1.64× faster. Each generated token reads all 14 GB of weights from memory, so the step is limited by memory bandwidth, and the two cards' bandwidth ratio is 3.35 / 2.039 ≈ 1.64.
• The approach: read a spec sheet as three ledgers — compute (SMs × operations per SM per clock × clock), memory (registers → shared memory → L2 → HBM, capacity and bandwidth) and interconnect (NVLink / PCIe) — then use the roofline to see which ledger a kernel hits first.
• The key line: attainable = min(peak compute, bandwidth × arithmetic intensity). The intensity where the two meet is the ridge point, about 989 / 3.35 ≈ 295 FLOP/byte for BF16 on H100.
• The usual mistake: quoting the starred Tensor Core figure as the peak. The H100 page lists 1,979 BF16 TFLOPS with 2:4 structured sparsity; a dense matmul peaks at half that, 989 TFLOPS.

Why It Matters

Start with a sizing question. A 7B-parameter model stored in BF16 (2 bytes per parameter) has 14 GB of weights. With one request at a time (batch 1), the model generates tokens one by one, and every layer's matmul needs all of that layer's weights, so all 14 GB move from HBM to the compute units once per token.

How much compute is that? At batch 1, each parameter takes part in one multiply-add, about 2 × 7e9 = 1.4e10 FLOPs. At H100's dense BF16 peak of 989 TFLOPS that takes 0.014 ms. Moving 14 GB at 3.35 TB/s takes 14 / 3350 ≈ 4.2 ms.

So bandwidth sets the speed. An A100 80GB (2.039 TB/s) needs 6.9 ms. The time ratio is 1.64, exactly the bandwidth ratio, and the 3.2× compute ratio plays no part.

Flip the workload to a large training matmul, such as 4096 × 4096 times 4096 × 4096, and every byte read feeds over a thousand operations; now compute sets the speed. Same card, different workload, different bottleneck.

This page splits a GPU into compute, memory and interconnect, shows where each spec-sheet number comes from and when to look at it. Everything here is paper arithmetic on a CPU: it gives upper bounds, and real speed has to be measured on a GPU.

Walk the Shapes First

Take an H100 SXM apart from the outside in (SXM is the board-mounted form factor; H100 PCIe is a different product), with A100 alongside. Figures come from NVIDIA's product pages and Hopper architecture overview, linked at the end.

Each SM has four partitions. On H100 each partition has one warp scheduler, 32 FP32 units and one Tensor Core. Every clock the scheduler picks a ready warp from those resident in its partition and issues its next instruction.

Multiply the table out and you get the spec-sheet peaks:

The last row surprises people: all SMs' registers add up to 33 MB, about the size of L2. The storage closest to the compute units is not small; it is spread across 132 SMs, and each thread sees only its own slice.

How to run: execute the code blocks below in order in one Python session; they use only the standard library. They check the spec-sheet arithmetic and measure no real GPU.

Core Implementations

Step 1: Inside an SM: hiding latency with many warps

This step: why an SM keeps dozens of warps resident, and how register use limits that number.

A CPU core uses large caches and out-of-order execution so one thread rarely waits for memory. A GPU does something else: after a warp issues a load from HBM, the data takes hundreds of clocks to return, so the scheduler does not wait — next clock it issues from another ready warp. With enough resident warps there is always one to issue, and the units stay busy.

Little's law estimates how much latency must be hidden: data in flight = bandwidth × latency. Assume a 600 ns memory latency (an order-of-magnitude figure for the arithmetic; real latency varies by part and access pattern and must be measured with a microbenchmark). To sustain 3.35 TB/s, an H100 needs 3.35e12 × 600e-9 ≈ 2 MB of reads outstanding at all times, about 15 KB per SM across 132 SMs. One warp-wide load of 32 floats is 128 bytes, so each SM needs about 120 such loads in flight. One or two warps cannot do that, which is why GPUs need so many threads.

How many warps fit depends on SM resources, most often registers. An SM has 65,536 32-bit registers, and the compiler decides how many each thread uses. At 128 registers per thread the SM fits 65536 / 128 = 512 threads, 16 warps, 25% of the 64-warp maximum, leaving fewer warps to switch between. Shared memory and threads per block cap residency the same way: compute a limit per resource and take the smallest.

CUDA cores and Tensor Cores are different execution units. A CUDA core instruction makes 32 threads each do one scalar operation. A Tensor Core instruction makes the whole warp cooperate on a small matrix multiply-accumulate; on Ampere one Tensor Core does 256 FP16 FMAs per clock, dozens of times a CUDA core's rate. Whether a matmul uses Tensor Cores depends on the dtype, the shapes and the library (cuBLAS, CUTLASS, ...). Elementwise ops, softmax and normalization cannot use them and run on CUDA cores.

import math

def bytes_in_flight(bandwidth_bytes_per_s, latency_s):
    """Little's law: bytes that must be outstanding to sustain the bandwidth."""
    return bandwidth_bytes_per_s * latency_s

def resident_warps_by_registers(regs_per_thread, regs_per_sm=65536, max_warps=64, warp_size=32):
    """Warps one SM can hold when registers are the only constraint."""
    return min(max_warps, regs_per_sm // (regs_per_thread * warp_size))

inflight = bytes_in_flight(3.35e12, 600e-9)                     # 600 ns is an assumed latency
assert round(inflight / 1e6, 2) == 2.01                          # about 2 MB
assert round(inflight / 132 / 1e3, 1) == 15.2                    # about 15 KB per SM
assert math.ceil(inflight / 132 / 128) == 119                    # about 119 128-byte loads per SM

assert resident_warps_by_registers(32) == 64                     # registers are not the limit
assert resident_warps_by_registers(128) == 16                    # 64 warps shrink to 16
assert resident_warps_by_registers(255) == 8                     # the 255-register maximum

Step 2: Where peak FLOPS comes from

This step: rebuild the spec-sheet TFLOPS from units × operations per clock × clock, and separate FP32 from Tensor Core and dense from sparse.

One fused multiply-add (a * b + c) counts as 2 FLOPs. The FP32 peak is:

SMs × FP32 units per SM × 2 × clock

A100: 108 × 64 × 2 × 1.41 GHz = 19.5 TFLOPS, matching the official value. H100 SXM: 132 × 128 × 2 × 1.98 GHz = 66.9 TFLOPS; NVIDIA lists 67.

For Tensor Cores, use their count and their operations per clock:

Tensor Cores × FLOPs per Tensor Core per clock × clock

An A100 Tensor Core does 256 FP16 FMAs per clock, 512 FLOPs: 432 × 512 × 1.41 GHz = 311.9 TFLOPS, official 312. Hopper doubles per-SM Tensor Core throughput to 1024 FLOPs per Tensor Core per clock. At 1.98 GHz the H100 would give about 1071, more than the official 989; only 1.83 GHz gives 989.4. The official Tensor Core peak is therefore computed at a lower clock than FP32. NVIDIA's product page does not state that clock; 1.83 GHz is inferred from the official figures, and the third-party summary linked below uses it too. Heavy matmuls draw more power and the GPU cannot hold its top clock — a reminder that every spec-sheet peak assumes a particular clock.

Three easy misreadings:

1. Dense or sparse. The H100 page lists 1,979 BF16 Tensor TFLOPS with an asterisk whose footnote says "with sparsity". That assumes 2:4 structured sparsity — two of every four weights are zero and stored in that format, so hardware skips them. Ordinary training and inference matmuls are dense: divide by 2 to get 989. Likewise the A100 page's 624 BF16 is 312 dense.
2. Which precision column. FP32, TF32, BF16 and FP8 each have their own peak: about 67, 495, 989 and 1979 dense TFLOPS on H100. A vector add runs on FP32 CUDA cores, so BF16 Tensor Core's 989 is not its ceiling.
3. Peak is not achieved. Small matrices, shapes that are not the multiples Tensor Cores want, or data that cannot arrive fast enough all fall far short. Measure what fraction you reach at fixed shapes and dtypes.

def peak_tflops(units, flops_per_unit_per_clock, clock_ghz):
    return units * flops_per_unit_per_clock * clock_ghz * 1e9 / 1e12

# FP32: one FMA per unit per clock = 2 FLOPs
assert round(peak_tflops(108 * 64, 2, 1.41), 1) == 19.5           # A100
assert round(peak_tflops(132 * 128, 2, 1.98), 1) == 66.9          # H100 SXM, official 67

# Tensor Cores: 256 FMA/clock each on A100, 512 FMA/clock each on Hopper
assert round(peak_tflops(432, 512, 1.41)) == 312                   # A100 BF16 dense
assert round(peak_tflops(528, 1024, 1.83)) == 989                  # H100 BF16 dense
assert round(peak_tflops(528, 1024, 1.98)) == 1071                 # the FP32 clock overshoots

sparse_on_datasheet = 1979                                          # the starred figure
assert sparse_on_datasheet / 2 == 989.5                             # dense = sparse / 2, about 989

Step 3: The memory hierarchy and the roofline

This step: list GPU storage from nearest to farthest, then use arithmetic intensity to tell compute-bound from bandwidth-bound. (Pro lesson)

Step 4: Architecture generations: what each one added

This step: a table of V100 through B200, and why precision keeps dropping. (Pro lesson)

Step 5: Multi-GPU interconnect: NVLink, PCIe and all-reduce time

This step: read interconnect bandwidth correctly and estimate how long one gradient sync takes. (Pro lesson)

In Real Systems

Check your own card : nvidia-smi shows the model, memory, current clock and power; nvidia-smi -q -d CLOCK shows max clocks; (Pro lesson)

Common Pitfalls

Estimating training time from the starred Tensor Core figure → that is the 2:4 sparse peak; (Pro lesson)

Say It in the Interview

A GPU is many SMs; each SM keeps dozens of warps resident and hides memory latency by switching between them, so heavy register or shared-me… (Pro lesson)

Hands-on Practice

The exercises run and are graded in the browser on the ML Coding page: GPU peak FLOPS : compute peaks from SM count, units per SM, operation… (Pro lesson)