Ring All-Reduce Time

Separate bidirectional from one-way bandwidth and estimate ideal gradient-sync time across GPU counts and links. A free ML coding interview task on NudgeCode: write it in Python and check it with tests in your browser. Medium · GPU, Interconnect, Collective · from GPU Hardware and Architecture.

Implement bytes_sent_per_rank(size_gb, ranks) = 2(R−1)/R × size (0 when R == 1) and ring_allreduce_ms(size_gb, ranks, link_gbs, bidirectional=True) = bytes sent / one-way bandwidth in ms, where a bidirectional spec gives link_gbs / 2 per direction. Example: 8 GPUs, 14 GB, NVLink 900 GB/s ≈ 54 ms; PCIe Gen5 128 GB/s ≈ 383 ms.