GPU Infrastructure Guide: Chips, Interconnects & Sizing

For many years, systems engineering focused on CPU IPC (Instructions Per Cycle), memory clock latencies, and multi socket NUMA architecture. In today’s world, the center of gravity has definitely moved. From deployment of transformer based large language models, to computer vision workloads and physical simulation rendering, the graphics processing unit (GPU) is no longer just an auxiliary coprocessor; it has become the main computing core. However, designing the GPU architecture requires a lot more than just selecting the device with the highest TFLOPS on paper. Memory bottlenecking, saturating the PCIe bus, and hypervisor overhead can seriously decrease throughput performance.

The Real Bottleneck: Memory Bandwidth vs. Computing Power

Another typical architectural mistake is that the assessment of GPUs should be based on compute power numbers such as FP16 or FP8 Tensor TFLOPS. In reality, when it comes to practical applications, performance bottlenecks have nothing to do with mathematical execution units and are constrained by memory bandwidth and caches.Take LLMs inference. In the generation part, the model needs to read each active weight from the VRAM to generate one new token. As the arithmetic intensity (the ratio of computing operations to memory accesses) is very low during the generation process, the speed of the execution depends almost entirely on memory bandwidth and speed:

TheoreticalGenerationSpeedMemoryBandwidth(GB/s)ModelSizeinVRAM(GB)

Even if a GPU uses standard GDDR6 with a 256 bit bus, the theoretical memory bandwidth can amount to 360-500 GB/s. While it is enough for batch processing and smaller quantized models, it is nothing compared to enterprise class architectures with HBM3 using multi thousand bit interfaces, reaching 2000-3000 GB/s.Moreover, the memory capacity should take into account not just the weights. Sizing calculations need to consider:

Model Parameters: Generally 2bytesperparameter for FP16/BF16, or 0.5–1byte for recent 4-bit/8-bit compression schemes (AWQ, GPTQ).

KV Cache Growth: Increases linearly with context window size, number of concurrent batches, and model layer count.

Activation Buffers: Temporary allocations during forward propagation

PCIe Saturation & NUMA Interconnect Topology

Accelerators don’t exist in a bubble; they require host CPUs to communicate for pre processing of data, data streams, and result serialization. This is why interconnection between host and accelerator is often a problem area. The default configuration for PCIe 4.0 x16 provides a theoretical unidirectional throughput of 31.5 GB/s (~63 GB/s bidirectional), while PCIe 5.0 doubles that to 63 GB/s unidirectional. Applications with continuous host to accelerator transfers like video analytics processing in real time, multi streaming frame ingestion, or gradients during distributed training would be bottlenecked by an under provisioned PCIe bus and would keep SMs idle during pipeline stalls.

Metric	PCIe 4.0 (x16)	PCIe 5.0 (x16)	NVLink 4.0 (Per GPU)
Unidirectional Throughput	~31.5 GB/s	~63.0 GB/s	Up to 450 GB/s
Typical Application	Single GPU inference, rendering	High volume batch ingest	Multi GPU unified memory training
Bottleneck Risk	Moderate if working on high volume data	Low for computations	Minimal if syncing GPUs

When using a multi socket platform, developers will need to connect GPUs directly to the respective CPU socket via PCIe bridges. Interconnect between sockets via UPI (Ultra Path Interconnect) or Infinity Fabric incurs significant overhead.

Dedicated Hardware vs. Multi Tenant Virtualization

When it comes to implementing any production solution, choosing the right deployment tier becomes as important as choosing the silicon. While cloud computing promoted shared and elastic instances, GPU workloads have some inherent disadvantages when it comes to being run in a virtual environment:

Pass Through PCIe Latency: Virtualized environments have additional hypervisor layers that can induce jitter and impact latency sensitive benchmarks and inference APIs.

Noisy Neighbors: Memory bus contention from neighboring instances running on shared vGPU slicing is likely to induce non determinism.

Thermal Throttling & Power Caps: Aggressive thermal throttling or sharing of power caps could restrict boost clock speeds in densely packed, multi tenant public cloud environments.

For sustained workloads where deterministic behavior and hardware isolation are required, allocating a bare metal or isolated dedicated GPU server environment provides root access to the PCIe lanes, memory bandwidth, and sustained clock speeds without any penalty of virtualization or shared hosts.

Aligning Workloads with Hardware Profiles

For optimal efficiency, it is essential for hardware profiles to align precisely with operational requirements: LLM Inference and Fine Tuning, Key Measure: Amount of memory and memory bandwidth.

Hardware Profile: Emphasis on large VRAM capacities (24GB-48GB+ per card) with FP8/INT4 matrix tensor calculations. The Ada Lovelace and Hopper generations’ tensor cores offer outstanding performance due to their optimized execution pipelines for FP8.

Massive 3D Rendering (OptiX, Cycles, V Ray), Key Measure: Accelerated BVH computations, RT cores and CUDA core density.

Hardware Profile: Standard workstation and server GPUs with shader cores and specialized hardware ray tracing hardware blocks. Raw FP32 processing delivers maximum efficiency.

Computer Vision and Edge Analytics, Key Measure: Hardware video decoders (NVDEC) and low precision INT8 computations.

Hardware Profile: GPUs with multi stream hardware video decoding chipsets capable of offloading H.264/HEVC/AV1 decoding directly from the CPU.

The Cost Effectiveness Equilibrium

The steady state billing of clouds reaches a turning point where the per unit cost quickly overtakes the costs associated with running month long dedicated instances. There is a lot of overhead to continuously running operations at 70%+ capacity utilization in on demand cloud instances, even more so considering the high network egress costs and IOPS limits.Through profiling of arithmetic intensity, VRAM budgeting, and PCIe interconnect analysis, the engineers can construct highly robust compute pipelines.