Architectural Determinants of Local LLM Inference: Evaluating Unified Memory vs. Discrete VRAM Throughput
When deploying Large Language Models (LLMs) on local hardware, the debate often settles into a simplified "Mac vs. PC" dichotomy. However, selecting hardware for generative AI is not about brand preference; it is an exercise in optimizing two specific physical constraints: memory capacity and memory bandwidth. To build or buy an efficient inference engine, one must understand how parameter counts, quantization levels, and architectural memory topologies dictate the ceiling of tokens per second (t/s).
The Physics of Inference: Capacity and Precision
An LLM is essentially a massive collection of weights, expressed as parameters. When we refer to a model like Qwen 3.8 with 27 billion parameters, we are describing the scale of its neural architecture. The hardware requirement for running such a model is determined by how these parameters are stored in memory.
The precision of these parameters—the number of bits used to represent each weight—is critical. At full precision (FP16/16-bit), a 27B parameter model requires approximately 54GB of VRAM. Through quantization, we can compress these weights to 4-bit precision, reducing the footprint to roughly 13.5GB plus overhead for the KV (Key-Value) cache and context window.
The fundamental constraint is that these weights cannot reside on an SSD or standard system RAM during active inference; they must be loaded into high-speed memory accessible by the GPU or Neural Engine. If a model exceeds the available VRAM (on PC) or Unified Memory (on Mac), the performance penalty is catastrophic. When "spilling" occurs—where parts of the model are forced into much slower system RAM—token generation speeds can drop by as much as 95%.
The Mathematical Ceiling: Bandwidth-Bound Inference
The most significant bottleneck in LLM decoding (the generation of tokens one by one) is memory bandwidth. For every single token generated, the hardware must read the entire model from memory into the compute cores. This makes LLM inference a memory-bandwidth-bound task rather than a compute-bound one.
We can approximate the theoretical maximum speed using a simple formula: $$\text{Max Tokens Per Second} \approx \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model Size in GB}}$$
Consider an M5 Max with a memory bandwidth of 614 GB/s. Running a 4-bit quantized version of Qwen 3.8 (~17GB total footprint), the theoretical ceiling is approximately 36 t/s. Real-world benchmarks often land slightly lower, around 31 t/s, due to architectural overhead and context management.
Conversely, an Nvidia RTX 5090 with a bandwidth exceeding 1 TB/s can push much higher speeds for smaller models. However, the limitation remains the capacity: if you attempt to run a massive model like GPT-OSS (120B) on an RTX 4090 (24GB VRAM), the speed will plummet to 1–2 t/s because the model cannot fit within the high-speed buffer.
Architectural Divergence: Unified Memory vs. Discrete VRAM
The "Mac vs. PC" debate is actually a conflict between two different memory topologies:
- Discrete VRAM (PC/Nvidia): The system splits memory into System RAM (high capacity, low bandwidth) and VRAM (low capacity, ultra-high bandwidth). This architecture excels at "Fast and Small" workloads. An RTX 4090 offers incredible throughput for models that fit within its 24GB envelope but fails on larger architectures.
- Unified Memory (Apple Silicon/DGX Spark): The GPU and CPU share a single, massive pool of high-speed memory. This allows for "Big and Slow" workloads. An M5 Ultra or a specialized device like the Nvidia DGX Spark can allocate 128GB or even 512GB to the GPU, enabling the execution of massive models (e.g., 120B+ parameters) that are physically impossible on consumer Nvidia cards.
The Impact of Mixture of Experts (MoE) and Neural Acceleration
Recent shifts in model architecture are altering these hardware requirements. The rise of Mixture of Experts (MoE) architectures, such as DeepSeek V4 Flash (284B total parameters, 13B active) or Llama S (118B total, 8B active), presents a unique opportunity. While the entire model must still reside in memory (requiring high capacity), only a fraction of the weights are "activated" per token. This reduces the effective bandwidth requirement per token, favoring hardware with massive unified memory pools but moderate bandwidth.
Furthermore, we must consider prompt processing (the prefill stage). When feeding large contexts or entire codebases into a model, the speed at which the machine reads input is vital. While Nvidia has historically dominated this space—with an RTX 4090 capable of ~3000 tokens/sec input—Apple’s M5 generation has closed the gap significantly through neural acceleration integrated directly into the GPU cores, increasing prompt processing speeds by 3x to 4x over previous generations.
Hardware Tiering for Local AI Deployment
Based on current benchmarks and architectural trends, hardware selection should follow this hierarchy:
- The Frontier (Ultra-High Capacity): Mac Studio M5 Ultra. With up to 1.2 TB/s bandwidth and massive unified memory, this is the gold standard for running the largest, most intelligent models at usable speeds.
- The Prosumer Workhorse (Balanced Performance): Nvidia DGX Spark or MacBook Pro M5 Max. These devices provide a sweet spot of high capacity and sufficient bandwidth for fine-tuning and large-scale inference.
- The Speed Demon (High Throughput/Small Models): Nvidia RTX 5090 / 4090. Ideal for developers focusing on low-latency applications, image generation, or video synthesis where the model fits within 24GB–32GB of VRAM.
- The Value Entry (Budget Inference): Used RTX 3090. With 24GB of VRAM and high bandwidth, it remains the most cost-effective way to access high-speed local inference for mid-sized models.
Ultimately, your hardware choice must be reverse-engineered from your model requirements. If you prioritize intelligence (parameter count), prioritize capacity via unified memory. If you prioritize latency (tokens per second) and training/fine-tuning capabilities, prioritize bandwidth and discrete VRAM.