Description
How can AI get more useful work from every byte of memory?
In this episode of Inside the Architecture, we explore five ways NVIDIA optimizes memory use across the AI factory. Learn how model weights, KV cache, and memory tiers work together, and why efficiency depends on the full system.
You’ll learn:
- How NVFP4 reduces memory footprints while preserving model accuracy
- How NVLink and expert parallelism distribute model weights across GPUs
- How NVIDIA Dynamo manages and reuses KV cache across inference workers
- How Vera accelerates tool execution and connects host memory with Rubin GPUs
- How NVIDIA CMX enables context reuse across servers to reduce repeated computation
No prior experience with AI memory infrastructure required.
Read the full technical deep dive here: https://nvda.ws/4zuWhz6
00:00 Five Ways to Get More Useful Work From Memory
00:28 Why AI Needs Memory
01:20 NVFP4: Reducing the Memory Footprint
04:48 NVLink: Distributing Model Weights Across GPUs
06:24 NVIDIA Dynamo: Managing and Reusing KV Cache
07:28 Vera: Reducing Tool Waits and Managing Context
09:52 NVIDIA CMX: Reusing Context Across Servers
11:56 Bringing the Full System Together