Description
When a model is too large for one GPU, you can still run it locally. In this tutorial, we serve GLM-5.3, a 753-billion-parameter mixture-of-experts model, across two NVIDIA DGX Stations with vLLM and tensor parallelism, keeping every weight in GPU memory with no CPU offload.
The GLM-5.3 NVFP4 checkpoint is about 464 GB, more than the 252 GB of memory on a single NVIDIA GB300 GPU. By connecting two DGX Stations with ConnectX-8 and splitting the model across both GPUs, the two systems serve one model through a single OpenAI-compatible API.
You'll see how to prepare both DGX Stations and collect the network values for the RDMA connection, download the pinned checkpoint and copy it to the second Station, and launch the same vLLM script on both machines. We then verify the service: a health check on the API, an RDMA check that confirms the GPUs communicate directly over the network instead of falling back to TCP sockets, and nvidia-smi showing one vLLM worker on each GPU. Next, we send a chat request through the OpenAI-compatible endpoint and check the answer.
Finally, we use NVIDIA AIPerf with SPEED-Bench Coding prompts to measure how per-user throughput and request latency change as concurrency increases. The video runs a quick test at concurrency 1 and 8, then shows the reference results at 1, 8, 16, and 32 concurrent requests. The companion post on the NVIDIA Technical Blog covers the full benchmark and every command used in this walkthrough.
Running a frontier-scale model on your own hardware is useful for working with sensitive data, prototyping agents locally, and iterating on prompts without per-token costs. To get started, connect your two DGX Stations with the Connect Two DGX Stations playbook, then follow the step-by-step commands in the companion blog.
➡️ Learn more: Connect Two DGX Stations playbook: https://build.nvidia.com/station/connect-two-stations
📝 Tech blog: https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm
📥 Download: GLM-5.3 NVFP4 on Hugging Face: https://huggingface.co/nvidia/GLM-5.3-NVFP4
Chapters:
00:00 Introduction
00:16 Why one GPU isn't enough
00:55 Step 1: Prepare the two DGX Stations
01:40 Step 2: Download and copy the model
02:21 Step 3: Launch the distributed vLLM service
03:45 Step 4: Verify serving and send a request
05:01 Step 5: Benchmark concurrency with NVIDIA AIPerf
06:20 Recap