NVIDIA GLM-5.3 Model Deployed Across Two DGX Stations Using vLLM

News

NVIDIA demonstrates serving the 753B-parameter GLM-5.3 model across two DGX Stations using tensor parallelism and vLLM for efficient local inference.

NVIDIA has showcased a method to locally serve the massive GLM-5.3 NVFP4 model, a quantized mixture-of-experts (MoE) model with 753 billion parameters, across two DGX Stations. This approach leverages tensor parallelism and vLLM to split the model’s weights and computation between two GB300 GPUs, overcoming the memory limitations of a single machine.

Challenges of Frontier-Scale Models

The GLM-5.3 model requires approximately 464 GB of GPU memory to store its weights, far exceeding the 252 GB of HBM3e memory available on a single GB300 GPU. Despite sparse activation, which reduces computation by activating only 40 billion parameters per token, all model weights must remain accessible for dynamic token routing to different experts.

Technical Implementation

To address these challenges, NVIDIA connected two DGX Stations via ConnectX-8 RDMA, enabling tensor parallelism (TP=2) to divide the model’s tensors across the GPUs. Station A hosts the OpenAI-compatible API endpoint, while Station B operates as a worker node. Both stations compute every request, with Station A returning the final output.

The deployment uses the vLLM framework, which supports multi-token prediction, FP8 KV-cache storage, and prefix caching to optimize throughput under concurrent loads. Benchmarks on the SPEED-Bench Coding dataset revealed a peak throughput of 607.8 tokens per second at a concurrency level of 16, dropping to 385.0 tokens per second at concurrency 32, with a mean request latency of 453.5 seconds.

Prerequisites and Setup

The setup requires two GB300 DGX Stations running Linux, connected and validated using NVIDIA’s “Connect Two DGX Stations” playbook. Each station must have at least 464 GB of free disk space for the model checkpoint and additional space for the vLLM container image and temporary files. The deployment also necessitates Docker, the NVIDIA Container Toolkit, and access to specific datasets and licenses.

During the configuration, users define environment variables for network and hardware settings, ensuring proper communication between the stations. The vLLM container is then launched with the GLM-5.3 model distributed across the GPUs, enabling efficient local inference.

Implications for AI Infrastructure

This demonstration highlights the growing need for distributed GPU setups to handle frontier-scale models that exceed the capacity of individual machines. By utilizing tensor parallelism and advanced frameworks like vLLM, organizations can deploy large-scale AI models locally without relying on cloud-based solutions.