NVIDIA AI Releases Nemotron-Labs-Sigma, a Continuous Diffusion Language Model with 14 checkpoints at 3B/8B

NVIDIA AI Releases Nemotron-Labs-Sigma, a Continuous Diffusion Language Model with 14 checkpoints at 3B/8B


NVIDIA has released Nemotron-Labs-Sigma, a collection of 14 open checkpoints at 3B and 8B scale. Nemotron-Labs-Sigma is a continuous diffusion language model family that denoises text in a 16-D embedding space. Most diffusion LLMs, such as LLaDA and Dream, work on discrete tokens. Sigma works in a smooth latent space instead. NVIDIA team states that this makes few-step distillation and trajectory steering practical for text.

TL;DR

Size: 3B and 8B parameters, 14 checkpoints. Context is 4,096 tokens (8,192 for Sigma-8B-instruct).

Runs on: 1 NVIDIA GPU in bfloat16 (Ampere, Lovelace, Hopper, Blackwell). About 120 to 140 tokens/s on 1 H100. No CPU support, no quantized variants listed.

Performance: Sigma-8B base averages 71.16 on NVIDIA’s 6-task suite, ahead of NLD-8B (70.22).

Best: 91.96 on GSM8K (Sigma-8B-instruct, 256 DDPM steps).

Worst: 17.00 on AIME25 (same model and sampler).

Bottom line: Best: the 8-step distilled 8B keeps a 70.34 average. Worst: the instruct average (59.14) trails NLD-8B (59.92).

What is Nemotron-Labs-Sigma?

The research team calls Sigma the first large-scale family of continuous diffusion language models. The collection has 4 variant types at each size:

Base: Sigma-3B and Sigma-8B.

Instruct: Sigma-3B-instruct and Sigma-8B-instruct.

Few-step distilled: distill-3B and distill-8B at NFE 2, 4, 8 and 16 (8 checkpoints).

Self-speculative: distill-spec-3B and distill-spec-8B.

How does continuous diffusion for language work?

Each token maps to a point in a learned 16-D embedding space. Training adds Gaussian noise to these embeddings and teaches the model to remove it. The embedding geometry is learned jointly with the denoiser.

Generation runs blockwise. Each denoised block is committed to a KV cache before the next starts. Blocks default to 64 tokens.

The backbone is a Ministral-style decoder-only Transformer with grouped-query attention, SwiGLU and rotary embeddings. Training starts from an autoregressive warm start of about 1T tokens. About 0.3T tokens of diffusion training follow.

At inference, classifier-free guidance and score temperature matter most. The model card page recommends guidance 3 and temperature 0.3 at 32 DDIM steps.

Why does a continuous space matter?

Discrete diffusion jumps between token states, so its trajectories are hard to steer. Sigma follows smooth ODE/SDE paths instead. Per the research paper, these degrade gracefully at low step counts and distill efficiently. Embedding-space steering also tunes the quality-diversity trade-off for pass@k sampling.

How does Sigma perform on benchmarks?

On the research’s 6-task base suite, Sigma-8B with 256 DDPM steps averages 71.16. NLD-8B in diffusion mode scores 70.22, Dream-7B 58.74 and LLaDA-8B 41.78. Qwen3-8B in AR mode scores 68.91. A cheaper 32-step DDIM sampler still gives Sigma-8B a 70.18 average.

Coding is the standout. Sigma-8B scores 65.12 on HumanEval versus 62.80 for NLD-8B. MBPP is weaker, at 65.74 versus 67.00.

After SFT, Sigma-8B-instruct scores 91.96 on GSM8K, 78.40 on MATH-500 and 82.50 on HumanEval. Hard reasoning stays limited, at 23.33 on AIME24 and 17.00 on AIME25. Its 10-task average of 59.14 sits just below NLD-8B at 59.92. All scores are NVIDIA’s own evaluations.

How fast are the distilled and speculative checkpoints?

Sigma-distill-8B-nfe8 decodes each 32-token block in 8 denoiser calls plus 1 readout. Its card reports a 6-task mean of 70.34, with 88.44 on GSM8K and 65.18 on HumanEval. That is close to the 256-step teacher.

NVIDIA’s launch thread frames the gain as 8 tokens per forward on math and 4 on code.

The distill-spec checkpoints add self-speculative decoding. A diffusion head drafts a 32-token block, and the model’s own AR head verifies it. Output exactly matches greedy AR decoding. On 1 GB200 with SGLang, the 8B spec model runs 3.46x faster than AR decoding on average. NLD-8B reaches 3.04x on the same setup. Peak speedups reach 4.20x at 8B and 3.18x at 3B.

How do you deploy Nemotron-Labs-Sigma?

All 14 checkpoints are on Hugging Face under the NVIDIA Nemotron Open Model License. The cards state the models are ready for commercial use. You need a CUDA GPU in bfloat16: Ampere, Lovelace, Hopper or Blackwell. CPU inference is not supported.

Loading needs PyTorch 2.4+, Transformers 4.55+ and trust_remote_code. On 1 H100, the base model generates about 120 to 140 tokens/s. The distilled card lists vLLM, SGLang and Docker Model Runner for serving. NVIDIA hosts no API.

How does Sigma compare with other 8B diffusion and AR models?

Benchmark rows use NVIDIA’s evaluations from the Sigma paper (Table 2 base, Table 5 instruct). Specs come from each model’s own card.

FeatureSigma-8BNLD-8BDream-7BLLaDA-8BQwen3-8BGenerationContinuous diffusion (16-D)Discrete diffusion, AR, self-specDiscrete diffusionDiscrete diffusionAutoregressiveParameters8B8B7B (card lists 8B)8B8.2BContext4,096 (instruct 8,192)Not disclosedNot disclosedNot disclosed32,768 (131,072 YaRN)LicenseNVIDIA Nemotron Open ModelNVIDIA Nemotron Open ModelApache 2.0MITApache 2.0HardwareNVIDIA GPU, bf16, no CPUNVIDIA GPU (tested DGX Spark, GB200)Not disclosedNot disclosedNot disclosedBase 6-task avg71.1670.2258.7441.7868.91 (AR)GSM8K88.5787.2677.1870.9686.73HumanEval65.1262.8054.8832.3264.63MBPP65.7467.0056.8040.8069.40Instruct 10-task avg59.1459.9240.5437.24Not disclosedAvailabilityOpen weights, HFOpen weights, HFOpen weights, HFOpen weights, HFOpen weights, HF

Key Takeaways

Sigma moves language diffusion into a learned 16-D continuous embedding space.

14 open checkpoints: 3B and 8B base, instruct, distilled and speculative.

Sigma-8B base averages 71.16, ahead of NLD-8B at 70.22.

Self-speculative 8B decodes 3.46x faster than AR on average.

Weak spots: 17.00 on AIME25, 4,096-token base context, GPU only.

Check out the Paper, Model Collection and Project Page. You can check the deployment options and specs here through our GPULeaderboard.com chrome extension.

Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

Asif Razzaq is the CEO of Marktechpost AI Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.



Source link

Leave a Reply

Your email address will not be published. Required fields are marked *

Pin It on Pinterest