Nemotron-3-Super Achieves 504K Token Needle Retrieval with Hybrid Mamba+MoE on 4x3090

Core summary

One sentence to understand this update

The Nemotron-3-Super-120B-A12B model, a hybrid Mamba+MoE architecture, demonstrated perfect needle retrieval up to 504K tokens on four 3090 GPUs, thanks to constant-size recurrent states.

Impact & opportunity

What this could mean

Builders can develop applications requiring extremely long context windows with models like Nemotron-3-Super, enabling advanced reasoning and information extraction on local hardware.