Together AI Unveils Polymorphic Inference and Together Compiler

News

Together AI introduces Polymorphic Inference and Together Compiler to optimize open-model AI inference for diverse and evolving workloads.

Together AI has announced the launch of Polymorphic Inference, a new approach to AI inference designed to adapt to the rapidly evolving landscape of open-source models, diverse architectures, and dynamic workloads. The company also introduced the Together Compiler, a key tool for achieving this vision by enabling continuous optimization of inference performance.

What is Polymorphic Inference?

Polymorphic Inference, derived from the Greek term for “many shapes,” represents a shift from the traditional Static Inference paradigm. Unlike Static Inference, which relies on fixed configurations optimized for specific conditions, Polymorphic Inference dynamically adapts to changes in model architectures, traffic patterns, and hardware platforms. Together AI highlighted several industry trends driving this shift, including:

The increasing velocity of open-source model releases, with 80% of OpenRouter traffic directed to models released in the last three months.The growing diversity of model architectures, with all top-ten open-source models using distinct designs.Rapidly evolving traffic shapes influenced by user behavior, product changes, and post-training optimizations.Frequent advancements in hardware, such as the transition from NVIDIA B200 to GB300 GPUs and the emergence of Vera Rubin deployments.

The company contrasts Polymorphic Inference with legacy Static Inference, which relied on closed models, traditional Transformer architectures, and static configurations that often failed to meet the demands of modern production environments.

Introducing Together Compiler

As part of its Polymorphic Inference initiative, Together AI unveiled the Together Compiler, an AI-powered tool designed to optimize inference workloads through continuous compilation. The compiler reframes traditional GPU optimization challenges as agentic search problems, enabling AI agents to iteratively rewrite and improve GPU code for diverse workloads.

According to Together AI, the Together Compiler delivers significant performance improvements. In company-reported tests, it achieved up to 1.7 times the speed of hand-written megakernels and processed over 2,100 tokens per second for Llama 1B models on NVIDIA H100 and B200 GPUs. The compiler also enables 252 times more optimization experiments compared to existing agent-based systems, producing deployable optimizations within 24 hours.

Practical applications of the Together Compiler include faster deployment times and reduced latency. For instance, a voice-agent deployment using the compiler reduced the time to generate the first 64 tokens from 281 milliseconds on NVIDIA B200 GPUs to 77 milliseconds on H100 GPUs, a 3.6-fold improvement. Additionally, Phonic, a conversational AI company, reported a nearly 50% reduction in median time to first speech for its Phonic-1 model.

Implications for AI Inference

Together AI’s Polymorphic Inference and Together Compiler address critical challenges in the AI inference ecosystem, such as adapting to diverse and rapidly changing conditions without manual tuning. By leveraging AI agents to optimize inference performance, the company aims to provide scalable and efficient solutions for open-source AI models deployed in production environments.

With trillions of tokens served daily through compiler-optimized inference, Together AI positions itself as a leader in advancing inference infrastructure for the open-source AI community.