KwaiKAT Team Releases KAT-Coder-V2.5: An Agentic Coding Model Trained on 100,000+ Verifiable Repository Environments

The KwaiKAT Team at Kuaishou has introduced the KAT-Coder-V2.5. It is a coding model trained to operate inside real, executable repositories rather than emit single-turn code. The served model is available through StreamLake. An open-weight variant, KAT-Coder-V2.5-Dev, was released separately on Hugging Face under Apache-2.0.

AutoBuilder: environments that actually run the intended tests

The research frames a verifiable task as a triplet. It needs a precise task description, an executable repository environment, and a set of validation tests. A patch is correct only if it passes all of them.

Tasks are mined from real pull requests and commits, following the SWE-bench lineage. The merged code change supplies a golden patch and the accompanying test change supplies a test patch. Raw issue text is discarded as a specification. Instead, descriptions are regenerated into three parts: a problem statement grounded in the golden patch, requirements derived from the test patch, and interface constraints inferred from both. A clarity check then drops anything ambiguous, incomplete, underspecified, or internally inconsistent.

AutoBuilder handles the environment side. A build agent analyzes the repository and writes a configuration script that installs dependencies and runs tests from a clean checkout. A verification agent executes that script in an isolated sandbox.

The acceptance rule is the interesting part. Verification does not read exit codes or grep log patterns. It parses structured test-framework output, and accepts an environment only when more than 90% of expected tests are collected and pass/fail outcomes reproduce across runs. Failures are fed back as structured information for iterative repair.

Combining a preconfigured base environment, build-system templates, and a retrievable library of distilled build recipes raised the construction success rate from 16.5% to 57.2%. The result is over 100,000 verifiable environments spanning 12 languages. Git history, commit metadata, and other exploitable traces are stripped so agents cannot read the reference solution out of the repo.

Data Scaling Flywheel: filtering on process

Filtering trajectories by final test success is misleading. Some passing runs rely on hard-coding, mechanism bypassing, or test-oriented shortcuts. Some failing runs contain valuable search, localization, and repair behavior.

KwaiKAT addresses both directions. For near misses, targeted process-level hints indicate what to inspect or verify without revealing the solution. That alone raises the pass rate of previously zero-pass tasks to roughly 20%. Because hinted trajectories contain information unavailable at inference, the verified patch is then fixed and a hint-free trajectory is regenerated from the original task context. Only samples that pass verification, show no hint leakage, and stay consistent with the patch are retained.

For passing runs, rule-based gates remove invalid, unstable, or exploitative trajectories. A scoring stage then rates exploration, localization, pre-edit reasoning, specification fidelity, repository conventions, patch minimality, verification quality, recovery behavior, and honesty.

A third mechanism targets harness overfitting. Tool names, argument conventions, output formats, and prompt templates are randomized while functionality is preserved. Because verification is anchored to test outcomes rather than harness traces, one task can be re-served under many harness configurations. Realistic perturbations are injected too: missing dependencies, transient command failures, truncated outputs, and noisy logs.

Explore the pipeline and the numbers

Infrastructure failures capped rewards before algorithmic limits

During KAT-Coder-V2 training, slow reward curves were initially blamed on the RL algorithm. However, an audit revealed that ~16% of trajectories failed due to sandbox infrastructure issues rather than the model policy, with boundary misalignments sometimes emptying observations for ~40 steps and corrupting rewards.

Three infrastructure fixes followed. First, an early-release image eviction policy lowered disk usage from 95% to 60%, reducing timeout-induced invalid rollouts from 6–7% to under 1%. Second, correcting environment variables during remote sandbox initialization stopped system overrides that flipped rewards on 6–7% of samples, cutting errors below 1%. Third, the Gateway Server bypassed mainstream chat endpoints—which caused 40% token drift at a ~200-turn scale by re-applying apply_chat_template and re-tokenizing—and called /generate directly to ensure rollout token alignment.

Together, these updates reduced the sandbox feedback error rate from roughly 16% to below 2% and cut training collapses by an order of magnitude.

Asymmetric PPO and a three-tier reward

The research team chose PPO with GAE over critic-free trajectory methods because production harnesses split sessions into structurally distinct samples, complicating group baselines.

Using asymmetric actor–critic, the Critic gets privileged training context (rewards, tests, coverage, patches, metadata, future turns), while the Actor sees only rollout state. The Critic and extra context are discarded at inference.

Rewards are three-tiered: Core Task Scores require all fail_to_pass and pass_to_pass tests to pass; Standard Behavior Constraints penalize duplication, bad tool calls, and debug remnants; Failed Trajectory Incentives score file retrieval via F2 and give partial test credit.

Five experts fuse via Multi-Teacher On-Policy Distillation using reverse KL, an off-policy start, and drift-aware truncation from Prune-OPD.

Results

Under a unified Claude Code harness, KAT-Coder-V2.5 leads its panel on PinchBench with 94.9, beating Opus 4.8 at 93.5. It places second on SWE-Bench Pro (65.2 vs 69.2) and the internal KAT Code Bench (53.1 vs 57.3).

However, it lags on Terminal-Bench 2.1, placing last with 60.7 behind GLM-5.1 (61.8) and Opus 4.8 (84.6). On SciCode, it scores 50.3, matching GLM-5.2.

Notably, the open-weight KAT-Coder-V2.5-Dev is a separate 35B-total / 3B-active MoE post-trained on Qwen3.6-35B-A3B using 127K SFT examples, then RL. Evaluated on a separate in-house protocol, its results are not comparable to the main flagship table.

Key Takeaways

KwaiKAT treats agentic coding as an infrastructure problem, not a model-scale problem.

AutoBuilder lifted environment construction success from 16.5% to 57.2%, yielding 100,000+ verifiable environments across 12 languages.

A sandbox audit found ~16% of RL trajectories failed because of the sandbox, not the policy; fixes cut that to below 2%.

KAT-Coder-V2.5 tops PinchBench at 94.9 and ranks second on SWE-Bench Pro at 65.2, behind Opus 4.8.

The open-weight KAT-Coder-V2.5-Dev is a separate 35B-A3B MoE under Apache-2.0, with its own benchmark numbers.

Check out the Paper, the Model Weight on Hugging Face, and the Product Page. All credit for this research goes to the researchers of this project.