Rohan Paul

Kimi K3 just released their technical paper.

one of the most detailed and exhaustive one.

A million-token context window does not create long-horizon agency if the training system cannot preserve a task across thousands of tool calls.

Kimi K3 is built around that constraint.

Reports that the architecture and training recipe improve overall scaling efficiency by roughly 2.5X over Kimi K2.

- Most layers no longer use full softmax attention. Each block runs 3 Kimi Delta Attention (KDA) layers and one global Multi-head Latent Attention (MLA) layer;

- No Position Encoding (NoPE): removes Rotary Position Embedding (RoPE) extension tricks, while bounded KDA decay makes the difficult causal tiles Tensor Core-friendly.

- Their agents are trained not to depend on one fixed scaffold. Tools, prompts, memory, context management, skills, and subagents are varied during RL, while partial rollouts and resumable microVM sandboxes allowed unfinished trajectories continue across training iterations with their environment state intact.

- The model attends across depth, not only across tokens. Attention Residuals let every layer selectively retrieve earlier block representations instead of squeezing the entire network history through one accumulated residual stream.

- The million-token context was trained as an actual dependency problem. The curriculum grows from 8K to 64K, then 256K to 1M, using synthetic multimodal tasks whose evidence is deliberately scattered across the full sequence.

- The 896-expert MoE required a complete stability redesign. Kimi K3 combines a half-width latent expert path, 16 active experts per token, RMSNorm, bounded SiTU-GLU activations, and Quantile Balancing that directly computes the routing bias needed for each expert’s target load.
https://bird.makeup/users/kimi_moonshot/statuses/2081760186235289764