Deploying Kimi K3 on Amazon SageMaker HyperPod and Amazon EKS

<p>Open weight models have become powerful enough to handle complex tasks such as multi-step agentic workflows, advanced reasoning, and long-horizon coding. However, as these models grow in capability, they also grow in size and hosting multi-trillion parameter architectures requires purpose-built infrastructure, high-end GPU compute, and optimized serving frameworks. On July 27, 2026, Moonshot AI released Kimi K3, a 2.8 trillion parameter Mixture of Experts (MoE) model that represents the <a href="https://platform.kimi.ai/docs/guide/kimi-k3-quickstart" target="_blank" rel="noopener noreferrer">first open-weight system</a> to reach the 3 trillion parameter class. Kimi K3 delivers <a href="https://artificialanalysis.ai/models/kimi-k2" target="_blank" rel="noopener noreferrer">frontier-level intelligence</a> while making its weights publicly available, so that organizations can self-host one of the most capable models in existence on their own infrastructure.</p> <p>This post walks through deploying Kimi K3 on AWS using two approaches: Amazon SageMaker HyperPod, and Amazon Elastic Kubernetes Service (Amazon EKS) cluster.</p> <h1><strong>About Kimi K3</strong></h1> <p>Kimi K3 is built on a differentiated architecture featuring Kimi Delta Attention (KDA), Gated Multi Head Latent Attention (MLA), and a Stable LatentMoE framework. The model distributes its 2.8 trillion parameters across 896 specialist experts, activating only 16 per token. This means approximately 104 billion parameters are active during any single forward pass, yielding a <a href="https://platform.kimi.ai/docs/guide/kimi-k3-quickstart" target="_blank" rel="noopener noreferrer">2.5x improvement</a> in scaling efficiency over its predecessor, Kimi K2.</p> <table class="styled-table" border="1px" cellpadding="10px"> <tbody> <tr> <td style="padding: 10px;border: 1px solid #dddddd"><strong>Attribute</strong></td> <td style="padding: 10px;border: 1px solid #dddddd"><strong>Value</strong></td> </tr> <tr> <td style="padding: 10px;border: 1px solid #dddddd">Total Parameters</td> <td style="padding: 10px;border: 1px solid #dddddd">2.8 Trillion</td> </tr> <tr> <td style="padding: 10px;border: 1px solid #dddddd">Active Parameters per Token</td> <td style="padding: 10px;border: 1px solid #dddddd">104 Billion</td> </tr> <tr> <td style="padding: 10px;border: 1px solid #dddddd">Architecture</td> <td style="padding: 10px;border: 1px solid #dddddd">Mixture of Experts (MoE)</td> </tr> <tr> <td style="padding: 10px;border: 1px solid #dddddd">Expert Count</td> <td style="padding: 10px;border: 1px solid #dddddd">896 (16 activated per token)</td> </tr> <tr> <td style="padding: 10px;border: 1px solid #dddddd">Context Window</td> <td style="padding: 10px;border: 1px solid #dddddd">1 Million Tokens</td> </tr> <tr> <td style="padding: 10px;border: 1px solid #dddddd">Modality</td> <td style="padding: 10px;border: 1px solid #dddddd">Native Multimodal (Text + Vision)</td> </tr> <tr> <td style="padding: 10px;border: 1px solid #dddddd">Release Date</td> <td style="padding: 10px;border: 1px solid #dddddd">July 27, 2026</td> </tr> </tbody> </table> <p>Kimi K3 excels at long-horizon coding tasks, agentic workflows, and complex reasoning. It supports native tool calling, structured output, and an always-on thinking mode for multi-step problem solving.</p> <h1><strong>Model availability and format</strong></h1> <p>The open weights for Kimi K3 are available on Hugging Face under the model identifier moonshotai/Kimi-K3. The weights are distributed in <strong>MXFP4</strong> (Microscaling Floating Point 4-bit) format, which provides an effective balance between model quality and memory efficiency for large-scale inference deployments.</p> <p>Given the model’s architecture and size, serving Kimi K3 requires a vLLM day-0 inference container for Kimi K3. At the time of writing, vllm co