The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten

We first covered Baseten last year when DeepSeek mania was at peak hype. Now they have raised a monster $13B round and become one of the new cohort of AI Infra decacorns that are (with Nvidia, Intel, and the semis complex) chief beneficiaries of the Inference Inflection. We return to Baseten at the peak of the 2026 edition of Open Weights debate. Ali has published a viral breakdown of Kimi K3:And since you last saw him, Philip has spoken at AI Engineer and written the definitive book on Inference Engineering spotted all over SF:Three years ago, inference engineering barely existed as a category.Today, it is one of the most critical disciplines in AI. Inference engineering inherently tackles a different question than standard model training: “How do you turn those weights from training into a product that is fast, reliable, and affordable at scale?” Focusing on these creates an entirely new optimization problem.baseten.com/inference-engi… ","username":"philipkiely","name":"Philip Kiely","profile_image_url":"https://pbs.substack.com/profile_images/1644827140641153024/ExLuda2F_normal.jpg","date":"2026-02-23T18:03:01.000Z","photos":[{"img_url":"https://substackcdn.com/image/fetch/$s_!1BR1!,w_1028,c_limit,f_auto,q_auto:best,fl_progressive:steep/l_play_button_usfui2,w_88,e_colorize:0/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F__ss-rehost__tw-video-preview-13_2025989166333616128.jpg","link_url":"https://t.co/QTNdMrypqR"}],"quoted_tweet":{},"reply_count":190,"retweet_count":230,"like_count":2367,"impression_count":1396109,"expanded_url":null,"video_url":"https://video.twimg.com/amplify_video/2025989166333616128/vid/avc1/1280x720/fBFnlcAf_0wCVPNv.mp4","video_preview_media_key":"13_2025989166333616128","belowTheFold":false}" data-component-name="Twitter2ToDOM">In one recent GLM-5.2 experiment, quantizing more of the model actually preserved its benchmark quality while increasing throughput by 20%, because the errors introduced in different layers could cancel each other out.Inference is no longer just the final step after training. It is becoming its own engineering discipline, with its own research problems, infrastructure, and increasingly specialized roles.In this episode, Baseten’s Philip Kiely and Ali Taha join swyx and Vibhu to explain what actually happens after a new open model is released and what it takes to turn “we generated a token” into a fast, reliable, production-ready API.We go deep on cache-aware routing, disaggregated prefill and decode, quantization, speculative decoding, KV-cache movement, model parallelism, GPU kernels, and the race to make frontier models up to 10× faster. Philip and Ali explain why inference optimizations can still produce gains of 20%, 100%, or even 200%; how quantization errors can cancel one another out; why identical weights can behave differently across clusters; and how Baseten grafted a Kimi vision encoder onto GLM-5.2 without changing the underlying language model.The conversation then expands beyond LLMs into NVIDIA Dynamo, mega kernels, Rubin, AI-specific chips, local inference, video generation, diffusion versus autoregressive models, and the enormous compute barrier to generating coherent long-form video. Finally, we explore the convergence of training and inference, continual learning through persistent KV cache, and the emerging loop where models help optimize the infrastructure that runs them.We discuss:What happens when a 200,000-token request enters an inference systemCache-aware routing and reusing previously computed KV cacheWhy prefill and decode are increasingly handled by different GPUsWhen dedicated deployments become cheaper and more reliable than shared APIsHow speculative decoding uses a smaller model to accelerate a larger oneTool calling, structured outputs, and what LLMs actually doWhat it takes to support a new open model on day zeroGrafting Kimi’s vision encoder onto GLM-5.2Retrofitting ineff