[AINews] Megakernels are so dead and so back

Part of our Inference Engineering Masterclass pod yesterday involved a spicy discussion about Megakernels:megakernels are deadwhy are megakernels useful? you spend two months writing a kernel to save time on launch overhead and poor inter-kernel overlap. you had PDL but then people said it wasn't perfect, that you could still get some marginal gains due to straggler CTAs and therefore- wait, sorry, I forgot, Rubin fixes that (kernel two needs 10 CTAs and kernel one has seven finished and three straggling, kernel two launches seven of its CTAs). given a long enough timeline, it all evens out. no serious inference provider is using a 67k loc hand-fused forward pass kernel in production, and the teams doing that are doing so out of pure research. dead.@swyx's pod and said some things that i... should not have said. \n\na lot has happened since then, i owe you all an apology.\n\ni'm sorry that i was right about every single thing. \n\na) re megakernels are dead\nwhy are megakernels useful? you spend two months","username":"waterloo_intern","name":"ali","profile_image_url":"https://pbs.substack.com/profile_images/2083657716690759680/zzYf-2oG_normal.jpg","date":"2026-08-03T23:49:24.000Z","photos":[{"img_url":"https://pbs.substack.com/media/HO1VZaKWAAANbcS.jpg","link_url":"https://t.co/L3oxkoWQSG"},{"img_url":"https://pbs.substack.com/media/HO1VcNeWMAAuRvS.jpg","link_url":"https://t.co/L3oxkoWQSG"},{"img_url":"https://pbs.substack.com/media/HO1ZrovXgAAwGgY.jpg","link_url":"https://t.co/L3oxkoWQSG"}],"quoted_tweet":{"full_text":"The Inference Engineering Masterclass: 10x faster models, quantization, speculative decoding, Rubin, & self-optimizing AI https://t.co/uRYIWWDebj\n\n@Baseten @philipkiely and @waterloo_intern explain what actually happens after a model is trained, why turning weights into a fast","username":"latentspacepod","name":"Latent.Space","profile_image_url":"https://pbs.substack.com/profile_images/1888346877428641792/rMxtG84Z_normal.jpg"},"reply_count":53,"retweet_count":67,"like_count":1276,"impression_count":380220,"expanded_url":null,"video_url":null,"video_preview_media_key":null,"belowTheFold":false}" data-component-name="Twitter2ToDOM">The full discussion, for those who care to listen through:Ali: A fused kernel can’t save you. Like here with tensor parallelism, half the matrix is on one GPU and the other half is on another, and if I need the entire matrix in order to do like a nonlinear operation in the next step, which is, for instance, like if I’m doing attention, I need the softmax, or I need to do like exponentiation, I need to have the entire row. So I need to know what the partial result was from GPU 2 and what the partial result was from GPU 1 in order to be able to do the softmax in the next stage. So I have to make them communicate with each other, even if I had a fused kernel, because of the nonlinearities within each one. Also with like mega kernels, like honestly, I’m very bearish. It was a good research direction, and it seems like intuitively, theoretically, it’s nice. You have a lot of launch overhead from launching- Just- one kernel- Yeah, just keep fusing it and moving the data. Just fuse everything together. But the kernel complexity itself is very difficult to write a very optimized mega kernel. It’s very difficult to do so. And not to name any companies, but like even the companies that have worked or people that I’ve spoken to who work at companies that do fused mega kernels, they very often don’t end up running those in production because the TensorRT-LLM and modular kernels that launch are faster because you can optimize each individual component, and you can just have them parallelize with each other. One of the tech leads at NVIDIA launched a Twitter post said like, “We’re pulling the curtain on Rubin, and here’s the specs.” And the third tweet showed, like not to get too technical into it, I and I need to read it much