Codebase Knowledge Base Series (04): Three Chunking Strategies — AST Precision Loses?

<h2> The Mistake Almost Everyone Makes </h2> <p>You need to split code into chunks to feed a vector model. What's the most "professional" way to do it?</p> <p>Most engineers answer instantly: chunk by AST (abstract syntax tree), one function per chunk. The reasoning is airtight — a function is the most natural semantic unit, AST aligns precisely with semantic boundaries, and it never cuts a function in half. By comparison, splitting by fixed line count (say, every 20 lines) looks crude and dumb: it hacks functions into fragments.</p> <p>Sounds unassailable. But the experimental data slaps that intuition across the face.</p> <p>On the same 266 lines of Python, I ran three chunking strategies. The result: <strong>AST function-level chunking scored the lowest.</strong> The "crude and dumb" fixed-line split scored a perfect 1.000.</p> <p>This isn't random noise. There's a mechanism underneath worth digging into.</p> <h2> The Three Strategies </h2> <p><strong>Dataset:</strong> 266 lines of Python covering 5 modules — auth, database, cache, payment, notification — 27 functions total. Embedding model is <code>bge-large-zh-v1.5</code> throughout; evaluation uses 12 natural-language queries.</p> <p>Three ways to cut:</p> <p><strong>Strategy 1: Fixed-line (20 lines each, overlap=3)</strong></p> <p>Ignore code structure, slice every 20 lines, overlap adjacent chunks by 3 lines to avoid losing boundary info.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Chunk 1: lines 1-20 Chunk 2: lines 18-37 ← overlaps 3 lines with the previous chunk Chunk 3: lines 35-54 ... </code></pre> </div> <p><strong>Strategy 2: File-level (one chunk for the whole file)</strong></p> <p>Brute force. The entire file is a single chunk.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Chunk 1: lines 1-266 ← everything crammed in </code></pre> </div> <p><strong>Strategy 3: AST function-level (one chunk per function)</strong></p> <p>Parse the syntax tree with Python's <code>ast</code> module, split precisely on function definitions.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="n">Chunk</span> <span class="mi">1</span><span class="p">:</span> <span class="k">def</span> <span class="nf">validate_jwt_token</span><span class="p">(...)</span> <span class="n">lines</span> <span class="mi">9</span><span class="o">-</span><span class="mi">18</span> <span class="n">Chunk</span> <span class="mi">2</span><span class="p">:</span> <span class="k">def</span> <span class="nf">hash_password</span><span class="p">(...)</span> <span class="n">lines</span> <span class="mi">20</span><span class="o">-</span><span class="mi">28</span> <span class="n">Chunk</span> <span class="mi">3</span><span class="p">:</span> <span class="k">def</span> <span class="nf">create_payment_intent</span><span class="p">(...)</span> <span class="bp">...</span> <span class="bp">...</span> </code></pre> </div> <p>First, let's see exactly how "crude" fixed-line chunking is. Take <code>validate_jwt_token</code> (lines 9-18, 10 lines total):<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Function 'validate_jwt_token' (lines 9-18, 10 lines) Splits across 2 chunks: Chunk lines 1-20: contains lines 9-18 (10/10 lines of function) Chunk lines 18-37: contains lines 18-18 (1/10 lines of function) Problem: Neither chunk contains the complete function. A query about 'validate_jwt_token' will retrieve a partial implementation. </code></pre> </div> <p>See it? The function's last line (line 18) got split into the next chunk. This is exactly what fixed-line chunking is criticized for — it doesn't know where function boundaries are, and it cuts wherever it wants.</p> <p>By all rights, that fragmentation should hurt retrieval. AST keeps every function whole. It should win, no conte