Let another AI write your teacher data

<p><strong>How I had a large AI, not people, write the "gold data" that trains a specialist</strong></p> <p>The "gold data" that trains a specialist AI is something humans write — I threw that assumption out. <strong>I had a large AI write it, and a small AI copy it out.</strong> Japanese→English, the direction the small model was worst at, jumped <strong>42% → 84%</strong> on straight imitation alone. Zero human translators.</p> <p>The ceiling above that sat not with the teacher but with <strong>the student's comprehension capacity</strong> (the teacher's own translations passed almost across the board). Lump every ceiling together as "the teacher is bad" and you pick the wrong move.</p> <p>And one pitfall. Hand the teacher surrounding context with the best of intentions, and the student learns to invent subjects. <strong>Build the teacher data under the same input conditions as production</strong> — the way kindness backfires only becomes visible once you measure.</p> <p>Training a small AI into "a specialist at one job" takes a large volume of <strong>gold data</strong>. A stack of model answers saying "for this input, this translation is the right answer." For translation, that's a pair — the source text and its model translation.</p> <p>And nearly everyone assumes the same thing — <strong>that those right answers are something humans produce</strong>. You hire professional translators, have them translate sentence by sentence, and use that as the model answers. The quality is solid, but it's slow. It's expensive. And for low-resource languages you can't get hold of a professional in the first place. This "produce the gold data" step looks like the biggest bottleneck in training a specialist model.</p> <p>I threw that assumption out. <strong>I had the gold data produced by another AI, not by people.</strong></p> <h2> Have the large AI write the model answers </h2> <p>The method is simple. What I want to train is a small, lightweight model (4B = 4 billion parameters). What I have on hand is a far larger, far smarter general-purpose model (27B class).</p> <p>So — <strong>have the large one write the right answers and the small one copy them out</strong>. Pour the source text into the large model, take the translations that come out as "model answers," and have the small model imitate them. In machine learning this is called <strong>distillation</strong>. The side producing the right answers is the <strong>teacher model</strong>; the side learning by imitating it is the <strong>student model</strong>. The title of this article — "Let another AI write your teacher data" — means exactly what it says: <strong>have the teacher model write the gold data (= the teacher data), and have the student copy it out</strong>.</p> <p>The effect showed up plainly. The un-fine-tuned small model was hopeless at Japanese→English, with accuracy of about <strong>42%</strong>. It got more than half of them wrong. So I had the large teacher write about 14,000 translations (clean, a single generation at temp0.1, and — as described below — with no context), rolled those into about 5,000 training examples, and had the small model imitate them —</p> <p>Accuracy: <strong>84%</strong>. <strong>+42 points</strong>.</p> <p>Worth adding: this 84% came out with no elaborate selection in between, just imitating the teacher's plain translations as they came. Tricks like "generate N candidates and pick the good one" are <strong>the next lever</strong>, the one you pull after this. Straight imitation on its own gets you this far.</p> <p>(The scoring is a reference-free binary judge. Rather than a human yardstick, an independent AI scores each translation "usable / not usable," and I compared before and after training under identical conditions. These are point estimates from a single run, so the absolute values wobble by a few points.)</p> <p>One training run took a model that was basically an amateur up to a level that draws level with the