Why a Scan Is Not a Document

In the previous piece I argued that a PDF keeps where the glyphs sit and throws away what they meant. A scan is a harder case, because a scan does not even have glyphs. It has pixels. Everything else is reconstructed by software that is guessing, and the guessing is much better than it used to be, which is exactly what makes it dangerous. More than twenty years of setting scientific books taught me what a table actually is. It is not a picture of rows and columns. It is a grid of relationships: this number belongs to this row label and this column header, and it means nothing at all detached from either. Publishers rebuild tables by hand when they come in as images, and they do it because there is no honest shortcut. That is the whole argument of this post. OCR converts pixels into characters. It does not convert a page back into a document. <h2> OCR errors are not random, and that is the problem </h2> If OCR failed noisily, this would be a solved problem. You would see garbage, you would know. It does not fail that way. It fails toward plausibility. A 6 becomes an 8. A 1 becomes a 7. A 5 becomes an S. A comma becomes a full stop, which in a European financial document turns 1.234 into 1,234 or the reverse. The engine is choosing the most likely character given the shape, and the most likely character is almost always a real one. In prose, this is survivable. You read a sentence with one wrong letter and your brain corrects it without stopping. That is why OCR feels so accurate: you are testing it on text where you are the error correction. In a table of figures there is no such redundancy. Every digit is load-bearing. Nothing in the surrounding context tells you that a margin should be 6.2 rather than 8.2. The wrong number is exactly as readable as the right one, carries the same confidence, and looks equally like a fact. <h2> Where scanned tables actually break </h2> Four failure modes, and I have seen all four in real documents. Column drift. A value lands one column to the left or right of where it belongs. Revenue for 2024 is now sitting under 2023. The total still adds up, because the same numbers are present, just attributed to the wrong period. Nothing looks wrong until someone builds a growth rate from it. Tables that continue across pages. In a printed book this is handled by repeating the header and adding a continuation rule. OCR sees two separate blobs on two separate pages, and unless something explicitly stitches them, half the table silently disappears or becomes its own orphan table with no headers at all. Merged and spanning cells. A header that spans three columns, or a row label that covers two rows, is a relationship expressed purely through geometry. Flatten it and the relationship is gone. The value that belonged to all three sub-columns now belongs to one of them, or to none. Rule lines read as characters. A thin vertical rule becomes a 1 or a pipe. A horizontal rule becomes a row of dashes or underscores that the parser treats as content. This one is easy to spot when you look and impossible to spot when you do not. <h2> The confidence score will not save you </h2> Most OCR engines return a confidence value, and it is tempting to gate on it. Below some threshold, flag; above it, trust. It does not work, for the same reason it does not work in retrieval. The confidence is per-character and it measures how sure the engine is about the shape it saw, not whether the resulting document makes sense. A crisp, clean, high-confidence 8 that was actually a 6 scores beautifully. A slightly smudged but correctly read digit scores worse. The score is orthogonal to the thing you care about. I measured a version of this problem in a different context and published the result: retrieval similarity scores could n