A small model’s problem was not accuracy, it was confidence. Untrained, it invented a value for 38% of the fields that should have been blank. After fine-tuning it did so 0.03% of the time, and found more real values too.
The job
Suppliers send product data as free text, and catalogs need named fields. The question was whether a model under one billion parameters, small enough to run on one in-house machine, could do this narrow job nearly as well as an expensive general-purpose model, at a fraction of the cost and without the data leaving the network. The honest test: does it leave a field blank when the document does not say?
In: 435952-B21 HP Xeon E5335 2.0GHz DL360 G5. Intel Xeon E5335 DL360 G5
(2.0GHz/4-core/8MB-2x4MB/80W) Full Processor Option Kit. Part Number(s)
Part# 435952-B21 Spare 437948-001...
Out: generation: G5 part_number: 435952-B21
processor_type: Xeon E5335 clock_speed: 2.0GHz
cache: 8MB-2x4MB bus_speed: (blank) interface: (blank)The real task has 18 named fields plus a catch-all. Reading the PDF itself and normalizing formats like “2.0GHz” versus “2GHz” are out of scope on purpose: one is a solved problem and the other suits simple rules, not a model.
The result
The team fine-tuned Qwen3 (0.6 billion parameters) on 4,390 documents from a public academic benchmark that ships with verified answers. Same test, same scoring, every model:
Recall: of the real values, how many it found. Higher is better.
F1 balances precision and recall. Higher is better.
Invented answers: when a field truly has no answer, how often the model makes one up. Lower is better.
“Invented answers” is the fabrication rate, the number the project is built around. Switch the metric above to see that only the fine-tuned model is strong on all three.
Field accuracy, with the do-nothing floor
Floor: a model that answers “blank” to everything scores 59.3%
- Only training moved every number the right way. The specialist tool was safe but found only about a third of the real values. A well-written prompt cut the untrained model’s inventions to 12.2% but lowered its recall.
- Plain accuracy would have hidden all of this. The untrained model scores below the do-nothing floor because its guesses cost more than silence would have. Accuracy is a different measure from recall and F1, and the two should not be compared.
What generalized, and what did not
The team re-tested the same documents with the field names changed, the layout restructured, and no field list at all:
Recall on the same documents, presented four ways.
F1 on the same documents, presented four ways.
Renaming or restructuring cost one to two points, which suggests a new customer’s field names may be a configuration change, not a retrain. With no field list, the fine-tuned model still found 71.8% of the values on its own judgment, where the untrained model produced nothing usable and a good prompt only reached 7%.
Where it fails. On a product category it had never seen, the fine-tuned model’s score dropped and it lost to the older, weaker specialist tool, 67.7% to 72.8%. This is the open problem.
Next: our own documents
Public data proved the idea, not that it works on our documents. The team gathered 1,500 products across six categories from a manufacturer catalog, each with a structured spec sheet (46,792 label-value pairs in all) and a supplier-style PDF.
To draft training answers, they compared candidate “teacher” models, three on five documents and then fourteen on eight, scored automatically and never by another model acting as judge. They chose gpt-4.1-mini: 89.8% quality against 90.0% for the best, at about one-eighth the cost. The top models sit within a point or two of each other, which is inside the noise of a test this small, so it is a cost win, not a quality win.
Even the best teacher scores only about 90%, and a model trained on wrong answers learns that guessing is rewarded. So the team cross-checks the teacher against the real catalog:
| Situation | What they do |
|---|---|
| Catalog and document agree | Use it, verified |
| Catalog has it, the document does not mention it | Label it blank: a free, correct example of “don’t guess” |
| Document mentions it, the catalog does not | Use the teacher’s answer |
Status and limits
Building training data from our own documents is in progress. Training on it, and measuring cost per document and the hand-off rate to a larger model, have not started, so the cross-check is the approach being built and not a validated result. Every accuracy number here is measured against a stand-in for truth, a public benchmark or a manufacturer’s own catalog, never against a client’s verified data.