Reading ancient handwritten Sanskrit manuscripts

The problem

These manuscripts exist as scans, but the text inside them is effectively locked. The handwriting is irregular, the script is Devanagari, with joined consonant clusters and vowel marks above and below the line, and the pages are laid out unusually: a main text block with notes in the margins, and some scans as elongated as 8512 × 715 pixels. Off-the-shelf OCR is built for printed text on evenly spaced lines, so it fails here.

The problem splits into four. A page cannot go straight into a text recognizer, so the lines have to be found first. Handwriting varies from page to page, so one model’s output is unreliable. Even a good recognizer turns valid Sanskrit words into non-words with a one-character slip. And a reader needs to know which words to doubt.

What I did

• Built the line-segmentation stage on a fine-tuned segmentation network (ocroseg), trained on masks annotated in Labelbox in two classes, central text block and margin notes. I turned its masks into deskewed per-line crops and rejoined fragments of one line that the network found separately.
• Trained the recognizer: three cross-fold line-recognition models, trained on line images degraded with noise to look like real aged pages, combined at inference by confidence-weighted voting.
• Added a SymSpell correction layer over a dictionary and bigram table built from a Sanskrit corpus, with a synthetic-typo generator to test it.
• Turned recognition confidence into a review workflow: per-word scores surface uncertain words, and reader corrections are saved and override the OCR output on later loads.
• Designed the service architecture so that slow page OCR never blocks a web request.

Key decisions

1. Segment small, crop big

   Segmentation quality dropped on small pages that had been shrunk hard, while the largest scans risked running out of memory. I segment on a downscale capped in area (at most 1500 × 1500 pixels), then multiply the box coordinates by the scale factor and cut each line from the full-resolution binarised original. The network sees a manageable image and the recognizer still sees full detail. Uniformly shrinking every page, and training on multi-scale copies of the dataset, were the alternatives I set aside.

2. Rotated boxes, padded

   Each predicted blob becomes a minimum-area rotated rectangle, so a tilted line is cut out and straightened rather than boxed with neighbouring lines. The box is padded in height so marks above and below the line are not clipped.

3. A trained segmenter

   Skewed, touching lines and margin notes are hard for straight-projection segmentation, so the line finder is a neural network trained on these pages’ own annotated masks. I did not record a comparison against a stock or classical segmenter.

4. Three voted models

   One handwriting model varies from run to run. Three cross-fold models vote on every line, weighted by per-character confidence, at three times the inference cost of one.

5. Denoising by comparison

   On adaptively binarised (Sauvola) scans I compared three ways to remove specks: connected-component filtering, generic image denoising and morphology. I chose erode, dilate, erode.

6. Suggest, don’t overwrite

   A dictionary corrector cannot see sentence context and can turn a rare but valid word into a common one. So the reader chooses among suggestions, and the confidence score tells them where to look, instead of the system silently rewriting the text.

7. Async from the start

   Page OCR takes far longer than a web request should. The API returns HTTP 202 with a task id and the client polls. Workers acknowledge late, so a crash does not lose a job, and take one task at a time, so no worker hoards the queue.

Making the errors reviewable

At about 81% character accuracy, errors are frequent, so the product could not pretend to be a finished transcription. The recognizer produces character probabilities, not word scores, so I average them between spaces to get a confidence per word and per line. The interface uses that to surface uncertain words for a human to fix.

Spelling suggestions come from a SymSpell service over a Sanskrit dictionary, up to edit distance 5, five per word, limited to Devanagari. A reader’s correction is stored as old word to new word and applied at confidence 1 on later loads, so the same fix is not repeated on every page. Correcting a word back to its original removes the mapping rather than stacking a loop.

Explored, not shipped

Inserted corrections written in manuscript margins are marked with a caret symbol. I experimented with detecting those marks, comparing encoder-decoder segmentation models and loss functions on synthetic training data. It stayed in notebooks, with no metrics recorded and no place in the pipeline, so the shipped system does not handle caret insertions.

Limits

• The ~81% figure is each model’s best score on its own validation fold, and it is a single model’s score. I did not measure the voted ensemble, line segmentation (no IoU or detection rate) or accuracy on a whole page, so I cannot say which stage contributes most of the error.
• The pipeline accepts page images only, and it was built and checked on 11 manuscript templates. I make no claim for other manuscripts.