In June 2021, the National Library of Australia contracted Project Computing Pty Ltd to automatically correct at least 60% of the OCR errors in the digitised text of 10 million newspaper articles from the Trove repository using overProof [paper]. The "first" (that is, those with the lowest article id) 10 million articles without any existing human text corrections were processed.

The majority of these articles were of category "Article" (7.4m), with 1.4m categorised as "Advertising" and 1.2m as "Detailed Lists, Results, Guides". Articles came from 32 different newspaper titles with the most common being The Argus (3.8m), Sydney Morning Herald (2.8m), The Hobart Mercury (1.2m), The Adelaide Advertiser (0.7m) and The Canberra Times (0.6m). There were 6.37 billion words in these 10 million articles, of which 1.65 billion were corrected.

The main reason to use overProof is to improve search recall on the digitised newspaper text: with an estimated OCR word-error rate of around 25%, by removing 60%+ of errors and reducing this error rate to around 10%, a very significant improvement in search recall can be achieved, particularly with longer words and multi-word searches. A secondary benefit is to improve the readability of text.

The corrected text has now replaced the publicly searchable version in the NLA newspapers database. "Before" and "after" versions of the text were stored on the overProof servers, and these versions are used by the comparative search engine (below) to show the improvement in recall when searching the corrected text.

This is a comparison of search recall, not word correction rates. For example, if an uncorrected document contains one instance of the correctly OCR'ed word Leichardt but also two error OCRs of the same word, a search for Leichardt will still find the document, and while the correction of the other two instances may improve relevance-ranking and some phrase search recall, it won't change the recall on a single word search for Leichardt.

For the public, it is the improvement in recall rather than word correction rate that is most important, because articles with no correctly OCR'ed versions of the target words cannot be easily found.

Recall improvement is a function of number of search words and the character-count of the words. A rough indicator of improvement is:

• Single word searches: recall improves by between 15% and 50%
• 2 word phrase searches: recall improves by between 35% and 150%
• 3+ word phrase searches: recall improves by over 60%

Consider the search for two randomly selected names, Reginald Evans and Dorothy Giles, typical of a search that may be conducted by someone researching family history:

Bella Lavender (aka Bella Guerin) was a prominent activist during the late 18th and early 19th century, and the first female graduate of Melbourne University. Newspaper references to her are likely to be of interest to social historians and other researchers. Automatic correction increases the number of articles found when searching for Bella Lavender from 22 to 41, with only two of the newly uncovered 19 articles being false positives (although one other is irrelevant, being adjacent names of horse "Bella" and "Lavender" in a form guide). A search on uncorrected text for Bella Guerin returns just 2 articles, but on the corrected text returns 14 articles of which 10 refer to "this" Bella Guerin and 4 refer to an actress in a play from the 1840s and 1850s.

A search for David Unaipon (aboriginal inventor, author and preacher) returns just 6 articles on uncorrected text. After correction, 18 articles are returned of which 2 are "false positives". Those 10 additional articles more than double the content found in this 10m article subset and if applied to the entire newspaper corpus, represent valuable needles in the Trove haystack.

Name components are some of the most difficult words for overProof to correct because they occur combined with a great many other names: there is little to differentiate statistically and contextually for example between "Reginald Evans" and "Reginald Lyons", both having comparable "language" probability, which is why the correction of Regruñid Lvmis above is difficult: the OCR-error model also finds little to differentiate between Evans and Lyons as more likely representations of the OCR'ed Lvmis.

General words tend to be easier, which is why, for example, the sample searches below for Lygon Street Carlton, kanaka trade and chocolate factory show very few false positives. Words part of quite common contexts, such as those making up Lygon Street Carlton, Sydney Harbour Bridge, and Flinders Street Station are typically corrected extremely reliably by overProof, even when the original OCR is severely corrupted.

The cost of automatic correction of text on a newspaper page is trivial compared to the up-front cost of digitisation and ongoing costs of storage and delivery. The speed of automatic correction is orders of magnitude faster than even a huge crowd-sourced team: at current rates of correction, NLA's existing newspaper corpus will take over 110 years to manually correct. By automatically correcting two-thirds of OCR errors, overProof can reduce the time needed for human corrections and as demonstrated by this comparative search engine, allow current and future generations of searchers to uncover a lot more "gold", and in so doing, greatly enhance the public utility of this magnificent resource.

The following table lists article counts found for sample searches before and after correction of the 10m article corpus by overProof. The meanings of the table columns are:

• Original: total count of articles found before correction by overProof
• Both original and corrected: count of common articles found before and after correction by overProof
• Only original: count of articles found only before correction by overProof. Some of these may have been "false positives" before correction (eg, "Franklin Mills" being incorrectly OCRed as "Franklin Miles" and hence falsely matching a search for "Miles Franklin"), some of these may have been incorrectly "corrected" by overProof to text statistically and contextually more likely, but nevertheless, incorrect in this instance. Some are also related to a name being corrected to a possessive form, as the index does not use stemming (eg, the original Mannix'« which matches Mannix, is accurately corrected to Mannix's, which does not).
• Only corrected: count of articles found only after correction by overProof, that is, the count of articles where the search term is found after correction but not before correction. Some of these may also be "false positives".
• Corrected: total count of articles found after correction by overProof
• Recall improvement: percentage increase in number of articles found after correction by overProof

Notes:

1. This search comparator uses an index built from the 10 million articles processed, not the entire NLA newspaper corpus. It is demonstrating the effect of overProof correction on just the articles it processed.
2. This search comparator uses a default configuration of a SOLR v8.9 search engine with the default SOLR word tokenizer and no word stemming. This is different from NLA's custom SOLR configuration, so although search result counts will be comparable, they will not be identical.
3. Because of the way articles were selected for processing, articles from the 18th and early 19th century predominate: there is comparatively little post-1940 content.