Data engineering for academic research · 15 September 2026
Historical bank rate extraction
Sole research assistant to Professor Rodney Ramcharan. This page is about the measurement — how the pipeline works, how I know it works, and what it still cannot do. The findings it produces are written up separately.
Academic exercise. Not investment advice. Precision has not been measured: the counts below describe how many rows passed the extraction's evidence tests, not a claim that every row is correct.
The question
[1]How do you measure competition in U.S. bank deposit and loan markets before electronic records exist? The rates were advertised in newspapers, so the question becomes an extraction problem — and then a harder one: once you have 18 million candidate rates out of newspaper OCR, how do you prove any of them are right?
| The evidence test is the product | 18,070,399 rate candidates in; 401,597 out, after every gate |
|---|---|
| A rate only counts inside its own advertisement | Blocks split on the printer's separator; institution and product must appear in the same block |
| Nothing is deleted, only tiered | Tier C rows are retained with their rejection reasons, so every discard is auditable |
| The pipeline validates against something it was never told about | It recovered a federal price ceiling it has no knowledge of |
Assumptions
[2]| Input | Value | Basis |
|---|---|---|
| Corpus | 1,303,758 documents | ProQuest TDM Studio OCR, 1950–2002, processed in two halves; 66 documents present in both were skipped on the second pass |
| Rate candidates extracted | 18,070,399 | Every number sitting next to a rate anchor |
| Offers passing every evidence test | 401,597 | Tier A and tier B combined |
| Against the previous run | 2.8x | v14 produced roughly 141,000 usable rows from the same corpus, without loosening the evidence standard |
| Lender location before entity resolution | 131,257 — 32.7% | Direct evidence only: a matched FDIC charter or an address printed in the advertisement |
| Institution name matching | 99.2% recall, zero false matches | Benchmarked against an FDIC lexicon on 3,600 junk strings |
| Publications | 81 titles, 50 producing usable rows | Across 51 states and territories |
| Environment | Runs inside TDM Studio | Corpus text never leaves the secure environment; only aggregate tables are exported, under a 30 MB allowance |
| Resolved institutions | 17,680 entities | Clustered from 33,857 distinct name strings, 1.9x compression |
| Lender location coverage | 401,597 of 401,597 rows | After entity resolution, on a four-rung evidence ladder recorded per entity |
How it is built
- Segment. Each document's hidden OCR layer is split into advertisement blocks on the printer's separator character. A rate only counts if its institution and its product appear inside the same block.
- Anchor. A number becomes a candidate rate only next to an anchor — "interest", "annual yield", "APR", "per annum". A percent sign alone is not enough; newspaper pages are full of them.
- Match the name. Institution names are folded to an OCR skeleton — 1 to l to i, 0 to o, 5 to s, 8 to b — and matched against an FDIC lexicon.
- Bound it. Every rate is checked against the official ceiling for its product, year and maturity class, plus plausibility bounds. Out-of-bounds rows are kept, flagged, and excluded from the usable set.
- Tier it. A passes every hard test. B is usable with up to three soft flags. C is retained with its reasons attached, so nothing is ever silently discarded.
- Aggregate. 18.1 million rows stream through counters in bounded memory, so the whole corpus reduces to tables small enough to carry out of the environment.
- Resolve entities. Name strings are clustered into institutions, and each one records the rung of evidence that placed it, so 33,857 strings become 17,680 lenders and every offer row gains a location.
- Filter before analysing. Three filters are applied before any result is quoted: the location must come from real evidence, the institution name must be distinctive, and the product must be specific rather than a generic savings line.
What survives
[3]The evidence funnel for the v17 run, with entity resolution applied.
- Documents processed
- 1,303,758
- Rate candidates extracted
- 18,070,399
- Offers passing every evidence test
- 401,597
- Previous run, same corpus
- ~141,000 — a 2.8x gain
- Distinct institution name strings
- 33,857
- Resolved institutions after clustering
- 17,680 — 1.9x compression
- Offer rows with a resolved lender location
- 401,597 — 100%, up from 32.7%
- Publications producing usable rows
- 50 of 81
The evidence ladder
[4]Location is not a yes/no field. Every resolved entity records which rung of evidence placed it, so an analysis can filter to the strength it needs rather than trusting a single flag.
| Evidence rung | Entities | What it means |
|---|---|---|
| fdic_consistent | 954 | Matched an FDIC charter, and the match agrees with where the advertisement ran |
| own_ad_address | 2,887 | A street address parsed out of the advertisement itself |
| single_market | 13,421 | The entity appears in only one newspaper market across the corpus — an inference, not direct evidence |
| fdic_unverified | 418 | An FDIC match that could not be corroborated |
| Total | 17,680 | Covering all 401,597 usable offer rows |
How I know the extraction works
[5]The counts above are a measure of throughput. The question that matters is whether any of it is real. The strongest evidence is that the pipeline recovered something nobody put into it.
- Modal advertised passbook rate, 1967–1981
- exactly 5.00%, fifteen years running
- Top-edge mass while the ceiling bound, 1950–1980
- 34.1% mean
- Top-edge mass after deregulation, 1987–2002
- 10.9% mean
- Peak, 1972
- 72.0%
- Daily compounding advertised, before 1966
- under 2%
- Daily compounding advertised, 1971
- 17.4%
- Daily compounding advertised, by 1999
- under 1%
- Mortgage series peak
- 12.75% in 1981 — the correct turning point
- Money-market products first appear
- 1975 — they did not exist in retail form before then
Until the Depository Institutions Deregulation Act phased them out between 1980 and 1986, federal law capped what a bank could pay on deposits. Advertised passbook rates pile up against that ceiling while it binds and scatter once it lifts. Top-edge mass — the share of a year's rates within a quarter point of that year's 95th percentile — is computed without assuming where the ceiling is; its position is found from the distribution. The extraction code contains no reference to Regulation Q, no ceiling schedule and no list of statutory rates. It read newspaper ink. The compounding series is the same story from the other side: a bank facing a binding nominal cap cannot offer more interest, but it can compound more often, and daily compounding was advertised hardest exactly while the cap bound.
What happened
[6]- Corpus
- 1,303,758 documents, 1950–2002
- Candidates extracted
- 18,070,399
- Usable offers
- 401,597
- Offers with direct lender-location evidence
- 131,257 — 32.7%
- Gain over the previous run
- 2.8x, same evidence standard
- Average printings per advertisement
- 1.87 — not yet exploited
Two things surprised me. The first was that the binding constraint on coverage was how the search was phrased, not how the text was read — keeping the interest-rate language but adding period wording such as "per annum" and "annual yield" was worth more than any parser change. The second was that the 2.8x gain over the previous run came from tightening the evidence tests and fixing what happens inside a crowded advertisement block, not from loosening the standard. The full findings, including the results this measurement supports and the ones it does not yet support, are written up separately.
As of 15 September 2026. Source: Own extraction-run statistics, v17, run finished 26 August 2026.
Limitations and next steps
[7]The 401,597 retained offers passed the stated evidence gates; that does not establish that every record is correct. The research is ongoing. These are the validation questions and methodological assumptions to address before drawing conclusions from the data.
What reconciles. Every row carries its tier, its flags and its source block, and tier C rows are kept with their rejection reasons attached — so every number above can be traced to the page that produced it and every discard can be inspected.
-
Validation pending
Extraction accuracy still needs independent measurement
The proportion of correctly extracted offers has not yet been measured against a manually verified sample. The next step is to compare sampled records with the original newspaper pages and report accuracy by evidence tier. Until that validation is complete, the counts on this page describe records passing the automated checks, not a measured number of correct observations.
-
Validation pending
Within-market rate variation needs further investigation
Within-market dispersion is approximately 1.6 percentage points. Conditioning on maturity across 1,745 cells changes that figure by less than 0.1 percentage point. Dispersion also rises from 0.82 points in the 1950s to 1.83 in the 1980s. These patterns do not establish whether the variation reflects genuine pricing differences, extraction noise or other differences between offers. Reviewing the underlying advertisements is needed before interpreting the result.
-
Methodological assumption
Location assignments use different levels of evidence
Before entity resolution, 131,257 of 401,597 offers — 32.7% — had direct lender-location evidence. Entity resolution assigns a location to every retained offer, but the evidence is not equally strong. Of 17,680 institutions, 954 use an FDIC charter consistent with the newspaper market and 2,887 use an address printed in the advertisement. Another 13,421 are assigned to the single newspaper market in which they appear. That last method is an inference, not independent address confirmation. Each entity retains its evidence rung so analyses can distinguish direct evidence from inferred locations.
-
Methodological assumption
Geographic coverage reflects the newspaper collection
Five states account for most geolocated records. This reflects the newspaper collection rather than a nationally representative sample of lenders or offers. Geographic comparisons need to account for that coverage; national conclusions would require additional evidence.
-
Validation pending
Generic institution names require filtering and review
Some extracted names contain only a generic label such as “SAVINGS BANK”, which is insufficient to identify a particular lender reliably. The entity export includes groups with 3,852 and 2,632 offers under that label. A total of 60,233 tier-B rows are flagged for non-distinctive institution names. These flags support filtering before institution-level analysis; ambiguous names and locations still require review.
-
Validation pending
The bank-type comparison has not recovered the expected gap
The historical thrift-versus-commercial-bank comparison produces a median gap of +0.006 percentage points, rather than the roughly quarter-point differential being tested. Bank type is currently inferred from institution names, so classification error is one possible explanation; it has not been established as the cause. A next step is to compare those classifications with charter information where FDIC identifiers are available, then repeat the comparison.
Academic exercise. Not investment advice. Precision has not been measured: the counts below describe how many rows passed the extraction's evidence tests, not a claim that every row is correct.