Historical bank rate extraction

How do you measure competition in U.S. bank deposit and loan markets before electronic records exist? The rates were advertised in newspapers, so the question becomes an extraction problem — and then a harder one: once you have 18 million candidate rates out of newspaper OCR, how do you prove any of them are right?

How it is built

• Segment. Each document's hidden OCR layer is split into advertisement blocks on the printer's separator character. A rate only counts if its institution and its product appear inside the same block.
• Anchor. A number becomes a candidate rate only next to an anchor — "interest", "annual yield", "APR", "per annum". A percent sign alone is not enough; newspaper pages are full of them.
• Match the name. Institution names are folded to an OCR skeleton — 1 to l to i, 0 to o, 5 to s, 8 to b — and matched against an FDIC lexicon.
• Bound it. Every rate is checked against the official ceiling for its product, year and maturity class, plus plausibility bounds. Out-of-bounds rows are kept, flagged, and excluded from the usable set.
• Tier it. A passes every hard test. B is usable with up to three soft flags. C is retained with its reasons attached, so nothing is ever silently discarded.
• Aggregate. 18.1 million rows stream through counters in bounded memory, so the whole corpus reduces to tables small enough to carry out of the environment.
• Resolve entities. Name strings are clustered into institutions, and each one records the rung of evidence that placed it, so 33,857 strings become 17,680 lenders and every offer row gains a location.
• Filter before analysing. Three filters are applied before any result is quoted: the location must come from real evidence, the institution name must be distinctive, and the product must be specific rather than a generic savings line.

The evidence funnel for the v17 run, with entity resolution applied.

Documents processed

1,303,758

Rate candidates extracted

18,070,399

Offers passing every evidence test

401,597

Previous run, same corpus

~141,000 — a 2.8x gain

Distinct institution name strings

33,857

Resolved institutions after clustering

17,680 — 1.9x compression

Offer rows with a resolved lender location

401,597 — 100%, up from 32.7%

Publications producing usable rows

50 of 81

Location is not a yes/no field. Every resolved entity records which rung of evidence placed it, so an analysis can filter to the strength it needs rather than trusting a single flag.

The counts above are a measure of throughput. The question that matters is whether any of it is real. The strongest evidence is that the pipeline recovered something nobody put into it.

Modal advertised passbook rate, 1967–1981

exactly 5.00%, fifteen years running

Top-edge mass while the ceiling bound, 1950–1980

34.1% mean

Top-edge mass after deregulation, 1987–2002

10.9% mean

Peak, 1972

72.0%

Daily compounding advertised, before 1966

under 2%

Daily compounding advertised, 1971

17.4%

Daily compounding advertised, by 1999

under 1%

Mortgage series peak

12.75% in 1981 — the correct turning point

Money-market products first appear

1975 — they did not exist in retail form before then

Until the Depository Institutions Deregulation Act phased them out between 1980 and 1986, federal law capped what a bank could pay on deposits. Advertised passbook rates pile up against that ceiling while it binds and scatter once it lifts. Top-edge mass — the share of a year's rates within a quarter point of that year's 95th percentile — is computed without assuming where the ceiling is; its position is found from the distribution. The extraction code contains no reference to Regulation Q, no ceiling schedule and no list of statutory rates. It read newspaper ink. The compounding series is the same story from the other side: a bank facing a binding nominal cap cannot offer more interest, but it can compound more often, and daily compounding was advertised hardest exactly while the cap bound.

Corpus

1,303,758 documents, 1950–2002

Candidates extracted

18,070,399

Usable offers

401,597

Geolocated offers

131,257

Gain over the previous run

2.8x, same evidence standard

Average printings per advertisement

1.87 — not yet exploited

Two things surprised me. The first was that the binding constraint on coverage was how the search was phrased, not how the text was read — keeping the interest-rate language but adding period wording such as "per annum" and "annual yield" was worth more than any parser change. The second was that the 2.8x gain over the previous run came from tightening the evidence tests and fixing what happens inside a crowded advertisement block, not from loosening the standard. The full findings, including the results this measurement supports and the ones it does not yet support, are written up separately.

As of 15 September 2026. Source: Own extraction-run statistics, v17, run finished 26 August 2026.

401,597 is a count of rows that passed the gates. It is not a claim that all of them are correct. These are the things I would raise before anyone leaned on this data, and the first one is the one that matters most.

1. Error

   Precision is unmeasured, and the working estimate is not defensible

   The internal estimate is roughly two thirds correct. That is a guess, not a measurement, and it is the first thing a referee will ask for. Converting it into a number means hand-labelling about 400 rows against the page images — a day of work. Until that exists, every count on this page should be read as rows that passed the evidence tests, not rows known to be right.

2. Error

   Within-market dispersion is 1.6 points and unexplained

   Rates for the same product in the same market in the same year vary far more than genuine price dispersion should. The obvious suspect was a three-month and a five-year certificate sharing one label, but conditioning on maturity across 1,745 cells moves the number by less than a tenth of a point. So it is either real promotional dispersion — in which case it is the phenomenon worth studying — or residual extraction noise in tier B. One pattern points at the second: dispersion rises from 0.82 points in the 1950s to 1.83 in the 1980s, tracking advertising density rather than rate volatility, and crowded pages are harder to parse.

3. Judgement call

   Lender location is now resolved for every row — but not every row at the same strength

   This was the binding constraint on the whole project: before entity resolution, only 131,257 of 401,597 offers — 32.7% — carried a location that could bear weight. Clustering 33,857 name strings into 17,680 institutions raised that to 100%. The honest qualification is that the rungs are not equal. Only 3,841 entities rest on direct evidence: 954 on an FDIC charter that agrees with where the advertisement ran, and 2,887 on a street address printed in the advertisement itself. The other 13,421 are placed because the entity appears in only one newspaper market across the whole corpus, which is a reasonable inference and not the same thing as a record. The rung is stored on every entity, so any analysis can choose its own standard rather than inherit mine.

4. Judgement call

   The geography is a newspaper footprint, not a national sample

   Five states supply the large majority of geolocated rows, because the underlying newspaper footprint does. Within-market and cross-market comparisons are defensible. National claims are not, and any geographic result has to carry that qualification rather than bury it in a footnote.

5. Judgement call

   Generic name strings still resolve into institutions that are not real lenders

   Entity resolution compresses 33,857 strings to 17,680, which is the right direction, but it cannot invent information that OCR destroyed. "SAVINGS BANK" survives as an entity carrying 3,852 offers, placed in Garden City, New York; another "SAVINGS BANK" carries 2,632 offers and lands in a city rendered "Patehogut" with a state code that does not match it. Neither is one bank. 60,233 tier-B rows already carry a flag saying the distinctive part of the name was lost and only a generic tail survived, and those have to be filtered out before any institution-level analysis — which is exactly why that filter is one of the three applied before a result is quoted.

6. Judgement call

   The test that failed, reported alongside the ones that worked

   Regulation Q let savings institutions advertise roughly a quarter point above commercial banks. Recovering that gap would have been a second clean validation, as strong as the ceiling itself. It is not there: the median thrift-minus-commercial gap is +0.006 points during exactly the years the differential was law. The likely cause is the classifier rather than the corpus — the thrift flag is inferred from the words in an institution's name, not from a charter registry, so a savings and loan advertising as a bank is classed commercial. The fix is specific: join on the FDIC certificate already carried for a fifth of offers and use the charter class instead. The same pipeline that recovered a federal price ceiling from newspaper ink cannot yet reliably tell a thrift from a bank, and saying so is the point.