Research library · Methodology

Acquisition, licensing, and the honest-absence rule

The research library is the text substrate for FinObservatory’s grounded analyst: a local, verifiable corpus of what central banks and financial institutions actually published. This page records how each corpus was acquired, the license language that governs it, what the catalog contains, and the rule that governs everything shown: absence is stated, never papered over. Every count on this page is computed from the catalog at render time. Full acquisition detail lives in the repository at data/raw/<corpus>/SOURCE.md per corpus and in docs/data_provenance.md; the catalog builder is scripts/build/build_text_catalogs.py.

What the catalog contains

text_catalog.parquet is a unified document index across the four corpora, 39,467 rows, one per document on local disk. Columns: corpus, doc_type, date, title_or_name, author_or_filer, path (relative to data/raw/), bytes, plus corpus-specific identifiers (url, cik, accession, sic, imf_year, imf_issue) nulled where not applicable. The catalog is metadata only. A second parquet, bis_speeches.parquet, additionally carries the full BIS speech text for local querying; that text is never rendered on any page (see the BIS terms below). Every unique catalog path was verified present on disk at build time.

BIS central bankers’ speeches (20,728 documents)

Acquired as the BIS’s own pre-compiled bulk download (speeches.zip, one csv with url, title, description, date, author, and text) from bis.org/cbspeeches. The first download arrived silently truncated and failed zip validation; it was replaced the same day with a verified re-download, an incident recorded in the source log. Two BIS-side future-date typos were corrected at parse time from the dates encoded in the speeches’ own bis.org URLs. The archive is a BIS editorial selection of English-language texts, not a census. License tier: noncommercial local-use. The recorded language:

“Use of the content is allowed for noncommercial purposes, and in line with the BIS terms and conditions.”

bis.org/cbspeeches/download.htm, recorded in data/raw/bis_speeches/SOURCE.md

“any extract of not more than 400 words of text or two tables or graphs … and in any case not exceeding 10% of the relevant publication.”

bis.org/terms_conditions.htm: a permitted limited extract is

Consequence on these pages: metadata (title, speaker, date) and a link to the speech’s bis.org page, nothing more. The grounded analyst may quote short attributed passages within the 400-word extract rule; the corpus itself is never republished.

Federal Reserve FOMC and Board materials (4,691 documents)

Fetched file-by-file from federalreserve.gov with a declared user agent and a throttle above one second per request, with a fetch manifest recording every URL and a checksum file over the corpus. Coverage: FOMC statements; minutes together with their predecessor records (historical minutes, executive committee minutes, memoranda of discussion, records of policy actions); verbatim meeting transcripts (1976 onward, released with a roughly five-year embargo, latest 2020-12-16); Board speeches; and the Beige Book. Statement coverage was verified complete against the Fed’s own meeting indexes after an initial crawler gap was caught by the anchor pass and backfilled the same day. License tier: public domain. The recorded language:

“Unless otherwise indicated, information on Board’s website is in the public domain and may be copied and distributed without permission.”

federalreserve.gov/disclaimer.htm, recorded in data/raw/fed_text/SOURCE.md

The federalreserve.gov URL mapping. Every fed_text row links to the exact URL it was fetched from, as recorded in the manifest, so no link on /library/fomc is reconstructed or guessed. The basenames nonetheless map to federalreserve.gov paths by stable rules, documented here for reproducibility: modern statements monetaryYYYYMMDDa.htm → /newsevents/pressreleases/<basename>; 1994–2005 statements YYYYMMDDdefault.htm → /fomc/<basename>; minutes fomcminutesYYYYMMDD.htm → /monetarypolicy/<basename>; modern speeches speakerYYYYMMDDa.htm → /newsevents/speech/<basename>; 1996–2005 speeches YYYYMMDD.htm → /boarddocs/speeches/<year>/<basename>; and all corpus PDFs (transcripts FOMCyyyymmddmeeting.pdf, historical minutes, Beige Books, statement and minutes PDFs) → /monetarypolicy/files/<basename>, except early web-era Beige Books at /fomc/beigebook/<year>/…. Checked against the manifest on 2026-07-11: these rules reproduce the recorded URL exactly for 4,117 of the 4,691 files with zero contradictions; the remaining 574 are 1996–2005 legacy variants (bare-date minutes, sequence-numbered speeches, _default-suffixed statements) for which the recorded manifest URL is authoritative. Had any URL been ambiguous the row would carry no link rather than a guessed one.

IMF Article IV staff reports and GFSR (4,202 documents)

PDFs from the IMF eLibrary, report years 2002–2026, in two kinds: 4,147 Article IV consultation staff reports (the Staff Country Reports series, eLibrary journal 002, 2004–2026) and 55 Global Financial Stability Report files (eLibrary book items, 2002–2026). Fetched slowly and politely from elibrary.imf.org with the fetch sized to research need, after imf.org itself proved closed to scripted access. The eLibrary catalog records the report year (plus an issue number for Article IV) but no per-document date, so IMF rows carry a null date; ordering them by date anywhere would be invented precision, so no page does. The catalog includes only PDFs verified on disk at build time: the fetch was still running at the first build cutoff, and the tail joins the catalog on rebuild, a pending count the build script prints rather than hides. License tier: local-use only, copyrighted. The recorded language:

“The IMF grants permission to visit its Sites and to download and copy information, documents, and materials from the Sites for personal, noncommercial usage only, without any right to resell or redistribute or to compile or create derivative works.”

IMF Copyright and Usage (imf.org/external/terms.htm, effective 2020-01-02), recorded in data/raw/imf_reports/SOURCE.md

“please do not copy publications from the IMF’s website for reproduction on a non-IMF website. Instead, please create links to items on IMF Sites.”

Same page, on republication

Consequence: metadata plus, for Article IV reports, a link to the report’s public eLibrary landing page, reconstructed from year and issue as elibrary.imf.org/view/journals/002/<year>/<issue>/002.<year>.issue-<issue>-en.xml with the issue zero-padded to three digits. The pattern was verified against every landing URL the fetcher itself recorded (4,148 of 4,148 Article IV pages match, zero exceptions, checked 2026-07-11). GFSR entries carry no issue number and no recorded landing page, and their recorded URL is a direct PDF endpoint rather than a public page, so per the honest-absence rule they show metadata without a link.

SEC EDGAR filings, US financial firms (9,846 documents)

10-K and 10-Q primary documents (2,592 and 7,254 respectively) for 314 US financial-sector filers, filing dates 2016-01-29 to 2026-05-15, fetched from SEC EDGAR under its published automated-access rules (identified user agent, under the request-rate cap). The corpus is deliberately a slice, not a census: financial SIC codes were enumerated banks-first and a 50 GB size cap ended the fetch partway through the commercial-bank codes, a boundary recorded exactly in the source log so the fetch can resume where it stopped. Amendments (10-K/A, 10-Q/A) are excluded by design, and EDGAR’s browse interface returns no filers for a few holding-company SIC codes, a gap recorded rather than patched. License tier: public domain (US government work, per data/raw/edgar_filings/SOURCE.md). Rows link to the filing’s sec.gov Archives index page, built from cik + accession.

The honest-absence rule

Every gap above is displayed as a gap. Transcripts stop at the embargo line and the page says so. IMF rows have no dates and are therefore absent from date-ordered lists, stated in place. The EDGAR slice boundary, the BIS editorial selection, the pre-1993 minutes-family nuance, and the sparse pre-1999 statement years are all labeled where they surface. Nothing is interpolated, no date is imputed, no missing document is silently skipped: if the library does not hold or may not show a thing, the page says exactly that. The same rule binds the grounded analyst this library will feed: answers cite documents that exist, or state that none does.

Back to the research library, FOMC materials, or central-banker speeches.