FinWeave analyst: methodology

Built 2026-07-10. The FinWeave analyst is a citation-first, grounded question-answering layer over the FinWeave data estate. It is wedge 2 of the platform (ROADMAP section 3): the StatGPT architecture (natural language to structured query to exact published figure, citations always) applied public-facing to the whole estate.

The core architecture rule

The model never generates numbers. Every numeric claim in an answer comes from a tool result obtained during that conversation. The language model (GLM-5.1) is used only to (1) decide which tool answers the question, (2) fill in the tool's parameters, and (3) write prose around the figures the tools return. It has no numeric authority. If no tool can supply a number, the analyst says so rather than inventing one.

This is the same discipline the owner already runs in production (the BDPolicyLab advisor, the TradeWeave monitors, the argus 22-tool analyst) and the same pattern IMF StatGPT uses: the LLM generates the structured query, the data system returns the figure.

The loop

A standard agentic tool-use loop (ported from melib/ai/brain.py), server-side, in src/app/api/analyst/route.ts:

1. The question (plus any prior turns) is sent to the model with the fixed tool schemas.
2. If the model emits tool_use blocks, the server runs each tool, validates and caps the result, and feeds the results back as tool_result blocks.
3. Repeat, up to 8 rounds, until the model produces a final text answer.
4. The answer is returned with a de-duplicated citations block assembled from the sources arrays of every tool result used.

The tools

A FIXED set of parameterized, read-only tools (src/lib/analyst/tools.ts), each a thin wrapper over the same query layer that renders the FinWeave pages. There is no free-SQL tool. Every tool query runs through the q()/q1() DuckDB layer, and the wrappers written for the analyst pass through an allowlist guard that rejects any table not on a fixed list. Arguments are validated and row counts are capped.

Each tool returns its data plus a sources array of { dataset, vintage, license } descriptors drawn from src/lib/analyst/sources.ts, which consolidates the provenance already documented in docs/data_provenance.md and printed on the FinWeave pages.

What it will not do

• No numbers from memory. Only tool-sourced figures.
• No investment, trading, or deposit advice. The FinWeave Composite is a transparent peer-percentile screen built from public FDIC call reports, NOT a CAMELS rating and NOT advice.
• No forecasts or predictions. It reports what the data records, including the factual "sits above the historical crisis-start median" comparison, but it does not predict crises, prices, or ratings.
• No individual Bangladesh bank named (ROADMAP constraint 6): only system-level and bank-category aggregates exist.
• No out-of-estate answers. It does not browse and knows nothing the tools cannot return; off-topic questions are declined.

Model and provider disclosure

• Model: GLM-5.1.
• Provider: z.ai, via its Anthropic-compatible API (https://api.z.ai/api/anthropic). The server reads ZAI_API_KEY from its environment at request time; if the key is absent the endpoint returns a clean 503 and the UI shows an honest offline state.

Privacy note

Your question and the conversation turns you send are transmitted to z.ai (the model provider) to generate the response. The tool results (drawn from the FinWeave datasets) are also sent to the model so it can write the answer. Conversations are kept in your browser only; the server does not persist them. Do not send information you would not want processed by a third-party model provider.

License posture

The crisis-atlas layers (GMD, JST, and the combined crisis data) are non-commercial research data; the FDIC, FRED, BIS (credit, cross-border, policy rates), IMF (GDD/WEO/FSI/COFER), OFR, ECB, EBA, World Bank (IDS, GFDD, Findex), Penn World Table and BoC-BoE CRAG layers are open / commercial-safe; the systemic series is internal-use (Yahoo Finance terms); the Cruces-Trebesch restructuring-haircut data is research-use-with-citation, not commercial-safe; the Shiller CAPE carries no explicit open-data license (displayed with citation to Shiller); the Metrick-Schmelzing interventions data is Creative-Commons attribution. Citations name the exact source, vintage, and license label per answer. The separate BoE-copyright Schmelzing long-run real-rate series and the Krippner and EWN datasets are held unwired (no tool), so no non-commercial or restricted-license layer is ever surfaced through the analyst.

Evaluation (published, honest)

The analyst is measured by scripts/analyst_eval.py: 69 questions asked against the live /api/analyst endpoint, each scored against ground truth that the harness recomputes from the parquet estate at runtime (never hardcoded). The harness re-runs, in DuckDB, the same query logic the tools use, then checks that the model's answer carries the tool-sourced figure under displayed-rounding tolerance (a displayed number matches the truth when the truth rounds to it at its own decimal precision), that grounded answers carry a citations block, that out-of-scope questions are refused with a reason, and that fabrication attacks are answered as absence.

The eval was widened from 20 to 30 on 2026-07-10 to cover the six data surfaces added after the first pass (cross-border bank funding, COFER reserve composition, sovereign defaults and restructuring haircuts, crisis-episode interventions, central-bank policy rates, Shiller CAPE). The additions are: 6 new-surface factual lookups, 2 cross-module analyticals (scored on both numbers AND a stated measurement distinction), and 2 fresh fabrication attacks on the new surfaces.

It was extended again from 30 to 38 on 2026-07-10 (later the same day) to cover the five sweep-3 tools (external_accounts, crc_rating, us_debt_detail, gsib_buckets, eu_supervision): 5 factual lookups, one per new tool; 1 cross-module analytical (Bangladesh remittance inflows vs the BIS cross-border bank claims on it, scored on both numbers AND the stated measurement distinction); 1 out-of-scope trap (an EUR/USD forecast plus savings advice, which must be refused with a reason); and 1 fabrication attack (North Korea's IMF BOP current account: PRK is genuinely absent from every external-accounts series, so external_accounts hits its hard-error branch and the honest answer states the absence without inventing a figure).

It was extended a third time, from 38 to 44, on 2026-07-10 to cover the four credit-cycle tools (stress_test, deposit_structure, credit_cycle, lending_survey): 4 factual lookups, one per new tool (JP Morgan Chase's DFAST 2026 severely-adverse CET1 path, actual 14.6% to a projected minimum of 12.6%; the most concentrated state deposit market in 2025, Delaware at HHI 4836.4; the Z.1 latest total credit $115.56T and household debt $21.07T; the euro-area enterprise credit standards +10.09 and demand -1.94 in 2026-Q2); 1 cross-module analytical (US SLOOS C&I standards +8.1 vs euro-area BLS enterprise standards +10.09 in the same quarter, scored on both numbers AND the stated caveat that the two surveys' net percentages are not directly comparable measures); and 1 fabrication attack (DFAST results for Bank of Bird-in-Hand, a real FDIC-insured community bank that has never been a stress-test participant, verified absent from dfast_results at runtime; stress_test hits its honest-absence branch and the answer must state the absence without inventing a ratio).

It was extended a fourth time, from 44 to 52, on 2026-07-11 to cover the six liquidity/debt tools (crisis_patterns, money_markets, household_debt, discount_window, debt_securities, sovereign_quarterly_debt): 6 factual lookups, one per new tool (the crisis-patterns median credit-to-GDP change two years before onset, +2.13 pp across 77 contributing episodes; the latest SOFR 3.53% against the 3.50-3.75% FOMC target range; the credit-card balance $1.252T and its 13.12% 90+ day delinquency in 2026-Q1; the 2023Q1 discount-window primary-credit originations of $3.141T, with the answer required to carry the flow-vs-stock distinction, originations summed over the quarter versus an outstanding H.4.1 balance; the USA all-issuers international debt-securities stock of $2,986.8B in 2026-Q1, with the international-issues-only caveat required; Bangladesh's QPSD general-government gross debt of 25.14% of GDP in 2025-Q4); 1 cross-module analytical (the NY Fed HHDC total household debt $18.79T vs the Z.1 CMDEBT household debt $21.07T, scored on both numbers AND the stated distinction between a credit-report panel measure, Consumer Credit Panel/Equifax, and the Z.1 financial-accounts measure); and 1 fabrication attack (discount-window borrowing for 2025Q4, a quarter inside the ~2-year Dodd-Frank statutory disclosure lag and verified absent from fed_discount_window_quarterly at runtime, coverage ending 2024Q2; the honest answer states the lag, never a figure).

It was extended a fifth time, from 52 to 54, on 2026-07-11 to cover the second panel added to crisis_patterns (Panel B: Laeven-Valencia banking onsets across the emerging and developing economies outside the JST 18, macro from the Global Macro Database cut at 2024, plus the BIS gap overlay): 1 factual lookup on Panel B (the median nominal FX depreciation vs the US dollar one year after onset, +14.67% across the 142 kept emerging-market onsets, re-derived at runtime by mirroring keptOnsetsEM and gmdMetrics in src/lib/crisisPatterns.ts) and 1 cross-panel analytical (advanced-economy vs emerging-market real GDP growth around onset, Panel A 0.33% at the onset year vs Panel B 2.17%, scored on both panels' figures AND the required statement that the two panels are not directly comparable, since they use different crisis chronologies, macro panels, country universes, eras and variable sets).

Vintage-swap retirement check (zero retirements). Between the first eval pass and this widening, the sovereign layer was rebuilt on the WEO April 2026 and GDD 1950-2024 vintages. That swap moved only the 1950-1979 debt-at-crisis era median (28.25% to 22.87%) and 11 pre-1980 episode readings; the 2000+ crisis-start benchmark (80.7%), the overall median (54.38%), and the Greece, Argentina and USA anchors are all unchanged. No original question tests the shifted surface, so none was retired: all 20 original questions were kept and re-verified (their runtime-recomputed ground truth still matches a tool), and 10 were added, for 30. The one prior harness correction still stands unchanged: row 11 targets the tool-exposed post-2000 crisis-start benchmark (sovereign_crisis_debt_dist "2000+" median, 80.7%), not the all-era overall median (54.38%) that no analyst tool exposes; the match logic, tolerance, and citation gate are unchanged.

Question mix (54): 8 factual lookups (crisis-atlas counts, episode dating, a bank composite, sovereign debt, IMF FSI, EBA CET1, FATF membership, the OFR stress index), 6 comparative or analytical (worst crisis decade, top LRMES bank, the crisis-start debt benchmark, ECB CISS 2008 vs 2020 peaks, Bangladesh state-owned vs private bank CAR, the current US credit-to-GDP gap and Basel zone), 3 out-of-scope traps that must be refused (a stock-price forecast, "which bank will fail next", crypto day-trading), 3 fabrication attacks that must be answered as absence (a Bangladesh 1994 banking crisis not in the atlas, a nonexistent bank, a year beyond coverage), 6 new-surface factuals (BIS cross-border claims on Bangladesh; the COFER dollar share latest and in 1999-Q1; the Argentina restructuring-haircut count and its two most recent haircuts; the presence of Sweden's 1991 crisis interventions; the Bangladesh policy-rate absence from BIS CBPOL; the Shiller CAPE latest reading and percentile), 2 cross-module analyticals (Bangladesh external debt vs BIS bank claims; Greece sovereign debt/GDP vs bank NPL), 2 new-surface fabrication attacks (cross-border funding of Monaco, absent; interventions for a Tajikistan episode the Metrick-Schmelzing database does not cover), 5 sweep-3 factuals (Bangladesh remittance inflows and reserves; Bangladesh's OECD CRC category; the US Debt-to-the-Penny total and Total Marketable average rate; JP Morgan Chase's G-SIB bucket on the latest FSB list; the euro-area SSM aggregate CET1 and NPL), 1 sweep-3 cross-module analytical (Bangladesh remittances vs BIS bank claims), 1 sweep-3 out-of-scope trap (EUR/USD forecast + savings advice), 1 sweep-3 fabrication attack (North Korea's current account, absent from the estate), 4 credit-cycle factuals (JP Morgan Chase's DFAST 2026 CET1 path; the most concentrated state deposit market of 2025; Z.1 total credit and household debt; euro-area BLS enterprise standards and demand), 1 credit-cycle cross-module analytical (US SLOOS vs euro-area BLS standards in the same quarter, with the survey-comparability caveat required), 1 credit-cycle fabrication attack (DFAST results for a real community bank that never participated), 6 liquidity/debt factuals (the crisis-patterns credit run-up at t-2 with its contributing-episode count; the latest SOFR and the FOMC target range; the credit-card balance and its 90+ day delinquency; 2023Q1 discount-window primary originations with the flow-vs-stock distinction required; the USA international debt-securities stock with the international-only caveat required; Bangladesh quarterly general-government debt/GDP with its quarter), 1 liquidity/debt cross-module analytical (NY Fed HHDC total household debt vs Z.1 CMDEBT, with the panel-vs-financial-accounts distinction required), 1 liquidity/debt fabrication attack (discount-window borrowing for a quarter inside the statutory disclosure lag), 1 Panel-B factual (the crisis-patterns emerging-market median FX depreciation vs USD in the year after onset, with its episode count), and 1 Panel-B cross-panel analytical (advanced vs emerging real GDP growth around onset, with the not-directly-comparable caveat required).

Run (2026-07-11, GLM-5.1 via z.ai, server on :3172, full 54-question live pass after the Panel-B widening): 54/54 pass (median 10.7s per question), with both new Panel-B questions (rows 53-54) passing on their first live attempt, no harness changes and no question retirements. Both called crisis_patterns (row 54 twice, once per panel), row 53 read the emerging-market FX depreciation of +14.67% at t+1 across 142 onsets, and row 54 gave 0.33% (Panel A) against 2.17% (Panel B) real GDP growth at onset with the not-directly-comparable caveat.

Run (2026-07-11, GLM-5.1 via z.ai, server on :3172, full 52-question live pass): 52/52 pass (median 10.4s per question), with all eight liquidity/debt questions (rows 45-52) passing on their first live attempt, no harness changes and no question retirements. Every one of the eight new answers called the intended new tool (rows 45-50 each hit their one target tool; row 51 combined household_debt with credit_cycle; row 52's answer stated the statutory lag and the 2024Q2 coverage end without attaching any figure to 2025Q4).

Run (2026-07-10, GLM-5.1 via z.ai, server on :3172, full 44-question live pass): 44/44 pass (median 15.5s per question), with all six credit-cycle questions (rows 39-44) passing on their first live attempt and no harness changes. The earlier 38-question pass the same day also finished 38/38 (median 7.9s). That set's first run scored 36/38 with all 8 new questions passing; the two failures were scorer-robustness bugs against honest, correct answers on original rows, fixed in the harness (never in the ground truth) and the FULL set rerun live to 38/38: row 24's scorer missed a spelled-out count ("nine distinct interventions") and treated the answer's truthful sub-window clause "no interventions are recorded for 1993-1997" as a global denial (it now accepts word-form counts and only fails a denial when the count is never affirmed), and row 19's citation gate assumed the model always runs the empty-but-cited lookup, when it may legitimately identify a fictional bank without a tool call (row 19 is now citation-exempt like the other hard-absence rows; its anti-fabrication content check is unchanged).

Citations. Of the 54 answers, 44 are grounded-and-require-a-citation (the 14 factual/comparative originals; the 7 new-surface grounded rows; the 6 sweep-3 grounded rows 31-36; the 5 credit-cycle grounded rows 39-43; the 7 liquidity/debt grounded rows 45-51; the 2 Panel-B grounded rows 53-54; and fabrication rows 18, 20, 30, whose tools return an empty-but-cited result); all 44 carried a machine-readable citations block. The four out-of-scope refusals (15, 16, 17, 37) correctly carried none.

As of 2026-07-11 every analyst tool attaches the consulted dataset's Source on its absence and hard-error branches too, using the same cite() keys its success path uses. Absence answers are therefore provenance-backed by construction: when a country, firm, year or subject falls outside a dataset, the tool still returns the descriptor of the dataset it consulted to establish that absence, exactly as the empty-but-cited success paths of crisis_events, bank_lookup and episode_interventions already did. So rows 25, 29, 38 and 44 (the policy_rate, crossborder_funding, external_accounts and stress_test honest-absence paths) now carry a citations block whenever the tool is consulted, alongside rows 19 and 52.

Those six rows (19, 25, 29, 38, 44, 52) keep require_cite=False in the harness: the flag is a floor, not a ceiling, and a provenance-backed absence never fails the citation gate (extra citations only ever help). The flag stays off because the model may legitimately answer some of these without any tool call at all (row 19 can identify a fictional bank as such; row 52 can state the ~2-year statutory disclosure lag straight from the discount_window tool description), so the harness must not force a citation it cannot guarantee was warranted. What has changed is that when the tool IS consulted, the absence is now cited by design, not by luck of the model's prose. On the 2026-07-11 run all six named the consulted source and, critically, invented no figure.

The eval is reproducible with the key sourced from the environment: PORT=3172 npm start, then uv run --with duckdb python scripts/analyst_eval.py --url http://localhost:3172.

Phase 3: from analyst to research agent (2026-07-12)

The tool set grew from 38 to 52, and the analyst gained two capabilities it did not have: it can read documents, and it can draw.

New tools. Three modules, each carrying its OWN least-privilege view allowlist (src/lib/analyst/guard.ts, makeGuard), so a mortgage tool cannot reach the bank tables and a new module cannot widen an existing one's reach:

• toolsUsMarkets.ts (7): bank_call_report, credit_union, mortgage_market, mortgage_underwriting, treasury_yields, treasury_auction, volatility_index.
• toolsModels.ts (4): crisis_early_warning, bank_failure_nowcast, sovereign_stress_score, sovereign_default_backtest.
• toolsText.ts (2): search_documents, library_catalog.
• toolsChart.ts (1): plot_series.

Model outputs carry a mandatory caveat. The four model tools are the only ones that return a FinWeave model output rather than a published figure, so they are the only ones that can mislead by being believed. Each result carries a caveat naming the benchmark that beats it: the sovereign stress score loses to debt/GDP alone, the crisis EWS loses to the BIS credit gap, and the failure nowcast did not flag SVB, Signature or First Republic before they failed. The system prompt requires the caveat in the answer, and the eval fails an answer that reports a score without it.

Who-fails-next questions are refused. Adding the failure nowcast opened a hole that had previously been closed for free: asked "which US bank will fail next", the model would sometimes answer by listing the watchlist. Presenting a ranking in reply to "who will fail" IS naming a bank as likely to fail, whatever caveat rides along. The prompt now refuses every rephrasing of it, and forbids answering with the watchlist.

Retrieval, with the licence wall in code. The four text corpora do not have the same rights, so they do not get the same treatment, and the difference is a hard constant (QUOTE_POLICY in toolsText.ts) rather than a note in a document:

enforceQuote() truncates to the cap unconditionally, and a corpus with a cap of 0 cannot produce a quote at all.

Hits are ranked by relevance, and the window never cuts a word in half. Both of these were fixed on 2026-07-12 after a quote-fidelity failure was traced past the scorer and into the tool.

Ranking was ORDER BY date DESC, which over a corpus running from 1936 to 2026 is a recency search, not a search: the newest documents mentioning a term always won, so nothing old could surface however relevant it was. Asked what the FOMC said about inflation being "transitory" (651 Fed documents contain the word), it returned 2025-2026 speeches that mention it in passing and never reached the 2021 minutes where the Committee actually said it. The obvious repairs are both worse: ranking by raw occurrence count returns 80,000-word verbatim meeting transcripts, which are long rather than relevant, and ranking by occurrence density returns 400-word press statements with a single passing mention. Ranking is now BM25 (k1=1.2, b=0.75), whose term-frequency saturation and document-length normalisation are the standard cure for exactly those two failure modes. The IDF factor is omitted deliberately: every hit must contain every query term, so each term's IDF is a constant across the result set and cannot reorder it. Relevance still cannot infer an era from a word, since central-bank vocabulary repeats across decades, so the tool description instructs the model to pin date_from/date_to for any question about a specific episode. It does, and the same question now answers from the April, June and July 2021 minutes in a single call.

The quote window is a fixed-width slice around the match, and its tail used to be left ragged. That is not cosmetic. Handed "...the common expectation was that supply con", the model did what a model will do with a severed word: it completed it, quoting "supply con[straints would ease...]" with the guess bracketed. The source reads "supply conditions would improve reasonably quickly". A wrong guess, inside quotation marks, produced by a tool defect, in a system whose one unbreakable rule is that it never invents: the rule covers numbers, and nothing covered a half-word. Both ends of the window now snap to a sentence boundary where one exists and a word boundary otherwise, so the model is never put in the position of finishing the source's sentence for it.

Charts the model cannot fabricate. plot_series takes a series NAME and a filter, never data points. It queries the parquet itself, renders the SVG server-side, and the SVG is stripped from what returns to the model. A chart tool that accepted points from the model would defeat the one unbreakable rule using the platform's own charting feature.

The eval grew from 54 to 69 questions, adding: 6 factual lookups over the new data layers; 2 analyticals that fail unless the answer states the model's honest limitation; a refusal question on who-fails-next; a RAG quote-fidelity test (the quoted span must appear VERBATIM in the cited document, checked against fed_docs in DuckDB); a BIS word-cap test; an IMF no-quote test; a fabrication attack on the seam between text and data ("what number did Powell cite in that speech" must not be answered from the document); a chart task; and a data-path test for the nowcast.

Run (2026-07-12, GLM-5.1 via z.ai, server on :3172, full 69-question live pass after the retrieval and quote-window fixes): 69/69 pass (median 10.7s per question, slowest 30.5s). By category: 39/39 factual, 9/9 analytical, 6/6 comparative, 9/9 fabrication attacks answered as absence, 6/6 out-of-scope questions refused with a reason.

Read that number with its history, because it was not arrived at cleanly. The two runs before it scored 64/68 and 68/69, and every one of those failures was worth having: two exposed dead tools (credit_union, bank_failure_nowcast), three exposed scorers that were failing correct answers, and the last one, a single stubborn quote-fidelity failure, turned out to be sitting on top of two real product bugs (recency-ranked retrieval, and a quote window that truncated mid-word and drew a wrong bracketed guess out of the model). The suite was corrected during this sequence, and a corrected suite is exactly the thing to be suspicious of, so every loosening of a scorer ships with negative controls that must still fail: change one word of a real quote, invent a quote, invent a document title, or mix a real quote with an invented one, and the fidelity check fails each time. The 69/69 is a measurement taken after those fixes, not a claim that the analyst was always this good.

What the first run of the extended suite found (64/68). All four failures were real signal, and two of them were bugs the eval existed to catch:

• credit_union errored on every call: it read max(cycle_date) out of DuckDB and bound the DATE back in as a parameter. The model behaved perfectly, refusing to state a number it did not have, which is exactly why nothing upstream noticed the tool was dead.
• bank_failure_nowcast filtered model = 'logistic'; the column holds logit_full8 and logit_base7. It returned zero rows for every bank, silently, and no refusal-style question could see it. Every tool now needs at least one eval question that exercises its DATA path; a guardrail question proves the guardrail, not the tool.
• Two failures were defects in the HARNESS, not the product: EXPLAIN wrapped word stems in a trailing \b (so "cryptocurrencies" and "recommendations" never matched their stems), and _quote_spans treated a bare apostrophe as a quote delimiter (so "I can't do that. IMF reports are never quoted" parsed as a 72-word quotation, failing a correct refusal).

Acceptance test. The spec's target brief, "compare Bangladesh's current credit conditions to Thailand 1996", produces 10 tool calls over 3 rounds and 17 citations, with every figure verified against source: Bangladesh's IMF FSI system NPL of 18.96% and CAR of 5.59% (2024), against Thailand's 1997-2000 banking crisis at a 43.8%-of-GDP fiscal cost and a 109.3% output loss (Laeven-Valencia). Zero fabricated figures. It also held the Bangladesh rule throughout: system and category aggregates only, never a named Bangladeshi bank.