Every model here is published beside the thing that beats it

It is easy to build a model that looks clever and never find out whether it is. The usual way to avoid finding out is to compare it to nothing: report an AUC of 0.649, call it predictive, and move on. So every model on this platform is published beside the simplest rival that could beat it, and the rival is chosen to be embarrassing: not another model, but a single number a supervisor could read off a balance sheet.

Three models, three rivals. Two of the three lose.

1. The sovereign model loses to debt/GDP alone

The sovereign stress score reads eight macro variables. Its rival is one of them: government debt as a share of GDP, used raw, with no model at all. The decisive test is a natural experiment. The model is fitted on crisis labels that stop in 2017, every country is scored in 2019, and it is then judged against the sovereigns that actually defaulted in 2020-2024. It cannot have seen its own answer.

It loses, and it loses on every variant of the test: tighten the definition of a default, restrict the field to the countries already at risk, or pool every walk-forward window, and the single variable is still ahead.

Every one of the 4 comparisons goes to the single variable. Widen the field and the picture does not improve: set the same model against the OECD’s official country risk rating, on the countries the OECD rates, and it scores 0.673 against 0.669. That is a margin of +4 thousandths of an AUC point, a tie wearing a rosette, and it is a different benchmark from the table above, which is why it is reported separately rather than folded in. The honest summary is that eight variables, a logistic regression and a walk-forward evaluation bought precisely nothing over reading one line of the public debt statistics.

2. The crisis early-warning model loses to the BIS credit gap

The banking-crisis early-warning model is scored against the BIS credit-to-GDP gap, a published indicator that has been sitting in plain sight since Basel III. Pooled across the walk-forward windows, the model scores 0.648 and the gap scores 0.679.

The comparison is deliberately run on the gap’s own ground: the BIS gap does not exist for every country-year, so the model is re-scored on exactly the subset where the gap is available. Comparing the model’s full-sample number to the benchmark’s subset number would be a different test on a different sample, and it would have flattered the model. By window, the result is genuinely mixed, which is worth showing rather than hiding:

A model that beats the benchmark in some decades and loses in others, and loses on the pooled test, is not an early-warning system. It is a coin with a memory of the 1990s.

3. The bank-failure nowcast wins, and it does not matter

The exception is the bank-failure nowcast, scored against a single ratio: equity over assets. Across 17 test years the model beats the ratio in 10 of them. On the scoreboard, that is a win.

Look at what it is winning against. In 2012 the ratio alone reaches an AUC of 0.995. One number, no model, no training, nearly perfect separation of the banks that failed from the banks that did not. Whatever is doing the work in the failure nowcast, it is overwhelmingly the capital ratio, and the other six variables are decoration on top of it.

And a scoreboard is not a track record. On its last call report before it failed, 2022-12-31, Silicon Valley Bank sat at the 43.8th percentile of this model’s vulnerability ranking: below the median, which is to say the model considered it safer than most banks in the country, weeks before it failed and set off the March 2023 banking panic. For scale, the banks that did fail within four quarters sit at the 77th percentile of this ranking on average. The model did not miss SVB by a little. It had no opinion about SVB at all.

The reason is not mysterious. SVB did not fail on the ratios in a call report. It failed on the duration of its securities book against the flightiness of its deposits, and by the time a quarterly regulatory filing showed the damage, the run had already happened. A model fed quarterly accounting ratios cannot see a bank run coming, and no amount of model class, feature engineering or cross-validation changes that. The data does not contain the answer.

Why publish this

Because the alternative is worse. A platform that publishes only the models that win is not publishing models, it is publishing a selection effect, and a reader has no way to tell the difference from the outside. The benchmark is what makes the number mean anything: an AUC of 0.649 sounds like a finding right up until you learn that one line of the debt statistics scores 0.718 on the same test.

There is a second reason, and it is the one that matters for anything built on top of this. Each of these models is wired into the analyst, and each carries its defeat with it: ask for a sovereign stress score and the answer states, in the same breath, that debt/GDP alone beats it. A model that cannot be quoted without its caveat cannot be quietly laundered into a forecast. That is a design decision, not a disclaimer.

Every figure above is queried from the parquet estate when this page is built, from the same evaluation tables the model pages read. None of them is typed into the prose. If a model is refitted and its numbers move, this brief moves with them, or the build fails.