Stock prices still move too much, but no longer five times too much

The claim

If a stock price is the expected present value of future dividends at a constant real discount rate, the price is an optimal forecast of p*, the present value of the dividends actually paid afterwards. An optimal forecast varies less than the variable it forecasts, so detrended prices must satisfy sigma(p) ≤ sigma(p*). Shiller built p* from realized dividends and reported, for Data Set 1 (Standard and Poor’s annual data, 1871-1979, deflated by a producer price index), sigma(p) = 50.12 against sigma(p*) = 8.968, the paper’s published figures, implying a ratio of 5.59. The headline range of 5 to 13 takes its upper end from Data Set 2, a modified Dow index over 1928-1979.

What we ran, on what data

The estate holds Shiller’s continuously maintained ie_data monthly file: 1,845 rows, 1871-01 to 2024-09, with prices and dividends deflated by CPI. The deflator was verified, not assumed: real_price equals the nominal price times the ratio of the file’s last CPI to the contemporaneous CPI, with a maximum relative deviation of 0 across the file. The 1981 paper deflated by the producer (wholesale) price index, and the file’s historical prices and dividends carry revisions made since 1981, so this is not the paper’s dataset even over the paper’s own years.

The annual series follow the paper’s construction. Price is the January real price. The file’s dividend column is a trailing annualized rate interpolated to monthly frequency, so a year’s dividend is the mean of its 12 monthly real values; years with fewer than 12 monthly dividend observations, 1 in this file (2024 (6 of 12 months)), are excluded rather than zero-filled. Dividends therefore run 1871-2023, the terminal price is January 2024, and n = 154. Both series are detrended by an exponential trend fitted to the window’s log January prices, growth 1.98% a year over the full window; the discount rate is the paper’s rbar = E(d)/E(p) on the detrended series, 3.85%; and p* is built backward through the paper’s recursion p*(t) = (p*(t+1) + d(t)) / (1 + rbar), with the terminal value set to the sample mean of the detrended price, the paper’s choice. The 1871-1979 re-run estimates its own trend (1.35%) and rbar (4.62%) inside the window, as the paper did. One robustness variant was declared before running and no others were run: terminal value equal to the actual detrended January 2024 price.

The result: the bound still fails, the factor shrinks

On the full window, sd(detrended p) = 46.04 and sd(p*) = 26.01: the bound fails by a ratio of 1.77 under the paper’s terminal condition, and by 1.25 with the terminal set to the actual 2024 detrended price. What changed relative to the paper’s window is the denominator, not the numerator: sd(p) is 48.18 over 1871-1979 and 46.04 over 1871-2024, while sd(p*) rises from 12.45 to 26.01.

The terminal condition matters this much because the detrended January 2024 price, 243.2, sits at 2.2 times the sample mean of 111.6. Every p* value is a discounted sum of the subsequent detrended dividends plus the discounted terminal value, so moving the terminal from 111.6 to 243.2 lifts the late-sample path of p* and raises its standard deviation from 26.01 to 36.90. Both choices are shown; picking one silently would be choosing the result.

The paper’s own window, on this vintage

Restricting to 1871-1979 and re-estimating the trend and rbar inside the window gives sd(p) = 48.18 and sd(p*) = 12.45, ratio 3.87 (n = 109). The paper’s published figures for the same window are sd(p) = 50.12 and sd(p*) = 8.968, ratio 5.59. Same years, same method, different number. Two data differences are documented above: this file is CPI-deflated where the paper used a producer price index, and the historical price and dividend series have been revised since 1981. How much of the gap each difference contributes cannot be decomposed here, because the estate holds one vintage under one deflator.

Source: Robert J. Shiller, Irrational Exuberance monthly data (Yale) Standard deviations of the detrended annual series, in detrended real index points. The last row quotes the paper's published sigmas; its trend and rbar are not quoted here, so those cells read n/a.

What this cannot tell you

• No significance claim. The ratios are point statistics with n = 154 and n = 109, and no standard errors were computed. Flavin (1983, Journal of Political Economy) showed that variance-bounds tests on highly autocorrelated series of this kind are biased toward violation in small samples, so a ratio above 1 is not by itself a rejection at any stated confidence level.
• The detrending looks ahead. The trend is estimated from the whole window, so the detrended value at any year uses prices from decades later. That is the paper’s own method, kept here for comparability; the Marsh and Merton (1986) and Kleidon (1986) critiques of variance-bounds tests target this construction, and nothing on this page answers them.
• The terminal value dominates the late sample. The full-sample ratio is 1.77 or 1.25 depending on a single assumption about p* in 2024, because the detrended terminal price (243.2) sits far above the sample mean (111.6). Both are reported; neither is privileged.
• This is not the paper’s dataset. The estate’s file is CPI-deflated and carries post-1981 revisions; the paper deflated by a producer price index. The paper’s own window returns 3.87 here against the 5.59 implied by its published sigmas, and the gap cannot be attributed between the deflator and the revisions with the data on hand.
• The factor of 13 is untestable here. It comes from the paper’s Data Set 2, a modified Dow index over 1928-1979, which the estate does not hold. Only Data Set 1’s bound is re-run on this page.
• The discount rate is assumed constant. rbar is one number per window, estimated from the sample itself as E(d)/E(p): 3.85% on the full window, 4.62% on the paper’s. Models in which the discount rate moves over time are a different hypothesis, and this test says nothing about them.