The U.S. Energy Information Administration (EIA) projects U.S. energy production, consumption, and prices each year in the Annual Energy Outlook (AEO). Every two years, EIA also produces the AEO Retrospective Review (AEO Retrospective). The AEO Retrospective compares recent history with the Reference case projections from previous editions of the AEO. It also shows the relationship between past AEO projections and actual energy indicators, and it informs discussions about the underlying models.

The projections presented in the AEO are not statements of what will happen but of what may happen given the assumptions in the underlying National Energy Modeling System (NEMS). These publically available assumptions include projections of oil prices and gross domestic product (GDP). The AEO Reference case projection generally assumes current laws and regulations, and it includes current views in economic and demographic trends and technology improvements.

Although the AEO Retrospective focuses on the Reference case projections, we review the projections in the context of the AEO’s full range of cases, which illustrate the uncertainty inherent in long-term projections. Beyond the Reference case, each year’s AEO typically includes cases that vary assumptions on economic growth, oil and natural gas supply, oil price, and renewable cost. In addition, recent AEOs have also included alternative cases that examine alternative policy developments.

Methodology overview

The AEO Retrospective summarizes the relationship between the Reference case projections and realized energy usage by comparing absolute differences and the average absolute percentage differences for many of the major measures published in AEOs from 1994 to 2020. The average absolute percentage difference is the simple mean of the absolute values of the percentage difference between the Reference case projections and the actual values. Most historical data are from the Monthly Energy Review (MER), which is available through the EIA open data API.

Table 1 summarizes 25 comparisons based on 21 primary measures projected in the AEOs for each year. Table 2 contains the standard deviations of the percentage forecast errors (between the Reference case projections and the actual values) for every horizon H=0 to H=15 computed with AEOs from 1994 to 2020. For price quantities, we computed the standard deviations of the log errors. The third column of Table 1 summarizes the percentage of occurrences in which a particular measure was overestimated relative to its actual value out of the total number of annual projections for which an actual value can be compared. A zero represents a measure that was always underestimated, and 100% represents a measure that was always overestimated.

Tables 3 through 23 show the detailed results that include the Reference case projections, historical values, and percentage differences between projected and actual values for all covered years. The detailed tables also provide the average absolute difference across all AEOs for each covered year.

Previous-year statistics can change from one year’s evaluation to the next because of updates to historical data in the MER and changes in the measurement of GDP. In addition, the statistics do not account for the continual improvements made to NEMS in an attempt to more accurately capture developing energy market trends based on more recent experience.

Key findings

As reflected in the relatively lower average absolute percentage differences in the second column of Table 1, measures of energy consumption are projected with greater accuracy than measures of net imports and of energy prices. Energy consumption tends to change at a slower pace than other indicators. Many energy-using devices, such as appliances, automobiles, and industrial equipment, are expensive to purchase and have long service lives. The substantial lead time for these large purchases, along with the effects of energy contracts and other mechanisms, tend to slow changes in consumer energy demand.

For the measures with wider deviations, many factors contribute to differences between the AEO Reference case projections and realized outcomes, but two primary contributors are the initial projections of future oil prices and the overall economic activity that are used in NEMS. These projections can greatly influence the other projections the model makes, which is why each AEO reviewed in this report includes alternative cases exploring differences in economic growth (Low Economic Growth case and High Economic Growth case) and in oil prices (Low Oil Price case and High Oil Price case).

In an unbiased projection, with a sufficiently large number of samples, overestimates and underestimates over time would occur in equal measure. In comparing the AEO Reference case projections with realized outcomes from 1994 to 2019, out of the 25 variables listed in Table 1, 17 more variables have historically been overestimated than underestimated.

Changes in industry-specific market conditions, significant technological breakthroughs, and new laws or regulations after the publication of an AEO can also lead to significant differences between projections and realized outcomes. For example, AEO Reference case projections from AEO1998 through AEO2017 projected the United States to continue to be a net importer of natural gas through 2017. In reality the United States became a net exporter of natural gas starting in 2017.

Table 1. Comparison of AEO Reference Case projections with realized outcomes, 1994–2019

Source: These statistics summarize the calculations in Tables 2 through 23. The data in Tables 2 through 23 are based on the 1994 through 2020 AEO Reference case projections. Historical Data are from the U.S. Energy Information Administration open data API (http://www.eia.gov/opendata/) (Washington, DC, September 2020), with the series listed under each appendix table, except for GDP data which are from the Bureau of Economic Analysis, U.S. Dept. of Commerce, August 2020, http://www.bea.gov/national/xls/gdplev.xlsx

1 The basis for GDP comparison is the projection differences in the cumulative average growth rate of real GDP from the first year shown for each AEO. The summary information for projection differences given for GDP growth rates is absolute percentage point differences; for all other AEO concepts, the comparison basis is absolute percent differences.

2 As of 2013, the wellhead price of natural gas was no longer reported by EIA. With the 2015 edition of the Retrospective, the natural gas price to the electric power sector replaced the wellhead price.

3 As natural gas net imports approached zero and then turned negative (indicating the change to U.S. net exporter status), the average absolute percent difference increased significantly.

4 Beginning in AEO2003, EIA electric generating projections incorporated combined heat and power (CHP) electricity generation in electricity generating plants. Prior to AEO2003, coal price projections reflected data collected, estimated, and reported to electric utilities and excluded CHP power generation.
Source: EIA