The efficient-markets hypothesis has underpinned many of the financial industry’s models for years. After the crash, what remains of it?

Efficiency and beyond

Financial economics - Efficiency and beyond | Briefing | The Economist

It may force the country’s indecisive leader to make a fateful choice

Next week’s election is South Africa’s most important since 1994

Its democratic system is not as robust as it seems

Why America is vulnerable to a despot

Critics will miss globalisation when it is gone

The world’s economic order is breaking down

On the campaign trail, both main candidates largely ignore the problem

America’s fiscal outlook is disastrous, but forgotten

America’s $61bn aid package buys Ukraine time

Briefing

And the soldier home from the hill

Correction: Uighurs

The other-worldly philosophers

span

gv_safe

gb_safe

gb_safe_from_high

gb_safe_from_high_med

pos_goldman_inclusion_2024

pos_israel-hamas_war

gs_busfin

america_department_commerce

gs_busfin_indus

gs_busfin_indus_financial

IN 1978 Michael Jensen, an American economist, boldly declared that “there is no other proposition in economics which has more solid empirical evidence supporting it than the efficient-markets hypothesis” (EMH). That was quite a claim. The theory's origins went back to the beginning of the century, but it had come to prominence only a decade or so before. Eugene Fama, of the University of Chicago, defined its essence: that the price of a financial asset reflects all available information that is relevant to its value.

From that idea powerful conclusions were drawn, not least on Wall Street. If the EMH held, then markets would price financial assets broadly correctly. Deviations from equilibrium values could not last for long. If the price of a share, say, was too low, well-informed investors would buy it and make a killing. If it looked too dear, they could sell or short it and make money that way. It also followed that bubbles could not form—or, at any rate, could not last: some wise investor would spot them and pop them. And trying to beat the market was a fool's errand for almost everyone. If the information was out there, it was already in the price.

On such ideas, and on the complex mathematics that described them, was founded the Wall Street profession of financial engineering. The engineers designed derivatives and securitisations, from simple interest-rate options to ever more intricate credit-default swaps and collateralised debt obligations. All the while, confident in the theoretical underpinnings of their inventions, they reassured any doubters that all this activity was not just making bankers rich. It was making the financial system safer and the economy healthier.

That is why many people view the financial crisis that began in 2007 as a devastating blow to the credibility not only of banks but also of the entire academic discipline of financial economics. That verdict is too simple. Granted, financial economists helped to start the bankers' party, and some joined in with gusto. But even when the EMH still seemed fresh, economists were picking holes in it. A strand of sceptical thought, behavioural economics, has been booming. There are even signs of a synthesis between the EMH and the sceptics. Academia thus moved on, even if Wall Street did not. Nonetheless, the extent to which politicians and regulators trying to reform finance can trust financial economists is an open question.

The EMH, to be sure, has loyal defenders. “There are models, and there are those who use the models,” says Myron Scholes, who in 1997 won the Nobel prize in economics for his part in creating the most widely used model in the finance industry—the Black-Scholes formula for pricing options. Mr Scholes thinks much of the blame for the recent woe should be pinned not on economists' theories and models but on those on Wall Street and in the City who pushed them too far in practice. 

Financial firms plugged in data that reflected a “view of the world that was far more benign than it was reasonable to take, emphasising recent inputs over more historic numbers,” says Mr Scholes. “Apparently, a lot of the models used for structured products were pretty good, but the inputs were awful.” Indeed, the vast majority of derivative contracts and securitisations have performed exactly as their models said they would. It was the exceptions that proved disastrous.

Mr Scholes knows whereof he speaks. Long-Term Capital Management (LTCM), a hedge fund he founded with, among others, Robert Merton, a fellow Nobel laureate, skidded off the road in 1998. Since then, he has been pointing out dangers ignored or underestimated in the finance industry, such as the risk that liquid markets can dry up far faster than is typically assumed. (That did not stop Platinum Grove, the latest hedge fund in which he is involved, taking a big hit during the recent meltdown.) 

He has also been “criticising for years” the “value-at-risk” (VAR) models used by institutional investors to work out how much capital they need to set aside as insurance against losses on risky assets. These models mistakenly assume that the volatility of asset prices and the correlations between prices are constant, says Mr Scholes. When, say, two types of asset were assumed to be uncorrelated, investors felt able to hold the same capital as a cushion against losses on both, because they would not lose on both at the same time. However, as Mr Scholes discovered at LTCM and as the entire finance industry has now learnt for itself, at times of market stress assets that normally are uncorrelated can suddenly become highly correlated. At that point the capital buffer implied by VAR turns out to be woefully inadequate.

Even as financial engineers were designing all sorts of clever products on the assumption that markets were efficient, academic economists were focusing more on how markets fall short. Even before the 1987 stockmarket crash gave them their first real-world reminder of markets' capriciousness, some of them were examining the flaws in the theory.

In 1980 Sanford Grossman and Joseph Stiglitz, another subsequent winner of a Nobel prize, pointed out a paradox. If prices reflect all information, then there is no gain from going to the trouble of gathering it, so no one will. A little inefficiency is necessary to give informed investors an incentive to drive prices towards efficiency. For Mr Scholes, it is the belief that markets tend to return prices to their efficient equilibrium when they move away from it that gives the EMH its continuing relevance.

Economists also began to study “institutional frictions” in markets. For instance, the EMH's devotees had assumed that smart investors would be able to trade against less well-informed “noise traders” and overwhelm them by driving prices to reflect true value. But it became clear that there were limits to their ability to arbitrage folly away. Andrei Shleifer, a Harvard economist, among others, pointed out that it could be too costly for informed investors to borrow enough to bet against the noise traders. Once it is admitted that prices can move away from fundamentals for a long time, informed investors may do best by riding the trend rather than fighting it. The trick then is to get out just before momentum shifts the other way. But in this world, rational investors may contribute to bubbles rather than preventing them.

In the early years of the EMH, researchers spent little time worrying about the workings of financial institutions—a weakness of macroeconomics too. In 2000, in his presidential address to the American Finance Association, Franklin Allen, of the University of Pennsylvania's Wharton School, asked: “Do financial institutions matter?” Lay people, he said, “might be surprised to learn that institutions play little role in financial theory.” Indeed they might. Mr Allen's explanation was partly that the dominant theories had been shaped at a time when America, especially, was spared financial crises.

In the past decade or so, financial economists have been paying more attention to institutional questions, such as how bankers should be paid. Many of these researchers broadly accept the EMH, but see their role as uncovering sources of inefficiency that can be addressed to make markets more efficient.

However, a second branch of financial economics is far more sceptical about markets' inherent rationality. Behavioural economics, which applies the insights of psychology to finance, has boomed in the past decade. In particular, behavioural economists have argued that human beings tend to be too confident of their own abilities and tend to extrapolate recent trends into the future, a combination that may contribute to bubbles. There is also evidence that losses can make investors extremely, irrationally risk-averse—exaggerating price falls when a bubble bursts.

Behavioural economists were among the first to sound the alarm about trouble in the markets. Notably, Robert Shiller of Yale gave an early warning that America's housing market was dangerously overvalued. This was his second prescient call. In the 1990s his concerns about the bubbliness of the stockmarket had prompted Alan Greenspan, then chairman of the Federal Reserve, to wonder if the heady share prices of the day were the result of investors' “irrational exuberance”. The title of Mr Shiller's latest book, “Animal Spirits” (written with George Akerlof, of the University of California, Berkeley), is taken from John Maynard Keynes's description of the quirky psychological forces shaping markets. It argues that macroeconomics, too, should draw lessons from psychology.

“In some ways, we behavioural economists have won by default, because we have been less arrogant,” says Richard Thaler of the University of Chicago, one of the pioneers of behavioural finance. Those who denied that prices could get out of line, or ever have bubbles, “look foolish”. Mr Scholes, however, insists that the efficient-market paradigm is not dead: “To say something has failed you have to have something to replace it, and so far we don't have a new paradigm to replace efficient markets.” The trouble with behavioural economics, he adds, is that “it really hasn't shown in aggregate how it affects prices.”

Yet EMH-ers and behaviouralists are increasingly asking the same questions and drawing on each other's ideas. For instance, Mr Thaler concedes that in some ways the events of the past couple of years have strengthened the EMH. The hypothesis has two parts, he says: the “no-free-lunch part and the price-is-right part, and if anything the first part has been strengthened as we have learned that some investment strategies are riskier than they look and it really is difficult to beat the market.” The idea that the market price is the right price, however, has been badly dented.

Mr Thaler also says that only some of the recent problems were behavioural. Many were due to things that are open to non-behavioural economics, “like better risk analysis, how we identify hidden correlations.” It will be no surprise if, thanks to the catalytic power of the bubble and market meltdown, the distinctions between the two camps disappear and a new paradigm emerges.

One economist leading the effort to define that new paradigm is Andrew Lo, of the Massachusetts Institute of Technology, who sees merit in both the rational and behavioural views. He has tried to reconcile them in the “adaptive markets hypothesis”, which supposes that humans are neither fully rational nor psychologically unhinged. Instead, they work by making best guesses and by trial and error. If one investment strategy fails, they try another. If it works, they stick with it. Mr Lo borrows heavily from evolutionary science. He does not see markets as efficient in Mr Fama's sense, but thinks they are fiercely competitive. Because the “ecology” changes over time, people make mistakes when adapting. Old strategies become obsolete and new ones are called for.

The finance industry is in the midst of a transformative period of evolution, and financial economists have a huge agenda to tackle. They should do so quickly, given the determination of politicians to overhaul the regulation of financial markets.

One task, also of interest to macroeconomists, is to work out what central bankers should do about bubbles—now that it is plain that they do occur and can cause great damage when they burst. Not even behaviouralists such as Mr Thaler would want to see, say, the Fed trying to set prices in financial markets. He does see an opportunity, however, for governments to “lean into the wind a little more” to reduce the volatility of bubbles and crashes. For instance, when guaranteeing home loans, Freddie Mac and Fannie Mae, America's giant mortgage companies, could be required to demand higher down-payments as a proportion of the purchase price, the higher house prices are relative to rents. 

Another priority is to get a better understanding of systemic risk, which Messrs Scholes and Thaler agree has been seriously underestimated. A lot of risk-managers in financial firms believed their risk was perfectly controlled, says Mr Scholes, “but they needed to know what everyone else was doing, to see the aggregate picture.” It turned out that everyone was doing very similar things. So when their VAR models started telling them to sell, they all did—driving prices down further and triggering further model-driven selling.

Several countries now expect to introduce a systemic-risk regulator. Financial economists may have useful advice to offer. Many of them see information as crucial. Data should be collected from individual firms and aggregated. The overall data should then be published. That would be better, they think, than a system based solely on the micromanagement of individual institutions deemed systemically significant. Mr Scholes favours relying less on VAR to calculate capital reserves against losses. Instead, each category of asset should have its own risk-capital reserves, which could not be shared with other assets, even if prices had not been correlated in the past. As experience shows, correlations can change suddenly.

Financial economists also need better theories of why liquid markets suddenly become illiquid and of how to manage the risk of “moral hazard”—the danger that the existence of government regulation and safety nets encourages market participants to take bigger risks than they might otherwise have done. The sorry consequences of letting Lehman Brothers fail, which was intended to discourage moral hazard, showed that the middle of a crisis is not the time to get tough. But when is?

Mr Lo has a novel idea for future crises: creating a financial equivalent of the National Transport Safety Board, which investigates every civil-aviation crash in America. He would like similar independent, after-the-fact scrutiny of every financial failure, to see what caused it and what lessons could be learned. Not the least of the difficulties in the continuing crisis is working out exactly what went wrong and why—and who, including financial economists, should take the blame.

Financial economics

Weekly insights into a continent that is reshaping our world

Our weekly newsletter will help you make sense of the sweeping changes taking place across Africa

Exclusive insights into a continent that is reshaping our world

Analysing Africa

Get to grips with a region where capital, climate and geopolitics collide

Go deeper with our Middle East newsletter. Every week our correspondents delve into the political, social and economic stories of a region that is rarely out of the news.

Sign up to our Middle East newsletter

Middle East Dispatch

Follow the action in Latin America, from the depths of the Brazilian Amazon to the Mexican-American border

Every week our correspondents share insights on politics, economics and business from an increasingly complex region

Stay informed with our weekly Latin America newsletter

El Boletín

A sharp round-up of the most important US political news

Sign up for updates from our correspondents and data journalists, poll tracker and election-forecast model as the 2024 race unfolds

The US in brief

Inside Europe, from the front lines in Ukraine to the corridors of power in Brussels

Stay informed with our weekly Europe newsletter

Café Europa

Understand a vast and complicated country that will increasingly shape the future

Stay informed with our weekly India newsletter

Essential India

The best of The Economist’s defence coverage, in one place

The War Room

A weekly conversation about culture. Because culture is a serious business

Exploring trends and connections across entertainment, the arts and politics

Broaden your perspective with our weekly culture newsletter

Plot Twist

Analysing the challenges facing Britain and what needs to be done to overcome them

Stay informed with our weekly Britain newsletter

Blighty

Understand what the world makes of China—and what China makes of the world

Stay informed with our weekly China newsletter

Drum Tower

Your essential guide to global business and technology

From supply chains to semiconductors, The Bottom Line newsletter has you covered

Need to track the megatrends shaping business and technology?

The Bottom Line

Catch up quickly on the global stories that matter

Sign up to enjoy a mind-expanding mix of stories, delivered six days a week

Catch up quickly on the stories that matter

The World in Brief

A behind-the-scenes look at how we chose and designed this week’s cover

How we chose this week’s cover image

Cover Story

Exclusive insight from our correspondents in America

Exclusive insight and reading recommendations from our correspondents in America

Checks and Balance

A weekly fix of our mind-expanding science coverage

Curious about the world? Enjoy a weekly fix of our mind-expanding science coverage

Simply Science

Climate-change analysis that you won’t read elsewhere in The Economist

The Climate Issue

Expert analysis of the biggest stories in economics and markets

Money Talks

Taking you behind the scenes of our data journalism

Off the Charts

Long reads and life from 1843 magazine

The Extraordinary Story

The very best of our journalism, handpicked for you each weekday

A daily newsletter with the best of our journalism

Handpicked stories, in your inbox

The Economist today

Highlights from the latest weekly issue, introduced by our editor

Efficiency and beyond

The efficient-markets hypothesis has underpinned many of the financial industry’s models for years. After the crash, what remains of it?

Briefing July 18th 2009

From the July 18th 2009 edition

More from Briefing

Next week’s election is South Africa’s most important since 1994

Why America is vulnerable to a despot

The world’s economic order is breaking down