Tuesday, 29 November 2016

Is technological progress at the heart of stagnation?

In the previous text I presented several economic hypotheses explaining why the developed world has entered what could possibly be a prolonged period of economic stagnation. In today's text (note: long read) I will present my own opinion, arguing that what we are experiencing is a temporary slowdown which could last for several decades, but one that could also provide the greatest opportunity for the next huge boost in living standards. I hypothesize that the underlying factor behind both the current temporary stagnation (particularly in productivity and real wages) and the upcoming rise in living standards is - technology

As I've emphasized several times on the blog before, I believe we are currently, for the past 30 years, in the period of the Third Industrial Revolution. And in our times, it's only heating up, with the potential to bring to some new disruptive innovations that could change our world as much as the previous two industrial revolutions had. The technological progress we are currently undergoing will, without doubt, be disruptive. As it always is. But in its disruptive nature it will bring greater benefits for the future generations. Automated work being replaced by robots will surely lead to job losses. But in a new economy, the job losses could be offset by a series of new opportunities and entirely new careers. Which ones in particular, we can't really tell at this moment. But just like the first two industrial revolutions brought completely new jobs and changed the world as we know it (more than 95% of jobs that exist today didn't exist before the 18th century), so will the Third bring in new jobs and new possibilities we can't even imagine. Social networks have already introduced new types of occupations (social network experts being the most dubious one). Various bloggers and youtubers have managed to turn their hobby into a money making venture. Firms are just beginning to exploit the Internet, and its users are just uncovering the various ways they can make money on it. None of this was even conceivable back in the 1990s when our remote Internet usage was often wrapped around in frustration with our dial-up connection (remind yourself of that glorious sound, I know you want to!). Today so much more opportunities await. 

The good thing about this inevitable change is that it tends to be gradual. This meaning that even if robots and automated work start replacing low-skilled jobs, this will all still happen during a prolonged period where it will be possible to maintain a generational switch.

What does this mean in particular? Let's take the example of the taxi market and the driverless car (or if you want - Uber, which is the intermediate step). Naturally if driverless cars all start hitting the streets they will almost immediately take away all the jobs from the taxi drivers. Which is likely to cause them to rebel, quite legitimately so. One always has an incentive to protect their job and their immediate interest. A good way to achieve a peaceful transition would be to allow the technological breakthrough to enter gradually by having the taxi drivers operating and overseeing the driverless cars at first. This would, on one hand, correspond to a permanent barrier to entry for any new driver, however all the current drivers would keep their jobs. Until retirement or until they find another job, whichever comes first. Each current driver would therefore still be driving/riding the driverless car and providing for example local advice to tourists. This would then be a perfect way to tell whether or not the passengers really enjoy the conversation and demand for the actual person to be in the car, or do they just prefer the robot to take them where they need to go without speaking to it. It's all about having choices! And in this way to minimize the despair of potential job losses imposed by the new technology. But I digress. 

The current slowdown is essentially of temporary effect as we're currently in a transitional period from the old industry-driven economy (including the service industry) to the new digitally-driven economy. The industry-driven economy still rests upon the old industrial classification paradigm: the primary sector (agriculture, fishing, and mining), the secondary sector (manufacturing, production), and the tertiary sector (services). So far in the history of the West we have witnessed the transition from an agricultural-dominant economy to a manufacturing-driven economy (the First Industrial Revolution in the 18th century), a shift from manufacturing to rapid industrialization in the 19th and beginning of the 20th century (the Second Industrial Revolution driven by mass innovation), and a shift from industrialization to services in the final part of the 20th century following a period of rapid globalization.

Now we are facing something different - a shift beyond the standard paradigm. Disruptive technological progress will rapidly change our patterns of production and of specialization. It will be nothing like the world we knew so far. Just like the first two industrial revolutions brought us to a state of the economy not known to us before. In the 16th and 17th century having a locomotion and machines was unimaginable. In the 18th century having electricity, cars, airplanes, and modern medicine was unimaginable. After WWI having computers and traveling to space was unimaginable. 40 years ago a cell phone was unimaginable, while a mere 15 years ago a smartphone was unimaginable. True, there were always visionaries who offered their overly enthusiastic views of the future by simply extrapolating the current levels of technological progress. In the 1960s visions of the future included flying cars, intergalactic travel, jet-packs, and personal robots, all by the year of 2000 (check out some of the futurist visions from that time - some of them actually did come to existence; also read this piece to see which predictions came true).

How the Internet has changed things - for the better

Why don't many people see this obvious advantage of the technological progress so far? A famous quotation from Nobel prize winner Robert Sollow: "You can see the computer age everywhere except in the productivity statistics" is actually true. There is a productivity paradox where the advances in computing power haven't really made workers more productive. This is contrary to the idea that automation of work should increase total factor productivity. Essentially the idea is that despite all the benefits the Internet has brought to us (instant global communication, entirely new business and marketing models, different consumer behavior patterns, social networking, even spontaneous mass gatherings), it has made only marginal improvements in well-being, at least compared to the non-internet age of the 1980s. The technology skeptics cite similar examples where modern technologies only offered marginal improvements over the products we enjoy today. For example, whereas the first invention of the car was a huge advantage over a horse, its further improvements, after reaching a certain level of speed and safety, were marginal. Airplanes are a similar example. Yes, today they tend to be much safer, but flight times are still similar to what they were 50 years ago. The smartphone was an improvement over the regular cell phone, but not as much as the cell phone was an improvement over the landline, and both not as much as the regular landline was an improvement over letters and telegrams.

However all these examples are missing the point. We are still at the very early stages of the Third Industrial Revolution - the Digital Revolution. We are slowly entering the Information age. The Internet has made much bigger changes than standard economic indicators would suggest. Particularly since most of them were indirect. The Internet has, without doubt, changed the patterns of firm specialization and has increased the rate of trial and error as well as innovation. The vast availability of information online can improve business strategies and force businesses to adapt to the Internet revolution. Those that don't, lose customers. No matter what industry they are in. In the upcoming decades this will become even more obvious. Furthermore, the Internet has had a key role in promoting economic opportunity for all, particularly for the underprivileged. To start a business all you need is a laptop and an internet connection (of course, this varies from country to country depending on the scope of regulations). Most importantly the Internet and the increasing socialization it has brought with it can be used to foster democracy and the empowerment of the middle classes. That is one of its, by far, biggest advantages. Social networks and the Internet can do more in overthrowing dictators and holding politicians accountable in democracies than the media ever could. (Obviously they can also be used in an opposite capacity - by distributing fake news and encouraging bubble behavior; but to be fair, fake news and living in bubbles happened way before the Internet). 

For all these reasons, in the upcoming Information Age, the Internet should be free for all - a global public good. Free access to the Internet should be a human right (the UN has done a lot in promoting this idea, there is even an initiative to implement the Internet as a basic human right, but some disagreements still remain). Nevertheless, its creation of economic opportunity, its role in fostering democracy by empowering the middle classes, its ease of access to education (online courses can do wonders!) are more than enough to declare it a human right and offer it as a global public good. This is something that the future might hold for us - free Internet, worldwide. (Although, don't be so sure on the "free" part. We have access to electricity and clean water, but both still come at a cost).

Learning from Japan

Even with all those advantages at hand, we have currently reached a ceiling with our pre-IT revolution models of economic growth. Japan is perhaps the best example. A huge booming economy for 30 years following the recovery after WWII, it was hit by a housing bubble burst in 1990 and has experienced very low to zero economic growth ever since. In the past 26 years Japan grew, on average, by 0.7%. Its lost decade of the 90s has turned into two lost decades and is now in the middle of a third. Its public debt is the largest in the world, and by a long shot (public debt to GDP is 230%, higher than even Greece, with debt to GDP at 170%). For the entire 26-year period inflation has been close to 0, borderline deflation, as have their interest rates. Needless to say Japan has done a series of monetary and fiscal stimuli to prompt up their economy throughout this period, but nothing has worked. The consequences of both are highly visible in their over-expanded monetary base and huge debt.

But in reality, there is nothing wrong with Japan. Yes its economic indicators are terrible, yes the population is ageing which is always a problem in countries with high debt levels, but Japan remains one of the richest countries in the world. Their GDP per capita (PPP) is around $38,000 which is quite a lot for a country with 127 million people. The only comparable in population size is the US with a GDP per capita (PPP) of $55,000. But beyond GDP, Japan ranks highest in many of the measures of living standards and well-being. By life expectancy they are no.1 in the world (84 years on average!), their human development index, prosperity index, happiness index all put them among the top performers. Their health care and education system are flourishing, they have low crime rates, and decreasing inequality. One conventional economic indicator - unemployment - always managed to stay low, at around 4%. And their levels of innovation and technological adaptation are arguably among the world's highest if not the highest. The vast majority of the population is therefore enjoying really good living standards. It seems that low GDP growth, ongoing deflation, and high public debt (as long as it is held by the domestic population in a country stable and rich enough to have a huge demand for its debt, particularly domestic in this case), don't really hamper living standards. The two and a half decades of stagnation have not apparently taken their toll on well-being.

So what's the story here? Japan has simply reached a level of very high living standards combined with a strive for technological innovation that has perhaps even worsened some productivity numbers and possibly GDP growth as well (although there are a number of reasons why GDP growth was low in Japan). The IT revolution took huge proportions in Japan. Anyone who's ever been there speaks of its technological superiority and a number of "cool gadgets". Few of these gadgets have raised GDP, but they have contributed to the well-being of the population, and more importantly, they have opened vast new opportunities for its population. It takes time for these to be seized in order to produce a high magnitude well-being effect.

The Third Industrial Revolution IS at the heart of the current stagnation...

The point is that we are currently experiencing a stagnation caused by a series of factors, one of which is certainly the technological revolution. I've written on that before on multiple occasions. Essentially, technological progress began by shifting jobs and changing the patterns of specialization and production. This will go on for several more decades. But the further it unfolds, the more benefits it will bring that will be immediately noticeable to us. In other words, think of the current stagnation in wages and productivity (and hence economic growth) as an indirect consequence of the upcoming and ongoing technological progress. It takes time for the people to recognize the new patterns of specialization and to exploit the opportunities for new jobs. With the start of the IT revolution in the 80s we've already noticed a series of new jobs being created. It all started with companies like IBM, Dell, HP, and Apple to provide the hardware. Then came Microsoft and revolutionized the software (Apple made it even cooler later on in the second coming of Steve Jobs). Then the Internet giants started emerging: Google, Facebook, Amazon, YouTube, eBay and many others. (The criticism these companies get is that while they replaced many old jobs in manufacturing, it wasn't actually a one-for-one replacement. They created much less jobs than what manufacturing companies created.)

All these new companies emerged in the very beginning of the IT revolution (some even before like IBM). Expect many more of these to come in the following decades. Don't necessarily expect new search engines, software manufactures, online retailers, or social networks. No, the new high tech companies will be about something completely different. They will most likely strive on the benefits provided by all these companies before them (we all have laptops, running on either Windows or Mac, and we all use Google, Facebook, YouTube, etc.).

...but it will also set the stage for the next big boost in living standards

The current level of technological development set the stage for the Next Big Thing. They didn't replace all the lost jobs, and were on that front mostly disruptive innovations. For now. However the foundations have been laid. These foundations are supported by the current (temporary) industry giants. And most importantly they provide the nurturing environment for growth, for new companies that one day will be even greater and will perhaps not only change the jobs market, they could profoundly change our way of life (e.g. robot manufacturing companies, or nanotech companies, or AI producers, or fusion energy companies - a bit too much? Or is it?). In 50 or 100 years from now we may look down on manual labor and automated work as relics of the past. And no one will complain as everyone will find such jobs meaningless and will have the time to pursue their most desired careers. It might sound a bit idealistic from today's point of view but who knows. Cars, trains, and airplanes sounded idealistic back when the First Industrial Revolution was underway, but today we know of no better form of transportation. At this point, we can only imagine.

Now, all this is just a theory. I can devise a multitude of examples and arguments in support of it, but I cannot really test it. Yet. Time will tell basically whether or not this thinking has any merit at all. I do however carry a slightly optimistic bias in believing that we live in awesome times and are on the verge of a breakthrough that most of us simply aren't aware of. My optimistic bias makes me a bit subjective towards the impact of technological progress on future living standards, but drawing simply from historical patterns and the possibilities being uncovered to us, the IT revolution is nothing to be feared.

Sunday, 20 November 2016

Explaining our current stagnation

Ever since the financial crisis of 2007-2009 and its subsequent (slow and modest) recovery many have claimed the world has entered into a state of prolonged stagnation. In addition to economic growth being relatively low (and therefore not enough to close the potential GDP gap caused by the crisis), real wages are also stagnating, unemployment is still high (although in relative decline), inflation is close to zero, while productivity growth is sending troublesome signals for some time now. This is particularly true of Europe, as it bears the strongest resemblance to Japan, and is on a good course to repeat Japan's (still ongoing) two decades of stagnation (more on emulating Japan in my next text). 

We all know the story. I, for one, have told it many times on the blog (see herehere, here, here, here, here or here). After the financial crisis, which usually tends to cause prolonged and slow recoveries, many governments adopted stimulus and bailout programs in 2009 that all but destroyed their public finances at the time. In retrospect this was a textbook Keynesian solution in times of crisis - in order to restore confidence and replace the lack of private sector investment the government should step in and provide as much liquidity and stimulus as possible. And most of them did. The US, the UK, almost all European countries, even some Asian countries (like China) were forced to adopt stimulus packages as an immediate response to boost confidence. In addition central banks did their part and lowered interest rates to historical lows to provide the much needed liquidity to the banking sector (whose reaction was mainly to hoard this cash, not let it flow in the system). But this doesn't work, the conventional wisdom teaches us, since you can only lower rates so much, forcing you to fall into a liquidity trap. And the only thing that can get you out of a liquidity trap is more government spending.

This solution was applied throughout. And its main consequence, in only a single year, was that debt levels have risen sharply (to be fair bank bailouts contributed to rising debt levels even more). They almost doubled in the US and in the UK, as in many European countries, while in countries like Ireland and Iceland they quadrupled. Budget deficits also went haywire. The UK in 2010 had the third biggest deficit in the world: -10% of GDP (behind only the revolution-undergoing Egypt and the shambolic Greece). Even for rich and usually fiscally responsible countries like the US or the UK, this was too much. The textbook Keynesian solution might have prevented a deeper slump as some economists claim (this we will never know as we cannot prove it), but it also dramatically increased budget deficits and public debt levels (this we can prove - see my discussion here). Austerity was imposed only after the stimulus and bailout packages (starting in 2010/11), it was not the initial reaction. Many argue that austerity was applied too quickly, before the economies actually recovered, but that too is a discussion for another time. 

The slow recovery is only part of a longer trend of stagnation? 

In order to understand the big picture of why the recovery was so slow, we must look at the trends that have occurred before the crisis. Many claim the stagnation (particularly in productivity and real wages) started long before. The first graph looks at the decline in the growth rate of total factor productivity (TFP) in the US, relative to its 1947-1973 trend. Natural logs are used to emphasize the relative stagnation of TFP. (Btw, the FT Alphaville blog has assembled in one place all the different hypotheses and ideas on why TFP growth started to decline since the 70s).

The quarterly TFP rates for the US from 1947 till 2010.
Graph taken from David Beckworth's blog
The second figure depicts a very modest, almost nonexistent, growth of real wages compared to productivity which has, as shown in the previous graph, experienced its own relative slowdown (the reason the graphs are not comparable is that the upper one uses natural logs, whereas in the lower one the second part of the picture only shows the rate of change from 1979 to 2009). Knowing that productivity rise fell short of its trend-based expectations, it is all the more striking to see the relative stagnation of real wages in the past 30-40 years
In addition, the wage growth was distributed unequally, where the trend for the bottom 90% of income earners was even worse throughout the observed period, not only for the US but for some other developed countries as well (shown here are Australia, Canada, France, Sweden and the UK): 
So what are the structural factors responsible for this 30-year long era of relative stagnation in productivity and real wages? Are these structural factors the same ones disabling our economies from fully recovering from the recent crisis?  

Economists came up with several competing hypotheses, each very interesting in its own way. I will present five of them briefly. 

1) The secular stagnation hypothesis

The first in line is the so-called secular stagnation theory. Its main proponent is Larry Summers. According to this theory, "economies suffer when higher propensity to save is coupled with a decreasing propensity to invest". Excessive savings will therefore lower aggregate demand which puts a downward pressure on inflation and on real interest rates (hence the low inflation and low interest rates we are experiencing) and lower economic growth. Furthermore even when high growth is achieved within this several-decade-long stagnation, it was usually a result of excessive borrowing that translates savings into unsustainable investments and causes bubbles.

I'm having difficulty in buying this argument given that the data shows a clear trend of declining savings rate in the US for the past 30-40 years before the crisis (see graph below). Besides, a bubble could not have lasted that long. It is true that higher savings was a response to the crisis (as you can also see in the graph), and it's also true that the immediate post-crisis consequence was massive deleveraging of a population overburdened with debt, but it certainly isn't a long term trend.

Perhaps Summers is referring to total gross savings, which has indeed been steadily increasing in the past 40 years, but in this case looking at the absolute value is wrong. Even gross savings as percentage of national income have been in a steady decline since the 1980s. To be fair, Summers is perhaps more preoccupied with the last couple of years (given that only in the last 7 years have we had historically low real interest rates):
"Secular stagnation occurs when neutral real interest rates are sufficiently low that they cannot be achieved through conventional central-bank policies. At that point, desired levels of saving exceed desired levels of investment, leading to shortfalls in demand and stunted growth. This picture fits with much of what we have seen in recent years. Real interest rates are very low, demand has been sluggish, and inflation is low, just as one would expect in the presence of excess saving. Absent many good new investment opportunities, savings have tended to flow into existing assets, causing asset price inflation."
However, hasn't China had a massive imbalance between savings and investments for the past 30 years? In times when its economic growth rates were in double digits for decades? It was this circumstance in particular that turned China into the greatest creditor nation in the world (S>I, meaning that EX>IM), while the US became the greatest debtor nation in the world. The US had a much higher propensity to invest than to save, and this imbalance led to a huge current account deficit (just the opposite of above; I>S, mean that IM>EX). The story of trade, unlike what the politicians make you believe, revolves around the relationship between savings and investments; here I fully agree with Summers.

2) The debt accumulation hypothesis  

The next one comes from Reinhart and Rogoff. They tend to blame massive debt accumulation. In other words, a period of sustained and bold optimism in which asset prices kept on rising, meaning that both the public and the private sector may borrow indefinitely. This happened not only in the US, but across the spectrum, as many countries ran large current account deficits prior to the crisis. This was particularly problematic in Europe as the desire to eliminate risk through the introduction of the common currency encouraged borrowing from abroad where high CA deficits were channeled into consumption rather than investment. What explains the slow recovery is deleveraging - a typical reaction to financial crises caused by an excessive accumulation of debt. In fact, in their excellent book "This Time Is Different" the authors point out that banking crises that arose due to an excessive accumulation of debt always imply a very slow and prolonged recovery, not only for the financial centers but also for the periphery.  

However this is still a theory focused only on the explanation of the post-crisis stagnation; it doesn't stretch long enough to explain the puzzling decline in productivity and real wages for the past 30 years. So we move on to the next one.

3) The global savings glut hypothesis

This famous hypothesis was proposed by Ben Bernanke, the former Fed chairmen, back in 2005. According to Bernanke reducing a financial surplus (households pushing savings) or running large deficits (governments or households financing consumption) will result in a (potentially decade-long) boom on the asset market. Bringing this up on an international level, financial surpluses from Asian households and governments were translated into investments and consumption in the West. The analogous story can be told in Europe: the 'interaction' between the savings in the "core" and the deficits in the "periphery".

In essence this hypothesis also begins with the period of low interest rates that reflected higher world savings. I wrote about it in my 2011 paper "The Political Economy of the US Financial Crisis": "There was strong demand for safe assets from Asian and oil exporting countries that contributed to depress the yield on long term government securities issued by advanced economies, the US in particular. A low US savings rate also contributed in steering assets from current account surplus countries into financing US investments and consumption. However, capital inflows were used to finance current consumption rather than investment into productive assets. The US current account deficit started to grow uncontrollably by the end of 1998 and reached its highest level around 2006, while at the same time oil exporting countries and emerging Asian countries experienced high surpluses in their current accounts. This period is matched by the likewise high growth in the US housing market. There is no proof that the current account deficit itself caused the housing boom, but there is evidence that the inflow of foreign capital was mainly used at the time being for the purchase of real-estate, adding to the housing bubble. Excess savings in Asia were being invested into safe assets such as US government securities, which contributed to a high level of capital inflows into the US. High inflows into the US brought about excessive risk taking and exposed domestic financial institutions, companies and households to exchange rate risk. Pushing excess savings towards assets increases the demand for these assets which resulted in an appreciation of asset prices. This put additional pressure on demand as well as on total output. An inflow of foreign savings, combined with low interest rates and expectations of constantly increasing asset prices resulted in the creation of an asset bubble in both houses and securities. An increasing demand for assets motivated the financial market in developing new instruments and securities (derivatives) whose main purpose was to diversify risks." 

4) The long-run decline in growth hypothesis 

The most recent hypothesis, attributed to economist Robert Gordon, does go back long enough to explain the fundamental decline in the total factor productivity growth. In fact, that's what the theory is all about. In his bestselling new book "The Rise and Fall of American Growth", Gordon paints a very pessimistic picture of an exhausted american growth model. He claims that all the life-altering innovations of the past (in particular from 1870 to 1970) will not be repeated in the future, meaning that our TFP as well as our economic growth rates may go down even more (he makes a prediction of long-run economic growth to fall down to only 0.2% - see graph below). Some of reasons of why this could be so (the so called 'headwinds' the economy is facing) are the rising inequality, an ageing population, poor education, and rising debt levels. In brief, Gordon's view is that the technological revolution will not increase our living standards. I personally disagree with this assessment, as I find it unnecessarily pessimistic. I will challenge it thoroughly in my next blog post.

Robert Gordon's projection of average growth until 2100.
What about the origin of the current stagnation? According to Gordon it was a mere exhaustion of innovation. None of the stuff produced in the late 20th and at the beginning of the 21st century (like the Internet, or iPhones, or Google and Facebook) can match themselves to the benefits given to our societies during the late 19th and 20th century - things like electricity, cars, penicillin, running water in homes, the telephone, etc. Many of the 'headwinds' such as the rise of inequality, the ageing of the baby boomers, or rising debt levels are attributed to this very problematic feature and are, according to Gordon, responsible for the relative decline as well as for the even worse future rates of growth. So the current stagnation is a mere beginning of a long trend of close to zero economic growth given that we will fail to emulate the technological breakthroughs of the 20th century. A truly depressing outlook. 

5) The 'low-hanging fruit' hypothesis

Finally, Tyler Cowen in his great book "The Great Stagnation" argues that the US simply ran out of low-hanging fruits which fueled american growth from the late 19th century onward. He makes an interesting claim that these low-hanging fruits brought the country to its current technological plateau and now it's stuck here for a while before a next major revolution happens. 

So what are these low-hanging fruits the US had and has by now exhausted? Cowen cites the three most important ones: free land and abundant resources (particularly in the late 19th and early 20th century as it attracted many talented Europeans to enjoy the relative abundance); technological breakthroughs from the 1880s to the 1970s (the same ones Gordon mentions: electricity, motors, cars, planes, telephone, plumbing, pharmaceuticals, mass production, radio,TV, etc.), and last but not least smart, uneducated kids (a vast amount of people that educated themselves and massively contributed to economic growth).

All of this is gone now. Moving a student from high school to college today will only reap marginal returns at high costs. Moving a child from a farm to high school back then significantly increased its skill-set and thus opened up room for innovation. This innovation came in the form of massive technological improvements which all greatly increased our living standards; not only in terms of faster transportation or handy appliances - it also significantly improved our health and increased life expectancy. No modern-day innovation can improve our living standards that significantly nor can it expand our life expectancy to a 100 years. The Internet, social networks, search engines and smartphones are all cool and useful stuff, but their impact on our living standards is not even comparable to that of electricity, engines, conveyor belt production, or pharmaceuticals. 

Yet! We have no idea how the Internet will change our life in the future and what opportunities the current technological plateau will open up for us. Remember, we are in the midst of the Third Industrial Revolution - the IT Revolution. It's benefits won't be obvious to us quite yet. My hypothesis is that the IT Revolution is at the heart of the current stagnation. I will defend this argument in more depth in the next post. 

Thursday, 10 November 2016

We called it! How we predicted a Trump victory with amazing precision

First of all apologies to my regular readers for not presenting our results here sooner. It's been overwhelming in the past two days - first with the prediction, then with the results, and then with the post-election frenzy. 

Anyway, we gave an almost perfect prediction! Not just a Trump victory, but also all the key swing states (PA, FL, NC, OH), and even that Hillary could get more votes but lose the electoral college vote.

Here are our results as I presented them in a Facebook post on the eve of the election:

For a more detailed explanation read our blog. The method is described there in greater detail, plus we call all the states.

The story got covered first by the academic sources. My own University of Oxford published it as part of their main election coverage, as did my alma mater, LSE on their EUROPP blog.

More news coverage soon to come!

Details of our prediction 

The results nevertheless came as an absolute shock to many, but it was the pollsters that took the biggest hit. All the major poll-based forecasts, a lot of models, the prediction markets, even the superforecaster crowd all got it wrong (we have summarized their predictions here). They estimated high probabilities for a Clinton victory, even though some were more careful than others in claiming that the race will be very tight. 

Our prediction survey, on the other hand, was spot on! We (by that I mean Oraclum Intelligence Systemspredicted a Trump victory, and we called all the major swing states in his favour: Pennsylvania (which no single pollster gave to him), Florida, North Carolina, and Ohio. We gave Virginia, Nevada, Colorado, and New Mexico to Clinton, along with the usual Red states and Blue states to each. We only missed three – New Hampshire, Michigan, and Wisconsin (although for Wisconsin we didn’t have enough survey respondents to make our own prediction so we had to use the average of polls instead). Therefore the only misses of our method were actually Michigan, where it gave Clinton a 0.5 point lead, and New Hampshire where it gave Trump a 1 point lead. Every other state, although close, we called right. For example in Florida we estimated 49.9% to Trump vs. 47.3% to Clinton. In the end it was 49.1 to 47.7. In Pennsylvania we have 48.2% to Trump vs. 46.7 for Clinton (it was 48.8. to 47.6. in the end). In North Carolina our method said 51% to Trump vs. 43.5% for Clinton (Clinton got a bit more, 46.7, but Trump was spot on at 50.5%). Our model even gave Clinton a higher chance to win the overall vote share than the electoral vote, which also proved to be correct. Overall for each state, on average, we were right within a single percentage point margin. Read the full prediction here.

It was a big risk to ‘swim against the current’ with our prediction, particularly in the US where the major predictors and pollsters were always so good at making correct forecasts. But we were convinced that the method was correct even though it offered, at first glance, very surprising results.

Read more about the method here

The graphics

Here is our final map:

For the swing states:

And here are the actual results (courtesy of 270towin.com):

Pretty good, right? 

Here is, btw, what the other poll-based forecasters were saying (more on that here):

In addition to these other forecasters we were tracking were even more confident in Hillary taking all the key states. As you can see no one gave PA to Trump, some were more careful about FL and NC, although they too were mostly expected to go to Hillary. However the reason I think PA was key in this election is because everyone thought Hillary's victory was certain there. Not to mention the shocks of losing MI and WI as well. If Hillary got these three states, even by losing the toss-up FL and NC, she would have won (278 EV). This is why, I believe, all the forecasters were so certain (some more than others) that Hillary will pull it off. Holding on to what was supposed to be her strongholds (all three states were last Red under Reagan in 1984) was to be enough for victory. Trump dominated the Rust Belt. Which is why I think this election was a good example of an economic vote. But more on that in another post. 

Thursday, 3 November 2016

New Scientist: "As US election looms, time is ripe for a new science of polling"

My article got published today at the New Scientist! One of the biggest science magazines in the world.

See the text here (there is no paywall, you just register and read it for free). It was even on the front page:

New Scientist website front page 03 Nov 2106
The text is about the scientific experiment behind our prediction survey. It starts by examining why the pollsters are getting it wrong lately and whether or not there is any science at all behind polling. Then it introduces our prediction survey idea and how we're doing an experiment on US elections to see whether or not science can actually improve polling. 

For those who don't bother to register in order to read it on the New Scientist webpage, I have copied the text here (enjoy!):

As US election looms, time is ripe for a new science of polling

"Growing scepticism about traditional methods for predicting election outcomes is fuelling a search for a more scientific approach to polling, says Vuk Vukovic

As the US prepares to vote for its new president next week, narrowing political polls have suggested that this crucial election may be too close to call.

Although such polls are hugely influential – affecting financial markets, for example – it is becoming clear that we should not set too much store by them. Their reliability is increasingly doubted in the wake of polls that got it wrong on big occasions, such as those relating to the UK’s 2015 general election and Brexit vote, and Donald Trump securing the Republican party nomination in the US.

Why might that be? These days, pollsters find it harder to get responses by calling voters on their home phones. A typical telephone survey now has a response rate of below 9 per cent, with fewer willing interviewees making the polls less likely to be representative of the wider voter population and, hence, less precise and reliable.

Telephone polls are usually carried out during the day, biasing the results towards stay-at-home parents, retirees and the unemployed. Most people, for some reason, do not respond to cellphone surveys as eagerly as they once did to those by landline.

Online polls have their own weaknesses: they tend to be biased towards particular voter groups, such as the young, better-educated and urban dwellers.

In both types of survey, pollsters try to compensate for biases, but the results of doing so can be dubious – as shownwhen four different pollsters gave four different results for the key swing state of Florida in the current US campaign based on the same data set. Furthermore, a recent study showed that the actual margin of error of a poll’s finding is about 7 per cent, instead of the typically reported 3 per cent. Not surprisingly, some critics argue that opinion polls are more art than science.
Turning to science

Putting polling back on a scientific footing will require experiments in the coming years, combining insights from various branches of sociology, economics, mathematics of networks and statistics.

I am one of a group of researchers at Oraclum, a start-up based in Cambridge, UK, involved in conducting precisely this type of experiment. Our new kind of poll is conducted online, meaning we have to make election predictions from unrepresentative and biased samples of voters.

However, we have added a twist that we hope will improve its power to predict an election outcome, in that we go beyond asking people who they will vote for. We also ask who they think will win and their view on who other people think will win. The idea is to incorporate wider influences, including peer groups, that shape an individual’s choice on voting day.

Why might this work? When people make choices, such as in elections, they usually succumb to their standard ideological preference. However, they also weigh up the chance that their favoured choice has. In other words, they think about how other people will vote. This is why people sometimes vote strategically and do not always pick their first choice, but can opt for the second or third to prevent their least-preferred option from winning.

It is going to take a fair few experiments to answer the question of whether contemporary polling can be considered scientific.

And though the current US election is widely condemned for its negative atmosphere, it provides a good chance for a new science of polling to begin to take shape.

If you are a US voter, you can help Oraclum test its polling method by participating in its survey and sharing it with your friends."

Vuk Vukovic is a researcher at Oraclum and a PhD student at the University of Oxford