Barriers to Scientific Progress – Part 2

How our information pipelines have become clogged.

Continuing this essay series on modern barriers to scientific progress, using Michael Bhaskar’s book Human Frontiers: The Future of Big Ideas in an Age of Small Thinking, this essay will explore how our modern information pipelines have become clogged in various ways.

As Bhaskar writes:

[Economist Paul] Romer assumed that if you invested in new ideas at a constantly growing rate, an equivalently constant rate of return would be exhibited. If you have more researchers and their productivity improves, this will automatically lead to more economic growth. Boosting the number of researchers or R&D spend should, therefore, boost economic growth. Put more into big ideas, and you should get more big ideas. In the mid-1990s a young Stanford economist, Charles I. Jones, began questioning that assumption. Just as Romer had put the study of ideas on a firm economic footing in his 1990 paper, so five years later Jones introduced a wrinkle. Jones noticed something interesting about growth theory: the output of ideas wasn't commensurate with R&D. Between 1950 and 1987 the number of US-based scientists and engineers had risen from under 200,000 to a million, a fivefold increase not reflected (even remotely) in the growth rate. Aggregate R&D went up, average growth rates did not. Jones had found evidence that ideas were, in a material sense, getting harder to realise … A more recent paper, “Are ideas getting harder to find?”, from Jones, along with colleagues Nick Bloom, John Van Reenen and Michael Webb, builds on this work. Here detailed empirical research develops an extraordinary new model giving concrete shape to the struggle for new ideas and showing how, over time, their generation has become more difficult. At its heart are Romer's insights that economic growth relies on the creation of new ideas, and the creation of new ideas relies on researchers and their productivity. But while the number of researchers and R&D expenditure is going up, their research productivity is going down. It takes more and more research to produce new modes of transport or new life-saving drugs. As [Jonathan] Huebner had clocked, our ability to generate new ideas seems to diminish with every passing year … A sample of firms reveals that research productivity is declining in 85 per cent of the companies surveyed. On average, research productivity was decreasing at 9 per cent per year: compounded, a huge fall … Since the 1930s, research effort has increased by a factor of twenty-three, averaging growth of 4.3 per cent per year. But total US research productivity has gone down by a factor of forty-one since the 1930s, a decline of 5.1 per cent a year. Another way of thinking about this is to say that every thirteen years we need to double our research effort just to stay on the same course … Bloom et al study the critical foundation of the 3IR and the digital economy: Moore's Law, that opposite of Eroom, named after the co-founder of Intel, Gordon Moore, and predicting that the number of transistors on an integrated circuit, a chip, doubles every two years. However, as the paper argues, the actual story is more complex. While Moore's Law looks like a rising technological curve, maintaining that curve requires more and more effort and expense. Research productivity in computer chips is declining at an average of 6.8 per cent a year: “Put differently, because of declining research productivity, it is around 18 times harder today to generate the exponential growth behind Moore's law than it was in 1971.” … Crop yields and agricultural R&D exhibit a similar pattern. Since the 1980s growth in agricultural production has been slowing, but the number of US researchers in the area doubled between 1970 and 2007. For crops including corn, soybeans, cotton and wheat, yields are generally on long-term rising curves, doubling between 1960 and 2015. But R&D expenditure from both government and non-government sources, including research on cross-breeding and hybridisation for better insect resistance and nutrient uptake, improved herbicides and pesticides, bioengineering and the automation of seed-related tasks, has also risen sharply. Just as with Moore's Law, it takes more and more researchers to maintain the same level of growth. Depending on the crop and the R&D effort, the increase factor ranges between 3 and 25. The average productivity growth in corn yields equates to minus 9.9 per cent over the period. Jones, Bloom, Van Reenen and Webb also pick up on the healthcare question. They examine ‘new molecular entities’ (NMEs) approved by the Food and Drug Administration. These may be chemical or biological, and they include virtually all new and significant drugs. Between 1970 and 2015 research inputs increased nine times, but over the same period research productivity fell by a factor of five. In other words, the number of new NMEs has risen, but it takes a vastly greater number of researchers to develop each NME now than in 1970. Eroom strikes again. But they also wanted to look at how diseases are treated more widely, and the relationship between this, idea production and R&D. Life expectancy rises in a linear, non-exponential fashion, but such arithmetical increases are correlated to exponential growth in research. The result is, once again, a decline in the efficiency of our efforts. The paper examines the two top killers in the US: heart disease and cancer. Using publications relating to a given disease as a proxy for research efforts, they find that the number of publications on cancer increased 3.5 times in the years 1975 to 2006, and publications on clinical trials 14.1 times. Despite these significant increases in research effort, the additional years of life saved is falling: between 1985 and 2006, declining research productivity means that the number of years of life saved per 100,000 people in the population by each publication of a clinical trial related to cancer declined from more than 8 years to just over one year. If you judge research output by verbiage, you don't find a problem; if you judge it by the rate at which ideas find purchase, you do … Moore's Law may require more and more input to keep going, but what of fields still in their freshest, most burstingly productive phase? Capturing such information isn't easy but Huebner and Vijg provide two negative examples, and the authors of the ideas paper do attempt to account for this in part. Sources like Encyclopædia Britannica and Wikipedia which both, in different ways, classify achievement also corroborate the general pattern. Further evidence comes from a second look at patents. Major new inventions prompt not just individual patents, but whole new classes of patents. These in turn necessitate the reclassification of existing patents as those novel classes ripple through the firmament of invention. Big ideas should show up in either increasing or decreasing rates of patent class creation and class reclassification. While there is some ambiguity in the data, the rate (at least for US patents) seems fairly constant over time. This buttresses the point about declining research productivity: despite having more researchers, knowledge and resource, we are not creating new patent classes any faster. If anything there is a slight bias towards a slowdown. Where the ideas paper looked only at existing technologies, whole new technologies are registered here; and here too we find diminishing returns.

This slowdown is affecting business formation:

New technology should trigger the creation of new businesses. The invention of the motor car, for example, produced jobs in everything from spark plug factories to car park construction. Big ideas create big new sectors, which open space for new companies. Hence the rate of big idea creation should result in rapid business formation and overall economic dynamism. Alas, the economy does not paint an encouraging picture of an era brimming with big ideas. Although America does better, the pace of successful business formation there is still poor. Startups struggle. Their number is falling and the failure rate is abysmal: 95 per cent fail to deliver the expected returns and 30–40 per cent burn through all their capital. Most surprisingly of all, the numbers of new firms and IPOs (Initial Public Offerings, i.e. stock market launches) both peaked decades ago. Rates of entrepreneurship have declined in major economies like the US, the UK and Germany, and they have declined faster among people with higher degrees … And the pattern holds even in the fabled tech sector: tech company creation peaked in 2000 and growth rates have been on the slide since the 1980s … Most advanced economies display a clear oligopolistic tendency: in banking, energy, telecoms, property, consumer goods and food retail, for example, the major players have remained essentially static for decades. In 1987 only one-third of firms were more than eleven years old; by the 2010s it was half. Moreover, these old firms account for 80 per cent of the total US workforce, up from 65 per cent in 1987. The proportion of young firms correspondingly continues to fall. Another of way of looking at this is that in the 1970s, about seventeen establishments opened for every 100 existing ones, while thirteen closed. After 2000 the corresponding figures have been more like thirteen and eleven – fewer entrants, fewer exits, less dynamism overall. The US government estimates that Fortune 500 companies doubled their share of the economy between 1955 and the present. Even here the top take more: the share of Fortune 500 revenue going to the top 100 grew from 57 per cent to 63 per cent between 1994 and 2013.

It also appears that creativity itself may be on the decline.  As Bhaskar writes:

Between 1980 and 2000, 305 of the 400 top twenty films every year were originals – that is, not a sequel, prequel, reboot, franchise rollout or spinoff. Between 2000 and 2020 only 189 of the same total were original; as time went on the number of originals in the top ten shrank to virtually nothing … Perhaps there is a connection with an observed decrease in creativity. Since the 1990s measures of creativity among US students have been falling, sparking talk of a “creativity crisis”. These measures include skills like original thinking (the ability to generate new ideas, and in great numbers) and also any decline in open-mindedness; that is, less of a creative attitude and a welcome view of new and different ideas. This is likely part of the broader dynamic that inhibits many forms of radical originality. It doesn't bode well for a new renaissance … Few would call this a golden age for original thought. Bemoaning the state of the humanities and social sciences, the anthropologist David Graeber saw an attenuation of ambitious thinking. Everyone still endlessly discusses the thinkers of the 1960s and 1970s without producing comparable work: “No major new works of social theory have emerged in the United States in the last thirty years.” The philosopher Agnes Callard agrees, writing, “When I am asked for sources of ‘big ideas’ in philosophy – the kind that would get the extra-philosophical world to stand up and take notice – I struggle to list anyone born after 1950.” Similarly a study of 500 Western polymathic intellectuals finds plenty born in the 1940s, from Julia Kristeva to Vaclav Smil, Jacqueline Rose to Bruno Latour, but encounters a precipitous fall from the 1950s on: “The drop around 1950 may be an alarm signal”, the author writes … Throughout history the collision of cultures has been a fecund ground [for progress]. Consider how the European Renaissance was sparked by trade with East, allowing the recovery of lost works, or how the Columbian Exchange [LINK to my Columbian Exchange essay] following the connection of the Americas with Eurasia sparked a pan-continental transformation in everything from diet to economics. Yet globalisation has homogenised the world. We wear the same clothes (jeans, suits and baseball caps), eat the same foods (pizza, noodles, beer, cola), work for and buy from the same companies (Apple, Walmart, VW, Nestlé) watch the same films (Disney), speak the same languages (English), worship the same idols (TikTok, Pokémon, the Beatles). Societal interchange has become so immediate and total that everything blurs into a single culture where only the far interstices are free of the bland blend. Exchange of ideas and cultures will inevitably start to diminish.

Indeed, some of the only social science ideas to become widely fashionable in recent years are Ibram X. Kendi’s “anti-racism” concept, and Robin DiAngelo’s “white fragility” idea, both of which are based on fundamentally false premises, as I’ve explored in previous essays.

Bhaskar continues:

In the late 1980s and early 1990s, Francis Fukuyama noticed the phenomenon, famously (and more optimistically) calling it the ‘End of History’. His hypothesis has been mocked as the epitome of liberal overreach, but it's also widely misunderstood. He neither believed nor argued that events, including those of the highest significance, would stop, just that the principles and institutions of government and politics were unlikely to develop much further. Fukuyama's reading of political philosophy suggested that a major evolution of economically efficient and psychologically satisfying capitalist liberal democracy was unlikely. Had Fukuyama more prosaically called his article and book “The End of Ideological Evolution and Political Big Ideas” it would, one suspects, have attracted less controversy. Moreover, Fukuyama never blithely assumed that liberal democracy was assured; not only might it fall prey to external threats, he pointed out, but liberal democracies could wither internally from contradictions, a lack of new ideas, misplaced focus, anomie, disillusion and distraction. Much of which we now see … There is no radically new or ambitious imagining of the world that doesn't replay existing concepts. For the most part programmes, policies and ideologies aim to adapt or perfect the present system.

Bhaskar writes that quantity is now trumping quality:

The school of rational optimism builds a powerful case that the world is, in fact, drastically improving. They argue – rightly – that on average, humanity lives longer, healthier, wealthier, safer lives than ever before. How does that picture square with the argument for a slowdown in the production of big ideas? In 1900 just 6.4 per cent of Americans completed high school, but now well over 90 per cent do. In the US, educational attainment increased by 0.8 years per decade over the period 1890–1970. [Note that, as I’ve explored in previous essays, students in America today tend to highly their own educational achievements higher than ever, but their standardized test scores have dropped of stagnated overall.]  People today also have more leisure time to pursue their interests: thanks to shorter working hours, more holiday and earlier retirement, the amount of life we spend working has gone down by a quarter since 1960 … [A]t least twenty-three elite US universities each have more than $5 billion in the bank and over a hundred sit on more than $1 billion … Around 7.8 million researchers work in professional science, by far a historic record. Other estimates suggest that 90 per cent of the scientists who have ever lived do so right now. Over the twentieth century, the number of knowledge-producing workers in the US increased by a factor of nineteen … Global R&D spending is at least $2.2 trillion. In 1973 the equivalent figure was just $100 billion; in 2000 $722 billion. The share of R&D as a proportion of global GDP has ebbed and flowed around 2 per cent, but is now lodged above 2 per cent (around 2.3 per cent per year, according to the OECD) and as the economy grows, so does this expenditure. The top 1000 corporate investors in R&D have a trend R&D growth rate of 4.8 per cent per year, putting in over $700 billion across sectors including IT software and hardware, healthcare, aerospace and defence, automotive, chemicals and energy. At this pace, spending doubles every fourteen years.

In the next essay in this series, we’ll explore the problems created by the modern tendency for over-specialization.