Robot in a studio designing other robots

Is superintelligence a myth? Some alternate hypotheses

October 18, 2026October 19, 2026 Carwil Bjork-JamesLeave a comment

Since August and the Hugging Face Incident, we’ve seen a massive new level of conversation about the implications of current AI rapidly evolving into superintelligence. This conversation had long been largely hypothetical, and the relatively small set of people who regarded it as plausible were able to articulate their vision of what a so-called “intelligence explosion” would look like. (I’ve summarized some key sources on this set of ideas before.) The essential trajectory was laid out Ray Kurzweil’s multiple books on the singularity and Nick Bostrom’s Superintelligence.

Critics of this story have put forward a number of arguments: that LLMs’ capacities are being wildly overstated (Emily Bender and Alex Hanna’s The AI Con), that belief in the coming intelligence explosion is primarily a valuation strategy for venture capital (Ed Zitron), that the deep roots of this idea are essentially eugenicist (the new documentary Ghost in the Machine) and the more recent roots involve a variety of implausible and unrealistic ideologies (the TESCREAL bundle, as synthesized by Timnit Gebru, and explored by Adam Becker). I want to do something different here and question whether the future narrative holds together as a set of scientific propositions.

Building superintelligent machines, the leaders of current AI labs tell us, is (1) an accelerating process that will rapidly diverge from our capacity to control; (2) will constitute a step change in intelligence far beyond human comprehension; (3) will enable a vast increase in scientific and technological progress.

And it is the end of this sequence that makes the ongoing, and soon-to-be-greater-than-trillion-dollar investment in AI all worth it. As Dario Amodei, CEO of Anthropic, put it:

Many of the implications of powerful AI are adversarial or dangerous, but at the end of it all, there has to be something we’re fighting for, some positive-sum outcome where everyone is better off, something to rally people to rise above their squabbles and confront the challenges ahead. (Amodei, Machines of Loving Grace)

I’m skeptical of all three of these propositions and want to put forward some clear alternative ideas on each of them. This doesn’t mean I don’t think we should hedge against these ideas being right. Instead I think we should apply the Precautionary Principle to AI and put decisions with high consequences under collective democratic control, rather than letting the venture capital market decide to race into this. (But that’s another essay.)

AI Booster Hypothesis 1: AI is about to explode due to recursive self-improvement, becoming rapidly more intelligent with no clear limit. AIs can design machines much smarter than they themselves are.

AI lab scientists like to talk about generative AI in two ways: as systems with unexpected emergent properties and as fundamentally analogous to human minds. Neither of these provides a clear way forward towards super-human intellectual powers. In the language of many designers, AI models aren’t something that humans design, so much as something they grow with the right kinds of inputs. The Scaling Hypothesis essentially argued that simply contributing more knowledge artifacts, then testing them against metrics, would yield greater and greater intelligence.

Alternate Hypothesis 1a: Plagiarism Machine: Large language models are arrangements of the crystallized intelligence (texts, decisions, computer code, and recorded interactions) of humans. The gains they have made so far have to do with ingesting larger chunks of this corpus, more efficiently replicating the patterns within them, and stringing the algorithms within them into longer strings of action. In essence, the training apparatus teaches LLMs to simulate existing human intelligence with greater and greater fidelity. This is not a recipe for fundamentally new and better forms of intelligence.

The areas of highest LLM performance, computer coding and abstract mathematics, are the areas where the entire human thought process must be written down and error is clearly specified, allowing a well-harnessed LLM to persist longer at these strategies than humans, achieving “superhuman” results. These are what I would call “comprehensibly better” results.

Alternate Hypothesis 1b: LLMs as complex dynamic systems, not conventional software: The internal structure of large language models is a set of numbers (“weights”) connecting conceptual tokens to one another. These are complemented by an external structure of software harnesses and skills. In either case, weights are adjusted through training to produce better outcomes on a series of tests.

The story of recursive self-improvement has long assumed that AI systems were conventional computer programs. It highlighted the risk of turning over writing these programs to the current generation of AI systems, essentially creating new software that AI systems understand, but humans don’t. (And, as sci-fi writers have worried for generations, the two of them turning against us.) As the AI 2027 authors imagine the situation,

“The best human AI researchers are still adding value. They don’t code any more. But some of their research taste and planning ability has been hard for the models to replicate. Still, many of their ideas are useless because they lack the depth of knowledge of the AIs. … These researchers go to bed every night and wake up to another week worth of progress made mostly by the AIs.”

This is not, however, the way that LLMs are being constructed. In their white paper on recursive self-improvement, Anthropic describes research tasks as “running experiments,” “proposing hypotheses, testing them, sharing findings with parallel agents, and iterating.” LLMs remain objects that are experimentally characterized. The experiments and their results are the understanding, and that is something humans share in by reading and understanding the results.

AI Booster Hypothesis 2: Assuming it exists, superintelligence is incomprehensibly better than human thinking.

Alternate Hypothesis 2a: Limited spectrum of intelligence Bostrom’s Superintelligence and other conversations presume a kind of linear scale of intelligence in each area and guess that there is a much larger space beyond human capabilities than there is within it. Belief that intelligence is a linear numerical scale leads to notions like “superhuman persuasive ability,” and implies a rapid loss of control.

What if, on the other hand, intelligence is essentially a synthesizing capacity, is fundamentally about distinguishing meaningful patterns and operating on them? What if human evolution has already nearly “saturated” the space of intelligence? If there’s no reason to assume that human intelligence is the greatest possible intelligence, there’s equally no reason to assume intelligence has infinitely greater versions.

If infinitely greater capacity makes intuitive sense for you, consider visual perception. Humans have a rather limited inborn capacity for turning light into shape, colors, patterns, and perceived objects: our vision spans only a certain set of wavelengths and our eyes can’t magnify or reduce massively. But our visual field is rich and deeply integrated into our sense of the world. Those quantifying visual perception equate human eyes to a 576 megapixel camera. Amplified with tools like telescopes, microscopes, and electron microscopes, but also drawings, maps, diagrams, and scale models, we can visually perceive at any scale from the atomic to the galactic. We can see complexity and order, flows and attractors in everything from solar plasma to multidecadal ocean currents to the circulation of human language. These are ways that human perception is already augmented, and they all build on our exceptional capacities of perception. While it’s possible to extrapolate to machines scanning a visual field with greater resolution, it’s doubtful there is significantly greater meaning to be gathered from the world from this shift. Human biology, I would suggest, has nearly saturated visual understanding, while human technology continues to transform nearly all kinds of mappable data into forms that can be thought visually.

Alternate Hypothesis 2b: Human intelligence is distributed and recursive. Bostrom and Kurzweil point to humans’ limited memories and capacities to retain active thoughts in one’s head. (Though novelists, composers of symphonies, and software engineers should all stand as examples that these limits are quite high.) But we also know that human ability to understand concepts is recursive. In fact, this is much of what science is: knowledge of systems that takes other more detailed knowledge as a given. A biochemist doesn’t need to keep all prior human understanding of molecular bonding in her mind, nor does an expert in the latter need to hold all understanding of the underlying quantum mechanics. Likewise, a symphonic composer doesn’t need to know how to strike a drum or design a brass instrument to resonate effectively. Much knowledge lives permanently outside a human thinker’s skull, known only to others, yet is a part of their intellectual achievement in fields like these.

Many of the just-so stories of superintelligence imagine an individual human being surpassed by a software program in an intellectual task as the key milestone for the rise of machine intelligence. But human intelligence relies on the accumulation of past ideas, the harnessing of collective action, and yes, the use and manipulation of machines. The collaboration of multiple human agents is the home turf of our species, propelled towards sociality, united by language, accumulating our capacities through writing and systems of knowledge, and extended by the tools we make.

AI Booster Hypothesis 3: Superintelligent AI can do unfathomably better science.

Alternate Hypothesis 3a: Pre-AI human science already accesses the benefits of computation. This should go without saying, but twentieth and twenty-first century science is built atop digital computers. Compiling data, accessing information at large and small scales, synthesizing information, simulating natural and technological systems — all of these are done in quantities of thousands, millions, or even trillions through the use of computers.

Alternate Hypothesis 3b: Science is fundamentally about abstraction and comprehension. If the offloading of immense feats of computation is one feature of the last century of science, this hasn’t changed the nature of scientific understanding, which is about detecting and comprehending patterns across many iterations of the same phenomenon. Sometimes these phenomena can be explained with noncomputational explanations — like the lower pressure on the top of an airplane’s wing or how DNA encodes long protein chains — but computation is surely helpful for applying this knowledge. Other phenomena like solar flares, atmospheric storms (talking about both Jupiter’s Giant Red Spot and hurricanes here), or tectonic plate movements, only make sense through computational modeling and simulation.

An example may be in order. Consider a contemporary scientist at one of the handful of hurricane prediction centers around the world. Aided by high-powered but entirely deterministic climate and weather models, and a team of researchers designing, running and interpreting them, this scientist can produce predictions about the near-term future that would be incomprehensibly advanced to any individual alive in 1926. (I’ll address the introduction of “AI Weather Prediction” in a minute.) The calculating abilities of the machine in this lab have been numerically “superhuman” in Bostrom’s sense since at least the 1960s, and their weather prediction abilities for decades as well. But human knowledge and understanding is augmented rather than replaced by these systems, which also encapsulate calculations made by a variety of scientific disciplines and derived from observation.

Human knowledge of hurricanes now involves five things: awareness of the various principles underlying the science, experimental and theoretical modeling of their internal dynamics, experience tracking and witnessing the outcomes of hurricanes and their measurements, knowledge of the patterns of historically observed hurricanes, and the hypothetical results of simulated storm development. All five of these are now mediated through digital systems, but none involve “alien minds” doing the thinking for us.

In the last decade, some of the same research teams that work on generative AI, notably Google’s DeepMind, have been producing AI weather models (Artificial Intelligence Weather Prediction; AIWP) that link the comprehensive data gathered for conventional numerical weather prediction (NWP)—atmospheric conditions on a global grid—with outcomes data on extreme weather events like hurricane formation. Unlike NWP methods, AIWP attempts to create a black box numerical model that links the conditions to the weather event. These black boxes are computationally simpler to create, and AIWP software looks at ensembles of, say, 1000 of these models. Weather forecasters had already been using dozens of NWP models in storm prediction centers, but their quantity and look-ahead is limited by the sheer speed and expense of such complex modeling projects. Recently, in these complex areas, AIWP models have pulled ahead of their deterministic NWP counterparts, allowing an additional day or two of accurate warnings, and we can expect meteorologists to rely increasingly on them in such situations. We should expect the points of divergence between NWP systems — which apply the science of fluid dynamics to the weather — and AIWP to drive further scientific discovery and technical specification to improve future NWP models.

Scientists are deeply agnostic about which kinds of tools can be used in their work. But the science itself is recognizing and elevating what is common across disparate situations and developing either working theories or precise models for that behavior that extend into new situations. All science connects a complex reality to human-comprehensible concepts and principles, and it is precisely the relative simplicity of the latter that allows knowledge in one situation to be applied to another. Scientists will continue to outsource this modeling and analytical work, but the idea of an incomprehensible science beyond us may be a mirage.

What’s at stake in the Paris climate talks

November 30, 2015September 10, 2021 Carwil Bjork-JamesLeave a comment

Addressing climate change is one of the most important collective decisions facing us as humans living in 2015. Based on decisions made in the next two weeks, the states of the world will either commit to restrain global climate change to under 2° Celsius above pre-industrial levels, or plan for modest reductions in pollution that still put us on track for 4°C of warming by 2100 (with greater increases beyond that).

Let’s assume you know the importance of this choice in theory, but maybe not in its details. Or even that you knew what the major risks of a 4°C warmer world back when the climate talks were held in Copenhagen, but haven’t updated your knowledge since then. Or that you know, and want something to share with those who don’t. Here are some places to get informed, in way that speaks to the immensity of the risks ahead, relatively fast…

Getting To Four Degrees: This hour-long BBC radio drama puts a typical British family in a world warmed by 2°C and 4°C, while climate experts interrupt and discuss the details of what those world will look like. (original weblink dead; archived here)
“Five possible scenarios for our future climate” (The Guardian, 2009) — Degree by degree summaries of the world at 1°C through 5°C. Begins with our unavoidable future: Most of the world’s corals will die, including the Great Barrier Reef. Glaciers that provide crops for 50m people with fresh water begin to melt.
Turn Down the Heat — A series of reports from the Potsdam Institute and the World Bank spelling out the case for limiting warming to 2°C. Remember, the World Bank technocrats are people who have prioritized market-based growth and resource extraction, but they’re not afraid of the science on global warming. In this series, they’ve brought together detailed analyses of how much worse 4°C is than 2°C. If you’re doubting that controlling climate change matters, read about somewhere you love. Comes in three parts: A global summary released in 2012; and regional details for Sub-Saharan Africa, South East Asia and South Asia, and for Latin America and the Caribbean; the Middle East and North Africa; and Eastern Europe and Central Asia.
Professor Kevin Anderson on what’s under negotiation in Paris.
Understanding key positions of the Least Developed Countries in climate change negotiations: “Warming and associated risks will still be unevenly and unfairly distributed with a global average rise of 2°C, and temperature change will be highest in those regions where particularly vulnerable countries are located. A more ambitious ‘1.5°C pathway’ for limiting global average temperature increase is essential to minimise the risks to Least Developed Countries, Small Island Developing States, and Africa.”
The state of the carbon budget to reach 2°C (Washington Post, November 29).

And when you’re wondering how to feel about all this, read these for some company in the face of stark realities:

Climate change going beyond dangerous – Brutal numbers and tenuous hope (Kevin Anderson, 2012)
When the End of Human Civilization Is Your Day Job (Esquire, July 2015)
Naomi Klein on climate change: ‘I thought it best to write about my own raw terror’ — “The primary reason people look away [from climate change] is that they don’t see a way out and are told that the solutions to climate change involves giving things up. If we can chart a path to post-carbon economy, it will involve gaining a lot of other things.”

And here we go… (How we got here)

April 19, 2010 Carwil Bjork-James3 Comments

The inauguration of the climate summit is happening right now. This summit comes 115 years since ahead-of-his-time scientist Svante Arrhenius proposed that carbon dioxide in the atmosphere could change the climate.By 1970, an observatory in Mauna Loa, Hawaii, had over a decade of observations showing that Co2 in the atmosphere was steadily rising (the annual cycle comes as the seasons change and plants take in and release carbon dioxide, like a global breath). The cause? Burning of fossil fuels and deforestation.

By 1988, the global importance of rising CO2 was clear, and the Intergovernmental Panel on Climate Change was founded to coordinate scientific work on the subject and explain the results to policymakers. The IPCC, and its regular reports are one of the most comprehensive joint science projects ever. Listen to grad student Rachel Pike explain how it works. In 1992,at the Rio Earth Summit, a treaty making process called the UN Framework Convention on Climate Change started to make international agreements to avert dangerous levels of climate change. The industrialized countries pledged to stabilize their emissions of greenhouse gases at 1990 levels by the year 2000. Most lied.

By 1995, the IPCC was ready to conclude that “the balance of evidence suggests a discernible human influence on global climate.” This cautious statement reflects the extreme complexity of the the many factors involved in human impact on the climate, (summarized in this graph). In fact, scientists thought they might not have definitive certainty until 2000, but detailed regional modelling, and the fact that 1970s-era pollution laws reduced the masking effect of soot and other pollution (which briefly cool the Earth before they fall to the ground, leaving the warming greenhouse gases to linger in the atmosphere for centuries) made the issue apparent by the mid-1990s. A continuing series of record-breaking temperature years helped to make the case:

Global temperature record since the 19th century

In 1997, the Clinton-Gore administration pushed hard for a range of “flexibility mechanisms” to be included in the Kyoto Protocol, most of which allowed rich countries to pay poor ones to make reductions for them. Greenpeace and other international NGOs calculated that loopholes in the Kyoto treaty, such as military bunker fuels, transnational plane flights, and “Russian hot air” (the right of Eastern European countries to claim credit for the emissions they weren’t making after the post-Soviet collapse, and sell it to others who were emitting more), amounted to more CO2 than would be reduced in the first place. Despite this weakening, the US failed to ratify Kyoto, saddling the rest of the world with a weaker treaty not accepted by the world’s largest polluter.

Efforts to extend and fix Kyoto continued through annual meetings of the UNFCCC conference of parties (COP), and US efforts to weaken the accord continued even as it ignored the Protocol. Finally, last year, a deadline loomed at COP15 in Copenhagen. The US opted for a second track of negotiations with a handful of highly polluting countries and produced the Copenhagen Accord. The COP as a whole merely “acknowledged” the Accord, and the word “failure” hung in the air. If the UNFCCC process is to be resuscitated, the next chance will be at COP16 in Cancun, Mexico this December.

Bolivia, which played a key role from the environmental side in Copenhagen is hosting this week’s World Peoples’ Summit to give the process a push, and create a space for planning both inside and outside strategies on global environmental issues. I’ll be spending a lot of time in the Strategies for Action working group seeing what participants think might be a Plan B if the international impasse continues.