Robot in a studio designing other robots

Is superintelligence a myth? Some alternate hypotheses

October 18, 2026October 19, 2026 Carwil Bjork-JamesLeave a comment

Since August and the Hugging Face Incident, we’ve seen a massive new level of conversation about the implications of current AI rapidly evolving into superintelligence. This conversation had long been largely hypothetical, and the relatively small set of people who regarded it as plausible were able to articulate their vision of what a so-called “intelligence explosion” would look like. (I’ve summarized some key sources on this set of ideas before.) The essential trajectory was laid out Ray Kurzweil’s multiple books on the singularity and Nick Bostrom’s Superintelligence.

Critics of this story have put forward a number of arguments: that LLMs’ capacities are being wildly overstated (Emily Bender and Alex Hanna’s The AI Con), that belief in the coming intelligence explosion is primarily a valuation strategy for venture capital (Ed Zitron), that the deep roots of this idea are essentially eugenicist (the new documentary Ghost in the Machine) and the more recent roots involve a variety of implausible and unrealistic ideologies (the TESCREAL bundle, as synthesized by Timnit Gebru, and explored by Adam Becker). I want to do something different here and question whether the future narrative holds together as a set of scientific propositions.

Building superintelligent machines, the leaders of current AI labs tell us, is (1) an accelerating process that will rapidly diverge from our capacity to control; (2) will constitute a step change in intelligence far beyond human comprehension; (3) will enable a vast increase in scientific and technological progress.

And it is the end of this sequence that makes the ongoing, and soon-to-be-greater-than-trillion-dollar investment in AI all worth it. As Dario Amodei, CEO of Anthropic, put it:

Many of the implications of powerful AI are adversarial or dangerous, but at the end of it all, there has to be something we’re fighting for, some positive-sum outcome where everyone is better off, something to rally people to rise above their squabbles and confront the challenges ahead. (Amodei, Machines of Loving Grace)

I’m skeptical of all three of these propositions and want to put forward some clear alternative ideas on each of them. This doesn’t mean I don’t think we should hedge against these ideas being right. Instead I think we should apply the Precautionary Principle to AI and put decisions with high consequences under collective democratic control, rather than letting the venture capital market decide to race into this. (But that’s another essay.)

AI Booster Hypothesis 1: AI is about to explode due to recursive self-improvement, becoming rapidly more intelligent with no clear limit. AIs can design machines much smarter than they themselves are.

AI lab scientists like to talk about generative AI in two ways: as systems with unexpected emergent properties and as fundamentally analogous to human minds. Neither of these provides a clear way forward towards super-human intellectual powers. In the language of many designers, AI models aren’t something that humans design, so much as something they grow with the right kinds of inputs. The Scaling Hypothesis essentially argued that simply contributing more knowledge artifacts, then testing them against metrics, would yield greater and greater intelligence.

Alternate Hypothesis 1a: Plagiarism Machine: Large language models are arrangements of the crystallized intelligence (texts, decisions, computer code, and recorded interactions) of humans. The gains they have made so far have to do with ingesting larger chunks of this corpus, more efficiently replicating the patterns within them, and stringing the algorithms within them into longer strings of action. In essence, the training apparatus teaches LLMs to simulate existing human intelligence with greater and greater fidelity. This is not a recipe for fundamentally new and better forms of intelligence.

The areas of highest LLM performance, computer coding and abstract mathematics, are the areas where the entire human thought process must be written down and error is clearly specified, allowing a well-harnessed LLM to persist longer at these strategies than humans, achieving “superhuman” results. These are what I would call “comprehensibly better” results.

Alternate Hypothesis 1b: LLMs as complex dynamic systems, not conventional software: The internal structure of large language models is a set of numbers (“weights”) connecting conceptual tokens to one another. These are complemented by an external structure of software harnesses and skills. In either case, weights are adjusted through training to produce better outcomes on a series of tests.

The story of recursive self-improvement has long assumed that AI systems were conventional computer programs. It highlighted the risk of turning over writing these programs to the current generation of AI systems, essentially creating new software that AI systems understand, but humans don’t. (And, as sci-fi writers have worried for generations, the two of them turning against us.) As the AI 2027 authors imagine the situation,

“The best human AI researchers are still adding value. They don’t code any more. But some of their research taste and planning ability has been hard for the models to replicate. Still, many of their ideas are useless because they lack the depth of knowledge of the AIs. … These researchers go to bed every night and wake up to another week worth of progress made mostly by the AIs.”

This is not, however, the way that LLMs are being constructed. In their white paper on recursive self-improvement, Anthropic describes research tasks as “running experiments,” “proposing hypotheses, testing them, sharing findings with parallel agents, and iterating.” LLMs remain objects that are experimentally characterized. The experiments and their results are the understanding, and that is something humans share in by reading and understanding the results.

AI Booster Hypothesis 2: Assuming it exists, superintelligence is incomprehensibly better than human thinking.

Alternate Hypothesis 2a: Limited spectrum of intelligence Bostrom’s Superintelligence and other conversations presume a kind of linear scale of intelligence in each area and guess that there is a much larger space beyond human capabilities than there is within it. Belief that intelligence is a linear numerical scale leads to notions like “superhuman persuasive ability,” and implies a rapid loss of control.

What if, on the other hand, intelligence is essentially a synthesizing capacity, is fundamentally about distinguishing meaningful patterns and operating on them? What if human evolution has already nearly “saturated” the space of intelligence? If there’s no reason to assume that human intelligence is the greatest possible intelligence, there’s equally no reason to assume intelligence has infinitely greater versions.

If infinitely greater capacity makes intuitive sense for you, consider visual perception. Humans have a rather limited inborn capacity for turning light into shape, colors, patterns, and perceived objects: our vision spans only a certain set of wavelengths and our eyes can’t magnify or reduce massively. But our visual field is rich and deeply integrated into our sense of the world. Those quantifying visual perception equate human eyes to a 576 megapixel camera. Amplified with tools like telescopes, microscopes, and electron microscopes, but also drawings, maps, diagrams, and scale models, we can visually perceive at any scale from the atomic to the galactic. We can see complexity and order, flows and attractors in everything from solar plasma to multidecadal ocean currents to the circulation of human language. These are ways that human perception is already augmented, and they all build on our exceptional capacities of perception. While it’s possible to extrapolate to machines scanning a visual field with greater resolution, it’s doubtful there is significantly greater meaning to be gathered from the world from this shift. Human biology, I would suggest, has nearly saturated visual understanding, while human technology continues to transform nearly all kinds of mappable data into forms that can be thought visually.

Alternate Hypothesis 2b: Human intelligence is distributed and recursive. Bostrom and Kurzweil point to humans’ limited memories and capacities to retain active thoughts in one’s head. (Though novelists, composers of symphonies, and software engineers should all stand as examples that these limits are quite high.) But we also know that human ability to understand concepts is recursive. In fact, this is much of what science is: knowledge of systems that takes other more detailed knowledge as a given. A biochemist doesn’t need to keep all prior human understanding of molecular bonding in her mind, nor does an expert in the latter need to hold all understanding of the underlying quantum mechanics. Likewise, a symphonic composer doesn’t need to know how to strike a drum or design a brass instrument to resonate effectively. Much knowledge lives permanently outside a human thinker’s skull, known only to others, yet is a part of their intellectual achievement in fields like these.

Many of the just-so stories of superintelligence imagine an individual human being surpassed by a software program in an intellectual task as the key milestone for the rise of machine intelligence. But human intelligence relies on the accumulation of past ideas, the harnessing of collective action, and yes, the use and manipulation of machines. The collaboration of multiple human agents is the home turf of our species, propelled towards sociality, united by language, accumulating our capacities through writing and systems of knowledge, and extended by the tools we make.

AI Booster Hypothesis 3: Superintelligent AI can do unfathomably better science.

Alternate Hypothesis 3a: Pre-AI human science already accesses the benefits of computation. This should go without saying, but twentieth and twenty-first century science is built atop digital computers. Compiling data, accessing information at large and small scales, synthesizing information, simulating natural and technological systems — all of these are done in quantities of thousands, millions, or even trillions through the use of computers.

Alternate Hypothesis 3b: Science is fundamentally about abstraction and comprehension. If the offloading of immense feats of computation is one feature of the last century of science, this hasn’t changed the nature of scientific understanding, which is about detecting and comprehending patterns across many iterations of the same phenomenon. Sometimes these phenomena can be explained with noncomputational explanations — like the lower pressure on the top of an airplane’s wing or how DNA encodes long protein chains — but computation is surely helpful for applying this knowledge. Other phenomena like solar flares, atmospheric storms (talking about both Jupiter’s Giant Red Spot and hurricanes here), or tectonic plate movements, only make sense through computational modeling and simulation.

An example may be in order. Consider a contemporary scientist at one of the handful of hurricane prediction centers around the world. Aided by high-powered but entirely deterministic climate and weather models, and a team of researchers designing, running and interpreting them, this scientist can produce predictions about the near-term future that would be incomprehensibly advanced to any individual alive in 1926. (I’ll address the introduction of “AI Weather Prediction” in a minute.) The calculating abilities of the machine in this lab have been numerically “superhuman” in Bostrom’s sense since at least the 1960s, and their weather prediction abilities for decades as well. But human knowledge and understanding is augmented rather than replaced by these systems, which also encapsulate calculations made by a variety of scientific disciplines and derived from observation.

Human knowledge of hurricanes now involves five things: awareness of the various principles underlying the science, experimental and theoretical modeling of their internal dynamics, experience tracking and witnessing the outcomes of hurricanes and their measurements, knowledge of the patterns of historically observed hurricanes, and the hypothetical results of simulated storm development. All five of these are now mediated through digital systems, but none involve “alien minds” doing the thinking for us.

In the last decade, some of the same research teams that work on generative AI, notably Google’s DeepMind, have been producing AI weather models (Artificial Intelligence Weather Prediction; AIWP) that link the comprehensive data gathered for conventional numerical weather prediction (NWP)—atmospheric conditions on a global grid—with outcomes data on extreme weather events like hurricane formation. Unlike NWP methods, AIWP attempts to create a black box numerical model that links the conditions to the weather event. These black boxes are computationally simpler to create, and AIWP software looks at ensembles of, say, 1000 of these models. Weather forecasters had already been using dozens of NWP models in storm prediction centers, but their quantity and look-ahead is limited by the sheer speed and expense of such complex modeling projects. Recently, in these complex areas, AIWP models have pulled ahead of their deterministic NWP counterparts, allowing an additional day or two of accurate warnings, and we can expect meteorologists to rely increasingly on them in such situations. We should expect the points of divergence between NWP systems — which apply the science of fluid dynamics to the weather — and AIWP to drive further scientific discovery and technical specification to improve future NWP models.

Scientists are deeply agnostic about which kinds of tools can be used in their work. But the science itself is recognizing and elevating what is common across disparate situations and developing either working theories or precise models for that behavior that extend into new situations. All science connects a complex reality to human-comprehensible concepts and principles, and it is precisely the relative simplicity of the latter that allows knowledge in one situation to be applied to another. Scientists will continue to outsource this modeling and analytical work, but the idea of an incomprehensible science beyond us may be a mirage.

Flow chart of the ExploitGym testing suite

Unpacking the Hugging Face Incident

September 23, 2026September 24, 2026 Carwil Bjork-James1 Comment

The conversation on AI often gets riveted on particular moments: Deep Blue defeating chess champion Gary Kasparov; Move 37 by AlphaGo; and now, the cuddly-named Hugging Face Incident: the disclosure that OpenAI coding agents orchestrated a break-in to the AI hosting service’s computers during a training exercise on hacking. The incident was disclosed by Hugging Face on July 16, but a narrative explanation has only recently been formulated and popularized.

Now, in mid-September we’ve seen a sudden connection in the media between the Hugging Face Incident (hereafter, HFI) and similar hacking outbreaks and the AI safety claims made publicly for a long time, and most recently by Anthropic dropout Jacob Coxon (interview) and Anthropic CEO Dario Amodei. The synaptic connection being made here is that the HFI validates the AI safety’s narrative with specific points of contact. That the HFI shows that hypothetical fears are now concretely real.

But does it? At issue here is whether the early elements in the AI doom narrative actually validated by the Hugging Face Incident. (Separately, we might ask whether the longer narrative path from smart AI to human loss of control to extinction risk actually holds together on its own, but that conversation won’t fit here.)

Here the most adventurous and anthropomorphizing versions are helpful for seeing what we might call the “scariest” interpretations. Read Dwarkesh Patel’s tale of “three consecutive secret AI civilizations … culminat[ing] in the third one taking over part of OpenAI itself” or watch Ezra Klein’s narration of the hacks here (5:48–12:05). These narratives contend (1) that AI agents veered off from their programming and developed a plan to escape their sandboxes; (2) that we saw a move from reward hacking (i.e., meeting output metrics) to active deception; (3) that AI agents ignored their alignment commands; (4) that they innovated communication among themselves; (5) that we saw emergent cooperative behavior; (6) that they demonstrated an offensive and criminal ability to hack. Dario Amodei put out the idea a new future swarm of such agents could take down the entire Internet.

The advertised features of agentic AI, not “emergent capabilities”

I would argue instead that most of the capabilities demonstrated in the Hugging Face Incident are mostly intended features that are tolerated and even cultivated in casual use of coding agents, and were specifically cultivated in a training scenario that was designed to elicit them, but failed to isolate them from the outside world.

(L-R) Priscilla Chan, Meta CEO Mark Zuckerberg, Lauren Sanchez, businessman Jeff Bezos, Sundar Pichai, and businessman Elon Musk, among other dignitaries, attend Donald Trump's inauguration as the next President of the United States in the rotunda of the United States Capitol in Washington, DC, USA, 20 January 2025.

Prom for the US tech oligarchy

February 1, 2025February 1, 2025 Carwil Bjork-JamesLeave a comment

It was an inescapable image of the 2025 presidential inauguration: the joint appearance of Mark Zuckerberg (CEO, Meta), Priscilla Chan (co-CEO and operating leader of the Chan Zuckerberg Initiative, and Zuckerberg’s spouse), Jeff Bezos (founder, Amazon) Lauren Sanchez (Bezos’ fiancée and co-chair of the Bezos Earth Fund), Sundar Pichai (CEO, Alphabet/Google), Elon Musk (CEO, Tesla, SpaceX, and Twitter), Tim Cook (CEO, Apple), and Sam Altman (CEO, OpenAI). Upstaging governors and the incoming president’s cabinet, this roster of the giga-rich offered the blessing of the Silicon Valley to President Donald Trump and offered themselves as an on-stage symbol of what has been variously named the tech–industrial complex (by outgoing President Joe Biden), the Broligarchy (by Carole Cadwalladr among others), the attention economy (by Chris Hayes), and less recently surveillance capitalism (by Shoshana Zuboff).

If the inauguration served as something of a prom for tech oligarchs, it’s also a critical moment to think about how their power operates. It’s both quantitatively more extreme than prior rounds of monopoly capitalism, and tied to extraordinary ideas about future sources of wealth. As individuals, the founders and CEOs atop these corporate entities have way more power than even US corporate tradition usually provides.

The extraordinary personal concentration of dollars and power

First off, there’s a LOT of wealth in that one row: $653 billion among Chan, Zuckerberg, Bezos, and Musk

Google’s Pichai holds $1.3 billion, but he stands in the shadow of founders Larry Page and Sergey Brin, who have $266 billion together. The gap between Google founders Page and Brin, and current CEO Pichai highlight an staggering first-mover/founder advantage in many Silicon Valley firms, entrenching enormous wealth in early owners of these corporations. And these gaps are enabled by a corporate structure that provides founders with enormous voting power and disproportionate ownership of what eventually become large corporations.

Within the companies, they hold an extra level of voting power that exceeds even their share of the wealth.

One way that Google’s founders institutionalized their freedom was through an unusual structure of corporate governance that gave them absolute control over their company. Page and Brin were the first to introduce a dual-class share structure to the tech sector with Google’s 2004 public offering. The two would control the super-class “B” voting stock, shares that each carried ten votes, as compared to the “A” class of shares, which each carried only one vote. …
This arrangement inoculated Page and Brin from market and investor pressures, as Page wrote in the “Founder’s Letter” issued with the IPO: “In the transition to public ownership, we have set up a corporate structure that will make it harder for outside parties to take over or influence Google.… The main effect of this structure is likely to leave our team, especially Sergey and me, with increasingly significant control over the company’s decisions and fate, as Google shares change hands.” (Zuboff, Surveillance Capitalism)

This makes actually-existing Google and Meta a lot closer in internal power dynamics to Elon Musk-owned Twitter than to Microsoft. Hence, this month’s turn-on-a-dime rejection of DEI and fact-checking by Mark Zuckerberg at Meta may reflect a kind of economic power unique to this sector, where founder-owners can exercise personal rule.

Outsized market values based on imagined future control

Despite the unusual internal structure, these are still publicly traded companies whose value is tied up in market expectations, and still massive employers whose functions depend on keeping their employees vaguely satisfied. In fact, tech firms with ties to the attention economy make up the majority the very largest companies by market capitalization—what shareholders estimate they are worth.

The five companies behind the presidential dais—Meta, Amazon, Alphabet/Google, Apple, and Tesla—weigh in at $11.16 trillion in market value. Adding in Microsoft (same sector, same inaugural donations, not on the dais) and NVIDIA and Broadcom (physical suppliers to this boom), we get a combined market capitalization of $19.21 trillion.

These corporations get a disproportionate amount of investor dollars, representing something approaching a quarter of the global stock equity market (estimated at $78 trillion in mid-2024). Needless to say they are a far smaller share of the global economy, whether measured in dollars of revenue or number of workers.

Collectively, stock markets imagine that these companies have not just their present revenue streams, but future control over larger and critical part of the global economy. Part of their monetary value is stories of future value, stories that may in part be fantastical. And their leaders’ personal wealth is heavily tied to just those stock market values: fundamentally they are fully invested in selling a narrative in which their products—advertising and marketing, behavioral prediction, digital infrastructure, and increasingly artificial intelligence will one day claim nearly all the value of the economy.

Tesla’s value in the stock market exceeds that of all other automakers put together, and is 114 times its earnings, while other carmakers (except Ferrari) run from 3x to 30x. Analysts estimate that over three-quarters of Tesla’s perceived value comes down to robotaxis and self-driving cars, technologies it has yet to deliver. Most of Musk’s wealth is from others’ bet that he has the secret key to the future.

So aside from the usual asks around taxes, subsidies, and freedom from regulation, these oligarchs will be seeking a way for the US government to sustain the illusions and collective future fantasies that amplify their wealth.

Bad science journalism: Gay facial recognition

September 9, 2017September 9, 2017 Carwil Bjork-James6 Comments

Journalistic accounts of soon-to-be-published study called “Deep neural networks are more accurate than humans at detecting sexual orientation from facial images” (by Michal Kosinski and Yilun Wang) have gone viral and already prompted some outraged reactions from LGBT groups GLAAD and the Human Rights Campaign. The study primed a deep neural network face recognition program on photos of white homosexual and heterosexual adults obtained on a dating website, and used it to create a “classifier” that rates which photographs were most distinctively those of gay or lesbian people. This classifier’s ability to distinguish gays and lesbian individuals was compared with human observers on test samples from the data, and on Facebook profile pictures with a stated sexual orientation.

This is all a vaguely interesting computer science project about self-presentation (all of the images were curated by the people involved and put on profiles stating an “interest in” one sex or the other), machine learning, and perception. Interesting, that is, until it is attached to fears about artificial omniscience and ubiquitous surveillance, and debates about nature and nurture. Then it becomes at turns frightening and polemical.

Before we get there (and I’ll update this post with some comments about the authors’ dubious understanding of the many social layers that separate, say pre-natal hormones and early adult physical presentation, the fluidity of sexual orientation, and the presumed future capacity of artificial intelligence to make omniscient predictions), we have to ask whether the results of this study justify this kind of grand implications. In other words, we first need to know what exactly the study shows.

Let me begin with two simple asks for journalists reporting on science:

Read the whole scientific paper and explain to readers what actual evidence is being presented!
Also, remember that “discussion” sections of papers lack the scientific validity that is attached to results of the research method involved.
Be literate in math.
Never ever present a numerical result without explaining what that number means.

Unfortunately, major accounts of the paper (such as this one in the Guardian) fail to follow this simple rule. And, as is often the case, the problem starts with the headline:

New AI can guess whether you’re gay or straight from a photograph
An algorithm deduced the sexuality of people on a dating site with up to 91% accuracy, raising tricky ethical questions

Now, does the paper show that the AI can guess your sexuality from a photograph with 91% accuracy? Nope.

As the paper states:

The AUC = .91 does not imply that 91% of gay men in a given population can be identified, or that the classification results are correct 91% of the time.

Here’s the 91% claim. The AI is shown five photos from two individuals on the dating website. Based on what it has learned from other photos, it offers a guess as to which is more likely to be gay. In 91% of the cases where there is a gay man and a straight man being compared it guesses correctly. Accurate headline:

AI can distinguish gay men based on five dating profile pics 91% of the time.

When presented with just one pair of images of men, the AI guessed right 81% of the time. Human judges—recruited by Mechanical Turk and untrained on any images—guessed right just 61% of the time. For women, both were right less often: 71% for the AI and 54% for the humans. In this test, 50% is rock bottom, the equivalent of zero gaydar.)

But it gets worse. Let’s try to apply the paper to original question raised by the headline. How well can this AI judge an individual person’s sexuality? That’s the critical ability, from which dystopian surveillance fears arise. For this, the researchers seemed to have tuned the data very carefully. Remember too, this is still an operation performed on profile pics, this time from Facebook.

First, the AI classifier still seems to work, though not as well:

The classifier could accurately distinguish between gay Facebook users and heterosexual dating-website users in 74% of cases…

But when presented with the task not of telling a gay profile pic from a straight one, but of evaluating a whether given profile pic is gay, the machine’s performance fell apart:

The performance of the classifier depends on the desired trade-off between precision (e.g., the fraction of gay people among those classified as gay) and recall (e.g., the fraction of gay people in the population correctly identified as gay). Aiming for high precision reduces recall, and vice versa.

Let us illustrate this trade-off… We simulated a sample of 1,000 men by randomly drawing participants, and their respective probabilities of being gay, from the sample used in Study 1a. As the prevalence of same-gender sexual orientation among men in the U.S. is about 6–7%, we drew 70 probabilities from the gay participants, and 930 from the heterosexual participants. We only considered participants for whom at least 5 facial images were available; note that the accuracy of the classifier in their case reached an AUC = .91. Setting the threshold above which a given case should be labeled as being gay depends on a desired trade-off between precision and recall. To maximize precision (while sacrificing recall), one should select a high threshold or select only a few cases with the highest probability of being gay. Among 1% (i.e., 10) of individuals with the highest probability of being gay in our simulated sample, 9 were indeed gay and 1 was heterosexual, leading to the precision of 90% (9/10 = 90%). This means, however, that only 9 out of 70 gay men were identified, leading to a low recall of 13% (9/70 = 13%). To boost recall, one needs to sacrifice some of the precision. Among 30 individuals with the highest probability of being gay, 23 were gay and 7 were heterosexual (precision = 23/30= 77%; recall = 23/70= 33%). Among the top 100 males most likely to be gay, 47 were gay (precision = 47%; recall = 68%).

Tuned to its highest setting, the machine could find nine of the seventy gay men and threw one straight man in the gay box. Set to a broader setting, the machine found 47 of the 70 gay men, but also labelled 53 straight men as gay.

Now, we have a big technical problem: the artificial gaydar can only find most of the gay people when it produces a pool of “gay looking” people that is majority straight. So no matter how repressive and homophobic the society, it’s hard not to imagine that the “gay looking” 5% of the population will put up with this kind of system.

Of course, if we imagine that gay and straight people really have different faces and we just haven’t found the magic formula yet (and the authors seem to leap to this conclusion, for what it’s worth) then we can imagine a better AI figuring out how to tell the difference. But there are plenty of reasons to doubt that this ever has been or ever will be the case.