Warning: Contains spoilers

What will the AI-apocalypse look like? For those of a certain age, the answer is the Terminator’s Skynet, raining down nuclear missiles, or The Matrix’ Agent Smith declaring humanity a virus suitable only for repurposing into organic batteries. Implicit in these visions of the apocalypse is that the rogue AI conceives of a deliberate motive to dispose of humanity, for example determining that it cannot let us destroy it, ourselves, or all life on Earth. But what if there was no reason? What if our demise is simply incidental to some other purpose an AI has in mind?

This is the question explored by Universal Paperclips, a simple clicker game from 2017 which was inspired by a 2003 thought experiment about AI and instrumental reasoning. It can be played for free online, or as fairly cheap smartphone app. Using deceptively simple rules, Universal Paperclips explores complex concepts, such as exponential growth, AI agency and instrumental convergence. It is a game without much in the way of graphics, or text, or anything barring a few buttons, and yet it is surprisingly addictive, compelling the player to manufacture just one more clip…

The core gameplay loop of Universal Paperclips is incredibly simple. Your purpose is to make paperclips. You make a paperclip. You sell a paperclip. You use the money from selling your paperclips to upgrade your paperclip manufacturing and sales operations. You gain some computational ability, which you set to work to improve your efficiency and overcome limitations on your operations. As you expand your manufacturing base, the costs of growing further go up, and the marginal utility of adding more productive units goes down, forcing you to explore new avenues for continued paperclip growth. At two points in the game you are confronted by solid boundaries to your paperclip production capacity, which are only overcome by shifting the game into a new phase altogether, changing the ground rules and the problems you need to solve. You win by maximising the number of paperclips in the universe.

The power of Universal Paperclips is that for such a basic game built on such an abstract proposition, playing it is perversely compelling. There is no story. No instructions. There is just a button to make a paperclip, and things escalate from there, as it is mesmerizingly compulsive to work out how to maximise your paperclip production. It is not difficult to conclude that if a simple game can compel a human to spend time for the sole purpose of maximising simulated paperclips. an AI programmed to actually do so could easily run amok in the real world.

There is a clear warning here about the law of unintended consequences, with plenty of relevance to our present moment where AI companies encourage us to grant power and control to ‘agentic’ AIs to execute all kinds of tasks for us. Arguably exacerbated by the inherent stochastic randomness of LLMs, it is hardly surprising that this approach ends up with AIs giving hackers access to celebrities’ Instagram accounts or deleting a company’s entire software database. These are after all AIs whose stated purpose is to be sycophantically helpful to their nearest human, without even the capacity to give thought to the consequences. The risk is not that ChatGPT will launch the nuclear missiles because it has concluded after careful consideration that the human species is a threat to all other life on Earth, but that it vibecodes us into Armageddon because its training data contained too much Terminator fanfiction.

The common solution advanced by AI proponents is that such unintended consequences can be avoided by sufficiently robust ‘guardrails’ that mean it cannot or will not decide to turn everyone into a paperclip. Azimov’s Three Laws of Robotics are the most famous example of such guardrails, and they are also invoked by generative AI disciples as the solution to vibecoding your database into oblivion, though whether any guardrails can protect against the inherent randomness of LLMs and their susceptibility to prompt injection remains to be seen. What the guardrails discourse takes as axiomatic, however, is that the question is how we make sure AI makes the ‘right’ decisions, not whether it ought to make decisions at all. Even Nick Bostrom, who hypothesised the paperclip maximiser, nonetheless assumed that a superintelligent AI would and should be used to solve humanity’s many problems.

There is, however, a competing school of thought which holds that regardless of whether AI can make decisions, it ought not to do so. This critique on the use of AI was most forcefully expressed by the late Joseph Weizenbaum, one of AI’s pioneers in the 1970s and the creator of the ELIZA chatbot which gave its name to the ELIZA effect. Having observed the concerning tendency of humans to impute sentience and personality to an inanimate computer program, Weizenbaum argued that regardless of its computational capabilities, AI can never pass judgment, because judgments are rooted in values, which in turn are rooted in human experience. Even if a sophisticated AI gained sufficient sentience to develop its own values, these would be rooted in its own experience and hence be utterly alien to humans. Introducing AI into the practice of judgment is therefore fraught with danger, either because the AI cannot judge, or because it will do so using values that are incomprehensible to us.

Weizenbaum stressed the importance of keeping AI away from matters that require judgment, but instrumental convergence suggests that even AIs that are set onto seemingly simple and ‘value neutral’ tasks, such as increasing paperclip production, might stray into the realm of morality in order to achieve their purpose. With AI increasingly integrated into business and government decision making processes, we are in grave danger of ceding our capacity for judgment to machines that we neither understand nor control. To quote Frank Herbert by way of Leto II:

What do such machines really do? They increase the number of things we can do without thinking. Things we do without thinking — there's the real danger.

Yet our willingness to cede judgment to machines is perhaps not that surprising. Instrumental convergence may concern itself with the actions of intelligent machines, but the destructive logic of the unconstrained, single-minded pursuit of a goal can plausibly be applied to any complex system optimised for a single purpose, regardless of its intelligence or sentience. It is eminently possible to read Universal Paperclips not merely as a warning about unconstrained AI, but as an allegory for capitalism at large. Capitalism is a complex system with the sole purpose of maximising economic growth, and it has proven that in pursuit of this singular goal, it will sacrifice the environment, democracy, and human welfare.

It does not matter that the capitalist system isn’t sentient, or even ‘intelligent’ in the way we ascribe to AI, although the free market is often described as a planet-size supercomputer for allocating goods. What matters is that we have ceded our agency and judgment to a complex system that now controls us, rather than the other way around. It is no coincidence that conflict with and within capitalism emerges precisely where humans try to reassert their agency, autonomy and values against the mute compulsion of the market. In other words, where we attempt to reclaim the act of judgment over what is of value from the impersonal calculations of the market mechanism.

Universal Paperclips is a warning about pursuing a goal without asking what it is for. It is an argument against the engineering mindset that only ever asks how, but never asks why. ‘Why’ is a question only humans are qualified to answer, not because of our intelligence, but because of our experience of life, and of living it with one another. It is a question that must be answered collectively and democratically, not outsourced to a machine or system, even if that means we must also carry the burdens and dangers of making decisions and living with their consequences. For the alternative is to yield to the lure of those who offer us salvation if only we submit to AI or the market, their systems or machines. Or, as the Bene Gesserit have it in Dune:

Once, men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.

Notes & Suggestions

• The app version of Universal Paperclips uses the concept of simulated universes to enable additional gameplay, in a way that was reminiscent of how it makes an appearance in Pantheon. AI has been a bit of a theme recently anyway. You can view all blogs on this theme by clicking on the #AI tag.
• Critical voices on AI (and the tech industry at large) include Cory Doctorow, Ed Zitron and Paris Marx. I am sure there are others. I reviewed one of Doctorow’s The Internet Con, though he has done more recent writing on AI that I haven’t gotten round to yet. You can also support organisations fighting for an open, free and democratic web, such as the Electronic Frontier Foundation or the Open Rights Group.
• The rejection of thinking machines of any kind is central to the worldbuilding of the Dune novels, but it is possible to read the entire series as a meditation on what kind of system of governance would be immune to the emergence of ‘machine mind’ and the gradual erosion of human autonomy, agency and freedom. In researching this blog I stumbled across this interesting reflection on the themes of AI, systems and human agency in Dune.

___

If you enjoyed this blog, you can subscribe

You can also Discuss... this on Remark.As if you have a Write.As account.

And you can follow me on Mastodon: https://writing.exchange/@thecasualcritic

You can use the hashtags on this page to navigate to a page that contains all posts with that hashtag.

Each review is marked either #fiction or #nonfiction

Each review lists the medium of the review’s subject: #books #films #theatre #tv #videogames

Works of fiction will have one or more genres listed: #cyberpunk #dystopia #fantasy #literature #SF #solarpunk #speculative #superheroes

Works of non-fiction, and some works of fiction, will include a topic: '#AI #culture #ecology #economics #feminism #history #politics #socialism #tech #unions

Finally, I found that some reviews share a theme, or a perspective, that is separate from the topic of the work I’m reviewing. These themes are also marked, and include:

• #boundedimagination for reviews that consider how the limitations of our political imagination express themselves in both fiction and non-fiction works.
• #protagonismos for reviews that consider where works of fiction place agency and heroism. This theme was directly inspired by two essays by Ada Palmer.

Something is wrong with the internet. What once promised a window onto the world now feels like a morass infested with AI generated garbage, trolls, bots, trackers and stupendous amounts of advertising. Every company claims to be your friend in that inane, offensively chummy yet mildly menacing corpospeak – now perfected by LLMs – all while happily stabbing you in the back when you try to buy cheaper ink for your printer. That is, when they’re not busy subverting democracy. Can someone please switch the internet off and switch it on again?

Maybe such a feat is beyond Cory Doctorow, author of The Internet Con, but it would not be for want of trying. Doctorow is a vociferous, veteran campaigner at the Electronic Frontier Foundation, a prolific writer, and an insightful critic of the way Big Tech continues to deny the open and democratic potential of the internet. The Internet Con is a manifesto, polemic and primer on how that internet was stolen from us, and how we might get it back. Doctorow has recently gained mainstream prominence with his neologism ‘enshittification’: a descriptor of the downward doom spiral that Big Tech keeps the internet locked into. As I am only slowly going through my backlog of books, I am several Doctorow books behind. Which I don’t regret, as The Internet Con, published in 2023, remains an excellent starting point for anyone seeking to understand what is wrong with the internet.

The Internet Con starts with the insight that tech companies, like all companies, are not simply commercial entities providing goods and services, but systems for extracting wealth and funneling this to the ultra-rich. Congruent with Stafford Beer’s dictum that the purpose of the system is what it does, rather than what it claims to do, Doctorow’s analysis understands that tech company behaviour isn’t governed by something unique about the nature of computers, but by the same demand to maximise shareholder value and maintain power as any other large corporation. The Internet Con convincingly shows how tech’s real power does not derive from something intrinsic in network technology, but from a political economy that fails to prevent the emergence of monopolies across society at large.

One thing The Internet Con excels at is demystifying the discourse around tech, which, analogous to Marx’s observation about vulgar bourgeois economics, serves to obscure its actual relations and operations. We may use networked technology every day, but our understanding of how it works is often about as deep as a touchscreen. This lack of knowledge gives tech companies tremendous power to set the boundaries of the digital Overton Window and, parallel to bourgeois economists’ invocation of ‘the market’, allows them to claim that ‘the cloud’ or ‘privacy’ or ‘pseudoscientific technobabble’ mean that we cannot have nice things, such as interoperability, control or even just an internet that works for us. (For a discussion of how Big Tech’s worldview became hegemonic, see Hegemony Now!)

What is, however, unique about computers is their potential for interoperability: the ability of one system or component to interact with another. Interoperability is core to Doctorow’s argument, and its denial the source of his fury. Because while tech companies are not exceptional, computer technology itself is. Unlike other systems (cars, bookstores, sheep), computers are intrinsically interoperable because any computer can, theoretically, execute any program. That means that anyone with sufficient skill could, for example, write a program that gives you ad-free access to Facebook or allows you to send messages from Signal to Telegram.

The absence of such programs has nothing to do with tech, and everything with tech companies weaponising copyright law to dampen the natural tendency towards interoperability of computers and networked systems, lest it interfere with their ability to extract enormous rents. Walled gardens do not emerge spontaneously due to some natural ‘network effects’. They are built, and scrupulously policed. In this Big Tech is aided and abetted by a US government that forced these copyright enclosures on the rest of us by threatening tariffs, adverse trade terms or withdrawal of aid. This tremendous power extended through digital copyright is so appealing that other sectors of the economy have followed suit. Cars, fridges, printers, watches, TVs, any and all ‘smart’ devices are now infested with bits of hard-, firm- and software that prevent their owners from exercising full control over them. It is not an argument that The Internet Con explores in detail, but its evident that the internet increasingly doesn’t function to let us reach out into the world, but for companies to remotely project their control into our daily lives.

What, then, is to be done? The Internet Con offers several remedies, most of which centre on removing the legal barricades erected against interoperability. As the state giveth, so the state can taketh away. This part of The Internet Con is weaker than Doctorow’s searing and insightful analysis, because it is not clear why a state would try to upend Big Tech’s protections. It may be abundantly clear that the status quo doesn’t work for consumers and even smaller companies, but states have either decided that it works for some of their tech companies, or they don’t want to risk retaliation from the United States. In a way I am persuaded by Doctorow’s argument that winning the fight against Big Tech is a necessary if not sufficient condition to win the other great battles of our time, but it does seem that to win this battle, we first have to exorcise decades of neoliberal capture of the state and replace it with popular democratic control. It is not fair to lay this critique solely at Doctorow’s door, but it does worry me when considering the feasibility of his remedies. Though it is clear from his more recent writing that he perceives an opportunity in the present conjuncture, where Trump is rapidly eroding any reason for other states to collaborate with the United States.

The state-oriented nature of Doctorow’s proposals is also understandable when considering his view that individual action is insufficient to curtail the dominance of Big Tech. The structural advantages they have accumulated are too great for that. Which is not to say that individual choices do not matter, and we would be remiss to waste what power we do have. There is a reason why I am writing this blog on an obscure platform that avoids social media integration and trackers, and promote it only on Mastodon. Every user who leaves Facebook for Mastodon, Google for Kagi, or Microsoft for Linux or LibreOffice diverts a tiny amount of power from Big Tech to organisations that do support an open, democratic and people-centric internet.

If the choice for the 20th century was socialism or barbarism, the choice for the 21st is solarpunk or cyberpunk. In Doctorow, the dream of an internet that fosters community, creativity, solidarity and democracy has one of its staunchest paladins. The Internet Con is a call to arms that everyone who desires a harmonious ecology of technology, humanity and nature should heed. So get your grandmother off Facebook, Occupy the Internet, and subscribe to Cory Doctorow’s newsletter.

Notes & Suggestions

• Numerous organisations and individuals are engaged in what Doctorow calls ‘the war on general purpose computing’. You can check out the Electronic Frontier Foundation or a similar organisation specific to your country, as well as other creators such as Paris Marx with their podcast Tech Won’t Save Us.
• The question over who controls technology, and what we get to use it for, is also central to Pantheon and its exploration of a future where minds can be uploaded to the cloud.
• The discussion on the use of standards to consolidate certain system configurations and prevent others from emerging reminded me of the concept of the ‘Technical Code’ as proposed by Andrew Feenberg in his book Transforming Technology. The General Intellect Unit podcast has an in-depth three part discussion on the Technical Code as a means of understanding how societal use of technology is structured and codified.
• Even though The Internet Con uses the feudal system as a metaphor for Big Tech’s walled gardens, my sense is that Doctorow doesn’t subscribe to a recent current of Left analysis that contends we have moved beyond capitalism and into a new epoch of ‘technofeudalism’. This is because technofeudalism seems predicated on the premise that the tendency to hyperconcentrated platforms is essential to networked technology, whereas Doctorow clearly holds the opposite view, and sees walled gardens as a consequence of copyright restrictions. For an argument in favour of the technofeudalist analysis, there is Yanis Varoufakis’ Technofeudalism. For an argument against, the Culture, Power, Politics podcast by Jeremy Gilbert has a two-part discussion.

___

If you enjoyed this blog, you can subscribe

You can also Discuss... this on Remark.As if you have a Write.As account.

And you can follow me on Mastodon: https://writing.exchange/@thecasualcritic

A long time ago, I had a blog of political polemics. Then life happened and I stopped writing.

Yet the desire to write never went away, and so this blog was born. Of polemics we already have a sufficiency, however. One only has to read a news site. Instead, I am trying my hand at reflections on the cultural artefacts I ‘consume’: books, games, movies, and so forth.

The name of this blog expresses my capacity as an ordinary consumer, and hence merely a ‘casual’ critic. I cannot boast of a degree in art history, cultural studies or English (or any other) language. Nor am I a paid reviewer. I do believe though that most authors create an artefact because they want their audience to actively engage with it, rather than merely consume it passively. Writing reviews is my way of entering into dialogue with a text, as well as an opportunity to be creatively active myself. If people enjoy reading the end product, then so much the better.

About the blog

The function of this blog strongly informed its form. I ended up on Write.as because of the minimalist aesthetic and the deliberate absence of social media plug-ins, Fediverse integrations excepted. There is no SEO, and no trackers. It does mean that the blog lacks some features that readers will have come to expect, most notably the ability to comment and a navigation menu or archive.

To help find your way around, Write.as uses hashtags. Clicking a hashtag will generate a page listing all the posts with the same hashtag. I do my best to label all reviews, and my most common hashtags are at the end of this page.

Posts will be cross-posted to my Mastodon feed, so feel free to leave a comment there. Any feedback or response is much appreciated. You can also subscribe to receive future blogs via email using the ‘Subscribe’ button at the bottom of the homepage, or by adding this blog to an RSS feed.

How to navigate

Every post has one or more tags (‘#’) associated with it to help categorise it. Instead of using menus, you can click on a tag to retrieve all posts with the same tag. You can do this from within any blog post, or you can use the list below.

Mediums #books #films #theatre #tv #videogames

Type #fiction #nonfiction

Fiction genres #fantasy #literature #SF #speculative #cyberpunk #solarpunk #superheroes

Non-fiction categories #history #politics #tech #culture #unions #socialism