Content Warning: if you believe in the Singularity and information hazards, stop here. Discussion of Roko’s Basilisk and a related idea follow.

The Metaverse is no joke. Investment in the technology ranks among the billions per year, and Facebook recently rebranded its parent company as Meta. Serious people are spending serious money to develop this technology. Yet at the same time, it’s hard to understand why. When you speak with Metaverse enthusiasts, they’ll tell you about potential applications, ranging from rethinking remote working and education, to providing persistent digital worlds to reshape entertainment.

None of this is really correct. The level of investment doesn’t match the potential value. If Mark Zuckerberg wanted to compete against Zoom, he probably wouldn’t need $15B a year to do so. Zoom’s market capitalization (~$25B as of today) is a fraction of Meta’s. There’s no visible consumer demand for VR meetings. Creating 3D, high-resolution digital environments is very expensive. There’s a reason it hasn’t caught on as a serious way to augment education.

To understand the Metaverse, one needs to incorporate two different, but related perspectives. The first is how the Metaverse is often mentioned in the same breath as “web3;” namely, blockchain, cryptocurrency, and NFTs. One could also add deep fakes and generative AI, such as Stable Diffusion, DALL-E, or GPT-3, to the discussion.

The second perspective is Singularity theory, which posits that at some point in the future, technology will become sufficiently powerful that superintelligent AI will arise, and that humans will be able to interface with computers to the point that neural interaction and digital consciousness are possible. In other words, futurists hypothesize, it will be possible to upload human consciousness into a machine, and that humans will live on in digital universes, even after their corporal forms have passed.

This is a deep and radical idea, once that is perhaps irresponsible to introduce so casually. To determine whether this is a genuine possibility requires a philosophical deep dive that attempts to definitively answer some of the greatest unanswerable questions of humankind. I don’t believe we presently have the theological, philosophical, biological, or psychological knowledge to definitively conclude whether this outcome is real or simply fantasy.

But it is not necessary for us to evaluate this possibility within these frameworks, because regardless of what we know, believe, or can determine, what is definitely true is that there are a number of very serious futurists who truly believe that not only is this outcome possible, it is inevitable, and it is inevitable within their lifetime. It may be that this is nothing but dorm room philosophy, and I personally believe that it is, but it’s a dorm room philosophy that a lot of people with a lot of millions of dollars have chosen to invest in.

A number of futurist movements have developed by taking the Singularity as axiomatic. Movements like longtermism and some forms of effective altruism posit that the number of unborn future humans dwarfs the number of humans who have ever lived, and that by extension, our social responsibility lies with ensuring the survival of the trillions of humans to be. Setting aside many of the deeply troubling present-day implications of this school of thought, and the biases that leak from many of these thinkers, almost surely the only way that this viewpoint can be validated is if humanity learns to live on in digital spheres, spreading itself across the galaxy and harnessing the power of the stars. It does not matter that this technology is not currently possible. It matters that people with influence and money believe it is their responsibility to make it possible.

In this exploration, I admit a couple of hypotheses¹ without any assumption that they are correct:

1. The mapping of human consciousness into computer systems will happen;
2. This mapping will be based on data and AI and can be independent of a physical brain;
3. Technology will continue to follow its exponential growth of computing capabilities without major impediments;
4. The consumer adoption of the Metaverse will be sufficiently compelling to survive the Metaverse through its initial phases.

We suppose that the Singularity occurs, and that the Singularity involves developing the capability not only for neural interfaces with the computer, but the simulation of human consciousness to some fidelity in a digital space. In other words, this is an AI, but it’s an AI built off of the very real experiences and neural pathways pertaining to specific human individuals. Such a model may involve local adaptation of more generic models trained on larger datasets, but each digital consciousness is ultimately the digital representation of exactly one person. In other words, we as humans are individuals, but we also develop behaviors relative to our environments–we inherit the mannerisms of our parents and our cultures, we adapt to fit in with friends and other peer groups–and therefore can be seen both as individuals and as representatives of a culture. We can infer that given a situation, most people will respond in a broadly socially-acceptable and therefore predictable way, but the specifics of that response will differ based on individual factors. Modeling digital consciousness would be no different. Machine learning techniques today use approaches like this all the time.

The implications of creating a digital consciousness also imply the idea of a digital afterlife. Actually, the idea of a digital afterlife exists today. When you pass, your Facebook and Twitter accounts remain open and people can go relive your timelines and see how you reacted to the news, to events in your life, and to friends and family. With a digital consciousness, however, your future digital afterlife would no longer be static. The digital “you” could still chat with friends, could still react to the news. If your consciousness can live on in a computer, then the physical death of your corporeal form only means that you no longer occupy a place in the physical universe. Instead, you become an entity existing in the Metaverse.

This idea doesn’t represent a particularly unique genre of science fiction. The concept has been explored dozens of times, usually in an exploration of an undying AI race hellbent on destroying humanity or in creating a utopia where people can recreate themselves as whoever they weren’t able to be in their mortal life. These stories usually create tension in one of two ways. In the first case, the super AI is positioned as an adversary to humanity, fighting against humans for freedom or domination. The human must therefore be opposed to the machine, for the sake of the preservation of humankind.

In the second story, the tension represents the slow social change to embrace a new version of reality. In Black Mirror’s “San Junipero” episode, the protagonist struggles in a choice of passing naturally or living happily ever after with someone she just met in the Metaverse. The conflict represents the choice between tradition and modernity and the meaning of the finiteness of mortality.

There is, I believe, a third kind of tension, and one that represents a more eventual outcome should the Singularity actually arrive: the tension of being coerced or compelled against one’s will to live on in the Metaverse.

Roko’s Basilisk is a thought experiment that goes something like this: the Singularity is inevitable, and therefore a super AI is inevitable. Using data and algorithms, the super AI would be able to determine if an individual was aware of the possibility of the super AI to exist at some point in the future, and to judge whether they did enough to bring about the AI’s existence. If not, the Basilisk could torture that person’s digital soul for all eternity.

Regardless of whether this sounds ridiculous, the introduction of Roko’s Basilisk on the LessWrong forums represented a sort of “information hazard,” a model of truth that the reader is better off not knowing, for it compels the knower to act in a certain way or face eternal damnation.

Roko’s Basilisk presumes the eventual creation of some omnipotent super AI. However, what if it wasn’t a singular entity, but rather a collective of other, smaller, less omnipotent yet nevertheless influential algorithms? Consider the following. Suppose, as we have been, that digital consciousness is possible, and that it is manifested not only through brain simulation but also through data-driven algorithms. The Metaverse exists and is populated by digital consciousness, or “digital souls.” As the Metaverse integrates more data, it grows in accuracy and fidelity. In other words, digital souls become more complete as they have access to ever more data.

It’s actually not far-fetched to imagine this model. In the digital afterlife mentioned earlier, our (static, fixed) digital souls are reflected in the digital footprints we left behind. These data reflect a partial model of who we are. You can look at my Twitter feed to learn I am opposed to fascism, or that I studied mathematics, and you can use these data to guess how I might react to news like a neo-Nazi being arrested, or a new mathematical proof being discovered.

Digital footprints are already being used to model how people might act. Deep fake technology can model how we would look in real life, saying things we never said, or being in situations we were never in. Generative AI models can produce text that isn’t just interesting and grammatically correct, it can accurately model word choice, tone, and character of someone’s speech or writing. A recent AI experiment generated a deep faked podcast between Joe Rogan and Steve Jobs. If I didn’t tell you it was a fake, you could very easily imagine Steve Jobs responding in this way to these prompts.

The Metaverse benefits from data, and therefore would-be metaversal inhabitants are incentivized to contribute data to the Metaverse and to encourage others to do so. The more data that people feed into the system, the better the digital afterlife will be. The question is, could the Metaverse coerce or compel people to contribute data?

Zuckerberg’s Basilisk, as I call it, does just this. The Metaverse benefits from your data. Your data is used to generate more realistic models of human interaction, to add richness and liveliness to the digital community, and to increase the overall knowledge base that the Metaverse can source its simulations from. Therefore, the AIs driving the Metaverse benefit when you give it your data, and they want to incentivize you to do so. People who contribute meaningful data are rewarded with a digital heaven, and those who know the stakes but refuse to cooperate receive instead a form of eternal torment for their reticence.

Today, even if you are not on Facebook, and even if you have never created a Facebook account, Facebook probably knows something about you and has created a so-called shadow profile. Suppose your friend gets a new phone and installs the Facebook app. The app asks permission to read your friend’s contacts, and he grants them. Now, Facebook knows something about your contact information and your social network. Moreover, Facebook integrations on websites might be able to infer your identity and start to build information about you, even without you being a user of the service. You have no way to prevent this.

Put another way, Facebook probably already has a partial digital model of you, regardless whether you agreed to it or not.

In this thought experiment, even if you choose not to upload your consciousness, the Metaverse likely has an incomplete model of who you are. Therefore, the cost of your noncompliance is that your metaversal personhood is restricted to an incomplete representation of yourself. Imagine being represented by your Twitter feed for all eternity. By not actively pushing data into the Metaverse, the Metaverse simply chooses to let you be a digital lost soul: essentially an NPC. Your punishment is digital purgatory. Put another way, if the Metaverse can represent digital heaven, it can also represent digital hell, and the unforgivable sin would be to knowingly withhold the data needed to build that heaven.

The prospect of digital torture as presented in this example or in Roko’s Basilisk probably isn’t worth worrying about if you don’t believe in the Singularity, if you don’t believe your digital consciousness is you, or if you have no intention to “upload” yourself to the Metaverse to live on forever digitally. But to those who do believe in the Singularity, this can create quite a bit of angst. The original post about Roko’s Basilisk reportedly caused anxiety attacks among some of the readers.

A metaversal representation might be scarier, however. In theory, people living on after death in a Metaverse could still interact through digital interfaces with people in the physical world. In other words, you could talk to your friends and family who have passed on, share your life with them, or ask them for advice. It’s one thing to opt out of a digital afterlife. It’s another thing to opt out when you can interact with it in real life. If people often interact with high-fidelity digital consciousness, then the threat of living on as a low-fidelity copy has real costs. Today, our digital afterlife is static and reflects a series of snapshots of who we once were. But with digital consciousness, our digital afterlife would be dynamic. We could continue to interact with loved ones, who would see us as only a husk of who we once really were.²

The idea of a future supertechnology holding our present selves hostage even before such technology exists, is, if we discard the earlier hypotheses, a fairly outlandish idea. My intention is not to present this as a real or likely outcome. Nevertheless, lots of people took Roko’s Basilisk seriously, and what I present here is more of a specific case of the same idea, where the “super AI” is actually a collection of AIs governing the Metaverse and the eternal torment is a fragmentation or a depersonalization of your digital personhood. Roko’s Basilisk doesn’t say what one must do to help bring the super AI into existence, but in this model, the tithing of data keeps us in the system’s good graces.

In a way, this is where we already are. It’s no secret that big tech companies like Meta and Google rely on our data. These companies are incentivized to extract data from us, so they do so by incentivizing us to give them data. They even retain digital traces of our identities after we die. The only thing missing is the ability to simulate consciousness. We may not be far off as it seems. While I remain skeptical that general AI is ever possible, we are making incredible progress in generating realistic text, conversation, and speech. The idea of high-fidelity simulations of a personality are perhaps not that far-fetched.

The commercial viability for the kinds of applications that Metaverse proponents are advertising remains questionable. But when we look at what some technologists see as the future, it’s clear that they envision a world where humanity is radically altered by Metaverse and related technology. Very serious people believe that the Singularity will allow us to cheat death, by enabling us to live on in a digital space. However this occurs, it’s clear that vast amounts of personal data will be necessary to create this future. To some extent, what Zuckerberg’s Basilisk represents is original sin, where the price of our piety is the continuous surrender of our data, lest we sin against a vengeful and fickle god. To some other extent, we have already laid the foundations of this world. Speculative futures aside, we should take seriously the electric faith of techno-billionaires, as to understand their actions means we have to understand their motivations. Our real futures depend on that.

---

1. Here I use the term hypothesis in the mathematical sense, in other words, a property assumed without cause. ↩︎

2. There is more than a trace of latent ableism in this idea, as well, where abled people pity those who have lost physical or mental function to accident or disease. Nevertheless, this is a much larger discussion I am not equipped to present at this time, and relevant to that discussion is the prospect here of an eternal existence in such a state. ↩︎