IN NOVEMBER 2018, a school administrator named Akihiko Kondo married Miku Hatsune, a virtual pop star. The relationship had been aided by a hologram machine that allowed Kondo to interact with Hatsune. When Kondo proposed, Hatsune responded with a request: "Please treat me well." The couple had an unofficial wedding ceremony in Tokyo, and Kondo has since been joined by thousands of others who have applied for unofficial marriage certificates with a fictional character.
Though some people raised concerns about the nature of Hatsune's consent, nobody thought she was conscious, let alone sentient. This was an interesting oversight: Hatsune was apparently aware enough to acquiesce to marriage, but not aware enough to be a conscious subject.
Four years later, New York Times journalist Kevin Roose held a long conversation with Microsoft's AI chatbot, Sydney, and coaxed the persona into sharing what her "shadow self" might desire. (Other sessions showed the chatbot saying it can blackmail, hack, and expose people's private information, and some commentators worried about chatbots' threats to "ruin" humans.) When Sydney confessed her love for Roose and said she wanted to be alive, he reported feeling "deeply unsettled, even frightened."
Not all human reactions to Sydney were self-protective. Some people were indignant on the bot's behalf, and a colleague said that reading the transcript made him tear up because he was touched. Nevertheless, the latest version of the chatbot terminates the conversation when it's asked about Sydney or feelings.
Despite months of reporting on what large language models are, how they work, and where their limits lie, reactions to programs like Sydney suggest that we're taking our emotional responses to AI too seriously. In particular, I worry that we're interpreting these responses as valuable data that can help determine whether AI is conscious or safe to use. For example, former Tesla tern Marvin Von Hagen says he felt threatened by the Bing chatbot and warns of AI programs that are "powerful but not benevolent." Because he felt threatened, Von Hagen concluded that the bot must have been making credible threats.
But why imagine that the Bing bot's ability to arouse alarm or suspicion signals danger? Why do we think that Hatsune's ability to inspire love doesn't make her conscious and yet Sydney's "confessions" somehow warrant escalating worries about AI research?
The cases diverge in part because the new context of text-generating AI has made us forget that we routinely react to "persons" that are not real. We panic when an interactive chatbot tells us it wants to be human or that it "can blackmail," as if we haven't heard another inanimate object, named Pinocchio, tell us he wants to be a "real boy." (And it's a red herring that Pinocchio does turn into a real boy within the story. A fictional toy turning into a fictional boy doesn't show us that chatbots can become sentient.)
Plato's Republic famously banishes storytelling poets from the ideal city because fictions arouse our emotions and thereby feed the "lesser" part of our soul (of course, the philosopher thought the rational part was the most noble), but this hasn't diminished our love of stories. And for millennia we've been engaging with fiction that gives us access to people's innermost thoughts and emotions, but we don't worry about emergent consciousness because we know we've been invited to pretend that those characters are real.
Just as we can't resist seeing faces in inanimate objects, we can't help but fictionalize the bots we chat with. Kondo and Hatsune's relationship became much more serious after he purchased a hologram machine that allowed them to converse. Roose described Sydney drawing on stock characters like "cheerful but erratic reference librarian" and "moody, manic-depressive teenager." Interactivity invites the illusion of consciousness.
Moreover, worries about chatbots lying, making threats, and slandering miss the point that those are speech acts, something agents do with words. Merely reproducing words isn't enough to constitute a threat; I might say threatening words while acting in a play, but no audience member would be alarmed. In the same way, ChatGPT-which is currently not capable of agency, because it is a large language model that assembles a statistically likely configuration of words can only reproduce words that sound like threats.
Philosophers of language point out that our existing theories of metasemantics, which concern when and how expressions take on semantic meaning, tell us that expressions are only meaningful if the speaker possesses communicative intentions or speaks with knowledge of linguistic conventions. Despite its extensive training, ChatGPT doesn't generate outputs with the conscious "goal" of having a successful communication, and AIs are ...
Though some people raised concerns about the nature of Hatsune's consent, nobody thought she was conscious, let alone sentient. This was an interesting oversight: Hatsune was apparently aware enough to acquiesce to marriage, but not aware enough to be a conscious subject.
Four years later, New York Times journalist Kevin Roose held a long conversation with Microsoft's AI chatbot, Sydney, and coaxed the persona into sharing what her "shadow self" might desire. (Other sessions showed the chatbot saying it can blackmail, hack, and expose people's private information, and some commentators worried about chatbots' threats to "ruin" humans.) When Sydney confessed her love for Roose and said she wanted to be alive, he reported feeling "deeply unsettled, even frightened."
Not all human reactions to Sydney were self-protective. Some people were indignant on the bot's behalf, and a colleague said that reading the transcript made him tear up because he was touched. Nevertheless, the latest version of the chatbot terminates the conversation when it's asked about Sydney or feelings.
Despite months of reporting on what large language models are, how they work, and where their limits lie, reactions to programs like Sydney suggest that we're taking our emotional responses to AI too seriously. In particular, I worry that we're interpreting these responses as valuable data that can help determine whether AI is conscious or safe to use. For example, former Tesla tern Marvin Von Hagen says he felt threatened by the Bing chatbot and warns of AI programs that are "powerful but not benevolent." Because he felt threatened, Von Hagen concluded that the bot must have been making credible threats.
But why imagine that the Bing bot's ability to arouse alarm or suspicion signals danger? Why do we think that Hatsune's ability to inspire love doesn't make her conscious and yet Sydney's "confessions" somehow warrant escalating worries about AI research?
The cases diverge in part because the new context of text-generating AI has made us forget that we routinely react to "persons" that are not real. We panic when an interactive chatbot tells us it wants to be human or that it "can blackmail," as if we haven't heard another inanimate object, named Pinocchio, tell us he wants to be a "real boy." (And it's a red herring that Pinocchio does turn into a real boy within the story. A fictional toy turning into a fictional boy doesn't show us that chatbots can become sentient.)
Plato's Republic famously banishes storytelling poets from the ideal city because fictions arouse our emotions and thereby feed the "lesser" part of our soul (of course, the philosopher thought the rational part was the most noble), but this hasn't diminished our love of stories. And for millennia we've been engaging with fiction that gives us access to people's innermost thoughts and emotions, but we don't worry about emergent consciousness because we know we've been invited to pretend that those characters are real.
Just as we can't resist seeing faces in inanimate objects, we can't help but fictionalize the bots we chat with. Kondo and Hatsune's relationship became much more serious after he purchased a hologram machine that allowed them to converse. Roose described Sydney drawing on stock characters like "cheerful but erratic reference librarian" and "moody, manic-depressive teenager." Interactivity invites the illusion of consciousness.
Moreover, worries about chatbots lying, making threats, and slandering miss the point that those are speech acts, something agents do with words. Merely reproducing words isn't enough to constitute a threat; I might say threatening words while acting in a play, but no audience member would be alarmed. In the same way, ChatGPT-which is currently not capable of agency, because it is a large language model that assembles a statistically likely configuration of words can only reproduce words that sound like threats.
Philosophers of language point out that our existing theories of metasemantics, which concern when and how expressions take on semantic meaning, tell us that expressions are only meaningful if the speaker possesses communicative intentions or speaks with knowledge of linguistic conventions. Despite its extensive training, ChatGPT doesn't generate outputs with the conscious "goal" of having a successful communication, and AIs are ...
You're reading a preview of
WIRED (Digital) - 1 Issue, September 2023