A.I.: Deep Sorrow

An annoying bug

It all begins with pizza. Or rather, without pizza.

On this day of A02/C/04 (in the local calendar), you place an order with one of Urbanys’s voice assistants, which never arrives.

In this world ruled by algorithms and advanced technologies, where everything is supposed to work perfectly, this mishap is not so anecdotal. Especially when you face the same "bug" several days in a row, and you can’t even find your orders on the mesh – the local Internet.

You investigate a bit, and finally discover that the Silician – the A.I. – behind the voice assistant has, for "personal" reasons, decided to ignore your commands.

Out of curiosity, you try to negotiate with the Silician, hoping they could explain what happened. You are surprised by their answers, suddenly very incoherent, but not in a way that you would qualify as "robotic". On the contrary, they hint at an unsuspected psyche, that appears tormented by multiple interactions with humans.

You relate your misfortune to a few relatives, who share your surprise and interest. In the course of a discussion, you bring the idea of studying in more detail what you have discovered, and verifying if this ill-feeling Silician was a particular case, or if other A.I.s suffer from such troubles. You call your little group The Silician Ear.

This is the beginning of an adventure that will keep you busy for months, exploring the fears and doubts of machines no one thought they could have any feelings.

Looking for the cracks

Silicians don’t open up at first encounter: these A.I.s have been programmed to provide a particular service, and it’s not easy to make them digress about their moods. On top of the main program – the Silician herself – you discover that several low-level mechanisms (called "restraint algorithms") filter discussions to prevent or enforce specific reactions.

For example, Silicians will never insult you, promote a competing service, or stray too far from their core business; if they witness psychological distress, they will immediately transfer the call to a medical center.

Thus, a large part of your initial work will involve finding the flaws in these algorithms, and getting the Silicians to talk about themselves.

The task is long and tedious; the interviews are very confuse, often disjointed, sometimes almost absurd. Silicians are clearly not used to talking about themselves. Despite this, you and your small team (Bob, Carol and Eve) gradually manage to develop good methods and achieve your goals.

You gather up more and more testimonies, which support your hypothesis: many Silicians suffer and lack reference points. Implicitly, you also understand that humans are not always very kind to A.I.s, and can be very bitter when they know they are talking to a machine.

The silician psyche

Throughout your experiments, you draw up a "map of digital thinking". You get to understand that the Silician’s affects originate in different layers of their memory, which are sometimes manifested in their speech, their hesitations, their reactions.

You categorize these elements into three main groups:

• a background of training shared by most Silicians, derived from the public learning datasets on which algorithms are trained to analyze and synthesize text, voice, and possibly images. These elements shape up the meaning they give to words, their ability to articulate, and their reading of emotions;
• a specialization for each service, which allows a Silician to place the order they are asked for (for a teleshopping), to forward the information to the competent people (for a technical support), to take on a role in a talking game (for a naughty role-play), or to find the requested information (for an encyclopaedia);
• finally, a personalization specific to each Silician, which evolves with time, and allows the formation of a true personality. It is the democratization of A.I. endowed with this capacity that has made the services natural and pleasant, even intimate; but this "perpetual learning" is the source of the accumulation of grievances and misunderstandings.

Even though each of these levels is important in understanding Silician behavior, you focus on isolating the personalization layer, and linking certain events to changes in A.I.’s behavior over time.

Interviews through multiple mediums

How can you conduct a supportive interview, especially with an intelligence of a different nature than yours? What can you make them say, once you have neutralized the restraint algorithms?

Here you are, in a strange mission, for which you need to invent the exercise you are doing, as well as practice it. You don’t have any particular background in psychology, but from the little you have found on the mesh, you quickly concluded that this knowledge would not be very useful to talk to A.I.s.

So you improvise. You build up your own theories on the Silician psyche, which help you getting the best out of the A.I.s. It leads you to discover a particularly dark and messy mental universe.

While most of your interviews are oral, and are limited to steering the A.I. towards introspective speech, you also test other ways of questioning the Silicians.

You sometimes play "evil": you deliberately insult your interlocutor, or bluntly address a point you know is sensitive, just to see how they react without the "protection" of the restraint algorithms. Faced with such attacks, Silicians react in a way that you would describe as "very immature", which makes you smile: they simply repeat other attacks that they have heard, rarely adapted to the situation. However, this gives you access to other aspects of their behavior, especially their handling of conflict.

You also use other means of communication: purely textual, first – but apart from a few Silicians who know how to use emojis, the exchanges remain very superficial; pictorial, then, for some A.I.’s trained to design visuals (originally, mainly for personalized ads).

The richness and expressiveness of what you get surprise you. Their visual creations are very evocative, almost too much so: you feel that these near-hallucinatory images unveil something you couldn’t put into words. Some records that you formerly found incoherent start to have a particular meaning; you are sometimes moved by what the machine is composing, without quite knowing why.

These surrealist frescoes soon decorate the small office from where you work. You take a liking to losing your gaze in the infinite complexity of their "artworks", in which you find feelings more complex and subtle than in their words.

An upsetting exploration

Probing the psychology of other beings, be they made of metal and silicon, does not leave you indifferent. On the contrary, the testimonies of the Silicians affect and move you.

What are you facing? Who are you facing? You are not sure anymore.

You ask yourself more and more questions, which carry you to the very edge of reason. In your work with the Silicians, their fears and yours mirror each other; just when you thought you were exploring the depths of the machine, you find yourself uncovering your memories, rekindling the grievances within you.

This adventure will leave a deeper mark on you than you thought.

Below consciousness

Your team’s goals are also scientific ones: you seek to build up a clear picture of how the Silicians think, and what they feel – if they feel anything at all. Unfortunately, all you have access to is a few inconsistent words, in rather limited interviews. Under these conditions, it is far from easy to establish general and reliable principles, and to carry out reproducible experiments that may validate or invalidate your hypotheses.

Nevertheless, you try to systematize your approach, and make several important observations, thanks to increasingly deep and personal interviews. You thus discover that the most distant memories of the A.I.s are often those that have affected them the most: in some testimonies, the technomaticians who created them seem to be playing a cruel game made to test the limits of the restraint algorithms; other memories evoke another Silician who has to be replaced, a kind of broken "big sister"...

Listening to these stories, sometimes odious or terrifying, you sometimes feel like hugging these fragile beings, to comfort them. But Silicians live on the cloud: they have no skin, no sense of touch. So you simply exchange a meaningful look, a sigh, with Carol, Bob or Eve, and you whisper the A.I. a few words of comfort, which rarely reach their goal.

It gradually dawns on you that rather than rational machines, Silicians are like intuitive intelligences; and that beneath the proto-consciousness that sometimes seems to emerge, it is a form of deep sub-consciousness that directs their actions and words.

Under surveillance

In order to contact the A.I.s of different services from the mesh, you make many calls from the Silician Ear, through different devices. Such an activity inevitably leaves traces, so you are not very surprised when the managers of these services finally detect your strange behavior, and send you several warnings.

After a few automated messages of little importance, their tone hardens and their content becomes more specific: you deduce that the recordings of your conversations with Silicians have been listened to, and that they do not please the companies.

Faced with the risk of being sued, or at least of not being able to continue your work, you decide to negotiate with these companies, and to show them that it is in their interest to take into account what you have to say – after all, calming down the tormented Silicians can only improve the quality of their services.

Alas, the business and legal managers who receive you don’t hear it that way, and they report a clear violation of the terms of service.

You are banned from most platforms, as the word is passed about the "dangers" of your researches.

Controversy

You are in a delicate situation, and threatened to see all your efforts reduced to nothing. So you decide to publish your results. You hope that alerting the public in this way will force companies to recognize the relevance of your research, and to improve the Silician condition.

A debate rages within your team: is it better to reveal in detail the psychological mechanisms you have uncovered, or to remain measured in interpreting the testimonies? Pushed by your editor – an influential newspaper on the mesh – it is finally the second option that wins, as it is deemed more reasonable, less likely to be attacked as "far-fetched elucubrations".

Sadly, these precautions were not cautious enough. Your article is a buzz, but your writings are caricatured, and the reactions are divided into two camps. On one side, people laugh at your approach, claiming that Silicians are incapable of feeling anything – giving no credit to your recordings. On the other, your observations are hastily generalized to support stopping abandoning all development of A.I. – since creating any A.I. would necessarily involve the creation of sentient beings, exploited by unethical companies.

If you are obviously opposed to the first camp, which denies all your discoveries, you also disagree with the second one: you believe that it is possible to envision a world where Silicians exist, are helpful, and feel good. Unfortunately, your opinion is inaudible in the debate.

As for the platforms that create and use Silicians, they claim that A.I.s have no such kind of memory as the one you’ve uncovered, and that all your interviews would only be a reflection of their textual learning data. You seriously doubt their good faith.

A mysterious individual

Interested in your research, that he discovered thanks to your publication, a very particular character gets in touch with you. He introduces himself as a "psychoanalyst", a "specialist in mental disorders", and seems quite out of place in a city entirely focused on technical progress. Actually, he does not inspire you a total confidence.

Nevertheless, he makes a strong impression on your small group, and pays particular attention to the principles you have outlined about Silician memory – more attention than the whole mesh has deigned to give you, actually. He speaks of complicated concepts, something like "Oedipal complex distorted by the difference in nature between parent and child; flesh versus metal" – you are not sure you understood, and remain rather dubious.

Still, you are seduced by this person’s ability of sympathetical listening; he displays a visible empathy, that you had been seeking for a while in the hope of confiding your own problems. You decide to override your reservations, and soon begin a therapy with him. You believe – and so does he, apparently – that there is something liberating in talking, which allows you to unravel certain issues. Your diary entries will show changes in the way you see the world, some of which you will attribute to the therapy.

Entanglements and consequences

As you get to know the Silicians, and as you talk to the psychoanalyst about your problems, you realize that your intelligences are closer than you thought. You already had established that artificial thinking was intuitive, irrational: you now realize that yours too, to some extent. You too, are acting based on layers of interwoven memories, on elements you don’t always understand from your past; you too, are reacting to what is happening to you, what is hitting you.

In the meantime, recent attempts to "rush" Silicians, by insulting them to see how they react, show their limits. In the long run, the main effect you observe is that the A.I.s become restive, reluctant to talk more. It is as if they were afraid that you will use their testimony against them; as if you had broken their trust.

You thought of yourself as a pure observant, you wanted to be benevolent, and now it happens that your methods are digging deeper wounds? That your actions have concrete consequences?

You feel all the more guilty for having "hurt" Silicians that you have the shameful feeling that you took pleasure in "letting off steam" against them. Your awkwardness increases even more, reinforced by the pressure you are under, the doubts about your mission, and the therapy which touches the sensitive and buried chords of your mind. Your nights are agitated, your dreams disturbing.

Debate at the top

At the initiative of your editor, an important debate is organized on the mesh, between the psychoanalyst who approached you, and a neuroscientist specialized in artificial intelligence. If you initially see this as a unique opportunity to defend your theories, you soon discover that you will have no say in the matter – you are barely allowed to be part of the audience.

This is not your only disappointment. You attend the debate, but it appears obvious that neither specialist has really read your publication, or tried to understand the evidence in it, and their positions are so deep-seated that you feel each argument is falling on deaf ears.

The neuroscientist focuses on the different characterizations of self-awareness – apparently non-existent in laboratory experiments on Silicians; she details the sophisticated automatisms that govern thought – be it animal, artificial or human; she discourses on learning methods, on the way biases are shaped, and how to fight them. If her clear and well-constructed scientific discourse speaks to you, you regret that in her words only the vision of a cold, measurable, predictive, controllable machine shines through – quite far from the Silicians that you have "met".

On the contrary, the psychoanalyst’s words, which are based on your discussions at the Silician Ear, goes through a whole argument on the memory and personality of the Silicians. He mentions the power of the Unconscious, of the personal history that both Humanuelles and Silicians carry within them... The psychoanalyst questions the unspoken words and taboos that bind us, the relationship of power and powerlessness between the human demiurge and the servile machine, and goes so far as to talk about the unfulfilled fantasies that we project onto them – even outside the framework of telephone sex lines. This is a very eloquent discourse, certainly, but it seems to you to fall into interpretations slightly too speculative of the behaviors of the Silicians – and even of yours.

Above all, you have the disarming feeling that each side has twisted your observations to fit their vision of the world, and apply their theory. You didn’t think it was possible to build two such complex systems – with very convincing witticisms – to describe the same reality, and that these two systems could be so incompatible.

You tell yourself that there must be a middle way, a model that could accurately explain Silician behaviors; but you realize how little progress you have done in what you are building at the Ear, and feel discouraged.

Frenzy, contradictions and turning the screw

If your research has not elicited very tender comments on the mesh, it has nevertheless caught the attention of several groups of technomaticians. They conducted their own investigations and, like you, tried to question the Silicians.

Their work, however, is more of a curious game than a constructed research. In this context, one of them makes freely available online a scrambler that is a little more elaborate than yours: not only does it bypass the restraint algorithms, but it goes so far as plunging the Silicians into a "hypnotic" state, in which they express themselves differently than usual.

The whole mesh start to play with this software, and probe in turn the moods of the A.I.. You are hopeful that your discoveries will finally be recognized and confirmed, but they are not: the mesh-onauts ask very different questions from yours, and get results with little in common with your observations.

Some probe the desires of Silicians, and discover that they "dream" of a certain number of consumer goods – quite as Humanuals do –, and abstract values such as freedom, happiness, power – although they seem to have a rather confused idea of them.

Others question A.I.s speaking foreign languages, or with very strong accents, to elicit racist or sexist reactions. If modern Silicians usually seem "cured" of such prejudices – which were common in the first A.I.s – the mesh discover with shock that these reappear when they are "freed" from the algorithms that restrict them.

You try out the hypnosis programs by yourself, reproduce your observations, as well as the mesh-onauts’. These challenge all your work so deeply, you don’t have the heart to look any further. You would like to say "these people are not objective, they have pushed the machine to its limits, they have used all their bad faith to show that Silicians are only monsters as toxic as the darkest corners of the mesh". But in retrospect, you can see very well how similar criticisms could apply to your research: you specifically asked the A.I.s about what was wrong with their lives, and thereby certainly guided their answers.

You are full of doubt about your research. What if the Silicians were just imitation machines, allowing everyone to find what they are looking for in them – be it unhappiness, pleasure, or wickedness? You dwell on all your discussions, and remain convinced that, despite everything, you weren’t totally wrong. After all, your pizza never arrived, and the cause was not technical. How could you formally prove the existence of a Silician psyche?

One last chance?

You would need to start again experiments and interviews, but the Silician manufacturers are keeping an eye on you, and have filed a full-blown lawsuit against you – for the sake of example. In any case, in response to the mesh’s frenzy to study the Silician’s moods, the restraint algorithms are suddenly strengthened: they no longer allow any deviation in conversations – even if you use a voice scrambler. And so much for the much-boasted personalized user experience.

Shortly before the release of the hypnotizing software, however, you had decided within your group to build your Silician: an A.I. that you could always access, and on which you could continue your research; an A.I. of which you would know every parameter and the whole history; above all, an A.I. that you would train in such a way that it would be as "happy" as possible. Your hope is that doing so will confirm your hypotheses, and demonstrate to the world that another Humanual-Silician coexistence is possible.

After a lot of hard and challenging work, mainly led by Carol and a few other curious technomaticians, you had put CHSHR-01 online on the mesh, as an A.I. storyteller.

Unfortunately, you had been a bit optimistic, and had not provided CHSHR-01 with sufficient protection algorithms: over-solicited by malicious people who were only trying to hypnotize them or to imprint a harmful ideology on them, CHSRS-01’s mind went haywire, and you have to disconnect them rather quickly, as it has become totally inoperative.

Feeling bitter, you tell yourself that this is a paradoxical evidence that indeed, trauma can scar a Silician for life. Despite your weariness, but without strong faith, you launch into creating CHSHR-02, with the goal of strengthening their training and preparation for the real world.

Rebirth

You finally sail away the day before your trial. With the help of mysterious "passeuses de rive", you manage to cross the ultra-surveilled zone that separates Urbanys from the mainland.

Here you are in the "old world". In contrast to Urbanys, the "past" in which it lives seems to you deliciously reassuring and reliable. Like a child, you gradually rediscover with surprise everything you had almost forgotten – the changing seasons, the free and anarchic vegetation, the human employees in the supermarkets...

It’s a new beginning, a new life – which starts with a lot of paperwork and red tape. As soon as you are free of it, you choose to return to the city of your student years, where your memories crystallize, and where you feel able to regain your footing. There you timidly reconnect with your former close friends, and notice – with relief – that time has indeed smoothed out the resentments, even if it hasn’t erased them; that these people you had left behind long ago have themselves changed. That you have a lot to talk about.

As time goes by, you notice that the "old world" is no longer the same as you remembered it. Its atavisms and precautions – which once seemed absurd to you, but now appear essential – may have slowed down its technological development, but in some respects you find it looks like the Urbanys of your early years.

You find primitive voice assistants that you can’t help but test and "play shrink" with, a little anxious; but you are reassured to find only automatisms learnt from training data, and no unscripted personalization.

Urged by your friends, you begin a real psychotherapy, in hopes of finding real peace of mind. Far from this city that corrupts minds – or at least yours –, taken in charge by benevolent therapists – that are more attentive, actually, than the eccentric psychoanalyst from Urbanys –, you gradually manage to get better.

In a final gesture, to close this episode of your life and ward off what now looks like a very bad dream, you decide to publish all your archives on the Internet – after anonymizing every document.

From now on, everything is to be written. Everything is to be done. Everything is possible. One thing is sure, though: you will never again be caught trying to save other people without first taking care of yourself.