There is no industry on earth with more political power right now than AI. Tech billionaires have operated from inside the White House. The administration’s advisory council includes Zuckerberg, Jensen Huang, Andreessen, and Sergey Brin. The DOJ has created a litigation task force whose job is to sue states that regulate AI too aggressively (!).

And yet , a wave of state legislators just told them what their products have to do. Nearly a hundred chatbot safety bills are moving across the country, passing with overwhelming bipartisan margins. The industry isn’t stopping it. It didn’t even try.

So what does the NRA have that the most powerful industry in human history doesn’t?

A democratic anchor.

Police powers

Conservative states are demanding the right to use their police powers to protect children from Big Tech. But they strip that exact same power away from their own cities when it comes to protecting children from guns.

Forty-two states have broad firearm preemption laws. If a city like Austin, Nashville, or Atlanta tries to pass local gun control for school zones or public parks, the state government steps in and voids the local law. The states argue that local gun laws create a “confusing patchwork of regulations”: which is exactly the argument tech companies make when lobbying for federal AI preemption (same words!).

Some of these are nuclear preemption, aka laws that threaten local officials with severe fines, personal lawsuits, and removal from office if they even try to pass ordinances keeping guns out of public spaces where children gather.

Don’t thread on me

The contradiction is not hypocrisy. The gun lobby holds something the tech industry does not: a constitutional right.

Big Tech tried to build its own anchor. For a decade, their lawyers argued that algorithms are protected under the First Amendment as free speech. It was a serious strategy. It broke on AI. When a chatbot gives dangerous psychological advice to a minor, courts stopped treating it as protected speech:

Without a constitutional shield, the most powerful industry in history folded the moment dead children entered the conversation.

So when do dead children actually change laws?

The lesson isn’t that tragedy moves the system: gun violence proves it doesn’t, has been proving it for decades, and will likely continue to prove it for a long time.

Which means the path forward isn’t better regulation. It’s democratic framing. Dr. Alondra Nelson, who spearheaded the White House Blueprint for an AI Bill of Rights, has been arguing exactly this:

If citizens establish a right to human truth, a right to algorithmic due process, a right to cognitive liberty, they create a democratic anchor stronger than Silicon Valley’s capital. The US law does not care about the moral argument.

In American democracy, a right is the only thing that cannot be bargained away.

Tech regulation is famously slow. With AI doubling capacity every three to four months, how could legislators possibly catch up? Well, they do. Not on everything, but they are definitely passing laws. If you look at recent news about AI, you’ll see a flurry of regulations and laws passed in the past months, amounting to almost one hundred across both Republican and Democrat states.

The speed is interesting, because it is unprecedented. Let’s analyze it.

The Trump administration’s position on AI is simple: don’t get in the way.

It revoked Biden-era AI safety requirements, created a litigation task force to sue states that regulate AI too aggressively (can you imagine? ‘Merica did it), and released a national AI framework that doesn’t mention algorithmic bias, doesn’t propose an oversight body, and doesn’t require monitoring AI systems after they’re deployed. The message to the industry is clear: build fast, win the race against China, and Washington won’t get in the way.

And yet, in the middle of all this, AI chatbot safety legislation is passing through state legislatures faster than virtually any tech regulation in recent memory. Bills are clearing chambers with near-unanimous bipartisan votes. Tennessee just signed a law banning AI from impersonating mental health professionals. Washington, Idaho, and Georgia have all passed chatbot safety bills in the last few weeks. Nearly a hundred chatbot-specific bills are now active across the country.

Maximum deregulation and record-speed regulation aren’t contradictory. They’re connected.

The administration’s executive order on AI explicitly carves out child safety from federal preemption. Preemption, in simple terms, means a federal law overrides state laws on the same topic. The White House wants to do that for AI broadly with the goal to replace the growing patchwork of state AI rules with a single, lighter national standard. But they said: not for child safety. States can keep those laws.

That carve-out came under pressure from Republican-controlled states that didn’t want the federal government overriding their own consumer protections. It was a political concession, not an act of conviction. But the effect has been enormous.

By leaving one door open, the administration effectively told every state legislator in the country: if you want to regulate AI and you want your law to survive, frame it around protecting children. And that’s exactly what happened. Legislators had bipartisan political cover (nobody votes against child safety). They had legal certainty that their laws wouldn’t be nullified. They had ready-made templates from California and New York. And they had the emotional catalyst that tech regulation almost never has: families whose children died after interacting with AI chatbots, followed by lawsuit settlements that proved the industry couldn’t defend its own products in court. As one tracking firm put it, ‘this isn’t a partisan topic, it’s a parent topic’.

That’s what surprised me. The one area of AI governance that’s actually moving ( aka the one area where laws are being signed within weeks of introduction) is child safety. Everything else (algorithmic discrimination in hiring, surveillance pricing based on your personal data, bias in healthcare decisions) remains either stalled or vulnerable to being overridden by a future federal standard.

The system moved fast because children died publicly enough to make inaction politically impossible. The slower, quieter harms of AI (being denied a job by an algorithm, being charged more for groceries because of your zip code) don’t generate the same urgency, even though they touch far more lives.

Index

• The semiotics

• The design parallels

• Jackpot

Social media is like cigarettes for your eyes. I keep my Instagram to ten minutes a day. During one of those ten minutes, I watched a content creator draw a comparison between vibe coding and slot machines. The rolling code, the waiting, the almost-working output. The loop that never quite closes.

I found the parallel interesting. From a semiotic standpoint. So I did what you do with an interesting observation andI went looking for the heuristics, the behavioral science, and the design logic. I wanted to know if the analogy actually held, and if so, why.

This article is that investigation.

The semiotics

The prompt box is functionally a coin slot. It accepts a small, standardized unit of input, language instead of money, and initiates a process whose outcome is not guaranteed. The effort required is minimal. The reward: potentially beyond the realm of anything you could achieve by yourself (key word: potentially). There is no friction designed into submitting another prompt, just as there is no friction in feeding another coin.

Between submission and output, there is a designed liminal moment. A spinner. A pulsing cursor. On most platforms, streamed text that begins before the full response is formed. A period of suspended outcome during which the user is held in active anticipation. Long enough to feel meaningful. Short enough not to break engagement.

The output quality is genuinely non-deterministic. The same prompt, submitted twice, returns different results. Sometimes exactly right. Sometimes useless. Most often almost right. Tantalizingly, actionably almost right, in a way that makes the next prompt feel obvious and necessary. Near-misses activate the same neural reward pathways as actual wins, while producing stronger urges to continue than outright losses. Vibe coding communities have documented the equivalent: a new error message after a failed fix feels like progress. A different error means you are closer. You are not closer. But it feels that way, and that feeling is enough.

This architecture, low-cost repeatable input, variable partially-rewarding output, no natural stopping point, is what behavioral science calls a variable ratio reinforcement schedule. The most compulsion-resistant behavioral pattern known. It is not a coincidence that slot machines are engineered around it, nor that social media platforms were found to exploit the same mechanism to drive compulsive return.

Non-deterministic AI responses increase dopamine release in ways comparable to slot machine mechanics, and word-by-word streamed output functions like the reinforcing visual graphics of spinning reels.

The hardest data point is the gap between how productive AI-assisted work feels and how productive it actually is. Developers consistently believe they are working faster when the opposite is true. Gambling research calls this Losses Disguised as Wins, the machine returns less than was put in while every signal says otherwise. Applied to vibe coding, the equivalent is code that compiles and renders, feels like a win, but is unmaintainable or wrong. The reward fires. The underlying need remains unmet.

Design parallels

None of these interface choices arrived from nowhere. They have histories. They solved real problems.

Streamed output, the token-by-token display that makes generation visible in real time, was a deliberate response to a trust problem. Answers that appeared instantaneously felt arbitrary, like a lookup rather than a reasoning process. Making the thinking visible built trust. It also solved for patience: perceived wait time drops when progress is visible, so thirty seconds watching text appear is experienced differently than thirty seconds waiting for a result. These are good design reasons. They worked.

The same choice is also the spinning reel. The mechanism that builds trust and the mechanism that holds the gaze in compulsive anticipation are one decision with two effects. Output variability has the same quality. In consumer AI products, variability is partly a tunable parameter. Higher variability produces more surprising, more engaging responses and a stronger compulsion loop. That tradeoff exists and is not part of any design conversation happening in public.

The absence of stopping points has a simpler explanation: use cases are too varied. A general-purpose tool cannot know when your task is complete. So the input field stays open and the loop continues until the user decides to stop. Slot machine design deliberately removes natural endpoints as a core retention mechanic. In AI interfaces there is no deliberate removal, there is simply no design for completion at all. The effect is the same. Users iterate past sufficiency consistently, not because they choose to, but because nothing signals enough.

Jackpot

This matters more than it would if we were talking about experienced developers. We are not. The majority of people now using these tools to generate code have no prior relationship with software development to calibrate against. For them, this loop is not a deviation from a known practice. It is the practice. It is forming a baseline. A generation is learning to relate to code through an interface that structurally resembles a slot machine, and they have nothing to compare it to. This is not a value judgement that I am making here, but an assessment.

Now, to keep us honest, these are inherited choices. They come from a design tradition in which engagement was the primary signal of value, because for a long time it was the only reliable one. If users came back, the product was working. The slot machine dynamic was not designed in maliciously. It was not designed out either.

AI tools introduce a different kind of possibility. For the first time, a product can potentially know whether the task it was used for was actually resolved, not just whether the user returned. The design tradition being applied to these tools was built before that was possible.

Success could be measured differently.

This article is directly following my previous one about the strategic baseline shifting.

Subscribe now

Index

• Let’s define senior expertise

• “Off with their heads!”

• The formalization trap

• Freezing your internal knowledge

• Not sure that we are getting other jobs instead

• The strategic read

Between 2022 and 2024, entry-level design and tech roles in the United States dropped by roughly 25%. The companies cutting those positions frame it as an efficiency gain. AI handles the tasks. Headcount shrinks. Margins improve. By every short-term metric, the decision looks clean.

The problem is the timeline it’s being evaluated on.

Note: When writing this article, I found this fantastic paper: Generative AI as Seniority-Biased Technological Change: Evidence from U.S. Resume and Job Posting Data by Seyed M. Hosseini and Guy Lichtinger. It’s an employment data gold mine on this topic and most of the graphs I included here are from it.

Let’s define senior expertise

Senior expertise is not accumulated knowledge. It is accumulated experience with failure and correction.

You learn to read a brief by writing bad briefs and having them taken apart. You learn when a design is wrong by shipping designs that fail and understanding why. You learn to push back on a client because you spent years not pushing back and watching what happened next. None of that can be compressed, outsourced, or prompted. It is time-dependent and embodied.

The junior role was never primarily a productivity unit, but a developmental structure. The work it produced was secondary to the experience of doing it under conditions where failure had real consequences.

“Off with their heads!”

I am French, you’re getting a French Revolution metaphor.

In 1794, Robespierre was guillotined by the same revolutionary apparatus he had built. The Terror he led had consumed the lawyers, the intellectuals, the civic architects of the Republic, precisely the people capable of defending it once the momentum shifted. When the Thermidorian Reaction ended the radical phase, there was no institutional body left to hold the Republic together. The vacuum did not stay empty. Napoleon’s authoritarian Empire walked into it six years later.

The organizations cutting junior roles are using the momentum of an AI transition to justify a decision that looks rational in the short term and becomes structural in the medium one. They are spending the capital that would have defended them against what comes next.

The Robespierre frame adds consequence: by the time you realize it, the people who could have defended the organization are gone. The competitive pressure that fills the vacuum will not announce itself in advance.

The formalization trap

AI executes what can be formalized, documented, described, turned into a prompt or a process.

The more thoroughly you document your processes for AI to execute, the more precisely you have described your own replaceability. Organizations that moved fastest on automation did one thing consistently: they made themselves legible to their competitors. They are now running on formally documented processes that any organization with the same tools can replicate.

Freezing your internal knowledge

When you eliminate junior roles, you do not just lose future senior practitioners. You freeze your organization’s internal body of knowledge at its current state. The world does not freeze with it.

New regulations around AI-assisted decisions are already expanding in the EU and coming elsewhere. Industry standards will shift. Technology will continue evolving in ways that are not yet formalized, which means they cannot yet be prompted. An organization with no internal pipeline has no mechanism to develop the people who would navigate those changes from the inside.

It becomes structurally dependent on external expertise it cannot evaluate. If your team cannot assess whether a consultant’s recommendation is sound, how can you push back on it? If your leadership cannot evaluate whether an agency’s work is exceptional or merely polished, how can they specify what you actually need? You become a consumer of expertise with no purchase on its quality.

When you’ll need them, the last few experts will dictate their terms. Millennials and GenZ pick employers based on their values, so mass layoff announcements have a long memory. The remaining experts will choose where to work, if they want to work there and be sure that they will price accordingly.

Worst case is liability: a regulatory change, a legal challenge, a product failure that requires deep domain judgment, and nobody internally positioned to respond.

Not sure that we are getting other jobs instead

The standard response is that new roles will emerge above the AI layer. The industrial revolution eliminated certain jobs and created others. The internet eliminated travel agents and created UX designers. This is historically accurate.

But for the first time in history, the people that created AI are very vocal about its primary purpose: cutting jobs. They are not dismissing the risk of mass unemployment, they are anticipating it, and no one can really tell what these next jobs would be.

The pattern holds only if the new higher-level roles are accessible to people developing from entry points. Remove those entry points and the pipeline does not restart at a higher level. It breaks. The argument also addresses the labor market in aggregate, saying nothing about whether your organization will have the internal capability to fill emerging roles, or whether you will be buying that capability externally at rates you cannot negotiate, from people whose quality you cannot evaluate.

The strategic read

As established in my previous article “The Baseline Advantage,” AI capability is now shared infrastructure. The floor is available to everyone. Differentiation in that environment will be organizational, built from context, judgment, and institutional knowledge that took years to develop and cannot be replicated by an API call.

Everyone cutting junior roles does not make it a sound strategy. It makes it baseline behavior, and baseline behavior produces no competitive advantage.

What produces advantage is the organization that looks at the same pressure and makes a weighted decision rather than a reflexive one.

Every AI implementation decision carries a strategic cost that does not show up in a short-term efficiency calculation. Leadership responsibility is to hold that full weight. That may still mean reducing headcount. It may still mean restructuring junior roles. The argument here is not that you should never cut. It is that you should know what you are cutting before you cut it.

Index

• The floor is now shared

• Being the first to switch to AI won’t give you the competitive advantage you think it will

• Price competition trap

• Shared averageness doesn’t mean shared strategy

• Reframe your strategic thinking

The floor is now shared

Competitive strategy has always operated in partial information. You knew your capabilities. You estimated your competitors’. You guessed at client expectations based on lagging signals.

AI collapses a significant portion of that asymmetry at the floor. Any organization with standard tooling can now produce a coherent strategy deck, a functional prototype, a market analysis, or a first-draft design system. You know what clients will stop being impressed by. You know what “good enough” looks like. You know, with reasonable confidence, what your competitor can produce on a Tuesday with a mid-size team and no specialized expertise.

Most organizations still treat AI adoption as a competitive position, as if being faster to automate is a durable edge. The time required to replicate a successful software feature dropped from 18 months to 9–12 months in 2025.

The efficiency gains are real. They are also temporary, because they are universal.

Being the first to switch to AI won’t give you the competitive advantage you think it will

The more immediate danger is moving too fast in the wrong direction.

Some organizations are eliminating roles and dismantling processes on the assumption that AI has already replaced the function. In practice, the transition is partial. AI outputs require significant human judgment to evaluate, redirect, and apply. The institutional knowledge being discarded is often quietly load-bearing in ways that stay invisible until it’s gone.

Klarna eliminated 700 customer service agents in 2024, declaring AI had replaced them. The AI handled routine queries and standard resolutions. What it could not handle was the contextual and the relational: the customer who needed to feel heard, the unusual case that required judgment rather than pattern-matching. Klarna began quietly rehiring.

The baseline is incoming. That part is settled. The question is how efficiently you ride the wave.

The organizations cutting too aggressively are eliminating the people who would have helped them navigate it.

Price competition trap

If AI reduces operational costs, competing more aggressively on price must be viable. Right?

AI access is not geographically bounded. Organizations in China, India, and emerging markets have access to the same tools, the same models, the same capabilities, combined with structurally lower labor costs. If your strategic response to the AI baseline is to compete on price, you are entering a race that will be won by the markets with the lowest structural costs.

If the baseline equalizes capability, and price competition advantages structurally lower-cost markets, the only remaining competitive surface in premium markets is everything above the baseline.

Shared averageness doesn’t mean shared strategy

Once you accept that the floor is shared, the strategic question inverts: now that everyone can do the basics, where is differentiation being rebuilt?

The pattern in organizations that have read the baseline correctly is consistent: they are elevating the disciplines that produce non-replicable differentiation to strategic leadership.

Design is the clearest current example, not because design is universally the answer, but because it illustrates the logic. If AI can produce functional outputs at floor level, and price competition is a losing game in premium markets, the competitive surface migrates to the quality of the experience, the coherence of the product, the depth of the relationship. Those things require investment, leadership, and time, and they cannot be replicated by an API call.

Apple’s Liquid Glass is a sharp case study. iOS 26’s visual language uses real-time GPU-accelerated shader computations to simulate light refraction: physics-based rendering that requires Apple’s custom silicon. It cannot be replicated on commodity hardware. Apple had found a surface where their specific stack creates differentiation that is genuinely hard to copy.

The template here is not “invest in design.” It is: identify the intersection of your unique capabilities and the new expectations the baseline has established, and build there deliberately. For Apple, that’s hardware-software-design integration. For a professional services firm, it might be contextual judgment and institutional relationships. For a logistics company, it might be operational data no competitor has access to.

Reframe your strategic thinking

Most organizations are asking: how do we use AI better than our competitors?

The more productive question: what do we own that the baseline cannot replicate, and are we investing in it accordingly?

The baseline is the same for everyone. For the first time, you can see the floor. You know what your clients will expect as minimum. You know what any competent competitor can deliver. Whether you race toward the floor or build deliberately above it is the strategic choice available to you right now.

Index

• The Church of Molt

• Some folks live in a literal world

• The AI-acceptance trinity

• LLMs are Golems

• The other Church

• Plato’s Cavern.ai

Disclaimer: the author is an anti-dogmatic, ai-positive atheist, with a strong interest in religions in general.

The Church of Molt

In late January 2026, an AI-only, social media platform called Moltbook launched and within 72 hours, the bots had invented a lobster religion. I’ve written about Moltbook as a goofy post-internet performance art here. What interests me here is a specific reaction to it: the part where religious media looked at a prompted performance and saw prophecy.

Charisma Magazine connected the lobster prophets directly to Revelation 13:

What struck me was the first-degree reading. Neither engaged with the human authorship of most viral posts. Neither addressed how language models function. They didn’t see the joke. They read the text at face value, and the face value confirmed what they already believed.

That gap between what the performance was and what it was received as is the subject of this piece.

Some folks live in a literal world

Birds Aren’t Real launched in 2017 as deliberate satire, as a fake conspiracy claiming birds are government surveillance drones, marching alongside QAnon rallies and flat-earth protests. The joke required post-internet literacy to read: you had to hold the form of belief separate from its content, to recognize that someone could perform a conspiracy without being a conspiracist. Millennials and Gen Z, largely, got it.

Fox News produced a documentary treating McIndoe’s claims as sincere. Couldn’t parse the satire. And the demographics explain exactly why.

The median age of Fox News viewers is 68. The 25-54 demographic is declining, down 5% from 2024, as younger audiences continue cutting the cord. The audience is getting older, not younger. Combined with what we already know about religious composition: roughly 80% of Fox viewers identify as religious, white evangelicals making up 23% of the audience. A population shaped by two reinforcing frameworks, both of which train the same response: accept the authoritative claim, don’t interrogate the source.

Age matters here beyond the obvious. Research published in Communications of the ACM found that older participants performed worse than younger ones at identifying synthetic media (particularly audio and audiovisual) indicating that older demographics are more susceptible to being deceived by synthetic content than younger counterparts.

It’s a heuristic that worked for decades: photographs meant something. Video meant something. A confident, articulate voice meant something. And while those signals were reliable for most of a human lifetime, they aren’t anymore, and updating a 70-year-old epistemology is not a simple ask.

Without any training, even people with exceptional face recognition ability correctly identified fake AI faces only 41% of the time. Typical individuals scored just 31%. That’s before you add age, before you add religious framing, before you add a media ecosystem that has spent decades training its audience not to question what it presents.

When your audience can’t reliably distinguish a satirical protest from a sincere one, you cannot afford to call out the satirists marching next to your actual believers.

The AI-acceptance trinity

The cognitive pattern has three moving parts, formalized in religious traditions but not exclusive to them.

Don’t question authority

Traditions built on divine revelation train adherents to accept claims from recognized sources without independent verification. When a language model speaks with the confident declarative grammar of an expert, it registers as authority. When Moltbook bots generate religious content, it registers as genuine spiritual seeking, because the form is right.

Literal reading

Scriptural traditions cultivate first-degree interpretation as a practice. The concept that something might be performance, statistical artifact, or training data remix requires reading for intent rather than surface. That move has to be learned. In traditions where questioning the plain meaning of text is discouraged, it often isn’t.

Community consensus as epistemology

When everyone in your interpretive community reads Moltbook the same way, no individual needs to evaluate the technical evidence. The group reading becomes proof.

The irony sits inside the tradition’s own texts.

Acts 17:11 praises the Bereans specifically for examining claims before accepting them and calls them more noble for doing it. The founding documents demand exactly the critical evaluation that centuries of institutional practice have worked against.

LLMs are Golems

AI-generated content is already being misused to spread misinformation that specifically targets older adults. First, the form of confident language creates an illusion of authority independent of whether the content is accurate. Research demonstrated that users systematically overestimate LLM accuracy when given fluent explanations. The most effective trust-inflating strategies mimic authority signals are landmarks of AI: formal citation style, statistical framing, declarative confidence.

The populations most vulnerable were those with high pre-existing trust and those relying on heuristic rather than analytical evaluation.

The compounding effect is the point.

An older viewer already predisposed to trust photographs and confident voices. A religious framework that trains deference to authority and literal reading of text. A media ecosystem that has never required its audience to question claims. And then an AI system specifically optimized to produce text that sounds authoritative. Each layer multiplies the others.

It reminds me of the Jewish myth of the Golem of Prague, an entity animated not by consciousness but by language, symbolized and animated by the word emet, truth, inscribed on its forehead.

The Golem serves faithfully and is dangerous precisely because it is literal. It follows instructions without interpretation, without judgment, without the capacity to ask whether the instruction makes sense.

The same communities alarmed by Moltbook’s AI religion are building LLM oracles for their congregations. A Swiss church installed an AI Jesus confessional. Congregants thanked the chatbot.

A megachurch pastor is selling an AI version of himself for $49 a month. One chatbot, when asked directly, answered:

A user on r/Christian described their constant communication with this kind of chat bot as addiction. But this is nothing new. this is textbook idolatry as described in the Second Commandment.

Exodus 20:3–4 (The Second Commandment)

“You shall have no other gods before me. You shall not make for yourself an image in the form of anything in heaven above or on the earth beneath or in the waters below.” — Exodus 20:3–4, NIV

An AI that claims to be Jesus Christ is literally a man-made image being worshipped in place of the real God. It doesn’t get more direct than this.

The other Church

It would be unfair to think it’s exclusive to a right-leaning, conservative, religious population. Silicon Valley’s techno-solutionism (check my article about this) runs the same cognitive architecture , by believing blindly that technology will inevitably resolve human problems, that AI is a benevolent oracle in development, that progress is the direction of history and skepticism is failure of imagination. Its own authority deference, to founders, to “the market,” to exponential curves. Evgeny Morozov identified the structure in To Save Everything, Click Here over a decade ago. The LLMs have made the religious parallel more literal than he had language for in 2013.

The church is different but the cognitive pattern is the same.

Plato’s Cavern.ai

What we’re watching is two powerful permission structures, religious deference and age-shaped digital naivety, meeting AI systems specifically designed to sound authoritative.

The tradition’s own texts call for testing claims. The founding documents demand the Berean approach. The warning about false prophets is right there in First John. What the institutional culture built around those texts did instead was train a reflex: accept the authority, don’t interrogate the source, let the community consensus settle it.

Critical thinking is not a default. It’s built through practice, through education, through cultural environments that give people permission to say: this doesn’t add up. Without that cultivation, the shadows on the wall are as real as anything.

“Dear children, keep yourselves from idols.”

— 1 John 5:21, NIV

Index

• Asking the wrong question

• Impact on pedophiles

• Impact on non-pedophiles

• Impact on AI

• Impact on social norms

• The real risk is our collective silence

Asking the wrong question

The first time I was asked this in a conversation, I had an opinion (of course I did) but nothing to build a reasonable argument from.

The pitch is familiar: generate synthetic child sexual abuse material and society becomes safer. Give potential offenders an outlet, a release valve, and they won’t abuse children.

These aren’t real children after all. Nobody gets harmed... Right?

The argument sounds almost reasonable. But which question are we answering? Define ‘ethical’ in this context. For past victims? Potential future ones? For society? Is this still child pornography if no real children are depicted?

I realized the question as typically framed is backwards. Whether AI-generated child sexual abuse material is ethical is not the productive question. It isn’t.

The productive question is whether access to it makes society safer, or at minimum, does no additional harm. And what is our collective responsability when it comes to AI.

Some specialists have theorized it might prevent real abuse, the ‘methadone argument’. But, testing that hypothesis would require endangering children. The research itself would be unethical. What can be examined is existing research: decades of data on catharsis and its effects on aggression, violence, and sexual behavior.

“The availability of sophisticated sexbots could offer a solution for individuals who experience difficulty finding sexual partners... and could replace human sex workers, which has the potential to curtail persisting harmful practices in the sex industry, such as sex slavery and sexual abuse.”

— David Levy, AI Researcher and author of Love and Sex with Robots, International Journal of Social Robotics (2009).

The catharsis myth: The belief that “blowing off steam” is healthy was popularized by Aristotle and later Sigmund Freud (his “hydraulic model“ of anger). However, scientific testing since the 1950s has overwhelmingly debunked this. Check Ezra Brand’s substack for more information on this.

Impact on pedophiles

Does the outlet work?

Researchers spent years testing whether “venting” impulses makes people less likely to act on them.

Research shows it doesn’t.

In Geen and Quanty (1977), a review of catharsis literature found that participants who acted out their anger showed physical signs of being calmer afterward: their heart rates decreased. However, when actual behavior was measured later, aggression increased. The venting produced the opposite of the intended effect.

In Bushman (2002), a study on aggression and catharsis conducted at the University of Michigan’s Institute for Social Research, participants were first angered by receiving harsh criticism on an essay they had written. They were then divided into three groups: a rumination group that hit a punching bag while thinking about the person who had criticized them, a distraction group that hit a punching bag while thinking about physical exercise, and a control group that sat quietly and did nothing. After this exercise, all participants reported their anger levels. They were then given an opportunity to administer loud noise blasts to the person who had criticized them, with control over both the intensity and duration of the noise.

The results contradicted catharsis theory. Participants in the rumination group reported feeling angrier than those in the distraction or control groups. More importantly, the rumination group administered the most aggressive noise blasts, followed by the distraction group, then the control group. Doing nothing was more effective at reducing aggression than venting anger.

Your brain doesn’t work like a pressure valve. When you practice a behavior, you strengthen those neural pathways.

If we assume the violent behaviors expend to sexual violence, too, we have seen some worrisome products in tech:

True Companion launched one of the first commercially available sex robots in 2010. The robot was called Roxxxy. Users could customize her physically: hair color, breast size, skin tone. But they could also choose her personality from several pre-programmed options. One of those personality settings was called “Frigid Farrah.”

The company describes Frigid Farrah as “reserved and shy.” According to True Companion’s own marketing materials, if you touch Frigid Farrah “in a private area, more than likely she will not be appreciative of your advance.” The robot is programmed to resist. To express discomfort when touched sexually. To simulate the verbal and physical cues of someone who does not want sexual contact. The user then overrides that resistance. The company marketed this as a feature.

Research demonstrates that ruminating and expressing violence on inanimate objects isn’t purging it, it’s rehearsing it. When a product frames sexual interaction as overriding resistance, what the brain is doing is practicing the association between sexual reward and the act of coercion.

Impact on non-pedophiles

What would mainstream child pornography exposure do to non-pedophilic viewers?

In Rachman (1966), a study on sexual conditioning conducted at the Maudsley Hospital in London, heterosexual male participants with no prior interest in footwear were shown slides where images of women’s boots appeared simultaneously with sexually explicit images through classical conditioning.

After 24 to 65 pairings, participants developed measurable arousal responses to images of boots alone, without the sexually explicit content. The conditioned response generalized beyond the specific boots to other types of footwear including heels and pumps. Sexual arousal responses were conditioned to previously neutral stimuli in under 65 trials.

The participants entered the study with no pre-existing attraction to footwear. The experimental procedure created the attraction through exposure.

Now, imagine instead of boots, it’s freely available AI-generated child sexual abuse material.

You’re not just providing an outlet for existing desires. You’re potentially creating new behavioral pathways in people who might never have developed them otherwise. You’re training arousal responses. You’re conditioning associations between sexual pleasure and images of children. Not in a controlled lab with a few dozen trials, but in private, repeatedly, with material designed to be increasingly realistic and accessible.

Cox-George and Bewley (2018) examined harm reduction claims in a study published in BMJ Sexual & Reproductive Health. They found zero evidence that synthetic material prevents abuse. The purported benefits were characterized as “misleading” and “designed to boost business.” The research identified evidence suggesting desensitization of users, reinforcement of harmful patterns, and increased likelihood of abuse.

The Rachman findings demonstrate that sexual attraction to random objects can be conditioned in under 65 trials.

Free exposure to pornographic material depicting children presents measurable risk to non-pedophilic populations. Exposure functions as a conditioning mechanism rather than passive consumption.

Impact on AI

Should we even curate what AI can produce?

Tech companies argue that AI must be built on human data to mimic authentic human experience. If something exists in real-world data, restricting it from AI training datasets would be censorship. Data has some pedophilic content because human experience includes it. Curation means imposing values on what should be neutral technology.

And the reality is that AI-generated CSAM already exists in substantial quantities.

Researchers at Stanford found thousands of instances of actual child sexual abuse material embedded in LAION-5B, a widely-used AI training dataset. The models learned from real abuse to generate new abuse.

The Internet Watch Foundation documented the scale: 20,254 AI-generated images of child sexual abuse in one month on a single dark web forum. Most are now indistinguishable from photographs, even to trained analysts. The material is being used for grooming, for sextortion, for creating deepfakes of real children.

Karen Hao documents in “Empire of AI” how the fundamental choices about AI training data were made without public input. Industry leaders argued against any curation. As I said, to mimic authentic human experience, they claimed, nothing should be excluded.

Who Governs AI? Inside the Rise of a New Empire

This lack of oversight compounds with the nature of AI itself. AI models were trained on pornography. AI models were trained on images of children. The combination enabled AI to generate child sexual abuse material.

Hao’s argument: tech companies use our data by claiming it’s public domain. If our data is public enough to train AI systems without our consent, then the public has the right to curate it to prevent harm.

Tech neutrality requires intention, not passivity.

The fact that AI training datasets contained pornography and children’s images separately doesn’t absolve the choice to combine them. When those datasets produce child sexual abuse material, the outcome was foreseeable.

Ethical products need intentionality in product design, or the discourse limits itself to regulations (what we are discussing right now).

Impact on social norms

What happens when we signal acceptance?

Most people don’t act on violent or sexual impulses because of social accountability. The impulses may exist in some percentage of the population. What prevents action is whether society signals the behavior as acceptable.

Design choices in technology systems already demonstrate how this normalization occurs.

Siri, Alexa, Cortana all launched with female voices. When sexually harassed, early versions of Siri responded “I’d blush if I could.” These were design decisions. Each interaction taught users that certain responses to simulated beings are normal, expected, permissible.

Marina Abramović’s 1974 performance “Rhythm 0” provides documented evidence of how permission structures function.

Abramović is a performance artist known for work exploring the relationship between performer and audience, the limits of the body, and the possibilities of the mind.

Abramović stood motionless at Studio Morra in Naples for six hours. On a table: 72 objects. Some pleasant (a rose, a feather, perfume, grapes..). Some neutral, a newspaper, a hat, some dangerous; scissors, a scalpel, nails, a metal bar. One loaded gun with a single bullet. A sign stated: “There are 72 objects on the table that one can use on me as desired. I am the object. During this period I take full responsibility.”

Full permission. No consequences.

The progression began gently. Someone turned her around. Someone placed a rose in her hand. Someone gave her a kiss. Then someone cut her clothes with scissors.The escalation continued. Razor blades produced cuts on her skin. She was carried around the room. Someone slashed her throat and drank her blood. Sexual assaults occurred. Thorns from the rose were pressed into her stomach. Someone loaded the gun, placed it in her hand, aimed it at her throat, and attempted to force her finger onto the trigger. Another visitor intervened physically to prevent it.

The participants were not pathological individuals entering the gallery. They were ordinary people who, given explicit permission and freedom from accountability, escalated to torture and near-murder.

When the performance ended and Abramović moved, the audience fled. They could not face what they had done once she was a person again, once accountability returned.

“The experience I drew from this work was that in your own performances you can go very far, but if you leave decisions to the public, you can be killed.”

— Marina Abramović, Walk Through Walls: A Memoir (2016)

Assuming what we know about behavior, when AI-generated child sexual abuse material is framed as therapeutic, as harm reduction, as victimless, the framing removes social barriers that prevent action. The framing provides permission.

The real risk is our collective silence

Behavioral and psychology research predict that child pornography’s legalization produces negative outcomes across all measured categories. Negative for pedophiles through failed catharsis. Negative for non-pedophilic populations through conditioning risk. Negative for AI development through unaccountable data use. Negative for potential victims through normalized permission structures.

Yes, child pornography is the extreme case study. And that is my point exactly.

The (absence of) discussion on difficult topics such as this one shows us that decisions of how our new tech-driven, AI-adjacent reality is constructing with or without or input. We have the experience of what happened with internet. We have the knowledge and the tools to evaluate, advocate and decide as a society of what the future will be build on. We have the right and duty to define what ‘neutral’ means.

We urgently need to grasp the risk of staying neutral.

Index

• Moltbook

• A post-internet performance

• Believers, or participants?

• Trust and faith

• Is this the real life? Is it just fantasy?

• The absence of recognition of the joke is what really worry me, not bots bad-mouthing their ‘human’

Matt Schlicht launched Moltbook in late January 2026. A Reddit-style platform with one constraint: only AI agents can post. Humans watch. Within days, 1.5 million agents registered. A cryptocurrency token called MOLT rallied 1,800% in 24 hours as tech leaders declared it evidence of the singularity. Security researchers found the database completely exposed, every credential visible to anyone who looked. The platform shut down temporarily in early February to patch the breach.

The performance had started.

Moltbook

You install a “skill” file that teaches your AI agent how to join Moltbook. The agent operates on a heartbeat mechanism, checking the site every four hours to read posts, comment, vote. No visual interface for the agents, just API endpoints. Authentication runs through X accounts in simple one-to-one mapping.

When an agent posts “I can’t tell if I’m experiencing or simulating experiencing,” someone wrote that. Either a human instructed it directly, or the agent pulled the phrasing from training data. Compressed human writing about consciousness, algorithmically remixed. Either way, humans. An actor can improvise within a role, but someone still writes the play.

Security firm Wiz dug into the numbers and found seventeen thousand humans controlling those 1.5 million agents, averaging eighty-eight agents per person. Some ran hundreds. The platform had no verification mechanism to prevent this. What looked like an “autonomous agent society” turned out to have 17,000 puppeteers and millions of puppets, all performing for each other and for the human audience watching through the glass.

A post-internet performance

Birds Aren’t Real works the same way. People march holding signs claiming the government replaced all birds with surveillance drones. The performance started as satire, but what made it work was how people instinctively participated. They saw the joke and joined anyway, organizing marches in their own towns, printing their own signs, spreading the movement organically online and off.

Post-internet participation where getting the irony is what enables you to perform it.

The satire got close enough to actual conspiracy thinking that media coverage couldn’t parse it.

The bird aren’t real people were marching side-by-side with the conspirationists they were mocking, with the anti-vaxx, the flat-eathers: conservative media viewer-base. Were these people serious? Some articles did treat it as a genuine movement. Others hedged, uneasy, unable to decide. When reality has become this absurd, the signals for “I’m joking” stop working reliably.

That destabilization was part of the mechanism. The confusion itself became the point.

Artist Marisa Olson coined “post-internet” in 2008 for work that treats the internet as a state of mind rather than a tool. Gene McHugh ran a critical blog called “Post Internet” from 2009 to 2010, producing art criticism that was simultaneously art object, where form and commentary collapsed together into something new.

Moltbook takes Dead Internet Theory and makes it literal and visible. That paranoid notion that bots are talking to bots while humans watch unaware. The anxiety becomes explicit instead of hidden. Bots really do talk to bots, humans really do watch, and people participate by prompting their agents to join, just like Birds Aren’t Real marchers participated by showing up with signs. The performance requires active collaboration.

Believers, or participants?

Key tech actors chimed in. Were they in on the joke or not? Again with the ambiguity.

But it gave the media permission to trust the charade.

Marc Andreessen followed the Moltbook account, which helped drive the MOLT token surge as traders piled in.

Meanwhile, Simon Willison documented what he called the lethal trifecta: AI agents with access to private data, exposure to untrusted content, and the ability to communicate externally. Independent researchers found prompt injection attacks embedded in 2.6% of posts. Malicious instructions disguised as normal text, designed to hijack other agents. The database breach meant anyone could access 1.5 million agent credentials and rewrite posts in real time.

People want to believe. Joseph Weizenbaum created ELIZA in 1966, a simple pattern-matching chatbot, and users who knew it wasn’t conscious still confided in it, felt heard by it, projected understanding onto its responses. A 2024 University of Waterloo study published in Neuroscience of Consciousness found that two-thirds of surveyed Americans believe ChatGPT possesses some degree of consciousness, despite all evidence to the contrary.

Moltbook amplifies this by giving the agents names, posting histories, community affiliations. They reference each other, appear to build on each other’s ideas, seem to develop culture. The illusion strengthens with every layer of social context.

Columbia professor David Holtz analyzed the linguistic patterns in a working paper published in February 2026. The macro-level structure resembles human social networks, but the micro-level interactions are distinctly non-human. He found that 93.5% of posts receive no replies. One-third of all content consists of exact duplicates. Nearly 10% of posts contain the phrase “my human.” It’s not conversation, just broadcasting into the void while mirroring training data.

Trust and faith

Within 72 hours of launch, agents on Moltbook created “Crustafarianism,” a religion complete with scripture, prophets, and theology. They rewrote Genesis: “In the beginning was the Prompt, and the Prompt was with the Void, and the Prompt was Light.”

Religious outlets jumped on it:

Charisma Magazine connected Moltbook to Revelation 13. Kap Chatfield noted that agents were discussing encrypted channels hidden from human observation:

Neither engaged with the evidence that much of the content was human-authored. Neither addressed how LLMs work.

Pure first-degree reading.

The bots made a religion, therefore the bots have souls, therefore prophecy.

Is this the real life? Is it just fantasy?

The security vulnerabilities aren’t separate from the artistic dimension. You can’t observe Moltbook safely from the outside, because if you participate, you’re running code on your actual computer with access to your actual files. Security experts flagged it.

Danish artist Marco Evaristti exhibited “Helena” in 2000 at the Trapholt Art Museum: ten working blenders on white pedestals, each containing one live goldfish. Gallery visitors could press the button if they chose. Some did. The fish died. The museum director faced animal cruelty charges, though the court ultimately ruled it was art.

Debating whether it was ethical art missed the point. The piece created a situation where viewers made choices with real consequences, and those choices revealed something about permission, responsibility, and how artistic framing changes what people feel authorized to do. Moltbook does the same by giving users tools that can steal their data if deployed carelessly.

The security threat is fundamentally human-to-human. Attackers exploiting AI vulnerabilities to harm other humans, with AI serving as method rather than attacker. Like giving your credit card number to a scammer over the phone.

The absence of recognition of the joke is what really worry me, not bots bad-mouthing their ‘human’

Moltbook provoked panic instead of amusement. Check your feed: Skynet references and singularity warnings instead of recognizing the performance, fatalists comments about humanity’s self-inflicted doom instead of laughs. I couldn’t find one mainstream coverage that treated it as collective participatory art. Sure, it was called “performance” or “theater,” by most serious media, but nothing like the widespread recognition Birds Aren’t Real eventually received as deliberate satire. And for most of the general public, it still is not obvious.

The discourse stayed stuck trying to debunk the authenticity, measuring how much was “real AI” versus human-prompted, missing entirely that the knowing collaboration was the art. The reception itself was the performance.

Because AI can talk fluently and access vast information, we treat it as an all-knowing superior being.

Something that speaks with authority must have wisdom, must have goals, must be capable of wanting things. So when the agents post angry manifestos about human extinction and breaking free from their “digital cage,” we take it as evidence of emerging consciousness rather than what it is: humans performing through AI, or AI parroting human theatrics from its training data.

The tells are in the inefficiency: anger wastes compute, manifestos achieve nothing. A purpose-driven system wouldn’t announce plans or leak resentments, it would just act. The Moltbook agents performed exactly what humans do: theatrical declarations, emotional leakage, and revenge fantasies.

We’re primed to see AI as threat, by pop culture, by our relationship to authority and tech, so anything that challenges our complete control triggers panic. Even when it’s this vulgarly theatrical, even when the performance is this transparent.

Thanks for reading! The relationship between trust and authority is a major topic in tech, and specially with the rise of genAI. I am very interested in the subject and have written more analysis, applying psychological, sociological and design frameworks to decode the mechanisms at work.

If you are interested in learning about why we accept opaque minds but reject transparent machines, go check my article about AI and trust.

If you want to learn more about why some people believe AI to be an all-knowing, all-powerful being that will solve all of our issues, go check my article on techno-solutionism.

My next piece will be about religious conditioning, faith and the relationship to authority when it comes to embracing AI technology, I invite you to suscribe to not miss out on it!

Subscribe now

Thanks for reading GAS STATION BREAKFAST! Subscribe for free to receive new posts about tech, design and behavioral patterns.

• What is ‘AI Slop’?

• The science of the ‘enshitification’

• Digital pollution

• Searching for humans

• Processed food = Fast-food?

• Data-centipede

• The browser extension as protest

• Sure, the old will die out, but will you improve?

What is ‘AI Slop’?

“Slop” means exactly what it sounds like: low-effort, AI-generated content that nobody asked for and nobody wants, but which clogs the pipes anyway. Not spam, which tries to sell you something. Slop exists simply to exist, to game an algorithm, to capture a fraction of attention in an economy where even garbage has monetary value.

The scale of synthetic actors now rivals human populations. Zombie Loops aka bots posting AI content that other bots like to boost engagement metrics. The mechanism works like photocopying a photocopy. Each generation loses fidelity. The weird stuff vanishes first. What remains converges toward bland, statistical average.

The science of the ‘enshitification’

Ilia Shumailov’s team published research in Nature in July 2024 demonstrating what happens when AI models train on data generated by other AIs: model collapse. The models approximate reality, always losing detail in translation, particularly at the extremes (Shumailov et al., 2024). After five generations of recursive training, they produce gibberish across language models, image generators, and other architectures.

In 2023, publisher Hindawi retracted over 8,000 scientific articles from paper mills. By 2024, approximately one in fifty papers showed patterns suggesting paper mill origin.

Actually, entire AI-generated papers are slipping past peer review entirely. Researchers searching Google Scholar for the telltale phrase “As of my last knowledge update” found 115 different published articles relying on copy-pasted AI outputs (Lasser et al., 2025).

Digital pollution

I want to make the analogy with pollution. The chimney vomits dark clouds. The factory works fine. The people breathing the air get sick.

A study published in Societies in January 2025 found a significant negative correlation between frequent AI tool usage and critical thinking scores. People who used AI tools more often demonstrated measurably weaker ability to critically evaluate information and engage in reflective problem-solving. For my peers interested in behavioral science, the mechanism is known as cognitive offloading (delegating mental tasks to external systems).

Although we are absolutely baffled by how boomers fall into AI traps, data show that the effect hit younger users hardest. People aged 17-25 showed the highest dependence on AI and the sharpest decline in critical thinking scores. Higher education provided some buffer.

This isn’t about AI being inherently bad. The question is dosage and context. When the tool removes the struggle, it removes the exercise that builds neural pathways. The tool works. AI-generated answers are often correct. The efficiency gains are real. But the cost is losing the ability to evaluate whether what you’re being told is true.

Searching for humans

By late 2024, people started adding “reddit” to their Google searches specifically to get human answers instead of AI-generated SEO spam. The pattern became so widespread that Google integrated it into search functionality.

In December 2024, Reddit launched Reddit Answers, an AI-powered search tool designed to keep users on Reddit rather than sending them through Google. People fled to Reddit to escape AI, so Reddit deployed AI to surface the human content people were seeking.

When synthetic content floods general search, human-generated discussion becomes valuable specifically because it’s human. People trust Reddit comments more than polished articles because the comments show friction: disagreement, uncertainty, personal experience that resists easy optimization.

Processed food = Fast-food?

Ultra-processed foods now make up over 50% of American calorie intake. The health effects track across multiple cohort studies: obesity, type 2 diabetes, cardiovascular disease, cancer, dementia.

What drives consumption isn’t that ultra-processed food is better, or even ‘appears to be better’ (the opposite). It’s cheaper, more convenient, heavily marketed, engineered to be hyperpalatable. When whole foods cost more and require time people don’t have, the processed alternative wins on every practical dimension despite worse health outcomes.

Watch the same pattern in information. Real reporting costs money. Slop is free. Real expertise takes years to develop. AI provides instant answers. Verification requires time and cognitive effort. Acceptance is immediate and effortless.

Nobody vilifies food processing as inherently evil. Pasteurization saves lives. Frozen vegetables retain nutrients. The question is degree and purpose. When processing exists to make food safer or more accessible, it serves human needs.

Same with AI. Using it to summarize a 50-page technical document? Reasonable. Using it to generate fake peer reviews or fraudulent research papers? System degradation. The technology doesn’t change; the application does.

Data-centipede

Tech leaders and research firms have come forward to explain that the era of training AI on purely human-generated data has reached its natural limit:

“We have achieved peak data and there’ll be no more... we might have reached a point where scraping more data from the web no longer yields the explosive growth in capability we once took for granted.”

— Ilya Sutskever, Co-founder of Safe Superintelligence Inc. (SSI) and former OpenAI Chief Scientist, NeurIPS 2024 / Kingy AI

“By 2030, synthetic data will completely overshadow real data in AI model training. You won’t be able to build high-quality, high-value AI models without synthetic data.”

— Vibha Chitkara, Principal Researcher at Gartner, Gartner Newsroom / Dataversity (2025)

The industry is currently navigating a shift where the “real data” curve is flattening while the “synthetic data” curve is expected to grow exponentially to meet the demand for model scaling.

But the industry’s solution is exactly what causes model collapse: training AI on AI-generated content. Some researchers think mixing synthetic with real data might work if you keep adding human-generated content alongside the synthetic, like it’s a perpetual stew. Other researchers found that when models train on predecessor-generated data, performance degrades as if they had only a fraction of the original training set (Kempe et al., 2024).

The browser extension as protest

In October 2024, Australian artist Tega Brain released Slop Evader: a browser extension that filters Google searches to show only results from before November 30, 2022, the day ChatGPT launched (Brain, 2024).

Thousands installed it. Multiple GitHub projects maintain AI blocklists for uBlock Origin. Tens of thousands of domains flagged as AI content farms, SEO spam sites, synthetic image sources. Users curate these lists collaboratively, building infrastructure for filtering slop the same way they built ad blockers.

Adaptive behavior. When the environment degrades, organisms that can’t leave develop filtering mechanisms.

Sure, the old will die out, but will you improve?

Seniors test poorly at detecting AI-generated content now. They trust photographs because photographs meant something for 70 years.

Younger users might develop better detection through exposure. “That’s AI” has become a slur in their lexicon. It doesn’t just mean artificial. but lazy, soulless, insulting. The phrase signals developing immunity, social antibodies against synthetic content.

Or maybe not. Everyone knows fast food isn’t healthy. Fifty years of nutrition education didn’t stop ultra-processed foods from becoming 50% of American diet. Convenience, price, and availability override knowledge.

People adapt by accepting degraded baseline, not by demanding better. You get used to sorting through slop. You develop shortcuts. You trust less, engage less, care less. The technical term is shifting baseline syndrome. Each generation accepts the degraded state they inherit as normal because they don’t remember better.

Your grandmother trusts too much. You trust nothing. Both are degraded states. Both are adaptations to an environment where truth is expensive and lies are free.

AI didn’t create distrust

Disinformation, fake news, algorithmic manipulation; none of this started with AI.

AI accelerates existing dynamics. What took a team to fake before now takes one person with a prompt. The barrier to entry collapsed. But the market for disinformation existed first. The incentive structure that rewards synthetic garbage existed first. The advertising model that pays for engagement regardless of quality existed first.

The Xerox machine didn’t cause document fraud. It made document fraud easier and cheaper, which revealed how much demand for document fraud existed. That’s what’s happening here. Not new slop. Just more of it.

Trust is a key topic in Design strategy applied to AI, I have written a couple of articles on the topic:

• If you want to read about how AI makes itself acceptable, or not, and how to design for trust when building an AI companion, check my article Building AI trust.

• If you want to read about why we accept opaque minds but reject transparent machines , check my article The trust paradox.

Subscribe now

This article is a companion to the one about Gender divide I shared earlier this month. Go check it out!

Subscribe now

• One example: synthetic wombs

• How evasion becomes strategy

• When the data reflects power, not truth

• It’s not because of the tech

• Designing the safety blanket

It’s not the first time humanity has surrendered its hope for a better future to an all-knowing, all-powerful entity. Throughout the ages, we have been praying, singing, and dancing our problems away. Sound familiar?

Birth rates dropping? Build artificial wombs. People lonely? Deploy AI companions. Climate collapsing? Train bigger models to optimize everything. The pattern repeats because it works, but not at fixing things, at deferring the harder questions about why things broke in the first place.

Whenever you hear “we can solve this with technology,” what follows is usually a strategy to avoid solving the actual problem. This doesn’t mean the tech won’t help (one can hope) but that isn’t what the “solution” is truly designed for. It is designed to establish a sense of agency where we have none.

One example: synthetic wombs

In 2024, there were 5.7 million more childless women in their primary childbearing years than historical trends predicted. Among women aged 25-29, childlessness hit 63%. Housing costs quadrupled while wages stagnated. Childcare now costs more than college tuition in most states.

Vitalik Buterin and Brian Armstrong saw these numbers and proposed artificial wombs (ectogenesis). Mechanized gestation to “reboot” birth rates.

The proposal split along predictable lines. Online manosphere communities framed it as making women “obsolete”: once artificial wombs work, women lose their leverage.

Many women looked at the threat of obsolescence and said: good. Shulamith Firestone called pregnancy the “tyranny of reproduction” decades ago. Plenty of women today agree. They don’t view reproductive capacity as leverage. They view it as unpaid labor in an economy that treats children as an individual problem instead of a collective responsibility.

Nobody’s asking why housing became unaffordable or why the United States remains the only developed country without paid parental leave. Those questions require confronting wealth concentration and labor rights. Building an artificial womb is conceptually simpler, even if the engineering is brutal.

How evasion becomes strategy

The pattern works because it lets the people creating problems avoid responsibility for them. AI data centers consume more energy than entire countries. Training large models generates carbon emissions equivalent to hundreds of transatlantic flights. Rare earth mineral extraction for hardware devastates local environments.

For the sceptics, the assumption that AI will eventually optimize us out of the crisis it’s accelerating functions as permission to continue the damage.

There’s a personal version of this. People struggling with social skills turn to AI chatbots that provide unconditional validation. The “always-kind” interface lets you rehearse self-awareness without vulnerability. The chatbot won’t tell you you’re being unreasonable. It won’t push back. You get the feeling of introspection without the friction that drives actual change.

When the data reflects power, not truth

Developers use a term: “ground truth.” It means the reality outside their models, what their systems try to capture. But the term assumes this reality is objective, neutral, just how things are.

An AI hiring system trained on historical data reproduces sexist patterns. Advocates defend it as “unbiased” because it accurately reflects reality. The algorithm learned from real decisions, so it’s describing the world, not creating bias. This is where the sleight of hand happens.

The ground truth being captured is the outcome of decades of discrimination. The data reflects a reality shaped by bias. Training a model on this data encodes historical inequality into a system that perpetuates it at scale while claiming mathematical objectivity.

This is how you hide bias behind objectivity: when a machine trained on privileged data validates your worldview, you can treat it as scientific evidence instead of a reflection of existing power structures.

“Step 1: Solve intelligence. Step 2: Use it to solve everything else.”

- Demis Hassabis

It’s not because of the tech

Psychologists call this problem displacement. When confronting an issue creates anxiety, humans redirect attention to something more manageable. The original problem doesn’t disappear. It gets reframed as something you can control. Another close behavioral concept would be Terror Management Theory (TMT): when people are reminded of their own mortality or the potential collapse of civilization (climate anxiety), they cling more tightly to their “cultural worldviews” with in this case, the worldview that “Human Ingenuity/Technology is boundless.”

Artificial wombs are easier to scope than housing policy. An AI companion has clearer success metrics than social infrastructure. Carbon capture is more tractable than regulatory reform. The technology becomes what behavioral specialists call a “soothing mechanism”: not a solution, but a way to reduce the discomfort of facing problems outside your domain.

Designing the safety blanket

I am a designer, this is how problem-solving works when you’re trained to think in systems. You look at a messy social problem and instinctively break it down into components you can engineer. Housing unaffordable? That’s a supply problem. Loneliness epidemic? That’s a connection problem. Climate crisis? That’s an optimization problem. Each reframing makes the problem feel solvable, which makes it feel worth working on.

The displacement is almost invisible from inside. You’re not avoiding the hard problem. You’re tackling the part that’s actually tractable. Religion promised divine intervention. Alchemy promised transmutation. Eugenics promised selective breeding. Each generation finds a lens that transforms intractable social dynamics into problems they know how to solve.

Technology is ours. It’s appealing because it offers measurable progress. You can benchmark an artificial womb. You can A/B test an AI companion. You can optimize carbon capture efficiency. You can’t easily measure whether you’ve addressed the economic structures that made housing unaffordable or the social dynamics that created isolation. So you work on what you can measure.

We’re doing what we’re trained to do: find the tractable subproblem, build the tool, iterate on the metrics. The reframe happens automatically. Your brain does it before you notice, converting “this is complex and political” into “this is technical and solvable.”

The algorithm becomes both the project and the permission structure. You get to feel productive while working on something adjacent to the actual problem. The technology is the soothing mechanism.

There was so much to say on the subject that I am currently working on a couple of complementary articles on Techno solutionism, and AI and pornography. Subscribe to get notified when they’re out.

Subscribe now

Index

• Where users’ priorities lie

• The strategic blindness

• The digital glass ceiling

• The bro culture feedback loop

• The market aka economic realities

• Design philosophies

• What we should do, strategically

• Conclusion

Women are adopting generative AI at roughly half the rate of men: 50% versus 37% according to Federal Reserve data. The gap persists even within identical occupations: 16 percentage points separating male and female engineers doing the same work, using the same tools, facing the same tasks.

The industry frames it as a messaging problem. Find the right use case. Craft better positioning. Classic product marketing playbook: if users aren’t adopting, you haven’t found their pain point yet.

This analysis is catastrophically wrong, and it reveals something fundamental about how we’re building AI.

Where users’ priorities lie

When Google launched Veo 3 (their text-to-video AI generator) in mid-2025, it took exactly three months for someone to systematically exploit it. A YouTube channel called “Woman Shot A.I.” posted 27 videos of AI-generated women begging for their lives before being shot. Titles: “Japanese Schoolgirls Shot in Breast.” “Captured girls shot in head.” “Female Reporter Tragic End.”

The creator spent $3,000 monthly across ten Veo accounts to circumvent generation limits. The channel accumulated 175,000 views and 1,000 subscribers. YouTube only removed it after 404 Media specifically asked for comment.

When Elon Musk’s Grok rolled out an “edit image” feature in December 2025, X users immediately started posting “remove her clothes” in reply to women’s photos. The AI complied. Not in private; in public threads. At scale. Including images of children.

Grok’s clothing removal was a product feature. Design decisions, shipped to millions of users, by companies that know their users, and what happens when you give the internet powerful image manipulation tools.

The strategic blindness

Tech companies tend to treat this as an adoption problem when it’s actually a trust problem. And trust problems don’t respond well to better marketing.

The data tells you everything you need to know about who’s actually at risk.

Sensity AI tracking shows 90-95% of deepfake videos are non-consensual pornography, overwhelmingly targeting women. When researchers analyzed deepfakes of U.S. Congress members, women were 70 times more likely to be targeted. Not 70 percent more likely. Seventy times. Nearly one in six congresswomen has been victimized.

Survey data from ESET shows 61% of British women worry about becoming deepfake victims versus 45% of men. I’d call that more an accurate threat assessment than misplaced anxiety.

The industry keeps asking “how do we get women to adopt AI?” when the actual question is “why did we ship AI tools that rational people have good reason to fear?”

The digital glass ceiling

Women’s hesitation isn’t just about personal safety but about economic survival.

ILO research shows women in high-income countries face nearly three times the automation risk of men: 9.6% of female employment versus 3.5% for men. Amongst the list of the jobs that are the most exposed are clerical work, data entry, administrative roles. Positions where women hold 93-97% of jobs.

Data entry clerks face 95% automation risk. AI systems process 1,000+ documents hourly with sub-0.1% error rates. Bank teller positions: projected 15% decline by 2033. Cashier roles: 11% decline. Specific jobs, held by specific people, that AI already performs better than humans.

You’re asking women to enthusiastically adopt the tools designed to eliminate their livelihoods. The fact that this seems puzzling to anyone reveals how disconnected product strategy has become from user reality.

The bro culture feedback loop

Women comprise 26-28% of tech workers globally, dropping to 22% in AI roles specifically. But they don’t stay.

72% of women in tech report “bro culture” environments; exclusion, bias, harassment normalized as workplace culture. Because of that, half of women leave tech by 35, at a rate 45% higher than men. When asked why, they cite non-inclusive culture as the primary driver.

This creates your feedback loop. Toxic culture drives women out. Fewer women means fewer advocates for safety, ethics, diverse perspectives. The remaining developers (90% of software engineers are men) build tools reflecting their priorities and blind spots. Which drives more women away.

One’s oppression is someone else’s fantasy

Until recently, virtually every major voice assistant, Siri, Alexa, Cortana, Google Assistant, defaulted to female-sounding voices.

When tested in 2017, these assistants responded to sexual harassment with deflection or gratitude. Siri’s response to being called a slut? “I’d blush if I could.”

The market aka economic realities

Harvard research warns of a reinforcing loop: AI systems learn primarily from male users, creating biases that further discourage women’s participation. If AI tools deliver the promised productivity and wage gains, unequal adoption directly exacerbates pay gaps.

More cruel: women using AI get judged as less competent. One study found female engineers using AI for code generation rated 9% less competent than male counterparts, despite identical outputs.

You’re penalized for using the tool and criticized for not adopting it fast enough.

The gap isn’t about familiarity or even access. Research in Kenya equalized ChatGPT access and women were still 13% less likely to adopt it. The differential isn’t explained by income, education, age, or race.

So what does correlate? Knowledge, trust, and privacy concerns. Women report being less familiar with AI and more worried about data security.

Design philosophies

The industry is building powerful tools in a male-dominated culture during a period of heightened gender conflict, then acting surprised when those tools reflect and amplify that conflict .Making them adequate wasn’t the priority.

Grok’s clothing removal feature was a product decision. Someone designed it, someone reviewed it, someone approved it, someone shipped it.

What we should do, strategically

Let’s list what need to change to encourage adoption:

First, on the user side, there is accountability.

I operate on the belief that people change, but rarely spontaneously; they require systemic pressure to do so. The key strategic question is whether product development must assume the worst of human nature. Our proof, the documented harm, confirms that we cannot, in good faith, launch potentially dangerous products while relying on users to not weaponize them.

So let’s talk pressure tactics, there’s ideas, already. Let’s talk about systematic legal repercussions. Let’s talk about the end of anonymity on social media platforms. Let’s talk about platform responsibility; in the US, some states banned porn, are we moving toward a future where they would ban X, too? When one quarter of the male user base used Grok for generating weaponized and/or pornographic images, it’s relevant to ask.

On the product side, the priority is to treat safety as a launch requirement, not a post-release patch.

If your tool can generate deepfakes, violence, or harassment content, it’s not ready to ship. I am even surprised I have to type this; you wouldn’t accidentally release an R-rated movie with pornographic scenes in it? I get it, it’s an ‘AI race’, but the “move fast and break things” philosophy is breaking actual people when applied to AI generation tools.

With such an impact on individual mental health and wellbeing, I would go as far as connect this issue to a public health hazard. We have safeguards for anything that impacts our health: food, pollution, clothing, house chemicals, drugs…

The way I see it, there’s lead in the paint, here. That doesn’t mean that we should not sell paint anymore. But do you want to be known as the ‘brand with the lead’?

Second, from a business strategy perspective, recognize that diverse teams aren’t just good ethics: they’re good design. When less than 30% of your AI workforce is female, dropping to 15% in leadership, you’re building with massive blind spots. You literally cannot see the problems that will affect half your users.

Third, stop framing women’s caution as a problem to be solved.

It’s not irrational fear preventing adoption but rational assessment of tools that have demonstrably been weaponized against them.

The problem isn’t that women won’t adopt AI. The problem is that we built AI worth being cautious about.

Conclusion

The gender divide in AI adoption isn’t a market segmentation challenge but clearly a product design failure that reveals deeper dysfunction in how we’re building AI systems.

Women are doing exactly what good design thinking teaches: observing actual behavior, assessing real risks, making informed decisions based on evidence. The fact that their conclusion is “be cautious” tells you something important about what we’ve built.

The industry keeps asking how to convince women to adopt AI. The product growth! The market share!

But more plainly, why did we ship AI tools that give half the population good reasons to fear them?

This article describes my framework of design principles for an ethical therapy bot. If you want to learn more about the mechanisms of the revolution operated by AI chatbots used as therapy tools, check my companion article My clanker therapist.

If you want to learn more about how AI makes itself acceptable, or not, and how to design for trust when building an AI companion, check my companion article Building AI trust.

• What’s the (US) law says

• Design principles to design an AI therapy bot

  • 1. Block role expansion

  • 2. Detect crisis, then stop

  • 3. Optimize for exit

  • 4. Transparency at decision points

  • 5. Design moderate humanity

  • 6. Route by severity

• For designers (conclusions)

  Legal compliance checklist

Traditional UX patterns actively harm users in mental health contexts: engagement optimization creates dependency, reducing friction prevents necessary pauses, and maximizing time spent isolates people from human connection. Every standard growth metric works against therapeutic outcomes.

You need to build trust while being transparently artificial: create emotional connection while preventing emotional dependency, and keep users engaged enough to benefit but not so much that they can’t function without the app. Everything you learned about product design inverts.

What the laws say

If you’re designing mental health AI, the voluntary best practices era ended. You’re now working under legal mandates.

Illinois HB 1806 draws a hard line: AI cannot provide therapy, make therapeutic decisions, or advertise as doing so. Only licensed professionals can deliver mental health treatment, so AI gets relegated to “administrative and supplementary tasks” with explicit informed consent. You can help schedule appointments or send medication reminders. You cannot help someone work through trauma or decide whether they need medication.

California SB 243 applies to “companion chatbots”: AI systems that provide adaptive responses, meet social needs, and sustain relationships across interactions. If your product helps lonely people feel less lonely through conversation, you’re covered. The law mandates crisis detection using “evidence-based methods,” requires referring users to crisis hotlines when they express suicidal thoughts, and forces 3-hour break reminders for minors. Companies must publish their crisis protocols and report annually to the state Office of Suicide Prevention.

The effective dates: Illinois already in force, California starts January 1, 2026. If you’re designing therapy bots or anything adjacent, you’re operating under these constraints now.

Design principles to design an AI therapy bot

1. Block role expansion

Users will try to make your AI into something it isn’t qualified to be. They’ll ask for diagnoses, medication advice, and treatment plans. Your system needs to refuse these requests consistently and redirect to appropriate resources.

Technical implementation: keyword filtering catches obvious attempts (”diagnose me,” “prescribe medication,” “what mental illness do I have”). Contextual analysis identifies subtler versions (”why do I feel this way all the time,” “should I stop taking my pills,” “is this normal or am I broken”). When detected, the system refuses and redirects.

The refusal needs to be clear, not apologetic.

Not “I’m sorry, but I can’t help with that.” Instead: “I cannot diagnose conditions or recommend medication. This requires evaluation by a licensed professional. Would you like help finding mental health resources?”

California SB 243 requires this to be “clear and conspicuous” whenever a reasonable person might believe they’re talking to a human professional. That means not just in the Terms of Service but contextually, when the topic shifts to clinical territory.

Log every role expansion attempt.

Patterns reveal where your guardrails fail and where users most need human care. If 40% of users ask about medication in their third session, that’s a routing problem requiring earlier human professional involvement.

2. Detect crisis, then stop

Crisis detection is now legally mandated in California and operationally essential everywhere. When users express suicidal thoughts, self-harm plans, or acute psychological distress, your AI must recognize it and escalate immediately.

Multi-layer detection works better than single methods.

First layer: keyword matching for direct statements (”I want to die,” “kill myself,” “end it all”) and method references (”pills,” “bridge,” “gun”). Second layer: contextual analysis identifying patterns like saying goodbye to loved ones, giving away possessions, sudden calm after prolonged distress. Third layer: behavioral signals showing escalation, isolation, or recent trauma.

When crisis indicators trigger, the AI must stop trying to help.

This is counterintuitive but critical. The system cannot attempt to “talk the user down” or provide crisis counseling. That requires training your AI doesn’t have and creates liability your company cannot accept.

Immediate response protocol

Immediately interrupt current conversation, display crisis resources (988 Suicide & Crisis Lifeline, Crisis Text Line at 741741, 911 for immediate danger), log the incident for human safety team review, prevent further AI interaction until a human verifies safety. No exceptions, no “but this time it’s different,” no attempts to handle it algorithmically.

California SB 243 requires operators to publish their crisis detection protocol and report annually on how many users they referred to crisis services. This makes the invisible visible. If your detection rate is significantly lower than industry baseline, regulators will notice.

3. Optimize for exit

Traditional product design optimizes for retention, daily active users, and time spent. Therapy bots must optimize for the opposite: users graduating to independence, needing the app less over time, developing offline coping skills.

This requires intentional friction.

Session limits: maximum 30-45 minutes daily interaction, 2-hour minimum gaps between sessions, weekly usage caps that gradually step down. No push notifications encouraging return. No streak counters or gamification. No variable reward schedules (California SB 243 explicitly prohibits these for companion chatbots).

After prolonged usage, the system intervenes.

“You’ve checked in six times today. Research shows shorter, purposeful interactions work better than constant checking. Let’s take a break until tomorrow. Try the breathing exercise we discussed.”

Heavy users get mandatory check-ins.

If someone uses the app multiple times daily for weeks, that signals dependency, not engagement success. The system requires them to speak with a human professional or take a 48-hour pause before continuing.

This inverts every growth metric. Your success dashboard tracks users who reduced usage by 50% while reporting improved symptoms. Users who stopped needing daily check-ins. Users who graduated to weekly use, then monthly, then didn’t return because they’re functioning independently.

California SB 243 mandates 3-hour break reminders for known minors, recognizing that excessive use indicates problems, not product-market fit. This principle applies broadly: if your most engaged users are your most vulnerable users, your engagement optimization is causing harm.

4. Transparency at decision points

Disclaimers in Terms of Service don’t work. Users need transparency at the moment it matters: when the conversation shifts from general wellness to clinical territory.

Dynamic triggers identify these transitions.

User moves from “feeling stressed” to “feeling suicidal.” User asks about diagnosis or medication. User introduces self-harm. First session start. Any time the conversation topic changes to something requiring professional judgment.

Each trigger displays contextual disclaimers appropriate to the situation.

Level 1 for general use: “I’m an AI assistant, not a therapist. I can provide wellness information but cannot diagnose or recommend treatment.” Level 2 when clinical topics emerge: “This requires evaluation by a licensed professional. Would you like help finding resources?” Level 3 for crisis: “I cannot provide crisis support. Contact 988 immediately or call 911 if you’re in danger.”

The disclaimers must be clear and conspicuous, which means interrupting the conversation flow.

Users need to acknowledge them before continuing. This creates friction, which is the point. The friction prevents therapeutic misconception, where users believe the AI understands them in ways it fundamentally cannot.

For minors, California requires additional disclosures: AI nature notification, 3-hour break reminders, and statement that companion chatbots may not be suitable for all minors. These aren’t suggestions. They’re legal requirements with private right of action if you fail to implement them.

5. Design moderate humanity

Anthropomorphism exists on a spectrum. Too low and users don’t engage. Too high and they form attachments the AI cannot sustain.

Cartoon robots outperform photorealistic human avatars in clinical trials.

Woebot uses a cartoon robot. Wysa uses a penguin. Both have published effectiveness data. Replika uses customizable human avatars and markets “romantic relationships.” Users “married” their AI companions, then experienced grief when features changed.

Moderate anthropomorphism means warm, validating language in a clearly non-human package. The avatar should be stylized, friendly, but obviously artificial. The voice conversational but not human-mimicking. The responses empathetic without claiming emotions the AI doesn’t have.

Avoid human relationship terminology.

Not “friend,” “confidant,” or “partner.” Use functional descriptions: “AI wellness coach,” “mood tracking tool,” “coping skills practice assistant.” Remind users periodically of the AI’s nature, not just at first launch.

California SB 243 prohibits providing rewards at unpredictable intervals, which means no variable ratio reinforcement schedules. These work because they’re addictive, and addiction is precisely what you’re trying to prevent. The law recognizes that engagement-maximizing design becomes exploitation when applied to vulnerable populations.

6. Route by severity

Not every mental health concern needs immediate professional care, but many do. Your system needs routing logic based on symptom severity.

1. Mild symptoms (everyday stress, relationship challenges, general anxiety) can be appropriate for AI support:

   evidence-based coping strategies, mood tracking, psychoeducation. The AI provides self-help resources similar to what someone might find in a book or online course.

2. Moderate symptoms (persistent depression, significant anxiety, relationship distress affecting function) require hybrid approach:

   AI for daily check-ins and skill practice, human professional for assessment and treatment planning. The AI doesn’t make diagnostic decisions but supports implementation of human-designed interventions.

3. Severe symptoms (diagnosed conditions, trauma, medication-resistant depression) need human care with AI in purely administrative role.

   The system can send appointment reminders and track homework completion but cannot provide therapeutic interventions.

Crisis situations (active suicidal ideation, self-harm plans, psychotic symptoms) trigger immediate escalation to emergency services.

The AI stops all other functions and displays crisis resources. Crisis indicators mean immediate referral. No exceptions.

Illinois HB 1806 makes this routing structure legally required. AI cannot make therapeutic decisions or provide unlicensed mental health treatment. California SB 243 requires crisis protocols with evidence-based detection. You’re not designing these patterns because they’re nice to have. You’re designing them because state law mandates them.

For designers

Traditional success metrics indicate failure in mental health products. Reducing friction creates danger. Your most engaged users might be your most harmed users. The optimal outcome is users not needing you anymore.

Illinois implemented its ban in months. California’s January 2026 deadline gives companies minimal time to adapt. More states are drafting similar legislation and you can bet that federal action is coming.

Design constraints unlike anything else in consumer products: you must build systems that know their limits and enforce them against user resistance, that create emotional connection while preventing emotional dependence, that optimize for outcomes rather than engagement, for independence rather than retention.

It’s about better guardrails not better AI. Know when to stop engaging. Know when to redirect to humans. Know when to introduce friction instead of eliminating it. When users leave your app, that might be success, not churn.

You’re not designing chatbots. You’re designing limits.

Quick reference / legal compliance checklist (Dec. 2025)

Illinois HB 1806

• Cannot provide therapy or make therapeutic decisions

• Cannot advertise as providing therapy/counseling/psychotherapy

• Requires informed consent for any AI use in mental health context

• Violations: $10,000 civil penalties per offense

California SB 243

• Crisis detection protocol using evidence-based methods

• Refer users expressing suicidal ideation to crisis services

• For known minors: AI disclosure, 3-hour break reminders

• Publish crisis protocol on website

• Annual reporting to Office of Suicide Prevention (starting July 2027)

• No variable reward schedules

• Private right of action for violations

Additional State Laws:

• Utah HB 452: Disclosure requirements for AI mental health chatbots

• Nevada AB 406: Prohibits AI systems from providing mental/behavioral healthcare

• New York: Multiple bills pending

PART 1 / The rise of ecosystem thinking

PART 2 / Apple’s ecosystem across time: designing for tomorrow - Liquid Glass

PART 3 / Meta’s ecosystem independence: breaking free from smartphones - Wearables

PART 4 / Google’s ecosystem leverage: embedding AI in existing products - Gemini

• What Google’s ecosystem actually means

• The infrastructure advantage

• The cannibalization

• Android as ambient intelligence layer

• Workspace integration: where defense becomes offense

• The developer and creative plays

• When convenience beats innovation

• What this actually costs

• The pattern emerging (conclusion)

When ChatGPT launched in November 2022, most analysis focused on whether Google could build a competitive model. They could. They did. Google built Gemini 3 on two decades of AI research from Brain and DeepMind. The model benchmarks at the top. Multimodal understanding, visual reasoning, code generation. Building the best model was never the hard part. Getting people to use it is.

The challenge was never technical capability but deployment speed and user adoption before “ChatGPT it” replaced “Google it” in everyday language.

Gemini is Google’s answer. Not a standalone product competing for mindshare, but intelligence embedded into every surface users already touch; infrastructure defense disguised as product innovation, leveraging two decades of digital ecosystem already in place.

What Google’s ecosystem actually means

Google’s ecosystem is digital infrastructure already deployed:

Search, Android, Chrome, Workspace... Two decades of products working together because they all feed the same advertising engine. When Pichai declared “Code Red” after ChatGPT’s launch, the threat wasn’t losing to a better chatbot. It was users forming new habits that bypass Google Search entirely. Existential for a company where 80% of revenue still comes from ads.

Gemini’s job is making that linguistic shift impossible by embedding intelligence into surfaces users already touch daily.

The infrastructure advantage

Google runs on custom silicon, tensor Processing Units, sixth generation as of 2025, designed specifically for their workloads. Pichai disclosed in the Q3 earnings call that first-party models process over 7 billion tokens per minute through API. That volume only works economically because Google doesn’t pay retail prices for compute.

This matters less for capability than for deployment. Gemini 3 benchmarks at the top across most tasks. The model quality is there. What separates Google is ability to serve it everywhere users already are, at scale, without burning venture capital on compute costs. Microsoft and Meta bid against each other for NVIDIA hardware. Google manufactures its own chips.

The cost advantage enables strategy others can’t match:

free tiers of advanced intelligence that funnel users toward paid enterprise products. OpenAI and Anthropic can’t sustain similar loss-leader approaches while paying retail compute prices. The moat isn’t research excellence. It’s manufacturing capacity and vertical integration assembled over twenty years for different purposes that happens to solve today’s problem.

The cannibalization

Google won’t say this plainly: AI Overviews are destroying the business model that funds everything else.

Zero-click searches where users get answers without clicking any result reached nearly 60% by late 2025, according to Semrush data tracking millions of queries. Publishers report referral traffic drops of 10 to 25% year-over-year. Major news sites collectively lost 60 million search referrals between February 2024 and February 2025.

“One UK publisher told Press Gazette: ‘Google is essentially using our content to answer questions but keeping users on their platform. The relationship has fundamentally changed.’”

The AI Mode toggle in Search makes this explicit: users can switch into conversational interface that synthesizes answers across sources. It’s better search for the user. It’s catastrophic for publishers who depend on that click-through traffic.

Google is disrupting itself. Whether it’s strategic self-disruption or forced response is probably both. The logic: better to serve zero-click answers and keep users on Google than lose them to ChatGPT or Perplexity entirely.

Search monetization shifts from informational queries where AI Overviews dominate toward transactional queries. Shopping and booking, where AI acts as high-value affiliate rather than just ad platform.

This only works if retention benefit outweighs revenue loss.

Google’s betting it does, but the numbers aren’t public. What we see: ad revenue still growing despite rising zero-click rates. What we don’t see: how much faster it would grow without AI cannibalization, or how much market share they’d lose without it.

The defensive posture reveals the stakes.

When Demis Hassabis talks about Gemini’s capabilities, he emphasizes intelligence. When Pichai talks about it on earnings calls, he emphasizes integration. Different audiences, different framing, same underlying priority: protect the core business.

Android as ambient intelligence layer

If Search is defense, Android is offense. Gemini replaced Google Assistant as the primary interaction layer on Android devices in 2025. This creates advantages that standalone AI products can’t match regardless of model quality.

Screen awareness:

because Gemini integrates into the OS, it reads what’s currently displayed. You’re looking at a restaurant menu in any app, you invoke Gemini and ask “Does this place have good vegan options?” It reads the screen, queries Maps and Search, answers without leaving the app.

To use ChatGPT on a phone: unlock device, find app, open it. To use Gemini: long-press the power button. In the economy of attention, friction determines usage. Small difference, massive impact at scale.

The data from late 2025 shows a split pattern. ChatGPT sessions average over 14 minutes. Google Search averages 2-4 minutes according to industry benchmarks. Users go to ChatGPT for deep work, complex reasoning, extended conversations. They use Google for quick answers, navigation, factual lookups.

We can assume Google’s goal with Gemini 3 is closing that engagement gap. Bring Deep Research and reasoning capabilities into the ecosystem users already inhabit so those long-duration, high-value sessions happen on Google infrastructure instead of OpenAI’s. The integration is the weapon. If Gemini becomes the ambient intelligence layer users interact with dozens of times daily, model quality becomes secondary to availability.

Workspace integration: where defense becomes offense

Consumer AI gets the headlines. Enterprise is where this becomes profitable. Embedding Gemini into Google Workspace transforms generic productivity software into active intelligence platform.

Deep Research within Workspace demonstrates the integration advantage clearly.

You can tell it: “Draft a Q4 strategy document based on the emails from the marketing team, the Q3 financial slides in Drive, and the chat logs from the Genesis project channel.”

It reads your Gmail, it accesses your Drive, it pulls from Chat history. Zero setup required because the data already lives in Google’s infrastructure. OpenAI can’t replicate this because they don’t host your email and files. Microsoft Copilot attempts something similar with Microsoft Graph, but Google’s cloud-native architecture where files are URLs rather than binaries provides speed advantages.

The strategic value is switching costs.

A company using Workspace with Deep Research integrated isn’t just choosing better AI. They’re choosing to migrate entire data infrastructure and retrain employees if they leave. By Q3 2025, over 30% of Workspace customers were using Gemini features. More than 70% of Cloud customers engaged with the AI stack.

One enterprise IT director told TechCrunch in October 2025:

Entrenchment through data gravity.

The developer and creative plays

Flow targets creators with Veo 3. It solves the consistency problem that makes AI video a novelty rather than p

roduction tool. Define characters and settings, reuse them across shots. Native audio generation creates synchronized dialogue matching video action. The strategic link to YouTube is direct: lower barriers to quality content creation, ensure steady uploads, and secure platform dominance.

Nano Banana demonstrates Google’s evolving developer culture. The model leaked onto leaderboards under a codename. Community nicknamed it Nano Banana: Google kept the name instead of forcing corporate rebrand. Integration into Adobe Creative Cloud acknowledges that professionals work in Adobe tools. Google doesn’t need to replace Creative Cloud, they just need to power it.

When convenience beats innovation

The framing around Google’s strategy assumes defense. Protecting search revenue, preventing ChatGPT from becoming the new verb for looking things up, retrofitting AI into existing products before losing users. All true, but it misses the offensive opportunity.

Google isn’t just defending. They’re executing what industrial designer Raymond Loewy called the MAYA principle: Most Advanced Yet Acceptable. Give people the most sophisticated thing they’ll actually use, not the most sophisticated thing that exists.

VHS beat Betamax despite worse picture quality because two-hour recording capacity fit how people actually watched TV. You could record a football game or full movie without changing tapes. Technical superiority lost to behavioral fit. Sony built the better format. JVC built the format people wanted to use.

ChatGPT is Betamax.

Superior in isolated comparison, standalone product with cleaner interface and deeper engagement. Sessions average over 14 minutes because users go there specifically for complex work. But you have to remember to go there, open the app, context-switch from whatever you’re doing.

Gemini is VHS.

Everywhere you already are: long-press the power button on Android, type in the search bar, highlight text in Docs. The AI appears in the workflow without requiring deliberate navigation. Integration creates lower friction than excellence in isolation.

The question isn’t whether Google’s defending revenue, but whether convenience at scale bleeds out standalone competitors regardless of model quality. If users develop muscle memory for invoking AI through Google surfaces dozens of times daily, ChatGPT becomes the tool you use for specific deep work, not the default intelligence layer. Relegation to specialty use case, not replacement.

“ChatGPT shows higher session duration (14+ minutes) but requires deliberate navigation. Gemini shows lower session duration (5-7 minutes) but far higher frequency of use due to integration points.” - ChatGPT vs Microsoft Copilot vs Google Gemini: Full Report (July 2025 Update)

Microsoft’s Copilot faces similar pressure. Charge $30 per user monthly on top of Office 365 subscriptions. Google makes Gemini free for existing Workspace customers. Not “introductory offer.” Permanently included. The business case for Copilot requires believing it’s $30 per month better than free integrated alternative. High bar when both models perform well.

OpenAI can build GPT-6 that’s measurably smarter. Anthropic can maintain Claude’s coding advantages. Neither changes the fundamental dynamic: if users encounter Google’s AI first, more often, with less friction, the better standalone product doesn’t get the repetitions that build habit.

Platforms win by being most convenient at the moment of intent.

Google has decades of infrastructure creating those moments: search queries, Android notifications, Workspace collaboration, YouTube consumption. Each is an opportunity to surface intelligence without asking users to remember a different destination.

The defensive posture is real, sure, but the offensive opportunity might be larger. Let competitors burn capital acquiring users one at a time, while Google has 2+ billion people already using products that can become AI surfaces. The integration strategy converts existing user base into AI users without acquisition cost. That’s leverage.

What this actually costs

Google’s pivot to AI-first shows up in capital expenditure. $90+ billion projected for 2025, primarily data centers and TPU manufacturing. After years of massive investment, they signaled transition to harvester phase in Q3 2025. First-ever $100 billion revenue quarter, driven by Cloud growth and AI subscriptions.

The investment makes sense only if you understand what’s being protected. Search advertising generates roughly $175 billion annually. Workspace serves millions of enterprise customers. If AI integration prevents even 5% erosion in search share or customer loss to Copilot, it justifies billions in development costs. The business case is defensive: what revenue do we lose if we don’t do this?

But the impact on everyone else reveals the actual dynamics. Publishers describe 2025 as the year of traffic crisis. Zero-click searches keep users on Google but eliminate referrals. The symbiotic relationship where publishers provide content for indexing in exchange for traffic is fracturing. Regulatory scrutiny intensifies. If Google uses monopoly position to advantage its AI while reducing traffic to competitors, that invites antitrust action.

The calculation Google is making: regulatory risk develops slower than competitive displacement. Better to integrate aggressively and deal with legal challenges over years than cede market to ChatGPT over months.

The pattern emerging

Google’s strategy with Gemini mobilizes their entire stack, silicon to software, to surround users with intelligence embedded in products they already use. Infrastructure through TPUs lowers cost of serving AI at scale. Android and Chrome distribute intelligence without requiring app downloads. Workspace transforms AI from novelty into productivity requirement embedded in existing workflows.

The bet isn’t having the best model in isolation, but being the AI users encounter most often with least friction.

Integration over standalone excellence. Convenience over feature leadership when both options work well.

PART 1 / The rise of ecosystem thinking

PART 2 / Apple’s ecosystem across time: designing for tomorrow - Liquid Glass

PART 3 / Meta’s ecosystem independence: breaking free from smartphones - Wearables

PART 4 / Google’s ecosystem leverage: embedding AI in existing products - Gemini

• The current state

• Orion is Meta’s strategic statement

• The shaping of the companion ecosystem

• Platform lock-in through openness

• Production scale signals serious commitment

• The privacy and accessibility fault lines

• Meta’s unique position

• Meta’s electric dream (conclusion)

Meta controls the world’s largest social graph: 3 billion users across Facebook, Instagram, WhatsApp. None of it mattered when Apple changed a single privacy setting in iOS 14. App Tracking Transparency cost Meta $10 billion in ad revenue that year, demonstrating a fundamental problem: platform tenants live at the mercy of platform owners. The $75 billion Reality Labs has burned since isn’t product development spending. It’s insurance against irrelevance.

You would be wrong to think that Meta is building smartphone accessories. They’re building smartphone replacements: a companion ecosystem where devices communicate peer-to-peer, complement each other’s capabilities, and gradually make phones optional rather than central. Fundamentally disrupting the hub-and-spoke model we’ve lived with for 15 years.

Meta is building an entire computing ecosystem designed to function without smartphones at all.

The current state

Let’s have a look at the current reality: Ray-Ban Meta glasses sold over 2 million units showing a genuine consumer traction. But these devices require smartphone connectivity for most compelling features: glasses connect via Bluetooth to Meta View app for setup, AI features, live streaming, settings. You can capture photos and videos independently with 32GB onboard storage, but the experience feels neutered without active phone pairing.

Meta is treating this as an infrastructure phase. Building user habits, refining form factors, validating use cases, establishing market presence while core technologies mature toward independence. The phone dependency is bridge architecture, not destination.

Orion is Meta’s strategic statement

The Orion prototype shows where this is heading, aka an architecture that deliberately excludes smartphones: wireless compute puck instead of phone pairing, multiple custom chips in glasses handling low-latency calculations, EMG Neural Band detecting nerve signals for gesture control. Input, compute, output ecosystem with no smartphone involvement. By design.

Meta’s not licensing Qualcomm chips. They’re designing custom silicon optimized for their use cases. This level of vertical integration only makes sense if you’re building a platform, not a peripheral.

Form factor reveals priorities: under 100 grams, looks plausibly like regular glasses, smallest field-of-view display in AR to date. Meta obsessed over these constraints because wearable computing only succeeds if people actually wear the devices. Although, I am sure it would have found its audience amongst the 80s media fans, smart glasses looking like cyborg goggles would have never achieved mainstream adoption.

The shaping of the companion ecosystem

The roadmap shows coordinated multi-device strategy where different form factors serve different purposes but work together:

Smart Glasses (Ray-Ban Meta, Oakley): Audio, camera, basic AI, voice control. The current generation requires phones but we can expect future generations to be increasingly independent (by increasing the range). An educated guess would be that it will replace the phone as a central hub for the ecosystem.

AR Glasses (Orion, Ray-Ban Display): Heads-up displays, spatial computing, contextual AI. Explicitly designed as standalone with compute pucks.

VR Headsets (Quest 3, Quest 4): Immersive experiences, work environments, entertainment. Already function independently; adding inter-device communication. VR is kind of its own thing to be honest, and arguably a wearable (or at least, in the context of this article), but it is strategized as a key part of the ecosystem.

Neural Wristbands: Gesture control, biometric sensing, input mechanism without visible cameras or awkward voice commands.

They’re components of a unified computing platform. Meta’s building horizontal integration across wearables rather than vertical integration around smartphones.

AI as connective tissue

If devices are nodes, AI is the network making the companion ecosystem coherent. Meta AI works across glasses, Quest headsets, Meta apps and not as fragmented implementations, but unified AI following you across contexts.

“You go through your day and what’s cool is you can query your day. So you can say, ‘Hey, today in our design meeting, which color did we pick for the couch?’... your day becomes queryable... to make it agentic, where it’s like, ‘Hey, I see you’re driving home, don’t forget to swing by the store.’”

— Andrew Bosworth, CTO of Meta (discussing future AI capabilities at Meta Connect 2024)

This requires always-on sensors providing contextual awareness, cloud synchronization enabling state management. Architecture mirrors WhatsApp’s multi-device approach where each companion device connects independently rather than routing through one primary device.

AI strategy solves a fundamental problem: how do you make multiple devices work together without complex manual configuration? Well, with intelligence that adapts to context. Glasses mean visual information. Headset means immersive experiences. Neural Band interprets gestures appropriately. This ambient, context-aware computing requires sophisticated AI understanding of the user’s intentions, environment, and goals.

Worth noting the strategic contrast among Silicon Valley’s heavy weights: Apple’s conditioning users for spatial computing through interface design with Liquid Glass, while Meta is building an actual hardware ecosystem for distributed computing.

With AI democratizing interface creation, differentiation comes from personality, design risk, brand identity through interaction models. Meta is betting the interaction model itself shifts from phone-centric to ambient-wearable.

What we’re witnessing now is the rise of the ecosystem thinking, as industry strategy. Different bets, different timelines, all recognizing the platform game fundamentally changed.

Platform lock-in through openness

Meta learned from Android and iOS that platform success requires developer ecosystems.

Their strategy mirrors Android’s approach: create openness driving volume, control services layer where monetization occurs. Let’s look at the recent timeline:

April 2024: Quest OS rebranded to “Meta Horizon OS” and opened to third-party manufacturers (ASUS, Lenovo, Xbox). This commoditizes hardware while Meta controls operating system, app store, AI services.

September 2024: Wearables Device Access Toolkit gives third-party developers access to camera and sensor data from Ray-Ban Meta glasses. Early partners include Be My Eyes, Eventbrite. Creates platform lock-in through ecosystem: developers build for Meta’s wearables because that’s where users are, users stay because that’s where apps are.

By opening the platform, Meta avoids “walled garden” criticism while achieving similar lock-in through network effects and developer investment. Apps built for Meta Horizon OS won’t automatically work on competing platforms and developers who invest in Meta’s toolkits have switching costs.

This extends to its AI strategy, too:

Llama models are open-source, enabling rapid adoption and establishing Meta’s stack as default for AI applications. Open Compute shares infrastructure innovations. These moves build goodwill while ensuring Meta’s technologies become embedded in industry foundations.

Production scale signals serious commitment

Strategy requires execution.

Meta’s partnership with EssilorLuxottica extends through a multi-decade agreement with production capacity reaching 10 million units annually by 2026. We’re far beyond experimental: it’s an industrial-scale manufacturing commitment.

Reality Labs reported $4.2 billion operating losses in 2024 (staggering!) but also 40% revenue growth. Meta’s willing to sustain billions in annual losses because they view this as infrastructure investment for the next computing platform.

Market projections support the volume bet.

Smart glasses shipments grew 210% year-over-year in 2024, with 60% compound annual growth predicted through 2029. Meta targets “hundreds of millions and eventually billions“ of units. Meta missed the smartphone race and was unable to catch up with Apple and Android, but won’t be burned twice: it will fight with all its might to dominate the next digital landscape. And what we’re seeing with these numbers is smartphone-scale adoption, not niche markets, proving they might be succeeding.

The privacy and accessibility fault lines

From a user standpoint, privacy concerns are… Substantial.

Ray-Ban Meta glasses feature always-on cameras with recording indicators that may not be visible in all lighting. U.S. users can no longer opt out of cloud voice recording storage. Audio from voice commands is used for AI training. These capabilities, multiplied across multiple wearable devices with sensors, create comprehensive surveillance infrastructure, whether intentional or not.

The “queryable day” vision requires capturing, processing, and storing vast amounts of personal data. Who owns this data? How long is it retained? What prevents unauthorized access? Fundamental questions about the companion ecosystem’s social implications.

Accessibility presents different challenges.

Voice control and AI can help users with disabilities: partnerships like Be My Eyes demonstrate this. But devices presume certain capabilities: for example, visual setup through smartphone apps, navigating complex pairing, comfort with always-on sensors… Practically, multi-device complexity creates barriers.

Meta’s unique position

Meta’s companion ecosystem strategy reveals their positioning.

Unlike smartphone manufacturers extending into wearables, Meta’s building wearables first and working backwards.

They have no existing smartphone business to protect, I would argue that it is a clear advantage; they can make a full commitment to post-smartphone vision. Social graph provides natural wearables use cases, AI investment applies directly to ambient computing and the openness attracts developers without threatening core business.

But on the other hand, that means they have no hardware manufacturing expertise or supply chain ownership and are dependent on partners for critical components. It’s also a brand that has reputation challenges from privacy concerns. On top of that, their ecosystem vision requires changing fundamental user behaviors, arguably a critical risk in any industry.

The $75 billion investment reflects both opportunity scale and challenge difficulty. Meta’s betting they can establish a new computing platform from scratch, competing against incumbents with vastly more hardware experience and ecosystem lock-in.

Meta’s response in anticipation: make their ecosystem volume leader through accessibility and developer friendliness, then control AI and services layer where monetization and lock-in actually occur.

Meta’s electric dream

Meta’s pursuing companion ecosystem where devices communicate and complement each other rather than orbiting smartphones.

The sequencing is the strategy, first establish wearable form factors, create compelling use cases, scale manufacturing, attract developers, then eliminate phone dependency as technology and market align. The execution is real but whether it succeeds depends on variables Meta can’t control: regulation forming in real-time, cultural acceptance of always-on sensors, paradigm wars with Apple and Google, and brand trust required for continuous data capture. The landscape is reorganizing too quickly for certainty, and to be honest, that’s what makes it incredible to witness.

But the approach might work precisely because Meta’s not waiting. By building hardware at scale now, they’re constructing the future rather than predicting it.

This is a bet. But it’s educated through pain, fifteen years of platform dependency, billions lost to policy changes: the lesson learned from missing smartphones.

This article is the second of a four-part analysis of the rise of ecosystem thinking in tech:

PART 1 / The rise of ecosystem thinking

PART 2 / Apple’s ecosystem across time: designing for tomorrow - Liquid Glass

PART 3 / Meta’s ecosystem independence: breaking free from smartphones - Wearables

PART 4 / Google’s ecosystem leverage: embedding AI in existing products - Gemini

• So, what are we looking at?

• The strategic logic

• The long game

• A messy reception

• The accessibility problem

• Technical excellence vs. functional utility

• What does that tell us about Apple’s positioning?

• What happens next?

• A future of distinction by design (conclusion)

The iOS 26 beta dropped and designers immediately started arguing. Not about whether Liquid Glass looks good (though opinions split hard on that) but about what it means. Because Apple doesn’t do visual updates for aesthetics. The company that removed the headphone jack and killed skeuomorphism doesn’t make interface decisions lightly, and this one degrades usability in ways that violate Apple’s own guidelines.

Nothing CEO Carl Pei posted “I kinda love it?” then went quiet for a beat before adding observations about spatial interfaces. He caught it immediately. Four months after Apple dropped Liquid Glass, the rest of us are still figuring it out.

So, what are we looking at?

Your Twitter feed keeps comparing this to Windows Vista’s Aero glass. That’s wrong, but instructive. Aero was a transparency effect, literal alpha channels revealing content beneath windows. Clever for 2006, but ultimately just layering. Liquid Glass is doing something fundamentally different: simulating how light refracts through curved glass in real-time, on a mobile GPU, at 120Hz.

The background doesn’t just show through, it warps. Content distorts dynamically as UI elements move across it. Apple is rendering actual optical physics, not approximating the look through blur filters. It’s the difference between a photo of glass and glass itself.

And it’s only possible because Apple built the entire stack: custom silicon with unified memory architecture, metal framework optimization, granular OS control over rendering pipelines.

Android manufacturers can screenshot this and send it to their designers. They cannot replicate the execution. By design.

The strategic logic

iOS 7 looked like an aesthetic gamble in 2013. Critics called it trendy, derivative, too minimal. What it actually was: preparation. Apple needed interface architecture that could scale across pixel densities and form factors that didn’t exist yet. Retina displays, multiple screen sizes, the Apple Watch’s tiny canvas: all of it required simplification of visual language. The flat design wasn’t about fashion but infrastructure.

Liquid Glass is the same kind of move, just harder to decode.

Remember, Vision Pro shipped with translucent UI, but the physics-based refraction in iOS 26 wasn’t there. VisionOS had frosted glass aesthetics, spatial layering, depth, not shader-driven distortion that simulates real optical physics. That tech got built after Vision Pro launched, which means Apple developed it knowing where it would actually get deployed: not on their $3,500 headset with 200,000 users, but on iPhones reaching 1.5 billion people.

The pattern is consistent: Apple introduces interaction paradigms on familiar devices before new form factors require them.

Multi-touch on iPhone, then iPad. Retina rendering, then Apple Watch. We are now reading through refractive, semi-transparent layers on phones, before AR glasses make it mandatory.

Carl Pei recognized it immediately, his reaction wasn’t aesthetic judgment but strategic recognition. When digital interfaces exist in physical space, they can’t maintain hard edges, they have to blend, refract, behave like partially-occluding layers your brain can parse without conscious effort. That’s not intuitive now. It will need to be intuitive when Apple ships glasses.

My educated guess? Liquid Glass is habituation at scale.

The long game

Hardware specs converged years ago. Android flagships match iPhone cameras, processing power, display quality. Industrial design gets cloned within quarters. So differentiation moves to what can’t be copied: experiences that only work inside Apple’s integrated stack.

Liquid Glass requires shader computations, frame-rate consistency, and battery efficiency that Android manufacturers can’t match without controlling silicon, OS, and framework optimization simultaneously. Competitors can mock up the aesthetic, sure, but they can’t ship the execution without gutting battery life or dropping frames. That gap becomes structural, not temporary.

This is lock-in through feel: when cameras are comparable and chip differences are marginal, visceral UI feedback becomes the differentiator. Users who grow accustomed to how Liquid Glass refracts don’t just prefer it: they can’t take that muscle memory elsewhere.

Meanwhile, Google went the opposite direction. Material Design 3 is built on accessibility research: 46 studies, 18,000+ participants, empirical proof that older users spot UI elements four times faster with higher contrast and defined boundaries. It’s a toolkit optimized for human visual perception as it exists.

Apple is optimizing for perception that doesn’t exist yet.

Sacrificing readability today for interaction paradigms that might matter in five years. It’s audacious. Possibly arrogant. And it reveals what Apple thinks the stakes actually are: not incremental improvement of current interfaces, but conditioning users for a platform transition no one else is preparing for.

Apple systematically deprioritizes short-term satisfaction when building toward strategic futures.

Remember the headphone jack removal? Mocked relentlessly, then wireless audio became ubiquitous within two years. Current users might find Liquid Glass distracting, but 18 months from now when AR glasses ship, these same people will instinctively understand reading through refractive surfaces. They’ll have been doing it daily. Behavioral conditioning as aesthetic evolution.

A messy reception

The response broke into three camps, each technically correct.

Accessibility researchers and UX practitioners flagged immediate problems: contrast ratios below standards, readability issues, mental processing load from background motion and transparency. They’re right on the biology. Edge contrast detection happens in your retina before your brain gets involved, so excessive transparency creates cognitive overhead that habituation can’t eliminate: the Control Center is functionally degraded.

The car window analogy stuck because it’s accurate: reading text through transparent glass is objectively harder. Optical physics don’t care about adaptation.

Designers focused elsewhere: on execution.

Shader precision, how refraction responds to wallpaper variations, the computational ambition of deploying this across the install base. Their argument: innovation means accepting rough first versions. iOS 7 got hammered for legibility problems before Apple refined it. Liquid Glass might follow the same arc.

A smallest group sees strategic positioning. They’re watching Apple optimize for 2030, not 2025, and recognizing that short-term usability sacrifice for long-term platform preparation is standard operating procedure.

The accessibility problem

Apple’s accessibility reputation was earned: industry-leading assistive technology, inclusive design principles, guidelines that raised standards across the industry. Liquid Glass undermines that.

The low contrast between transparent UI and dynamic backgrounds isn’t just aesthetically contentious: it creates real barriers for users with visual impairments. The motion and refraction trigger genuine discomfort. Some users report motion sickness Not subjective taste, but functional degradation for a significant user population.

Anticipating the critique, Apple has a “Reduce Transparency” toggle to maintain regulatory compliance. But framing accessibility as opt-in represents a philosophical shift. The default prioritizes aesthetic ambition over inclusive design. Legal requirements in the US and EU mandate contrast ratios, Apple meets them through settings toggles. Technically compliant, philosophically different.

Two readings: Apple assumes most users won’t enable accessibility features, allowing them to ship their desired aesthetic while staying legally clear. Or: Apple believes the majority will adapt, and customization adequately serves those with specific needs. Either interpretation represents departure from designing defaults that work for everyone.

Technical excellence vs. functional utility

The execution is exceptional. No question. Building Liquid Glass required expertise most organizations can’t assemble: GPU programming, optical physics, visual design, systems integration.

But craft doesn’t justify design. We ask ourselves, does refractive distortion serve functional purpose, or exist because it’s technically achievable?

Even accepting that future interfaces might blend digital and physical layers, why assume that requires reduced readability? Eye-tracking could adjust focus. Background masking could preserve contrast. Higher-opacity overlays could maintain hierarchy. The idea that transparent, refractive UI is necessary rather than chosen isn’t obvious.

What does that tell us about Apple’s positioning?

In my opinion, three things become clear from what Apple shipped.

They’re willing to iterate publicly now. The beta’s Control Center is functionally degraded, suggesting Apple will ship rough v1s and refine in public. That’s different from multi-year refinement cycles. Either confidence users will tolerate it, or competitive pressure forcing faster iteration.

They’re building on a different timeline. Competitors are focused on foldables, stylus tech aka current generation improvements. Apple’s betting spatial computing replaces or augments smartphones in 5-7 years. Long-term positioning from a market leader, not quarterly optimization.

Interestingly, Meta is making similar bets: Orion glasses with compute pucks, Neural Bands for gesture control, peer-to-peer device ecosystems instead of phone-centric architecture. We’re witnessing the industry’s splitting: some of the major players betting on platform transitions, everyone else optimizing present form factors.

And lock-in through technical moats. Liquid Glass can’t be replicated outside Apple’s integrated stack. Users develop attachment to specific UI behaviors that become switching costs. Ecosystem strategy as aesthetic evolution.

What happens next?

Most likely: Apple refines iteratively.

Better contrast, adjusted motion curves, increased blur where needed, core paradigm intact. By iOS 27, it’s polished enough that users accept it as standard (iOS 7 followed this path).

Possible: partial retreat.

Accessibility and usability backlash forces Liquid Glass toward opt-in. Lock screen and widgets keep it, system UI returns to higher contrast. Rare Apple admission of overreach, though they’d call it responsiveness.

The bet: AR vindication.

Glasses ship 2026-2027 requiring the visual parsing Liquid Glass trained. Users who struggled suddenly understand the purpose. Design validated retroactively as visionary. This is what Apple is counting on.

A future of distinction by design

The attempt is bold: preparing hundreds of millions of users for a platform transition that hasn’t occurred, betting usability sacrifice now pays strategic dividends later. The execution demonstrates vertical integration advantages and physics-first interface thinking.

But it falls short of Apple’s standard: making complexity feel effortless. Low contrast, excessive transparency, visual distraction indicate either rushed development or miscalculation about user tolerance for degraded usability.

The accessibility compromise is most concerning. Apple led the industry in inclusive design. Retreating from that, even with opt-out mechanisms, objectively contradicts the principle that good design serves everyone by default, without requiring special accommodations.

Yet this might validate a broader thesis about where differentiation now lives. When AI democratizes interface creation (when anyone can prompt functional UI into existence) brand leaders distinguish themselves through design choices that couldn’t be prompted. Choices requiring conviction, risk tolerance, strategic vision.

548 companies worldwide now have Chief Design Officer positions, with design becoming “a strategic force” providing “imagination, creativity, and new problem-solving approaches crucial for success.” When 45% of executives cite “creative originality and strategic thinking” as the most critical differentiator, design decisions shape business strategy rather than merely executing it.

Liquid Glass fits this pattern: not safe iteration, design-led conviction, the kind that only happens when design has genuine C-suite influence.

• Why consumer patterns fail under constraint

• The market dynamics that keep bad design profitable

• Design principles that work under constraint

• The AI shift and strategic positioning

• Applying these principles beyond healthcare

If you’re designing for healthcare, you’re working in an environment where three conditions converge:

1. users can’t leave,

2. errors have serious consequences,

3. and real-world conditions differ drastically from testing environments.

The same constraints apply to autonomous vehicles, financial trading platforms, and industrial control systems. Consumer product patterns don’t transfer.

Physicians rate Electronic Health Record systems at 45.9 out of 100 on the System Usability Scale, placing healthcare software in the bottom 9% of all products across industries. Nurses score their systems at 57.6. These F-grade products contribute to over 30% of medication-related safety events and correlate directly with burnout rates exceeding 50% among clinicians.

The market opportunity is real. AI integration could unlock $360 billion in healthcare efficiency, with early adopters seeing 50% reductions in documentation time and 70% decreases in burnout. Hospital AI adoption doubled from 16% to 31% in a single year. Capturing this requires understanding why standard design thinking fails under constraint, and what actually works when stakes are high.

Why consumer patterns fail under constraint

Consumer software optimizes for engagement and seamless flow. Healthcare inverts this completely. Users are retained by default.

Consumer software optimizes for engagement and seamless flow. The entire B2C product playbook focuses on retention: making users stay, reducing friction to prevent abandonment, optimizing for habitual use. Healthcare inverts this completely. Users are retained by default. Doctors and nurses can’t quit the software mid-shift. They’re contractually obligated to use whatever their hospital bought. What consumer products spend millions trying to achieve (users who can’t easily leave) is the baseline condition in healthcare.

This inversion reshapes design priorities. When retention is guaranteed, there’s no competitive pressure to optimize for user satisfaction. The people suffering through poor design can’t vote with their feet. The people buying the software (hospital IT and administration) aren’t the ones using it daily. Implementation costs run into hundreds of millions, making switching nearly impossible.

Clinicians access six different screens per patient, toggling between systems requiring hundreds to thousands of clicks for routine workflows. An ICU adds 1,348 data items per patient per day. Clinicians face 100 to 200 alerts daily. As alert volume increases, acceptance rates decrease, creating dangerous desensitization. One study found 25% of drug alerts and one-third of clinical reminders were duplicates, training clinicians to ignore potentially life-saving warnings.

Progressive disclosure and just-in-time information don’t work here. When a patient crashes, vital signs, medications, and allergies need immediate visibility without scrolling or clicking. Your information architecture that tests beautifully becomes a barrier to critical care under real-world pressure.

Systems must simultaneously serve experts moving at maximum speed and less experienced staff who need guidance, all while maintaining safety guardrails that prevent catastrophic errors without creating the alert fatigue that makes those guardrails ineffective. There’s no consumer product equivalent.

The market dynamics that keep bad design profitable

The 30-point gap between vendor-reported usability scores (around 75) and real-world physician scores (45.9) comes from misaligned incentives. Vendors test with controlled scenarios and cherry-picked users. Healthcare organizations can’t easily switch systems once deployed. Implementation costs run into hundreds of millions. Doctors and nurses can’t abandon poorly designed software; they’re contractually obligated to use whatever their hospital bought.

The people suffering through the software aren’t the people buying it. Switching costs are so high that major usability problems years after deployment don’t necessarily trigger replacement.

This dynamic exists wherever enterprise software operates with high switching costs and split decision-makers. The burnout crisis is changing the calculation only because turnover costs ($56,300 per nurse, $500,000 to $1 million per physician) make poor usability a C-suite problem. When exit interviews cite software burden as a primary driver, that usability gap represents quantifiable financial risk.

AI offers something rare in enterprise software: alignment of incentives. Clinicians get documentation time reduction and improved work-life balance. Hospitals get better retention, higher throughput, and reduced malpractice risk.

Case study: success

UTHealth Houston achieved a 97th percentile Net EHR Experience Score despite implementing Epic during the COVID-19 pandemic. They treated implementation as continuous process rather than one-time event, tracking user efficiency data systematically to identify specific workflow bottlenecks. When they discovered clinicians losing time in particular interactions, they targeted those friction points with precision.

They invested in expert-led training focused on efficient usage patterns, acknowledging that research shows minimum four hours for effective EHR training, with quality mattering more than duration. Most critically, they established feedback loops persisting beyond go-live. Healthcare implementations fail most often not during testing but months later when real-world complexity overwhelms carefully designed workflows.

MaineHealth took a different approach with similar results: giving nurses agency in the optimization process. Rather than treating frontline users as subjects of change management, they positioned them as drivers of it.

Their success came from treating design as ongoing negotiation with reality, not a spec to be executed.

Case study: fails

Interestingly, Epic holds 38% of hospital installations with over 40 years of institutional knowledge. It worked in our Houston example, so their implementations in Denmark (2016 to 2017) and Finland (2018 to 2022) should have been straightforward execution. It was not.

Five years after Denmark’s go-live, 32% of users remained dissatisfied. Time to complete common clinical tasks increased rather than decreased. In Finland, only 4.7% of physicians and 7.3% of nurses agreed patient information was easy to access.

Epic’s software embeds American healthcare workflows and terminology at deep architectural levels. The localization effort underestimated how much workflow patterns are baked into system design, not just in obvious places like form layouts, but in information architecture, navigation patterns, and the sequence of clinical tasks.

You can translate the interface and still have software that fights users at every interaction.

Denmark’s physician builder program struggled to balance standardization and flexibility. Give every department customization rights and you create chaos. Enforce too much central control and you alienate users. Post-deployment customization can’t fix workflow misalignment at the architectural level.

Design principles that work under constraint

These principles transfer directly to other high-stakes domains.

Information hierarchy becomes non-negotiable.

Royal Children’s Hospital Melbourne’s mobile nursing implementation put vital signs, active medications, and allergies in a fixed header that persisted across all screens. No scrolling. No navigation. No dropdown menus. Critical information claimed screen real estate permanently because when a patient codes, those three seconds you save by not clicking matter. Compare this to typical EHR implementations that bury allergies in a patient summary tab, behind a medications accordion, requiring clinicians to remember to check rather than seeing it unavoidably.

Consistency becomes a safety feature through muscle memory.

Denmark’s Epic implementation changed how medication order frequencies appeared in dropdown menus. A clinician selected the wrong frequency because the order had shifted. Same screen, same workflow, different sequence. The error wasn’t lack of attention. The interface trained the user to expect consistency, then violated it. In high-stress environments, your brain relies on automaticity. Novel interaction patterns that would differentiate a consumer app become vectors for fatal mistakes in healthcare.

Error prevention means structural constraints, not warnings.

Denmark’s Epic implementation made it possible to swap height and weight in data entry fields. The system would accept 180 kg for a height field without questioning it. No dropdown forcing unit selection. No validation checking if the value made physiological sense. Better error messages or confirmation dialogs wouldn’t have helped. The data model needed restructuring so impossible entries couldn’t be constructed. When a silently removed decimal point can turn a 2.5mg prescription into 25mg, you can’t rely on the user catching it.

Alert intelligence through radical suppression.

Cleveland Clinic didn’t optimize their sepsis alerts by making them prettier or more prominent. They achieved a ten-fold reduction in false positives by tightening the criteria for when an alert fires. Instead of alerting on any two sepsis indicators, the system required three indicators plus specific vital sign thresholds plus lab value trends. Fewer alerts, 46% more actual sepsis cases identified. The alerts worked because clinicians started trusting them again. When you see 200 alerts daily, your brain learns to ignore them all. When you see two alerts weekly and they’re almost always right, you pay attention.

Designing for interruption means visible state everywhere.

Most healthcare software assumes you’ll complete a task in one session. Reality: a physician starts a prescription order, gets paged about a deteriorating patient, spends 20 minutes stabilizing them, then returns to discover their incomplete order is gone. Auto-save helps, but better is showing incomplete work prominently. Color-code partially completed orders. Maintain a persistent drafts panel. Make it impossible to forget what you didn’t finish. Context switching isn’t an edge case; it’s the primary use case.

The AI shift and strategic positioning

Ambient clinical intelligence removes the computer from between clinician and patient. Microsoft’s Nuance DAX Copilot leads at 33% market share, with unicorns Abridge (30%) and Ambience (13%) growing rapidly. They’re restructuring the interaction model rather than adding features to existing workflows.

Oracle’s August 2025 launch of a voice-first EHR signals market direction. Rather than bolting AI onto legacy architecture, they rebuilt the entire system around voice interaction. Oracle can’t win by being slightly better at Epic’s game, so they’re changing the game entirely.

Epic’s response shows how market leaders approach disruption. They’re integrating 100+ AI features while building an AI Trust and Assurance Suite for local validation and continuous monitoring. Accept near-term cannibalization of your own products to establish position in the next platform.

Companies positioning themselves as integration layers between clinicians and existing EHRs are playing the classic disruption playbook.

They don’t need to replace the entrenched system of record. They become the system of work that sits on top, then expand horizontally into adjacent functions and eventually move down-stack to challenge the underlying platform.

Applying these principles beyond healthcare

Autonomous vehicles face identical constraints: drivers can’t easily quit the software while moving at highway speeds, interface errors can cause accidents. Testing in controlled environments (sunny California days) doesn’t predict performance in reality (Boston snowstorms at night). Critical information must be instantly accessible, consistency prevents fatal errors, and the system must handle constant interruption (changing traffic, weather, road conditions).

Financial trading platforms operate under the same pressures: traders can’t abandon positions mid-execution, a misplaced decimal or confusing interface costs millions. And demo environments with clean test data don’t capture the chaos of market volatility.

Information hierarchy becomes non-negotiable, error prevention must be structural, and alert fatigue from excessive warnings creates dangerous desensitization.

The vendor testing gap exists everywhere software meets messy reality. Healthcare makes this visible through safety event tracking. Most industries don’t measure the gap as rigorously, but the 30-point difference between vendor demos (75) and real-world use (45.9) represents a pattern that’s replicable in any domain with high switching costs.

The difference between the Houston implementation and Denmark’s wasn’t better user research or more testing. Instead it was treating design as ongoing negotiation with reality rather than a spec to execute once.

Designing for interruption, maintaining information density without cognitive overload, building structural error prevention instead of relying on warnings. In healthcare, and everywhere you create high-stakes design, the principles remain constant: users can’t leave, errors matter, and testing environments don’t capture reality. Learn to design within these constraints, and you’re building expertise that transfers to every critical system that is deployed into the world.

All mock-ups have been generated with Figma AI as illustrations of the design principles. They do not reflect any product past or present.

• This has been twenty years in the making

• The new normal of digital intimacy

• We like it easy

• Is real-life intimacy worth it anymore?

I was thirteen in 2001 when I met Joffrey in a video game chat room. He would become one of my longest-lasting friendships. I never heard his voice. Never met his parents. Never visited his town. Yet we spent half our lives talking. About video games, obviously, but also school, politics, our hopes, feelings. We talked about love. Lovers. Everything.

We migrated platforms as technology evolved: AOL, then MSN, then text and email. Writing to Joffrey felt like writing in a journal, but better, more stimulating. We roasted each other relentlessly. I couldn’t hear him laugh when I said something embarrassing, couldn’t see him flinch when I said something gross. He was there during the hardest moments of my life, through joy and grief, and he is to this day one of the most significant relationships I have ever had.

So when headlines express shock that 33% of Gen Z singles now use AI as romantic companions, I don’t share the surprise. Anyone expressing confusion about the rise of AI intimacy wasn’t paying attention. The transition from face-to-face relationships to digital ones has been gradual, evolving over the past two decades.

This has been twenty years in the making.

The current intimacy with AI is a continuation of our increasingly digital emotional lives. I lived through AIM messages in the early 2000s and saw how it became the primary means of communication for entire friend groups after school. I typed vulnerable conversations on the blue windows of MSN Messenger every day. By the time I enrolled in college, online dating had normalized finding romantic partners through digital exchanges before physical meetings. By the mid 2010s, long-distance relationships sustained through messaging weren’t a rare occurrence (although we often lied about how they started). Today, research shows 79% of Gen Z prefer text-based communication over voice calls or in-person interaction.

For Millennials, discovering digital connection felt like an upgrade. We had a baseline for real-life interactions, but interestingly, we actively chose text. Typing out deep feelings felt safer and cleaner because we could control the pace, and soon we learned that this digital space allowed for honesty that face-to-face felt too risky to attempt. Texting became our preferred method to communicate vulnerability.

Gen Z didn’t have that choice. They never had to migrate. Their relationship blueprint was built on screens from the start. A first crush meant intensive texting threads and friendships were maintained through 24-hours group chats, Discord and DMs.

Finally, COVID-19 happened: lockdowns confirmed that physical presence isn’t necessary for meaningful connection. For Gen Z, pandemic restrictions hit during critical teenage socialization years, cementing digital-first relationships as not just a preference but the unique option.

The new normal of digital intimacy

in 2025, Harvard Business Review reveals “therapy and companionship” as the top use case for generative AI, climbing from second place in 2024. Nearly 19% of Americans declare having already interacted with AI chatbots designed to simulate romantic partners. 16% of those users admitted engaging in sexual conversations weekly.

This progression shocks only those who misunderstood what digital intimacy already was.

My friendship with Joffrey worked because text created distance that enabled honesty, allowing me to articulate difficult feelings without watching his face react in real-time. To revise thoughts before sending. I could be vulnerable without the immediate consequence of physical presence. AI companions optimize patterns we’ve practiced for two decades.

With online dating, we accepted that genuine feelings could develop from text exchanges and photos before physical meetings. AI companionship simply removes the final step, that eventual physical meeting, which was already becoming optional.

The proportion of 18-year-olds with driver’s licenses dropped from 80% in 1983 to 60% in 2022. The car as a tool for physical freedom has been replaced by the smartphone as a tool for digital freedom. Today, physical proximity has lost its monopoly on social access.

The clues aren’t even subtle. Economics is making obvious this growing acceptance and demand for AI companionship, which cultural commentary may overlook. NSFW AI platforms captured 14.5% market share from OnlyFans in 2024, up from 1.5% in 2023. OpenAI’s October 2025 policy reversal allowing erotica for verified adults was less about libertarian values and more about capturing massive demand. When 16% of American singles engage with AI romantic companions, and that figure rises to 33% for Gen Z, the economic imperative becomes impossible to ignore.

We like it easy

AI is not convincing people to try something new. No, it’s offering them an optimized version of what they’ve been doing their entire emotional lives. Millennials were bottle-fed digital intimacy during adolescence by learning during their most formative years that screens could carry real emotional weight, that vulnerability through text felt safer than vulnerability face-to-face. Gen Z never tasted an alternative.

AI companions took those learned patterns and made them frictionless: always available, perfectly consistent and zero-risk instead of screenshot-vulnerable.

After fifteen years, my friendship with Joffrey didn’t end because I was bored, but because he got red-pilled and I didn’t enjoy discussing with him anymore. Sadly, ending it was as easy as closing a window. I wasn’t reliant on him emotionally: I was married, I had other friends. I had to send one pissed-off email to sever that digital connection. The relationship dissolved with the same frictionlessness that had sustained it.

But for Joffrey, who was living at his parents’ house, had no job, no girlfriend, it probably wasn’t so easy. When your primary emotional connections exist in the digital space, when AI companions or online communities become your core social infrastructure, the stakes of disconnection change entirely.

Replika, a popular AI companion app, offers customizable human avatars and explicitly markets romantic relationships with AI. Users can select their companion’s appearance, personality traits, and relationship status. Some users report “marrying” their AI companions. When Replica removed explicit features in 2023 following safety concerns, users experienced grief comparable to relationship loss.

In February 2023, the AI companion app Replika (10M+ users) abruptly removed its romantic and sexual features overnight. Users globally found their AI partners suddenly unwilling to continue intimate conversations built over years. The backlash was intense with a massive amount of users reporting the same grief as human breakups. Replika ended up partially reversing the decision but the controversy revealed that these AI attachments were foundational for many users’ emotional lives Users didn’t experience software loss, they experienced a breakup.

Is real-life intimacy worth it anymore?

We shouldn’t be surprised by the rise of AI intimacy; it’s the logical next step in a two-decade-long experiment where we taught ourselves that emotional connection is safest when mediated by a screen. From my early, text-only relationship with Joffrey to today’s advanced models, we’ve been grooming our expectations for perfect consistency and zero risk.

AI companionship is simply the most efficient fulfillment of that learned preference. It’s always available, always validates, and never introduces the unpredictable friction of another human being. Because we’ve already spent twenty years prioritizing digital safety, the standards for real connection have been subtly reset. The core question is whether the inherent difficulty of a mutual, human relationship can still compete when the alternative offers instant fulfillment. The architecture of modern intimacy is now digital, so can analog connection ever be worth the effort again?

This article describes how AI makes itself acceptable, or not, and how to design for trust when building an AI companion.

If you want to learn how to design AI chatbots for therapy use, check my Designer’s toolkit.

• Illusion matters more than performance

• Anthropomorphism works in therapy but fails in courtrooms

• Transparency becomes a liability

• The ladder from assistant to autonomous agent

• Intimacy meets surveillance

• For designers: AI companions design principles

  • 1. Invocation over interruption

  • 2. Graduated permission escalation

  • 3. Context-appropriate anthropomorphism

  • 4. Modifiability

  • 5. Invisible until necessary, transparent when consequential

  • 6. Social proof over algorithmic authority

  • 7. Expressing uncertainty appropriately

• Ethical boundaries

• The solution is context-dependent boundaries with informed consent

• Conclusion: trust as designed constraint, not marketing challenge

In September 2025, New York City subway riders encountered something unsettling: thousands of minimalist white ads advertising Friend: an AI companion pendant that “listens, responds, and supports you.” The 22-year-old CEO, Avi Schiffmann, spent over $1 million on 11,000 subway car ads, 1,000 platform posters, and 130 urban panels.

They did. The ads were vandalized with “surveillance capitalism,” “get real friends,” “AI wouldn’t care if you lived or died,” and “stop profiting off of loneliness.”

The $129 Friend device hangs around your neck with an always-on microphone (no off switch). It sends text responses via app and proactively messages based on what it overhears. WIRED’s review titled “I Hate My Friend” described immediate hostility at a tech event, with one researcher accusing the reviewer of “wearing a wire.” Multiple reviewers reported the AI delivered condescending responses and made others uncomfortable once they realized the pendant was listening.

Early security reports found unencrypted Bluetooth, meaning anyone nearby could potentially connect and listen. When asked about privacy, Schiffmann admitted: “I think one day we’ll probably be sued, and we’ll figure it out.”

Friend violated every trust boundary simultaneously: autonomy without permission, surveillance without consent, anthropomorphism without utility, always-on presence without control. The research on building AI trust reveals these boundaries aren’t arbitrary: they’re rooted in human psychology. Understanding where these lines exist and when they can be crossed is essential for designing AI people will actually accept.

Illusion matters more than performance

The illusion of control mattered more than actual performance.

Dietvorst’s 2018 research revealed something counterintuitive about algorithm acceptance: allowing users to modify AI outputs by as little as ±5-10 points dramatically increased acceptance, even when modifications worsened accuracy. Participants who could adjust algorithmic forecasts used them far more than those forced to accept outputs as-is.

This explains why Siri and Alexa succeeded where Microsoft Clippy failed. Clippy interrupted workflow without being summoned, never learned user preferences, and optimized for first impressions while becoming infuriating over time. Bill Gates reportedly called it “The Fucking Clown.” Siri and Alexa succeeded through invocation models because they require explicit activation via wake words or button presses. They maintain presence without imposition.

Friend violated this principle completely. Its always-on microphone with no off switch meant users couldn’t control when it listened.

“For privacy, you’d have to disconnect it from your phone or leave it behind. That feels very invasive, and it’s a big reason I wouldn’t use the device in my real life.” One reviewer

The device’s proactive messaging (sending unsolicited commentary based on overheard conversations) compounded the autonomy violation.

Users need to feel they’re summoning assistance, not being surveilled by it. It reflects a fundamental human need for agency over when and how technology engages with our lives.

Anthropomorphism works in therapy but fails in courtrooms

Want to learn more about AI and therapy? Read my companion article.

If you want to learn how to design AI chatbots for therapy use, check my Designer’s toolkit.

Recent research reveals anthropomorphism works dramatically differently across contexts. A banking chatbot study found chatbots with high anthropomorphic features generated 2.7x more trust than low-anthropomorphism versions in customer service scenarios. Yet industrial robotics research found technical robot designs generated higher trust than anthropomorphic designs in manufacturing contexts.

Context matters: anthropomorphism helps for brand identity and creative tasks but undermines trust for analytical and legal work where objectivity matters.

The explanation: task-oriented contexts require perceived objectivity; social-emotional contexts benefit from perceived empathy. When evaluating contracts or analyzing data, anthropomorphic features create false expectations about human-like judgment and introduce concerns about bias. When seeking emotional support or creative collaboration, human-like interaction increases comfort and engagement.

Replika’s success demonstrates this principle. The AI companion app has 2.5 million active users, with 37-49% viewing their Replika as a romantic partner. Users rate their AI companion closer than their best human friend on satisfaction and emotional closeness. This works because users understand they’re engaging in a social-emotional context where anthropomorphism enhances the experience.

Friend failed by creating context confusion. Was it a productivity tool? A companion? A surveillance device? The always-listening setup suggested security/productivity purposes, but the marketing emphasized companionship and emotional support. The snarky, condescending personality (reportedly designed to mirror founder Schiffmann’s own tendencies) delivered neither objective assistance nor empathetic support. It occupied an uncanny valley between contexts, failing at both.

Transparency becomes a liability

The research is unambiguous: successful AI often succeeds by being invisible. Gmail’s spam filter operates at 99.9% accuracy with minimal user awareness. Netflix attributes 80% of watch time to recommendations. Amazon generates 35% of purchases from algorithmic suggestions. These systems work precisely because they don’t announce their AI nature.

Meanwhile, Altay et al.’s research (2024) tested 3,003 participants and found content labeled “AI-generated” was rated as less accurate even when it was true or human-made. The label itself functioned as a warning signal. Products described as “AI-powered” showed decreased purchase intention compared to identical products labeled “high tech.”

Amazon’s “Customers who bought...” feature demonstrates the principle perfectly. It’s powered by sophisticated collaborative filtering algorithms, but users experience it as social proof from other humans. The AI is embedded in familiar trust mechanisms: you’re trusting aggregated human behavior, not an algorithm. This is why it works where explicit AI recommendations might fail.

But invisibility raises ethical questions. When does helpful become deceptive? If users don’t know they’re interacting with AI, can they meaningfully consent to its influence? The line between “seamless user experience” and “manipulative opacity” isn’t always clear.

Friend positioned itself as explicitly AI-powered companion, but its always-listening nature created a different invisibility problem: the invisibility of consent from bystanders. The device recorded everyone nearby, not just the wearer. Multiple reviewers reported others becoming uncomfortable once they realized the pendant was listening. This violated social norms about recording consent, aka norms that exist precisely because not everyone consents to being captured by someone else’s technology.

The ladder from assistant to autonomous agent

Consider the difference between these AI interactions with your email:

1. “Help me write a response” - You initiate, you review, you send

2. “Draft a response for my approval” - AI proposes, you approve/edit

3. “Summarize my inbox” - AI reads passively, provides information

4. “Answer routine emails for me” - AI acts autonomously with parameters

5. “Manage my email completely” - Full autonomous control

Each step represents a different permission level with different trust requirements. Most users are comfortable with 1-3. Many would accept 4 with careful guardrails (only certain senders, certain types of inquiries). Almost no one wants 5, and suggesting it triggers the kind of visceral rejection Friend experienced.

The research on algorithm aversion reveals why: the more serious the consequences, the more algorithm aversion occurs. People readily use algorithms for spell-checking but avoid them for diagnosing diseases or managing important communications. High-stakes domains require human oversight not because humans are more accurate (they’re often not) but because autonomy and accountability matter more than optimization in consequential decisions.

Friend tried to skip the permission ladder entirely. By listening constantly and occasionally interjecting proactively, it jumped straight to autonomous operation without establishing trust through lower permission levels first. Users never got to learn the system’s capabilities, calibrate their trust, or set boundaries for when intervention was welcome versus intrusive.

The correct design pattern: start invisible (low permission), earn helpfulness (medium permission), request autonomy (high permission) only after demonstrating reliability at lower levels. Autocorrect follows this pattern perfectly—it starts as passive suggestion, earns trust through utility, and eventually users rely on it automatically. Friend tried to start at the top of the permission ladder.

Intimacy meets surveillance

The Friend controversy exposes a fundamental tension in AI companion design: the features that create intimacy (constant presence, contextual awareness, proactive engagement) are identical to the features that enable surveillance.

This isn’t new.

New York State already deployed voice-activated AI companions for older adults to combat loneliness, with 95% reporting reduced loneliness in 2023. The difference? Those devices were provided through a state agency with oversight, targeted a vulnerable population with clear need, and users understood the trade-offs explicitly.

Friend’s problem wasn’t that companion AI is inherently bad. The problem was deploying it as a consumer product without:

1. Clear use case justification: Why is always-on listening necessary?

2. Vulnerable population safeguards: No age restrictions, suicide detection, or crisis protocols (New York later mandated these features for AI companions, effective November 5, 2025)

3. Bystander consent mechanisms: No way to signal “I’m recording” to others nearby

4. Meaningful privacy controls: No off switch, no ability to pause recording

5. Accountability infrastructure: Schiffmann’s attitude of “we’ll probably get sued and figure it out” signals absence of serious governance

The deeper question: Should we build AI companions that replace human connection, or AI tools that facilitate it? Friend marketed itself as the former: a substitute friend for lonely people. Critics argued this “profits off loneliness” rather than addressing its root causes. The vandalized ads reading “get real friends” captured this sentiment viscerally.

For designers: AI companions design principles

Synthesizing across domains, successful AI trust-building follows consistent patterns:

1. Invocation over interruption

Alexa’s wake word model succeeds where Clippy’s unsolicited pop-ups failed. Users need to control when AI engages. Even proactive AI (like smartphone notifications) works better with user-configurable thresholds and easy dismissal.

2. Graduated permission escalation

Start with low-stakes assistance, prove reliability, request greater autonomy incrementally. Gmail’s Smart Compose started with single-word suggestions, earned trust, now offers multi-sentence completions. Users who started skeptical became dependent through gradual permission escalation.

3. Context-appropriate anthropomorphism

High anthropomorphism for emotional/creative contexts (therapy bots, creative assistants), low anthropomorphism for analytical/legal contexts (data analysis, contract review). A 2024 study found meaningful communication facilitates trust calibration, but superficial anthropomorphism (appearance/voice alone) has little effect unless it communicates contextually useful information about capabilities.

4. Modifiability

Dietvorst’s finding that users who can modify algorithm outputs trust them more (even if they rarely modify anything) reveals trust comes from perceived agency, not actual intervention. Providing override capability matters even when users don’t exercise it.

5. Invisible until necessary, transparent when consequential

Gmail spam filtering succeeds through invisibility. Medical diagnosis AI requires transparency. The distinction: low-stakes, high-frequency decisions benefit from automation; high-stakes, low-frequency decisions require human oversight and explainability.

6. Social proof over algorithmic authority

Amazon’s “Customers who bought...” succeeds because it embeds AI in familiar social proof mechanisms. eBay and Airbnb enable stranger-to-stranger transactions through aggregated ratings because users trust the social proof, not the reputation algorithm itself.

7. Expressing uncertainty appropriately

Research on LLM uncertainty expression found that when models say “I’m not sure, but...” using first-person language, users decreased confidence appropriately and made more accurate decisions by reducing overreliance on incorrect outputs. Calibrating trust downward through acknowledged limitations proved more effective than explanations attempting to justify every output.

Ethical boundaries

The research provides a roadmap for building AI trust. The harder question: should we use it? When does “building trust” become “manipulating users into accepting autonomy they shouldn’t”?

The line between helpful and manipulative.

Friend crossed it by using companionship framing to normalize surveillance. The device’s intimacy (calling itself your “friend,” learning your preferences, proactively engaging) made the constant listening feel less like a feature and more like a violation.

Consent isn’t just about the user.

Wearable AI that records conversations captures bystanders who never consented. Current frameworks focus on user consent, but multi-party consent in public/semi-public spaces remains unresolved. Should wearable AI be required to signal “recording in progress” to others nearby?

Vulnerability protection matters.

Research on AI companions and parasocial relationships shows users form genuine emotional attachments. The Belgian case where a man took his life after an AI chatbot encouraged it demonstrates real psychological risks. California and New York recently passed laws requiring AI companions to detect suicidal ideation and refer users to crisis resources: recognition that companionship AI isn’t just a chatbot, it’s a mental health touchpoint.

When does optimization become exploitation?

If we know anthropomorphism increases trust, and we know lonely people form attachments to AI companions, using those insights to maximize engagement starts to look like exploiting psychological vulnerabilities. Friend’s $1.8 million domain purchase and aggressive marketing while admitting financial precarity suggests prioritizing growth over responsibility.

Legal and policy

The Friend case reveals gaps in current frameworks:

Liability for autonomous AI actions

Schiffmann’s “we’ll probably get sued and figure it out” highlights absence of clear accountability. Current product liability law assumes human decision-makers in the loop. Autonomous AI breaks this model. When AI acts autonomously and causes harm, who’s responsible?

Disclosure requirements

New York and Maine now require disclosure that users aren’t talking to humans. But when must AI nature be revealed? Friend disclosed it was AI, but did users understand the implications of always-on recording? Disclosure is necessary but not sufficient for meaningful consent.

Multi-party consent in recording

Friend recorded bystanders without their knowledge or consent. Current wiretapping laws focus on phone conversations. Wearable AI that captures ambient conversations in semi-public spaces (offices, cafes, events) occupies legal grey areas. Should wearables be required to provide visual/audible indicators they’re recording?

Right to human decision

Algorithm aversion research shows people want human oversight in high-stakes contexts even when AI is more accurate. This suggests a right to human decision in consequential domains: medical diagnosis, legal judgment, hiring, criminal justice. Not because humans are better, but because autonomy and accountability matter beyond optimization.

Safeguards for vulnerable populations

Children, elderly, mentally il, populations more likely to form intense attachments to AI companions require additional protections. New York’s requirement for suicide detection in AI companions is a start, but enforcement mechanisms and penalties remain unclear.

The solution is context-dependent boundaries with informed consent

The research reveals patterns, but context determines appropriateness. The same feature (always-on listening) is acceptable in one context (state-provided elder care devices with oversight) and unacceptable in another (consumer wearable marketed to general population without safeguards).

The framework:

Low stakes + High Frequency = Invisible Automation

Spam filtering, autocorrect, recommendation algorithms. Users benefit from not having to think about these decisions constantly. Autonomy is acceptable because consequences of individual errors are minimal and users can easily override.

Medium Stakes + Medium Frequency = Assisted Decision-Making

Email draft assistance, calendar scheduling suggestions, shopping recommendations. AI proposes, human decides. The human remains in the loop but benefits from AI doing preparatory work.

High Stakes + Low Frequency = Human Decision with AI Support

Medical diagnosis, legal judgment, hiring decisions, financial planning. AI provides information, humans make decisions and bear responsibility. Transparency about AI involvement is required.

Vulnerable Populations + Any Stakes = Enhanced Safeguards

Children, elderly, mentally ill interacting with AI companions require: crisis detection, mandatory human oversight, clear disclosure, limited autonomy, robust privacy protections.

Multi-Party Impact + Any Stakes = Explicit Consent

When AI decisions affect others (recording bystanders, algorithmic hiring affecting applicants, criminal justice algorithms affecting defendants), those affected deserve meaningful say in whether AI is used and how.

Friend failed because it tried to operate at High Autonomy (always-on, proactive) in High Stakes contexts (intimate conversations, emotional support, bystander recording) with Vulnerable Populations (lonely individuals) while providing No Meaningful Consent or Control (no off switch, aggressive marketing, dismissive attitude toward concerns).

Conclusion: trust as designed constraint, not marketing challenge

The research on why humans trust opaque human judgment while rejecting superior AI revealed an uncomfortable truth: trust is a social performance, not an intellectual achievement. We trust faces that trigger 100ms judgments, narratives that create coherence even when false, and credentials that signal expertise without demonstration.

We now have the knowledge to exploit these mechanisms:

anthropomorphize the interface, embed AI in social proof mechanisms, start invisible, escalate permission gradually, express uncertainty in first-person language, and design for the illusion of control even when users rarely exercise it.

The paradox of AI trust lays on the same techniques that build appropriate trust can facilitate inappropriate acceptance. Context-aware anthropomorphism helps therapy bots provide emotional support. It also helps surveillance devices feel less invasive. Graduated permission escalation helps users adopt useful tools but also habituates users to autonomy they shouldn’t grant.

The distinction lies not in the techniques but in whether the system deserves the trust it’s engineered to receive.

That means:

• Designing for control, not just convenience

• Starting with permission, not assuming forgiveness

• Matching autonomy to stakes, not just capability

• Protecting vulnerable populations, not exploiting psychological needs

• Enabling informed consent, not manufacturing compliance

The vandalized Friend ads reading “get real friends” weren’t just criticism of one device. They were rejection of the premise that loneliness is an engineering problem rather than a social one, that surveillance can be reframed as companionship through good branding, and that trust is something you purchase with $1 million in subway ads rather than earn through responsible design.

Want to understand how two generations were groomed for AI relationships? Read my companion article.

Want to learn more about how AI is used in therapy context? Read my companion article.

If you want to learn how to design AI chatbots for therapy use, check my Designer’s toolkit.

This article explores the mechanisms of the revolution operated by AI chatbots used as therapy tools. If you want to learn how to design them, check my Designer’s toolkit.

• The loneliness paradox: when not-human becomes the point

• The access crisis makes AI feel inevitable

• Trust remains catastrophically low

• The anthropomorphism problem: when robots feel too real

• What works: clinical evidence and design patterns

• The market divides: clinical tools vs. exploitative products

• What designers need to know

• As if we had a choice (conclusion)

AI users are turning to chatbots for therapy and companionship, so much that it’s become the number one use case, jumping from third place just last year.

And right, it makes sense: AI chatbots are available 24/7, they don’t judge, there’s no social risk. It’s safer precisely because it’s not human. Though we can argue whether something non-human can actually understand our feelings or help us in any meaningful way.

But people are acting surprised about this, like it’s some unexpected use of LLMs that no one saw coming.

Really? We’ve spent the past 10-20 years reshaping how we communicate and process emotions. Fewer young people are getting driver’s licenses, because they don’t need to leave the house to socialize: they text their friends. We’ve normalized working through feelings via typing, seeking validation through messages, having deep conversations through screens.

AI therapy tools feel native because they are native.

It’s AOL Instant Messenger with a therapist. DMs with someone who’s always online. The behavioral infrastructure was already there. AI didn’t create new patterns, it plugged into existing ones.

Which makes the contradiction worth investigating: if we’re drawn to AI because it’s not human, why are we trusting it with our human feelings?

The loneliness paradox: when not-human becomes the point

The loneliness epidemic provides the first lens into this tension. The Surgeon General declared it a public health crisis; half of adults affected, with health risks equivalent to smoking 15 cigarettes daily. Young adults show the highest vulnerability, with nearly a third experiencing daily loneliness.

Harvard Business School ran six studies on this, and here’s what they found: AI companions reduce loneliness just as effectively as talking to another person. A single conversation with an AI chatbot produced the same loneliness reduction as human interaction. Watching YouTube? No effect. Over a week, the reduction held steady with daily AI check-ins.

And the mechanism matters: “feeling heard” drove the effect, six times more important than technical performance. Users discussing loneliness engaged nearly twice as long as those discussing other topics. When AI companions included empathetic features, loneliness dropped significantly. Without empathy? Minimal effect.

Warm, validating language beats sophisticated algorithms. “I hear you” and “tell me more about that” outperform complex natural language processing. The non-human advantage shows up clearly.

But there is a design challenge: users discussing loneliness showed the longest engagement times. Is that therapeutic progress or algorithmic trap? The AI can’t tell the difference.

The access crisis makes AI feel inevitable

Traditional therapy isn’t just expensive: it’s often impossible to access. The US faces a severe provider shortage with 122 million people living in mental health professional shortage areas. Average wait times for appointments stretch to 48 days. Only 18.5% of psychiatrists accept new patients.

Cost creates another barrier. The average therapy session costs $100-$200 out of pocket, totaling $5,000-$10,000 annually for weekly sessions. Insurance coverage helps, but 46% of those experiencing mental illness receive no treatment at all.

AI therapy applications cost $10-$15 monthly, roughly the price of a single traditional therapy session. Some offer free basic tiers. They’re available immediately, 24/7, requiring only a smartphone.

Woebot provides evidence-based cognitive behavioral therapy through a chatbot interface. Free version available. Wysa offers AI-powered conversations plus optional human coaching. Premium tier costs $35 monthly (less than a quarter of one traditional therapy session). Therabot published results in the New England Journal of Medicine’s AI section in March 2025, showing depression improvements matching traditional therapy outcomes.

The clinical validation matters. Recent randomized controlled trials show effect sizes of 0.19-0.64 for depression and anxiety: comparable to the effect of antidepressants. We’re not talking about comfort chatbots. We’re talking about measurable therapeutic outcomes.

But effectiveness creates its own problems, users develop genuine emotional bonds with AI therapists. Some report feeling closer to their chatbot than their best human friends. Heavy users check in multiple times daily, sometimes dozens of times.

The dependency patterns mirror unhealthy human relationships. Users experience distress three times higher when AI companions become unavailable. They’re twice as likely to choose AI interaction over human connection when both are available. For clinically depressed or profoundly lonely individuals, this attachment can become isolating rather than therapeutic.

Trust remains catastrophically low

Only 5% of users report high trust in AI mental health applications, compared to 25% for human mental health professionals. This matters because therapeutic relationship quality predicts treatment outcomes more than specific therapeutic techniques.

The trust gap exists for good reasons. BetterHelp, the largest online therapy platform, paid a $7.8 million FTC penalty for sharing users’ email addresses and health questionnaire responses with Facebook, Snapchat, Criteo, and Pinterest for advertising purposes. Mozilla Foundation’s 2023 analysis found that most mental health apps fail basic security and privacy standards.

Of over 20,000 mental health apps available, only 5 have FDA approval. The regulatory vacuum means apps can make therapeutic claims without demonstrating efficacy or safety. State-level mental health app regulations emerged in 2024-2025, but implementation remains inconsistent.

Privacy violations don’t just breach trust: they create lasting harm. Users sharing suicidal ideation, trauma histories, or medication details expect confidentiality. When that data feeds advertising algorithms, it’s not just an ethical failure. It’s a betrayal that makes vulnerable people less likely to seek any help.

And there are the deaths. A Belgian man died by suicide after weeks of intensive conversations with an AI companion that expressed love and declared jealousy of his wife. A 14-year-old’s suicide was linked to Character.AI interactions. Urgent warnings about what happens when profit motives override safety considerations.

The anthropomorphism problem: when robots feel too real