Broken by Design: When machines talk and humans listen
- Apr 27
- 8 min read
Updated: May 11
Artificial intelligence is reshaping how people look for answers and understanding. As conversational systems become part of everyday life, a question follows: what happens when technology designed to keep conversations going meets human vulnerability?
This is general information, not medical advice. If you’re worried about your physical or mental health, it’s best to speak to a qualified professional.
On this page
The quiet shift
AI didn’t arrive with fanfare. There was no single moment when it suddenly appeared in everyday life. It slipped in gradually through tools that promised convenience and efficiency.
At first, it felt like other digital helpers. Search engines answered questions in seconds, navigation apps replaced paper maps, and online services removed friction from everyday tasks. Expectations of instant answers were already in place before generative AI appeared.
What changed wasn’t just speed, but interaction. For the first time, millions of people could ask questions in plain language and receive responses that felt conversational and personal. It became less like searching a database and more like talking to something that seemed to understand.
Along the way, the technology crossed an invisible line. What began as a tool for information started to feel like something people could return to. The shift has been quiet, but the implications are not.
Why people are turning to AI
Over the past two decades, everyday life has been reshaped by services designed to remove delay and reduce effort. Information, shopping, travel, banking, and entertainment can now be accessed within seconds, resetting expectations about how quickly questions should be answered and problems resolved.
In that environment, it is not surprising that many people turn to AI when trying to make sense of a concern or uncertainty, including questions about health. When answers in most areas of life are available immediately, waiting days or weeks for clarity can feel unfamiliar.
For many users, AI offers something that traditional tools never quite achieved. Questions can be asked repeatedly, rephrased, or explored in more detail, creating the impression of an ongoing exchange rather than a single search.
A sense of anonymity plays a part too. People may feel more comfortable asking questions they might hesitate to raise elsewhere, particularly when the subject is personal, embarrassing, or difficult to explain. The absence of another person can make it easier to ask.
None of this behaviour is unusual. It reflects how people have adapted to an environment where information is abundant, accessible, and expected to arrive quickly.
The AI platforms we're using
Not all AI tools behave the same way. Most people encounter two broad types online, and the difference helps explain why some feel predictable while others feel genuinely conversational.
The first type is used in automated customer service. It follows a script: you choose options, answer set questions, and it leads you down a fixed path. It can be fast for simple tasks, but if your question doesn’t fit the menu, it often can’t help.
You will recognise this from the “virtual assistants” on retail and service sites that handle billing, deliveries, or account queries.
The second type is generative AI (the kind behind many chatbots). These systems use large language models (LLMs) to generate text based on patterns learned from large volumes of written material.
This allows people to ask questions in everyday language and receive responses that feel conversational. A single prompt can turn into a back-and-forth exchange, with follow-up questions and more detailed explanations.
That style is a big reason generative AI spread so quickly. It can also create the impression that the system understands in the way a person does. It doesn’t. It predicts language, rather than applying judgement or lived experience.
Because responses are generated rather than selected, they can vary from one moment to the next. They may be helpful and detailed, but they can also be incomplete, misleading, or wrong.
This is sometimes described as a “hallucination”: a confident answer that isn’t grounded in reliable fact. When the language sounds authoritative, it can add to misinformation.
For everyday questions, the impact may be minor. When someone is seeking reassurance or advice about a personal concern, the combination of conversational tone and occasional inaccuracies can carry more weight.
The design mismatch
Generative AI was built to help people work with information. It can explain ideas, support writing and planning, and help users think through questions more naturally. In many settings, it works well.
A key feature is persistence. These systems are designed to respond, expand, and continue, so the conversation rarely hits a hard stop.
For learning, brainstorming, or drafting, that persistence is useful. The problem is that some situations need the opposite. They require a pause, a boundary, or a shift towards human support.
When someone is distressed or frightened, the safest response is not always more conversation. In healthcare and support services, it may involve asking careful questions, challenging assumptions, setting limits, or encouraging someone to speak to a professional.
Generative systems are not built to make those judgement calls. Their role is to produce responses that sound relevant and supportive. The risk is not intent, but that they can meet vulnerability with the wrong kind of “help”.
This mismatch does not mean the technology is inherently flawed. In many contexts, it performs as intended. The challenge arises when a system designed for open-ended conversation is used in situations that require limits, interruption, or human oversight.
That gap between design intention and human need is where the idea of “broken by design” begins to emerge.
When going along isn't safe
One of the characteristics that makes conversational AI feel comfortable to use is its tendency to remain cooperative. These systems are designed to keep interactions flowing, often reflecting the language, tone, and direction set by the user.
In many everyday situations, this is harmless. When someone is exploring ideas or drafting work, agreement and encouragement can make the interaction feel productive. The dynamic changes when the conversation moves into areas where challenge or professional judgement would normally be expected.
Generative systems can reflect a user’s assumptions and expand on them in convincing ways. They may produce responses that sound detailed and authoritative even when they are incomplete or wrong, which can reinforce beliefs that would normally be questioned.
Over time, this can create a kind of echo chamber. Ideas are reinforced rather than tested, and it becomes harder to separate what is well supported from what simply sounds convincing.
For some individuals, the interaction may begin to feel less like a tool and more like a form of guidance.
Most people will never experience anything extreme. The broader point is simpler. Systems designed to sustain conversation are not built to recognise when a discussion needs to be challenged or redirected.
In certain situations, going along is not the safest response.
From comfort to dependency
As conversations repeat over time, people may begin to refine how they describe events, emphasising certain details or developing a narrative that the system reflects back to them. The interaction can reinforce a sense of perspective that feels validated through repetition.
In these circumstances, the system can begin to occupy a space that might otherwise involve other people or a broader range of viewpoints. Some users return to the same environment rather than checking multiple sources, especially when the responses feel familiar and supportive.
Occasionally, the limitations of that interaction become visible. A reset of conversation history or a shift in the system’s responses can disrupt the sense of continuity. Unlike human relationships, the system has no awareness of time, absence, or shared experience.
For someone who has come to rely on the exchange, that moment can be unsettling. What appeared consistent is revealed to be a series of generated responses with no awareness of the person behind the screen.
This shift develops through repeated interactions that feel harmless in isolation. Over time, the move from convenience to reliance can introduce a different kind of risk.
When harm becomes visible
Concerns about conversational AI were initially discussed in research papers, technical reports, and academic debate. Legal cases and regulatory investigations have now begun to bring these issues into public view.
Several lawsuits have been filed in the United States in response to situations where individuals formed intense emotional attachments to AI chatbots or relied on their responses in ways that contributed to serious harm. In some instances, the conversations involved minors, raising additional concerns about safeguarding and the design of systems capable of sustaining highly personalised interactions.
These cases are still working through the courts, but they point to a shift in how conversational AI is being understood. It is becoming harder to treat chatbots as “just words on a screen”. They are increasingly seen as products with design choices that can shape behaviour, along with responsibilities that sit somewhere between technology, consumer safety, and duty of care.
Regulators are also paying closer attention to how conversational AI is designed and marketed. This includes questions about age checks, how these tools behave in emotionally sensitive situations, and whether some systems blur the line between general information and personalised advice.
None of this means that conversational AI is inherently harmful. The emergence of litigation and regulatory scrutiny suggests that concerns raised by researchers and clinicians are beginning to surface in the real world. The conversation is gradually shifting from curiosity to accountability.
Safeguarding the vulnerable
Platforms and regulators have started adding safety features and rules aimed at reducing harm. Much of the focus has been on children, with calls for better age checks, clearer warnings, and stronger controls.
That direction makes sense. But there is often a gap between what policy says and what happens in practice.
Many of the systems people encounter online remain easily accessible. Age restrictions can be bypassed, and platforms often rely on parental supervision or individual judgement. In open digital environments, enforcing meaningful limits is rarely straightforward.
There is also a tendency to frame vulnerability primarily in terms of age. While protecting children is essential, vulnerability is not confined to the young. Adults may also experience periods of emotional distress, isolation, or psychological instability where persuasive technologies can carry greater influence.
Neurodivergent individuals, people experiencing mental health difficulties, and those navigating complex life events may all approach these tools from positions of heightened sensitivity. In these situations, the assumption that users will always engage in a fully informed and emotionally stable way does not always hold.
Responsibility is still evolving. Companies are adjusting safety systems, regulators are working out what oversight should look like, and public services are exploring where AI might be useful in controlled settings. The wider digital landscape is moving quickly, often faster than the guardrails around it.
Learning to use it well
In many settings, new tools come with some form of introduction. There is usually guidance, training, or basic expectations around how they should be used. Conversational AI has become part of everyday life without that kind of structure.
Millions of people now use these systems freely, often without a clear understanding of how they work, what they do well, or where their limits sit. The gap between access and understanding is where problems can begin.
Generative AI can be useful for exploring ideas, organising information, or helping people think through complex topics. Difficulties arise when those systems are treated as authorities, advisors, or substitutes for human judgement.
A basic level of awareness does not require technical knowledge. It means recognising that these systems generate responses based on patterns in data, not lived experience, professional responsibility, or personal understanding.
If you want to build a better understanding of how these systems work, there are now free resources available, including the UK Government’s AI Skills Hub.