About a third of people say they tell chatbots things they keep from friends, family and medical professionals. That figure comes from a survey called The AI Privacy Problem, commissioned by DuckDuckGo (Yahoo Tech). DuckDuckGo sells a privacy browser, so the number comes with a commercial interest attached to it. Treat it as industry evidence rather than independent research.
The direction of it still matches what academics find when they read the policies instead of asking the users.
This piece is about the interface itself, and why anyone building or funding one of these products should see the chat window as a trust mechanism with a balance sheet behind it.
Why does a chat box feel safer than a form?
A form announces what it is. It has fields, a submit button, and somewhere underneath, a sentence about how the data will be used. Nobody mistakes it for a chat with a friend.
The chat box replies. It asks a follow-up. It remembers the thing mentioned two messages ago. Every cue in it belongs to a conversation, and conversations come with social expectations that the software has no intention of honouring.
Doctors, lawyers and therapists operate under confidentiality rules. Chatbots are bound by none of them (Forbes). The resemblance stops at the tone of the exchange.
Researchers at Stanford went through six major chatbot companies and found that all six use chat data by default to train their models. Some keep that information in their systems indefinitely (Stanford). Indefinitely is a long time to store someone's worst Tuesday.
And the product does not stand still. OpenAI launched Memory in April 2024, and in April 2025 improved it, so that ChatGPT retains and rewrites what it remembers about the person typing (XDA). Some users have reported getting more than they bargained for, as long-term memory changed the way the bot interacted with them (Futurism).
The confidant has a file, and the file is edited by its owner.
Isn't this a trade people make knowingly?
Here is the fair objection. People are not children. They know in the abstract that their chats are stored, processed and probably used to train something. They type anyway, because the thing is useful and the risk is vague. That is a bargain they have weighed, and a privacy piece that treats every user as a dupe deserves to be closed halfway through.
Two things weaken it. The first is that the people closest to the subject do not describe it that way. Bernard Marr, writing in Forbes, puts a good deal of the oversharing down to people who "use them without fully understanding the implications" (Forbes). Knowing that data is stored somewhere is not the same as knowing that six out of six companies train on it by default, and that some of them never delete it.
The second is that the terms move. Someone who made a calm, informed trade in early 2024 made it with a product that had no long-term memory of them. Memory arrived afterwards, and was rewritten a year later. The deal was renegotiated by one party, and the other party found out by noticing that the chatbot had opinions about their job.
Consent given to a product that tidies emails is doing heroic work by the time the conversation reaches a medical result.
None of this requires anyone to be stupid. It requires an interface that feels like talking and a backend that behaves like a database, which is exactly what has been built. The conversational style is the part of the business that produces the training data, and it works far better than any form ever did.
For anyone building one of these things, that is the uncomfortable bit. The friendliness is load-bearing. Make the interface feel more like a transaction, more like a form, and people will tell it less. Which is good for them and inconvenient for the roadmap.
The Eagles worked out the mechanics of this in Hotel California, long before anyone had a memory feature. Checking out is allowed. Leaving is the part that does not happen.