Privacy and Security: Your Prompts, Injection and Deepfakes

Treat anything you type into a consumer chatbot as something you handed to a company, under that company's current terms, which can change. For most uses that is fine. For a confidential informant's name, a leaked document, a medical record or an unproven accusation against a named person, it is a decision you should make on purpose.

What happens to your prompts

Where your words go depends on whether the company may train on them, how long it keeps them, and who else could make it hand them over.

Settings like these are company policy, not law, and they change. Check the current page before you rely on it, and check again a few months later. Anthropic's own announcement notes that data already used in training stays in models trained on it, even if you change your setting later.

When a court overrides the delete button

In 2025 a federal court in the New York Times copyright case ordered OpenAI to preserve consumer ChatGPT and API content, including chats users had deleted. That obligation ran until September 26, 2025, and covered Free, Plus, Pro and Team users and API customers without zero data retention agreements. Enterprise and education customers were not affected. Later the court ordered OpenAI to produce a de-identified sample of 20 million logs to the plaintiffs.

The lesson for anyone handling sensitive material: "deleted" means deleted under normal operations. A lawsuit, a subpoena or a government demand can freeze data that would otherwise disappear. If a source's safety depends on something never existing on a third party's server, do not put it there.

Prompt injection

A language model cannot reliably tell the difference between instructions from you and instructions that arrive inside the material it is working on. Everything is just text. OWASP, the security standards group, lists prompt injection as the number one risk in its 2025 Top 10 for LLM applications.

Direct injection is a user typing something that makes the model ignore its rules, often called jailbreaking. Indirect injection is worse for everyone else: instructions hidden in a web page, email, PDF, résumé or image that the model reads while doing a task for someone. OWASP's examples include a web page with hidden text that makes the model insert a link leaking the conversation, and a résumé with split instructions that combine to give the applicant a glowing review.

The danger grows with what the AI is allowed to do. A chatbot that only writes text can be tricked into saying something false. An "agent" that can read your inbox, browse, fill forms and send messages can be tricked into doing things. OWASP's recommended defenses include giving the model the least access it needs, handling sensitive actions in ordinary code instead of letting the model decide, clearly marking untrusted content, and requiring a human to approve risky actions. None of these makes injection impossible. They limit the damage.

Deepfakes and fraud

Voice cloning now needs only a short sample. The FBI's December 2024 public service alert describes criminals using cloned voices of relatives in fake emergencies, AI audio to get into bank accounts, and AI video of executives or police in live video calls.

Recent cases and rulings:

• Politics. In January 2024, robocalls with an AI-generated imitation of President Biden's voice told New Hampshire voters to skip the primary. The FCC fined the political consultant behind them $6 million in September 2024, and the carrier that transmitted them agreed to a $1 million fine. He also faced state criminal charges for voter suppression and impersonating a candidate; a New Hampshire jury acquitted him on all of them in June 2025.
• Corporate. In early 2024 an employee of the engineering firm Arup joined a video call with what looked like senior managers and sent about $25 million to criminals. Everyone else on the call was a deepfake.
• Regulation. On February 8, 2024, the FCC ruled that AI-generated voices count as "artificial" under the Telephone Consumer Protection Act, so AI-voiced robocalls without consent are illegal.

States have added their own laws. Idaho passed three in 2024: HB 664 on deceptive synthetic media about candidates (lesson 8 covers it), HB 465 on AI-generated child sexual abuse material, and HB 575 on disclosing explicit synthetic media of a person without consent.