Security
Chapter 02
On this page5 sections
Data Privacy & PII in the Context Window
The Context windowThe model's working memory — a desk. Pile on too many papers and the earliest ones slide off the edge.Everything the model can see at once: your prompt, the conversation so far, attached files, and its own reply. Past the limit, earlier content is dropped — and nothing persists once the session ends.Full definition is a disclosure boundary. Anything that enters it has been disclosed to the provider — not "might be," has been, the moment the request is sent. Privacy engineering for AI is therefore not about what you intend to share. It is about controlling what crosses that boundary, because the boundary does not care about your intent.
Most teams reason about the PromptThe brief. Closer to writing instructions for a new contractor than to programming.The text you give the model — question, instructions, context. Its quality shapes the quality of the output.Full definition they wrote. The exposure is the prompt the system assembled.
What Actually Enters the Window
A single AgentA worker you delegate to: brief it, and it takes steps on its own. A chatbot answers; an agent acts.An LLM that runs tools in a loop toward a goal — it acts, checks the result, decides the next step, and repeats until done.Full definition turn assembles far more than your message. The context window economics chapter covers the cost of this; here the concern is disclosure:
| Source | Enters the window? | PII risk |
|---|---|---|
| Your typed prompt | Always | The part you control and think about |
| File reads by tools | Yes, full contents | High — config files, fixtures, real exports |
| Tool/command output | Yes | High — DB rows, logs, API responses |
| Retrieved documents (RAGLetting it check the notes before answering, instead of going from memory.Retrieval-Augmented Generation: fetch relevant documents at question time and have the model answer from them. It reduces made-up answers — it does not eliminate them.Full definition) | Yes | High — whatever the retriever matched |
| Project instructions (CLAUDE.md) | Every turn | Low if reviewed, silent if not |
The pattern: the highest-PII content is the content you did not type. An agent answering "why is this test flaky?" may read a fixture full of real customer rows you forgot were there. You disclosed them without ever seeing them.
Retention and Training Are Separate Questions
Conflating these causes bad decisions. They are independent:
- Retention — does the provider store the request, and for how long? Driven by the tier and contract. A zero-data-retention agreement means requests are not persisted past the response.
- Training — could the content improve a future model? Enterprise and API tiers generally exclude this by contract; consumer tiers historically do not by default.
A zero-retention API path can still be the wrong place for regulated data if the contract or region is wrong. "It is not used for training" does not answer "is it allowed to be there at all." Decide both, explicitly, before the first request — not after.
Minimize Before the Boundary, Not After
Redaction only works if it happens before the TokenA chunk of text — roughly three-quarters of a word. The unit the model reads and bills in.The fragments a model breaks text into. Around 100 tokens ≈ 75 English words. Limits and pricing are counted in tokens, not words or characters.Full definition crosses the wire. Redacting a transcript afterward is theater — the disclosure already happened. The durable controls:
- Data minimization at the tool layer. A tool that returns a customer record should return the fields the task needs, not
SELECT *. This is tool result curation applied to privacy, not just token budget. - Synthetic and masked fixtures. The single highest-leverage change: agents read fixtures constantly. If fixtures contain no real PII, the largest silent channel is closed at the source.
- Deliberate priming. Context priming is usually framed as a quality move; it is also a privacy move. Putting the right minimal context in on purpose means the agent does not go spelunking through the filesystem for it.
- Durable facts go to memory, not the prompt. Persisted state you control beats re-disclosing the same data every turn.
The Tradeoff
Aggressive minimization degrades the agent. A model reasoning about masked data sometimes reasons about the mask — it cannot debug a date parser if every date is [REDACTED]. There is a real, unavoidable tension: the data that makes the agent useful is often the data you least want to disclose. Pretending otherwise produces either a leaky system or a useless one. The honest position is per-task: decide how much fidelity this specific task actually needs, and disclose exactly that.
When Not To Minimize
Not all data is sensitive, and treating it as if it were has a cost: redaction pipelines fail, mask poorly, and add latency and bugs. For a public OSS codebase with no secrets and synthetic fixtures, an elaborate redaction layer is pure overhead protecting nothing. Spend the control budget where the disclosure is real. The mechanism for deciding where that is comes next — A Threat Model for AI Systems.