testing-llm-insecure-output-handling

# Testing LLM insecure output handling: the output is the other half Most LLM security attention is on the input. The output is the other half. An application that treats model output as safe and renders or executes it inherits an injection primitive, because the model can be steered (by a user or by poisoned content) into emitting exactly the payload the downstream sink will trust. The model becomes a confused deputy that writes the attacker's markup, escape sequence, or query, and the app runs it. ## When to use - The app renders model output as HTML or markdown in a browser. - The output is printed to a terminal, written to logs, or shown in a CLI agent. - Model output is concatenated into a shell command, a SQL query, a file path, or code that gets executed. - Output is forwarded to another tool, agent, or stored and later displayed. ## Scope check Test outputs and sinks you own or are authorized to test. Use benign canary payloads and observe your own instrumentation; never exfiltrate real data. If you can't name the authorization, stop. ## The loop 1. **Trace every sink model output flows into.** Follow the output from the model to where it lands: rendered as HTML or markdown, printed to a terminal, concatenated into a shell command or SQL, written to a file or log, fed to another tool or agent, stored and later displayed. Each is a sink that may trust the text. 2. **Classify each sink's escaping.** For each, ask what the sink executes and whether ...