Treat model output as untrusted input

A support chatbot was persuaded to include an HTML image tag in its answer. The tag carried an onerror handler, which is the oldest trick on the web. The page rendered it.

The interesting part is not the customer’s browser. It is that the same content also reached the support agent’s console, so the payload ran in front of a member of staff with a session that mattered. The analysis lists the failures in order: improper input sanitisation, “improper chatbot output sanitization, the web server not verifying content produced by the chatbot, running unverified code, and loading content from arbitrary web resources.”

Not one of those is a novel AI vulnerability. They are 2005 web vulnerabilities that arrived through a new door, because somebody classified the model’s output as content from our system rather than as a string an attacker influenced.

Every crossing is a sink

Here is the frame that makes this tractable. A model emits text. That text then crosses into something that interprets text: a browser, a shell, a SQL parser, a terminal emulator, a Python process. Every one of those crossings is an injection sink you already know how to defend, and every one of them is now being reached from a direction your existing rules never covered.

The advisories bear this out. The same underlying mistake has been filed as CWE-74 against Microsoft 365 Copilot, CWE-77 against GitHub Copilot, CWE-78 against Cursor, and CWE-94 against Vanna. Four products, four interpreters, one error.

The clearest single case is Langroid, CVE-2026-25879, rated 9.8 critical. Its SQLChatAgent “executes SQL produced by an LLM, which is influenceable by prompt injection.” Given a database role with enough privilege, an attacker shaping the agent’s input can coerce primitives like COPY ... FROM PROGRAM, “achieving RCE on the database host.” The model was asked to write a query. It wrote one, and the query was a program.

Rendering is a sink too, and a quiet one. Johann Rehberger’s write-up of GitHub Copilot Chat explains the mechanism exactly: the assistant “interprets and renders markdown text that is returned from the large language model (LLM), including images”, so instructions hidden in a source file can make the model emit an image link that the client fetches automatically, with chat context appended “as a query parameter.”

Even the terminal counts. A demonstrated attack has an AI tool emitting ANSI escape codes, where “The Terminal app interprets the sequence and issues DNS requests containing the stolen data.” No victim in the wild, but a working proof of concept, disclosed to the vendor in December 2024.

The move

Every place model output leaves your process, treat it as a string an attacker wrote, and pass it through the defence you would already use there.

SINK                    WHAT YOU ALREADY DO FOR USER INPUT

browser, HTML           render as text. if you must allow markup,
                          sanitize, then a strict CSP as a second layer
browser, markdown       strip image and link auto-fetch.
                          a link a human clicks is fine
SQL                     parameterised queries. a generated query is
                          reviewed and bound, never concatenated
shell                   never. pass an argv array, no shell string
terminal                strip ANSI escapes before printing
python / eval           no. if you truly need it, another sandbox,
                          another machine, no credentials
file paths              resolve and confirm inside the allowed root

Nothing in the right-hand column is new. That is the point of the chapter. You do not need an AI security practice for this, you need to apply your existing one to a source you had not classified as hostile.

Being honest about the ranking

The outline for this move treats output handling as a headline risk, and I have to correct that.

OWASP’s 2026 revision moved Improper Output Handling from fifth to tenth, the largest fall on the list. Selling this as a rising threat would be wrong.

But read what that ranking is made of before you relax. The 2026 list is weighted 75% practitioner vote and 25% incident data, so a fall in rank measures attention, not frequency. The CWE spread above accumulated during exactly the period the rank was falling. My reading is that this dropped because it is unglamorous and well understood, not because it stopped happening, and the Lenovo case is what unglamorous and well understood looks like in production.

What it costs

The safe browser API is not the available one. The HTML Sanitizer API is exactly the right tool and MDN’s own banner says it is “not Baseline because it does not work in some of the most widely-used browsers”, with Safari not shipping it at all. Meanwhile setHTMLUnsafe, the variant whose name warns you, has been supported everywhere for years. So you will be reaching for a library rather than the platform, and carrying its update burden.

And escaping everything makes a worse product. Markdown rendering is genuinely nicer, and tables and code blocks are most of why your assistant feels good to use. The trade is not render-versus-plaintext, it is which elements can cause a fetch. Keep the formatting, kill the automatic requests.

Try this week

Trace one response. Pick your most-used AI feature and follow a single model output from the API response to the pixel, writing down every function it passes through.

You are looking for the moment it stops being data. dangerouslySetInnerHTML, a template that is not escaping, a query built by concatenation, a subprocess call with shell=True. There is usually exactly one, and it is usually somewhere nobody has looked since it was written.

Then ask the same question about the second consumer, because Lenovo’s was the expensive one: does this output reach an internal tool, an admin dashboard, a log viewer, or a support console? Those render with more privilege and less scrutiny than anything you show a customer.