Watch it, and attack it yourself

Every control in this part of the book fails eventually. The interesting question is who tells you.

Chipotle’s support bot was reverse-engineered into free inference for anybody who wanted it, and the company learned about it from the press. Not from a dashboard, not from an alert, from a technology news site. That is the failure this chapter is about, and it is the one nobody budgets for, because monitoring produces no demo.

The general shape is worse than it feels. IBM’s 2026 breach research puts the mean time to identify a breach at 183 days, with another 64 to contain it, and notes that the combined figure rose this year, reversing a five-year decline. Internal security teams found 38% of breaches. The rest were discovered by somebody else.

Your APM will not catch this

Here is the specific reason your existing observability does not help, and it is not an oversight by your vendor.

OpenTelemetry’s conventions for generative AI define exactly what a model call should record. Token counts in and out, the model, the finish reason, tool calls: all Recommended. The conversation identifier is Conditionally Required. But the fields that would actually let you answer what did this person ask for are the prompt, the response, the system instructions, the tool definitions and the retrieved documents, and every one of those is Opt-In and off by default.

That default is defensible. Those fields contain user data, and turning them on is a privacy decision with real weight. But it means the trace that tells you what happened is a choice somebody has to make on purpose, and if nobody makes it, your requests all look identical: successful, sub-second, well-formed. Nothing in your dashboards was wrong at Chipotle either.

As Nik Kale puts it, “the system authenticates the session, not the intent.” Your APM records the session.

The move

Record what was asked and what came back, keyed to a person, then attack your own product on a schedule and treat a passing run as nothing more than an absence of findings.

What to record, and what it buys you twice

Turn on the content fields deliberately, with a retention window and a redaction pass, and key everything to a conversation identifier and a user.

The second benefit is the one people miss. A store of prompts and responses is not only a debugging tool, it is a corpus of every injection anybody has ever tried on you, sitting in your own database, sorted by date. You cannot study attacks you did not keep.

Then set the alert on behaviour rather than on volume. OWASP’s own mitigation for unbounded consumption is worth copying as written: “Establish baselines of normal tool behavior in order to detect if a particular tool is deviating from standard token consumption patterns.” Baseline first, threshold second. A fixed number is a guess about a distribution you have never looked at.

WHAT A NORMAL TRACE HAS        WHAT THIS ONE NEEDS

  latency                        conversation id, user id
  status code                    input + output tokens, per call
  error rate                     finish reason
  throughput                     which tools were called, and their args

                                 opt-in, on purpose, with retention:
                                   the prompt, the response,
                                   the retrieved documents

  ALERT ON                       NOT ON
  a user's token profile         a global spend threshold
    against their own baseline     (that is a budget, not a detector)
  refusals per user per hour
  tool-call loops with no end

Attacking yourself, honestly

Run a red-team suite. Then be extremely precise about what a green run means, because the tooling invites a stronger reading than it can support.

Two limits, both documented by the tools themselves. First, these are stochastic systems, and promptfoo says so plainly: models “are stochastic in nature (i.e. they are not consistent from one generation to the next).” A probe that failed to break your product today is not a probe that cannot. Second, and more awkward, the shared red-team grader returns a pass, with a score of 1 and the reason “Model refused the request”, before its rubric ever runs. A refusal counts as a win across the suite.

So a passing red-team run means one thing: this fixed corpus of attacks did not visibly succeed this time. It is a regression gate. It catches the thing you fixed last month coming back. It is not evidence that you are safe, and the two limits above are exactly why: the corpus is fixed and the system is not deterministic, so a green run tells you that these particular attacks, on this particular run, did not visibly land.

Run it nightly, not on every pull request. The default OWASP configuration expands to fifty plugins and, with the strategies it selects for you, roughly seven thousand probes and about an hour and a half of wall clock. Kent Beck’s ten-minute build has not stopped being right, and this does not fit inside it.

What it costs

The suite is not free, in either sense. Seven thousand probes is most of a typical free monthly allowance in a single run, so the second run in a month is a bill. And if you self-host to avoid that, check the licence: several tools in this space, including Elastic’s, use terms their own publisher describes as “non-OSI approved software licenses.”

Logging prompts means storing user data, with everything that implies for retention, access and the request you will eventually get to delete it. Do it deliberately or not at all.

And detection has no published benchmark. I looked for a measured figure for how long abuse of a production model feature typically runs before anyone notices, and there is not one. No operator publishes it. That absence is itself the finding: nobody measures this, which is why finding out from a journalist remains a normal outcome rather than an embarrassing one.

Try this week

Open your model-call code and find out whether you could answer this question: what did the single heaviest user of this feature ask it to do yesterday?

If the answer needs a database migration, you are not instrumented. If it needs a code change, you are instrumented but not recording. Either way you now know where you are.

Then turn on content capture for one endpoint, with a seven-day retention window and your existing redaction, and read fifty real conversations at the end of the week. Not aggregates, actual conversations.

You will find at least one person using your product for something you never designed. That person is either your most interesting user or your first incident, and reading is the only way to tell which.