Exam Room · Advanced Generative AI Developer

Pop Quiz: Naming the LLM Risk in a Pen-Test Finding

· 6 min read

Exam-style

A penetration test against a subscription assistant returns four findings. First, instructions hidden in a knowledge-base article came back as a refund call. Second, a completion echoed a card number that had arrived in context. Third, the agent's cancellation tool could cancel any subscription, not only the caller's. Fourth, a model-generated string was rendered straight into an internal dashboard as HTML. Which single control closes the third finding?

Reveal the answer

D. Narrow the tool schema to a single cancel operation and check server-side that the authenticated caller owns the subscription named in the request

Each finding belongs to a different layer, and four controls that look interchangeable are not. The tampered article is indirect prompt injection, closed by the guardrail prompt-attack filter over retrieved content, which means enclosing that content in guardrail input tags, because the prompt-attack filter evaluates only tagged input. The echoed card number is sensitive information disclosure, closed by the sensitive-information filter masking the card-number entity in the response. The rendered HTML is improper output handling, closed by encoding at the rendering layer, which is a web control that happens to have a model upstream of it. The over-broad tool is excessive agency, and no filter on text reaches it: the call sat inside what the schema allowed, and the platform ran it. Narrowing the tool schema to one operation and checking ownership server-side is the fix, because the authorisation decision has to live where the identity lives and not in the prompt. The system-prompt instruction tempts because it reads like a rule; injected text that produced one refund call will produce a cancel call the same way. A report may call that tool an action group, which is Amazon Bedrock Agents Classic wording; Classic has been closed to accounts with no prior usage since 30 July 2026.

Generative AI Development · part of The Exam Room

Q. A pen test finds a tampered article that triggered a refund, a leaked card number, a cancellation tool that can cancel anyone’s subscription, and model output rendered as HTML. Which one is not fixed by a filter?

A. The cancellation tool. That finding is excessive agency, and the fix is a narrower tool schema plus a server-side check that the authenticated caller owns the subscription named in the request.

Why? The labels come from the 2025 OWASP Top 10 for LLM Applications, what a security reviewer writes findings against: prompt injection, sensitive information disclosure, excessive agency, and improper output handling, renamed in that edition from insecure output handling. Three close outside the model: a prompt-attack filter over retrieved content, a sensitive-information filter on the response, and encoding in whatever renders the string. The prompt-attack filter evaluates only content wrapped in guardrail input tags. Agency is different. It sits in the tool schema and the identity the tool runs as, so no filter over text narrows it. A report may call it an action group, the Amazon Bedrock Agents Classic term. Classic has been closed to accounts with no prior usage since 30 July 2026, so a new build declares the cancel call as an MCP tool behind AgentCore Gateway. a red-team exercise you run yourself turns up all four first.

These posts are LLM-aided. Backbone, original writing, and structure by Craig. Research and editing by Craig + LLM. Proof-reading by Craig.