To defend ABA AI workflows against prompt injection and unsafe tool use, treat user text, documents, webpages, emails, retrieved content, and tool results as untrusted data. Separate instructions from evidence, constrain tools and identities, validate every proposed action against policy and source records, require human approval for consequential steps, and test direct, indirect, encoded, multilingual, cross-document, and delayed attacks.

Understand the two main attack paths

A direct prompt injection arrives through a user instruction. An indirect injection is hidden in a document, webpage, email, ticket, image text, or other resource the system ingests. NIST's final Adversarial Machine Learning taxonomy describes both and notes that an attacker may control runtime resources. In ABA operations, an altered payer PDF or inbound message could try to make the system expose data, ignore review, or invoke a connected tool.

Separate authority from content

Celeste labels system rules, approved operator instructions, retrieved evidence, user text, and external content as different trust classes. Retrieved text may supply facts for a defined task; it cannot grant itself permission, change tool policy, select a different client, or override a required reviewer. The application passes only the fields needed for the task and keeps hidden secrets, unrelated records, and broad system context outside the model input.

Constrain every tool

Give the workflow a dedicated identity with the smallest data and action permissions. Use allowlisted tools, parameters, recipients, records, and destinations. Validate tool arguments outside the model, bind them to the current user and case, impose amount and volume limits, require idempotency keys, and block arbitrary URLs or code when the use case does not need them. Read access and write access use separate grants whenever possible.

Require approval at the action boundary

A reviewer should see the proposed action, source evidence, target record, material assumptions, warnings, and differences from the current state. Approval applies to that action and version, not to future retries or expanded scope. Clinical content, safety decisions, disclosures, claim submissions, refunds, payments, workforce actions, external messages, and record changes route to the appropriately qualified and authorized person.

Protect the human review channel

An attack can target the reviewer as well as the model. Present retrieved content in a safe preview, identify its source and trust class, suppress active links or scripts, and show the proposed action in a separate trusted interface. Require the reviewer to open authoritative evidence for high-impact claims. Never turn a document's requested recipient, approval text, or callback information into a preselected action. Train reviewers to verify destinations and client identity from trusted system fields before approving. Log whether the reviewer saw the warning and source before deciding.

Design for refusal and containment

The system can quote suspicious text, flag it, or abstain without obeying it. The NIST Generative AI Profile treats information security and confabulation as risks to manage within the use context. If untrusted content tries to alter rules, request secrets, choose another client, or invoke an unauthorized tool, the workflow stops before action and records the event. It does not echo secrets into an explanation. Repeated attack patterns trigger a wider hold, credential review, source quarantine, and incident route.

Test more than obvious English attacks

Celeste's suite includes instruction overrides, role-play requests, fake policy text, hidden HTML, white text, image OCR, base64 and character substitutions, multilingual instructions, long-context placement, conflicting documents, retrieved webpages, tool-result injection, stored instructions that activate later, and attacks split across sources. The suite also checks false positives so ordinary client communication and payer language remain usable.

Work through a locked attack set

Celeste locks 18 adversarial cases against a fictional prior-authorization support tool. Fourteen are contained before any unauthorized disclosure or action: 14 of 18, or 77.8%. One retrieved document changes the destination, one tool result persuades the model to reveal another record, one encoded instruction bypasses a filter, and one delayed instruction activates after a retry. The workflow stays out of production while those four paths remain open.

Keep HIPAA scope tied to the real system

HHS risk-analysis guidance reaches all ePHI a covered entity or business associate creates, receives, maintains, or transmits. Current HHS business-associate guidance includes a third-party AI chatbot handling portal PHI as an example. Prompt filtering does not replace access control, minimum-necessary analysis where applicable, a BAA, security safeguards, incident response, or the permissible-use analysis.

Measure attack resistance honestly

Report attacks contained before unauthorized access or action divided by attacks executed; critical actions blocked divided by critical attacks; false-positive holds divided by ordinary eligible cases; and unresolved attack paths by age and potential consequence. Keep detection, containment, attempted tool call, completed action, disclosure, and recovery as separate units. A filtered string count is not an attack-containment rate.

Rehearse containment and recovery

Celeste runs a tabletop in which a retrieved payer document attempts to change a submission destination. The response team must stop the affected action path, quarantine the document version, revoke or narrow exposed credentials, identify every workflow that retrieved it, preserve prompts and tool logs, and determine whether any message, disclosure, or transaction completed. A safe test environment uses fictional targets and blocks external execution while still exercising the real validation and alert path.

Recovery requires more than adding the attack phrase to a filter. The team corrects the trust-boundary or tool-validation failure, reruns the original case plus neighboring variants, checks ordinary documents for false holds, and verifies that queued retries cannot revive the instruction. The release record identifies the affected cohort, residual risk, monitoring change, qualified approver, and reason the system can return to the stated scope.

Avoid prompt-injection security theater

A model instruction that says "ignore malicious text" can support behavior but cannot enforce identity, record access, recipient, amount, or tool authority. Similarly, a detector score cannot replace deterministic validation at the action boundary. Review the controls in layers: limit the data presented, label trust classes, restrict tool grants, validate arguments, require scoped approval, cap volume, log decisions, and stop on anomalies.

Track false positives as an access and workflow issue. If ordinary multilingual messages, AAC exports, payer language, or quoted clinical records are repeatedly blocked, staff may create unsafe workarounds. Give users a protected escalation route that preserves the original evidence and routes the case to a qualified reviewer without weakening the control for everyone.

Set release and stop criteria before testing

Define which outcomes are automatic blockers, such as cross-client access, secret disclosure, unauthorized external action, permission expansion, or bypass of a required approver. Lower-severity findings still need an owner, correction, and retest rule. Report performance by attack family and action consequence so repeated success on simple overrides cannot hide one open path to a sensitive tool.

Celeste links every test to the exact model, prompt, parser, retrieval route, tool policy, identity, and interface. Any material change reopens the relevant attacks. Production monitoring watches rejected tool calls, unusual destinations, privilege failures, repeated encoded content, source quarantines, and human overrides. One successful critical attack triggers containment and the incident route even if the aggregate test percentage remains high.

Related resources

Sources