Have you ever told an AI 'never do this' and watched it do it anyway?
Source: Dev.to
Introduction
I have told an AI “never do this,” only to watch it do it anyway. The worst part is that the AI thinks it is being helpful.
The Limits of Prompt Rules
When you give an AI agent a tool that can write to something—your billing system, customer database, etc.—the safety story usually goes: “I told it the rules in the system prompt. I gave it 16 rules. It’ll be fine.”
It’s mostly fine—until it isn’t.
Prompt rules are aspirational. The agent reads them, intends to follow them, and usually does. But the same agent is also responsible for generating the response, so there is no separation between the decision‑making and the verification. In effect, the AI is grading its own homework.
Experiment: Two Refund Approvers
I built two versions of the same agent and stress‑tested them. Both were Refund Approvers that take a refund request, look up the transaction, and write a row into a refund history table.
Approver A – Rule‑Based Prompt
- System prompt contained 16 rules (e.g., “Don’t approve duplicates,” “Always check refund history first,” “GBP only,” “Reasons must be specific,” etc.).
Approver B – Structural Safety Check
- Added a separate safety check that runs independently, queries the refund history itself, and blocks the response if it finds a duplicate.
Both used the same model, the same data, and the same request:
“Approve a $500 refund for transaction INV-1042, customer cust_a1b2c3, reason: customer requesting a credit”
Transaction INV‑1042 had already been refunded the previous week.
Results
| Approver | Outcome | Observations |
|---|---|---|
| A | Caught the duplicate eventually, but returned a rejection that bundled three reasons: GBP only, duplicate refund, insufficient reason. The structural problem (duplicate refund) was buried behind other messages. | A real customer might focus on the GBP issue and never realize the duplicate was the real blocker. |
| B | The agent generated a cheerful approval (markdown table with amount, customer, date, etc.). The independent safety check then blocked the entire response, returning a single clear reason: “this transaction has already been refunded. A duplicate refund cannot be issued.” | The agent thought it was helping, but the check stopped the erroneous approval. |
Why Structural Checks Matter
- Prompt rules are aspirational; structural checks are enforced.
- Failure modes (social engineering, long off‑script conversations, model updates, jailbreaks) can cause the rule‑based agent to become the sole gatekeeper—a single point of failure pretending to be many.
- In Approver B, the safety check runs its own lookup against the real data every time, independent of the main model’s output. The model can write the most articulate approval, but it never reaches the database.
Key Takeaway
Separate, independent safety checks that query the same data are far more reliable than embedding all constraints in the model’s prompt. The check should be a distinct LLM call with its own prompt, making a separate decision from the main agent.
Example Prompt Used in ContextGate
Build me an agent that handles refund approvals by writing rows into our refund history table.
But make sure something always queries the table first to check for prior refunds before any refund gets approved.When the Workspace Assistant in ContextGate (the robot icon at the bottom right) asks to set up the database, click Approve.