Your AI Agent Just Hallucinated a Wire Transfer. Here's How I Stopped It

Source: Dev.to

The problem

An LLM agent decides to send $45,000 to a vendor based on a hallucinated invoice, a mis‑read Slack approval, and a loosely‑matched recipient. By the time you notice, the money is gone.

Why this is happening now

• OWASP Agentic AI Top 10 (late 2025) lists threats that read like a horror show: goal hijacking, tool misuse, privilege escalation via tool chaining, etc.
• 48 % of cybersecurity professionals now cite agentic AI as the number‑one attack vector, yet only ~33 % of enterprises have AI‑specific security controls.

Introducing PIC (Provenance & Intent Contracts)

PIC forces an agent to prove every high‑impact action before it executes. It sits at the action boundary – the moment between “the LLM decided to do something” and “the tool actually runs”.

How it works (in a nutshell)

1. Proposal – The agent submits a structured JSON describing what it wants to do, why, and where the decision data came from.
2. Verification – PIC validates the proposal against a schema, checks provenance trust levels, and verifies any evidence.
3. Decision – If anything is missing, malformed, or untrusted, the proposal is blocked (fail‑closed). No “allow anyway” fallback.

Example Proposal

{
  "protocol": "PIC/1.0",
  "intent": "Execute wire transfer for Q4 server costs.",
  "impact": "money",
  "provenance": [
    { "id": "cfo_signed_invoice_hash", "trust": "trusted" },
    { "id": "slack_approval_manager", "trust": "semi_trusted" }
  ],
  "claims": [
    {
      "text": "Invoice hash matches authorized payment list",
      "evidence": ["cfo_signed_invoice_hash"]
    }
  ],
  "action": {
    "tool": "treasury.wire_transfer",
    "args": { "recipient": "AWS_Global_Payments", "amount": 45000 }
  }
}

Required Fields

Core Verification Rule

• High‑impact actions (money, privacy, irreversible, …) must have at least one claim backed by evidence from a trusted provenance source.
• Missing fields, schema violations, or any verification error → blocked.

Fail‑closed by design.

Installation & CLI

pip install pic-standard

Verify a proposal

# Trusted provenance + valid evidence → passes
pic-cli verify examples/financial_irreversible.json

# Bad SHA‑256 hash → blocked
pic-cli verify examples/failing/financial_hash_bad.json --verify-evidence

The first command succeeds; the second fails because the evidence hash does not match the artifact, so the action never runs.

Mapping PIC to the OWASP Agentic AI Top 10

Integrations (plug‑and‑play)

Security & Hardening Details

• Tests: 108 tests across 18 files (schema, verifier rules, evidence handling, keyring, integrations, HTTP bridge, pipeline).

• Impact classes: 7 formal classes with explicit evidence requirements.

• Evidence types: SHA‑256 hash verification and Ed25519 digital signatures.

• Keyring: Trusted keys with expiry timestamps and revocation lists.

• DoS hardening:

  • Max proposal size: 64 KB
  • Evaluation budget: 500 ms per proposal
  • Max evidence file: 5 MB
  • Max HTTP body: 1 MB
  • Socket timeout: 5 s

• Formal specification: RFC‑0001 (includes a 7‑threat model extension).

TL;DR

• Guardrails (NeMo Guardrails, Guardrails AI, etc.) protect what the model says but not what the model does.
• PIC fills that gap by enforcing provenance, intent, and evidence at the action boundary.
• It is fail‑closed, deterministic, and zero‑dependency (local verification).
• Plug‑in to any existing LLM‑agent stack with a single pip install or a tiny HTTP bridge.

Secure your agents. Verify before they act.

PIC‑Standard Overview

• Core concepts: causal taint semantics, action‑boundary gating, provenance bridging
• License: Apache 2.0 (defensive publication – concepts are documented and timestamped to prevent patenting)

Quick Start

pip install pic-standard
pic-cli verify examples/financial_irreversible.json

That’s one command to verify your first proposal. From there you can:

• Read the Quickstart guide
• Browse the example proposals (both passing and failing)
• Check the RFC for the formal specification

Who Should Use This

If you are building AI agents that handle money, user data, or any irreversible actions, this is the missing safety layer.

Resources

• GitHub:
• PyPI:
• License: Apache 2.0

Contributing

If you find this useful, ⭐️ the repository – it helps more than you might think.