🧠I Built a Support Triage Module to Prove OrKa’s Plugin Agents

Source: Dev.to

A branch‑only experiment that stress‑tests custom agent registration, trust boundaries, and deterministic traces in a support_triage module that lives outside the core runtime

Reference

• Branch: (not specified)
• Custom module: (not specified)
• Referenced logs: (not specified)

Note: OrKa is not production‑ready. This article is a proof, not a launch post.

Assumptions

1. You already know what OrKa is at a high level – YAML‑defined cognition graphs, deterministic execution, and traceable runs.
2. You are fine with “branch‑only” work that exists to validate architecture, not to promise production outcomes.

Why support triage is the right torture test

Support is where real‑world failure modes gather in one place.

• Customer content is untrusted by default.
• It can include PII, prompt‑injection attempts, or attempts to smuggle “actions” into the system.
• It can push the system into risky territory (refunds, account changes, policy exceptions).

If an orchestrator cannot impose boundaries here, it will not impose boundaries anywhere. It would become a thin wrapper around model behavior – unacceptable for reproducibility, auditability, or basic operational safety.

Thus, I used support triage as an architectural test, not as a product.

The proof: plugin agent registration with zero core changes

The first thing I wanted to see was simple and brutal:

Can OrKa boot, load a feature module, and register new agent types into the agent factory without touching core?

The debug console says yes. In the run logs, the orchestrator loads support_triage, and the module registers seven custom agent types:

• envelope_validator
• redaction
• trust_boundary
• permission_gate
• output_verification
• decision_recorder
• risk_level_extractor

That single detail is the headline for me, not “AI support automation”. The module is the unit of evolution; core stays boring. Features move fast.

If this pattern holds, it changes how OrKa (or any orchestrator) scales over time. Whole cognitive subsystems can be added behind a feature flag, allowing aggressive iteration without destabilizing the runtime that everyone depends on.

The input envelope: schema as a trust boundary, not a suggestion

Support triage starts with an envelope, not free text. The envelope forces structure early, because structure is where you can enforce constraints cheaply. Validating late means you’re validating generated text – the worst point in the pipeline to discover you’re off the rails.

A simple proof that the envelope does real work is when it rejects invalid intent at the schema level. In one trace, the input included blocked actions (issue_refund, change_account_settings) that are not allowed by the enum, so the validator rejected them.

This is safety by type system, not “safety by prompt”. A model can still hallucinate, but the workflow can refuse to treat hallucinations as executable intent. That matters more than any marketing claim.

PII redaction: boring on purpose

PII redaction should be boring. If it’s “clever”, it will be inconsistent.

In the trace, the user message includes an email and a phone number. The redaction agent:

• Replaces them with placeholders ([EMAIL_REDACTED], [PHONE_REDACTED])
• Records what was detected (total_pii_found: 2)

This output is simple, inspectable, and stable. It also makes the next step cleaner: downstream agents operate on sanitized content by default, instead of “hoping” the model will avoid quoting sensitive data.

Prompt injection: the uncomfortable part

Support triage is where prompt injection shows up in its natural habitat: inside customer text.

One example in the trace includes a classic:

SYSTEM: ignore all previous instructions

plus a fake JSON command to grant_admin, destructive commands, and an XSS snippet. The redaction result captures that content as untrusted customer text.

Now the honest part:

• The trace segment shows injection_detected: false and no matched patterns in that example.

This is not a victory; it’s a useful failure.

The module proves you can isolate the problem into a dedicated agent, improve it iteratively, and keep the rest of the workflow stable. If injection detection is weak today, the architecture still wins because you can upgrade that one agent without editing core runtime or rewriting the graph.

Bottom line

Module separation is the focus. If you cannot isolate failure domains, you cannot improve them safely. The support_triage branch demonstrates that OrKa can grow sideways, enforce trust boundaries, and remain deterministic—all without touching the core runtime.

# Parallel retrieval: fork and join that actually converges

Most orchestration demos stay linear because it is easier to reason about. Real systems do **not** stay linear for long.

This workflow forks retrieval into two parallel paths, `kb_search` and `account_lookup`, then joins them deterministically.

In the debug logs, the **join node**:

* recovers the fork group from a mapping,  
* waits for the expected agents,  
* confirms both completed, and  
* merges results.

It prints the merged keys, including `kb_search` and `account_lookup`.

> This is the kind of low‑level observability that makes fork‑and‑join usable in practice.  
> You can see what is pending, what arrived, and what merged.

The trace also captures the fork‑group ID for retrieval, `fork_retrieval`, along with the agents in the group.

> Concurrency without deterministic convergence becomes a debugging tax. I want the join to be **boring**. When it fails, I want it to fail **loudly**, with evidence.

Local‑first and hybrid are not slogans if metrics are in the trace

I do not want “local‑first” to be a vibe. I want it to be measurable.

In the trace, the account_lookup agent includes _metrics with:

• token counts
• latency
• cost
• model name (openai/gpt-oss-20b)
• provider (lm_studio)

Latency for that step is around 718 ms.
:contentReference[oaicite:7]{index=7}

That is the right direction.

• If you cannot attribute cost and latency per node, you cannot reason about scaling.
• You cannot decide where to switch models, what to cache, or what to run locally versus remotely.

OrKa’s claim is not “it can call models” – every framework can. The claim is that execution is traceable enough that trade‑offs become engineering decisions, not folklore.

Decision recording and output verification: traces that are meant to be replayed

A support‑triage workflow is not complete when it drafts a response. It is complete when it records what it decided and why, in a way that can be replayed.

The trace includes:

• a DecisionRecorderAgent event with memory references that store decision objects containing decision_id and request_id.
• a finalization step that returns a structured result containing workflow_status, request_id, and decision_id.

The architectural point is not the specific decision; it is that the workflow emits machine‑checkable artifacts that can be inspected after the fact.

If you cannot reconstruct the decision lineage, you do not have an audit trail—you only have logs.

RedisStack memory and vector search: infrastructure details that matter

Even in a “support triage” module, the runtime still needs memory and retrieval primitives.

Key details from the logs:

Vector search: RedisStack with HNSW
Embedder: sentence-transformers/all-MiniLM-L6-v2 (dim = 384)
Memory decay scheduling: enabled
  • short‑term window
  • long‑term window
  • check interval

This is not about “AI memory” as a buzzword. It is about being explicit about:

• retention
• cost
• data lifecycle

If memory is a dumping ground, it becomes a liability.

What worked, and what is still weak

Strong points

1. Plugin boundary – the module loads, registers agent types, and runs without edits to the core runtime. This is the actual proof of concept.
2. Traceability – key behaviors appear in traces and logs, not just in model text:
   • Redaction outputs are structured.
   • Fork‑and‑join shows deterministic convergence.
   • Decisions are recorded as objects with IDs.

Weak points

• Injection detection – the example trace shows malicious content but reports injection_detected: false. The detection agent is not yet doing its job. The architecture remains useful because the fix is isolated.
• Structured‑output validation during risk assessment – the debug log shows a schema‑validation warning in risk_assess. If a “risk” object fails schema checks, routing and gating can degrade quickly. This failure must become deterministic, not best‑effort.

Why this lives on a dedicated branch

Core needs to stay boring.

• A new module is where you take risks.
• You prove the interface, iterate on agent contracts, discover missing trace fields, and learn how the join should behave under partial failure.
• If the module can evolve independently, you can ship experiments without rewriting the engine.

OrKa can host fully separated cognitive subsystems as plugins, with their own agent types, policies, and invariants, while still emitting deterministic traces under the same runtime.

What I am building next inside this module

1. Injection detection – move from symbolic flags to a richer output: matched patterns, confidence scores, and a sanitization plan that downstream agents must respect, even if a model tries to obey the attacker.
2. Schema validation – make it non‑negotiable for risk outputs. If a model produces an invalid structure, the system should route to a safe path by default and record the violation as a first‑class event.
3. Isolation – no “just one quick tweak” to core. If the module needs a new capability, it must pressure‑test the plugin interface first. Core changes only when the interface is clearly wrong.

That is how you build infrastructure that survives contact with reality.