The Agent Factory: Building Consistent Agents at Scale

Source: Dev.to

Agents Everywhere – The Scaling Problem

Deploying the same agent to multiple events, customers, or product lines is harder than it looks.

• How do you guarantee every instance has the right configuration?
• When something improves, how do you propagate the change without editing each agent by hand?

I developed the Agent Factory pattern while scaling a conference demo from one event to many. This post explains the pattern and shows how to apply it to any project where agents must be deployed consistently across multiple contexts.

What Is an Agent, Exactly?

Strip away the API surface and every agent is built from three things:

In most agent frameworks all three legs are configured once at creation time. This makes agents predictable and auditable, but it also means that when you need the same agent experience across multiple contexts, each context needs its own correctly‑configured instance. Managing that manually doesn’t scale.

A Real‑World Example: Pokemon Card Vending Machine

I used Algolia’s Agent Studio to build a demo for a Pokemon‑card vending machine that we run at Algolia’s conference booths. Attendees chat with an AI agent to:

• Find cards
• Check values
• Claim what they received

The agent pulls all data from a searchable index via an MCP‑like tool.

Each conference gets a fresh machine stocked with that event’s cards, so all three legs must adapt:

• Tool – must point at the event’s index (shoptalk-2026, not etail-palm-springs-2026).
• Prompt – must reference the correct event name, booth number, and context.
• Model – might change based on cost or event (e.g., avoid Gemini at an AWS event).

All three parts can vary per event. How do you stamp out a correctly‑configured agent for each event without doing it by hand?

Why Not a Single Shared Agent?

A single shared agent with access to every event’s indices sounds convenient, but agents are static. A shared agent can’t know which event it’s serving, so it either:

• Searches everything indiscriminately, or
• Requires guardrails that are impossible to enforce reliably.

Thus, one agent per event is the only way to provide proper context. The challenge is keeping them consistent.

The Agent Factory Pattern

Treat agent creation as a repeatable process:

1. Define the three legs as templates.
2. Render them together for each new context.
3. Register the resulting agent ID.
4. Manage the fleet as a unit.

The diagram below (conceptual) shows the flow:

[Template Store] → render (event variables) → [Agent Studio API] → create agent
                                                          ↓
                                                   return agent_id
                                                          ↓
                                            store agent_id in registry

Core Insight

The model, tools, and prompt aren’t independent – they reference the same context. The index name in the tool configuration must match what the prompt tells the agent it’s searching. Rendering them separately lets them drift.

Template‑Based Configuration

Because Agent Studio exposes APIs for agent creation, we store everything as templates with shared placeholder variables and render all three legs together in a single pass.

agent-config.json

{
  "name": "Demo Agent — {{event_name}}",
  "provider": "your-llm-provider",
  "model": "your-model",
  "instructions": "PROMPT.md",
  "tools": [
    {
      "type": "algolia_index_search",
      "index": "catalog_{{event_id}}",
      "description": "Product catalog for the {{event_name}} event."
    }
  ]
}

PROMPT.md (excerpt)

You are an agent at {{event_name}} (booth {{booth}}).
Use Algolia search to help attendees find cards in the vending machine.

• {{event_id}}, {{event_name}}, and {{booth}} are resolved across both files together.
• A missing or mismatched variable fails before any API call is made.
• When the prompt improves or the tool configuration changes, both are updated atomically.

Storing the Agent ID

Agent Studio hosts each agent as a managed endpoint and returns a unique ID at creation time. Your application loads the correct agent at runtime using this ID, but the ID isn’t predictable in advance – you must store it somewhere accessible (e.g., a database, a config file, or another Algolia index). The factory writes each returned agent_id back to an event registry, closing the loop between creation and runtime.

Full Lifecycle Management

Agent creation is only the beginning. When the prompt changes, new models drop, or an event retires, the factory handles updates and deletions.

# New event
python agent.py create shoptalk-2026 "Shoptalk 2026" 701 --publish

# Prompt improvement — preview then push to all existing agents
python agent.py update --all --dry-run
python agent.py update --all --publish

# Retire an event
python agent.py delete etail-palm-springs-2026

• update --all is the beating heart of the factory. One command re‑renders the current templates against every event that has an agent. A prompt improvement or tool fix becomes one command rather than N error‑prone dashboard edits.

Benefits of a File‑Based, Version‑Controlled Setup

• History – Full audit trail of what changed and why.
• Rollback – Ability to revert a prompt that regressed.
• Clarity – Clear record of the exact configuration that produced each agent.

Storing the prompt and configuration as files gives you all the advantages of Git (or any VCS) while keeping the deployment process deterministic and repeatable.

TL;DR

• An agent = model + tools + prompt.
• When you need many agents, template the three legs together.
• Render the templates per context, create the agent via API, and store the returned ID.
• Use a small CLI (the Agent Factory) to create, update, and delete agents in bulk.

With this pattern you can scale from one demo to dozens of events—or any other multi‑tenant scenario—while keeping every agent consistent, auditable, and easy to evolve.

Managing Agent Updates at Scale

Running agents at any given time lets you compare behavior before and after a model upgrade, review a prompt change the same way you’d review a code change, and reason about what actually improved things.

This is especially useful now as the agent space is moving fast—model capabilities improve, better prompting patterns emerge, and the tools themselves evolve. That discipline is easy to skip when you’re managing agents through a dashboard.

The Factory Pattern Split

As the factory matured, a clear split emerged in the code:

That split is the pattern stated as code. I extracted the generic layer into a standalone open‑source CLI — algolia-agent — and a thin demo wrapper that calls the CLI’s API and only contains the three things unique to its domain.

The algolia-agent CLI

The CLI makes the factory pattern accessible for any project:

algolia-agent init        # defines all three parts of your agent interactively
algolia-agent create      # renders templates and calls the API
algolia-agent update      # diffs and pushes changes to existing agents
algolia-agent publish     # promotes a draft to live

init workflow

The init command walks through the definition in order:

1. Pick a provider and model
2. Select tools (or not—an agent backed by a model and a prompt is valid)
3. Write a prompt

Real‑world Prompt Fix

A user at the booth asked:

“Do you have any cards worth more than $50?”

The first version of the prompt had no guidance on numeric filters. The agent searched for “cards worth more than 50 dollars” as a keyword query. It got results, but not the right ones—keyword search matched card descriptions that mentioned the word “worth,” not cards filtered by actual value.

The fix

Add one line to the prompt:

For “more than” / “less than” questions on numeric fields, use comparison operators in searchParams filters rather than facets.

Example:

{
  "searchParams": {
    "filters": "estimated_value > 50"
  }
}

After adding this, the agent correctly translated the question into a filter against the estimated_value attribute, returning only cards actually priced above $50.

Because I was running agents for multiple active events, a single algolia-agent update --all pushed the improvement to every instance simultaneously. The agent at the next conference was better than the one at the previous conference without anyone touching it individually. That’s the payoff of treating the prompt as a versioned artifact and the fleet as a unit.

Safety Nets

The hardest part of managing multiple agents isn’t creating them—it’s keeping them consistent after the fact. Before pushing any change to the fleet, you want to preview exactly what will change:

• How the prompt shifted
• Which tools were added or removed
• Whether the model changed

A wrong change pushed to N agents is N times harder to debug than one caught before deployment. In my CLI, I included a --dry-run flag to show a diff of what would change before calling the API.

Takeaways

• Agent anatomy: three legs—model, set of tools, and prompt.
• Separation of concerns: core agent configuration vs. event‑specific context.
• Factory pattern: scaffold the agent definition, render it per context, and perform consistent CRUD operations across every instance.
• One‑command propagation: when something improves—a better prompt, a model upgrade, a tool fix—one command updates every agent at once.

Feel free to try out my algolia-agent for yourself. I’d also love to hear if the factory pattern is useful to you, or how it can be improved!