The Self-Evolving Agent: How to Build Closed-Loop AI Systems That Write and Optimize Their Own Code

Source: Dev.to

We have all been there. You spend hours meticulously crafting the perfect system prompt or tool description for your AI agent. It performs beautifully in your initial tests. But a week later, production data throws a curveball. The team’s coding standards shift, edge cases emerge, or the underlying LLM updates, and suddenly your agent’s performance degrades.

To fix it, you have to manually inspect the logs, diagnose the failure pattern, rewrite the prompt, and run manual tests.

This is an open-loop system. It relies entirely on an external controller—you, the human engineer—to close the loop between performance feedback and behavioral adjustment.

But what if your agent could close this loop itself? What if it could measure its own performance, reflect on its failures, and autonomously rewrite its own instructions, tool descriptions, and code to adapt to new environments?

This isn’t science fiction; it is autonomous evolution. In this article, we will unpack the engineering principles behind self-improving agents and build a complete, production-grade Python library that allows an agent to autonomously optimize its own skills using DSPy and genetic algorithms.

(The concepts and code demonstrated here are drawn from my ebook Hermes Agent, The Self-Evolving AI Workforce)

The Thermodynamics of Software: The Closed Learning Loop

To understand why autonomous evolution is necessary, let’s borrow an analogy from classical physics: the steam engine.

A primitive steam engine requires a human operator to constantly adjust valves to keep the pressure and speed stable under changing loads. This is an open-loop system. The invention that truly unlocked the Industrial Revolution was James Watt’s centrifugal governor. This simple mechanical device used feedback: as the engine spun faster, centrifugal force threw flyballs outward, which mechanically choked the steam valve, slowing the engine down. If the engine slowed, the balls fell, opening the valve.

The engine did not need a human to think; it had an internal feedback mechanism that modulated its own inputs based on its current load.

+-------------------------------------------------------------+
|                      CLOSED LEARNING LOOP                   |
|                                                             |
|   +------------------+           +----------------------+   |
|   |  Current Skill   | --------> |  Fitness Evaluation  |   |
|   |   (Prompt/Code)  |           | (Heuristic / LLM)    |   |
|   +------------------+           +----------------------+   |
|            ^                                |               |
|            |                                v               |
|   +------------------+           +----------------------+   |
|   |    Validated     |           |  Persistent Memory   |   |
|   |    Mutation    |           |  (Feedback / Scores) |   |
|   +------------------+           +----------------------+   |
|            ^                                |               |
|            |                                v               |
|   +------------------+           +----------------------+   |
|   |    Constraint    |  response")

    def forward(self, task: str) -> dspy.Prediction:
        # Inject the instruction dynamically into the predictor's context
        with dspy.settings.context(instruction=self.instruction.get()):
            return self.predictor(task=task)

class ConstraintValidator:
    """Ensures evolved skills do not break safety, structural, or length constraints."""
    def __init__(self, max_chars: int = 1500):
        self.max_chars = max_chars

    def validate(self, original_skill: str, evolved_skill: str) -> Tuple[bool, str]:
        if len(evolved_skill) > self.max_chars:
            return False, f"Evolved skill length ({len(evolved_skill)}) exceeds limit of {self.max_chars} characters."

        # Prevent wiping out core functional hooks
        if "DO NOT" in original_skill and "DO NOT" not in evolved_skill:
            return False, "Evolved skill stripped out critical safety constraints ('DO NOT' clauses)."

        return True, "Passed all structural constraints."

class SyntheticDatasetBuilder:
    """Generates synthetic test cases based on the skill's description to evaluate performance."""
    def __init__(self, model_name: str):
        self.model_name = model_name

    def generate(self, skill_text: str, num_examples: int = 5) -> List[Dict[str, str]]:
        console.print(f"[bold blue]\[Dataset][/bold blue] Generating {num_examples} synthetic test cases using {self.model_name}...")
        # In practice, this calls an LLM to generate diverse inputs and expected outputs
        # We return a structured mock dataset representing a code-review task
        return [
            {
                "task": "def add(a,b):\nreturn a+b", 
                "expected": "Error: Missing spaces around operators, missing docstring, missing type hints."
            },
            {
                "task": "import os\ndef run_sys(cmd):\n    os.system(cmd)", 
                "expected": "Error: Security vulnerability: os.system call detected. Use subprocess with safety checks."
            },
            {
                "task": "class user:\n    def __init__(self, name):\n        self.name=name", 
                "expected": "Error: Class name 'user' should follow CamelCase naming conventions."
            },
            {
                "task": "def calculate_area(radius):\n    return 3.14 * radius ** 2",
                "expected": "Error: Missing type hints and docstrings. Consider using math.pi instead of a hardcoded float."
            },
            {
                "task": "def get_data(timeout=10):\n    pass",
                "expected": "Error: Missing docstring, missing return type hint."
            }
        ][:num_examples]

# --- Main SkillEvolver Implementation ---

class SkillEvolver:
    """
    Orchestrates the autonomous evolution of an agent's skill.
    Loads a skill -> Generates a test suite -> Iteratively mutates instruction -> Validates -> Saves.
    """
    def __init__(
        self,
        skill_name: str,
        initial_instruction: str,
        iterations: int = 3,
        eval_model: str = "gpt-4o-mini",
        max_instruction_length: int = 1000,
    ):
        self.skill_name = skill_name
        self.instruction = initial_instruction
        self.iterations = iterations
        self.eval_model = eval_model

        self.validator = ConstraintValidator(max_chars=max_instruction_length)
        self.dataset_builder = SyntheticDatasetBuilder(model_name=eval_model)

        self.history: List[Dict[str, Any]] = []
        self.best_instruction = initial_instruction
        self.best_score = 0.0

    def heuristic_fitness(self, expectation: str, actual_output: str) -> float:
        """
        Fast, cheap evaluation metric.
        Measures semantic overlap and length penalties to score agent responses.
        """
        words_expected = set(expectation.lower().split())
        words_actual = set(actual_output.lower().split())

        if not words_actual:
            return 0.0

        intersection = words_expected.intersection(words_actual)
        overlap_score = len(intersection) / max(len(words_expected), 1)

        # Length penalty: discourage overly verbose or completely empty answers
        length_ratio = len(actual_output) / max(len(expectation), 1)
        penalty = 1.0 if (0.5  float:
        """Runs the entire evaluation dataset against a specific instruction set."""
        total_score = 0.0
        # Configure DSPy with the current instruction
        module = SkillModule(instruction)

        for example in dataset:
            # Simulate prediction output based on the instruction strength
            # In a live environment, this calls: module(task=example["task"])
            # For demonstration, we simulate a response that improves if the instruction contains specific keywords
            simulated_response = "Error: "
            if "type hints" in instruction.lower():
                simulated_response += "missing type hints, "
            if "docstring" in instruction.lower():
                simulated_response += "missing docstring, "
            if "security" in instruction.lower() or "vulnerability" in instruction.lower():
                simulated_response += "security vulnerability detected, "
            if "naming" in instruction.lower() or "camelcase" in instruction.lower():
                simulated_response += "naming conventions violated, "

            simulated_response = simulated_response.strip(", ")

            score = self.heuristic_fitness(example["expected"], simulated_response)
            total_score += score

        return round(total_score / len(dataset), 3)

    def simulate_mutation(self, current_instruction: str, feedback: str) -> str:
        """
        Simulates the GEPA optimizer mutating the instruction text.
        In production, this calls an LLM with a metaprompt instructing it to mutate
        the prompt based on historical failure feedback.
        """
        # Simulated mutations adding critical behavioral requirements based on feedback
        mutations = [
            current_instruction + "\n- Ensure you check for missing type hints and docstrings in every function.",
            current_instruction + "\n- Actively detect security vulnerabilities like hardcoded credentials or dangerous system calls.",
            current_instruction + "\n- Verify class names follow CamelCase and functions follow snake_case naming conventions.",
        ]
        # Cycle through mutations based on history length
        return mutations[len(self.history) % len(mutations)]

    def evolve(self) -> Dict[str, Any]:
        """Runs the closed-loop optimization cycle."""
        console.print(f"\n[bold green]\[Evolution Loop][/bold green] Starting autonomous evolution for skill: '{self.skill_name}'")
        console.print(f"  Initial Instruction length: {len(self.instruction)} characters")

        # 1. Build the evaluation dataset
        dataset = self.dataset_builder.generate(self.instruction, num_examples=5)

        # 2. Evaluate baseline performance
        self.best_score = self.evaluate_skill_performance(self.instruction, dataset)
        console.print(f"  [bold yellow]Baseline Fitness Score:[/bold yellow] {self.best_score:.3f}\n")

        current_instruction = self.instruction

        # 3. Optimization Loop
        for generation in range(1, self.iterations + 1):
            console.print(f"[bold magenta]\[Generation {generation}/{self.iterations}][/bold magenta]")

            # Generate a mutated instruction candidates
            feedback = f"Improve coverage of PEP 8 rules and security flags. Current score: {self.best_score}"
            mutated_candidate = self.simulate_mutation(current_instruction, feedback)

            # Validate constraints
            is_valid, validation_msg = self.validator.validate(self.instruction, mutated_candidate)
            if not is_valid:
                console.print(f"  [bold red]Mutation Rejected:[/bold red] {validation_msg}")
                continue

            # Evaluate mutated candidate
            candidate_score = self.evaluate_skill_performance(mutated_candidate, dataset)
            console.print(f"  Proposed Mutation Score: {candidate_score:.3f}")

            # Selection step
            if candidate_score > self.best_score:
                improvement = ((candidate_score - self.best_score) / max(self.best_score, 0.01)) * 100
                console.print(f"  [bold green]Success![/bold green] Score improved by +{improvement:.1f}%")
                self.best_score = candidate_score
                self.best_instruction = mutated_candidate
                current_instruction = mutated_candidate
            else:
                console.print("  [yellow]Mutation discarded (no performance improvement).[/yellow]")

            self.history.append({
                "generation": generation,
                "score": candidate_score,
                "instruction_preview": mutated_candidate[-80:]
            })
            print("-" * 60)
            time.sleep(0.5)

        # Calculate final improvement
        total_improvement = self.best_score - self.evaluate_skill_performance(self.instruction, dataset)

        console.print("\n[bold green]\[Evolution Complete][/bold green]")
        console.print(f"  Final Best Score: [bold green]{self.best_score:.3f}[/bold green]")
        console.print(f"  Absolute Improvement: [bold green]+{total_improvement:.3f}[/bold green]")

        return {
            "skill_name": self.skill_name,
            "original_instruction": self.instruction,
            "evolved_instruction": self.best_instruction,
            "score_improvement": total_improvement,
            "history": self.history
        }

# --- Execution Example ---
if __name__ == "__main__":
    # Define a basic, naive code review prompt
    naive_review_prompt = (
        "You are an AI code reviewer. Analyze the provided Python code and list any "
        "errors or bad practices you find. Keep your answers concise. DO NOT output code unless requested."
    )

    evolver = SkillEvolver(
        skill_name="pep8-reviewer",
        initial_instruction=naive_review_prompt,
        iterations=3,
        eval_model="gpt-4o-mini"
    )

    results = evolver.evolve()

    print("\n=== EVOLVED INSTRUCTION RESULT ===")
    print(results["evolved_instruction"])
    print("==================================")

Enter fullscreen mode


Exit fullscreen mode

Step-by-Step Code Breakdown: How It Works

Let’s dissect the engineering patterns implemented in the code above:

1. Dynamic Instruction Injection (SkillModule)

We wrap our agent’s instruction inside a DSPy Module. Instead of hardcoding prompts, we use a dynamic variable (self.instruction = dspy.Value(instruction)). This allows our optimizer to swap out the underlying instructions on the fly during evaluation loops without having to re-instantiate the core prediction pipeline.

2. Guardrails Against Evolutionary Drift (ConstraintValidator)

When language models write their own prompts, they can easily drift. An optimizer trying to maximize a score might strip out safety checks to save tokens, or write instructions that are 10,000 words long.

The ConstraintValidator acts as a hard gate. If a mutation exceeds our maximum character limit or strips out critical safety phrases (like "DO NOT" clauses), the mutation is instantly killed.

3. Automatically Generating the Curriculum (SyntheticDatasetBuilder)

An evolutionary system is only as good as its test suite. If you don’t have a dataset, the agent cannot evaluate itself.

The SyntheticDatasetBuilder solves this cold-start problem. It takes the original skill description, calls an LLM, and asks: “What are 5 highly diverse inputs that would thoroughly test an agent trying to perform this skill, and what are the ideal outputs?” This creates an instant bootstrapping dataset to drive the evolution loop.

4. The Heuristic Fitness Score (heuristic_fitness)

To keep the evolution fast and cost-effective, we use a heuristic score that evaluates output length penalties and keyword alignment against the expected target.

By comparing the actual output to the synthetic target, we get a continuous, smooth fitness landscape. This allows the genetic algorithm to make incremental progress rather than dealing with binary pass/fail metrics.

Practical Engineering Trade-Offs

When deploying self-evolving architectures in production, you will face several critical design decisions.

Dataset Size: Overfitting vs. Computational Cost

The Trap: If your evaluation dataset is too small (e.g., 2 examples), the optimizer will aggressively overfit to those specific examples, resulting in a mutated prompt that performs terribly on real-world production data.

The Cost: If your dataset is too large (e.g., 200 examples), running 10 iterations of evolution will require 2,000 LLM calls, resulting in high latency and API bills.

The Sweet Spot: Use a three-way split (Train, Validation, and Holdout) of 15 to 30 highly diverse examples. Use the Validation set for the rapid mutation steps, and run the Holdout set only once at the very end to prove the evolved skill genuinely generalizes.

Mutation Limits

Do not let your agents run infinite evolution loops in production. Set a strict iteration cap (typically 5 to 10 generations). After a certain point, prompt optimization reaches a plateau of diminishing returns, and further mutations risk over-optimizing for the evaluation dataset at the expense of general reasoning capabilities.

The Future: Online Self-Improvement

The implementation we built today runs in an offline development environment. But the ultimate goal of autonomous agent architecture is online evolution.

Imagine an agent running in production. When a human user corrects the agent’s output, that correction is automatically flagged, transformed into a new training example, and saved to a persistent database. Every midnight, a cron job spins up the SkillEvolver library, evaluates the day’s failures, runs a genetic optimization loop, and deploys a newly evolved, more robust prompt for the next morning.

By building closed loops, persistent memory, and self-evaluation directly into our software, we stop writing static code and start planting the seeds for systems that grow, adapt, and evolve on their own.

Let’s Discuss

The Safety Dilemma: If an agent is allowed to autonomously modify its own tool descriptions and instructions to maximize performance, how do we mathematically guarantee it will never bypass safety constraints or drift into malicious behaviors?

Heuristics vs. LLMs: In your experience, can simple heuristic metrics (like keyword overlap, length, and regex) reliably guide prompt optimization, or is an expensive LLM-as-Judge strictly necessary to achieve meaningful improvements?

Leave your thoughts in the comments below!

The concepts and code demonstrated here are drawn directly from the comprehensive roadmap laid out in the ebook Hermes Agent, The Self-Evolving AI Workforce: details link, you can find also my programming ebooks with AI here: Programming & AI eBooks.