I Built a Tool to Test Whether Multiple LLMs Working Together Can Beat a Single Model

Source: Dev.to

The Question

Can you get a better answer by having multiple LLMs collaborate than by just asking one directly?

That’s the thesis behind Occursus Benchmark — an open‑source benchmarking platform that systematically tests multi‑model LLM synthesis pipelines against single‑model baselines across 4 providers and 22 orchestration strategies.

Screenshots

What It Does

Occursus Benchmark runs the same task through 22 different orchestration strategies — from a simple single‑model call to a 13‑call graph‑mesh collaboration — and scores every output using dual blind judging (two frontier models score independently on a 0‑100 scale, then the scores are averaged). This tells you whether adding pipeline complexity actually improves quality, or just burns tokens and money.

The tool supports four LLM providers:

You can toggle models on and off; the tool auto‑assigns them to pipeline roles (generator, critic, synthesizer, reviewer).

The 22 Pipelines

Notable Pipeline Architectures

Two Ways to Call LLMs: API or Subscription

One of the key features is dual provider mode — you choose how the tool connects to cloud LLMs.

API Mode (Default)

• Standard REST API calls using your API keys.
• Full control over temperature, max_tokens, and concurrency.
• Costs roughly $0.01‑$0.05 per call depending on the model.

Subscription CLI Mode

Routes calls through your existing paid subscriptions at $0 extra cost:

This makes running large benchmark suites economically viable — a full 22‑pipeline × 8‑task run with dual judging makes ~700+ LLM calls. At API rates that’s $50‑$100; with subscription mode it’s essentially free.

Trade‑off: Subscription CLIs don’t expose temperature or max_tokens, so advanced toggles (token‑budget management, adaptive temperature) only work in API mode.

How It Works

1. Toggle models on/off – 6 preset models across 4 providers, simple checkboxes.
2. Select pipelines and tasks – Choose which strategies to benchmark against which problems.
3. Click Run – The tool auto‑assigns models to roles:
   • Claude or GPT as the primary generator and synthesizer
   • The other as critic and alternative generator
   • Gemini for diversity in multi‑model pipelines
   • Ollama for speed
4. Watch results stream in – Real‑time Server‑Sent Events update a score matrix, bar charts, and statistics as each cell completes.
5. Dual blind judge – Both Claude and GPT score every output independently. Scores are averaged into a single 0‑100 result. Neither judge knows which pipeline produced the output.

4 Task Suites

Thesis‑Suite Highlights

Bottom line: Occursus Benchmark lets you empirically answer whether multi‑model collaboration truly beats a single, powerful LLM — and it does so with a reproducible, open‑source framework.

ts and calculate EBITDA

These are problems where a single LLM routinely drops constraints, misses cross‑domain connections, or loses track of conflicting information.

Enhancement Toggles

Beyond pipeline selection, the tool offers toggles that modify how every pipeline behaves:

• Chain‑of‑Thought – Forces step‑by‑step reasoning before the final answer.
• Token Budget Management – Reserves 60 % of the token budget for the synthesis step (prevents verbose intermediate steps from starving the final answer).
• Adaptive Temperature – Auto‑classifies tasks (factual / code / analytical / creative) and sets the optimal temperature.
• Repeat Runs – Run each cell 1 / 3 / 5 times, reporting mean ± std dev for statistical significance.
• Cost Tracking – Displays estimated $ per pipeline using published per‑token pricing.

Early Results

We have now run two full benchmark rounds across the thesis task suite (8 hard tasks including cross‑domain synthesis, multi‑file code refactoring, constraint satisfaction, and needle‑in‑haystack analysis). Below are the combined results.

Latest Run: Research Pipelines on Thesis Tasks

Adaptive Debate won – its specialist debaters (logical reasoner, factual verifier, strategic planner) consistently improved answers through two rounds of domain‑specific critique. Reflexion and Graph‑Mesh were close behind.

Surprise: Self‑MoA underperformed the baseline by 4.8 points, contradicting the Princeton paper’s finding. On these hard tasks, same‑model temperature diversity wasn’t enough – the specialist critique from different model perspectives in Adaptive Debate added genuine value.

Combined Top 5 Pipelines (Across Both Runs)

The baseline single model scored 81.8 – 82.1 across both runs – meaning pipelines need to score above ~82 to justify their additional LLM calls.

Key Findings

• Multi‑model pipelines DO beat single‑model on hard tasks, but only by a narrow margin (1‑2 points) rather than the 10‑20‑point gaps reported in research papers.
• Gains come from:
  • Selection pressure (Sample & Vote, Ranked Merge): generating multiple candidates and picking the best consistently outperforms single‑shot.
  • Specialist critique (Adaptive Debate, Constraint Checker): domain‑specific feedback catches errors that a generic single model misses.
  • Self‑correction (Reflexion): articulating why something failed produces better retries than a simple critique‑revise loop.

What Doesn’t Work Well

Tech Stack

• Backend: Python, FastAPI (fully async)
• Providers: Ollama, OpenAI, Anthropic, Gemini – with auto‑routing by model name and exponential‑backoff retries
• Frontend: Vanilla HTML/JS/CSS, Chart.js, Server‑Sent Events
• Storage: SQLite (WAL mode), CSV/JSON export
• Pipelines: 10 module files implementing 22 strategies, all sharing a common BasePipeline interface

Try It

git clone https://github.com/rich1398/Multi-Model-Benchmarking.git
cd Multi-Model-Benchmarking
pip install -r requirements.txt
python app.py

1. Open http://localhost:8000.
2. Configure your API keys (or just use Ollama for free local testing).
3. Run your first benchmark.

GitHub:

This is an active research project. The next benchmark run will test all 22 pipelines against the thesis task suite with enhancement toggles enabled. If you have ideas for pipeline architectures that might beat single‑model baselines, open an issue or PR.