API Token Speed Benchmark: Compare LLM API Provider Performance

Source: Dev.to

When developing AI applications, understanding the performance characteristics of different LLM API providers is crucial for making informed decisions. The API Token Speed Benchmark tool provides comprehensive metrics to compare token generation speed, latency, and throughput across multiple providers.

Why Benchmark LLM API Providers?

Different LLM API providers offer varying performance characteristics that can significantly impact your application’s user experience. Factors like Time To First Token (TTFT), tokens‑per‑second throughput, and total generation time vary between providers, models, and even specific API endpoints.

Benchmarking helps you:

• Identify the fastest provider for your specific use case
• Compare latency and throughput across different models
• Verify API connectivity and authentication
• Test new API endpoints or experimental models
• Optimize cost‑performance trade‑offs

Key Performance Metrics

The benchmark tool measures several critical performance indicators:

• TTFT (Time To First Token) – latency before the first token arrives, indicating how quickly the model starts generating a response
• TPS (Tokens Per Second) – generation throughput, showing how fast tokens are produced
• Total Time – complete generation duration from request to final token
• Input/Output Tokens – token counts from API usage data, with fallback estimation at 4 characters per token

Getting Started with Benchmarking

The tool requires Python 3 with the requests library and reads configuration from ~/.openclaw/openclaw.json.

1. List Available Targets

python3 main.py --targets

Run Benchmark on a Specific Target

python3 main.py run --label 

Compare All Targets

python3 main.py run --all

Verify API Connectivity

python3 main.py check --label 

Configuration and Security

The tool reads configuration from ~/.openclaw/openclaw.json. Targets are defined in the models.providers section with baseUrl, apiKey, api format, and model configurations.

Security Best Practice: Never hard‑code API keys in configuration files. Use environment‑variable placeholders, e.g.:

{
  "models": {
    "providers": {
      "my-provider": {
        "baseUrl": "https://api.example.com/",
        "apiKey": "${ANTHROPIC_API_KEY}",
        "api": "openai-completions",
        "models": [
          {
            "id": "model-name",
            "api": "openai-completions"
          }
        ]
      }
    }
  }
}

Advanced Options

The benchmark tool offers several options for fine‑tuning your tests:

• --repeat N – number of runs per prompt level (default: 1)
• --category – run specific prompt categories (short, medium, long)
• --quiet – suppress progress output
• --timeout N – request timeout in seconds (default: 120)
• --table – output as formatted table instead of JSON

Interpreting Results

The benchmark output provides detailed metrics for each test run. Pay attention to:

• Consistency across multiple runs
• Performance differences between prompt lengths
• TTFT vs. throughput trade‑offs
• Token count accuracy and estimation methods

Practical Use Cases

Consider benchmarking when:

• Choosing between API providers for a new project
• Evaluating performance improvements after model updates
• Testing geographic latency differences
• Comparing cost vs. performance across pricing tiers
• Validating API stability before production deployment

Supported API Formats

The tool supports multiple API formats:

• anthropic-messages – Anthropic’s message‑based API format
• openai-completions – OpenAI’s completions API format
• openai-responses – OpenAI’s responses API format

This flexibility allows you to benchmark across different providers using their native API formats while maintaining a consistent testing methodology.

Conclusion

API benchmarking is an essential practice for developers working with LLM services. By systematically measuring and comparing performance across providers, you can make data‑driven decisions that optimize your application’s responsiveness and user experience.

Whether you’re building chatbots, content‑generation tools, or complex AI applications, understanding the performance characteristics of your chosen API providers will help you deliver better products to your users.

Skill can be found at: benchmark/SKILL.md