Optimizing .NET 8 API Consumption at Scale: A Technical Deep Dive into Concurrency, Batching, and Resilient Retry Mechanisms

Source: Dev.to

Introduction

When architecting systems that rely on external APIs, it is paramount to anticipate and mitigate potential scaling bottlenecks, such as rate limiting. This article details the technical strategies employed using .NET 8 to successfully scale the consumption of a third‑party API from an initial volume of 500+ requests to over 10,000, significantly reducing processing time and error rates.

Initial Implementation

The first version worked for low volumes but failed to scale:

• Processing time: ~30 minutes for 10,000 requests
• Failure rate: ~50 % (mostly 429 Too Many Requests errors)

The workload consisted of iterating over 10,000 unique URLs, each requiring an individual API call, e.g.:

https://www.example.com/get?id=123

Core Optimizations

Batching

Instead of processing all requests sequentially or flooding the API with concurrent calls, a structured batching strategy was introduced. Requests are grouped into batches, and each batch is sent only after the previous one completes.

Dynamic Delay Calculation

To maximize throughput without violating the rate limit, the delay between batches is calculated dynamically:

// Delay for the next batch (in seconds)
double delayNext = 10.0 - processingTimePrevious;

If processingTimePrevious exceeds 10 seconds, the delay becomes zero, allowing immediate dispatch of the next batch.

Controlled Concurrency with Resilience

A global concurrency limit is enforced using the .NET 8 SemaphoreSlim primitive:

private static readonly SemaphoreSlim _semaphore = new SemaphoreSlim(50); // max 50 concurrent requests

public async Task SendAsync(HttpRequestMessage request)
{
    await _semaphore.WaitAsync();
    try
    {
        return await _httpClient.SendAsync(request);
    }
    finally
    {
        _semaphore.Release();
    }
}

• Limits concurrent outbound calls to 50.
• Prevents overwhelming the external API and controls memory/thread usage.

Asynchronous Processing and Queuing

The pipeline leverages async/await throughout, allowing the thread pool to remain responsive while I/O‑bound API calls are in flight. Requests are queued internally, and workers dequeue items respecting the semaphore limit.

Resilience Policies (Microsoft.Extensions.Resilience)

Retry Policy

var retryPolicy = Policy
    .Handle()
    .OrResult(r => r.StatusCode == HttpStatusCode.TooManyRequests)
    .WaitAndRetryAsync(
        retryCount: 3,
        sleepDurationProvider: attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt))
    );

• Retries transient failures with exponential back‑off.
• Specifically targets 429 responses.

Circuit Breaker Policy

var circuitBreaker = Policy
    .Handle()
    .OrResult(r => r.StatusCode == HttpStatusCode.TooManyRequests)
    .CircuitBreakerAsync(
        handledEventsAllowedBeforeBreaking: 20,
        durationOfBreak: TimeSpan.FromSeconds(30)
    );

• Opens the circuit after 20 consecutive failures, pausing requests for 30 seconds before allowing traffic again.

Performance Gains

Combining batching, dynamic delays, and resilient retry logic yielded substantial improvements:

Conclusion and Future Scalability

By integrating concurrency control, batching, and robust retry mechanisms in .NET 8, a sluggish API processing pipeline was transformed into a more efficient system, cutting processing time by roughly half. This experience underscores the importance of:

• Understanding third‑party API constraints.
• Leveraging batching for scalability.
• Applying resilient patterns such as retries and circuit breakers.

Future Scalability Efforts

• Adaptive batch sizing based on real‑time latency metrics.
• Distributed processing (e.g., using Azure Functions or Kubernetes) to handle even larger volumes.
• Monitoring and auto‑tuning of rate‑limit thresholds.

These optimizations provide a solid foundation for handling large‑scale API requests, with ample room for refinement as request volumes continue to grow.