How switching to SQS Batch operations improves Performance an Billing

Published: 1 month ago (December 23, 2025 at 02:36 PM EST)

4 min read

Source: Dev.to

Abstract

In this post, we explore how refactoring SQS message processing from individual SendMessage calls to Batch SendMessage operations can significantly improve application performance and reduce SQS billing costs by lowering IOPS usage.

The idea

When monitoring a Golang application with DataDog, we can measure SQS message sending in detail. By comparing a traditional loop‑based send approach versus batch sending, we can see clear differences in timing, network calls, and resource usage.

Full Datadog tracing of SQS is not supported for all languages

Set the following environment variables on this service to enable complete payload tagging:

DD_TRACE_CLOUD_REQUEST_PAYLOAD_TAGGING=all
DD_TRACE_CLOUD_RESPONSE_PAYLOAD_TAGGING=all

Documentation:

For Golang, you can leverage Datadog attribute tags to inspect payload metadata.

Regular SQS message send operations

Sending messages one by one involves multiple network calls and extra overhead. The tracing diagram below shows how the timing looks when using a loop operation.

Example: Sending 7 messages individually took 175 ms with 7 separate HTTP requests. The first call typically dominates the timing due to DNS lookup and connection setup.

Because the service runs in the same K8s cluster, we can assume the experiment is clean and no additional overhead is present.

Sending messages in a batch

AWS SQS allows sending up to 10 messages per batch. Sending 20 messages in 2 batches demonstrates significant efficiency gains:

Sent 3× more messages.
Made 10× fewer HTTP requests.
Total processing time reduced by ≈ 3×.

Batch send timing diagram

Response examples

When a batch send is performed, the response contains a status for each message, including any errors. The batch can be partially successful; parsing the response allows you to efficiently replay or handle failures.

{
  "Successful": [
    { "ID": "0", "MessageID": "655f3404-fbe4-4c51-8868-b5c604bd5f6d", "Error": null },
    { "ID": "1", "MessageID": "daf36653-9abb-490b-b620-608efa24a219", "Error": null },
    { "ID": "2", "MessageID": "93f4dcfd-0500-4076-90f2-3b880b32c943", "Error": null },
    { "ID": "3", "MessageID": "f6c7b079-98f5-4290-b293-2ac6e43ed6f2", "Error": null },
    { "ID": "4", "MessageID": "2b4a96bc-b4ec-4711-9473-d887dd3213f7", "Error": null },
    { "ID": "5", "MessageID": "1bd30cd9-f9c1-4b47-8d6d-2e23ce771841", "Error": null },
    { "ID": "6", "MessageID": "8eed75ef-2563-442e-a191-6b3dff29d635", "Error": null },
    { "ID": "7", "MessageID": "c65a36ce-7ce0-444c-9974-96648dcae0ea", "Error": null },
    { "ID": "8", "MessageID": "75379265-5f9-4a60-8c3a-0537cffdaa80", "Error": null },
    { "ID": "9", "MessageID": "59239903-d4d9-498f-9a08-6d7d7ae8beba", "Error": null },
    { "ID": "10", "MessageID": "9a614c58-113b-487d-a8f1-7509f93b42f9", "Error": null },
    { "ID": "11", "MessageID": "1077de5c-8f0f-4d5b-a0fe-dca45712bfdf", "Error": null },
    { "ID": "12", "MessageID": "8b0f5836-0e01-4a88-9793-4bac2a6d879a", "Error": null }
  ],
  "Failed": []
}

AWS Console behavior

Batch sending does not change how messages appear in SQS. Each message is stored individually, so consumers don’t need any changes to handle batches.

SQS console view

The same messages, with the same structure, are posted and visible in SQS. Other optimization techniques can be applied to adjust consumer batch size when polling messages from SQS.

Golang implementation example

entry := &sqs.SendMessageBatchRequestEntry{
    Id:          aws.String(fmt.Sprintf("%d", i+idx)), // Unique ID within batch
    MessageBody: aws.String(string(b)),
}

if taskConfig.MessageGroupId != "" {
    entry.SetMessageGroupId(taskConfig.MessageGroupId)
}
if taskConfig.MessageDeduplicationId != "" {
    entry.SetMessageDeduplicationId(taskConfig.MessageDeduplicationId)
}
if taskConfig.DelaySeconds > 0 {
    entry.SetDelaySeconds(taskConfig.DelaySeconds)
}
entries = append(entries, entry)
}

// ...

type BatchResult struct {
    Successful []BatchResultEntry
    Failed     []BatchResultEntry
}

Code Example

// BatchResultEntry represents a single entry in a batch result
type BatchResultEntry struct {
    ID        string
    MessageID string
    Error     error
}

// Send batch
input := &sqs.SendMessageBatchInput{
    QueueUrl: stp.url,
    Entries:  entries,
}

output, err := stp.c.SendMessageBatchWithContext(ctx, input)
if err != nil {
    err = handleSqsErrors(err)
    // Mark all entries in this batch as failed
    for idx := range batch {
        result.Failed = append(result.Failed, BatchResultEntry{
            ID:    fmt.Sprintf("%d", i+idx),
            Error: err,
        })
    }
    continue
}

Dedicated message details

Exactly this messageID was returned in a batch‑success response section.

Additional Things to Check and Optimize

Deduplication Technique

Before sending the messages, perform deduplication. This reduces SQS IOPS usage, decreases processing latency, and lessens the load on the consumer side. It also avoids unnecessary storage reads, rewrites, etc.

Distributed Tracing Frameworks Can Consume SQS Batch Slots

Some distributed‑tracing frameworks propagate meta‑information through async transports like SQS. If you are using them, check the integrations—they can affect the maximum batch size.

Datadog uses one batch element to propagate meta‑information with tracing, which will be consumed and applied to the same trace.
AWS X‑Ray (a proprietary AWS technology) does not utilize any slots in an SQS batch; it sends span/trace info via a UDP server.

Limitations

Message size payload: 1 MiB
Batch size: 10 messages
Payload serialization: JSON

Conclusions

Switching the implementation from a loop‑based SendMessage to a batch SendMessageBatch:

Significantly decreases overall processing time.
Reduces network round‑trips.
Lowers SQS billing (fewer API calls, roughly a 10× reduction).