Solved: DynamoDB errors in ap-southeast-2

Source: Dev.to

TL;DR

DynamoDB errors in ap‑southeast‑2, often showing up as ProvisionedThroughputExceededException or connection timeouts, are frequently caused by localized network “grey failures” within a specific Availability Zone—not capacity issues. Solutions range from a quick instance reboot to robust architectural fixes such as tuning AWS SDK client time‑outs and implementing a DynamoDB Gateway VPC Endpoint for private network connectivity.

Why It Happens

• AWS regions = collections of Availability Zones (AZs).
• A “grey failure” in a single AZ can disrupt DynamoDB connectivity even when the overall region status is green.
• The AWS SDK resolves dynamodb.ap-southeast-2.amazonaws.com to an IP that is latency‑optimized for the caller’s AZ. If that specific front‑end experiences a transient network glitch, only instances in that AZ see failures.

Pro Tip: Never assume a region is a monolithic, single‑point‑of‑failure service. Architect for failure within any individual AZ.

The Incident (A Real‑World Story)

“2:47 AM. PagerDuty screaming. Our primary auth service in Sydney (ap‑southeast‑2) was throwing ProvisionedThroughputExceededException and connection timeouts to DynamoDB. CloudWatch metrics for prod‑users‑table were flat—no capacity exhaustion. Half of our login attempts were failing.”

After an hour of debugging we discovered:

• Only instances in AZ ap‑southeast‑2a were failing.
• Instances in 2b and 2c were healthy.

This is the classic signature of an AWS “grey failure”: a localized, often network‑related hiccup that doesn’t turn the AWS Status page red.

Three Playbooks – From Quick Fix to Long‑Term Remedy

Play #2 – Example: Aggressive SDK Configuration (Python/Boto3)

# Example in Python using Boto3
from botocore.config import Config
from boto3 import resource

# Configure a more aggressive timeout and retry strategy
#   • Connect timeout: 1 s
#   • Read timeout:    1 s
#   • Retries: 5 attempts with backoff
config = Config(
    connect_timeout=1,
    read_timeout=1,
    retries={'max_attempts': 5}
)

# Pass this config when creating your client or resource
dynamodb = resource('dynamodb',
                    region_name='ap-southeast-2',
                    config=config)

table = dynamodb.Table('prod-users-table')
# All calls using `table` now inherit the new timeouts.

This change can turn a 30‑second user‑visible outage into a fast‑fail‑and‑retry scenario that most users never notice.

Play #3 – Architecting the Problem Out of Existence

DynamoDB Gateway VPC Endpoint

• Private, direct connection between your VPC and DynamoDB.
• Traffic stays on the AWS private network—never touches the public internet.
• Improves reliability, reduces latency, and adds a security boundary (no need for NAT/IGW egress).

Implementation steps (high‑level):

1. Open the VPC console → Endpoints → Create Endpoint.
2. Choose Service category: AWS services and select com.amazonaws.ap-southeast-2.dynamodb.
3. Attach the endpoint to the relevant subnet(s) and route tables.
4. (Optional) Add a policy to restrict which DynamoDB tables can be accessed.
5. Update your application’s SDK configuration to use the VPC endpoint (usually automatic once DNS resolves to the endpoint).

Bottom Line

• Grey failures in a single AZ can masquerade as capacity problems.
• Quick fix: Restart the affected instance.
• Short‑term resilience: Tune SDK time‑outs and retries.
• Long‑term robustness: Deploy a DynamoDB Gateway VPC Endpoint.

By layering these approaches, you can keep your authentication service (or any DynamoDB‑backed workload) humming even when a single AZ hiccups. 🚀

VPC Endpoint for DynamoDB

Creating a VPC endpoint bypasses public DNS resolution and the unpredictable network paths that cause “grey‑failures.” Your traffic stays inside the VPC, making it both reliable and secure.

How to set it up

1. Create a Gateway Endpoint in your VPC.
2. Associate the endpoint with the route tables of the subnets that host your application instances.
3. Update Security Groups to allow traffic to the DynamoDB service via the endpoint’s prefix list.

It’s a bit more work, but it virtually eliminates this class of problems while keeping database traffic off the Internet.

Solution Options

TL;DR

When you encounter a weird, region‑specific DynamoDB error:

1. Don’t immediately blame your code or capacity planning.
2. Check which AZs are failing.
3. Consider a VPC endpoint if the issue is recurring or impacts production reliability.

Remember: the cloud is just someone else’s computer, and sometimes the network cable between those computers gets a little loose.

---

👉 Read the original article on TechResolve.blog

☕ Support my work
If this article helped you, you can buy me a coffee:
👉