S3 Vectors: 90% Cheaper Than Pinecone? Our Migration Guide

Source: Dev.to

Last week, I got a Slack message from our Finance Team that made my stomach drop

“Why is our Pinecone bill $4,200 this month?”

We’re running a mid‑sized RAG application with about 50 million vectors, and our database costs had quietly become our second‑largest AWS expense.

Then AWS announced S3 Vectors in December, promising “store and query vectors at up to 90 % lower cost than specialized databases.” I was skeptical—vector databases are fast, purpose‑built, and reliable. Could object storage really compete?

We spent two weeks migrating one of our production indexes from Pinecone to S3 Vectors. Below is what we learned, what worked, and when you should (and shouldn’t) make the switch.

The Vector Database Pricing Problem

Specialized vector databases (Pinecone, Weaviate, Qdrant) are engineering marvels. They deliver sub‑10 ms query latency and can handle billions of vectors, but that performance comes at a cost.

Monthly Cost Comparison (50 M vectors, 768 dimensions)

For our workload—storing product embeddings for semantic search with ~50 k queries per day—Pinecone cost us roughly $420/month. After migration, S3 Vectors landed at $42/month, a 90 % reduction, exactly as advertised.

Reality check: This isn’t an apples‑to‑apples comparison. Pinecone delivers consistent single‑digit‑millisecond latencies. S3 Vectors gives you sub‑second for infrequent queries and ~100 ms for frequent ones. The question isn’t “which is better” but “which matches your needs.”

Understanding S3 Vectors Architecture

S3 Vectors introduces a new bucket type specifically designed for vector data. Think of it as S3’s answer to the vector‑database market, but with a fundamentally different architectural approach.

Key Concepts

• Vector Buckets – Optimized bucket type with dedicated APIs for vector operations.
• Vector Indexes – Organize vectors within buckets; each index can hold up to 2 billion vectors.
• Strong Consistency – Immediately access newly written data—no eventual‑consistency delays.
• Integrated Metadata – Store up to 50 metadata keys per vector for powerful filtering.

What Makes It Different

Traditional vector databases keep everything in memory or on fast SSDs, pre‑computing indexes and scaling horizontally—like keeping an entire library on your desk. You get instant access, but you pay for the desk space.

S3 Vectors flips the model. Built on S3’s object‑storage foundation, vectors live on cheap disk‑based storage. AWS adds clever caching and optimizations to deliver reasonable query performance without the memory overhead—more like a well‑organized warehouse: retrieval takes a bit longer, but storage is cheap.

The Migration Process: Step‑by‑Step

We migrated our product‑search index (52 M vectors, 768 dimensions from OpenAI’s text-embedding-3-large) from Pinecone to S3 Vectors. Below is the exact process we followed.

Step 1 – Create Your S3 Vector Bucket

# Create a vector bucket
aws s3api create-vector-bucket \
    --bucket my-vectors \
    --region us-east-1

# Create a vector index
aws s3api create-vector-index \
    --bucket my-vectors \
    --index-name product-embeddings \
    --dimensions 768 \
    --distance-metric cosine

We chose cosine similarity because it matches what we used in Pinecone. Adjust the --distance-metric flag if you need Euclidean or dot‑product.

Step 2 – Export Data from Pinecone

Pinecone doesn’t have a built‑in export feature, so you need to fetch all vectors yourself:

import pinecone
import json

# Initialize Pinecone
pinecone.init(api_key="YOUR_API_KEY")
index = pinecone.Index("product-embeddings")

# Fetch all vectors (paginated)
vectors = []
for ids in fetch_all_ids():          # Implement your pagination logic
    batch = index.fetch(ids=ids)
    vectors.extend(batch["vectors"].values())

# Save to file for backup
with open("vectors_backup.json", "w") as f:
    json.dump(vectors, f)

Pro tip: Exporting 52 M vectors took ~3 hours for us. Run it off‑hours and add retry logic—network hiccups happen.

Step 3 – Transform & Upload to S3 Vectors

S3 Vectors expects a slightly different payload format:

import boto3

s3_client = boto3.client("s3")

def upload_batch(vectors_batch):
    # S3 Vectors expects a list of dicts with id, values, and optional metadata
    formatted = [
        {
            "id": v["id"],
            "values": v["values"],
            "metadata": v.get("metadata", {})
        }
        for v in vectors_batch
    ]

    response = s3_client.insert_vectors(
        Bucket="my-vectors",
        IndexName="product-embeddings",
        Vectors=formatted
    )
    return response

BATCH_SIZE = 1_000
for i in range(0, len(vectors), BATCH_SIZE):
    batch = vectors[i:i + BATCH_SIZE]
    upload_batch(batch)
    print(f"Uploaded {i + BATCH_SIZE}/{len(vectors)} vectors")

Performance: We sustained ~1 000 vectors/second, so the full upload took roughly 14 hours. Run it as a background job.

Step 4 – Update Your Application Code

The API differences are minimal. Below is a before/after comparison for a typical query.

# BEFORE: Pinecone query
results = index.query(
    vector=query_embedding,
    top_k=10,
    include_metadata=True,
    namespace="products"
)

# AFTER: S3 Vectors query
s3_client = boto3.client("s3")
response = s3_client.query_vectors(
    Bucket="my-vectors",
    IndexName="product-embeddings",
    QueryVector=query_embedding,
    TopK=10,
    IncludeMetadata=True,
    FilterExpression="namespace = 'products'"
)
results = response["Matches"]

Only the client library and parameter names change; the surrounding logic stays the same.

When to Switch (and When Not To)

TL;DR

• Cost: S3 Vectors can slash vector‑storage spend by ~90 % (e.g., $420 → $42).
• Performance: Expect sub‑second latency for typical workloads; sub‑100 ms for frequent queries.
• Migration effort: Roughly 1 day of export + 1 day of upload for 50 M vectors (parallelizable).
• Fit: Ideal for large, relatively static embeddings with modest query rates; not a drop‑in replacement for ultra‑low‑latency, high‑throughput use cases.

If your RAG app’s query volume is modest and you’re looking to tame your vector‑database bill, give S3 Vectors a try. For latency‑critical, high‑throughput workloads, stick with a purpose‑built vector database. Happy vectoring!

Migration from Pinecone to Amazon S3 Vectors

1️⃣ Before & After: Querying Pinecone vs. S3 Vectors

Pinecone (Python SDK)

# BEFORE: Pinecone query
response = pinecone_index.query(
    vector=query_embedding,
    top_k=10,
    filter={"category": "electronics"}
)

# Parse results
results = [{
    "id": match.id,
    "score": match.score,
    "metadata": match.metadata
} for match in response.matches]

S3 Vectors (Boto3)

# AFTER: S3 Vectors query
response = s3_client.query_vectors(
    Bucket='my-vectors',
    IndexName='product-embeddings',
    QueryVector=query_embedding,
    MaxResults=10,
    MetadataFilters={
        'category': {'StringEquals': 'electronics'}
    }
)

# Parse results (format is slightly different)
results = [{
    "id": match['Id'],
    "score": match['Score'],
    "metadata": match['Metadata']
} for match in response['Matches']]

2️⃣ Step 5: Test and Validate

We ran both systems in parallel for a week, comparing results:

3️⃣ Performance Benchmarks: The Real Numbers

Query Latency Comparison

The latency increase is noticeable but acceptable for catalog‑search use cases where sub‑100 ms response times feel instantaneous to users.

When Latency Matters

• Real‑time recommendation engines
• Chatbots with instant responses
• High‑frequency trading systems

For a chatbot that performs 10 vector queries per message, the extra ~100 ms per query adds up to roughly 1 second of perceived delay—enough to feel sluggish.

4️⃣ Cost Breakdown: Where the Savings Come From

The biggest cost driver is storage: Pinecone keeps vectors in memory/fast SSDs, while S3 Vectors leverages cheap disk‑based storage with intelligent caching. For infrequently accessed data, the cost advantage is massive.

5️⃣ When to Use S3 Vectors vs. Dedicated Vector Databases

Decision Matrix

Choose S3 Vectors When

• Query frequency is low‑to‑moderate (≤ 100 QPS sustained)
• Budget is a primary constraint and you store millions of vectors
• 100‑200 ms latency is acceptable for your application
• You’re already heavily invested in AWS and want native integration
• Data durability is critical (S3’s 11‑nine durability)

Stick with Dedicated Vector DBs When

• You need consistent single‑digit‑millisecond latency
• High query throughput (≥ 1 000 QPS)
• Complex filtering & faceting are core features
• Building user‑facing features where speed directly impacts UX
• Advanced capabilities like hybrid search or custom distance metrics matter

6️⃣ Integration with AWS Services

Bedrock Knowledge Bases

# Create a Bedrock Knowledge Base with S3 Vectors
aws bedrock create-knowledge-base \
    --name "product-knowledge" \
    --role-arn "arn:aws:iam::account:role/bedrock-kb-role" \
    --knowledge-base-configuration '{
        "type": "VECTOR",
        "vectorKnowledgeBaseConfiguration": {
            "embeddingModelArn": "arn:aws:bedrock:...",
            "vectorStoreConfiguration": {
                "s3VectorConfiguration": {
                    "bucketName": "my-vectors",
                    "indexName": "product-embeddings"
                }
            }
        }
    }'

OpenSearch Integration

Create a tiered architecture: hot data lives in OpenSearch for low latency, while cold data resides in S3 Vectors for cost savings. AWS can automatically move data based on access patterns.

7️⃣ Gotchas and Limitations

8️⃣ Real‑World Migration Checklist

1. Measure current query patterns

   • Avg. QPS during peak hours
   • P95 / P99 latency requirements
   • Hot vs. cold data access

2. Calculate ROI

   • Current monthly vector‑DB cost
   • Estimated S3 Vectors cost (use AWS calculator)
   • Engineering effort (≈ 2‑3 weeks)

3. Run a proof of concept

   • Migrate a small, non‑critical index
   • Compare latency, accuracy, and cost

4. Plan data migration

   • Export, transform, and bulk‑load as shown in steps 1‑3

5. Update application code

   • Switch SDK calls (see before/after examples)

6. Monitor in production

   • Track latency, error rates, and cost savings

Following this checklist will help ensure a smooth transition from Pinecone to S3 Vectors with minimal disruption.