AWS Secrets Manager: How to Set Up Secrets and Fetch Them in a Python Lambda

Source: Dev.to

Managing sensitive information such as database passwords, API keys, and tokens is a critical part of building secure cloud applications. Hardcoding secrets in source code or configuration files is a common anti‑pattern that leads to security vulnerabilities.

AWS Secrets Manager provides a secure, scalable, and auditable way to store and retrieve secrets dynamically at runtime. In this article we cover:

• What AWS Secrets Manager is
• How to create and store secrets
• IAM permissions required
• How to fetch secrets in a Python AWS Lambda
• Best practices

1. What Is AWS Secrets Manager?

AWS Secrets Manager is a managed service that helps you:

• Securely store secrets (credentials, API keys, tokens)
• Encrypt secrets using AWS KMS
• Control access via IAM
• Rotate secrets automatically (for supported services)
• Retrieve secrets programmatically at runtime

Typical use cases:

• Database credentials (RDS, Aurora)
• Third‑party API keys
• JWT signing secrets
• OAuth client secrets

2. Creating a Secret in AWS Secrets Manager

Step 1: Open AWS Secrets Manager

1. Log in to the AWS Console.
2. Navigate to Secrets Manager.
3. Click Store a new secret.

Step 2: Choose Secret Type

Select Other type of secret to store custom key/value pairs, e.g.:

DB_USERNAME = admin
DB_PASSWORD = StrongPassword@123
DB_HOST     = mydb.cluster-xyz.us-east-1.rds.amazonaws.com

Secrets Manager stores these values as encrypted JSON.

Step 3: Configure Encryption

• Choose the default AWS‑managed KMS key, or
• Select a customer‑managed KMS key for stricter compliance.

Step 4: Name the Secret

Give the secret a clear, environment‑aware name, for example:

• myapp/dev/database
• myapp/staging/database
• myapp/prod/database

This naming strategy avoids accidental cross‑environment access.

Step 5: Review and Create

Click Store. Your secret is now securely stored.

3. IAM Permissions for Lambda

Your Lambda function must have permission to read secrets.

IAM Policy Example

Attach the following policy to the Lambda execution role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:myapp/dev/database*"
    }
  ]
}

Important: Always scope the Resource to specific secrets instead of using "*".

4. Fetching Secrets in a Python Lambda

Step 1: Python Dependencies

AWS Lambda already includes boto3 and botocore. No additional libraries are required.

Step 2: Python Code to Fetch Secrets

import json
import boto3
from botocore.exceptions import ClientError

def get_secret(secret_name, region_name="us-east-1"):
    client = boto3.client(
        service_name="secretsmanager",
        region_name=region_name
    )

    try:
        response = client.get_secret_value(SecretId=secret_name)
    except ClientError as e:
        raise RuntimeError(f"Unable to retrieve secret: {e}")

    # Secrets are usually stored as JSON strings
    if "SecretString" in response:
        return json.loads(response["SecretString"])
    else:
        # Binary secrets (rare case)
        return response["SecretBinary"]

Step 3: Using Secrets in Lambda Handler

def lambda_handler(event, context):
    secret_name = "myapp/dev/database"

    secrets = get_secret(secret_name)

    db_user = secrets["DB_USERNAME"]
    db_password = secrets["DB_PASSWORD"]
    db_host = secrets["DB_HOST"]

    # Example usage
    print(f"Connecting to DB at {db_host} with user {db_user}")

    return {
        "statusCode": 200,
        "body": "Secrets fetched successfully"
    }

5. Performance Consideration (Important)

Each call to GetSecretValue is a network call. To reduce latency and API usage, cache secrets at module level:

_cached_secrets = None

def get_cached_secret(secret_name):
    global _cached_secrets
    if _cached_secrets is None:
        _cached_secrets = get_secret(secret_name)
    return _cached_secrets

6. Environment‑Based Secret Management

Use an environment variable to control which secret is loaded:

import os

secret_name = os.environ["SECRET_NAME"]   # e.g., myapp/dev/database
secrets = get_cached_secret(secret_name)

This enables:

• Same codebase across DEV / STAGE / PROD
• Environment‑specific secrets
• Safer deployments

7. Security and Best Practices

• Never hardcode secrets.
• Use least‑privilege IAM policies.
• Keep separate secrets per environment.
• Enable automatic rotation where supported.
• Cache secrets inside Lambda for performance.
• Log carefully—never log secret values.
• Prefer Secrets Manager over SSM Parameter Store for highly sensitive data.