Never Commit Secrets Again: Generate .env Files from AWS Secrets Manager

Source: Dev.to

TL;DR

Store secrets in AWS Secrets Manager. Generate .env files on demand with a Python script. Never commit credentials again.

The Problem

Every team commits secrets eventually. GitHub detected over 12 million exposed credentials last year through their secret scanning.

The usual approaches all have failure modes:

• .gitignore fails when developers forget to add it, or clone fresh and ask for the file via Slack.
• SOPS encryption still puts files in git, adds key management overhead, and creates merge‑conflict nightmares.
• .env.example templates get stale and require manual copying.

We needed something better: secrets that live outside the repository entirely, with a frictionless developer experience.

The Solution

Secrets live in AWS Secrets Manager. Developers run one command to generate their .env file:

make env
# .env is generated locally, ready to use

The file is .gitignored. It never touches version control. When secrets change in AWS, developers regenerate and get the latest values.

Implementation

1. Organize Secrets in AWS

Structure your secrets by application and environment:

/myapp/dev/database   → {"DB_HOST": "...", "DB_PASSWORD": "..."}
/myapp/dev/api-keys   → {"STRIPE_KEY": "...", "SENDGRID_KEY": "..."}
/myapp/prod/database  → {"DB_HOST": "...", "DB_PASSWORD": "..."}
/myapp/prod/api-keys  → {"STRIPE_KEY": "...", "SENDGRID_KEY": "..."}

Create secrets using the AWS CLI:

aws secretsmanager create-secret \
  --name /myapp/dev/database \
  --secret-string '{"DB_HOST":"localhost","DB_PASSWORD":"devpass123"}'

2. The Python Script

generate_env.py generates .env files from Secrets Manager:

#!/usr/bin/env python3
"""
Generate .env file from AWS Secrets Manager.

Usage:
    python generate_env.py dev
    python generate_env.py prod --force
"""

import argparse
import json
import os
import sys
from pathlib import Path

import boto3
from botocore.exceptions import ClientError, NoCredentialsError

# Configuration
APP_NAME = "myapp"
AWS_REGION = os.environ.get("AWS_REGION", "us-east-1")
ENV_FILE = ".env"

SECRET_KEYS = ["database", "api-keys", "third-party"]

def get_secret(secret_name: str, region: str) -> dict:
    """Fetch a secret from AWS Secrets Manager."""
    client = boto3.client("secretsmanager", region_name=region)

    try:
        response = client.get_secret_value(SecretId=secret_name)
        return json.loads(response.get("SecretString", "{}"))
    except ClientError as e:
        if e.response["Error"]["Code"] == "ResourceNotFoundException":
            print(f"  Warning: Secret '{secret_name}' not found")
            return {}
        raise

def validate_aws_credentials() -> bool:
    """Check if AWS credentials are configured."""
    try:
        sts = boto3.client("sts")
        identity = sts.get_caller_identity()
        print(f"Authenticated as: {identity['Arn']}")
        return True
    except NoCredentialsError:
        print("Error: AWS credentials not found.")
        print("\nFix with one of:")
        print("  1. aws configure")
        print("  2. Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY")
        print("  3. Use IAM role (if on AWS)")
        return False

def fetch_all_secrets(environment: str, region: str) -> dict:
    """Fetch all secrets for the environment."""
    all_secrets = {}
    for key in SECRET_KEYS:
        secret_path = f"/{APP_NAME}/{environment}/{key}"
        print(f"  Fetching: {secret_path}")
        all_secrets.update(get_secret(secret_path, region))
    return all_secrets

def generate_env_content(secrets: dict) -> str:
    """Generate .env content from secrets."""
    lines = [
        "# Auto-generated from AWS Secrets Manager",
        "# DO NOT COMMIT THIS FILE",
        "",
    ]
    for key, value in sorted(secrets.items()):
        if isinstance(value, str) and " " in value:
            value = f'"{value}"'
        lines.append(f"{key}={value}")
    return "\n".join(lines) + "\n"

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("environment", choices=["dev", "staging", "prod"])
    parser.add_argument("-f", "--force", action="store_true")
    parser.add_argument("-o", "--output", default=ENV_FILE)
    args = parser.parse_args()

    print(f"Generating .env for '{args.environment}'\n")

    if not validate_aws_credentials():
        sys.exit(1)

    secrets = fetch_all_secrets(args.environment, AWS_REGION)

    if not secrets:
        print(f"\nError: No secrets found at /{APP_NAME}/{args.environment}/*")
        sys.exit(1)

    print(f"\nFound {len(secrets)} secret values")

    content = generate_env_content(secrets)
    path = Path(args.output)

    if path.exists() and not args.force:
        if input(f"{args.output} exists. Overwrite? [y/N]: ").lower() != "y":
            sys.exit(0)

    path.write_text(content)
    print(f"Generated: {args.output}")

if __name__ == "__main__":
    main()

3. Shell Wrapper and Makefile

Shell wrapper (generate-env.sh):

#!/bin/bash
# generate-env.sh
set -e
ENV=${1:-dev}

if ! python3 -c "import boto3" 2>/dev/null; then
    pip3 install boto3 --quiet
fi

python3 "$(dirname "$0")/generate_env.py" "$ENV" "${@:2}"

Makefile targets:

.PHONY: env env-dev env-prod env-dry

env:
	@./scripts/generate-env.sh dev

env-dev:
	@./scripts/generate-env.sh dev

env-prod:
	@./scripts/generate-env.sh prod

env-dry:
	@./scripts/generate-env.sh dev --dry-run

4. GitHub Actions with OIDC

No stored credentials are needed. Use OIDC to assume an AWS role:

name: Deploy

on:
  push:
    branches: [main]

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions
          aws-region: us-east-1

      - name: Generate .env
        run: |
          pip install boto3
          python scripts/generate_env.py prod --force

      - name: Deploy
        run: |
          # Your deployment commands
          echo "Deploying..."

      - name: Cleanup
        if: always()
        run: rm -f .env

5. GitLab CI

Same pattern with GitLab’s OIDC:

deploy:
  stage: deploy
  image: python:3.11-slim
  script:
    - pip install boto3
    - |
      export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"
      $(aws sts assume-role-with-web-identity
      --role-arn ${AWS_ROLE_ARN}
      --role-session-name "gitlab-${CI_PIPELINE_ID}"
      --web-identity-token ${CI_JOB_JWT_V2}
      --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
      --output text))
    - python scripts/generate_env.py prod --force
    - echo "Deploying..."
  after_script:
    - rm -f .env

6. IAM Permissions

Developers need a policy that allows reading the relevant secrets, e.g.:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": [
        "arn:aws:secretsmanager:*:*:secret:/myapp/*"
      ]
    }
  ]
}