WHAT IS A SECRET?

Source: Dev.to

1. What is a secret?

A secret is any piece of sensitive information that must be protected from unauthorized access.

Examples

• API keys
• Access tokens
• Database passwords
• Private keys (.pem)
• Confluent Cloud credentials
• Terraform backend credentials
• OAuth tokens
• SSH keys

Goal: Keep secrets encrypted at rest, encrypted in transit, and never stored in plaintext (e.g., in repos, logs, or artifacts).

---

2. Why secrets management is critical

A senior DevOps engineer must prevent:

• Credential leaks
• Unauthorized access
• Accidental commits to Git
• Hard‑coding in Terraform, Kubernetes, Docker, or CI/CD pipelines

Consequences of secret leaks

• Environment compromise
• Data breaches
• Unauthorized AWS usage costing thousands
• Repository takeovers

This is why secure secret stores are used instead of plain files.

---

3. Secret storage options DevOps must know

---

4. GitHub Secrets — Used for CI/CD only

Where used

• GitHub Actions CI/CD pipelines

What it stores

• AWS access key + secret key
• Docker registry token
• Terraform Cloud token
• Confluent Cloud credentials
• Any deployment API keys

How it works

• GitHub encrypts the secret with libsodium.
• Only GitHub Actions that run in your repository can access it.
• Not available to forked PRs.

Security rules for senior DevOps

• Never store database passwords here for applications.
• Never store long‑lived AWS keys (prefer OIDC).
• Rotate keys every 90 days.
• Give repositories minimum access.
• Avoid storing complex JSON — use AWS Parameter Store instead.

What GitHub Secrets do not replace

• AWS Secrets Manager
• HashiCorp Vault
• Kubernetes Secrets
• Application runtime secrets

GitHub Secrets are only for CI/CD.

---

5. AWS Secrets Manager — Production‑grade secret storage

Where used

• Production microservices on AWS.

Features

• Automatic rotation (via Lambda)
• Version history
• Multi‑account access with IAM
• Replication across regions
• Built‑in KMS encryption

Typical use cases

• RDS master password
• Confluent API secret
• Stripe keys
• OAuth tokens
• DB credentials for ECS tasks

Access via IAM

ecsTaskExecutionRole:
  can access secret: arn:aws:secretsmanager:...

Code example (ECS task environment)

{
  "name": "DB_PASSWORD",
  "valueFrom": "arn:aws:secretsmanager:us-east-2:xxx:secret:db_pass"
}

When to choose Secrets Manager

• You need rotation.
• You need strict auditing.
• You manage cross‑account applications.

---

6. AWS SSM Parameter Store

Overview

• Stores SecureString parameters.

Cost

• $0 for the Standard tier (Secrets Manager costs ~$0.40 per secret per month).

Ideal for

• Dev / QA / non‑critical secrets.

Use cases

• Microservice configuration values
• Non‑rotating tokens
• S3 bucket names
• Feature flags

Not recommended for

• Production database passwords (no rotation support).

---

7. HashiCorp Vault — The most advanced system

Why Vault is used

• Supports AWS, GCP, Azure, Kubernetes, on‑prem.
• Dynamic secrets (temporary DB credentials).
• Encryption‑as‑a‑service (Transit).
• PKI certificate generation.
• Fine‑grained access policies.
• Audit logs.
• Can run on‑prem or as HCP Vault Cloud.

Dynamic secrets example

Vault can generate a PostgreSQL username/password that is:

• Valid for 1 hour
• Automatically deleted afterward

Perfect for short‑lived CI/CD tasks and high‑security environments (banks, healthcare, fintech).

Notable users

• Uber
• Stripe
• Goldman Sachs
• Netflix

---

8. Kubernetes Secrets (optional but DevOps must know)

Storage

• Stored inside etcd (encrypted with KMS in production).

Used for

• API keys
• DB passwords
• TLS certificates

Mounted as

• Environment variables
• Files

---

9. Terraform & Secrets — Senior level knowledge

Where not to store secrets

• In Git repositories
• In .tf files or modules
• In Terraform state

How secrets must be passed

• terraform.tfvars (locally only)
• CI/CD environment variables
• AWS SSM Parameter Store
• AWS Secrets Manager

Example bad code (do not do)

password = "MySecret123"

Better practice

password = var.db_password

Recommended best practice

password = data.aws_secretsmanager_secret_version.db_password.secret_string

---

10. How secrets flow in a real CI/CD pipeline

1. Store secrets in AWS Secrets Manager.
2. EC2/ECS/Lambda uses an IAM role to access those secrets.
3. GitHub Actions stores only:
   • AWS Access Key
   • AWS Secret Key
   • Confluent API key
4. Terraform deploys infrastructure, referencing secrets by ARN.
5. Applications retrieve secrets via the AWS SDK using IAM role permissions.

---

11. What NOT TO DO (Senior DevOps knowledge)

• ❌ Never store secrets in a GitHub repository.
• ❌ Never store secrets in Slack or Teams.
• ❌ Never store secrets in a Docker image.
• ❌ Never store secrets in YAML files.
• ❌ Never store secrets in Terraform state.
• ❌ Never store secrets in code comments.
• ❌ Never echo secrets in CI logs.
• ❌ Never send secrets via email.

If a secret is leaked, rotate it immediately.

---

12. Interview‑level explanation (you can say this)

“In my pipelines, GitHub Secrets are used only for CI/CD credentials.
For application runtime secrets, I use AWS Secrets Manager or SSM Parameter Store depending on rotation requirements.
I avoid hard‑coding secrets in Terraform by pulling them from the secret stores at runtime.
For enterprise multi‑cloud environments, I integrate HashiCorp Vault with AWS IAM and Kubernetes service accounts for secure authentication and dynamic secrets.
All secrets are KMS‑encrypted and never exposed in logs.”

---

Secrets flow — High‑level diagram

                        ┌──────────────────────────┐
                        │   Developer Machine      │
                        │   (Push Git Changes)     │
                        └─────────────┬────────────┘
                                      │
                                      ▼
                         ┌────────────────────────┐
                         │   GitHub Repository    │
                         └─────────────┬──────────┘
                                      │
                                      ▼
                         ┌──────────────────────────────┐
                         │   GitHub Actions Runner        │
                         │   (CI/CD Workflow Execution)  │
                         └──────────────┬────────────────┘
        SECRETS ENTER HERE FROM GITHUB →│
                                      ▼
   ┌───────────────────────────────────────────────────────────────────────────────┐
   │                     GitHub Secrets Storage                                   │
   │   - AWS_ACCESS_KEY_ID                                                         │
   │   - AWS_SECRET_ACCESS_KEY                                                     │
   └───────────────────────────────────────────────────────────────────────────────┘