Monthly Golden Image Build Process using Packer & Ansible

Source: Dev.to

Introduction

In IT operations, imagine we’re working in a cloud organization that deploys hundreds of EC2 instances every month. Each instance needs to be secure and compliant. Manually configuring each instance is a nightmare. Instead, you want a golden image – a reusable AMI that’s pre‑hardened and provisioned with all necessary tools.

This is the point at which Packer becomes relevant.

Problem Statement

In most organizations, EC2 instances are launched frequently to support various workloads. But each instance must be:

• Secure and compliant
• Equipped with monitoring and security agents
• Consistently configured

Real‑Time Scenario

Suppose you’re part of a security‑conscious enterprise. Every EC2 instance must:

• Follow CIS benchmarks
• Have CrowdStrike and Qualys agents installed

Instead of configuring each instance post‑launch, you want to create a golden AMI that’s already hardened and provisioned. This image will serve as the base for all future deployments – saving time and ensuring consistency.

Tools Involved

• Packer
• AWS EC2
• Ansible
• GitLab CI/CD
• Amazon SSM

Architecture Diagram

Workflow Overview

1. Launch a temporary EC2 instance from a base image.
2. Run provisioning scripts to:
   • Apply OS hardening (CIS benchmarks, firewall rules, SSH configs).
3. Create a new AMI from the configured instance.
4. Terminate the temporary instance.

Implementation Steps

Step 1 – Install Packer

Step 2 – Create Packer Template with Ansible Provisioning

The following Packer template automates the creation of a custom Amazon Machine Image (AMI) by launching a temporary EC2 instance in a specific AWS VPC and subnet, using a designated SSH key pair for secure access.

packer {
  required_plugins {
    amazon = {
      version = ">= 1.2.8"
      source  = "github.com/hashicorp/amazon"
    }
    ansible = {
      version = "~> 1"
      source  = "github.com/hashicorp/ansible"
    }
  }
}

variable "ami_prefix" {
  type    = string
  default = ""
}

variable "reference_image" {
  type    = string
  default = ""
}

locals {
  timestamp = regex_replace(timestamp(), "[- TZ:]", "")
}

variable "privatekey" {
  type    = string
  default = ""
}

source "amazon-ebs" "amazon_linux" {
  ami_name                     = "${var.ami_prefix}-${local.timestamp}"
  instance_type                = "t2.micro"
  region                       = "ap-south-1"
  vpc_id                       = "vpc-07b2ce11f9b189f3b"
  subnet_id                    = "subnet-063ebc661edd9fb37"
  security_group_id            = "sg-04e9ae673095b02e9"
  ssh_interface                = "private_ip"
  associate_public_ip_address  = true
  ssh_keypair_name             = "runner_key"
  ssh_private_key_file         = var.privatekey

  source_ami_filter {
    filters = {
      name                 = "${var.reference_image}"
      root-device-type     = "ebs"
      virtualization-type = "hvm"
    }
    most_recent = true
    owners      = [""]
  }

  ssh_username = "ec2-user"
}

build {
  name    = "learn-packer"
  sources = ["source.amazon-ebs.amazon_linux"]

  provisioner "shell" {
    inline = [
      "sleep 20",
      "echo '--- Running AMI pre‑check ---'",
      "set -e",
      "sudo mkdir -p /usr/lib",
      "# Ensure SFTP subsystem path exists",
      "if [ ! -f /usr/lib/sftp-server ]; then",
      "  if [ -f /usr/libexec/openssh/sftp-server ]; then",
      "    sudo ln -s /usr/libexec/openssh/sftp-server /usr/lib/sftp-server",
      "    echo 'Linked /usr/libexec/openssh/sftp-server -> /usr/lib/sftp-server';",
      "  else",
      "    echo 'Warning: sftp-server not found, installing openssh-server...';",
      "    sudo yum install -y openssh-server || sudo apt-get install -y openssh-server;",
      "  fi;",
      "fi",
      "# Basic network sanity check",
      "sudo yum clean all || true",
      "sudo yum update -y || true",
      "echo '--- Pre‑check complete ---'"
    ]
  }

  provisioner "shell" {
    inline = [
      "sudo mkdir -p /tmp/.ansible",
      "sudo chmod 777 /tmp/.ansible"
    ]
  }

  provisioner "ansible" {
    playbook_file   = "./playbook/main.yml"
    use_proxy       = false
    extra_arguments = [
      "--vault-password-file=/home/gitlab-runner/.vault_pass",
      "-e", "ansible_remote_tmp=/tmp/.ansible",
      "-e", "ansible_local_tmp=/tmp/.ansible",
      "-e", "ansible_scp_if_ssh=True",
      "-e", "ansible_python_interpreter=/usr/bin/python3",
      "-e", "ansible_ssh_transfer_method=scp"
    ]
  }
}

Ansible Main Playbook

---
- name: Create users and provide sudo access
  hosts: all
  become: true
  gather_facts: true
  vars_files:
    - ../vars/useradd.yml
    - ../vars/vault.yml
  roles:
    - ../roles/useradd
    - ../roles/sudo

- name: Set hostnames
  hosts: all
  become: true
  gather_facts: false
  vars_files:
    - ../vars/var.yml
  roles:
    - ../roles/hostnamectl

- name: Enable or set miscellaneous services
  hosts: all
  become: true
  gather_facts: false
  roles:
    - ../roles/ssh
    - ../roles/login_banner
    - ../roles/services
    - ../roles/timezone
    # - ../roles/fs_integrity
    # - ../roles/selinux
    # - ../roles/firewalld
    # - ../roles/log_management
    - ../roles/rsyslog
    # - ../roles/cron

./roles/journald

---
- hosts: all
  become: true
  gather_facts: true
  vars_files:
    - ../vars/useradd.yml
    - ../vars/vault.yml
  roles:
    - ../roles/useradd

Outcome

This template builds a custom AMI by:

• Launching a VM in a specific VPC and subnet.
• Using a defined SSH key pair for access.
• Running shell scripts and Ansible to configure the instance.
• Saving the final image with a unique name for future use.

Note: I’ve also configured a CI/CD variable in GitLab to securely store the private‑key content used for SSH access in the Packer build.

GitLab CI/CD Variable for Private Key

In GitLab, CI/CD variables allow us to store sensitive data (passwords, tokens, SSH keys) securely. I created a variable (e.g., PRIVATE_KEY) that contains the entire private‑key content (not just the path). This variable is injected into the pipeline at runtime, allowing tools like Packer to use it without hard‑coding the key or exposing it in the repository.

packer validate -var-file="ami.pkrvars.hcl" -var "privatekey=runner_key.pem" aws-linux.pkr.hcl

Step 3: GitLab Pipeline Stages

In this pipeline, a GitLab CI/CD job automates the AMI creation process by:

1. Securely injecting an SSH private key from a CI/CD variable.
2. Validating and building a Packer template.
3. Provisioning an EC2 instance in a specific VPC/subnet.
4. Saving the final image for future use.

default: 
  tags:
    - gitlab_runner

stages:
  - image_build

Image Build: 
  stage: image_build
  script: 
    - echo "$SSH_PRIVATE_KEY" > runner_key.pem
    - chmod 400 runner_key.pem
    - packer init .
    - echo "Validating packer template..."
    - packer validate -var-file="ami.pkrvars.hcl" -var "privatekey=runner_key.pem" aws-linux.pkr.hcl
    - echo "Building AMI..."
    - packer build -var-file="ami.pkrvars.hcl" -var "privatekey=runner_key.pem" aws-linux.pkr.hcl

Conclusion

In this article we tackled a common challenge in cloud operations—ensuring every EC2 instance is secure, compliant, and consistently configured without manual intervention.

By combining Packer, Ansible, and GitLab CI/CD, we built a fully automated pipeline that:

• Launches a temporary EC2 instance.
• Applies CIS hardening and installs security agents.
• Saves a golden AMI for future use.
• Secures credentials using GitLab CI/CD variables.

This approach not only boosts security and compliance but also saves hours of manual effort, reduces human error, and ensures every deployment starts from a trusted baseline.

Thanks,

Susseta Bose