Prometheus + Node Exporter on Two EC2 Instances

Source: Dev.to

1️⃣ Architecture Overview (What we are building)

EC2 #1 — TARGET (Ubuntu)

• Purpose: expose system metrics
• Tool: Node Exporter
• Port: 9100

EC2 #2 — MONITOR (Ubuntu)

• Purpose: collect and display metrics
• Tool: Prometheus
• Port: 9090

Browser
   ↓
Prometheus (Ubuntu, :9090)
   ↓ scrape
Node Exporter (Ubuntu, :9100)

2️⃣ AWS SECURITY GROUP SETUP (LAB MODE)

⚠️ This is NOT secure for production – use only for training & demos.

2.1 Create Security Group (same steps for both EC2s)

AWS Console → EC2 → Security Groups → Create security group

Outbound Rules – keep default: All traffic → 0.0.0.0/0

Attach this SG to:

• Monitor EC2
• Target EC2

3️⃣ TARGET EC2 (Ubuntu) – Install Node Exporter

3.1 Connect to TARGET EC2

ssh ubuntu@<target-ip>

3.2 Download Node Exporter

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz

3.3 Extract & install

tar -xvf node_exporter-1.7.0.linux-amd64.tar.gz
cd node_exporter-1.7.0.linux-amd64
sudo mv node_exporter /usr/local/bin/

3.4 Start Node Exporter (foreground demo)

node_exporter

You should see:

Listening on :9100

3.5 Verify Node Exporter

ss -tulnp | grep 9100

Test metrics:

curl http://localhost:9100/metrics | head

✅ Node Exporter is ready

4️⃣ MONITOR EC2 (Ubuntu) – Install Prometheus

4.1 Connect to MONITOR EC2

ssh ubuntu@<monitor-ip>

4.2 Download Prometheus

cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.48.1/prometheus-2.48.1.linux-amd64.tar.gz

4.3 Extract files

tar -xvf prometheus-2.48.1.linux-amd64.tar.gz
cd prometheus-2.48.1.linux-amd64

4.4 Create directories

sudo mkdir -p /etc/prometheus
sudo mkdir -p /var/lib/prometheus

4.5 Install binaries

sudo mv prometheus promtool /usr/local/bin/
prometheus --version

4.6 Move config files

sudo mv prometheus.yml /etc/prometheus/
sudo mv consoles console_libraries /etc/prometheus/

Verify:

ls /etc/prometheus

Expected output:

prometheus.yml
consoles
console_libraries

5️⃣ Configure Prometheus (Ubuntu)

5.1 Edit config

sudo nano /etc/prometheus/prometheus.yml

5.2 Replace the entire file with the following

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: []

rule_files: []

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: [":9100"]

Save (CTRL+O, Enter, CTRL+X).

5.3 Validate config (VERY IMPORTANT)

promtool check config /etc/prometheus/prometheus.yml

Expected output:

SUCCESS

6️⃣ Start Prometheus (Ubuntu)

prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus

Look for:

Server is ready to receive web requests.

7️⃣ Access Prometheus UI

Open a browser and navigate to:

http://<monitor-ip>:9090

Then go to Status → Targets.

✅ Expected result

prometheus   UP
node         UP

This confirms:

• Networking works
• Security group works
• Metrics are being scraped

8️⃣ Live Demonstration Queries (Ubuntu Lab)

Go to the Graph tab.

8.1 Check targets

up

8.2 CPU usage (%)

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

8.3 Memory usage (%)

(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) 
/ node_memory_MemTotal_bytes * 100

8.4 Disk usage (%)

100 * (1 - (node_filesystem_avail_bytes{mountpoint="/"} 
/ node_filesystem_size_bytes{mountpoint="/"}))

Node Exporter now exposes system metrics that Prometheus can scrape and visualise.

📦 Overview

• Node Exporter runs on the TARGET EC2 and exposes metrics on port 9100.
• Prometheus scrapes those metrics at regular intervals.
• If the targets are UP, monitoring is working.
• Security Groups control network access – they are not a Linux‑level firewall.

10️⃣ What We Deliberately Allowed (Lab Mode)

📊 Grafana Placement & Setup (Ubuntu, AWS EC2)

🔹 Where does Grafana go?

Grafana is installed on the MONITOR EC2, together with Prometheus.

Final architecture (very important)

TARGET EC2 (Ubuntu)
└── Node Exporter
    └── :9100 (/metrics)

MONITOR EC2 (Ubuntu)
├── Prometheus
│   └── :9090 (scrapes node exporter)
└── Grafana
    └── :3000 (visualizes Prometheus data)

Why Grafana goes on the MONITOR EC2

• Grafana does NOT collect metrics – it only visualises them.
• Prometheus is the data source.
• Placing Grafana next to Prometheus gives:
  • Simpler networking
  • A real‑production pattern
  • Easier teaching

🧩 STEP‑BY‑STEP: Install Grafana on Ubuntu (MONITOR EC2)

🧠 Common Issues & Fixes

Happy monitoring! 🎉