Linux Health Sentinel Phase 2: From Metrics to Meanings with Grafana Loki

Source: Dev.to

The Concept: Metrics vs. Logs

Metrics tell you there is a problem; logs tell you what the problem is.

The Architecture

• Loki (The Library) – runs on your laptop, stores logs and provides search capabilities.
• Promtail (The Spy) – runs on the Vagrant VM, tails log files and ships them to Loki.

Prerequisites

• Ubuntu laptop with Grafana + Prometheus running.
• Vagrant VM from Phase 1 (or any local VM).
• Basic networking between host and VM.

Step 1: Setting up the Library (Loki)

# Download and unzip Loki
wget https://github.com/grafana/loki/releases/latest/download/loki-linux-amd64.zip
sudo apt update && sudo apt install unzip -y
unzip loki-linux-amd64.zip
chmod +x loki-linux-amd64

# Download the default config file
wget https://raw.githubusercontent.com/grafana/loki/main/cmd/loki/loki-local-config.yaml

Run Loki:

./loki-linux-amd64 -config.file=loki-local-config.yaml

Note: This setup is for local learning only and runs without authentication. Do not expose Loki directly to the internet.

Step 2: Deploying the Spy (Promtail)

Install Promtail

curl -O -L "https://github.com/grafana/loki/releases/download/v3.5.9/promtail-linux-amd64.zip"
unzip promtail-linux-amd64.zip
chmod a+x promtail-linux-amd64

Configure Promtail

# Download the basic config file
wget https://raw.githubusercontent.com/grafana/loki/main/clients/cmd/promtail/promtail-local-config.yaml

Edit promtail-local-config.yaml and replace the client URL with your laptop’s IP address (use hostname -I to find it):

clients:
  - url: http://:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          host: vagrant-vm
          __path__: /var/log/*log

Run Promtail:

./promtail-linux-amd64 -config.file=promtail-local-config.yaml

Step 3: Visualisation in Grafana

1. Open Grafana (http://localhost:3000).
2. Add Data Source → select Loki.
   • Set URL to http://localhost:3100.
   • Click Save & Test.
3. Go to the Explore tab (compass icon).
4. Use the Label Browser to select job="varlogs" or host="vagrant-vm".
5. Click Run Query.

Enabling Live Log Streaming

• Click the Live button in the top‑right of the Grafana UI.
• Adjust Auto‑Refresh to 5 s or 10 s for near‑real‑time updates.

Generating Test Logs

sudo logger "Sentinel Test: Can you hear me, Grafana?"
sudo logger "Hello Loki, this is a test"
sudo logger "Sentinel Alert: Testing log flow to Grafana"
sudo logger -p user.err "Simulating a critical system error"
sudo logger "Hello Loki, this is test-2."

Conclusion

Next up: teaching the sentinel to notify us via Slack, Discord, or email when it detects trouble, using Alerting.