Solved: Real-time Server Health Monitoring: Grafana to Telegram Integration

Source: Dev.to

Why This Matters

As a DevOps Engineer you know the drill: constant vigilance, endless email alerts, complex dashboards, and the occasional “user‑reported” incident. This reactive approach:

• Increases downtime and damages reputation.
• Generates alert fatigue.
• Wastes budget on SaaS features you may never use.

Imagine receiving critical server‑health alerts instantly on your phone via a familiar messaging app. This tutorial walks you through building that exact workflow.

Prerequisites

1. Grafana – operational instance (v7.0 or newer) with dashboards & data sources configured (e.g., Prometheus, Node Exporter).
2. Telegram – account with the desktop or mobile app installed.
3. Telegram Bot – ability to create/manage bots via BotFather.
4. CLI access – a server (or local machine) where you can run curl.
5. Basic server‑admin knowledge and familiarity with Grafana alert rule creation.

Step 1 – Create a Dedicated Telegram Bot

curl -s "https://api.telegram.org/botYOUR_BOT_TOKEN/getUpdates" \
  | grep -oP '"chat":{"id":\K[^,]*'

The output will be one or more numeric IDs:

• Private chat → your personal chat ID (positive number).
• Group chat → group chat ID (usually negative, e.g., -123456789).

Copy the appropriate ID for the next step.

Step 2 – Configure Grafana’s Telegram Notification Channel

1. Open Notification Channels

   • In Grafana, click the Alerting (bell) icon → Notification channels.

2. Add a New Channel

   • Click Add channel.

3. Fill in the Settings

4. Test the Channel

   • Click Test at the bottom. You should receive a “Test notification” in the Telegram chat.
   • If you don’t, verify:
     • Bot Token & Chat ID are correct.
     • Grafana server can reach api.telegram.org (outbound internet access).

5. Save the channel.

Step 3 – Create Alert Rules that Use the Telegram Channel

1. Navigate to a Dashboard containing the metric you want to monitor (e.g., CPU usage, memory, disk I/O).
2. Open the Panel’s Alert Tab (or create a new panel).
3. Define the Alert Condition (e.g., “when node_cpu_seconds_total > 80 % for 5 minutes”).
4. Add Notification → select the Telegram Alerts channel you just created.
5. Save the Dashboard – the alert rule is now active and will push notifications to Telegram when the condition fires.

Advanced: Custom Telegram Message Templates (Go Templating)

Grafana lets you craft rich messages using Go’s templating syntax. Example template:

{{ define "telegram.default.message" }}
🚨 *{{ .Title }}* 🚨
{{ if .State }}*State:* {{ .State }}{{ end }}
{{ if .Message }}*Message:* {{ .Message }}{{ end }}

*Labels:* {{ range $key, $value := .Labels }}{{ $key }}={{ $value }} {{ end }}
*Annotations:* {{ range $key, $value := .Annotations }}{{ $key }}={{ $value }} {{ end }}

{{ if .EvalMatches }}
*Metrics:* {{ range .EvalMatches }}
- {{ .Metric }} = {{ .Value }}{{ end }}
{{ end }}

[View Dashboard]({{ .RuleUrl }})
{{ end }}

• Place this in the Message Template field of the Telegram channel.
• Use *bold*, _italic_, or `code` for Markdown formatting (Telegram supports a subset).

Troubleshooting Checklist

Recap

By following these steps you now have:

• A Telegram bot that can receive messages.
• A Grafana notification channel wired to that bot.
• Alert rules that push real‑time server‑health notifications straight to your mobile device.

Enjoy faster incident response, reduced alert fatigue, and a leaner monitoring stack—no costly SaaS subscriptions required. 🎉

Setting Up Grafana Alerts for Server Health (Telegram Integration)

1. Edit a Panel

1. Hover over the panel you want to alert on (e.g., a CPU usage graph).
2. Click the Edit icon (pencil).

2. Access the Alert Tab

In the panel editor, click the Alert tab (bell icon).

3. Create an Alert Rule

1. Click Create Alert.
2. Fill in the required fields:

4. Define Query and Conditions

Query

Select your data source and write the query that fetches the metric.

Example (Prometheus):

avg(node_cpu_seconds_total{mode="idle",instance="server-x:9100"}) by (instance)

You would then transform this to trigger when 100 - (AVG_IDLE_CPU_PERCENTAGE) > 80.

Conditions

Set the alert threshold.

WHEN last() OF A IS ABOVE 80

5. Configure Notification

1. In the Send to section, select your Telegram Alerts channel from the dropdown.
2. (Optional) Add a custom message for this alert – it will override the channel’s default template.

6. Save the Alert

Click Save at the top‑right of the panel editor.

7. Repeat for Other Metrics

Create similar alerts for:

• Memory usage
• Disk space
• Network errors

Tailor thresholds and messages to your operational needs.

Using Go Templating for Rich Telegram Messages

Grafana’s alerting engine supports Go templating, allowing you to craft dynamic, context‑rich notifications.

Example Message Template

Add the following to the Message Template field of your Telegram notification channel (or individual alert rules):

{{ range .Alerts }}
🔥 ALERTING 🔥
{{ .Labels.alertname }} on {{ .Labels.instance }} is FIRING!
Severity: {{ .Labels.severity | default "critical" }}
Summary: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Value: {{ .ValueString }}
Dashboard: {{ .DashboardURL }}
Panel: {{ .PanelURL }}
{{ end }}

Template Breakdown

• {{ .Alerts }} – slice of all alerts firing for the notification.
• {{ .Labels }} – access labels from Prometheus metrics or Grafana alerts (e.g., alertname, instance, severity).
• {{ .Annotations }} – custom key‑value pairs defined in the alert rule (e.g., summary, description).
• {{ .ValueString }} – the actual value that triggered the alert.
• {{ .DashboardURL }} / {{ .PanelURL }} – direct links back to Grafana for quick investigation.

Tip: Define annotations in the alert rule, e.g.:

annotations:
  summary: "CPU utilization is too high"
  description: "Investigate processes consuming excessive CPU. Consider scaling up or optimizing application."

Common Issues & Troubleshooting

Next Steps

• Alert Grouping & Silencing: Reduce noise by grouping related alerts or silencing during maintenance windows.
• On‑Call Integration: Connect Telegram alerts to Grafana OnCall, PagerDuty, or other escalation tools for 24/7 coverage.
• Advanced Templates: Enrich messages with additional context (e.g., runbooks, links to ticketing systems).

You’ve now integrated Grafana with Telegram, giving your team instant, actionable server‑health notifications without relying on email or costly SaaS solutions.