7.Stream Kinesis Data to CloudWatch Using Terraform

Source: Dev.to

Lab Information

The monitoring team wants to improve observability into the streaming infrastructure. Implement a solution using Amazon Kinesis and CloudWatch so that an alert is triggered immediately if write throughput exceeds provisioned limits.

Tasks (using Terraform):

1. Create a Kinesis Data Stream

   • Name: xfusion-kinesis-stream
   • Shard count: 1

2. Enable Monitoring

   • Enable shard‑level metrics to track ingestion and throughput errors.

3. Create a CloudWatch Alarm

   • Name: xfusion-kinesis-alarm
   • Monitor the WriteProvisionedThroughputExceeded metric.
   • Trigger if the metric exceeds a threshold of 1.

4. Ensure Alerting

   • Configure the alarm to detect write throughput issues exceeding provisioned limits.

5. Terraform Files

   • main.tf – provision the Kinesis stream and CloudWatch alarm.
   • outputs.tf – output the stream and alarm names.

Lab Solutions

main.tf

# Kinesis Data Stream
resource "aws_kinesis_stream" "xfusion_kinesis" {
  name        = "xfusion-kinesis-stream"
  shard_count = 1

  shard_level_metrics = [
    "IncomingBytes",
    "IncomingRecords",
    "WriteProvisionedThroughputExceeded"
  ]
}

# CloudWatch Alarm for write throughput exceeded
resource "aws_cloudwatch_metric_alarm" "xfusion_kinesis_alarm" {
  alarm_name          = "xfusion-kinesis-alarm"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "WriteProvisionedThroughputExceeded"
  namespace           = "AWS/Kinesis"
  period              = 60
  statistic           = "Sum"
  threshold           = 1

  dimensions = {
    StreamName = aws_kinesis_stream.xfusion_kinesis.name
  }

  alarm_description = "Alarm when Kinesis write throughput exceeds provisioned limits"
}

outputs.tf

output "kke_kinesis_stream_name" {
  value = aws_kinesis_stream.xfusion_kinesis.name
}

output "kke_kinesis_alarm_name" {
  value = aws_cloudwatch_metric_alarm.xfusion_kinesis_alarm.alarm_name
}

Terraform Commands (required)

terraform init
terraform validate
terraform apply
# type "yes" when prompted

Step‑by‑Step Explanation (Why & What Happens)

What is Amazon Kinesis?

A data pipe that receives streaming data such as:

• Application logs
• Clickstream data
• IoT events

What is a Shard?

A lane on a highway. Each shard has a write limit; exceeding it causes errors. In this lab, shard_count = 1 creates a single lane.

Why Enable Shard‑Level Metrics?

shard_level_metrics = [
  "IncomingBytes",
  "IncomingRecords",
  "WriteProvisionedThroughputExceeded"
]

Enables detailed visibility inside the shard; without it, CloudWatch cannot see shard‑level problems, making the alarm ineffective.

WriteProvisionedThroughputExceeded Metric

Indicates that producers are sending data faster than the shard can accept, leading to rejected writes, potential data loss, and retries.

Purpose of the CloudWatch Alarm

Transforms metric monitoring into actionable alerts. The alarm configuration:

• comparison_operator = "GreaterThanThreshold"
• threshold = 1
• evaluation_periods = 1
• period = 60 (seconds)
• statistic = "Sum"

Means: every minute, sum the number of write‑throughput errors; if the sum exceeds 1, trigger the alarm immediately.

Why Use Dimensions?

dimensions = {
  StreamName = aws_kinesis_stream.xfusion_kinesis.name
}

Ensures the alarm watches only this specific stream, not all Kinesis streams in the account.

What Happens During terraform apply?

1. Terraform creates the Kinesis stream.
2. Shard‑level metrics are enabled.
3. Terraform creates the CloudWatch alarm.
4. The alarm starts monitoring immediately.
5. Terraform stores the resources in its state file.
6. Output values (kke_kinesis_stream_name, kke_kinesis_alarm_name) are displayed.

Common Mistakes to Avoid

• Forgetting to enable shard_level_metrics.
• Using an incorrect metric namespace.
• Omitting the StreamName dimension.
• Setting an incorrect alarm threshold.
• Creating extra .tf files beyond main.tf and outputs.tf.

Resources & Next Steps

• Full Code Repository: KodeKloud Learning Labs
• More Deep Dives: Whispering Cloud Insights
• Discussion Community: DEV Community

Credits

• Lab content sourced from KodeKloud.