Event-Driven Architecture on a Budget: Kotlin Coroutines + Redis Streams

Source: Dev.to

What We Will Build

Today we are building a lightweight event‑driven pipeline using Kotlin coroutines and Redis Streams. By the end, you will have a working producer that buffers and batches events, a consumer built on Kotlin Flows, and a clean abstraction layer that lets you swap in Kafka later without rewriting your application.

Prerequisites

• Kotlin 1.9+ with coroutines (kotlinx-coroutines-core)
• A running Redis 6.2+ instance (Docker works fine: docker run -p 6379:6379 redis:7)
• Lettuce Redis client with coroutine support (lettuce-core)
• Basic familiarity with Kotlin coroutines and Flow

Step 1: Build the Buffered Producer

The producer uses a Kotlin Channel for local back‑pressure, then flushes events to Redis Streams in batches.

class EventProducer(
    private val redis: RedisCoroutinesCommands,
    private val stream: String,
    bufferSize: Int = 1024
) {
    private val channel = Channel(bufferSize)

    suspend fun emit(event: Map) {
        channel.send(event)
    }

    fun startFlusher(scope: CoroutineScope, batchSize: Int = 100) {
        scope.launch {
            val batch = mutableListOf()
            for (event in channel) {
                batch.add(event)
                if (batch.size >= batchSize || channel.isEmpty) {
                    batch.forEach { redis.xadd(stream, it) }
                    batch.clear()
                }
            }
        }
    }
}

The Channel suspends the caller when the buffer is full instead of dropping events. This back‑pressure propagation makes the setup production‑safe.

Step 2: Build the Consumer With Flow

Each consumer reads from a Redis consumer group via XREADGROUP and emits messages into a Kotlin Flow.

fun CoroutineScope.consumeStream(
    redis: RedisCoroutinesCommands,
    stream: String,
    group: String,
    consumer: String
): Flow = flow {
    runCatching { redis.xgroupCreate(stream, group, "0") }

    while (currentCoroutineContext().isActive) {
        val messages = redis.xreadgroup(
            Consumer.from(group, consumer),
            XReadArgs.Builder.count(50).block(Duration.ofSeconds(2)),
            XReadArgs.StreamOffset.lastConsumed(stream)
        )
        messages.forEach { emit(it) }
    }
}

A consumer coroutine costs roughly 200 bytes of heap. In practice, 500 concurrent consumers can run on a single 512 MB container handling ~12 000 events/sec with p99 latency under 15 ms.

Step 3: Abstract the Event Bus

Wrap everything behind an interface from day one to avoid costly rewrites later.

interface EventBus {
    suspend fun publish(topic: String, event: Map)
    fun subscribe(topic: String, group: String): Flow
}

Your RedisEventBus implements this interface. When you outgrow Redis Streams, you can write a KafkaEventBus and swap it with minimal changes.

Gotchas

• Retention is memory‑bound. Redis Streams live in RAM. If you need retention beyond 48–72 hours or your stream depth exceeds available memory, Kafka becomes more economical. Use MAXLEN with XADD to cap memory usage.
• No schema registry. Manage schema evolution yourself (e.g., add a version field "v" to the event map).
• Cross‑datacenter replication is fragile. Redis replication works for single‑region setups; multi‑region scenarios are better suited to Kafka.
• channel.isEmpty is not atomic. The batch flush check is best‑effort; under very high throughput you may flush smaller batches than expected. This is fine for correctness.
• Consumer group creation is not idempotent by default. The runCatching around xgroupCreate prevents errors when the group already exists.

When to Migrate to Kafka

Monitor two metrics: sustained event rate and retention depth. When you consistently hit ~40 K events/sec or need retention beyond 72 hours, start planning a migration. Below that threshold, Kafka adds roughly $400/month extra infrastructure cost and operational overhead you may not need.

Conclusion

Start with Redis Streams if you are under 50 K events/sec. Use Kotlin coroutine Channels for back‑pressure and Flows for consumption. Abstract your event bus behind an interface from the beginning. This approach gives you production‑ready, event‑driven workflows at a fraction of the cost and complexity of Kafka.