Module 2 Summary - Workflow Orchestration with Kestra Part 1

Source: Dev.to

Part 1: Introduction to Workflow Orchestration & Kestra Fundamentals

What is Workflow Orchestration?

Think of a music orchestra with various instruments that need to work together. The conductor helps them play in harmony. Similarly, a workflow orchestrator coordinates multiple tools and platforms to work together.

A workflow orchestrator typically:

• Runs workflows containing predefined steps
• Monitors and logs errors with additional handling when they occur
• Automatically triggers workflows based on schedules or events

In data engineering, we often need to move data from one place to another with modifications. The orchestrator manages these steps while providing visibility into the process.

Kestra Overview

Kestra is an open‑source, event‑driven, infinitely‑scalable orchestration platform. Key features include:

Core Concepts

• Flow – A container for tasks and orchestration logic (defined in YAML)
• Tasks – Individual steps within a flow
• Inputs – Dynamic values passed at runtime
• Outputs – Data passed between tasks and flows
• Triggers – Mechanisms that automatically start flow execution
• Execution – A single run of a flow with a specific state
• Variables – Key‑value pairs for reusable values across tasks
• Plugin Defaults – Default values applied to tasks of a given type
• Concurrency – Controls how many executions can run simultaneously

Running Kestra with Docker Compose

# Navigate to the project directory
cd 02-workflow-orchestration

# Start Kestra and its PostgreSQL database
docker compose up -d

Access the UI at the appropriate URL (typically http://localhost:8080).

Executing Python Code in Kestra

Kestra can run Python code either:

• From a dedicated file, or
• Directly inside the workflow YAML

This flexibility lets you choose the right tools for your pipelines without limitations.