· software
Data Engineering Isn’t About Tools — It’s About Thinking Like This
Introduction Data engineering is often misunderstood as a discipline driven mainly by tools. New learners are frequently advised to master Airflow, Spark, Kafk...
-
- · software
Why Idempotency Is So Important in Data Engineering
In data engineering, failures are the norm: jobs crash, networks timeout, Airflow retries tasks, Kafka replays messages, and backfills rerun months of data. In...
- · software
GitHub Spark: Improvements, DPA coverage, & dedicated SKU
Spark is covered by the GitHub Data Protection Agreement As of October 27th, Spark is covered by the GitHub Data Protection Agreement, which means data handlin...
- · software
Day 10: Partitioning vs Bucketing - The Spark Optimization Guide Every Data Engineer Needs
Why Partitioning Matters in Spark Example python df.write.partitionBy'year', 'month'.parquet'/sales' This creates folders such as: year=2024/month=01/ Benefits...