· software
Starting Dusty — A Tiny DSL for ETL & Research Data Cleaning
!Cover image for Starting Dusty — A Tiny DSL for ETL & Research Data Cleaninghttps://media2.dev.to/dynamic/image/width=1000,height=420,fit=cover,gravity=auto,fo...
-
- · software
Navigating the Future: Key Data Engineering Trends for 2024 and Beyond
In the rapidly evolving landscape of data, data engineering stands as the backbone of every data‑driven organization. As businesses increasingly rely on data fo...
- · software
Continuous Journey through Dagster - bugs and testing
My Recent Contributions Fixing ECS Pipes Client Execution Issue: Users encountered an IndexError when launching tasks using the PipesECSClient, causing pipelin...
- · software
Data Pipeline Tools Compared: Key Criteria to Pick the Right One
Data’s all around us — from CRM systems and cloud apps to spreadsheets and data warehouses. When teams are wrangling numbers across 15+ platforms and spending m...
- · software
Advanced Configuration and Performance Tuning for Residential Proxies in the Scrapy Framework
When using Scrapy for large‑scale, high‑frequency data scraping, simple proxy settings quickly become insufficient. Random IP rotation and fixed delays can caus...
- · ai
Overfitting & Underfitting — Beyond Textbook Definitions (Part 5)
Part 5 of The Hidden Failure Point of ML Models series Most ML beginners think they understand overfitting and underfitting. In real production systems, overfi...
- · software
Trying Out Dagster for Data Orchestration
Intro I can't jump right into the pipeline without a brief intro and highlighting the most obvious differentiating factor that Dagster has – Assets. In Dagst...