Continuous Journey through Dagster - bugs and testing
Source: Dev.to
My Recent Contributions
Fixing ECS Pipes Client Execution
Issue: Users encountered an IndexError when launching tasks using the PipesECSClient, causing pipelines to crash in ECS environments.
Fix: Added proper exception handling and bounds checking to ensure the client launches tasks smoothly without crashing on index errors.
[Issue #32936]
Resolving Asset Specs Mapping Dependencies
Issue: Logic error in AssetsDefinition.map_asset_specs caused failures when adding dependencies while input definitions were already set.
Fix: Adjusted core logic to correctly handle mapping of asset specs even when inputs are pre‑configured.
[Issue #32913]
[WIP] Correcting Asset Sensor Event Processing
Issue: The asset_sensor only processed the last materialization event when multiple partitions materialized simultaneously due to a race condition.
Current Work: Modified sensor logic to capture and process every materialization event regardless of concurrency. Precise approach with careful testing is required.
[Issue #32853]
[WIP] Implementing Merge Support for Polars & Delta Lake
Use Case: dagster-deltalake I/O manager supports writing data but lacks merge operation support when using Polars.
Implementation: Updating dagster_deltalake/handler.py to support merge mode. When write mode is set to merge, a DeltaTable object is created and the merge operation is executed instead of the standard write_deltalake() call.
[Issue #32644]
CI Issues
Situation:
- Unit tests pass locally.
- After pushing to GitHub, the CI pipeline fails, preventing proper code review.
Possible causes include environment configuration mismatches, stricter linting rules in CI, or hidden dependency issues.
Next Steps
- Seek guidance from the Dagster team and community.
- Understand differences between the CI environment and a standard local setup to replicate and fix the failures.
- Continue reading code and fixing errors, as sometimes that is more efficient than exhaustive testing.