Dagster on our own server
Everything ran on Excel macros, stored procedures and Python scripts that someone had to kick off by hand. I got us an on-site server, set up Dagster connected to GitHub, and started moving the pipelines over. Now there’s version control, one place to see what ran, easier teamwork, and a foundation for ML to come next.
How it works
Tap a step for the details.
Excel macros, stored procedures and Python scripts, all run by hand.
I asked for an on-site server to give the pipelines a home.
Dagster (a tool that runs data jobs) set up and connected to GitHub.
Pipelines moved across one at a time.
See what ran, work together, and lay the ground for ML later.