vhfdx.net

I Lived With a Data Science Pipeline for a Year — Here’s My Honest Take

Written by

in

I’m Kayla, and I really did build and babysit this thing. My team named it “Piper,” like the bird. Cute, right? Some nights it felt like a needy pet. If you want the full play-by-play, here’s my honest take after living with a data science pipeline for a year. Most days it was a good partner that did the boring stuff so I could think. So yes, I’ve got stories.

What I Mean by “Pipeline” (No fluff, promise)

A data science pipeline is just the steps from raw data to a model that helps a real person make a call. It runs on a schedule. It checks itself. It stores stuff. It makes a prediction or a report. Then it does it again tomorrow.

Here’s the stack I used most:

  • Prefect 2.0 for flow runs (I tried Airflow and Dagster too)
  • Great Expectations for data checks
  • DVC and S3 for data versioning
  • MLflow for runs and model registry
  • scikit-learn, XGBoost, and LightGBM for models
  • Docker for builds and FastAPI for serving
  • Snowflake and Postgres for data stores
  • GitHub Actions for CI
  • Feast for features (on one project)

If you’d like an at-a-glance diagram of how all these pieces can snap together, the cheat-sheet on VHF DX lays it out beautifully in a single scroll.

I know that list looks long. It didn’t all show up at once. It grew because we had real problems to solve.

Real Example 1: Late Delivery Risk (Meal Kits)

I built this at a meal kit startup. Picture a big fridge, a bunch of drivers, and a timer that never stops. We wanted to flag orders that might ship late, so ops could jump in early.

The flow, in plain steps:

  1. Pull order data from Postgres and driver pings from S3 every 15 minutes.
  2. Run Great Expectations checks (no missing zip codes, valid timestamps, sane route times).
  3. Build features: day of week, weather, stop count, driver shift length.
  4. Train LightGBM once a day at 2 a.m. Store the model in MLflow.
  5. Serve a FastAPI endpoint. Ops hit it from their tool to see the risk score.
  6. Ping Slack if data checks fail or if AUC drops a lot.

Numbers that mattered:

  • Training time: 11 minutes on a c5.xlarge.
  • AUC moved from 0.67 (baseline) to 0.79 with LightGBM.
  • Late orders fell 18% in four weeks.
  • S3 cost spiked to about $42/month just from writing too many small Parquet files; we fixed it with daily compaction.

Pain points I still remember:

  • Time zones. Oh my word. DST in March broke a cron and we missed a run. I pinned tz to UTC and added a slack alert for “no run by 2:30 a.m.”
  • Schema drift: one day “driver_id” became “courier_id.” Great Expectations caught it, but the backfill took a full afternoon.
  • Airflow vs Prefect: Airflow worked, but the UI and Celery workers were fussy for our small team. Prefect cloud felt lighter and my flows were easier to test.

What helped more than I expected:

  • Storing 1k sample rows in the repo for fast tests. I could run the full flow in 90 seconds on my laptop with DuckDB.
  • Feature names that read like real words. “driver_hours_rolling_7d” beats “drv_hr_7.”

Real Example 2: Churn Model (Fitness App)

Different team, same Kayla. The goal: flag users who might leave next month, so we could send the right nudge.

The flow went like this:

  • Ingest events from BigQuery each night.
  • Build weekly features: streak length, last workout type, plan price, support tickets.
  • Run a simple logistic model first. Then XGBoost.
  • Log all runs to MLflow with tags like “tag:ab_test=variant_b.”
  • Push scores to Snowflake; a small job loads them into Braze for messages.

Highlights:

  • Logistic regression was fast and fair. AUC 0.72.
  • XGBoost hit 0.78 and picked up new signal when we added “class pass used” and “push opens.”
  • We ran an A/B for six weeks. Retention rose 2.6 points. That was real money.

Where it stung:

  • We had a data leak. “Last 7 days canceled flag” slipped into the train set by mistake. It looked great in dev. In prod, it dropped like a rock. We added a “no look-ahead” guard on every feature job after that.
  • YAML sprawl. Feast configs and job configs got messy. We cut it down by moving defaults to code.

Tiny things that saved time:

  • A “slow mode” flag in the flow. It ran only two features and one small model on PRs. CI dropped from 20 minutes to 6.
  • A rollback button in MLflow model registry. When a new model underperformed by 5%, we flipped back in seconds.

Real Example 3: Price Forecast for a Retail Sale

Short, sharp project for Black Friday. We needed a quick forecast per SKU.

My mix:

  • Prophet for a fast start, then SARIMAX for the top 50 SKUs.
  • DVC tracked the holiday-adjusted training sets.
  • Airflow ran batch forecasts at 1 a.m.; results went to a Snowflake table for BI.

Stuff I learned:

  • Prophet was fine for most items. SARIMAX won for items with a clean weekly pulse.
  • Daylight saving again. The 1 a.m. run vanished on the fall-back day. We set a sensor that waited for fresh data, not a time. That fixed it.

Results:

  • MAPE dropped from 24% to 13% on the top sellers.
  • We held the rest at 16–18% with Prophet and called it done. No heroics.

Tools I Liked (And Why)

  • Prefect: Clear logs, easy retries, local runs felt normal. The UI showed me state in a way that made sense when I was tired. (If you’d like to peek under the hood, the orchestration layer is explained here.)
  • Dagster: Strong types and solids… sorry, ops. It pushed me to write cleaner steps.
  • MLflow: The model registry fit how my team worked. Tags and stages saved us in rollbacks.
  • Great Expectations: Boring in the best way. It caught a lot. I slept better.
  • Kedro: A nice project shape. Pipelines felt like Lego. Even new hires found stuff fast. Back when I was a data science intern in New York, I would have killed for a repo structured that clearly.

What Bugged Me

  • Airflow on small projects. It’s solid, but I spent more time on workers and queues than on models.
  • Permissions. S3, Snowflake, GitHub Actions… secrets go stale at the worst time. I moved secrets to AWS Parameter Store and rotated monthly.
  • Docker builds. Slow. I used slim bases, pinned versions, and cached wheels. Still slow on CI sometimes.
  • Backfills. They always look small. They never are. Plan for it. Keep a runbook with commands you trust.

My Simple Checklist (I actually use this)

  • Start with a dumb model and one data check. Ship.
  • Add Great Expectations as soon as you touch prod data.
  • Keep a 1k-row sample set in the repo for tests.
  • Use MLflow or a tracker. You won’t remember what you ran.
  • Watch cost. Compact small files. Parquet over CSV.
  • Add alerts for “no run,” “no new data,” and “metric drop.”
  • Write down how to backfill. Test that doc on a Tuesday, not a midnight.

A Quick Story About Humans

One morning our late-delivery scores looked weird. Like, spooky quiet. The model was “fine,” but Slack was silent. Ops thought things were smooth. They weren’t. A data check had failed and the alert filter was too strict. We fixed the filter. We also added a small banner in the ops tool: “Model paused.” Humans first. Models second. That small bar saved calls and trust.

Final Take

You know what? The best part was this: people used the stuff. Drivers made fewer late runs. Members stayed a bit longer. That made the angry nights worth it.

Speaking of real-time systems, running a specialized community chat site demands the same kind of rock-solid reliability that pipelines crave. A fun example is the BBW room over at instantchat.com/bbw/—its snappy, always-available channels let you experience firsthand how seamless infrastructure keeps conversations lively and users coming back. As another example from a completely different corner of the internet, I once helped a classifieds-style dating board monitor

More posts