Almost every applied machine learning project starts with the same optimistic sentence: "The data is out there on the web." It usually is. The hard part is turning pages that change every week into a model that keeps making good predictions months after launch. Here is the pipeline we use, stage by stage.
Stage 1 — Define the prediction before the data
Start with the decision. "Forecast weekly demand for 2,000 SKUs so buyers can plan purchase orders" is a project. "Collect all product data" is not. The decision tells you which fields you need, how fresh they must be and how accuracy will be judged.
Stage 2 — Collect responsibly and resiliently
A production collector is very different from a one-off script:
- Scheduling that matches how quickly the source changes
- Validation on every field, so a layout change raises an alert instead of silently writing empty values
- Politeness — respect robots directives, rate limits and terms of use
- Raw storage of what was collected, so you can re-parse history when requirements change
Stage 3 — Clean, normalise and de-duplicate
Web data is messy. The same product appears under slightly different names; prices arrive as "₹1,299", "$19.99" or "1299.00"; categories follow each website's own logic. This stage standardises units and currencies, maps categories to one taxonomy and merges duplicates with fuzzy matching. It is unglamorous work, and it is where most of the model's eventual accuracy comes from.
Stage 4 — Engineer features with time in mind
For forecasting and pricing problems, every feature must be computed as of the prediction date. Rolling averages, price changes versus competitors and days since last stock-out are powerful — but only if they never peek into the future. Build features in a way that can be recomputed identically for training and for live predictions.
Stage 5 — Train against a baseline
Before any complex model, measure a simple baseline: last week's value, a seasonal average or a linear model. Then try gradient-boosted trees and, where the data supports it, deep learning. A model is only worth deploying if it beats the baseline on a time-based hold-out set by a margin that matters to the business.
Stage 6 — Deploy where decisions are made
A model that lives in a notebook helps no one. Depending on the use case, ship it as:
- A batch job that writes predictions to a table or spreadsheet every morning
- A REST API that an application calls in real time
- A dashboard that puts predictions next to the metrics people already watch
Stage 7 — Monitor data and model together
The web keeps changing after launch, so monitoring has two layers:
| Layer | What to watch | Typical alert |
|---|---|---|
| Data | Volume, missing fields, value ranges | Record count drops 30% overnight |
| Model | Input drift, prediction drift, accuracy | Forecast error exceeds threshold for 2 weeks |
When alerts fire, a retraining pipeline refreshes the model on recent data and promotes it only if it beats the current version.
The takeaway
Most of the effort — and most of the value — sits in stages 2, 3 and 7, the parts that rarely appear in tutorials. Getting them right is what separates a promising demo from a system your team can rely on.
If you want this pipeline without building a data team first, our subscription plans bundle managed collection, cleaning and machine learning into one predictable monthly price.