Data processing
Build the path data travels: collect, clean, load, verify.
SYNOPSIS
data [--collect] [--clean] [--load] [--reconcile]DESCRIPTION
Moving data takes less time than proving it is right, so we decide how to reconcile it during design.
We choose batch or stream to fit the job, and we build in where to restart from when a run fails.
SCOPE
- Collect
- Reliable intake from scattered sources: files, databases, APIs.
- Clean
- Format and value mismatches resolved by rules, with exceptions kept apart.
- Load
- Bulk migration and incremental loads, centered on PostgreSQL.
- Reconcile
- Source against result by counts, totals, and samples, written up as a report.
DELIVERABLES
- Pipeline code
- Cleaning rules document
- Reconciliation report
- Runbook with re-run procedures
FIT
Good fit
- A system replacement means moving a large amount of data
- Data from several sources has to end up in one place
Not a fit
- You only need analytics screens and dashboards