Index / Data Plumbing
PB-013

Export Cleanup Pipeline

Turns a raw export into a validated dataset, with every row it could not clean listed rather than silently dropped.

The spec — copy this

Set up a new bot for me I can trigger on any raw export. Walk me through connecting my document store and spreadsheet, then configure it: take the export and produce a clean dataset plus a data-quality report. Normalize dates to a single format and flag any that are impossible, out of range, or ambiguous between day-first and month-first; standardize picklist and free-text variants to the canonical values I define; split or join name and address fields consistently; normalize phone numbers and mark the invalid ones; and deduplicate on the keys I specify with a stated confidence level on every proposed merge. Never silently drop, merge, or auto-correct a row — everything it cannot clean goes to an exceptions file with the reason and the original value preserved, and the counts have to reconcile: rows in, rows clean, rows excepted, and they must add up on the face of the report. Show me the fill rate for every column before and after. Ask me for my canonical values, my dedupe keys and confidence threshold, my valid date ranges, and which columns are required versus optional, run it against an export I have already cleaned by hand so I can diff its output against mine, then save it.
Paste into an assistant that can connect to your systems.◆ Ethics note below — read before you run it

Connect first

The spec asks for these as it goes — however you normally connect them works. Nothing needs to be set up in advance, and a system named here is usually an example rather than a requirement. If yours has an API or an export, the spec generally adapts.

What it never does
  • Silently drop, merge, or auto-correct a row — anything it cannot clean is listed with its reason and original value
  • Auto-merge a duplicate below your stated confidence threshold
  • Write back to the source system
  • Produce a report whose row counts do not reconcile
Before you run this

Exports of matter data carry client information and often protected health information. Handle the cleaned file and the exceptions file with the same care as the source, and confirm your vendor's retention terms before uploading either.

Category
Data Plumbing
Contributed by
Jacob Malherbe @jacobmalherbeMass Tort Ad Agency
Approval gate
A named human approves before anything sends, files, or publishes.
Last verified
2026-08-19

More in Data Plumbing