Custom import tool with column mapping, validation, diff preview, and rollback. Built for your schema, your rules, your team.
- Scope
- Ship
- Iterate
monthly subscription
Who this is for
You're the ops lead who runs monthly CSV uploads into a system that has no real guard rails. A bad file already corrupted production data once. The post-mortem had action items; the action items went nowhere. Now every import is a small act of faith.
The pain today
- Bad CSVs silently corrupting production with no warning
- No column mapping step — operators pray the headers match
- No diff preview before commit, so imports feel like a coin flip
- Validation rules buried in scripts that only one person understands
- Scheduled exports emailed to an address on someone's old laptop
The outcome you get
- Drag-and-drop upload with automatic header detection and column mapping
- Three-layer validation: syntactic, semantic, and your own business rules
- Diff preview showing adds, updates, and deletes before anything commits
- One-click rollback that restores the previous state within a configurable window
- Scheduled exports to S3, SFTP, webhook, or email with retry on failure
The five-step import pattern that prevents data disasters
Most import disasters share the same shape: operator uploads, system commits, something in the file was wrong, production is now wrong, nobody notices until Tuesday. The fix is a five-step UX pattern I build into every import tool.
Upload: accept the file, detect encoding, parse headers, sample rows. Map: let the operator match spreadsheet columns to your canonical schema — this step alone catches half the corruption events that happen with naive importers. Validate: run rules against every row before committing anything. Preview: show the operator exactly what would change — 12 rows added, 7 updated with before-and-after diffs, 2 flagged with errors. Commit: operator explicitly confirms after reviewing the preview.
Rollback sits underneath all of it. If something looks wrong post-commit, one click restores the previous state from a snapshot. The extra 60 seconds of preview and confirmation saves hours of manual recovery.
Drag-and-drop upload with automatic header detection and column mapping
Column mapping and header detection
The column mapping step is what separates a serious import tool from a script someone wrote on a Friday afternoon. Operators upload files from different sources — a CRM export, a supplier spreadsheet, a data dump from a legacy system. Headers are never perfectly consistent. "Customer ID", "customer_id", "CustID" are all the same thing to a human and all different things to a naive parser.
I build automatic header detection that compares incoming column names against your canonical schema using fuzzy matching. The tool suggests mappings; the operator confirms or overrides. Mappings can be saved as templates so repeat uploads from the same source don't require manual remapping every time.
For custom formats (fixed-width files, XML, EDI), a short additional spec phase covers the parser. Large files are streamed chunk by chunk, so a 1 GB file processes without memory problems.
120k+: Properties indexed and searchable.
Three-layer validation and per-row error reporting
Validation runs in three layers. Syntactic: does the file parse, do column counts match after mapping, are required fields present. Semantic: do values match expected types, do foreign key IDs reference existing records, do enum values appear in allowed lists. Business rules: do operator-defined conditions hold — price must be greater than zero, customer status must be active, quantity cannot exceed stock on hand.
Errors report per-row with the specific column and reason in plain language. Operators see a summary table: 847 rows valid, 3 rows with errors, reason listed for each. They can fix errors in the UI for small sets, download an error report for large sets, or abort and re-upload a corrected file. No guesswork, no silent failures.
Common rules — required, unique, type, min/max, regex, referential integrity — configure from a UI with no code. Complex multi-field rules and external API checks ("verify this company ID exists in our CRM before import") need a short plugin, which I write as part of the build.
Async processing and queue architecture
Imports of 100k or more rows need to run in the background, not lock the operator's browser tab for 20 minutes. The pattern: upload triggers a background job via a queue (Redis, SQS, or equivalent), the operator gets a live progress indicator, the system processes in configurable chunk sizes, and completion pushes via server-sent events or email.
Resumable imports matter at scale. If the job dies mid-run, the processor restarts from the last checkpoint rather than beginning again from row one. This makes 500k-row imports as reliable as 500-row imports.
Exports follow the same async pattern. The operator clicks export, a job enqueues, the completed file lands in S3 or arrives by email. Scheduled exports run on daily, weekly, monthly, or cron-expression schedules with exponential-backoff retries on failure and an alert when retries are exhausted.
How Imohub handles 120k+ records through similar patterns
Imohub ingests property data from 20-plus sources: scraped feeds, partner APIs, and manual uploads from agents. Each source has its own schema, its own naming conventions, its own quirks. The import pipeline normalizes everything to a canonical schema, detects duplicates, and indexes 120k-plus property records to under 0.5-second query response while running infrastructure at 70% lower cost than the legacy stack.
The patterns are identical to what I build for smaller-scale import tools: validation layers, diff review for operator-uploaded files, async processing for bulk feeds, rollback for recovery from bad batches. The volume changes. The discipline does not.
For a single-tenant internal tool with one or two data sources, the build is simpler. The architecture decisions — parse before commit, validate in layers, give operators a preview — are the same regardless of scale.
Downstream writes and transactional consistency
An import tool does not live alone. Once validated data commits to your primary store, it usually needs to reach downstream systems: a search index, a CRM, an analytics warehouse, a third-party API. How those writes happen matters.
For synchronous downstream writes, I wrap them in the same database transaction as the primary insert. All succeed or all roll back together. A partial import that leaves your database updated but your search index stale causes exactly the kind of subtle inconsistency that takes days to diagnose.
For async downstream writes — indexing to Elasticsearch, syncing to HubSpot, pushing to a webhook — I use an outbox pattern. The import writes to an outbox table in the same transaction as the primary data. A separate worker drains the outbox into downstream systems with retries and dead-letter handling. Downstream failures do not corrupt your primary data, and every event is recoverable. It is not glamorous architecture, but it works reliably across the cases where I have shipped it.
Pricing and timeline
Data import and export tools fit the Applications Standard tier at $4,999/mo. Complex ETL builds — multiple sources, transformation logic, scheduled jobs with custom delivery integrations — move to the Pro tier at $5,499/mo.
First-version timeline for a single-source import tool with validation, preview, and rollback is typically three to four weeks. The subscription continues as new data sources, validation rules, or export destinations get added.
The guarantee is a 14-day money-back window, cancel anytime after, and Work Made for Hire — code is yours the moment you pay. For a one-off data migration (move everything from system A to system B, done) a fixed-price project under the Websites pricing structure may be more appropriate. Ask and I will scope it.
Recent proof
A comparable engagement, delivered and documented.
Rebuilt a real estate portal at a fraction of the cost
Rebuilt Imóveis SC's real estate portal as ImoHub, a faster, more scalable successor, handling 120k+ properties with sub-second search and drastically reduced AWS costs.
Read the case studyFrequently asked questions
The questions prospects ask before they book.
A single-source import tool with drag-and-drop upload, column mapping, validation, diff preview, rollback, and scheduled exports typically takes three to four weeks from kickoff to a working first version. That assumes your schema is defined and I have access to a sample dataset. More sources, more complex transformation logic, or multi-tenant requirements add scope. I give a firm timeline estimate after a short scoping call.
Standard CSV and TSV, Excel (.xlsx and .xls), and JSON out of the box. Custom formats — fixed-width files, EDI, XML, or proprietary exports from legacy systems — are supported with a short additional spec phase. Large files are streamed chunk by chunk, so a 1 GB upload does not cause memory problems or timeouts for the operator.
Each import creates a snapshot of the affected rows before committing. If something looks wrong after commit, rollback restores from the snapshot in a single click. The retention window is configurable — typically seven days of rollback history stored as S3-backed snapshots. Beyond that window, recovery requires restoring from full backups, which is slower and involves a manual request rather than a self-service button.
Yes. Common rules — required fields, unique constraints, type checks, min/max values, regex patterns, and referential integrity — configure from a UI without touching code. Complex rules that span multiple fields or require checking an external system (for example, confirming that a submitted company ID exists in your CRM before import) need a small plugin. I write those plugins as part of the build; your engineering team can maintain or extend them afterward.
Exports support S3, SFTP, webhook (POST to a URL you provide), and email attachment. Multiple destinations per schedule are supported — the same nightly export can land in S3 and trigger a webhook simultaneously. Exports can be encrypted before delivery (GPG or AES) for PII-sensitive data. Failed exports retry with exponential backoff and alert on permanent failure so a missed delivery does not go unnoticed until someone complains.
Imports can pre-mask PII fields in the diff preview — operators see enough to verify the row without the full sensitive value being visible in the UI. The committed database value is unmasked. Exports can encrypt the file before delivery and can apply field-level redaction for audit exports. Every import and export event writes to an audit log capturing who triggered it, when, what file, and how many rows were affected, which covers most compliance review requirements.
Both. Most customers want the import tool embedded in their existing internal dashboard, not as a separate URL. I build the import interface as a self-contained module that mounts into your existing app. If you have no existing app, I build a minimal wrapper with authentication. The underlying processing logic is backend-agnostic and can connect to any database I work with — PostgreSQL, MySQL, MongoDB, or an API endpoint you expose.