pn-new-crm/docs/superpowers/specs/2026-06-01-bulk-import-design.md

Bulk CSV/XLSX Importer — Design Spec

Status: Approved (2026-06-01) · ready for implementation plan Driver: Replace the static admin/import mockup with a real self-serve importer. Primary purpose: one-time cutover migration of legacy NocoDB/portal data into the new CRM at launch. Tracker: docs/launch-readiness.md · feature-completeness batch.

Purpose & scope

A visual importer that ingests CSV/XLSX exports of the legacy system and loads them into the CRM with column-mapping, dry-run preview, dedup, and per-batch undo. Built for the cutover migration but engineered as a reusable engine (it can serve ongoing ops later without a rewrite).

In scope — seven entities, imported in dependency order so foreign keys resolve by natural key:

#	Entity	Dedup match-key	FKs resolved by natural key
1	Companies	name (case-insensitive)	—
2	Clients	primary email → fallback canonical phone	—
3	Yachts	name + owner (or HIN if present)	owner → client email / company name
4	Berths	mooringNumber (canonical ^[A-Z]+\d+$)	—
5	Interests/deals	default create-new (flag likely dupes by client+berth+stage)	client → email, primary berth → mooring
6	Tenancies	client + berth + startDate	client → email, berth → mooring
7	Expenses	date + amount + description (or none)	—

Berths are included for UI consistency even though scripts/import-berths-from-nocodb.ts already covers them via CLI.

Non-goals (v1): full pre-update snapshot/revert of updated rows (undo covers inserts only); streaming multi-GB files (migration files are small); scheduling/automation of imports; importing attachments/PDFs (handled by the Initiative 5 MinIO backfill scripts, separate).

Architecture — generic engine + per-entity adapter registry

One pipeline parameterised by a per-entity adapter, mirroring the existing src/lib/reports/custom/registry.ts and settings-registry patterns.

src/lib/import/registry.ts exports IMPORT_ENTITY_KEYS and IMPORT_REGISTRY: Record<ImportEntityKey, ImportAdapter>. Each adapter:

interface ImportAdapter {
  key: ImportEntityKey;
  label: string;
  order: number; // dependency order (companies=1 … expenses=7)
  dependsOn: ImportEntityKey[];
  /** Target fields drive the column-mapping UI + zod validation. */
  targetFields: ImportField[]; // { key, label, required, type, zod }
  /** Natural key used for dedup + as the FK-resolution lookup value. */
  matchKey: (row: MappedRow) => string | null;
  /** Resolve FK ids by natural key against the live DB. Returns ids or a
   *  per-field resolution error. */
  resolveForeignKeys: (row: MappedRow, ctx: ImportCtx) => Promise<FkResult>;
  /** Dedup lookup — find an existing row by matchKey within the port. */
  findExisting: (portId: string, matchKey: string) => Promise<{ id: string } | null>;
  /** Writes delegate to the EXISTING service helpers so audit logging,
   *  validation, and polymorphic-ownership rules come for free. */
  insert: (row: ResolvedRow, ctx: ImportCtx) => Promise<{ id: string }>;
  update: (existingId: string, row: ResolvedRow, ctx: ImportCtx) => Promise<void>;
}

Adding an entity = adding one adapter + registering it. No engine change.

Pipeline (BullMQ import queue, concurrency 1)

The queue + worker already exist (src/lib/queue/workers/import.ts is currently a documented no-op). We replace the no-op body with the real processor and add a producer.

1. Upload & parse. Drag-drop CSV/XLSX → parse (papaparse for CSV; ExcelJS already installed for XLSX) → raw rows. The uploaded file is stored via getStorageBackend() under a temp prefix so the worker can re-read it; cleaned up after commit or on expiry.
2. Map columns. Auto-suggest mappings by fuzzy header match to the adapter's targetFields; user overrides; save mapping as a per-port template (import_mappings) for re-runs.
3. Dry-run (no writes). Per row: apply mapping → zod-validate → resolveForeignKeys → findExisting → classify as will-insert | will-update | will-skip | error(line, reason). Surface counts + a sample of rows + a downloadable line-numbered error report.
4. Commit. Producer enqueues the job; the worker streams rows applying the chosen conflict policy (skip-matches / update-matches / error-on-match) via the adapter's insert/update. Per-row try/catch so valid rows still land; every action recorded in import_batch_rows; import_batches updated with live progress + final counts.
5. History + Undo. Admin list of batches (status, counts, error-report download). Undo deletes the rows a batch inserted, in reverse dependency order, refusing if any inserted row now has dependents created outside the batch. Updates are marked non-revertible in v1.

Data model (3 new tables; no changes to entity tables)

• import_batches — id, port_id, entity_type, filename, storage_key, status (uploaded|dry_run|committing|completed|failed|undone), total_rows, inserted, updated, skipped, errored, mapping_json, conflict_policy, created_by, created_at, completed_at.
• import_batch_rows — id, batch_id, row_number, action (inserted|updated|skipped|errored), entity_id (nullable), error (nullable). Powers the error report + undo. Migration-scale volume is fine.
• import_mappings — id, port_id, entity_type, name, mapping_json, created_by, created_at. Saved column mappings, reusable across runs.

Migration added via the project's psql-applied numbered migration flow; restart next dev after (prepared-statement cache caveat per CLAUDE.md).

Validation, errors, conflict policy

• Per-row zod from each adapter's targetFields; failures collected with row number + field + message, never aborting the whole file.
• Downloadable error report (CSV: row, field, message) from any dry-run or completed batch.
• Conflict policy chosen per import, surfaced at the dry-run step (three distinct behaviours for a matched row):
  • skip-matches — insert new, leave matched rows untouched. Default; safe to re-run.
  • update-matches — insert new, overwrite matched rows with the file's values (correct earlier mistakes).
  • error-on-match — treat a match as a row error to review, importing nothing for it (strictest).

UI

A 4-step wizard mirroring the existing bulk-add-berths wizard:

1. Pick entity (registry-driven, shown in dependency order with a hint) + upload file.
2. Map columns (auto-suggested; load a saved mapping; save current).
3. Dry-run preview — counts (new / update / skip / error), sample table, error-report download, pick conflict policy.
4. Commit — progress bar (worker reports % via batch counts) → result summary with link to History.

Plus an Import History tab: batch list + status + counts + error report + Undo. Replaces the static mockup at src/app/(dashboard)/[portSlug]/admin/import/page.tsx.

Permissions & tenancy

Gate behind a new data.import permission (admin-tier). Every query + write is port_id-scoped; FK resolution only matches within the port.

Testing (TDD)

• Per-adapter unit tests (one suite each): column mapping, zod validation (valid + each failure mode), matchKey, resolveForeignKeys (hit / miss / ambiguous), findExisting dedup.
• Dry-run classifier integration test on a seeded DB: a fixture file yielding one of each class (insert / update / skip / error).
• Commit worker integration test: each conflict policy; partial-failure (valid rows land, errored rows reported); idempotent re-run.
• Undo test: deletes inserted rows; refuses when an inserted row has an outside dependent.

Decisions locked (defaults the user approved 2026-06-01)

• Rollback depth: inserts-only undo; updates non-revertible in v1.
• Partial failure: valid rows commit, errors reported (not all-or-nothing).
• Berths: included in the UI importer despite the existing CLI.
• All seven entities in scope.
• Purpose: one-time cutover migration (engine reusable for ongoing ops).