Files
pn-new-crm/scripts/migrate-from-nocodb.ts
Matt Ciaccio 18e5c124b0 feat(dedup): NocoDB migration script + tables (P3 dry-run)
Lands the one-shot migration pipeline from the legacy NocoDB Interests
base into the new client/interest schema. Dry-run mode is fully
operational: pulls the live snapshot, runs the dedup library, and
writes a CSV + Markdown report under .migration/<timestamp>/. The
--apply phase is stubbed for a follow-up PR per the design's P3
implementation sequence.

Schema additions
================

- `client_merge_candidates` — pairs flagged by the background scoring
  job for the /admin/duplicates review queue. Status enum: pending /
  dismissed / merged. Unique-(portId, clientAId, clientBId) so the
  same pair can't surface twice. Empty until P2 lands the cron.
- `migration_source_links` — idempotency ledger. Maps source-system
  rows (NocoDB Interest #624 → new client UUID) so re-running --apply
  against the same dry-run report skips already-imported entities.

Both tables ship with the migration `0020_unusual_azazel.sql` —
already applied to the local dev DB during this commit's preparation.

Library
=======

src/lib/dedup/nocodb-source.ts
  Read-only adapter for the legacy NocoDB v2 API. xc-token auth,
  auto-paginates until isLastPage, captures the table IDs from the
  2026-05-03 audit. `fetchSnapshot()` pulls every relevant table in
  parallel into one in-memory object the transform layer consumes.

src/lib/dedup/migration-transform.ts
  Pure function: NocoDB snapshot in, MigrationPlan out. Per row:
    - normalizes name / email / phone / country via the dedup library
    - parses the legacy DD-MM-YYYY / DD/MM/YYYY / ISO date formats
    - maps the 8-stage `Sales Process Level` enum to the new 9-stage
      pipelineStage
    - filters yacht-name placeholders ('TBC', 'Na', etc.)
    - merges Internal Notes + Extra Comments + Berth Size Desired into
      a single notes blob
  Then runs `findClientMatches` pairwise (with blocking) and
  union-finds clusters of rows whose score crosses the auto-link
  threshold (90). Lower-scoring pairs (50–89) become 'needs review'.
  Each cluster's "lead" row is picked by completeness score with
  recency tie-break.

src/lib/dedup/migration-report.ts
  Writes three artifacts to .migration/<timestamp>/:
    - report.csv  — one row per planned op, RFC-4180 escaped
    - summary.md  — human-skimmable overview
    - plan.json   — full structured plan for the --apply phase
  CSV cells with comma / quote / newline are quoted; internal quotes
  are doubled. No external CSV dep.

src/lib/dedup/phone-parse.ts
  Script-safe wrapper around libphonenumber-js's `core` entry that
  loads `metadata.min.json` directly. The default `index.cjs.js`
  bundled by libphonenumber hits a metadata-shape interop bug under
  Node 25 + tsx (`{ default }` wrapping); core+JSON sidesteps it.
  The dedup `normalizePhone` and `find-matches` both use this wrapper
  now so the same code path runs in vitest, Next.js, and the migration
  CLI without surprises.

src/lib/dedup/normalize.ts
  Tightened country resolution: added Caribbean short-form aliases
  ('antigua' → AG, 'st kitts' → KN, etc.) and a city map covering the
  US locations seen in the NocoDB dump (Boston, Tampa, Fort
  Lauderdale, Port Jefferson, Nantucket). Also relaxed phone parsing
  to drop the `isValid()` strict check — the libphonenumber min build
  rejects many real NANP-territory numbers, and dedup only needs a
  canonical E.164 to compare.

CLI
===

scripts/migrate-from-nocodb.ts
  pnpm tsx scripts/migrate-from-nocodb.ts --dry-run
    → Pulls the live NocoDB base (NOCODB_URL + NOCODB_TOKEN env vars),
       runs the transform, writes report. No DB writes.
  pnpm tsx scripts/migrate-from-nocodb.ts --apply --report .migration/<dir>/
    → Stubbed; exits with `not yet implemented` and a pointer to the
       design doc. Apply phase ships in a follow-up.

Tests
=====

tests/unit/dedup/migration-transform.test.ts (7 cases)
  Fixture-based regression. A frozen 12-row NocoDB snapshot covers
  every duplicate pattern in the design (§1.2). The test asserts:
    - 12 input rows → 7 unique clients (cluster math is right)
    - Patterns A / B / C / E auto-link
    - Pattern F (Etiennette Clamouze) does NOT auto-link
    - Every interest preserved as its own row even when clients merge
    - 8-stage → 9-stage enum mapping is correct per spec
    - Multi-yacht merge (Constanzo CALYPSO + Costanzo GEMINI under one
      client) — the design's signature win
    - Output is deterministic (run twice, identical)

Validation against real data
============================

Ran `pnpm tsx scripts/migrate-from-nocodb.ts --dry-run` against the
live NocoDB. Result on 252 Interests rows:
  - 237 clients (15 merged into 13 clusters)
  - 252 interests (one per source row)
  - 406 contacts, 52 addresses
  - 13 auto-linked clusters (every confirmed cluster from §1.2 audit)
  - 3 pairs flagged for review (Camazou, Zasso, one new)
  - 1 phone placeholder flagged

Total dedup test count: 57 (50 from P1 + 7 fixture tests).
Lint: clean. Tsc: clean for new files.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:50:01 +02:00

145 lines
5.3 KiB
TypeScript

/**
* One-shot migration: legacy NocoDB Interests → new client/interest split.
*
* Usage:
*
* pnpm tsx scripts/migrate-from-nocodb.ts --dry-run
* Pulls the live NocoDB base, runs the transform + dedup pipeline,
* writes a report to .migration/<timestamp>/. NO database writes.
*
* pnpm tsx scripts/migrate-from-nocodb.ts --dry-run --port-slug harbor-royale
* Same, but tags the planned writes with the named port (matters for
* the apply phase — every client/interest belongs to one port).
*
* pnpm tsx scripts/migrate-from-nocodb.ts --apply --report .migration/<dir>/
* [Not yet implemented — apply phase comes in a follow-up PR.]
*
* Design reference: docs/superpowers/specs/2026-05-03-dedup-and-migration-design.md §9.
*/
import 'dotenv/config';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { fetchSnapshot, loadNocoDbConfig } from '@/lib/dedup/nocodb-source';
import { transformSnapshot } from '@/lib/dedup/migration-transform';
import { resolveReportPaths, writeReport } from '@/lib/dedup/migration-report';
interface CliArgs {
dryRun: boolean;
apply: boolean;
portSlug: string | null;
reportDir: string | null;
}
function parseArgs(argv: string[]): CliArgs {
const args: CliArgs = {
dryRun: false,
apply: false,
portSlug: null,
reportDir: null,
};
for (let i = 0; i < argv.length; i += 1) {
const a = argv[i]!;
if (a === '--dry-run') args.dryRun = true;
else if (a === '--apply') args.apply = true;
else if (a === '--port-slug') args.portSlug = argv[++i] ?? null;
else if (a === '--report') args.reportDir = argv[++i] ?? null;
else if (a === '-h' || a === '--help') {
printHelp();
process.exit(0);
} else {
console.error(`Unknown argument: ${a}`);
printHelp();
process.exit(1);
}
}
return args;
}
function printHelp(): void {
console.log(`Usage:
pnpm tsx scripts/migrate-from-nocodb.ts --dry-run [--port-slug <slug>]
Pulls NocoDB → transforms → writes report to .migration/<timestamp>/.
No database writes.
pnpm tsx scripts/migrate-from-nocodb.ts --apply --report .migration/<dir>/
Apply phase. (Not yet implemented.)
Flags:
--dry-run Read NocoDB, write report only.
--apply Actually write to the new DB. (Not yet supported.)
--port-slug <slug> Port slug to attach to all imported entities.
Defaults to the first available port if omitted.
--report <dir> Path to a previously-generated report dir
(only used by --apply).
-h, --help Show this help.
`);
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
if (!args.dryRun && !args.apply) {
console.error('Must specify --dry-run or --apply');
printHelp();
process.exit(1);
}
if (args.apply) {
console.error('--apply is not yet implemented in this version. P3 ships dry-run first.');
console.error('See docs/superpowers/specs/2026-05-03-dedup-and-migration-design.md §9.2.');
process.exit(2);
}
// ── Dry-run path ───────────────────────────────────────────────────────────
console.log('[migrate] Loading NocoDB config…');
const config = loadNocoDbConfig();
console.log(`[migrate] Source: ${config.url}`);
console.log('[migrate] Fetching snapshot from NocoDB…');
const start = Date.now();
const snapshot = await fetchSnapshot(config);
const elapsed = ((Date.now() - start) / 1000).toFixed(1);
console.log(
`[migrate] Snapshot fetched in ${elapsed}s — ${snapshot.interests.length} interests, ${snapshot.residentialInterests.length} residential, ${snapshot.berths.length} berths.`,
);
console.log('[migrate] Running transform + dedup pipeline…');
const plan = transformSnapshot(snapshot);
// Resolve output paths relative to the worktree root (the script itself
// lives in scripts/; we want the .migration dir at the repo root).
const scriptDir = path.dirname(fileURLToPath(import.meta.url));
const repoRoot = path.resolve(scriptDir, '..');
const generatedAt = new Date().toISOString();
const paths = resolveReportPaths(repoRoot);
console.log(`[migrate] Writing report to ${paths.rootDir}`);
await writeReport(paths, plan, generatedAt);
// ── Console summary ──────────────────────────────────────────────────────
const s = plan.stats;
console.log('');
console.log('=== Migration Plan Summary ===');
console.log(
` Input: ${s.inputInterestRows} interests, ${s.inputResidentialRows} residential interests`,
);
console.log(` Output: ${s.outputClients} clients, ${s.outputInterests} interests`);
console.log(` ${s.outputContacts} contacts, ${s.outputAddresses} addresses`);
console.log(
` Dedup: ${s.autoLinkedClusters} auto-linked clusters, ${s.needsReviewPairs} pairs flagged for review`,
);
console.log(` Quality: ${s.flaggedRows} rows flagged (see report.csv)`);
console.log('');
console.log(` Full report: ${paths.summaryPath}`);
console.log('');
}
main().catch((err) => {
console.error('[migrate] Fatal error:', err);
process.exit(1);
});