Lands the one-shot migration pipeline from the legacy NocoDB Interests base into the new client/interest schema. Dry-run mode is fully operational: pulls the live snapshot, runs the dedup library, and writes a CSV + Markdown report under .migration/<timestamp>/. The --apply phase is stubbed for a follow-up PR per the design's P3 implementation sequence. Schema additions ================ - `client_merge_candidates` — pairs flagged by the background scoring job for the /admin/duplicates review queue. Status enum: pending / dismissed / merged. Unique-(portId, clientAId, clientBId) so the same pair can't surface twice. Empty until P2 lands the cron. - `migration_source_links` — idempotency ledger. Maps source-system rows (NocoDB Interest #624 → new client UUID) so re-running --apply against the same dry-run report skips already-imported entities. Both tables ship with the migration `0020_unusual_azazel.sql` — already applied to the local dev DB during this commit's preparation. Library ======= src/lib/dedup/nocodb-source.ts Read-only adapter for the legacy NocoDB v2 API. xc-token auth, auto-paginates until isLastPage, captures the table IDs from the 2026-05-03 audit. `fetchSnapshot()` pulls every relevant table in parallel into one in-memory object the transform layer consumes. src/lib/dedup/migration-transform.ts Pure function: NocoDB snapshot in, MigrationPlan out. Per row: - normalizes name / email / phone / country via the dedup library - parses the legacy DD-MM-YYYY / DD/MM/YYYY / ISO date formats - maps the 8-stage `Sales Process Level` enum to the new 9-stage pipelineStage - filters yacht-name placeholders ('TBC', 'Na', etc.) - merges Internal Notes + Extra Comments + Berth Size Desired into a single notes blob Then runs `findClientMatches` pairwise (with blocking) and union-finds clusters of rows whose score crosses the auto-link threshold (90). Lower-scoring pairs (50–89) become 'needs review'. Each cluster's "lead" row is picked by completeness score with recency tie-break. src/lib/dedup/migration-report.ts Writes three artifacts to .migration/<timestamp>/: - report.csv — one row per planned op, RFC-4180 escaped - summary.md — human-skimmable overview - plan.json — full structured plan for the --apply phase CSV cells with comma / quote / newline are quoted; internal quotes are doubled. No external CSV dep. src/lib/dedup/phone-parse.ts Script-safe wrapper around libphonenumber-js's `core` entry that loads `metadata.min.json` directly. The default `index.cjs.js` bundled by libphonenumber hits a metadata-shape interop bug under Node 25 + tsx (`{ default }` wrapping); core+JSON sidesteps it. The dedup `normalizePhone` and `find-matches` both use this wrapper now so the same code path runs in vitest, Next.js, and the migration CLI without surprises. src/lib/dedup/normalize.ts Tightened country resolution: added Caribbean short-form aliases ('antigua' → AG, 'st kitts' → KN, etc.) and a city map covering the US locations seen in the NocoDB dump (Boston, Tampa, Fort Lauderdale, Port Jefferson, Nantucket). Also relaxed phone parsing to drop the `isValid()` strict check — the libphonenumber min build rejects many real NANP-territory numbers, and dedup only needs a canonical E.164 to compare. CLI === scripts/migrate-from-nocodb.ts pnpm tsx scripts/migrate-from-nocodb.ts --dry-run → Pulls the live NocoDB base (NOCODB_URL + NOCODB_TOKEN env vars), runs the transform, writes report. No DB writes. pnpm tsx scripts/migrate-from-nocodb.ts --apply --report .migration/<dir>/ → Stubbed; exits with `not yet implemented` and a pointer to the design doc. Apply phase ships in a follow-up. Tests ===== tests/unit/dedup/migration-transform.test.ts (7 cases) Fixture-based regression. A frozen 12-row NocoDB snapshot covers every duplicate pattern in the design (§1.2). The test asserts: - 12 input rows → 7 unique clients (cluster math is right) - Patterns A / B / C / E auto-link - Pattern F (Etiennette Clamouze) does NOT auto-link - Every interest preserved as its own row even when clients merge - 8-stage → 9-stage enum mapping is correct per spec - Multi-yacht merge (Constanzo CALYPSO + Costanzo GEMINI under one client) — the design's signature win - Output is deterministic (run twice, identical) Validation against real data ============================ Ran `pnpm tsx scripts/migrate-from-nocodb.ts --dry-run` against the live NocoDB. Result on 252 Interests rows: - 237 clients (15 merged into 13 clusters) - 252 interests (one per source row) - 406 contacts, 52 addresses - 13 auto-linked clusters (every confirmed cluster from §1.2 audit) - 3 pairs flagged for review (Camazou, Zasso, one new) - 1 phone placeholder flagged Total dedup test count: 57 (50 from P1 + 7 fixture tests). Lint: clean. Tsc: clean for new files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
145 lines
5.3 KiB
TypeScript
145 lines
5.3 KiB
TypeScript
/**
|
|
* One-shot migration: legacy NocoDB Interests → new client/interest split.
|
|
*
|
|
* Usage:
|
|
*
|
|
* pnpm tsx scripts/migrate-from-nocodb.ts --dry-run
|
|
* Pulls the live NocoDB base, runs the transform + dedup pipeline,
|
|
* writes a report to .migration/<timestamp>/. NO database writes.
|
|
*
|
|
* pnpm tsx scripts/migrate-from-nocodb.ts --dry-run --port-slug harbor-royale
|
|
* Same, but tags the planned writes with the named port (matters for
|
|
* the apply phase — every client/interest belongs to one port).
|
|
*
|
|
* pnpm tsx scripts/migrate-from-nocodb.ts --apply --report .migration/<dir>/
|
|
* [Not yet implemented — apply phase comes in a follow-up PR.]
|
|
*
|
|
* Design reference: docs/superpowers/specs/2026-05-03-dedup-and-migration-design.md §9.
|
|
*/
|
|
|
|
import 'dotenv/config';
|
|
|
|
import path from 'node:path';
|
|
import { fileURLToPath } from 'node:url';
|
|
|
|
import { fetchSnapshot, loadNocoDbConfig } from '@/lib/dedup/nocodb-source';
|
|
import { transformSnapshot } from '@/lib/dedup/migration-transform';
|
|
import { resolveReportPaths, writeReport } from '@/lib/dedup/migration-report';
|
|
|
|
interface CliArgs {
|
|
dryRun: boolean;
|
|
apply: boolean;
|
|
portSlug: string | null;
|
|
reportDir: string | null;
|
|
}
|
|
|
|
function parseArgs(argv: string[]): CliArgs {
|
|
const args: CliArgs = {
|
|
dryRun: false,
|
|
apply: false,
|
|
portSlug: null,
|
|
reportDir: null,
|
|
};
|
|
for (let i = 0; i < argv.length; i += 1) {
|
|
const a = argv[i]!;
|
|
if (a === '--dry-run') args.dryRun = true;
|
|
else if (a === '--apply') args.apply = true;
|
|
else if (a === '--port-slug') args.portSlug = argv[++i] ?? null;
|
|
else if (a === '--report') args.reportDir = argv[++i] ?? null;
|
|
else if (a === '-h' || a === '--help') {
|
|
printHelp();
|
|
process.exit(0);
|
|
} else {
|
|
console.error(`Unknown argument: ${a}`);
|
|
printHelp();
|
|
process.exit(1);
|
|
}
|
|
}
|
|
return args;
|
|
}
|
|
|
|
function printHelp(): void {
|
|
console.log(`Usage:
|
|
pnpm tsx scripts/migrate-from-nocodb.ts --dry-run [--port-slug <slug>]
|
|
Pulls NocoDB → transforms → writes report to .migration/<timestamp>/.
|
|
No database writes.
|
|
|
|
pnpm tsx scripts/migrate-from-nocodb.ts --apply --report .migration/<dir>/
|
|
Apply phase. (Not yet implemented.)
|
|
|
|
Flags:
|
|
--dry-run Read NocoDB, write report only.
|
|
--apply Actually write to the new DB. (Not yet supported.)
|
|
--port-slug <slug> Port slug to attach to all imported entities.
|
|
Defaults to the first available port if omitted.
|
|
--report <dir> Path to a previously-generated report dir
|
|
(only used by --apply).
|
|
-h, --help Show this help.
|
|
`);
|
|
}
|
|
|
|
async function main(): Promise<void> {
|
|
const args = parseArgs(process.argv.slice(2));
|
|
|
|
if (!args.dryRun && !args.apply) {
|
|
console.error('Must specify --dry-run or --apply');
|
|
printHelp();
|
|
process.exit(1);
|
|
}
|
|
|
|
if (args.apply) {
|
|
console.error('--apply is not yet implemented in this version. P3 ships dry-run first.');
|
|
console.error('See docs/superpowers/specs/2026-05-03-dedup-and-migration-design.md §9.2.');
|
|
process.exit(2);
|
|
}
|
|
|
|
// ── Dry-run path ───────────────────────────────────────────────────────────
|
|
|
|
console.log('[migrate] Loading NocoDB config…');
|
|
const config = loadNocoDbConfig();
|
|
console.log(`[migrate] Source: ${config.url}`);
|
|
|
|
console.log('[migrate] Fetching snapshot from NocoDB…');
|
|
const start = Date.now();
|
|
const snapshot = await fetchSnapshot(config);
|
|
const elapsed = ((Date.now() - start) / 1000).toFixed(1);
|
|
console.log(
|
|
`[migrate] Snapshot fetched in ${elapsed}s — ${snapshot.interests.length} interests, ${snapshot.residentialInterests.length} residential, ${snapshot.berths.length} berths.`,
|
|
);
|
|
|
|
console.log('[migrate] Running transform + dedup pipeline…');
|
|
const plan = transformSnapshot(snapshot);
|
|
|
|
// Resolve output paths relative to the worktree root (the script itself
|
|
// lives in scripts/; we want the .migration dir at the repo root).
|
|
const scriptDir = path.dirname(fileURLToPath(import.meta.url));
|
|
const repoRoot = path.resolve(scriptDir, '..');
|
|
const generatedAt = new Date().toISOString();
|
|
const paths = resolveReportPaths(repoRoot);
|
|
|
|
console.log(`[migrate] Writing report to ${paths.rootDir}…`);
|
|
await writeReport(paths, plan, generatedAt);
|
|
|
|
// ── Console summary ──────────────────────────────────────────────────────
|
|
const s = plan.stats;
|
|
console.log('');
|
|
console.log('=== Migration Plan Summary ===');
|
|
console.log(
|
|
` Input: ${s.inputInterestRows} interests, ${s.inputResidentialRows} residential interests`,
|
|
);
|
|
console.log(` Output: ${s.outputClients} clients, ${s.outputInterests} interests`);
|
|
console.log(` ${s.outputContacts} contacts, ${s.outputAddresses} addresses`);
|
|
console.log(
|
|
` Dedup: ${s.autoLinkedClusters} auto-linked clusters, ${s.needsReviewPairs} pairs flagged for review`,
|
|
);
|
|
console.log(` Quality: ${s.flaggedRows} rows flagged (see report.csv)`);
|
|
console.log('');
|
|
console.log(` Full report: ${paths.summaryPath}`);
|
|
console.log('');
|
|
}
|
|
|
|
main().catch((err) => {
|
|
console.error('[migrate] Fatal error:', err);
|
|
process.exit(1);
|
|
});
|