feat(dedup): NocoDB migration script + tables (P3 dry-run)
Lands the one-shot migration pipeline from the legacy NocoDB Interests base into the new client/interest schema. Dry-run mode is fully operational: pulls the live snapshot, runs the dedup library, and writes a CSV + Markdown report under .migration/<timestamp>/. The --apply phase is stubbed for a follow-up PR per the design's P3 implementation sequence. Schema additions ================ - `client_merge_candidates` — pairs flagged by the background scoring job for the /admin/duplicates review queue. Status enum: pending / dismissed / merged. Unique-(portId, clientAId, clientBId) so the same pair can't surface twice. Empty until P2 lands the cron. - `migration_source_links` — idempotency ledger. Maps source-system rows (NocoDB Interest #624 → new client UUID) so re-running --apply against the same dry-run report skips already-imported entities. Both tables ship with the migration `0020_unusual_azazel.sql` — already applied to the local dev DB during this commit's preparation. Library ======= src/lib/dedup/nocodb-source.ts Read-only adapter for the legacy NocoDB v2 API. xc-token auth, auto-paginates until isLastPage, captures the table IDs from the 2026-05-03 audit. `fetchSnapshot()` pulls every relevant table in parallel into one in-memory object the transform layer consumes. src/lib/dedup/migration-transform.ts Pure function: NocoDB snapshot in, MigrationPlan out. Per row: - normalizes name / email / phone / country via the dedup library - parses the legacy DD-MM-YYYY / DD/MM/YYYY / ISO date formats - maps the 8-stage `Sales Process Level` enum to the new 9-stage pipelineStage - filters yacht-name placeholders ('TBC', 'Na', etc.) - merges Internal Notes + Extra Comments + Berth Size Desired into a single notes blob Then runs `findClientMatches` pairwise (with blocking) and union-finds clusters of rows whose score crosses the auto-link threshold (90). Lower-scoring pairs (50–89) become 'needs review'. Each cluster's "lead" row is picked by completeness score with recency tie-break. src/lib/dedup/migration-report.ts Writes three artifacts to .migration/<timestamp>/: - report.csv — one row per planned op, RFC-4180 escaped - summary.md — human-skimmable overview - plan.json — full structured plan for the --apply phase CSV cells with comma / quote / newline are quoted; internal quotes are doubled. No external CSV dep. src/lib/dedup/phone-parse.ts Script-safe wrapper around libphonenumber-js's `core` entry that loads `metadata.min.json` directly. The default `index.cjs.js` bundled by libphonenumber hits a metadata-shape interop bug under Node 25 + tsx (`{ default }` wrapping); core+JSON sidesteps it. The dedup `normalizePhone` and `find-matches` both use this wrapper now so the same code path runs in vitest, Next.js, and the migration CLI without surprises. src/lib/dedup/normalize.ts Tightened country resolution: added Caribbean short-form aliases ('antigua' → AG, 'st kitts' → KN, etc.) and a city map covering the US locations seen in the NocoDB dump (Boston, Tampa, Fort Lauderdale, Port Jefferson, Nantucket). Also relaxed phone parsing to drop the `isValid()` strict check — the libphonenumber min build rejects many real NANP-territory numbers, and dedup only needs a canonical E.164 to compare. CLI === scripts/migrate-from-nocodb.ts pnpm tsx scripts/migrate-from-nocodb.ts --dry-run → Pulls the live NocoDB base (NOCODB_URL + NOCODB_TOKEN env vars), runs the transform, writes report. No DB writes. pnpm tsx scripts/migrate-from-nocodb.ts --apply --report .migration/<dir>/ → Stubbed; exits with `not yet implemented` and a pointer to the design doc. Apply phase ships in a follow-up. Tests ===== tests/unit/dedup/migration-transform.test.ts (7 cases) Fixture-based regression. A frozen 12-row NocoDB snapshot covers every duplicate pattern in the design (§1.2). The test asserts: - 12 input rows → 7 unique clients (cluster math is right) - Patterns A / B / C / E auto-link - Pattern F (Etiennette Clamouze) does NOT auto-link - Every interest preserved as its own row even when clients merge - 8-stage → 9-stage enum mapping is correct per spec - Multi-yacht merge (Constanzo CALYPSO + Costanzo GEMINI under one client) — the design's signature win - Output is deterministic (run twice, identical) Validation against real data ============================ Ran `pnpm tsx scripts/migrate-from-nocodb.ts --dry-run` against the live NocoDB. Result on 252 Interests rows: - 237 clients (15 merged into 13 clusters) - 252 interests (one per source row) - 406 contacts, 52 addresses - 13 auto-linked clusters (every confirmed cluster from §1.2 audit) - 3 pairs flagged for review (Camazou, Zasso, one new) - 1 phone placeholder flagged Total dedup test count: 57 (50 from P1 + 7 fixture tests). Lint: clean. Tsc: clean for new files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -12,7 +12,7 @@
|
||||
import { z } from 'zod';
|
||||
|
||||
import { ALL_COUNTRY_CODES, getCountryName, type CountryCode } from '@/lib/i18n/countries';
|
||||
import { parsePhone } from '@/lib/i18n/phone';
|
||||
import { parsePhoneScriptSafe as parsePhone } from './phone-parse';
|
||||
|
||||
// ─── Names ──────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -228,10 +228,18 @@ export function normalizePhone(
|
||||
|
||||
// 7. Parse via the existing i18n helper (libphonenumber-js under the hood).
|
||||
const parsed = parsePhone(cleaned, defaultCountry);
|
||||
if (!parsed.e164 || !parsed.isValid) {
|
||||
if (!parsed.e164) {
|
||||
// Couldn't even produce a canonical form — genuinely garbage.
|
||||
return { e164: null, country: null, display: null, flagged: 'unparseable' };
|
||||
}
|
||||
|
||||
// Note: we deliberately don't gate on `parsed.isValid`. The
|
||||
// libphonenumber-js `min` build returns isValid=false for many real
|
||||
// numbers (NANP territories share +1; some country metadata is
|
||||
// truncated). For dedup we only need a canonical E.164 string to
|
||||
// compare; strict validity is the form layer's problem, not ours.
|
||||
// If a string-only test (e.g. \"abc-not-a-phone\") gets here, parse
|
||||
// returns null e164 anyway and the branch above handles it.
|
||||
return {
|
||||
e164: parsed.e164,
|
||||
country: parsed.country,
|
||||
@@ -261,6 +269,13 @@ const COUNTRY_ALIASES: Record<string, CountryCode> = {
|
||||
'st barth': 'BL',
|
||||
'st barths': 'BL',
|
||||
'st barthelemy': 'BL',
|
||||
// Caribbean short-forms whose canonical Intl names are awkward
|
||||
// ("Antigua and Barbuda", "Saint Vincent and the Grenadines", etc.).
|
||||
antigua: 'AG',
|
||||
barbuda: 'AG',
|
||||
'st kitts': 'KN',
|
||||
'saint kitts': 'KN',
|
||||
nevis: 'KN',
|
||||
};
|
||||
|
||||
/**
|
||||
@@ -275,6 +290,20 @@ const CITY_TO_COUNTRY: Record<string, CountryCode> = {
|
||||
'kansas city': 'US',
|
||||
'sag harbor': 'US',
|
||||
'new york': 'US',
|
||||
// Cities that came out unresolved from the 2026-05-03 NocoDB dry-run.
|
||||
// Using lowercase (post-normalize keys).
|
||||
boston: 'US',
|
||||
tampa: 'US',
|
||||
'fort lauderdale': 'US',
|
||||
'port jefferson': 'US',
|
||||
nantucket: 'US',
|
||||
// US state abbreviations that often appear standalone or as suffix:
|
||||
' fl': 'US',
|
||||
' ma': 'US',
|
||||
' ny': 'US',
|
||||
' tx': 'US',
|
||||
' ca': 'US',
|
||||
// International
|
||||
london: 'GB',
|
||||
paris: 'FR',
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user