Files
pn-new-crm/src/lib/db/schema/clients.ts

278 lines
11 KiB
TypeScript
Raw Normal View History

import {
pgTable,
text,
boolean,
feat(dedup): NocoDB migration script + tables (P3 dry-run) Lands the one-shot migration pipeline from the legacy NocoDB Interests base into the new client/interest schema. Dry-run mode is fully operational: pulls the live snapshot, runs the dedup library, and writes a CSV + Markdown report under .migration/<timestamp>/. The --apply phase is stubbed for a follow-up PR per the design's P3 implementation sequence. Schema additions ================ - `client_merge_candidates` — pairs flagged by the background scoring job for the /admin/duplicates review queue. Status enum: pending / dismissed / merged. Unique-(portId, clientAId, clientBId) so the same pair can't surface twice. Empty until P2 lands the cron. - `migration_source_links` — idempotency ledger. Maps source-system rows (NocoDB Interest #624 → new client UUID) so re-running --apply against the same dry-run report skips already-imported entities. Both tables ship with the migration `0020_unusual_azazel.sql` — already applied to the local dev DB during this commit's preparation. Library ======= src/lib/dedup/nocodb-source.ts Read-only adapter for the legacy NocoDB v2 API. xc-token auth, auto-paginates until isLastPage, captures the table IDs from the 2026-05-03 audit. `fetchSnapshot()` pulls every relevant table in parallel into one in-memory object the transform layer consumes. src/lib/dedup/migration-transform.ts Pure function: NocoDB snapshot in, MigrationPlan out. Per row: - normalizes name / email / phone / country via the dedup library - parses the legacy DD-MM-YYYY / DD/MM/YYYY / ISO date formats - maps the 8-stage `Sales Process Level` enum to the new 9-stage pipelineStage - filters yacht-name placeholders ('TBC', 'Na', etc.) - merges Internal Notes + Extra Comments + Berth Size Desired into a single notes blob Then runs `findClientMatches` pairwise (with blocking) and union-finds clusters of rows whose score crosses the auto-link threshold (90). Lower-scoring pairs (50–89) become 'needs review'. Each cluster's "lead" row is picked by completeness score with recency tie-break. src/lib/dedup/migration-report.ts Writes three artifacts to .migration/<timestamp>/: - report.csv — one row per planned op, RFC-4180 escaped - summary.md — human-skimmable overview - plan.json — full structured plan for the --apply phase CSV cells with comma / quote / newline are quoted; internal quotes are doubled. No external CSV dep. src/lib/dedup/phone-parse.ts Script-safe wrapper around libphonenumber-js's `core` entry that loads `metadata.min.json` directly. The default `index.cjs.js` bundled by libphonenumber hits a metadata-shape interop bug under Node 25 + tsx (`{ default }` wrapping); core+JSON sidesteps it. The dedup `normalizePhone` and `find-matches` both use this wrapper now so the same code path runs in vitest, Next.js, and the migration CLI without surprises. src/lib/dedup/normalize.ts Tightened country resolution: added Caribbean short-form aliases ('antigua' → AG, 'st kitts' → KN, etc.) and a city map covering the US locations seen in the NocoDB dump (Boston, Tampa, Fort Lauderdale, Port Jefferson, Nantucket). Also relaxed phone parsing to drop the `isValid()` strict check — the libphonenumber min build rejects many real NANP-territory numbers, and dedup only needs a canonical E.164 to compare. CLI === scripts/migrate-from-nocodb.ts pnpm tsx scripts/migrate-from-nocodb.ts --dry-run → Pulls the live NocoDB base (NOCODB_URL + NOCODB_TOKEN env vars), runs the transform, writes report. No DB writes. pnpm tsx scripts/migrate-from-nocodb.ts --apply --report .migration/<dir>/ → Stubbed; exits with `not yet implemented` and a pointer to the design doc. Apply phase ships in a follow-up. Tests ===== tests/unit/dedup/migration-transform.test.ts (7 cases) Fixture-based regression. A frozen 12-row NocoDB snapshot covers every duplicate pattern in the design (§1.2). The test asserts: - 12 input rows → 7 unique clients (cluster math is right) - Patterns A / B / C / E auto-link - Pattern F (Etiennette Clamouze) does NOT auto-link - Every interest preserved as its own row even when clients merge - 8-stage → 9-stage enum mapping is correct per spec - Multi-yacht merge (Constanzo CALYPSO + Costanzo GEMINI under one client) — the design's signature win - Output is deterministic (run twice, identical) Validation against real data ============================ Ran `pnpm tsx scripts/migrate-from-nocodb.ts --dry-run` against the live NocoDB. Result on 252 Interests rows: - 237 clients (15 merged into 13 clusters) - 252 interests (one per source row) - 406 contacts, 52 addresses - 13 auto-linked clusters (every confirmed cluster from §1.2 audit) - 3 pairs flagged for review (Camazou, Zasso, one new) - 1 phone placeholder flagged Total dedup test count: 57 (50 from P1 + 7 fixture tests). Lint: clean. Tsc: clean for new files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:50:01 +02:00
integer,
timestamp,
jsonb,
index,
uniqueIndex,
primaryKey,
} from 'drizzle-orm/pg-core';
import { sql } from 'drizzle-orm';
import { ports } from './ports';
export const clients = pgTable(
'clients',
{
id: text('id')
.primaryKey()
.$defaultFn(() => crypto.randomUUID()),
portId: text('port_id')
.notNull()
.references(() => ports.id),
fullName: text('full_name').notNull(),
/** ISO-3166-1 alpha-2 nationality code. */
feat(i18n): country/phone/timezone/subdivision primitives + form wiring Cross-cutting i18n polish for forms across the marina + residential + company domains. Introduces a single source of truth for country/phone/timezone/ subdivision data and replaces every nationality-as-free-text and timezone- as-string Input with a dedicated combobox. PR1 Countries — ALL_COUNTRY_CODES (~250 ISO-3166-1 alpha-2), Intl.DisplayNames for localized labels, detectDefaultCountry() with navigator-region fallback to US, CountryCombobox with regional-indicator flag glyphs + compact mode for inline use. PR2 Phone — libphonenumber-js wrapper (parsePhone / formatAsYouType / callingCodeFor), PhoneInput with flag dropdown + national-format AsYouType + paste-detect that flips the country dropdown for pasted international strings. PR3 Timezones — country->IANA map (250 entries, multi-zone for AU/BR/CA/CD/ ID/KZ/MN/MX/RU/US), formatTimezoneLabel ("Europe/London (UTC+1)"), TimezoneCombobox with Suggested/All grouping driven by countryHint. PR4 Subdivisions — wraps the iso-3166-2 npm package (~5000 ISO 3166-2 codes for every country), per-country cache, SubdivisionCombobox with "Pick a country first" / "No regions available" empty states. PR5 Schema deltas (migration 0015) — clients.nationality_iso, clientContacts {value_e164, value_country}, clientAddresses {country_iso, subdivision_iso}, residentialClients {phone_e164, phone_country, nationality_iso, timezone, place_of_residence_country_iso, subdivision_iso}, companies {incorporation_ country_iso, incorporation_subdivision_iso}, companyAddresses {country_iso, subdivision_iso}. Plus shared zod validators (validators/i18n.ts) used by every entity validator + route handler. PR6 ClientForm + ClientDetail — CountryCombobox replaces nationality Input, TimezoneCombobox replaces timezone Input (driven by nationalityIso hint), PhoneInput conditionally rendered for phone/whatsapp contacts. Inline editors (InlineCountryField / InlineTimezoneField / InlinePhoneField) for the detail-page overview rows + ContactsEditor. PR7 Residential client form + detail — phone -> PhoneInput, nationality/ timezone/place-of-residence-country/subdivision rows in both create sheet and inline-editable detail view. Subdivision wipes when country flips since codes are country-scoped. PR8 Company form + detail — incorporation country -> CountryCombobox, incorporation region -> SubdivisionCombobox in both modes. PR9 Public inquiry endpoint — accepts pre-normalized phoneE164/phoneCountry and i18n fields from newer website builds, server-side parsePhone() fallback for legacy raw-international submissions. Old Nuxt builds keep working unchanged. Tests: 4 unit suites for the primitives (25 tests), 1 integration spec for the public phone-normalization path (3 tests), 1 smoke spec asserting the combobox triggers render in all three create sheets. Test totals: vitest 713 -> 741 (+28). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 18:13:08 +02:00
nationalityIso: text('nationality_iso'),
preferredContactMethod: text('preferred_contact_method'), // email, phone, whatsapp
preferredLanguage: text('preferred_language'),
feat(i18n): country/phone/timezone/subdivision primitives + form wiring Cross-cutting i18n polish for forms across the marina + residential + company domains. Introduces a single source of truth for country/phone/timezone/ subdivision data and replaces every nationality-as-free-text and timezone- as-string Input with a dedicated combobox. PR1 Countries — ALL_COUNTRY_CODES (~250 ISO-3166-1 alpha-2), Intl.DisplayNames for localized labels, detectDefaultCountry() with navigator-region fallback to US, CountryCombobox with regional-indicator flag glyphs + compact mode for inline use. PR2 Phone — libphonenumber-js wrapper (parsePhone / formatAsYouType / callingCodeFor), PhoneInput with flag dropdown + national-format AsYouType + paste-detect that flips the country dropdown for pasted international strings. PR3 Timezones — country->IANA map (250 entries, multi-zone for AU/BR/CA/CD/ ID/KZ/MN/MX/RU/US), formatTimezoneLabel ("Europe/London (UTC+1)"), TimezoneCombobox with Suggested/All grouping driven by countryHint. PR4 Subdivisions — wraps the iso-3166-2 npm package (~5000 ISO 3166-2 codes for every country), per-country cache, SubdivisionCombobox with "Pick a country first" / "No regions available" empty states. PR5 Schema deltas (migration 0015) — clients.nationality_iso, clientContacts {value_e164, value_country}, clientAddresses {country_iso, subdivision_iso}, residentialClients {phone_e164, phone_country, nationality_iso, timezone, place_of_residence_country_iso, subdivision_iso}, companies {incorporation_ country_iso, incorporation_subdivision_iso}, companyAddresses {country_iso, subdivision_iso}. Plus shared zod validators (validators/i18n.ts) used by every entity validator + route handler. PR6 ClientForm + ClientDetail — CountryCombobox replaces nationality Input, TimezoneCombobox replaces timezone Input (driven by nationalityIso hint), PhoneInput conditionally rendered for phone/whatsapp contacts. Inline editors (InlineCountryField / InlineTimezoneField / InlinePhoneField) for the detail-page overview rows + ContactsEditor. PR7 Residential client form + detail — phone -> PhoneInput, nationality/ timezone/place-of-residence-country/subdivision rows in both create sheet and inline-editable detail view. Subdivision wipes when country flips since codes are country-scoped. PR8 Company form + detail — incorporation country -> CountryCombobox, incorporation region -> SubdivisionCombobox in both modes. PR9 Public inquiry endpoint — accepts pre-normalized phoneE164/phoneCountry and i18n fields from newer website builds, server-side parsePhone() fallback for legacy raw-international submissions. Old Nuxt builds keep working unchanged. Tests: 4 unit suites for the primitives (25 tests), 1 integration spec for the public phone-normalization path (3 tests), 1 smoke spec asserting the combobox triggers render in all three create sheets. Test totals: vitest 713 -> 741 (+28). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 18:13:08 +02:00
/** IANA timezone, e.g. 'Europe/Warsaw'. Validated client + server. */
timezone: text('timezone'),
source: text('source'), // website, manual, referral, broker
sourceDetails: text('source_details'),
feat(schema): berths.archived_at + clients.source_inquiry_id + email_bounces Step 3 schema additions per PRE-DEPLOY-PLAN § 1.4. berths.archived_at (+ archived_by, archive_reason) — soft-delete column so retired moorings can be hidden from the public feed and admin lists without losing historical interest joins. Partial index `idx_berths_active` on (port_id) WHERE archived_at IS NULL keeps the active-only list path fast. Already wired: - /api/public/berths and /api/public/berths/[mooringNumber] now filter out archived rows. - berths.service.listBerths defaults to active-only with an ?includeArchived=true escape hatch for the archive bin. clients.source_inquiry_id — text column with ON DELETE SET NULL FK to website_submissions(id). Preserves the linkage from a website inquiry to the client that came out of the "Convert to client" triage flow (P-4.5). Drives the conversion-funnel-by-source chart (Step 6). The Drizzle column ships without `.references()` to avoid the cross-file circular import; the FK lives in the migration SQL. email_bounces table — bounce-monitoring storage. The DSN poller worker (forthcoming, depends on this table existing) writes one row per parsed bounce; consumers join via (original_send_type, original_send_id). Three secondary indexes cover the expected access patterns (port + recent bounces; lookup by bounced address; lookup by original send). Schema additions plus the migration SQL are ready for `pnpm db:push` (or the migration runner once its journal is backfilled — separate concern, journal currently stops at 0042 despite migrations through 0065 existing on disk). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 15:33:20 +02:00
/** When this client came out of a "Convert inquiry to client" triage
* step, points back at the originating `website_submissions` row.
* Drives the conversion-funnel-by-source chart. Migration 0065
* installs the FK with ON DELETE SET NULL Drizzle doesn't reflect
* it here to avoid the cross-file circular import. */
sourceInquiryId: text('source_inquiry_id'),
archivedAt: timestamp('archived_at', { withTimezone: true }),
feat(client-archive): smart-archive backend foundation (dossier + archive + restore) The first slice of the smart-archive project. Replaces the dumb DELETE client flow with a deliberate "look before you leap" pattern: - New columns on clients: archived_by, archive_reason, archive_metadata (jsonb capturing every decision made during archive, so restore can attempt reversal). Migration 0043. - client-archive-dossier.service builds a structured snapshot of "what's at stake" for a given client: pipeline interests, berths under offer (with next-in-line interests for the notification), yachts owned, active reservations, outstanding invoices, signed/in-flight Documenso envelopes, portal user, company memberships. Classifies the client as low-stakes or high-stakes based on pipeline stage (HIGH_STAKES_STAGES = deposit_10pct + later) so the bulk wizard knows which clients to prompt individually. - client-archive.service.archiveClientWithDecisions takes the operator's decisions and applies them in a single transaction. Persists the decision log into archive_metadata for restore. Auto-handles portal user revocation + company membership end-dating; everything else is caller-driven. Surfaces external cleanups (Documenso void) for the caller to queue. - client-restore.service.getRestoreDossier classifies each persisted decision as autoReversible / reversibleWithPrompt / locked based on the current state of the world (berth still available? new owner has active interests on the yacht? etc). restoreClientWithSelections applies reversals + un-archives the client. - 4 API routes wire the services to HTTP. The existing /restore endpoint is upgraded to use the smart restore but stays backwards-compatible: clients archived before this feature have no archive_metadata so the dossier returns empty, and a POST with no body just un-archives them — same as before. UI work + bulk variant + hard-delete + Documenso cleanup queueing land in follow-on commits. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 17:13:08 +02:00
/** Better-auth user id of the operator who archived this client. */
archivedBy: text('archived_by'),
/** Free-text reason captured at archive time. Required when archiving a
* client at deposit_10pct or later (compliance trail). Optional when
* archiving an early-stage lead. */
archiveReason: text('archive_reason'),
/** Per-decision metadata captured during smart-archive flow. Used by
* the restore wizard to attempt reversal. Shape:
* { decisions: Array<{ kind, refId, ...specifics }>, decidedAt, decidedBy }
* See src/lib/services/client-archive.service.ts for the canonical
* payload schema. */
archiveMetadata: jsonb('archive_metadata'),
feat(dedup): runtime surfaces — merge service, at-create suggestion, admin queue (P2) Adds the live dedup pipeline on top of the P1 library + P3 migration script. The new `client/interest` model now actively prevents duplicate client records at creation time and gives admins a queue to triage the borderline pairs the at-create check missed. Three layers, per design §7: Layer 1 — At-create suggestion ============================== `GET /api/v1/clients/match-candidates` Accepts free-text email / phone / name from the in-flight client form, normalizes them via the dedup library, and returns scored matches against the port's live client pool. Filters out low-confidence noise (the background scoring queue picks those up separately). Strict port scoping; never leaks across tenants. `<DedupSuggestionPanel>` (`src/components/clients/dedup-suggestion-panel.tsx`) Debounced React Query hook. Renders nothing for short inputs or no useful match. On a high-confidence match it interrupts visually with an amber-tinted card and a "Use this client" primary button. Medium confidence falls back to a softer "possible match — check before creating" treatment. `<ClientForm>` Renders the panel above the form (create path only — skipped on edit). New `onUseExistingClient` callback fires when the user picks the existing client; the form closes and the parent decides what to do (typically: navigate to that client's detail page or open the create-interest dialog pre-filled). Layer 2 — Merge service ======================= `mergeClients` (`src/lib/services/client-merge.service.ts`) The atomic merge primitive that everything else calls. Single transaction. Per §6 of the design: - Locks both rows (FOR UPDATE) so concurrent merges of the same loser fail with a clear error rather than racing. - Snapshots the full loser state (contacts / addresses / notes / tags / interest+reservation IDs / relationship rows) into the `client_merge_log.merge_details` JSONB column for the eventual undo flow. - Reattaches every loser-side row to the winner: interests, reservations, contacts (skipping duplicates by `(channel, value)`), addresses, notes, tags (deduped), relationships. - Optional `fieldChoices` — per-scalar overrides letting the user keep the loser's value for fullName / nationality / preferences / timezone / source. - Marks the loser archived with `mergedIntoClientId` set (a redirect pointer for stragglers; never hard-deleted within the undo window). - Resolves any matching `client_merge_candidates` row to status='merged'. - Writes audit log entry. Schema additions: - `clients.merged_into_client_id` (nullable text, indexed) — the redirect pointer set on archive. Tests: 6 cases against a real DB — happy path moves rows + writes log; self-merge / cross-port / already-merged refused; duplicate-contact deduped on reattach; fieldChoices copies loser values to winner. Layer 3 — Admin review queue ============================ `GET /api/v1/admin/duplicates` Pending merge candidates (status='pending') for the current port, with both client summaries hydrated for side-by-side rendering. Skips pairs where one side is already archived/merged. `POST /api/v1/admin/duplicates/[id]/merge` Confirms a candidate. Body picks the winner; the other side becomes the loser. Calls into `mergeClients` — the only path that writes `client_merge_log`. `POST /api/v1/admin/duplicates/[id]/dismiss` Marks the candidate dismissed. Future scoring runs skip the same pair until a score change recreates the row. `<DuplicatesReviewQueue>` (`/admin/duplicates`) Side-by-side card UI for each pending pair. Click a card to pick the winner; the other side is automatically the loser. Toolbar: "Merge into selected" + "Dismiss". No per-field merge editor in this PR — that's a future polish; the simple "pick the better row" flow handles ~80% of cases. Test coverage ============= 11 new integration tests (76 added in this branch total): - 6 mergeClients (atomicity, refusal cases, contact dedup, fieldChoices) - 5 match-candidates API (shape, port scoping, confidence tiers, Pattern F false-positive guard) Full vitest: 926/926 passing (was 858 before the dedup branch). Lint: clean. tsc: clean for new files (only pre-existing errors in unrelated `tests/integration/` files remain, same as before this PR). Out of scope, deferred ====================== - Background scoring cron that populates `client_merge_candidates` (the queue is empty until this lands; manual seeding works for now via the at-create flow). - Side-by-side per-field merge editor with checkboxes (the simple "pick the winner" UX shipped here covers ~80% of real cases). - Admin settings UI for tuning the dedup thresholds. Defaults from the design (90 / 50) are baked in for now. - `unmergeClients` (the snapshot is captured in client_merge_log; the undo endpoint just hasn't been wired yet). These are all natural follow-up PRs that don't block shipping the runtime UX. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:59:04 +02:00
/** When this client was merged into another (the "loser" of a dedup
* merge), this points at the surviving client. Used by the
* /admin/duplicates review queue to redirect any stragglers, and by
fix(audit-tier-1): timeouts, lifecycle, per-port Documenso, FK constraints Closes the second wave of HIGH-priority audit findings: * fetchWithTimeout helper (new src/lib/fetch-with-timeout.ts) wraps Documenso, OCR, currency, Umami, IMAP, etc. — a hung upstream can no longer pin a worker concurrency slot indefinitely. OpenAI client passes timeout: 30_000. ImapFlow gets socket / greeting / connection timeouts. * SIGTERM / SIGINT handler in src/server.ts drains in-flight HTTP, closes Socket.io, and disconnects Redis before exit; compose stop_grace_period bumped to 30s. Adds closeSocketServer() helper. * env.ts gains zod-validated PORT and MULTI_NODE_DEPLOYMENT, and filesystem.ts now reads from env (a typo can no longer silently disable the multi-node guard). * Per-port Documenso template + recipient IDs land in system_settings with env fallback (PortDocumensoConfig now exposes eoiTemplateId, clientRecipientId, developerRecipientId, approvalRecipientId). document-templates.ts uses the per-port config and threads portId into documensoGenerateFromTemplate(). * Migration 0042 wires the eleven HIGH-tier missing FK constraints (documents/files/interests/reminders/berth_waiting_list/ form_submissions) plus polymorphic CHECK round 2 (yacht_ownership_history.owner_type, document_sends.document_kind), invoices.billing_entity_id NOT EMPTY, and clients.merged_into self-FK. Drizzle schema columns updated to .references(...) where possible so the misleading "FK wired in relations.ts" comments are gone. Test status: 1168/1168 vitest, tsc clean. Refs: docs/audit-comprehensive-2026-05-05.md HIGH §§5,6,7,8,9,10 + MED §§14,15,16,18. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 19:52:58 +02:00
* the unmerge flow to restore. Null for live clients. The Postgres
* self-FK is installed via migration 0042; Drizzle's table builder
* doesn't accept self-references in the column factory so the
* constraint isn't reflected here in `.references(...)`. */
feat(dedup): runtime surfaces — merge service, at-create suggestion, admin queue (P2) Adds the live dedup pipeline on top of the P1 library + P3 migration script. The new `client/interest` model now actively prevents duplicate client records at creation time and gives admins a queue to triage the borderline pairs the at-create check missed. Three layers, per design §7: Layer 1 — At-create suggestion ============================== `GET /api/v1/clients/match-candidates` Accepts free-text email / phone / name from the in-flight client form, normalizes them via the dedup library, and returns scored matches against the port's live client pool. Filters out low-confidence noise (the background scoring queue picks those up separately). Strict port scoping; never leaks across tenants. `<DedupSuggestionPanel>` (`src/components/clients/dedup-suggestion-panel.tsx`) Debounced React Query hook. Renders nothing for short inputs or no useful match. On a high-confidence match it interrupts visually with an amber-tinted card and a "Use this client" primary button. Medium confidence falls back to a softer "possible match — check before creating" treatment. `<ClientForm>` Renders the panel above the form (create path only — skipped on edit). New `onUseExistingClient` callback fires when the user picks the existing client; the form closes and the parent decides what to do (typically: navigate to that client's detail page or open the create-interest dialog pre-filled). Layer 2 — Merge service ======================= `mergeClients` (`src/lib/services/client-merge.service.ts`) The atomic merge primitive that everything else calls. Single transaction. Per §6 of the design: - Locks both rows (FOR UPDATE) so concurrent merges of the same loser fail with a clear error rather than racing. - Snapshots the full loser state (contacts / addresses / notes / tags / interest+reservation IDs / relationship rows) into the `client_merge_log.merge_details` JSONB column for the eventual undo flow. - Reattaches every loser-side row to the winner: interests, reservations, contacts (skipping duplicates by `(channel, value)`), addresses, notes, tags (deduped), relationships. - Optional `fieldChoices` — per-scalar overrides letting the user keep the loser's value for fullName / nationality / preferences / timezone / source. - Marks the loser archived with `mergedIntoClientId` set (a redirect pointer for stragglers; never hard-deleted within the undo window). - Resolves any matching `client_merge_candidates` row to status='merged'. - Writes audit log entry. Schema additions: - `clients.merged_into_client_id` (nullable text, indexed) — the redirect pointer set on archive. Tests: 6 cases against a real DB — happy path moves rows + writes log; self-merge / cross-port / already-merged refused; duplicate-contact deduped on reattach; fieldChoices copies loser values to winner. Layer 3 — Admin review queue ============================ `GET /api/v1/admin/duplicates` Pending merge candidates (status='pending') for the current port, with both client summaries hydrated for side-by-side rendering. Skips pairs where one side is already archived/merged. `POST /api/v1/admin/duplicates/[id]/merge` Confirms a candidate. Body picks the winner; the other side becomes the loser. Calls into `mergeClients` — the only path that writes `client_merge_log`. `POST /api/v1/admin/duplicates/[id]/dismiss` Marks the candidate dismissed. Future scoring runs skip the same pair until a score change recreates the row. `<DuplicatesReviewQueue>` (`/admin/duplicates`) Side-by-side card UI for each pending pair. Click a card to pick the winner; the other side is automatically the loser. Toolbar: "Merge into selected" + "Dismiss". No per-field merge editor in this PR — that's a future polish; the simple "pick the better row" flow handles ~80% of cases. Test coverage ============= 11 new integration tests (76 added in this branch total): - 6 mergeClients (atomicity, refusal cases, contact dedup, fieldChoices) - 5 match-candidates API (shape, port scoping, confidence tiers, Pattern F false-positive guard) Full vitest: 926/926 passing (was 858 before the dedup branch). Lint: clean. tsc: clean for new files (only pre-existing errors in unrelated `tests/integration/` files remain, same as before this PR). Out of scope, deferred ====================== - Background scoring cron that populates `client_merge_candidates` (the queue is empty until this lands; manual seeding works for now via the at-create flow). - Side-by-side per-field merge editor with checkboxes (the simple "pick the winner" UX shipped here covers ~80% of real cases). - Admin settings UI for tuning the dedup thresholds. Defaults from the design (90 / 50) are baked in for now. - `unmergeClients` (the snapshot is captured in client_merge_log; the undo endpoint just hasn't been wired yet). These are all natural follow-up PRs that don't block shipping the runtime UX. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:59:04 +02:00
mergedIntoClientId: text('merged_into_client_id'),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(),
},
(table) => [
index('idx_clients_port').on(table.portId),
index('idx_clients_name').on(table.portId, table.fullName),
fix(audit): backlog sweep — partial archived indexes, custom-fields per-entity gate, polish Wave through the 2026-05-07 backlog of small/concrete audit-final-deferred items (deferring the Documenso Phases 2-7 build and items needing design decisions or live external instances). DB schema: - Migration 0046 converts 5 composite (port_id, archived_at) indexes to partial WHERE archived_at IS NULL — clients, interests, yachts, and both residential tables. Smaller, faster planner choice for the dominant list-query shape. Multi-tenant isolation: - document_sends now verifies recipient.interestId belongs to the port before landing on the audit row (the surrounding clientId check was already port-scoped; interestId pollution was the gap). Routes / API: - /api/v1/custom-fields/[entityId] requires entityType query param and gates on the matching resource permission (clients/interests/berths/ yachts/companies). Fixes the cross-resource gap where a user with clients.view could read company custom-field values. - Admin user list trash button wrapped in PermissionGate (edit was already gated; remove was not). Service polish: - berth-recommender accepts string-shaped JSONB booleans ('true'/'false') so admin UIs that wrap values as strings don't silently fall through to defaults. - expense-pdf renderReceiptHeader anchors all text positions to a captured baseY rather than reading mutating doc.y after rect+stroke. Headers no longer drift on the first receipt page after a soft page break. - berth-pdf apply: collect non-finite numeric coercion drops + warn-log them so partial silent drops are observable (was invisible because the no-fields-supplied check only fires when ALL drop). - Storage cache fingerprint comment documenting the encrypted-secret invariant + the explicit invalidation hook. UI polish: - invoice-detail typed: replaced two `any` casts with a proper InvoiceDetailData / LineItem / LinkedExpense interface set. - YachtForm now accepts initialOwner prop. Wired through: - client-yachts-tab passes { type: 'client', id: clientId } - interest-form passes { type: 'client', id: selectedClientId } - Interest-form yacht picker now includes company-owned yachts where the selected client is a member (fetches client.companies and feeds YachtPicker an array filter). Plus an inline "Add new" button that opens YachtForm pre-bound to the client. - YachtPicker accepts ownerFilter as single OR array for "match any" semantics. BACKLOG.md updated with what landed vs what's still deferred (and why each deferred item is genuinely larger than this push warrants). Tests: 1185/1185 vitest, tsc clean.
2026-05-07 21:45:42 +02:00
index('idx_clients_archived')
.on(table.portId)
.where(sql`${table.archivedAt} IS NULL`),
feat(i18n): country/phone/timezone/subdivision primitives + form wiring Cross-cutting i18n polish for forms across the marina + residential + company domains. Introduces a single source of truth for country/phone/timezone/ subdivision data and replaces every nationality-as-free-text and timezone- as-string Input with a dedicated combobox. PR1 Countries — ALL_COUNTRY_CODES (~250 ISO-3166-1 alpha-2), Intl.DisplayNames for localized labels, detectDefaultCountry() with navigator-region fallback to US, CountryCombobox with regional-indicator flag glyphs + compact mode for inline use. PR2 Phone — libphonenumber-js wrapper (parsePhone / formatAsYouType / callingCodeFor), PhoneInput with flag dropdown + national-format AsYouType + paste-detect that flips the country dropdown for pasted international strings. PR3 Timezones — country->IANA map (250 entries, multi-zone for AU/BR/CA/CD/ ID/KZ/MN/MX/RU/US), formatTimezoneLabel ("Europe/London (UTC+1)"), TimezoneCombobox with Suggested/All grouping driven by countryHint. PR4 Subdivisions — wraps the iso-3166-2 npm package (~5000 ISO 3166-2 codes for every country), per-country cache, SubdivisionCombobox with "Pick a country first" / "No regions available" empty states. PR5 Schema deltas (migration 0015) — clients.nationality_iso, clientContacts {value_e164, value_country}, clientAddresses {country_iso, subdivision_iso}, residentialClients {phone_e164, phone_country, nationality_iso, timezone, place_of_residence_country_iso, subdivision_iso}, companies {incorporation_ country_iso, incorporation_subdivision_iso}, companyAddresses {country_iso, subdivision_iso}. Plus shared zod validators (validators/i18n.ts) used by every entity validator + route handler. PR6 ClientForm + ClientDetail — CountryCombobox replaces nationality Input, TimezoneCombobox replaces timezone Input (driven by nationalityIso hint), PhoneInput conditionally rendered for phone/whatsapp contacts. Inline editors (InlineCountryField / InlineTimezoneField / InlinePhoneField) for the detail-page overview rows + ContactsEditor. PR7 Residential client form + detail — phone -> PhoneInput, nationality/ timezone/place-of-residence-country/subdivision rows in both create sheet and inline-editable detail view. Subdivision wipes when country flips since codes are country-scoped. PR8 Company form + detail — incorporation country -> CountryCombobox, incorporation region -> SubdivisionCombobox in both modes. PR9 Public inquiry endpoint — accepts pre-normalized phoneE164/phoneCountry and i18n fields from newer website builds, server-side parsePhone() fallback for legacy raw-international submissions. Old Nuxt builds keep working unchanged. Tests: 4 unit suites for the primitives (25 tests), 1 integration spec for the public phone-normalization path (3 tests), 1 smoke spec asserting the combobox triggers render in all three create sheets. Test totals: vitest 713 -> 741 (+28). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 18:13:08 +02:00
index('idx_clients_nationality_iso').on(table.nationalityIso),
feat(dedup): runtime surfaces — merge service, at-create suggestion, admin queue (P2) Adds the live dedup pipeline on top of the P1 library + P3 migration script. The new `client/interest` model now actively prevents duplicate client records at creation time and gives admins a queue to triage the borderline pairs the at-create check missed. Three layers, per design §7: Layer 1 — At-create suggestion ============================== `GET /api/v1/clients/match-candidates` Accepts free-text email / phone / name from the in-flight client form, normalizes them via the dedup library, and returns scored matches against the port's live client pool. Filters out low-confidence noise (the background scoring queue picks those up separately). Strict port scoping; never leaks across tenants. `<DedupSuggestionPanel>` (`src/components/clients/dedup-suggestion-panel.tsx`) Debounced React Query hook. Renders nothing for short inputs or no useful match. On a high-confidence match it interrupts visually with an amber-tinted card and a "Use this client" primary button. Medium confidence falls back to a softer "possible match — check before creating" treatment. `<ClientForm>` Renders the panel above the form (create path only — skipped on edit). New `onUseExistingClient` callback fires when the user picks the existing client; the form closes and the parent decides what to do (typically: navigate to that client's detail page or open the create-interest dialog pre-filled). Layer 2 — Merge service ======================= `mergeClients` (`src/lib/services/client-merge.service.ts`) The atomic merge primitive that everything else calls. Single transaction. Per §6 of the design: - Locks both rows (FOR UPDATE) so concurrent merges of the same loser fail with a clear error rather than racing. - Snapshots the full loser state (contacts / addresses / notes / tags / interest+reservation IDs / relationship rows) into the `client_merge_log.merge_details` JSONB column for the eventual undo flow. - Reattaches every loser-side row to the winner: interests, reservations, contacts (skipping duplicates by `(channel, value)`), addresses, notes, tags (deduped), relationships. - Optional `fieldChoices` — per-scalar overrides letting the user keep the loser's value for fullName / nationality / preferences / timezone / source. - Marks the loser archived with `mergedIntoClientId` set (a redirect pointer for stragglers; never hard-deleted within the undo window). - Resolves any matching `client_merge_candidates` row to status='merged'. - Writes audit log entry. Schema additions: - `clients.merged_into_client_id` (nullable text, indexed) — the redirect pointer set on archive. Tests: 6 cases against a real DB — happy path moves rows + writes log; self-merge / cross-port / already-merged refused; duplicate-contact deduped on reattach; fieldChoices copies loser values to winner. Layer 3 — Admin review queue ============================ `GET /api/v1/admin/duplicates` Pending merge candidates (status='pending') for the current port, with both client summaries hydrated for side-by-side rendering. Skips pairs where one side is already archived/merged. `POST /api/v1/admin/duplicates/[id]/merge` Confirms a candidate. Body picks the winner; the other side becomes the loser. Calls into `mergeClients` — the only path that writes `client_merge_log`. `POST /api/v1/admin/duplicates/[id]/dismiss` Marks the candidate dismissed. Future scoring runs skip the same pair until a score change recreates the row. `<DuplicatesReviewQueue>` (`/admin/duplicates`) Side-by-side card UI for each pending pair. Click a card to pick the winner; the other side is automatically the loser. Toolbar: "Merge into selected" + "Dismiss". No per-field merge editor in this PR — that's a future polish; the simple "pick the better row" flow handles ~80% of cases. Test coverage ============= 11 new integration tests (76 added in this branch total): - 6 mergeClients (atomicity, refusal cases, contact dedup, fieldChoices) - 5 match-candidates API (shape, port scoping, confidence tiers, Pattern F false-positive guard) Full vitest: 926/926 passing (was 858 before the dedup branch). Lint: clean. tsc: clean for new files (only pre-existing errors in unrelated `tests/integration/` files remain, same as before this PR). Out of scope, deferred ====================== - Background scoring cron that populates `client_merge_candidates` (the queue is empty until this lands; manual seeding works for now via the at-create flow). - Side-by-side per-field merge editor with checkboxes (the simple "pick the winner" UX shipped here covers ~80% of real cases). - Admin settings UI for tuning the dedup thresholds. Defaults from the design (90 / 50) are baked in for now. - `unmergeClients` (the snapshot is captured in client_merge_log; the undo endpoint just hasn't been wired yet). These are all natural follow-up PRs that don't block shipping the runtime UX. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:59:04 +02:00
index('idx_clients_merged_into').on(table.mergedIntoClientId),
],
);
export const clientContacts = pgTable(
'client_contacts',
{
id: text('id')
.primaryKey()
.$defaultFn(() => crypto.randomUUID()),
clientId: text('client_id')
.notNull()
.references(() => clients.id, { onDelete: 'cascade' }),
channel: text('channel').notNull(), // email, phone, whatsapp, other
value: text('value').notNull(),
feat(i18n): country/phone/timezone/subdivision primitives + form wiring Cross-cutting i18n polish for forms across the marina + residential + company domains. Introduces a single source of truth for country/phone/timezone/ subdivision data and replaces every nationality-as-free-text and timezone- as-string Input with a dedicated combobox. PR1 Countries — ALL_COUNTRY_CODES (~250 ISO-3166-1 alpha-2), Intl.DisplayNames for localized labels, detectDefaultCountry() with navigator-region fallback to US, CountryCombobox with regional-indicator flag glyphs + compact mode for inline use. PR2 Phone — libphonenumber-js wrapper (parsePhone / formatAsYouType / callingCodeFor), PhoneInput with flag dropdown + national-format AsYouType + paste-detect that flips the country dropdown for pasted international strings. PR3 Timezones — country->IANA map (250 entries, multi-zone for AU/BR/CA/CD/ ID/KZ/MN/MX/RU/US), formatTimezoneLabel ("Europe/London (UTC+1)"), TimezoneCombobox with Suggested/All grouping driven by countryHint. PR4 Subdivisions — wraps the iso-3166-2 npm package (~5000 ISO 3166-2 codes for every country), per-country cache, SubdivisionCombobox with "Pick a country first" / "No regions available" empty states. PR5 Schema deltas (migration 0015) — clients.nationality_iso, clientContacts {value_e164, value_country}, clientAddresses {country_iso, subdivision_iso}, residentialClients {phone_e164, phone_country, nationality_iso, timezone, place_of_residence_country_iso, subdivision_iso}, companies {incorporation_ country_iso, incorporation_subdivision_iso}, companyAddresses {country_iso, subdivision_iso}. Plus shared zod validators (validators/i18n.ts) used by every entity validator + route handler. PR6 ClientForm + ClientDetail — CountryCombobox replaces nationality Input, TimezoneCombobox replaces timezone Input (driven by nationalityIso hint), PhoneInput conditionally rendered for phone/whatsapp contacts. Inline editors (InlineCountryField / InlineTimezoneField / InlinePhoneField) for the detail-page overview rows + ContactsEditor. PR7 Residential client form + detail — phone -> PhoneInput, nationality/ timezone/place-of-residence-country/subdivision rows in both create sheet and inline-editable detail view. Subdivision wipes when country flips since codes are country-scoped. PR8 Company form + detail — incorporation country -> CountryCombobox, incorporation region -> SubdivisionCombobox in both modes. PR9 Public inquiry endpoint — accepts pre-normalized phoneE164/phoneCountry and i18n fields from newer website builds, server-side parsePhone() fallback for legacy raw-international submissions. Old Nuxt builds keep working unchanged. Tests: 4 unit suites for the primitives (25 tests), 1 integration spec for the public phone-normalization path (3 tests), 1 smoke spec asserting the combobox triggers render in all three create sheets. Test totals: vitest 713 -> 741 (+28). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 18:13:08 +02:00
/** E.164-normalized phone number (only set when channel='phone'/'whatsapp'). */
valueE164: text('value_e164'),
/** ISO-3166-1 alpha-2 of the country this number was parsed against. */
valueCountry: text('value_country'),
label: text('label'), // primary, secondary, work, personal, broker, assistant
isPrimary: boolean('is_primary').notNull().default(false),
notes: text('notes'),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(),
},
(table) => [
index('idx_cc_client').on(table.clientId),
index('idx_cc_email')
.on(table.channel, table.value)
.where(sql`${table.channel} = 'email'`),
index('idx_cc_phone')
.on(table.channel, table.value)
.where(sql`${table.channel} = 'phone'`),
// At most one is_primary=true per (client_id, channel). Prevents
// ambiguity when the /clients list pulls "the" primary phone/email.
uniqueIndex('idx_cc_one_primary_per_channel')
.on(table.clientId, table.channel)
.where(sql`${table.isPrimary} = true`),
],
);
export const clientRelationships = pgTable(
'client_relationships',
{
id: text('id')
.primaryKey()
.$defaultFn(() => crypto.randomUUID()),
portId: text('port_id')
.notNull()
.references(() => ports.id),
clientAId: text('client_a_id')
.notNull()
.references(() => clients.id, { onDelete: 'cascade' }),
clientBId: text('client_b_id')
.notNull()
.references(() => clients.id, { onDelete: 'cascade' }),
relationshipType: text('relationship_type').notNull(), // referred_by, broker_for, family_member, same_vessel, custom
description: text('description'),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
},
(table) => [index('idx_cr_port').on(table.portId)],
);
export const clientNotes = pgTable(
'client_notes',
{
id: text('id')
.primaryKey()
.$defaultFn(() => crypto.randomUUID()),
clientId: text('client_id')
.notNull()
.references(() => clients.id, { onDelete: 'cascade' }),
authorId: text('author_id').notNull(), // user ID
content: text('content').notNull(),
mentions: text('mentions').array(), // array of mentioned user IDs
isLocked: boolean('is_locked').notNull().default(false),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(),
},
(table) => [index('idx_cn_client').on(table.clientId)],
);
export const clientTags = pgTable(
'client_tags',
{
clientId: text('client_id')
.notNull()
.references(() => clients.id, { onDelete: 'cascade' }),
tagId: text('tag_id').notNull(), // references tags.id - defined later in system.ts
},
(table) => [primaryKey({ columns: [table.clientId, table.tagId] })],
);
export const clientMergeLog = pgTable(
'client_merge_log',
{
id: text('id')
.primaryKey()
.$defaultFn(() => crypto.randomUUID()),
portId: text('port_id')
.notNull()
.references(() => ports.id),
survivingClientId: text('surviving_client_id')
.notNull()
.references(() => clients.id),
mergedClientId: text('merged_client_id').notNull(), // the client that was merged away (may no longer exist)
mergedBy: text('merged_by').notNull(), // user ID
mergeDetails: jsonb('merge_details').notNull(), // which fields were kept from which record
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
},
(table) => [index('idx_cml_port').on(table.portId)],
);
feat(dedup): NocoDB migration script + tables (P3 dry-run) Lands the one-shot migration pipeline from the legacy NocoDB Interests base into the new client/interest schema. Dry-run mode is fully operational: pulls the live snapshot, runs the dedup library, and writes a CSV + Markdown report under .migration/<timestamp>/. The --apply phase is stubbed for a follow-up PR per the design's P3 implementation sequence. Schema additions ================ - `client_merge_candidates` — pairs flagged by the background scoring job for the /admin/duplicates review queue. Status enum: pending / dismissed / merged. Unique-(portId, clientAId, clientBId) so the same pair can't surface twice. Empty until P2 lands the cron. - `migration_source_links` — idempotency ledger. Maps source-system rows (NocoDB Interest #624 → new client UUID) so re-running --apply against the same dry-run report skips already-imported entities. Both tables ship with the migration `0020_unusual_azazel.sql` — already applied to the local dev DB during this commit's preparation. Library ======= src/lib/dedup/nocodb-source.ts Read-only adapter for the legacy NocoDB v2 API. xc-token auth, auto-paginates until isLastPage, captures the table IDs from the 2026-05-03 audit. `fetchSnapshot()` pulls every relevant table in parallel into one in-memory object the transform layer consumes. src/lib/dedup/migration-transform.ts Pure function: NocoDB snapshot in, MigrationPlan out. Per row: - normalizes name / email / phone / country via the dedup library - parses the legacy DD-MM-YYYY / DD/MM/YYYY / ISO date formats - maps the 8-stage `Sales Process Level` enum to the new 9-stage pipelineStage - filters yacht-name placeholders ('TBC', 'Na', etc.) - merges Internal Notes + Extra Comments + Berth Size Desired into a single notes blob Then runs `findClientMatches` pairwise (with blocking) and union-finds clusters of rows whose score crosses the auto-link threshold (90). Lower-scoring pairs (50–89) become 'needs review'. Each cluster's "lead" row is picked by completeness score with recency tie-break. src/lib/dedup/migration-report.ts Writes three artifacts to .migration/<timestamp>/: - report.csv — one row per planned op, RFC-4180 escaped - summary.md — human-skimmable overview - plan.json — full structured plan for the --apply phase CSV cells with comma / quote / newline are quoted; internal quotes are doubled. No external CSV dep. src/lib/dedup/phone-parse.ts Script-safe wrapper around libphonenumber-js's `core` entry that loads `metadata.min.json` directly. The default `index.cjs.js` bundled by libphonenumber hits a metadata-shape interop bug under Node 25 + tsx (`{ default }` wrapping); core+JSON sidesteps it. The dedup `normalizePhone` and `find-matches` both use this wrapper now so the same code path runs in vitest, Next.js, and the migration CLI without surprises. src/lib/dedup/normalize.ts Tightened country resolution: added Caribbean short-form aliases ('antigua' → AG, 'st kitts' → KN, etc.) and a city map covering the US locations seen in the NocoDB dump (Boston, Tampa, Fort Lauderdale, Port Jefferson, Nantucket). Also relaxed phone parsing to drop the `isValid()` strict check — the libphonenumber min build rejects many real NANP-territory numbers, and dedup only needs a canonical E.164 to compare. CLI === scripts/migrate-from-nocodb.ts pnpm tsx scripts/migrate-from-nocodb.ts --dry-run → Pulls the live NocoDB base (NOCODB_URL + NOCODB_TOKEN env vars), runs the transform, writes report. No DB writes. pnpm tsx scripts/migrate-from-nocodb.ts --apply --report .migration/<dir>/ → Stubbed; exits with `not yet implemented` and a pointer to the design doc. Apply phase ships in a follow-up. Tests ===== tests/unit/dedup/migration-transform.test.ts (7 cases) Fixture-based regression. A frozen 12-row NocoDB snapshot covers every duplicate pattern in the design (§1.2). The test asserts: - 12 input rows → 7 unique clients (cluster math is right) - Patterns A / B / C / E auto-link - Pattern F (Etiennette Clamouze) does NOT auto-link - Every interest preserved as its own row even when clients merge - 8-stage → 9-stage enum mapping is correct per spec - Multi-yacht merge (Constanzo CALYPSO + Costanzo GEMINI under one client) — the design's signature win - Output is deterministic (run twice, identical) Validation against real data ============================ Ran `pnpm tsx scripts/migrate-from-nocodb.ts --dry-run` against the live NocoDB. Result on 252 Interests rows: - 237 clients (15 merged into 13 clusters) - 252 interests (one per source row) - 406 contacts, 52 addresses - 13 auto-linked clusters (every confirmed cluster from §1.2 audit) - 3 pairs flagged for review (Camazou, Zasso, one new) - 1 phone placeholder flagged Total dedup test count: 57 (50 from P1 + 7 fixture tests). Lint: clean. Tsc: clean for new files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:50:01 +02:00
/**
* Pairs of clients flagged by the background scoring job as potential
* duplicates. The `/admin/duplicates` review queue reads from here.
*
* Lifecycle:
* - Background job inserts a row when a pair scores >= the
* `dedup_review_queue_threshold` system setting.
* - User reviews in the admin UI and either merges (status='merged')
* or dismisses (status='dismissed').
* - Subsequent runs of the scoring job skip pairs already
* `dismissed` so the same false-positive doesn't keep reappearing.
* A future score increase recreates the row.
*
* Pairs are stored canonically with `clientAId < clientBId` (string
* comparison) so the same pair only generates one row regardless of
* scoring direction.
*/
export const clientMergeCandidates = pgTable(
'client_merge_candidates',
{
id: text('id')
.primaryKey()
.$defaultFn(() => crypto.randomUUID()),
portId: text('port_id')
.notNull()
.references(() => ports.id),
clientAId: text('client_a_id')
.notNull()
.references(() => clients.id, { onDelete: 'cascade' }),
clientBId: text('client_b_id')
.notNull()
.references(() => clients.id, { onDelete: 'cascade' }),
score: integer('score').notNull(),
/** Human-readable rule list, e.g. ["email match", "phone match"]. */
reasons: jsonb('reasons').notNull(),
status: text('status').notNull().default('pending'), // pending | dismissed | merged
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
resolvedAt: timestamp('resolved_at', { withTimezone: true }),
resolvedBy: text('resolved_by'),
},
(table) => [
index('idx_cmc_port_status').on(table.portId, table.status),
// Same pair shouldn't surface twice - enforce uniqueness on the
feat(dedup): NocoDB migration script + tables (P3 dry-run) Lands the one-shot migration pipeline from the legacy NocoDB Interests base into the new client/interest schema. Dry-run mode is fully operational: pulls the live snapshot, runs the dedup library, and writes a CSV + Markdown report under .migration/<timestamp>/. The --apply phase is stubbed for a follow-up PR per the design's P3 implementation sequence. Schema additions ================ - `client_merge_candidates` — pairs flagged by the background scoring job for the /admin/duplicates review queue. Status enum: pending / dismissed / merged. Unique-(portId, clientAId, clientBId) so the same pair can't surface twice. Empty until P2 lands the cron. - `migration_source_links` — idempotency ledger. Maps source-system rows (NocoDB Interest #624 → new client UUID) so re-running --apply against the same dry-run report skips already-imported entities. Both tables ship with the migration `0020_unusual_azazel.sql` — already applied to the local dev DB during this commit's preparation. Library ======= src/lib/dedup/nocodb-source.ts Read-only adapter for the legacy NocoDB v2 API. xc-token auth, auto-paginates until isLastPage, captures the table IDs from the 2026-05-03 audit. `fetchSnapshot()` pulls every relevant table in parallel into one in-memory object the transform layer consumes. src/lib/dedup/migration-transform.ts Pure function: NocoDB snapshot in, MigrationPlan out. Per row: - normalizes name / email / phone / country via the dedup library - parses the legacy DD-MM-YYYY / DD/MM/YYYY / ISO date formats - maps the 8-stage `Sales Process Level` enum to the new 9-stage pipelineStage - filters yacht-name placeholders ('TBC', 'Na', etc.) - merges Internal Notes + Extra Comments + Berth Size Desired into a single notes blob Then runs `findClientMatches` pairwise (with blocking) and union-finds clusters of rows whose score crosses the auto-link threshold (90). Lower-scoring pairs (50–89) become 'needs review'. Each cluster's "lead" row is picked by completeness score with recency tie-break. src/lib/dedup/migration-report.ts Writes three artifacts to .migration/<timestamp>/: - report.csv — one row per planned op, RFC-4180 escaped - summary.md — human-skimmable overview - plan.json — full structured plan for the --apply phase CSV cells with comma / quote / newline are quoted; internal quotes are doubled. No external CSV dep. src/lib/dedup/phone-parse.ts Script-safe wrapper around libphonenumber-js's `core` entry that loads `metadata.min.json` directly. The default `index.cjs.js` bundled by libphonenumber hits a metadata-shape interop bug under Node 25 + tsx (`{ default }` wrapping); core+JSON sidesteps it. The dedup `normalizePhone` and `find-matches` both use this wrapper now so the same code path runs in vitest, Next.js, and the migration CLI without surprises. src/lib/dedup/normalize.ts Tightened country resolution: added Caribbean short-form aliases ('antigua' → AG, 'st kitts' → KN, etc.) and a city map covering the US locations seen in the NocoDB dump (Boston, Tampa, Fort Lauderdale, Port Jefferson, Nantucket). Also relaxed phone parsing to drop the `isValid()` strict check — the libphonenumber min build rejects many real NANP-territory numbers, and dedup only needs a canonical E.164 to compare. CLI === scripts/migrate-from-nocodb.ts pnpm tsx scripts/migrate-from-nocodb.ts --dry-run → Pulls the live NocoDB base (NOCODB_URL + NOCODB_TOKEN env vars), runs the transform, writes report. No DB writes. pnpm tsx scripts/migrate-from-nocodb.ts --apply --report .migration/<dir>/ → Stubbed; exits with `not yet implemented` and a pointer to the design doc. Apply phase ships in a follow-up. Tests ===== tests/unit/dedup/migration-transform.test.ts (7 cases) Fixture-based regression. A frozen 12-row NocoDB snapshot covers every duplicate pattern in the design (§1.2). The test asserts: - 12 input rows → 7 unique clients (cluster math is right) - Patterns A / B / C / E auto-link - Pattern F (Etiennette Clamouze) does NOT auto-link - Every interest preserved as its own row even when clients merge - 8-stage → 9-stage enum mapping is correct per spec - Multi-yacht merge (Constanzo CALYPSO + Costanzo GEMINI under one client) — the design's signature win - Output is deterministic (run twice, identical) Validation against real data ============================ Ran `pnpm tsx scripts/migrate-from-nocodb.ts --dry-run` against the live NocoDB. Result on 252 Interests rows: - 237 clients (15 merged into 13 clusters) - 252 interests (one per source row) - 406 contacts, 52 addresses - 13 auto-linked clusters (every confirmed cluster from §1.2 audit) - 3 pairs flagged for review (Camazou, Zasso, one new) - 1 phone placeholder flagged Total dedup test count: 57 (50 from P1 + 7 fixture tests). Lint: clean. Tsc: clean for new files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:50:01 +02:00
// canonical (a < b) ordering.
uniqueIndex('idx_cmc_pair').on(table.portId, table.clientAId, table.clientBId),
],
);
export const clientAddresses = pgTable(
'client_addresses',
{
id: text('id')
.primaryKey()
.$defaultFn(() => crypto.randomUUID()),
clientId: text('client_id')
.notNull()
.references(() => clients.id, { onDelete: 'cascade' }),
portId: text('port_id')
.notNull()
.references(() => ports.id, { onDelete: 'cascade' }),
label: text('label').notNull().default('Primary'),
streetAddress: text('street_address'),
city: text('city'),
feat(i18n): country/phone/timezone/subdivision primitives + form wiring Cross-cutting i18n polish for forms across the marina + residential + company domains. Introduces a single source of truth for country/phone/timezone/ subdivision data and replaces every nationality-as-free-text and timezone- as-string Input with a dedicated combobox. PR1 Countries — ALL_COUNTRY_CODES (~250 ISO-3166-1 alpha-2), Intl.DisplayNames for localized labels, detectDefaultCountry() with navigator-region fallback to US, CountryCombobox with regional-indicator flag glyphs + compact mode for inline use. PR2 Phone — libphonenumber-js wrapper (parsePhone / formatAsYouType / callingCodeFor), PhoneInput with flag dropdown + national-format AsYouType + paste-detect that flips the country dropdown for pasted international strings. PR3 Timezones — country->IANA map (250 entries, multi-zone for AU/BR/CA/CD/ ID/KZ/MN/MX/RU/US), formatTimezoneLabel ("Europe/London (UTC+1)"), TimezoneCombobox with Suggested/All grouping driven by countryHint. PR4 Subdivisions — wraps the iso-3166-2 npm package (~5000 ISO 3166-2 codes for every country), per-country cache, SubdivisionCombobox with "Pick a country first" / "No regions available" empty states. PR5 Schema deltas (migration 0015) — clients.nationality_iso, clientContacts {value_e164, value_country}, clientAddresses {country_iso, subdivision_iso}, residentialClients {phone_e164, phone_country, nationality_iso, timezone, place_of_residence_country_iso, subdivision_iso}, companies {incorporation_ country_iso, incorporation_subdivision_iso}, companyAddresses {country_iso, subdivision_iso}. Plus shared zod validators (validators/i18n.ts) used by every entity validator + route handler. PR6 ClientForm + ClientDetail — CountryCombobox replaces nationality Input, TimezoneCombobox replaces timezone Input (driven by nationalityIso hint), PhoneInput conditionally rendered for phone/whatsapp contacts. Inline editors (InlineCountryField / InlineTimezoneField / InlinePhoneField) for the detail-page overview rows + ContactsEditor. PR7 Residential client form + detail — phone -> PhoneInput, nationality/ timezone/place-of-residence-country/subdivision rows in both create sheet and inline-editable detail view. Subdivision wipes when country flips since codes are country-scoped. PR8 Company form + detail — incorporation country -> CountryCombobox, incorporation region -> SubdivisionCombobox in both modes. PR9 Public inquiry endpoint — accepts pre-normalized phoneE164/phoneCountry and i18n fields from newer website builds, server-side parsePhone() fallback for legacy raw-international submissions. Old Nuxt builds keep working unchanged. Tests: 4 unit suites for the primitives (25 tests), 1 integration spec for the public phone-normalization path (3 tests), 1 smoke spec asserting the combobox triggers render in all three create sheets. Test totals: vitest 713 -> 741 (+28). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 18:13:08 +02:00
/** ISO 3166-2 subdivision code (e.g. 'PL-MZ', 'US-CA'). Optional. */
subdivisionIso: text('subdivision_iso'),
postalCode: text('postal_code'),
/** ISO-3166-1 alpha-2 country code. */
feat(i18n): country/phone/timezone/subdivision primitives + form wiring Cross-cutting i18n polish for forms across the marina + residential + company domains. Introduces a single source of truth for country/phone/timezone/ subdivision data and replaces every nationality-as-free-text and timezone- as-string Input with a dedicated combobox. PR1 Countries — ALL_COUNTRY_CODES (~250 ISO-3166-1 alpha-2), Intl.DisplayNames for localized labels, detectDefaultCountry() with navigator-region fallback to US, CountryCombobox with regional-indicator flag glyphs + compact mode for inline use. PR2 Phone — libphonenumber-js wrapper (parsePhone / formatAsYouType / callingCodeFor), PhoneInput with flag dropdown + national-format AsYouType + paste-detect that flips the country dropdown for pasted international strings. PR3 Timezones — country->IANA map (250 entries, multi-zone for AU/BR/CA/CD/ ID/KZ/MN/MX/RU/US), formatTimezoneLabel ("Europe/London (UTC+1)"), TimezoneCombobox with Suggested/All grouping driven by countryHint. PR4 Subdivisions — wraps the iso-3166-2 npm package (~5000 ISO 3166-2 codes for every country), per-country cache, SubdivisionCombobox with "Pick a country first" / "No regions available" empty states. PR5 Schema deltas (migration 0015) — clients.nationality_iso, clientContacts {value_e164, value_country}, clientAddresses {country_iso, subdivision_iso}, residentialClients {phone_e164, phone_country, nationality_iso, timezone, place_of_residence_country_iso, subdivision_iso}, companies {incorporation_ country_iso, incorporation_subdivision_iso}, companyAddresses {country_iso, subdivision_iso}. Plus shared zod validators (validators/i18n.ts) used by every entity validator + route handler. PR6 ClientForm + ClientDetail — CountryCombobox replaces nationality Input, TimezoneCombobox replaces timezone Input (driven by nationalityIso hint), PhoneInput conditionally rendered for phone/whatsapp contacts. Inline editors (InlineCountryField / InlineTimezoneField / InlinePhoneField) for the detail-page overview rows + ContactsEditor. PR7 Residential client form + detail — phone -> PhoneInput, nationality/ timezone/place-of-residence-country/subdivision rows in both create sheet and inline-editable detail view. Subdivision wipes when country flips since codes are country-scoped. PR8 Company form + detail — incorporation country -> CountryCombobox, incorporation region -> SubdivisionCombobox in both modes. PR9 Public inquiry endpoint — accepts pre-normalized phoneE164/phoneCountry and i18n fields from newer website builds, server-side parsePhone() fallback for legacy raw-international submissions. Old Nuxt builds keep working unchanged. Tests: 4 unit suites for the primitives (25 tests), 1 integration spec for the public phone-normalization path (3 tests), 1 smoke spec asserting the combobox triggers render in all three create sheets. Test totals: vitest 713 -> 741 (+28). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 18:13:08 +02:00
countryIso: text('country_iso'),
isPrimary: boolean('is_primary').notNull().default(true),
createdAt: timestamp('created_at', { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp('updated_at', { withTimezone: true }).notNull().defaultNow(),
},
(table) => [
index('idx_ca_client').on(table.clientId),
index('idx_ca_port').on(table.portId),
uniqueIndex('idx_ca_primary')
.on(table.clientId)
.where(sql`${table.isPrimary} = true`),
],
);
export type Client = typeof clients.$inferSelect;
export type NewClient = typeof clients.$inferInsert;
export type ClientContact = typeof clientContacts.$inferSelect;
export type NewClientContact = typeof clientContacts.$inferInsert;
export type ClientRelationship = typeof clientRelationships.$inferSelect;
export type NewClientRelationship = typeof clientRelationships.$inferInsert;
export type ClientNote = typeof clientNotes.$inferSelect;
export type NewClientNote = typeof clientNotes.$inferInsert;
export type ClientMergeLog = typeof clientMergeLog.$inferSelect;
export type NewClientMergeLog = typeof clientMergeLog.$inferInsert;
export type ClientAddress = typeof clientAddresses.$inferSelect;
export type NewClientAddress = typeof clientAddresses.$inferInsert;
feat(dedup): NocoDB migration script + tables (P3 dry-run) Lands the one-shot migration pipeline from the legacy NocoDB Interests base into the new client/interest schema. Dry-run mode is fully operational: pulls the live snapshot, runs the dedup library, and writes a CSV + Markdown report under .migration/<timestamp>/. The --apply phase is stubbed for a follow-up PR per the design's P3 implementation sequence. Schema additions ================ - `client_merge_candidates` — pairs flagged by the background scoring job for the /admin/duplicates review queue. Status enum: pending / dismissed / merged. Unique-(portId, clientAId, clientBId) so the same pair can't surface twice. Empty until P2 lands the cron. - `migration_source_links` — idempotency ledger. Maps source-system rows (NocoDB Interest #624 → new client UUID) so re-running --apply against the same dry-run report skips already-imported entities. Both tables ship with the migration `0020_unusual_azazel.sql` — already applied to the local dev DB during this commit's preparation. Library ======= src/lib/dedup/nocodb-source.ts Read-only adapter for the legacy NocoDB v2 API. xc-token auth, auto-paginates until isLastPage, captures the table IDs from the 2026-05-03 audit. `fetchSnapshot()` pulls every relevant table in parallel into one in-memory object the transform layer consumes. src/lib/dedup/migration-transform.ts Pure function: NocoDB snapshot in, MigrationPlan out. Per row: - normalizes name / email / phone / country via the dedup library - parses the legacy DD-MM-YYYY / DD/MM/YYYY / ISO date formats - maps the 8-stage `Sales Process Level` enum to the new 9-stage pipelineStage - filters yacht-name placeholders ('TBC', 'Na', etc.) - merges Internal Notes + Extra Comments + Berth Size Desired into a single notes blob Then runs `findClientMatches` pairwise (with blocking) and union-finds clusters of rows whose score crosses the auto-link threshold (90). Lower-scoring pairs (50–89) become 'needs review'. Each cluster's "lead" row is picked by completeness score with recency tie-break. src/lib/dedup/migration-report.ts Writes three artifacts to .migration/<timestamp>/: - report.csv — one row per planned op, RFC-4180 escaped - summary.md — human-skimmable overview - plan.json — full structured plan for the --apply phase CSV cells with comma / quote / newline are quoted; internal quotes are doubled. No external CSV dep. src/lib/dedup/phone-parse.ts Script-safe wrapper around libphonenumber-js's `core` entry that loads `metadata.min.json` directly. The default `index.cjs.js` bundled by libphonenumber hits a metadata-shape interop bug under Node 25 + tsx (`{ default }` wrapping); core+JSON sidesteps it. The dedup `normalizePhone` and `find-matches` both use this wrapper now so the same code path runs in vitest, Next.js, and the migration CLI without surprises. src/lib/dedup/normalize.ts Tightened country resolution: added Caribbean short-form aliases ('antigua' → AG, 'st kitts' → KN, etc.) and a city map covering the US locations seen in the NocoDB dump (Boston, Tampa, Fort Lauderdale, Port Jefferson, Nantucket). Also relaxed phone parsing to drop the `isValid()` strict check — the libphonenumber min build rejects many real NANP-territory numbers, and dedup only needs a canonical E.164 to compare. CLI === scripts/migrate-from-nocodb.ts pnpm tsx scripts/migrate-from-nocodb.ts --dry-run → Pulls the live NocoDB base (NOCODB_URL + NOCODB_TOKEN env vars), runs the transform, writes report. No DB writes. pnpm tsx scripts/migrate-from-nocodb.ts --apply --report .migration/<dir>/ → Stubbed; exits with `not yet implemented` and a pointer to the design doc. Apply phase ships in a follow-up. Tests ===== tests/unit/dedup/migration-transform.test.ts (7 cases) Fixture-based regression. A frozen 12-row NocoDB snapshot covers every duplicate pattern in the design (§1.2). The test asserts: - 12 input rows → 7 unique clients (cluster math is right) - Patterns A / B / C / E auto-link - Pattern F (Etiennette Clamouze) does NOT auto-link - Every interest preserved as its own row even when clients merge - 8-stage → 9-stage enum mapping is correct per spec - Multi-yacht merge (Constanzo CALYPSO + Costanzo GEMINI under one client) — the design's signature win - Output is deterministic (run twice, identical) Validation against real data ============================ Ran `pnpm tsx scripts/migrate-from-nocodb.ts --dry-run` against the live NocoDB. Result on 252 Interests rows: - 237 clients (15 merged into 13 clusters) - 252 interests (one per source row) - 406 contacts, 52 addresses - 13 auto-linked clusters (every confirmed cluster from §1.2 audit) - 3 pairs flagged for review (Camazou, Zasso, one new) - 1 phone placeholder flagged Total dedup test count: 57 (50 from P1 + 7 fixture tests). Lint: clean. Tsc: clean for new files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:50:01 +02:00
export type ClientMergeCandidate = typeof clientMergeCandidates.$inferSelect;
export type NewClientMergeCandidate = typeof clientMergeCandidates.$inferInsert;