fix(audit): non-Documenso backlog sweep — port-binding, NULLS NOT DISTINCT, custom merge tokens, company docs
Wave through the remaining audit-final-deferred items that aren't blocked
on the back-burnered Documenso work.
Multi-tenant isolation:
- Storage proxy ProxyTokenPayload gains optional `p` (port slug) claim;
verifier asserts `key.startsWith(${p}/)`. Defense-in-depth against a
buggy issuer in some future code path that mixes port scopes — every
storage key generated by generateStorageKey() already prefixes the
slug. document-sends opts in for 24h emailed download links; other
callers continue working unchanged via the optional field.
DB schema reconciliation:
- Migration 0047 rebuilds system_settings unique index with NULLS NOT
DISTINCT (Postgres 15+) so global settings (port_id IS NULL) are
uniquely keyed by `key` alone. Surfaced + dedupe'd 65 duplicate
(storage_backend, NULL) rows that had accumulated from race-prone
delete-then-insert patterns in ocr-config / settings / residential-
stages / ai-budget services. All four services converted to true
onConflictDoUpdate upserts so the race window is closed.
API uniformity:
- Response shape standardization: 16 routes converted from
`{ success: true }` to 204 No Content. CLAUDE.md documents the
convention (`{ data: <T> }` for content, 204 for empty mutations,
portal-auth retains `{ success: true }` for the frontend's auth chain).
- req.json() → parseBody() migration across 9 admin/CRM routes
(custom-fields, expenses/export ×3, currency convert,
search/recently-viewed, admin/duplicates, berths/pdf-{upload-url,
versions, parse-results}). Uniform 400 error shapes for
ZodError-flagged bodies.
Custom-fields merge tokens (shipped end-to-end):
- merge-fields.ts gains CUSTOM_MERGE_TOKEN_RE + helpers for the
`{{custom.<fieldName>}}` shape.
- document-templates validator accepts the dynamic shape alongside
the static catalog tokens.
- document-sends.service mergeCustomFieldValues resolver fetches
per-port custom_field_definitions for client/interest/berth contexts
and substitutes stored values keyed by `{{custom.fieldName}}`.
- custom-fields-manager amber banner updated to reflect that merge
tokens now expand (search index + entity-diff remain documented
design limitations).
/api/v1/files cross-entity filtering:
- Validator + listFiles + uploadFile accept companyId AND yachtId
alongside clientId. file-upload-zone propagates both.
- New CompanyFilesTab component mirrors ClientFilesTab; restored as a
visible Documents tab in company-tabs.tsx (was a hidden stub).
Inline TODOs:
- Reviewed remaining two TODOs (per-user reminder schedule, import
worker handlers). Both are placeholders for future feature surfaces,
not bugs — per-port digest works for every customer; nothing
currently enqueues import jobs (verified). Annotated in BACKLOG.
BACKLOG.md updated to reflect what landed and what's still pending
(Documenso-related items still bundled with the back-burnered phases).
Tests: 1185/1185 vitest, tsc clean.
2026-05-08 02:20:27 +02:00
|
|
|
import { NextRequest, NextResponse } from 'next/server';
|
|
|
|
|
import { z } from 'zod';
|
feat(dedup): runtime surfaces — merge service, at-create suggestion, admin queue (P2)
Adds the live dedup pipeline on top of the P1 library + P3 migration
script. The new `client/interest` model now actively prevents duplicate
client records at creation time and gives admins a queue to triage
the borderline pairs the at-create check missed.
Three layers, per design §7:
Layer 1 — At-create suggestion
==============================
`GET /api/v1/clients/match-candidates`
Accepts free-text email / phone / name from the in-flight client
form, normalizes them via the dedup library, and returns scored
matches against the port's live client pool. Filters out
low-confidence noise (the background scoring queue picks those up
separately). Strict port scoping; never leaks across tenants.
`<DedupSuggestionPanel>` (`src/components/clients/dedup-suggestion-panel.tsx`)
Debounced React Query hook. Renders nothing for short inputs or
no useful match. On a high-confidence match it interrupts visually
with an amber-tinted card and a "Use this client" primary button.
Medium confidence falls back to a softer "possible match — check
before creating" treatment.
`<ClientForm>`
Renders the panel above the form (create path only — skipped on
edit). New `onUseExistingClient` callback fires when the user
picks the existing client; the form closes and the parent decides
what to do (typically: navigate to that client's detail page or
open the create-interest dialog pre-filled).
Layer 2 — Merge service
=======================
`mergeClients` (`src/lib/services/client-merge.service.ts`)
The atomic merge primitive that everything else calls. Single
transaction. Per §6 of the design:
- Locks both rows (FOR UPDATE) so concurrent merges of the same
loser fail with a clear error rather than racing.
- Snapshots the full loser state (contacts / addresses / notes /
tags / interest+reservation IDs / relationship rows) into the
`client_merge_log.merge_details` JSONB column for the eventual
undo flow.
- Reattaches every loser-side row to the winner: interests,
reservations, contacts (skipping duplicates by `(channel, value)`),
addresses, notes, tags (deduped), relationships.
- Optional `fieldChoices` — per-scalar overrides letting the user
keep the loser's value for fullName / nationality / preferences /
timezone / source.
- Marks the loser archived with `mergedIntoClientId` set (a redirect
pointer for stragglers; never hard-deleted within the undo window).
- Resolves any matching `client_merge_candidates` row to status='merged'.
- Writes audit log entry.
Schema additions:
- `clients.merged_into_client_id` (nullable text, indexed) — the
redirect pointer set on archive.
Tests: 6 cases against a real DB — happy path moves rows + writes log;
self-merge / cross-port / already-merged refused; duplicate-contact
deduped on reattach; fieldChoices copies loser values to winner.
Layer 3 — Admin review queue
============================
`GET /api/v1/admin/duplicates`
Pending merge candidates (status='pending') for the current port,
with both client summaries hydrated for side-by-side rendering.
Skips pairs where one side is already archived/merged.
`POST /api/v1/admin/duplicates/[id]/merge`
Confirms a candidate. Body picks the winner; the other side
becomes the loser. Calls into `mergeClients` — the only path that
writes `client_merge_log`.
`POST /api/v1/admin/duplicates/[id]/dismiss`
Marks the candidate dismissed. Future scoring runs skip the same
pair until a score change recreates the row.
`<DuplicatesReviewQueue>` (`/admin/duplicates`)
Side-by-side card UI for each pending pair. Click a card to pick
the winner; the other side is automatically the loser. Toolbar:
"Merge into selected" + "Dismiss". No per-field merge editor in
this PR — that's a future polish; the simple "pick the better row"
flow handles ~80% of cases.
Test coverage
=============
11 new integration tests (76 added in this branch total):
- 6 mergeClients (atomicity, refusal cases, contact dedup,
fieldChoices)
- 5 match-candidates API (shape, port scoping, confidence tiers,
Pattern F false-positive guard)
Full vitest: 926/926 passing (was 858 before the dedup branch).
Lint: clean. tsc: clean for new files (only pre-existing errors in
unrelated `tests/integration/` files remain, same as before this PR).
Out of scope, deferred
======================
- Background scoring cron that populates `client_merge_candidates`
(the queue is empty until this lands; manual seeding works for
now via the at-create flow).
- Side-by-side per-field merge editor with checkboxes (the simple
"pick the winner" UX shipped here covers ~80% of real cases).
- Admin settings UI for tuning the dedup thresholds. Defaults from
the design (90 / 50) are baked in for now.
- `unmergeClients` (the snapshot is captured in client_merge_log;
the undo endpoint just hasn't been wired yet).
These are all natural follow-up PRs that don't block shipping the
runtime UX.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:59:04 +02:00
|
|
|
import { and, eq, inArray } from 'drizzle-orm';
|
|
|
|
|
|
|
|
|
|
import type { AuthContext } from '@/lib/api/helpers';
|
fix(audit): non-Documenso backlog sweep — port-binding, NULLS NOT DISTINCT, custom merge tokens, company docs
Wave through the remaining audit-final-deferred items that aren't blocked
on the back-burnered Documenso work.
Multi-tenant isolation:
- Storage proxy ProxyTokenPayload gains optional `p` (port slug) claim;
verifier asserts `key.startsWith(${p}/)`. Defense-in-depth against a
buggy issuer in some future code path that mixes port scopes — every
storage key generated by generateStorageKey() already prefixes the
slug. document-sends opts in for 24h emailed download links; other
callers continue working unchanged via the optional field.
DB schema reconciliation:
- Migration 0047 rebuilds system_settings unique index with NULLS NOT
DISTINCT (Postgres 15+) so global settings (port_id IS NULL) are
uniquely keyed by `key` alone. Surfaced + dedupe'd 65 duplicate
(storage_backend, NULL) rows that had accumulated from race-prone
delete-then-insert patterns in ocr-config / settings / residential-
stages / ai-budget services. All four services converted to true
onConflictDoUpdate upserts so the race window is closed.
API uniformity:
- Response shape standardization: 16 routes converted from
`{ success: true }` to 204 No Content. CLAUDE.md documents the
convention (`{ data: <T> }` for content, 204 for empty mutations,
portal-auth retains `{ success: true }` for the frontend's auth chain).
- req.json() → parseBody() migration across 9 admin/CRM routes
(custom-fields, expenses/export ×3, currency convert,
search/recently-viewed, admin/duplicates, berths/pdf-{upload-url,
versions, parse-results}). Uniform 400 error shapes for
ZodError-flagged bodies.
Custom-fields merge tokens (shipped end-to-end):
- merge-fields.ts gains CUSTOM_MERGE_TOKEN_RE + helpers for the
`{{custom.<fieldName>}}` shape.
- document-templates validator accepts the dynamic shape alongside
the static catalog tokens.
- document-sends.service mergeCustomFieldValues resolver fetches
per-port custom_field_definitions for client/interest/berth contexts
and substitutes stored values keyed by `{{custom.fieldName}}`.
- custom-fields-manager amber banner updated to reflect that merge
tokens now expand (search index + entity-diff remain documented
design limitations).
/api/v1/files cross-entity filtering:
- Validator + listFiles + uploadFile accept companyId AND yachtId
alongside clientId. file-upload-zone propagates both.
- New CompanyFilesTab component mirrors ClientFilesTab; restored as a
visible Documents tab in company-tabs.tsx (was a hidden stub).
Inline TODOs:
- Reviewed remaining two TODOs (per-user reminder schedule, import
worker handlers). Both are placeholders for future feature surfaces,
not bugs — per-port digest works for every customer; nothing
currently enqueues import jobs (verified). Annotated in BACKLOG.
BACKLOG.md updated to reflect what landed and what's still pending
(Documenso-related items still bundled with the back-burnered phases).
Tests: 1185/1185 vitest, tsc clean.
2026-05-08 02:20:27 +02:00
|
|
|
import { parseBody } from '@/lib/api/route-helpers';
|
feat(dedup): runtime surfaces — merge service, at-create suggestion, admin queue (P2)
Adds the live dedup pipeline on top of the P1 library + P3 migration
script. The new `client/interest` model now actively prevents duplicate
client records at creation time and gives admins a queue to triage
the borderline pairs the at-create check missed.
Three layers, per design §7:
Layer 1 — At-create suggestion
==============================
`GET /api/v1/clients/match-candidates`
Accepts free-text email / phone / name from the in-flight client
form, normalizes them via the dedup library, and returns scored
matches against the port's live client pool. Filters out
low-confidence noise (the background scoring queue picks those up
separately). Strict port scoping; never leaks across tenants.
`<DedupSuggestionPanel>` (`src/components/clients/dedup-suggestion-panel.tsx`)
Debounced React Query hook. Renders nothing for short inputs or
no useful match. On a high-confidence match it interrupts visually
with an amber-tinted card and a "Use this client" primary button.
Medium confidence falls back to a softer "possible match — check
before creating" treatment.
`<ClientForm>`
Renders the panel above the form (create path only — skipped on
edit). New `onUseExistingClient` callback fires when the user
picks the existing client; the form closes and the parent decides
what to do (typically: navigate to that client's detail page or
open the create-interest dialog pre-filled).
Layer 2 — Merge service
=======================
`mergeClients` (`src/lib/services/client-merge.service.ts`)
The atomic merge primitive that everything else calls. Single
transaction. Per §6 of the design:
- Locks both rows (FOR UPDATE) so concurrent merges of the same
loser fail with a clear error rather than racing.
- Snapshots the full loser state (contacts / addresses / notes /
tags / interest+reservation IDs / relationship rows) into the
`client_merge_log.merge_details` JSONB column for the eventual
undo flow.
- Reattaches every loser-side row to the winner: interests,
reservations, contacts (skipping duplicates by `(channel, value)`),
addresses, notes, tags (deduped), relationships.
- Optional `fieldChoices` — per-scalar overrides letting the user
keep the loser's value for fullName / nationality / preferences /
timezone / source.
- Marks the loser archived with `mergedIntoClientId` set (a redirect
pointer for stragglers; never hard-deleted within the undo window).
- Resolves any matching `client_merge_candidates` row to status='merged'.
- Writes audit log entry.
Schema additions:
- `clients.merged_into_client_id` (nullable text, indexed) — the
redirect pointer set on archive.
Tests: 6 cases against a real DB — happy path moves rows + writes log;
self-merge / cross-port / already-merged refused; duplicate-contact
deduped on reattach; fieldChoices copies loser values to winner.
Layer 3 — Admin review queue
============================
`GET /api/v1/admin/duplicates`
Pending merge candidates (status='pending') for the current port,
with both client summaries hydrated for side-by-side rendering.
Skips pairs where one side is already archived/merged.
`POST /api/v1/admin/duplicates/[id]/merge`
Confirms a candidate. Body picks the winner; the other side
becomes the loser. Calls into `mergeClients` — the only path that
writes `client_merge_log`.
`POST /api/v1/admin/duplicates/[id]/dismiss`
Marks the candidate dismissed. Future scoring runs skip the same
pair until a score change recreates the row.
`<DuplicatesReviewQueue>` (`/admin/duplicates`)
Side-by-side card UI for each pending pair. Click a card to pick
the winner; the other side is automatically the loser. Toolbar:
"Merge into selected" + "Dismiss". No per-field merge editor in
this PR — that's a future polish; the simple "pick the better row"
flow handles ~80% of cases.
Test coverage
=============
11 new integration tests (76 added in this branch total):
- 6 mergeClients (atomicity, refusal cases, contact dedup,
fieldChoices)
- 5 match-candidates API (shape, port scoping, confidence tiers,
Pattern F false-positive guard)
Full vitest: 926/926 passing (was 858 before the dedup branch).
Lint: clean. tsc: clean for new files (only pre-existing errors in
unrelated `tests/integration/` files remain, same as before this PR).
Out of scope, deferred
======================
- Background scoring cron that populates `client_merge_candidates`
(the queue is empty until this lands; manual seeding works for
now via the at-create flow).
- Side-by-side per-field merge editor with checkboxes (the simple
"pick the winner" UX shipped here covers ~80% of real cases).
- Admin settings UI for tuning the dedup thresholds. Defaults from
the design (90 / 50) are baked in for now.
- `unmergeClients` (the snapshot is captured in client_merge_log;
the undo endpoint just hasn't been wired yet).
These are all natural follow-up PRs that don't block shipping the
runtime UX.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:59:04 +02:00
|
|
|
import { db } from '@/lib/db';
|
|
|
|
|
import { clients, clientMergeCandidates } from '@/lib/db/schema/clients';
|
fix(audit-tier-2-routes): manual NextResponse.json error sweep + admin form banners
Two final waves of error-surface hygiene closing the audit's MED §12 +
HIGH §15 + HIGH §17 findings:
* 50 route files swept (61 sites): manual NextResponse.json({error,
status: 4xx|5xx}) early-returns replaced by typed throws +
errorResponse(err) at the catch.
- Super-admin gates (13 sites) use new requireSuperAdmin(ctx, action)
helper from src/lib/api/helpers.ts so denials hit the audit log.
- Path-param + body validation 400s become ValidationError throws.
- 404s become NotFoundError or CodedError('NOT_FOUND') for AI
feature-flag paths.
- 11 manual 5xx returns now re-throw so error_events captures the
request-id (the admin error inspector becomes usable from real
incidents).
- website-analytics 200-with-error anti-pattern flipped to 409 +
UMAMI_NOT_CONFIGURED. 502 upstream paths use UMAMI_UPSTREAM_ERROR.
- 11 sites intentionally preserved: storage/[token] anti-enumeration
token-failure paths, webhook-secret 401, "Unknown port" 400 in
public intake.
* 7 admin forms (roles, users, ports, webhooks, custom-fields,
document-templates, tags) gain a formatErrorBanner() helper from
src/lib/api/toast-error.ts that builds a multi-line "Error code / Reference ID"
banner — the rep can copy the request id when reporting a failed
save. Banners get whitespace-pre-line so newlines render.
Test status: 1168/1168 vitest, tsc clean.
Refs: docs/audit-comprehensive-2026-05-05.md MED §12 (auditor-F Issue 1)
+ HIGH §15 (auditor-F Issue 2) + HIGH §17 (auditor-H Issue 2).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 20:36:59 +02:00
|
|
|
import { errorResponse, NotFoundError, ValidationError } from '@/lib/errors';
|
feat(dedup): runtime surfaces — merge service, at-create suggestion, admin queue (P2)
Adds the live dedup pipeline on top of the P1 library + P3 migration
script. The new `client/interest` model now actively prevents duplicate
client records at creation time and gives admins a queue to triage
the borderline pairs the at-create check missed.
Three layers, per design §7:
Layer 1 — At-create suggestion
==============================
`GET /api/v1/clients/match-candidates`
Accepts free-text email / phone / name from the in-flight client
form, normalizes them via the dedup library, and returns scored
matches against the port's live client pool. Filters out
low-confidence noise (the background scoring queue picks those up
separately). Strict port scoping; never leaks across tenants.
`<DedupSuggestionPanel>` (`src/components/clients/dedup-suggestion-panel.tsx`)
Debounced React Query hook. Renders nothing for short inputs or
no useful match. On a high-confidence match it interrupts visually
with an amber-tinted card and a "Use this client" primary button.
Medium confidence falls back to a softer "possible match — check
before creating" treatment.
`<ClientForm>`
Renders the panel above the form (create path only — skipped on
edit). New `onUseExistingClient` callback fires when the user
picks the existing client; the form closes and the parent decides
what to do (typically: navigate to that client's detail page or
open the create-interest dialog pre-filled).
Layer 2 — Merge service
=======================
`mergeClients` (`src/lib/services/client-merge.service.ts`)
The atomic merge primitive that everything else calls. Single
transaction. Per §6 of the design:
- Locks both rows (FOR UPDATE) so concurrent merges of the same
loser fail with a clear error rather than racing.
- Snapshots the full loser state (contacts / addresses / notes /
tags / interest+reservation IDs / relationship rows) into the
`client_merge_log.merge_details` JSONB column for the eventual
undo flow.
- Reattaches every loser-side row to the winner: interests,
reservations, contacts (skipping duplicates by `(channel, value)`),
addresses, notes, tags (deduped), relationships.
- Optional `fieldChoices` — per-scalar overrides letting the user
keep the loser's value for fullName / nationality / preferences /
timezone / source.
- Marks the loser archived with `mergedIntoClientId` set (a redirect
pointer for stragglers; never hard-deleted within the undo window).
- Resolves any matching `client_merge_candidates` row to status='merged'.
- Writes audit log entry.
Schema additions:
- `clients.merged_into_client_id` (nullable text, indexed) — the
redirect pointer set on archive.
Tests: 6 cases against a real DB — happy path moves rows + writes log;
self-merge / cross-port / already-merged refused; duplicate-contact
deduped on reattach; fieldChoices copies loser values to winner.
Layer 3 — Admin review queue
============================
`GET /api/v1/admin/duplicates`
Pending merge candidates (status='pending') for the current port,
with both client summaries hydrated for side-by-side rendering.
Skips pairs where one side is already archived/merged.
`POST /api/v1/admin/duplicates/[id]/merge`
Confirms a candidate. Body picks the winner; the other side
becomes the loser. Calls into `mergeClients` — the only path that
writes `client_merge_log`.
`POST /api/v1/admin/duplicates/[id]/dismiss`
Marks the candidate dismissed. Future scoring runs skip the same
pair until a score change recreates the row.
`<DuplicatesReviewQueue>` (`/admin/duplicates`)
Side-by-side card UI for each pending pair. Click a card to pick
the winner; the other side is automatically the loser. Toolbar:
"Merge into selected" + "Dismiss". No per-field merge editor in
this PR — that's a future polish; the simple "pick the better row"
flow handles ~80% of cases.
Test coverage
=============
11 new integration tests (76 added in this branch total):
- 6 mergeClients (atomicity, refusal cases, contact dedup,
fieldChoices)
- 5 match-candidates API (shape, port scoping, confidence tiers,
Pattern F false-positive guard)
Full vitest: 926/926 passing (was 858 before the dedup branch).
Lint: clean. tsc: clean for new files (only pre-existing errors in
unrelated `tests/integration/` files remain, same as before this PR).
Out of scope, deferred
======================
- Background scoring cron that populates `client_merge_candidates`
(the queue is empty until this lands; manual seeding works for
now via the at-create flow).
- Side-by-side per-field merge editor with checkboxes (the simple
"pick the winner" UX shipped here covers ~80% of real cases).
- Admin settings UI for tuning the dedup thresholds. Defaults from
the design (90 / 50) are baked in for now.
- `unmergeClients` (the snapshot is captured in client_merge_log;
the undo endpoint just hasn't been wired yet).
These are all natural follow-up PRs that don't block shipping the
runtime UX.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:59:04 +02:00
|
|
|
import {
|
|
|
|
|
listPendingMergeCandidates,
|
|
|
|
|
mergeClients,
|
|
|
|
|
type MergeFieldChoices,
|
|
|
|
|
} from '@/lib/services/client-merge.service';
|
|
|
|
|
|
fix(audit): non-Documenso backlog sweep — port-binding, NULLS NOT DISTINCT, custom merge tokens, company docs
Wave through the remaining audit-final-deferred items that aren't blocked
on the back-burnered Documenso work.
Multi-tenant isolation:
- Storage proxy ProxyTokenPayload gains optional `p` (port slug) claim;
verifier asserts `key.startsWith(${p}/)`. Defense-in-depth against a
buggy issuer in some future code path that mixes port scopes — every
storage key generated by generateStorageKey() already prefixes the
slug. document-sends opts in for 24h emailed download links; other
callers continue working unchanged via the optional field.
DB schema reconciliation:
- Migration 0047 rebuilds system_settings unique index with NULLS NOT
DISTINCT (Postgres 15+) so global settings (port_id IS NULL) are
uniquely keyed by `key` alone. Surfaced + dedupe'd 65 duplicate
(storage_backend, NULL) rows that had accumulated from race-prone
delete-then-insert patterns in ocr-config / settings / residential-
stages / ai-budget services. All four services converted to true
onConflictDoUpdate upserts so the race window is closed.
API uniformity:
- Response shape standardization: 16 routes converted from
`{ success: true }` to 204 No Content. CLAUDE.md documents the
convention (`{ data: <T> }` for content, 204 for empty mutations,
portal-auth retains `{ success: true }` for the frontend's auth chain).
- req.json() → parseBody() migration across 9 admin/CRM routes
(custom-fields, expenses/export ×3, currency convert,
search/recently-viewed, admin/duplicates, berths/pdf-{upload-url,
versions, parse-results}). Uniform 400 error shapes for
ZodError-flagged bodies.
Custom-fields merge tokens (shipped end-to-end):
- merge-fields.ts gains CUSTOM_MERGE_TOKEN_RE + helpers for the
`{{custom.<fieldName>}}` shape.
- document-templates validator accepts the dynamic shape alongside
the static catalog tokens.
- document-sends.service mergeCustomFieldValues resolver fetches
per-port custom_field_definitions for client/interest/berth contexts
and substitutes stored values keyed by `{{custom.fieldName}}`.
- custom-fields-manager amber banner updated to reflect that merge
tokens now expand (search index + entity-diff remain documented
design limitations).
/api/v1/files cross-entity filtering:
- Validator + listFiles + uploadFile accept companyId AND yachtId
alongside clientId. file-upload-zone propagates both.
- New CompanyFilesTab component mirrors ClientFilesTab; restored as a
visible Documents tab in company-tabs.tsx (was a hidden stub).
Inline TODOs:
- Reviewed remaining two TODOs (per-user reminder schedule, import
worker handlers). Both are placeholders for future feature surfaces,
not bugs — per-port digest works for every customer; nothing
currently enqueues import jobs (verified). Annotated in BACKLOG.
BACKLOG.md updated to reflect what landed and what's still pending
(Documenso-related items still bundled with the back-burnered phases).
Tests: 1185/1185 vitest, tsc clean.
2026-05-08 02:20:27 +02:00
|
|
|
const confirmMergeSchema = z.object({
|
|
|
|
|
winnerId: z.string().min(1),
|
|
|
|
|
fieldChoices: z.record(z.string(), z.string()).optional(),
|
|
|
|
|
});
|
|
|
|
|
|
feat(dedup): runtime surfaces — merge service, at-create suggestion, admin queue (P2)
Adds the live dedup pipeline on top of the P1 library + P3 migration
script. The new `client/interest` model now actively prevents duplicate
client records at creation time and gives admins a queue to triage
the borderline pairs the at-create check missed.
Three layers, per design §7:
Layer 1 — At-create suggestion
==============================
`GET /api/v1/clients/match-candidates`
Accepts free-text email / phone / name from the in-flight client
form, normalizes them via the dedup library, and returns scored
matches against the port's live client pool. Filters out
low-confidence noise (the background scoring queue picks those up
separately). Strict port scoping; never leaks across tenants.
`<DedupSuggestionPanel>` (`src/components/clients/dedup-suggestion-panel.tsx`)
Debounced React Query hook. Renders nothing for short inputs or
no useful match. On a high-confidence match it interrupts visually
with an amber-tinted card and a "Use this client" primary button.
Medium confidence falls back to a softer "possible match — check
before creating" treatment.
`<ClientForm>`
Renders the panel above the form (create path only — skipped on
edit). New `onUseExistingClient` callback fires when the user
picks the existing client; the form closes and the parent decides
what to do (typically: navigate to that client's detail page or
open the create-interest dialog pre-filled).
Layer 2 — Merge service
=======================
`mergeClients` (`src/lib/services/client-merge.service.ts`)
The atomic merge primitive that everything else calls. Single
transaction. Per §6 of the design:
- Locks both rows (FOR UPDATE) so concurrent merges of the same
loser fail with a clear error rather than racing.
- Snapshots the full loser state (contacts / addresses / notes /
tags / interest+reservation IDs / relationship rows) into the
`client_merge_log.merge_details` JSONB column for the eventual
undo flow.
- Reattaches every loser-side row to the winner: interests,
reservations, contacts (skipping duplicates by `(channel, value)`),
addresses, notes, tags (deduped), relationships.
- Optional `fieldChoices` — per-scalar overrides letting the user
keep the loser's value for fullName / nationality / preferences /
timezone / source.
- Marks the loser archived with `mergedIntoClientId` set (a redirect
pointer for stragglers; never hard-deleted within the undo window).
- Resolves any matching `client_merge_candidates` row to status='merged'.
- Writes audit log entry.
Schema additions:
- `clients.merged_into_client_id` (nullable text, indexed) — the
redirect pointer set on archive.
Tests: 6 cases against a real DB — happy path moves rows + writes log;
self-merge / cross-port / already-merged refused; duplicate-contact
deduped on reattach; fieldChoices copies loser values to winner.
Layer 3 — Admin review queue
============================
`GET /api/v1/admin/duplicates`
Pending merge candidates (status='pending') for the current port,
with both client summaries hydrated for side-by-side rendering.
Skips pairs where one side is already archived/merged.
`POST /api/v1/admin/duplicates/[id]/merge`
Confirms a candidate. Body picks the winner; the other side
becomes the loser. Calls into `mergeClients` — the only path that
writes `client_merge_log`.
`POST /api/v1/admin/duplicates/[id]/dismiss`
Marks the candidate dismissed. Future scoring runs skip the same
pair until a score change recreates the row.
`<DuplicatesReviewQueue>` (`/admin/duplicates`)
Side-by-side card UI for each pending pair. Click a card to pick
the winner; the other side is automatically the loser. Toolbar:
"Merge into selected" + "Dismiss". No per-field merge editor in
this PR — that's a future polish; the simple "pick the better row"
flow handles ~80% of cases.
Test coverage
=============
11 new integration tests (76 added in this branch total):
- 6 mergeClients (atomicity, refusal cases, contact dedup,
fieldChoices)
- 5 match-candidates API (shape, port scoping, confidence tiers,
Pattern F false-positive guard)
Full vitest: 926/926 passing (was 858 before the dedup branch).
Lint: clean. tsc: clean for new files (only pre-existing errors in
unrelated `tests/integration/` files remain, same as before this PR).
Out of scope, deferred
======================
- Background scoring cron that populates `client_merge_candidates`
(the queue is empty until this lands; manual seeding works for
now via the at-create flow).
- Side-by-side per-field merge editor with checkboxes (the simple
"pick the winner" UX shipped here covers ~80% of real cases).
- Admin settings UI for tuning the dedup thresholds. Defaults from
the design (90 / 50) are baked in for now.
- `unmergeClients` (the snapshot is captured in client_merge_log;
the undo endpoint just hasn't been wired yet).
These are all natural follow-up PRs that don't block shipping the
runtime UX.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:59:04 +02:00
|
|
|
/**
|
|
|
|
|
* GET /api/v1/admin/duplicates
|
|
|
|
|
*
|
|
|
|
|
* Pending merge candidates for the current port, sorted by score.
|
|
|
|
|
* Each row hydrates its two client summaries so the review-queue UI
|
|
|
|
|
* can render side-by-side cards without an N+1 fetch.
|
|
|
|
|
*/
|
|
|
|
|
export async function listHandler(_req: Request, ctx: AuthContext): Promise<NextResponse> {
|
|
|
|
|
try {
|
|
|
|
|
const pairs = await listPendingMergeCandidates(ctx.portId);
|
|
|
|
|
if (pairs.length === 0) return NextResponse.json({ data: [] });
|
|
|
|
|
|
|
|
|
|
const ids = Array.from(new Set(pairs.flatMap((p) => [p.clientAId, p.clientBId])));
|
|
|
|
|
const clientRows = await db
|
|
|
|
|
.select({
|
|
|
|
|
id: clients.id,
|
|
|
|
|
fullName: clients.fullName,
|
|
|
|
|
archivedAt: clients.archivedAt,
|
|
|
|
|
mergedIntoClientId: clients.mergedIntoClientId,
|
|
|
|
|
createdAt: clients.createdAt,
|
|
|
|
|
})
|
|
|
|
|
.from(clients)
|
|
|
|
|
.where(inArray(clients.id, ids));
|
|
|
|
|
const clientById = new Map(clientRows.map((c) => [c.id, c]));
|
|
|
|
|
|
|
|
|
|
const data = pairs
|
|
|
|
|
.map((p) => {
|
|
|
|
|
const a = clientById.get(p.clientAId);
|
|
|
|
|
const b = clientById.get(p.clientBId);
|
2026-05-04 22:57:01 +02:00
|
|
|
if (!a || !b) return null; // FK orphan - shouldn't happen, but be defensive
|
feat(dedup): runtime surfaces — merge service, at-create suggestion, admin queue (P2)
Adds the live dedup pipeline on top of the P1 library + P3 migration
script. The new `client/interest` model now actively prevents duplicate
client records at creation time and gives admins a queue to triage
the borderline pairs the at-create check missed.
Three layers, per design §7:
Layer 1 — At-create suggestion
==============================
`GET /api/v1/clients/match-candidates`
Accepts free-text email / phone / name from the in-flight client
form, normalizes them via the dedup library, and returns scored
matches against the port's live client pool. Filters out
low-confidence noise (the background scoring queue picks those up
separately). Strict port scoping; never leaks across tenants.
`<DedupSuggestionPanel>` (`src/components/clients/dedup-suggestion-panel.tsx`)
Debounced React Query hook. Renders nothing for short inputs or
no useful match. On a high-confidence match it interrupts visually
with an amber-tinted card and a "Use this client" primary button.
Medium confidence falls back to a softer "possible match — check
before creating" treatment.
`<ClientForm>`
Renders the panel above the form (create path only — skipped on
edit). New `onUseExistingClient` callback fires when the user
picks the existing client; the form closes and the parent decides
what to do (typically: navigate to that client's detail page or
open the create-interest dialog pre-filled).
Layer 2 — Merge service
=======================
`mergeClients` (`src/lib/services/client-merge.service.ts`)
The atomic merge primitive that everything else calls. Single
transaction. Per §6 of the design:
- Locks both rows (FOR UPDATE) so concurrent merges of the same
loser fail with a clear error rather than racing.
- Snapshots the full loser state (contacts / addresses / notes /
tags / interest+reservation IDs / relationship rows) into the
`client_merge_log.merge_details` JSONB column for the eventual
undo flow.
- Reattaches every loser-side row to the winner: interests,
reservations, contacts (skipping duplicates by `(channel, value)`),
addresses, notes, tags (deduped), relationships.
- Optional `fieldChoices` — per-scalar overrides letting the user
keep the loser's value for fullName / nationality / preferences /
timezone / source.
- Marks the loser archived with `mergedIntoClientId` set (a redirect
pointer for stragglers; never hard-deleted within the undo window).
- Resolves any matching `client_merge_candidates` row to status='merged'.
- Writes audit log entry.
Schema additions:
- `clients.merged_into_client_id` (nullable text, indexed) — the
redirect pointer set on archive.
Tests: 6 cases against a real DB — happy path moves rows + writes log;
self-merge / cross-port / already-merged refused; duplicate-contact
deduped on reattach; fieldChoices copies loser values to winner.
Layer 3 — Admin review queue
============================
`GET /api/v1/admin/duplicates`
Pending merge candidates (status='pending') for the current port,
with both client summaries hydrated for side-by-side rendering.
Skips pairs where one side is already archived/merged.
`POST /api/v1/admin/duplicates/[id]/merge`
Confirms a candidate. Body picks the winner; the other side
becomes the loser. Calls into `mergeClients` — the only path that
writes `client_merge_log`.
`POST /api/v1/admin/duplicates/[id]/dismiss`
Marks the candidate dismissed. Future scoring runs skip the same
pair until a score change recreates the row.
`<DuplicatesReviewQueue>` (`/admin/duplicates`)
Side-by-side card UI for each pending pair. Click a card to pick
the winner; the other side is automatically the loser. Toolbar:
"Merge into selected" + "Dismiss". No per-field merge editor in
this PR — that's a future polish; the simple "pick the better row"
flow handles ~80% of cases.
Test coverage
=============
11 new integration tests (76 added in this branch total):
- 6 mergeClients (atomicity, refusal cases, contact dedup,
fieldChoices)
- 5 match-candidates API (shape, port scoping, confidence tiers,
Pattern F false-positive guard)
Full vitest: 926/926 passing (was 858 before the dedup branch).
Lint: clean. tsc: clean for new files (only pre-existing errors in
unrelated `tests/integration/` files remain, same as before this PR).
Out of scope, deferred
======================
- Background scoring cron that populates `client_merge_candidates`
(the queue is empty until this lands; manual seeding works for
now via the at-create flow).
- Side-by-side per-field merge editor with checkboxes (the simple
"pick the winner" UX shipped here covers ~80% of real cases).
- Admin settings UI for tuning the dedup thresholds. Defaults from
the design (90 / 50) are baked in for now.
- `unmergeClients` (the snapshot is captured in client_merge_log;
the undo endpoint just hasn't been wired yet).
These are all natural follow-up PRs that don't block shipping the
runtime UX.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:59:04 +02:00
|
|
|
// Skip pairs where one side has already been merged or archived.
|
|
|
|
|
if (a.mergedIntoClientId || b.mergedIntoClientId) return null;
|
|
|
|
|
return {
|
|
|
|
|
id: p.id,
|
|
|
|
|
score: p.score,
|
|
|
|
|
reasons: p.reasons,
|
|
|
|
|
createdAt: p.createdAt,
|
|
|
|
|
clientA: { id: a.id, fullName: a.fullName, createdAt: a.createdAt },
|
|
|
|
|
clientB: { id: b.id, fullName: b.fullName, createdAt: b.createdAt },
|
|
|
|
|
};
|
|
|
|
|
})
|
|
|
|
|
.filter((row): row is NonNullable<typeof row> => row !== null);
|
|
|
|
|
|
|
|
|
|
return NextResponse.json({ data });
|
|
|
|
|
} catch (error) {
|
|
|
|
|
return errorResponse(error);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* POST /api/v1/admin/duplicates/[id]/merge
|
|
|
|
|
*
|
|
|
|
|
* Body: { winnerId: string, fieldChoices?: MergeFieldChoices }
|
|
|
|
|
*
|
|
|
|
|
* Confirms a merge candidate. The winner is the one the user picked
|
|
|
|
|
* to keep; the other side becomes the loser. Calls into the merge
|
|
|
|
|
* service which is the only path that touches client_merge_log.
|
|
|
|
|
*/
|
|
|
|
|
export async function confirmMergeHandler(
|
fix(audit): non-Documenso backlog sweep — port-binding, NULLS NOT DISTINCT, custom merge tokens, company docs
Wave through the remaining audit-final-deferred items that aren't blocked
on the back-burnered Documenso work.
Multi-tenant isolation:
- Storage proxy ProxyTokenPayload gains optional `p` (port slug) claim;
verifier asserts `key.startsWith(${p}/)`. Defense-in-depth against a
buggy issuer in some future code path that mixes port scopes — every
storage key generated by generateStorageKey() already prefixes the
slug. document-sends opts in for 24h emailed download links; other
callers continue working unchanged via the optional field.
DB schema reconciliation:
- Migration 0047 rebuilds system_settings unique index with NULLS NOT
DISTINCT (Postgres 15+) so global settings (port_id IS NULL) are
uniquely keyed by `key` alone. Surfaced + dedupe'd 65 duplicate
(storage_backend, NULL) rows that had accumulated from race-prone
delete-then-insert patterns in ocr-config / settings / residential-
stages / ai-budget services. All four services converted to true
onConflictDoUpdate upserts so the race window is closed.
API uniformity:
- Response shape standardization: 16 routes converted from
`{ success: true }` to 204 No Content. CLAUDE.md documents the
convention (`{ data: <T> }` for content, 204 for empty mutations,
portal-auth retains `{ success: true }` for the frontend's auth chain).
- req.json() → parseBody() migration across 9 admin/CRM routes
(custom-fields, expenses/export ×3, currency convert,
search/recently-viewed, admin/duplicates, berths/pdf-{upload-url,
versions, parse-results}). Uniform 400 error shapes for
ZodError-flagged bodies.
Custom-fields merge tokens (shipped end-to-end):
- merge-fields.ts gains CUSTOM_MERGE_TOKEN_RE + helpers for the
`{{custom.<fieldName>}}` shape.
- document-templates validator accepts the dynamic shape alongside
the static catalog tokens.
- document-sends.service mergeCustomFieldValues resolver fetches
per-port custom_field_definitions for client/interest/berth contexts
and substitutes stored values keyed by `{{custom.fieldName}}`.
- custom-fields-manager amber banner updated to reflect that merge
tokens now expand (search index + entity-diff remain documented
design limitations).
/api/v1/files cross-entity filtering:
- Validator + listFiles + uploadFile accept companyId AND yachtId
alongside clientId. file-upload-zone propagates both.
- New CompanyFilesTab component mirrors ClientFilesTab; restored as a
visible Documents tab in company-tabs.tsx (was a hidden stub).
Inline TODOs:
- Reviewed remaining two TODOs (per-user reminder schedule, import
worker handlers). Both are placeholders for future feature surfaces,
not bugs — per-port digest works for every customer; nothing
currently enqueues import jobs (verified). Annotated in BACKLOG.
BACKLOG.md updated to reflect what landed and what's still pending
(Documenso-related items still bundled with the back-burnered phases).
Tests: 1185/1185 vitest, tsc clean.
2026-05-08 02:20:27 +02:00
|
|
|
req: NextRequest,
|
feat(dedup): runtime surfaces — merge service, at-create suggestion, admin queue (P2)
Adds the live dedup pipeline on top of the P1 library + P3 migration
script. The new `client/interest` model now actively prevents duplicate
client records at creation time and gives admins a queue to triage
the borderline pairs the at-create check missed.
Three layers, per design §7:
Layer 1 — At-create suggestion
==============================
`GET /api/v1/clients/match-candidates`
Accepts free-text email / phone / name from the in-flight client
form, normalizes them via the dedup library, and returns scored
matches against the port's live client pool. Filters out
low-confidence noise (the background scoring queue picks those up
separately). Strict port scoping; never leaks across tenants.
`<DedupSuggestionPanel>` (`src/components/clients/dedup-suggestion-panel.tsx`)
Debounced React Query hook. Renders nothing for short inputs or
no useful match. On a high-confidence match it interrupts visually
with an amber-tinted card and a "Use this client" primary button.
Medium confidence falls back to a softer "possible match — check
before creating" treatment.
`<ClientForm>`
Renders the panel above the form (create path only — skipped on
edit). New `onUseExistingClient` callback fires when the user
picks the existing client; the form closes and the parent decides
what to do (typically: navigate to that client's detail page or
open the create-interest dialog pre-filled).
Layer 2 — Merge service
=======================
`mergeClients` (`src/lib/services/client-merge.service.ts`)
The atomic merge primitive that everything else calls. Single
transaction. Per §6 of the design:
- Locks both rows (FOR UPDATE) so concurrent merges of the same
loser fail with a clear error rather than racing.
- Snapshots the full loser state (contacts / addresses / notes /
tags / interest+reservation IDs / relationship rows) into the
`client_merge_log.merge_details` JSONB column for the eventual
undo flow.
- Reattaches every loser-side row to the winner: interests,
reservations, contacts (skipping duplicates by `(channel, value)`),
addresses, notes, tags (deduped), relationships.
- Optional `fieldChoices` — per-scalar overrides letting the user
keep the loser's value for fullName / nationality / preferences /
timezone / source.
- Marks the loser archived with `mergedIntoClientId` set (a redirect
pointer for stragglers; never hard-deleted within the undo window).
- Resolves any matching `client_merge_candidates` row to status='merged'.
- Writes audit log entry.
Schema additions:
- `clients.merged_into_client_id` (nullable text, indexed) — the
redirect pointer set on archive.
Tests: 6 cases against a real DB — happy path moves rows + writes log;
self-merge / cross-port / already-merged refused; duplicate-contact
deduped on reattach; fieldChoices copies loser values to winner.
Layer 3 — Admin review queue
============================
`GET /api/v1/admin/duplicates`
Pending merge candidates (status='pending') for the current port,
with both client summaries hydrated for side-by-side rendering.
Skips pairs where one side is already archived/merged.
`POST /api/v1/admin/duplicates/[id]/merge`
Confirms a candidate. Body picks the winner; the other side
becomes the loser. Calls into `mergeClients` — the only path that
writes `client_merge_log`.
`POST /api/v1/admin/duplicates/[id]/dismiss`
Marks the candidate dismissed. Future scoring runs skip the same
pair until a score change recreates the row.
`<DuplicatesReviewQueue>` (`/admin/duplicates`)
Side-by-side card UI for each pending pair. Click a card to pick
the winner; the other side is automatically the loser. Toolbar:
"Merge into selected" + "Dismiss". No per-field merge editor in
this PR — that's a future polish; the simple "pick the better row"
flow handles ~80% of cases.
Test coverage
=============
11 new integration tests (76 added in this branch total):
- 6 mergeClients (atomicity, refusal cases, contact dedup,
fieldChoices)
- 5 match-candidates API (shape, port scoping, confidence tiers,
Pattern F false-positive guard)
Full vitest: 926/926 passing (was 858 before the dedup branch).
Lint: clean. tsc: clean for new files (only pre-existing errors in
unrelated `tests/integration/` files remain, same as before this PR).
Out of scope, deferred
======================
- Background scoring cron that populates `client_merge_candidates`
(the queue is empty until this lands; manual seeding works for
now via the at-create flow).
- Side-by-side per-field merge editor with checkboxes (the simple
"pick the winner" UX shipped here covers ~80% of real cases).
- Admin settings UI for tuning the dedup thresholds. Defaults from
the design (90 / 50) are baked in for now.
- `unmergeClients` (the snapshot is captured in client_merge_log;
the undo endpoint just hasn't been wired yet).
These are all natural follow-up PRs that don't block shipping the
runtime UX.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:59:04 +02:00
|
|
|
ctx: AuthContext,
|
|
|
|
|
params: { id?: string },
|
|
|
|
|
): Promise<NextResponse> {
|
|
|
|
|
try {
|
|
|
|
|
const id = params.id ?? '';
|
fix(audit): non-Documenso backlog sweep — port-binding, NULLS NOT DISTINCT, custom merge tokens, company docs
Wave through the remaining audit-final-deferred items that aren't blocked
on the back-burnered Documenso work.
Multi-tenant isolation:
- Storage proxy ProxyTokenPayload gains optional `p` (port slug) claim;
verifier asserts `key.startsWith(${p}/)`. Defense-in-depth against a
buggy issuer in some future code path that mixes port scopes — every
storage key generated by generateStorageKey() already prefixes the
slug. document-sends opts in for 24h emailed download links; other
callers continue working unchanged via the optional field.
DB schema reconciliation:
- Migration 0047 rebuilds system_settings unique index with NULLS NOT
DISTINCT (Postgres 15+) so global settings (port_id IS NULL) are
uniquely keyed by `key` alone. Surfaced + dedupe'd 65 duplicate
(storage_backend, NULL) rows that had accumulated from race-prone
delete-then-insert patterns in ocr-config / settings / residential-
stages / ai-budget services. All four services converted to true
onConflictDoUpdate upserts so the race window is closed.
API uniformity:
- Response shape standardization: 16 routes converted from
`{ success: true }` to 204 No Content. CLAUDE.md documents the
convention (`{ data: <T> }` for content, 204 for empty mutations,
portal-auth retains `{ success: true }` for the frontend's auth chain).
- req.json() → parseBody() migration across 9 admin/CRM routes
(custom-fields, expenses/export ×3, currency convert,
search/recently-viewed, admin/duplicates, berths/pdf-{upload-url,
versions, parse-results}). Uniform 400 error shapes for
ZodError-flagged bodies.
Custom-fields merge tokens (shipped end-to-end):
- merge-fields.ts gains CUSTOM_MERGE_TOKEN_RE + helpers for the
`{{custom.<fieldName>}}` shape.
- document-templates validator accepts the dynamic shape alongside
the static catalog tokens.
- document-sends.service mergeCustomFieldValues resolver fetches
per-port custom_field_definitions for client/interest/berth contexts
and substitutes stored values keyed by `{{custom.fieldName}}`.
- custom-fields-manager amber banner updated to reflect that merge
tokens now expand (search index + entity-diff remain documented
design limitations).
/api/v1/files cross-entity filtering:
- Validator + listFiles + uploadFile accept companyId AND yachtId
alongside clientId. file-upload-zone propagates both.
- New CompanyFilesTab component mirrors ClientFilesTab; restored as a
visible Documents tab in company-tabs.tsx (was a hidden stub).
Inline TODOs:
- Reviewed remaining two TODOs (per-user reminder schedule, import
worker handlers). Both are placeholders for future feature surfaces,
not bugs — per-port digest works for every customer; nothing
currently enqueues import jobs (verified). Annotated in BACKLOG.
BACKLOG.md updated to reflect what landed and what's still pending
(Documenso-related items still bundled with the back-burnered phases).
Tests: 1185/1185 vitest, tsc clean.
2026-05-08 02:20:27 +02:00
|
|
|
const body = await parseBody(req, confirmMergeSchema);
|
feat(dedup): runtime surfaces — merge service, at-create suggestion, admin queue (P2)
Adds the live dedup pipeline on top of the P1 library + P3 migration
script. The new `client/interest` model now actively prevents duplicate
client records at creation time and gives admins a queue to triage
the borderline pairs the at-create check missed.
Three layers, per design §7:
Layer 1 — At-create suggestion
==============================
`GET /api/v1/clients/match-candidates`
Accepts free-text email / phone / name from the in-flight client
form, normalizes them via the dedup library, and returns scored
matches against the port's live client pool. Filters out
low-confidence noise (the background scoring queue picks those up
separately). Strict port scoping; never leaks across tenants.
`<DedupSuggestionPanel>` (`src/components/clients/dedup-suggestion-panel.tsx`)
Debounced React Query hook. Renders nothing for short inputs or
no useful match. On a high-confidence match it interrupts visually
with an amber-tinted card and a "Use this client" primary button.
Medium confidence falls back to a softer "possible match — check
before creating" treatment.
`<ClientForm>`
Renders the panel above the form (create path only — skipped on
edit). New `onUseExistingClient` callback fires when the user
picks the existing client; the form closes and the parent decides
what to do (typically: navigate to that client's detail page or
open the create-interest dialog pre-filled).
Layer 2 — Merge service
=======================
`mergeClients` (`src/lib/services/client-merge.service.ts`)
The atomic merge primitive that everything else calls. Single
transaction. Per §6 of the design:
- Locks both rows (FOR UPDATE) so concurrent merges of the same
loser fail with a clear error rather than racing.
- Snapshots the full loser state (contacts / addresses / notes /
tags / interest+reservation IDs / relationship rows) into the
`client_merge_log.merge_details` JSONB column for the eventual
undo flow.
- Reattaches every loser-side row to the winner: interests,
reservations, contacts (skipping duplicates by `(channel, value)`),
addresses, notes, tags (deduped), relationships.
- Optional `fieldChoices` — per-scalar overrides letting the user
keep the loser's value for fullName / nationality / preferences /
timezone / source.
- Marks the loser archived with `mergedIntoClientId` set (a redirect
pointer for stragglers; never hard-deleted within the undo window).
- Resolves any matching `client_merge_candidates` row to status='merged'.
- Writes audit log entry.
Schema additions:
- `clients.merged_into_client_id` (nullable text, indexed) — the
redirect pointer set on archive.
Tests: 6 cases against a real DB — happy path moves rows + writes log;
self-merge / cross-port / already-merged refused; duplicate-contact
deduped on reattach; fieldChoices copies loser values to winner.
Layer 3 — Admin review queue
============================
`GET /api/v1/admin/duplicates`
Pending merge candidates (status='pending') for the current port,
with both client summaries hydrated for side-by-side rendering.
Skips pairs where one side is already archived/merged.
`POST /api/v1/admin/duplicates/[id]/merge`
Confirms a candidate. Body picks the winner; the other side
becomes the loser. Calls into `mergeClients` — the only path that
writes `client_merge_log`.
`POST /api/v1/admin/duplicates/[id]/dismiss`
Marks the candidate dismissed. Future scoring runs skip the same
pair until a score change recreates the row.
`<DuplicatesReviewQueue>` (`/admin/duplicates`)
Side-by-side card UI for each pending pair. Click a card to pick
the winner; the other side is automatically the loser. Toolbar:
"Merge into selected" + "Dismiss". No per-field merge editor in
this PR — that's a future polish; the simple "pick the better row"
flow handles ~80% of cases.
Test coverage
=============
11 new integration tests (76 added in this branch total):
- 6 mergeClients (atomicity, refusal cases, contact dedup,
fieldChoices)
- 5 match-candidates API (shape, port scoping, confidence tiers,
Pattern F false-positive guard)
Full vitest: 926/926 passing (was 858 before the dedup branch).
Lint: clean. tsc: clean for new files (only pre-existing errors in
unrelated `tests/integration/` files remain, same as before this PR).
Out of scope, deferred
======================
- Background scoring cron that populates `client_merge_candidates`
(the queue is empty until this lands; manual seeding works for
now via the at-create flow).
- Side-by-side per-field merge editor with checkboxes (the simple
"pick the winner" UX shipped here covers ~80% of real cases).
- Admin settings UI for tuning the dedup thresholds. Defaults from
the design (90 / 50) are baked in for now.
- `unmergeClients` (the snapshot is captured in client_merge_log;
the undo endpoint just hasn't been wired yet).
These are all natural follow-up PRs that don't block shipping the
runtime UX.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:59:04 +02:00
|
|
|
|
|
|
|
|
const [candidate] = await db
|
|
|
|
|
.select()
|
|
|
|
|
.from(clientMergeCandidates)
|
|
|
|
|
.where(
|
|
|
|
|
and(
|
|
|
|
|
eq(clientMergeCandidates.id, id),
|
|
|
|
|
eq(clientMergeCandidates.portId, ctx.portId),
|
|
|
|
|
eq(clientMergeCandidates.status, 'pending'),
|
|
|
|
|
),
|
|
|
|
|
);
|
|
|
|
|
if (!candidate) throw new NotFoundError('Merge candidate');
|
|
|
|
|
|
|
|
|
|
const loserId =
|
|
|
|
|
body.winnerId === candidate.clientAId
|
|
|
|
|
? candidate.clientBId
|
|
|
|
|
: body.winnerId === candidate.clientBId
|
|
|
|
|
? candidate.clientAId
|
|
|
|
|
: null;
|
|
|
|
|
if (!loserId) {
|
fix(audit-tier-2-routes): manual NextResponse.json error sweep + admin form banners
Two final waves of error-surface hygiene closing the audit's MED §12 +
HIGH §15 + HIGH §17 findings:
* 50 route files swept (61 sites): manual NextResponse.json({error,
status: 4xx|5xx}) early-returns replaced by typed throws +
errorResponse(err) at the catch.
- Super-admin gates (13 sites) use new requireSuperAdmin(ctx, action)
helper from src/lib/api/helpers.ts so denials hit the audit log.
- Path-param + body validation 400s become ValidationError throws.
- 404s become NotFoundError or CodedError('NOT_FOUND') for AI
feature-flag paths.
- 11 manual 5xx returns now re-throw so error_events captures the
request-id (the admin error inspector becomes usable from real
incidents).
- website-analytics 200-with-error anti-pattern flipped to 409 +
UMAMI_NOT_CONFIGURED. 502 upstream paths use UMAMI_UPSTREAM_ERROR.
- 11 sites intentionally preserved: storage/[token] anti-enumeration
token-failure paths, webhook-secret 401, "Unknown port" 400 in
public intake.
* 7 admin forms (roles, users, ports, webhooks, custom-fields,
document-templates, tags) gain a formatErrorBanner() helper from
src/lib/api/toast-error.ts that builds a multi-line "Error code / Reference ID"
banner — the rep can copy the request id when reporting a failed
save. Banners get whitespace-pre-line so newlines render.
Test status: 1168/1168 vitest, tsc clean.
Refs: docs/audit-comprehensive-2026-05-05.md MED §12 (auditor-F Issue 1)
+ HIGH §15 (auditor-F Issue 2) + HIGH §17 (auditor-H Issue 2).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 20:36:59 +02:00
|
|
|
throw new ValidationError('winnerId must match one of the candidate clients');
|
feat(dedup): runtime surfaces — merge service, at-create suggestion, admin queue (P2)
Adds the live dedup pipeline on top of the P1 library + P3 migration
script. The new `client/interest` model now actively prevents duplicate
client records at creation time and gives admins a queue to triage
the borderline pairs the at-create check missed.
Three layers, per design §7:
Layer 1 — At-create suggestion
==============================
`GET /api/v1/clients/match-candidates`
Accepts free-text email / phone / name from the in-flight client
form, normalizes them via the dedup library, and returns scored
matches against the port's live client pool. Filters out
low-confidence noise (the background scoring queue picks those up
separately). Strict port scoping; never leaks across tenants.
`<DedupSuggestionPanel>` (`src/components/clients/dedup-suggestion-panel.tsx`)
Debounced React Query hook. Renders nothing for short inputs or
no useful match. On a high-confidence match it interrupts visually
with an amber-tinted card and a "Use this client" primary button.
Medium confidence falls back to a softer "possible match — check
before creating" treatment.
`<ClientForm>`
Renders the panel above the form (create path only — skipped on
edit). New `onUseExistingClient` callback fires when the user
picks the existing client; the form closes and the parent decides
what to do (typically: navigate to that client's detail page or
open the create-interest dialog pre-filled).
Layer 2 — Merge service
=======================
`mergeClients` (`src/lib/services/client-merge.service.ts`)
The atomic merge primitive that everything else calls. Single
transaction. Per §6 of the design:
- Locks both rows (FOR UPDATE) so concurrent merges of the same
loser fail with a clear error rather than racing.
- Snapshots the full loser state (contacts / addresses / notes /
tags / interest+reservation IDs / relationship rows) into the
`client_merge_log.merge_details` JSONB column for the eventual
undo flow.
- Reattaches every loser-side row to the winner: interests,
reservations, contacts (skipping duplicates by `(channel, value)`),
addresses, notes, tags (deduped), relationships.
- Optional `fieldChoices` — per-scalar overrides letting the user
keep the loser's value for fullName / nationality / preferences /
timezone / source.
- Marks the loser archived with `mergedIntoClientId` set (a redirect
pointer for stragglers; never hard-deleted within the undo window).
- Resolves any matching `client_merge_candidates` row to status='merged'.
- Writes audit log entry.
Schema additions:
- `clients.merged_into_client_id` (nullable text, indexed) — the
redirect pointer set on archive.
Tests: 6 cases against a real DB — happy path moves rows + writes log;
self-merge / cross-port / already-merged refused; duplicate-contact
deduped on reattach; fieldChoices copies loser values to winner.
Layer 3 — Admin review queue
============================
`GET /api/v1/admin/duplicates`
Pending merge candidates (status='pending') for the current port,
with both client summaries hydrated for side-by-side rendering.
Skips pairs where one side is already archived/merged.
`POST /api/v1/admin/duplicates/[id]/merge`
Confirms a candidate. Body picks the winner; the other side
becomes the loser. Calls into `mergeClients` — the only path that
writes `client_merge_log`.
`POST /api/v1/admin/duplicates/[id]/dismiss`
Marks the candidate dismissed. Future scoring runs skip the same
pair until a score change recreates the row.
`<DuplicatesReviewQueue>` (`/admin/duplicates`)
Side-by-side card UI for each pending pair. Click a card to pick
the winner; the other side is automatically the loser. Toolbar:
"Merge into selected" + "Dismiss". No per-field merge editor in
this PR — that's a future polish; the simple "pick the better row"
flow handles ~80% of cases.
Test coverage
=============
11 new integration tests (76 added in this branch total):
- 6 mergeClients (atomicity, refusal cases, contact dedup,
fieldChoices)
- 5 match-candidates API (shape, port scoping, confidence tiers,
Pattern F false-positive guard)
Full vitest: 926/926 passing (was 858 before the dedup branch).
Lint: clean. tsc: clean for new files (only pre-existing errors in
unrelated `tests/integration/` files remain, same as before this PR).
Out of scope, deferred
======================
- Background scoring cron that populates `client_merge_candidates`
(the queue is empty until this lands; manual seeding works for
now via the at-create flow).
- Side-by-side per-field merge editor with checkboxes (the simple
"pick the winner" UX shipped here covers ~80% of real cases).
- Admin settings UI for tuning the dedup thresholds. Defaults from
the design (90 / 50) are baked in for now.
- `unmergeClients` (the snapshot is captured in client_merge_log;
the undo endpoint just hasn't been wired yet).
These are all natural follow-up PRs that don't block shipping the
runtime UX.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:59:04 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
const result = await mergeClients({
|
|
|
|
|
winnerId: body.winnerId,
|
|
|
|
|
loserId,
|
|
|
|
|
mergedBy: ctx.userId,
|
fix(audit-tier-4): tenant-isolation defense-in-depth
Closes the audit's HIGH §10 + MED §§17–22 isolation footguns. None of
these are user-impactful TODAY — every site is preceded by a port-
scoped read or pre-validated by ctx.portId — but each is a future-
refactor accident waiting to happen, so the SQL itself now pins the
tenant boundary:
* mergeClients gains a callerPortId option; the route caller passes
ctx.portId. removeInterestBerth now requires portId and verifies
both the interest and the berth share it before deleting the
junction row. All three callers updated.
* Six service mutations now scope the WHERE to (id, portId):
form-templates update + delete, invoices.detectOverdue per-row
update, notifications.markRead, clients.deleteRelationship.
company-memberships uses an inArray sub-select against port
companies (no port_id column on the table itself), covering
updateMembership / endMembership / setPrimary.
* Port-scoped file lookups in portal.getDocumentDownloadUrl,
reports.getDownloadUrl (file presign), berth-reservations.activate
(contractFileId attach guard), and residential.getResidentialInterestById
(residentialClient join).
Test status: 1168/1168 vitest, tsc clean.
Refs: docs/audit-comprehensive-2026-05-05.md HIGH §10 + MED §§17–22
(auditor-B3 Issues 1–5,7).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 20:48:13 +02:00
|
|
|
callerPortId: ctx.portId,
|
fix(audit): non-Documenso backlog sweep — port-binding, NULLS NOT DISTINCT, custom merge tokens, company docs
Wave through the remaining audit-final-deferred items that aren't blocked
on the back-burnered Documenso work.
Multi-tenant isolation:
- Storage proxy ProxyTokenPayload gains optional `p` (port slug) claim;
verifier asserts `key.startsWith(${p}/)`. Defense-in-depth against a
buggy issuer in some future code path that mixes port scopes — every
storage key generated by generateStorageKey() already prefixes the
slug. document-sends opts in for 24h emailed download links; other
callers continue working unchanged via the optional field.
DB schema reconciliation:
- Migration 0047 rebuilds system_settings unique index with NULLS NOT
DISTINCT (Postgres 15+) so global settings (port_id IS NULL) are
uniquely keyed by `key` alone. Surfaced + dedupe'd 65 duplicate
(storage_backend, NULL) rows that had accumulated from race-prone
delete-then-insert patterns in ocr-config / settings / residential-
stages / ai-budget services. All four services converted to true
onConflictDoUpdate upserts so the race window is closed.
API uniformity:
- Response shape standardization: 16 routes converted from
`{ success: true }` to 204 No Content. CLAUDE.md documents the
convention (`{ data: <T> }` for content, 204 for empty mutations,
portal-auth retains `{ success: true }` for the frontend's auth chain).
- req.json() → parseBody() migration across 9 admin/CRM routes
(custom-fields, expenses/export ×3, currency convert,
search/recently-viewed, admin/duplicates, berths/pdf-{upload-url,
versions, parse-results}). Uniform 400 error shapes for
ZodError-flagged bodies.
Custom-fields merge tokens (shipped end-to-end):
- merge-fields.ts gains CUSTOM_MERGE_TOKEN_RE + helpers for the
`{{custom.<fieldName>}}` shape.
- document-templates validator accepts the dynamic shape alongside
the static catalog tokens.
- document-sends.service mergeCustomFieldValues resolver fetches
per-port custom_field_definitions for client/interest/berth contexts
and substitutes stored values keyed by `{{custom.fieldName}}`.
- custom-fields-manager amber banner updated to reflect that merge
tokens now expand (search index + entity-diff remain documented
design limitations).
/api/v1/files cross-entity filtering:
- Validator + listFiles + uploadFile accept companyId AND yachtId
alongside clientId. file-upload-zone propagates both.
- New CompanyFilesTab component mirrors ClientFilesTab; restored as a
visible Documents tab in company-tabs.tsx (was a hidden stub).
Inline TODOs:
- Reviewed remaining two TODOs (per-user reminder schedule, import
worker handlers). Both are placeholders for future feature surfaces,
not bugs — per-port digest works for every customer; nothing
currently enqueues import jobs (verified). Annotated in BACKLOG.
BACKLOG.md updated to reflect what landed and what's still pending
(Documenso-related items still bundled with the back-burnered phases).
Tests: 1185/1185 vitest, tsc clean.
2026-05-08 02:20:27 +02:00
|
|
|
fieldChoices: body.fieldChoices as MergeFieldChoices | undefined,
|
feat(dedup): runtime surfaces — merge service, at-create suggestion, admin queue (P2)
Adds the live dedup pipeline on top of the P1 library + P3 migration
script. The new `client/interest` model now actively prevents duplicate
client records at creation time and gives admins a queue to triage
the borderline pairs the at-create check missed.
Three layers, per design §7:
Layer 1 — At-create suggestion
==============================
`GET /api/v1/clients/match-candidates`
Accepts free-text email / phone / name from the in-flight client
form, normalizes them via the dedup library, and returns scored
matches against the port's live client pool. Filters out
low-confidence noise (the background scoring queue picks those up
separately). Strict port scoping; never leaks across tenants.
`<DedupSuggestionPanel>` (`src/components/clients/dedup-suggestion-panel.tsx`)
Debounced React Query hook. Renders nothing for short inputs or
no useful match. On a high-confidence match it interrupts visually
with an amber-tinted card and a "Use this client" primary button.
Medium confidence falls back to a softer "possible match — check
before creating" treatment.
`<ClientForm>`
Renders the panel above the form (create path only — skipped on
edit). New `onUseExistingClient` callback fires when the user
picks the existing client; the form closes and the parent decides
what to do (typically: navigate to that client's detail page or
open the create-interest dialog pre-filled).
Layer 2 — Merge service
=======================
`mergeClients` (`src/lib/services/client-merge.service.ts`)
The atomic merge primitive that everything else calls. Single
transaction. Per §6 of the design:
- Locks both rows (FOR UPDATE) so concurrent merges of the same
loser fail with a clear error rather than racing.
- Snapshots the full loser state (contacts / addresses / notes /
tags / interest+reservation IDs / relationship rows) into the
`client_merge_log.merge_details` JSONB column for the eventual
undo flow.
- Reattaches every loser-side row to the winner: interests,
reservations, contacts (skipping duplicates by `(channel, value)`),
addresses, notes, tags (deduped), relationships.
- Optional `fieldChoices` — per-scalar overrides letting the user
keep the loser's value for fullName / nationality / preferences /
timezone / source.
- Marks the loser archived with `mergedIntoClientId` set (a redirect
pointer for stragglers; never hard-deleted within the undo window).
- Resolves any matching `client_merge_candidates` row to status='merged'.
- Writes audit log entry.
Schema additions:
- `clients.merged_into_client_id` (nullable text, indexed) — the
redirect pointer set on archive.
Tests: 6 cases against a real DB — happy path moves rows + writes log;
self-merge / cross-port / already-merged refused; duplicate-contact
deduped on reattach; fieldChoices copies loser values to winner.
Layer 3 — Admin review queue
============================
`GET /api/v1/admin/duplicates`
Pending merge candidates (status='pending') for the current port,
with both client summaries hydrated for side-by-side rendering.
Skips pairs where one side is already archived/merged.
`POST /api/v1/admin/duplicates/[id]/merge`
Confirms a candidate. Body picks the winner; the other side
becomes the loser. Calls into `mergeClients` — the only path that
writes `client_merge_log`.
`POST /api/v1/admin/duplicates/[id]/dismiss`
Marks the candidate dismissed. Future scoring runs skip the same
pair until a score change recreates the row.
`<DuplicatesReviewQueue>` (`/admin/duplicates`)
Side-by-side card UI for each pending pair. Click a card to pick
the winner; the other side is automatically the loser. Toolbar:
"Merge into selected" + "Dismiss". No per-field merge editor in
this PR — that's a future polish; the simple "pick the better row"
flow handles ~80% of cases.
Test coverage
=============
11 new integration tests (76 added in this branch total):
- 6 mergeClients (atomicity, refusal cases, contact dedup,
fieldChoices)
- 5 match-candidates API (shape, port scoping, confidence tiers,
Pattern F false-positive guard)
Full vitest: 926/926 passing (was 858 before the dedup branch).
Lint: clean. tsc: clean for new files (only pre-existing errors in
unrelated `tests/integration/` files remain, same as before this PR).
Out of scope, deferred
======================
- Background scoring cron that populates `client_merge_candidates`
(the queue is empty until this lands; manual seeding works for
now via the at-create flow).
- Side-by-side per-field merge editor with checkboxes (the simple
"pick the winner" UX shipped here covers ~80% of real cases).
- Admin settings UI for tuning the dedup thresholds. Defaults from
the design (90 / 50) are baked in for now.
- `unmergeClients` (the snapshot is captured in client_merge_log;
the undo endpoint just hasn't been wired yet).
These are all natural follow-up PRs that don't block shipping the
runtime UX.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 14:59:04 +02:00
|
|
|
});
|
|
|
|
|
|
|
|
|
|
return NextResponse.json({ data: result });
|
|
|
|
|
} catch (error) {
|
|
|
|
|
return errorResponse(error);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* POST /api/v1/admin/duplicates/[id]/dismiss
|
|
|
|
|
*
|
|
|
|
|
* Mark a merge candidate as dismissed. The background scoring job
|
|
|
|
|
* skips dismissed pairs on subsequent runs (a future score increase
|
|
|
|
|
* can re-create them).
|
|
|
|
|
*/
|
|
|
|
|
export async function dismissHandler(
|
|
|
|
|
_req: Request,
|
|
|
|
|
ctx: AuthContext,
|
|
|
|
|
params: { id?: string },
|
|
|
|
|
): Promise<NextResponse> {
|
|
|
|
|
try {
|
|
|
|
|
const id = params.id ?? '';
|
|
|
|
|
const result = await db
|
|
|
|
|
.update(clientMergeCandidates)
|
|
|
|
|
.set({
|
|
|
|
|
status: 'dismissed',
|
|
|
|
|
resolvedAt: new Date(),
|
|
|
|
|
resolvedBy: ctx.userId,
|
|
|
|
|
})
|
|
|
|
|
.where(
|
|
|
|
|
and(
|
|
|
|
|
eq(clientMergeCandidates.id, id),
|
|
|
|
|
eq(clientMergeCandidates.portId, ctx.portId),
|
|
|
|
|
eq(clientMergeCandidates.status, 'pending'),
|
|
|
|
|
),
|
|
|
|
|
)
|
|
|
|
|
.returning({ id: clientMergeCandidates.id });
|
|
|
|
|
|
|
|
|
|
if (result.length === 0) throw new NotFoundError('Merge candidate');
|
|
|
|
|
return NextResponse.json({ data: { id: result[0]!.id, status: 'dismissed' } });
|
|
|
|
|
} catch (error) {
|
|
|
|
|
return errorResponse(error);
|
|
|
|
|
}
|
|
|
|
|
}
|