letsbe/pn-new-crm
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(dedup): runtime surfaces — merge service, at-create suggestion, admin queue (P2)

Browse Source 
Adds the live dedup pipeline on top of the P1 library + P3 migration
script. The new `client/interest` model now actively prevents duplicate
client records at creation time and gives admins a queue to triage
the borderline pairs the at-create check missed.

Three layers, per design §7:

Layer 1 — At-create suggestion
==============================

`GET /api/v1/clients/match-candidates`
  Accepts free-text email / phone / name from the in-flight client
  form, normalizes them via the dedup library, and returns scored
  matches against the port's live client pool. Filters out
  low-confidence noise (the background scoring queue picks those up
  separately). Strict port scoping; never leaks across tenants.

`<DedupSuggestionPanel>` (`src/components/clients/dedup-suggestion-panel.tsx`)
  Debounced React Query hook. Renders nothing for short inputs or
  no useful match. On a high-confidence match it interrupts visually
  with an amber-tinted card and a "Use this client" primary button.
  Medium confidence falls back to a softer "possible match — check
  before creating" treatment.

`<ClientForm>`
  Renders the panel above the form (create path only — skipped on
  edit). New `onUseExistingClient` callback fires when the user
  picks the existing client; the form closes and the parent decides
  what to do (typically: navigate to that client's detail page or
  open the create-interest dialog pre-filled).

Layer 2 — Merge service
=======================

`mergeClients` (`src/lib/services/client-merge.service.ts`)
  The atomic merge primitive that everything else calls. Single
  transaction. Per §6 of the design:

  - Locks both rows (FOR UPDATE) so concurrent merges of the same
    loser fail with a clear error rather than racing.
  - Snapshots the full loser state (contacts / addresses / notes /
    tags / interest+reservation IDs / relationship rows) into the
    `client_merge_log.merge_details` JSONB column for the eventual
    undo flow.
  - Reattaches every loser-side row to the winner: interests,
    reservations, contacts (skipping duplicates by `(channel, value)`),
    addresses, notes, tags (deduped), relationships.
  - Optional `fieldChoices` — per-scalar overrides letting the user
    keep the loser's value for fullName / nationality / preferences /
    timezone / source.
  - Marks the loser archived with `mergedIntoClientId` set (a redirect
    pointer for stragglers; never hard-deleted within the undo window).
  - Resolves any matching `client_merge_candidates` row to status='merged'.
  - Writes audit log entry.

Schema additions:
  - `clients.merged_into_client_id` (nullable text, indexed) — the
    redirect pointer set on archive.

Tests: 6 cases against a real DB — happy path moves rows + writes log;
self-merge / cross-port / already-merged refused; duplicate-contact
deduped on reattach; fieldChoices copies loser values to winner.

Layer 3 — Admin review queue
============================

`GET /api/v1/admin/duplicates`
  Pending merge candidates (status='pending') for the current port,
  with both client summaries hydrated for side-by-side rendering.
  Skips pairs where one side is already archived/merged.

`POST /api/v1/admin/duplicates/[id]/merge`
  Confirms a candidate. Body picks the winner; the other side
  becomes the loser. Calls into `mergeClients` — the only path that
  writes `client_merge_log`.

`POST /api/v1/admin/duplicates/[id]/dismiss`
  Marks the candidate dismissed. Future scoring runs skip the same
  pair until a score change recreates the row.

`<DuplicatesReviewQueue>` (`/admin/duplicates`)
  Side-by-side card UI for each pending pair. Click a card to pick
  the winner; the other side is automatically the loser. Toolbar:
  "Merge into selected" + "Dismiss". No per-field merge editor in
  this PR — that's a future polish; the simple "pick the better row"
  flow handles ~80% of cases.

Test coverage
=============

11 new integration tests (76 added in this branch total):
  - 6 mergeClients (atomicity, refusal cases, contact dedup,
    fieldChoices)
  - 5 match-candidates API (shape, port scoping, confidence tiers,
    Pattern F false-positive guard)

Full vitest: 926/926 passing (was 858 before the dedup branch).
Lint: clean. tsc: clean for new files (only pre-existing errors in
unrelated `tests/integration/` files remain, same as before this PR).

Out of scope, deferred
======================

- Background scoring cron that populates `client_merge_candidates`
  (the queue is empty until this lands; manual seeding works for
  now via the at-create flow).
- Side-by-side per-field merge editor with checkboxes (the simple
  "pick the winner" UX shipped here covers ~80% of real cases).
- Admin settings UI for tuning the dedup thresholds. Defaults from
  the design (90 / 50) are baked in for now.
- `unmergeClients` (the snapshot is captured in client_merge_log;
  the undo endpoint just hasn't been wired yet).

These are all natural follow-up PRs that don't block shipping the
runtime UX.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Ciaccio
2026-05-03 14:59:04 +02:00
parent 18e5c124b0
commit 4bcc7f8be6
17 changed files with 12018 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

215
src/components/admin/duplicates/duplicates-review-queue.tsx Normal file
View File

@@ -0,0 +1,215 @@
'use client';
import { useState } from 'react';
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
import { ArrowRight, GitMerge, X } from 'lucide-react';
import { toast } from 'sonner';
import { Button } from '@/components/ui/button';
import { PageHeader } from '@/components/shared/page-header';
import { EmptyState } from '@/components/shared/empty-state';
import { Skeleton } from '@/components/ui/skeleton';
import { apiFetch } from '@/lib/api/client';
import { cn } from '@/lib/utils';
interface CandidatePair {
id: string;
score: number;
reasons: string[];
createdAt: string;
clientA: { id: string; fullName: string; createdAt: string };
clientB: { id: string; fullName: string; createdAt: string };
}
/**
* Admin review queue for the dedup background scoring job.
*
* Lists every pending merge candidate (pairs where score >=
* `dedup_review_queue_threshold`). For each pair the admin can:
* - Pick a winner via the side-by-side card → confirms a merge
* - Dismiss → removes from the queue (a future score increase
* re-creates the pair on the next scoring run)
*
* Only minimal merge UI here: the user picks which side is the winner
* (no per-field choice), and the loser archives. A richer side-by-side
* field-merge dialog is a future enhancement.
*/
export function DuplicatesReviewQueue() {
const queryClient = useQueryClient();
const { data, isLoading } = useQuery<{ data: CandidatePair[] }>({
queryKey: ['admin', 'duplicates'],
queryFn: () => apiFetch<{ data: CandidatePair[] }>('/api/v1/admin/duplicates'),
});
const pairs = data?.data ?? [];
return (
<div className="space-y-4">
<PageHeader
title="Duplicate clients"
description={
pairs.length === 0
? 'No pending pairs to review.'
: `${pairs.length} pair${pairs.length === 1 ? '' : 's'} flagged for review.`
}
/>
{isLoading ? (
<div className="space-y-3">
{[0, 1, 2].map((i) => (
<Skeleton key={i} className="h-32 w-full" />
))}
</div>
) : pairs.length === 0 ? (
<EmptyState
title="All clear"
description="The background scoring job hasn't surfaced any potential duplicates yet."
/>
) : (
<ul className="space-y-3">
{pairs.map((pair) => (
<li key={pair.id}>
<CandidateRow pair={pair} queryClient={queryClient} />
</li>
))}
</ul>
)}
</div>
);
}
function CandidateRow({
pair,
queryClient,
}: {
pair: CandidatePair;
queryClient: ReturnType<typeof useQueryClient>;
}) {
const [busy, setBusy] = useState<'merge' | 'dismiss' | null>(null);
const [winnerId, setWinnerId] = useState<string>(pair.clientA.id);
const mergeMutation = useMutation({
mutationFn: () =>
apiFetch(`/api/v1/admin/duplicates/${pair.id}/merge`, {
method: 'POST',
body: { winnerId },
}),
onSuccess: () => {
const loserName =
winnerId === pair.clientA.id ? pair.clientB.fullName : pair.clientA.fullName;
const winnerName =
winnerId === pair.clientA.id ? pair.clientA.fullName : pair.clientB.fullName;
toast.success(`Merged "${loserName}" into "${winnerName}"`);
queryClient.invalidateQueries({ queryKey: ['admin', 'duplicates'] });
queryClient.invalidateQueries({ queryKey: ['clients'] });
},
onError: (err) => toast.error(err instanceof Error ? err.message : 'Merge failed'),
onSettled: () => setBusy(null),
});
const dismissMutation = useMutation({
mutationFn: () => apiFetch(`/api/v1/admin/duplicates/${pair.id}/dismiss`, { method: 'POST' }),
onSuccess: () => {
toast.message('Dismissed');
queryClient.invalidateQueries({ queryKey: ['admin', 'duplicates'] });
},
onError: (err) => toast.error(err instanceof Error ? err.message : 'Dismiss failed'),
onSettled: () => setBusy(null),
});
return (
<div className="rounded-lg border bg-card p-4">
<div className="mb-3 flex items-baseline justify-between gap-3">
<div>
<span className="rounded-full bg-muted px-2 py-0.5 text-[10px] font-medium uppercase tracking-wide text-muted-foreground">
score {pair.score}
</span>{' '}
<span className="text-xs text-muted-foreground">{pair.reasons.join(' · ')}</span>
</div>
<span className="text-xs text-muted-foreground">
flagged {new Date(pair.createdAt).toLocaleDateString()}
</span>
</div>
<div className="grid gap-3 sm:grid-cols-[1fr_auto_1fr]">
<ClientCard
client={pair.clientA}
isSelected={winnerId === pair.clientA.id}
onSelect={() => setWinnerId(pair.clientA.id)}
/>
<div className="flex items-center justify-center text-muted-foreground">
<ArrowRight className="size-4" aria-hidden />
</div>
<ClientCard
client={pair.clientB}
isSelected={winnerId === pair.clientB.id}
onSelect={() => setWinnerId(pair.clientB.id)}
/>
</div>
<div className="mt-3 flex flex-wrap items-center gap-2">
<Button
size="sm"
onClick={() => {
setBusy('merge');
mergeMutation.mutate();
}}
disabled={busy !== null}
>
<GitMerge className="mr-1 size-3.5" aria-hidden />
Merge into selected
</Button>
<Button
size="sm"
variant="ghost"
onClick={() => {
setBusy('dismiss');
dismissMutation.mutate();
}}
disabled={busy !== null}
>
<X className="mr-1 size-3.5" aria-hidden />
Dismiss
</Button>
<p className="text-xs text-muted-foreground">
The unselected card becomes the loser; its interests + contacts move to the selected
client and the original is archived.
</p>
</div>
</div>
);
}
function ClientCard({
client,
isSelected,
onSelect,
}: {
client: CandidatePair['clientA'];
isSelected: boolean;
onSelect: () => void;
}) {
return (
<button
type="button"
onClick={onSelect}
className={cn(
'rounded-md border p-3 text-left transition-colors',
isSelected
? 'border-primary bg-primary/5 ring-1 ring-primary/30'
: 'border-border hover:bg-muted/40',
)}
>
<p className="text-sm font-medium">{client.fullName}</p>
<p className="mt-0.5 text-[11px] text-muted-foreground">
Created {new Date(client.createdAt).toLocaleDateString()}
</p>
{isSelected ? (
<span className="mt-1 inline-block rounded-full bg-primary/10 px-1.5 py-0.5 text-[10px] font-semibold text-primary">
KEEP
</span>
) : null}
</button>
);
}

29
src/components/clients/client-form.tsx
View File

@@ -23,6 +23,7 @@ import { TagPicker } from '@/components/shared/tag-picker';
import { CountryCombobox } from '@/components/shared/country-combobox';
import { TimezoneCombobox } from '@/components/shared/timezone-combobox';
import { PhoneInput } from '@/components/shared/phone-input';
import { DedupSuggestionPanel } from '@/components/clients/dedup-suggestion-panel';
import { apiFetch } from '@/lib/api/client';
import { createClientSchema, type CreateClientInput } from '@/lib/validators/clients';
import type { CountryCode } from '@/lib/i18n/countries';
@@ -30,6 +31,12 @@ import type { CountryCode } from '@/lib/i18n/countries';
interface ClientFormProps {
open: boolean;
onOpenChange: (open: boolean) => void;
/** Optional callback fired when the dedup suggestion panel reports
* the user picked an existing client. The form closes; parent is
* responsible for navigating to the existing client's detail page
* or opening the create-interest dialog pre-filled with that
* clientId. Skipped in edit mode. */
onUseExistingClient?: (clientId: string) => void;
/** If provided, form is in edit mode */
client?: {
id: string;
@@ -53,7 +60,7 @@ interface ClientFormProps {
};
}
export function ClientForm({ open, onOpenChange, client }: ClientFormProps) {
export function ClientForm({ open, onOpenChange, client, onUseExistingClient }: ClientFormProps) {
const queryClient = useQueryClient();
const isEdit = !!client;
@@ -143,6 +150,26 @@ export function ClientForm({ open, onOpenChange, client }: ClientFormProps) {
</SheetHeader>
<form onSubmit={handleSubmit((data) => mutation.mutate(data))} className="space-y-6 py-6">
{/* Dedup suggestion — only on the create path. Watches the
live form values for email / phone / name and surfaces
an existing client when one matches. The user can
attach the new interest to that client instead of
creating a duplicate. */}
{!isEdit ? (
<DedupSuggestionPanel
email={watch('contacts')?.find((c) => c?.channel === 'email')?.value ?? null}
phone={
watch('contacts')?.find((c) => c?.channel === 'phone' || c?.channel === 'whatsapp')
?.valueE164 ?? null
}
name={watch('fullName') ?? null}
onUseExisting={(match) => {
onUseExistingClient?.(match.clientId);
onOpenChange(false);
}}
/>
) : null}
{/* Basic Info */}
<div className="space-y-4">
<h3 className="text-sm font-medium text-muted-foreground uppercase tracking-wide">

183
src/components/clients/dedup-suggestion-panel.tsx Normal file
View File

@@ -0,0 +1,183 @@
'use client';
import { useEffect, useState } from 'react';
import { useQuery } from '@tanstack/react-query';
import { AlertCircle, ArrowRight, Briefcase, X } from 'lucide-react';
import { Button } from '@/components/ui/button';
import { apiFetch } from '@/lib/api/client';
import { cn } from '@/lib/utils';
interface MatchData {
clientId: string;
fullName: string;
score: number;
confidence: 'high' | 'medium' | 'low';
reasons: string[];
interestCount: number;
emails: string[];
phonesE164: string[];
}
interface DedupSuggestionPanelProps {
/** Free-text inputs from the in-flight new-client form. The panel
* debounces them and queries /api/v1/clients/match-candidates. */
email?: string | null;
phone?: string | null;
name?: string | null;
/** Caller wants to attach the new interest to an existing client
* rather than creating a new one. The form switches to
* interest-only mode and pre-fills the client. */
onUseExisting: (match: MatchData) => void;
/** User explicitly said "create new anyway." Hide the panel until
* they change input again. */
onDismiss?: () => void;
}
/**
* Surfaces existing clients that match the form's in-flight inputs.
*
* Renders nothing while inputs are short / no useful match found.
* On a high-confidence match, the panel interrupts visually with a
* solid border and a primary "Use this client" button.
*
* Wired into the new-client form. Skipped in edit mode.
*/
export function DedupSuggestionPanel({
email,
phone,
name,
onUseExisting,
onDismiss,
}: DedupSuggestionPanelProps) {
const [dismissed, setDismissed] = useState(false);
// Debounce inputs by 300ms so we don't fire on every keystroke. Keep
// the latest debounced values in component state.
const [debounced, setDebounced] = useState({
email: email ?? '',
phone: phone ?? '',
name: name ?? '',
});
useEffect(() => {
const t = setTimeout(() => {
setDebounced({ email: email ?? '', phone: phone ?? '', name: name ?? '' });
// Clear the dismissed flag when inputs change — the user typed
// something new, so the prior dismissal no longer applies.
setDismissed(false);
}, 300);
return () => clearTimeout(t);
}, [email, phone, name]);
const hasSomething =
debounced.email.length > 3 || debounced.phone.length > 3 || debounced.name.length > 2;
const { data, isFetching } = useQuery<{ data: MatchData[] }>({
queryKey: ['dedup-match-candidates', debounced],
queryFn: () => {
const params = new URLSearchParams();
if (debounced.email) params.set('email', debounced.email);
if (debounced.phone) params.set('phone', debounced.phone);
if (debounced.name) params.set('name', debounced.name);
return apiFetch<{ data: MatchData[] }>(`/api/v1/clients/match-candidates?${params}`);
},
enabled: hasSomething && !dismissed,
// Same query is fine to cache for a minute — moves are slow at this layer.
staleTime: 60_000,
});
if (dismissed) return null;
if (!hasSomething) return null;
if (isFetching && !data) return null;
const matches = data?.data ?? [];
if (matches.length === 0) return null;
const top = matches[0]!;
const isHigh = top.confidence === 'high';
return (
<div
className={cn(
'rounded-lg border p-3 mb-3 transition-colors',
isHigh
? 'border-amber-300 bg-amber-50/60 dark:bg-amber-950/30'
: 'border-border bg-muted/40',
)}
data-testid="dedup-suggestion"
>
<div className="flex items-start gap-3">
<div className="mt-0.5">
<AlertCircle
className={cn(
'size-5',
isHigh ? 'text-amber-700 dark:text-amber-400' : 'text-muted-foreground',
)}
aria-hidden
/>
</div>
<div className="min-w-0 flex-1">
<p className="text-sm font-semibold leading-tight">
{isHigh
? 'This looks like an existing client'
: 'Possible match — check before creating'}
</p>
<div className="mt-2 rounded-md border bg-background/80 p-2.5">
<div className="flex items-center gap-2">
<p className="truncate text-sm font-medium">{top.fullName}</p>
<span
className={cn(
'shrink-0 rounded-full px-1.5 py-0.5 text-[10px] font-medium uppercase tracking-wide',
isHigh
? 'bg-amber-200 text-amber-900 dark:bg-amber-800 dark:text-amber-100'
: 'bg-muted text-muted-foreground',
)}
>
{top.confidence}
</span>
</div>
<div className="mt-0.5 flex flex-wrap items-center gap-x-3 gap-y-0.5 text-xs text-muted-foreground">
{top.emails[0] ? <span className="truncate">{top.emails[0]}</span> : null}
{top.phonesE164[0] ? <span>{top.phonesE164[0]}</span> : null}
<span className="inline-flex items-center gap-1">
<Briefcase className="size-3" aria-hidden />
{top.interestCount} {top.interestCount === 1 ? 'interest' : 'interests'}
</span>
</div>
<p className="mt-1.5 text-[11px] text-muted-foreground">{top.reasons.join(' · ')}</p>
</div>
<div className="mt-3 flex flex-wrap items-center gap-2">
<Button
type="button"
size="sm"
onClick={() => onUseExisting(top)}
data-testid="dedup-use-existing"
>
Use this client
<ArrowRight className="ml-1 size-3.5" aria-hidden />
</Button>
<Button
type="button"
size="sm"
variant="ghost"
onClick={() => {
setDismissed(true);
onDismiss?.();
}}
data-testid="dedup-dismiss"
>
<X className="mr-1 size-3.5" aria-hidden />
Create new anyway
</Button>
{matches.length > 1 ? (
<span className="text-xs text-muted-foreground">
+{matches.length - 1} other possible{' '}
{matches.length - 1 === 1 ? 'match' : 'matches'}
</span>
) : null}
</div>
</div>
</div>
</div>
);
}