feat/dedup-migration
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Matt Ciaccio
|
18e5c124b0 |
feat(dedup): NocoDB migration script + tables (P3 dry-run)
Lands the one-shot migration pipeline from the legacy NocoDB Interests base into the new client/interest schema. Dry-run mode is fully operational: pulls the live snapshot, runs the dedup library, and writes a CSV + Markdown report under .migration/<timestamp>/. The --apply phase is stubbed for a follow-up PR per the design's P3 implementation sequence. Schema additions ================ - `client_merge_candidates` — pairs flagged by the background scoring job for the /admin/duplicates review queue. Status enum: pending / dismissed / merged. Unique-(portId, clientAId, clientBId) so the same pair can't surface twice. Empty until P2 lands the cron. - `migration_source_links` — idempotency ledger. Maps source-system rows (NocoDB Interest #624 → new client UUID) so re-running --apply against the same dry-run report skips already-imported entities. Both tables ship with the migration `0020_unusual_azazel.sql` — already applied to the local dev DB during this commit's preparation. Library ======= src/lib/dedup/nocodb-source.ts Read-only adapter for the legacy NocoDB v2 API. xc-token auth, auto-paginates until isLastPage, captures the table IDs from the 2026-05-03 audit. `fetchSnapshot()` pulls every relevant table in parallel into one in-memory object the transform layer consumes. src/lib/dedup/migration-transform.ts Pure function: NocoDB snapshot in, MigrationPlan out. Per row: - normalizes name / email / phone / country via the dedup library - parses the legacy DD-MM-YYYY / DD/MM/YYYY / ISO date formats - maps the 8-stage `Sales Process Level` enum to the new 9-stage pipelineStage - filters yacht-name placeholders ('TBC', 'Na', etc.) - merges Internal Notes + Extra Comments + Berth Size Desired into a single notes blob Then runs `findClientMatches` pairwise (with blocking) and union-finds clusters of rows whose score crosses the auto-link threshold (90). Lower-scoring pairs (50–89) become 'needs review'. Each cluster's "lead" row is picked by completeness score with recency tie-break. src/lib/dedup/migration-report.ts Writes three artifacts to .migration/<timestamp>/: - report.csv — one row per planned op, RFC-4180 escaped - summary.md — human-skimmable overview - plan.json — full structured plan for the --apply phase CSV cells with comma / quote / newline are quoted; internal quotes are doubled. No external CSV dep. src/lib/dedup/phone-parse.ts Script-safe wrapper around libphonenumber-js's `core` entry that loads `metadata.min.json` directly. The default `index.cjs.js` bundled by libphonenumber hits a metadata-shape interop bug under Node 25 + tsx (`{ default }` wrapping); core+JSON sidesteps it. The dedup `normalizePhone` and `find-matches` both use this wrapper now so the same code path runs in vitest, Next.js, and the migration CLI without surprises. src/lib/dedup/normalize.ts Tightened country resolution: added Caribbean short-form aliases ('antigua' → AG, 'st kitts' → KN, etc.) and a city map covering the US locations seen in the NocoDB dump (Boston, Tampa, Fort Lauderdale, Port Jefferson, Nantucket). Also relaxed phone parsing to drop the `isValid()` strict check — the libphonenumber min build rejects many real NANP-territory numbers, and dedup only needs a canonical E.164 to compare. CLI === scripts/migrate-from-nocodb.ts pnpm tsx scripts/migrate-from-nocodb.ts --dry-run → Pulls the live NocoDB base (NOCODB_URL + NOCODB_TOKEN env vars), runs the transform, writes report. No DB writes. pnpm tsx scripts/migrate-from-nocodb.ts --apply --report .migration/<dir>/ → Stubbed; exits with `not yet implemented` and a pointer to the design doc. Apply phase ships in a follow-up. Tests ===== tests/unit/dedup/migration-transform.test.ts (7 cases) Fixture-based regression. A frozen 12-row NocoDB snapshot covers every duplicate pattern in the design (§1.2). The test asserts: - 12 input rows → 7 unique clients (cluster math is right) - Patterns A / B / C / E auto-link - Pattern F (Etiennette Clamouze) does NOT auto-link - Every interest preserved as its own row even when clients merge - 8-stage → 9-stage enum mapping is correct per spec - Multi-yacht merge (Constanzo CALYPSO + Costanzo GEMINI under one client) — the design's signature win - Output is deterministic (run twice, identical) Validation against real data ============================ Ran `pnpm tsx scripts/migrate-from-nocodb.ts --dry-run` against the live NocoDB. Result on 252 Interests rows: - 237 clients (15 merged into 13 clusters) - 252 interests (one per source row) - 406 contacts, 52 addresses - 13 auto-linked clusters (every confirmed cluster from §1.2 audit) - 3 pairs flagged for review (Camazou, Zasso, one new) - 1 phone placeholder flagged Total dedup test count: 57 (50 from P1 + 7 fixture tests). Lint: clean. Tsc: clean for new files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Matt Ciaccio
|
ba5fb6db5e |
feat(sales): EOI queue route + invoice→deposit auto-advance + won/lost outcomes
Three independent strengthenings of the sales spine that the prior coherence
sweep made it possible to do cleanly.
1. EOI queue page
- Sidebar entry under Documents → "EOI queue".
- Route /[port]/documents/eoi renders DocumentsHub with the existing
eoi_queue tab pre-selected (filters in-flight EOIs only).
- .gitignore: tightened root-only `eoi/` ignore so the documents/eoi
route is no longer silently excluded.
2. Invoice ↔ deposit link
- invoices.interestId (FK, ON DELETE SET NULL) + invoices.kind
('general' | 'deposit'). Indexed on (port_id, interest_id).
- createInvoiceSchema requires interestId when kind === 'deposit';
the service validates the linked interest belongs to the same port
before insert.
- recordPayment auto-advances pipelineStage to deposit_10pct (via
advanceStageIfBehind) when a paid invoice is kind=deposit and has
an interestId. No-op if the interest is already further along.
- "Create deposit invoice" link added to the Deposit milestone on the
interest detail. Links to /invoices/new?interestId=…&kind=deposit;
the form prefills the billing entity from the linked interest's
client and shows a context banner.
3. Won / lost terminal outcomes
- interests.outcome ('won' | 'lost_other_marina' | 'lost_unqualified'
| 'lost_no_response' | 'cancelled') + outcomeReason text +
outcomeAt timestamp. Indexed on (port_id, outcome).
- setInterestOutcome / clearInterestOutcome services + POST/DELETE
/api/v1/interests/:id/outcome endpoints (gated by change_stage
permission). Setting an outcome moves the interest to `completed`
in the same write; clearing reopens to `in_communication` (or a
caller-specified stage).
- Mark Won / Mark Lost icon buttons on the interest detail header,
plus an outcome badge that replaces the stage pill once a terminal
outcome is set, plus a Reopen button.
- Funnel + dashboard math updated to exclude lost/cancelled outcomes
from active calculations (KPIs.activeInterests, pipelineValueUsd,
getPipelineCounts, computePipelineFunnel, getRevenueForecast).
The funnel now also returns a `lost` summary so callers can
surface leakage without polluting conversion percentages.
Schema changes shipped via 0019_lazy_vampiro.sql; applied to dev DB
manually via psql because drizzle-kit push hits a pre-existing zod
parsing issue on the companies index. Dev server may need a restart
to flush prepared-statement caches.
tsc clean. vitest 832/832 pass. ESLint clean on every file touched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Matt Ciaccio
|
fbb1f1f366 |
scaffold(mobile): branch setup — audit harness, spec, plan, gitignore + client-portal cleanup
Pre-execution baseline for the mobile foundation PR: - Mobile audit harness (tests/e2e/audit/mobile.spec.ts + mobile-audit Playwright project) — visits every page at four anchor iPhone viewports (375/393/402/440), screenshots full-page to .audit/mobile/, generates index.md - Design spec (docs/superpowers/specs/2026-04-29-mobile-optimization-design.md) — adaptive shell + responsive content; full active-iPhone-range coverage; foundation + per-page migration phases - Implementation plan (docs/superpowers/plans/2026-04-29-mobile-foundation.md) — 24 TDD tasks for the foundation PR - .gitignore: ignore /client-portal/ (legacy nested Nuxt repo) and /.audit/ (regenerable screenshots) - Remove phantom client-portal gitlink (mode 160000 with no .gitmodules) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Matt Ciaccio
|
e8d61c91c4 |
feat(platform): residential module + admin UI + reliability fixes
Residential platform - New schema: residentialClients, residentialInterests (separate from marina/yacht clients) with migration 0010 - Service layer with CRUD + audit + sockets + per-port portal toggle - v1 + public API routes (/api/v1/residential/*, /api/public/residential-inquiries) - List + detail pages with inline editing for clients and interests - Per-user residentialAccess toggle on userPortRoles (migration 0011) - Permission keys: residential_clients, residential_interests - Sidebar nav + role form integration - Smoke spec covering page loads, UI create flow, public endpoint Admin & shared UI - Admin → Forms (form templates CRUD) with validators + service - Notification preferences page (in-app + email per type) - Email composition + accounts list + threads view - Branded auth shell shared across CRM + portal auth surfaces - Inline editing extended to yacht/company/interest detail pages - InlineTagEditor + per-entity tags endpoints (yachts, companies) - Notes service polymorphic across clients/interests/yachts/companies - Client list columns: yachtCount + companyCount badges - Reservation file-download via presigned URL (replaces stale <a href>) Route handler refactor - Extracted yachts/companies/berths reservation handlers to sibling handlers.ts files (Next.js 15 route.ts only allows specific exports) Reliability fixes - apiFetch double-stringify bug fixed across 13 components (apiFetch already JSON.stringifies its body; passing a stringified body produced double-encoded JSON which failed zod validation) - SocketProvider gated behind useSyncExternalStore-based mount check to avoid useSession() SSR crashes under React 19 + Next 15 - apiFetch falls back to URL-pathname → port-id resolution when the Zustand store hasn't hydrated yet (fresh contexts, e2e tests) - CRM invite flow (schema, service, route, email, dev script) - Dashboard route → [portSlug]/dashboard/page.tsx + redirect - Document the dev-server restart-after-migration gotcha in CLAUDE.md Tests - 5-case residential smoke spec - Integration test updates for new service signatures Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Matt Ciaccio
|
475b051e29 |
feat(portal): replace magic-link with email/password + admin-initiated activation
The client portal no longer uses passwordless / magic-link sign-in. Each
client now has a `portal_users` row with a scrypt-hashed password,
created by an admin from the client detail page; the admin's invite
mails an activation link that the client uses to set their own password.
Forgot-password is wired through the same token mechanism.
Schema (migration `0009_outgoing_rumiko_fujikawa.sql`):
- `portal_users` — one per client account, separate from the CRM
`users` table (better-auth) so the auth realms stay isolated. Email
is globally unique, password is null until activation.
- `portal_auth_tokens` — single-use activation / reset tokens. Stores
only the SHA-256 hash so a DB compromise never leaks live tokens.
Services:
- `src/lib/portal/passwords.ts` — scrypt hash/verify (no new deps;
uses node:crypto), token mint+hash helpers.
- `src/lib/services/portal-auth.service.ts` — createPortalUser,
resendActivation, activateAccount, signIn (timing-safe),
requestPasswordReset, resetPassword. Auth failures throw the new
UnauthorizedError (401); enumeration-safe behaviour everywhere.
Routes:
- POST /api/portal/auth/sign-in — sets the existing portal JWT cookie.
- POST /api/portal/auth/forgot-password — always 200.
- POST /api/portal/auth/reset-password — token + new password.
- POST /api/portal/auth/activate — token + initial password.
- POST /api/v1/clients/:id/portal-user — admin invite (and `?action=resend`).
- Removed: /api/portal/auth/request, /api/portal/auth/verify (magic link).
UI:
- /portal/login — replaced email-only magic-link form with email +
password + "forgot password" link.
- /portal/forgot-password, /portal/reset-password, /portal/activate — new.
- New shared `PasswordSetForm` component used by activate + reset.
- New `PortalInviteButton` rendered on the client detail header.
Email send:
- `createTransporter` now wires SMTP auth when SMTP_USER+SMTP_PASS are
set (gmail app-password or marina-server creds, configured via env).
- `SMTP_FROM` env var lets the sender address be overridden without
pinning it to `noreply@${SMTP_HOST}`.
Tests:
- Smoke spec 17 (client-portal) updated to the new flow: 7/7 green.
- Smoke specs 02-crud-spine, 05-invoices, 20-critical-path updated to
match the post-refactor client + invoice forms (drop companyName,
use OwnerPicker + billingEmail).
- Vitest 652/652 still green; type-check clean.
Drops the dead `requestMagicLink` from portal.service.ts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
Matt Ciaccio
|
ed40662b99 |
chore: gitignore docker-compose.override.yml and .remember/
The override file is a local-only port remap for when the default dev postgres port is already bound by another project. .remember/ is skill-maintained session-state storage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
| a13d7503cc |
Fix Docker build and production server infrastructure
- Add SKIP_ENV_VALIDATION to bypass Zod env check during next build - Bundle custom server.ts with esbuild so production uses Socket.io - Create worker entry point (src/worker.ts) with all BullMQ workers - Add esbuild build scripts for server and worker bundles - Fix Dockerfile.worker to include its own build stage - Fix pre-commit hook to work without global pnpm - Add CLAUDE.md with project conventions and quick reference Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
|||
| 67d7e6e3d5 |
Initial commit: Port Nimara CRM (Layers 0-4)
Full CRM rebuild with Next.js 15, TypeScript, Tailwind, Drizzle ORM, PostgreSQL, Redis, BullMQ, MinIO, and Socket.io. Includes 461 source files covering clients, berths, interests/pipeline, documents/EOI, expenses/invoices, email, notifications, dashboard, admin, and client portal. CI/CD via Gitea Actions with Docker builds. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
