letsbe/pn-new-crm
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
256 Commits 7 Branches 0 Tags
a792d9a182181341ffe978c443d8d55f98d05677
Commit Graph

8 Commits

Author SHA1 Message Date
Matt Ciaccio
18e5c124b0 feat(dedup): NocoDB migration script + tables (P3 dry-run) 
Lands the one-shot migration pipeline from the legacy NocoDB Interests
base into the new client/interest schema. Dry-run mode is fully
operational: pulls the live snapshot, runs the dedup library, and
writes a CSV + Markdown report under .migration/<timestamp>/. The
--apply phase is stubbed for a follow-up PR per the design's P3
implementation sequence.

Schema additions
================

- `client_merge_candidates` — pairs flagged by the background scoring
  job for the /admin/duplicates review queue. Status enum: pending /
  dismissed / merged. Unique-(portId, clientAId, clientBId) so the
  same pair can't surface twice. Empty until P2 lands the cron.
- `migration_source_links` — idempotency ledger. Maps source-system
  rows (NocoDB Interest #624 → new client UUID) so re-running --apply
  against the same dry-run report skips already-imported entities.

Both tables ship with the migration `0020_unusual_azazel.sql` —
already applied to the local dev DB during this commit's preparation.

Library
=======

src/lib/dedup/nocodb-source.ts
  Read-only adapter for the legacy NocoDB v2 API. xc-token auth,
  auto-paginates until isLastPage, captures the table IDs from the
  2026-05-03 audit. `fetchSnapshot()` pulls every relevant table in
  parallel into one in-memory object the transform layer consumes.

src/lib/dedup/migration-transform.ts
  Pure function: NocoDB snapshot in, MigrationPlan out. Per row:
    - normalizes name / email / phone / country via the dedup library
    - parses the legacy DD-MM-YYYY / DD/MM/YYYY / ISO date formats
    - maps the 8-stage `Sales Process Level` enum to the new 9-stage
      pipelineStage
    - filters yacht-name placeholders ('TBC', 'Na', etc.)
    - merges Internal Notes + Extra Comments + Berth Size Desired into
      a single notes blob
  Then runs `findClientMatches` pairwise (with blocking) and
  union-finds clusters of rows whose score crosses the auto-link
  threshold (90). Lower-scoring pairs (50–89) become 'needs review'.
  Each cluster's "lead" row is picked by completeness score with
  recency tie-break.

src/lib/dedup/migration-report.ts
  Writes three artifacts to .migration/<timestamp>/:
    - report.csv  — one row per planned op, RFC-4180 escaped
    - summary.md  — human-skimmable overview
    - plan.json   — full structured plan for the --apply phase
  CSV cells with comma / quote / newline are quoted; internal quotes
  are doubled. No external CSV dep.

src/lib/dedup/phone-parse.ts
  Script-safe wrapper around libphonenumber-js's `core` entry that
  loads `metadata.min.json` directly. The default `index.cjs.js`
  bundled by libphonenumber hits a metadata-shape interop bug under
  Node 25 + tsx (`{ default }` wrapping); core+JSON sidesteps it.
  The dedup `normalizePhone` and `find-matches` both use this wrapper
  now so the same code path runs in vitest, Next.js, and the migration
  CLI without surprises.

src/lib/dedup/normalize.ts
  Tightened country resolution: added Caribbean short-form aliases
  ('antigua' → AG, 'st kitts' → KN, etc.) and a city map covering the
  US locations seen in the NocoDB dump (Boston, Tampa, Fort
  Lauderdale, Port Jefferson, Nantucket). Also relaxed phone parsing
  to drop the `isValid()` strict check — the libphonenumber min build
  rejects many real NANP-territory numbers, and dedup only needs a
  canonical E.164 to compare.

CLI
===

scripts/migrate-from-nocodb.ts
  pnpm tsx scripts/migrate-from-nocodb.ts --dry-run
    → Pulls the live NocoDB base (NOCODB_URL + NOCODB_TOKEN env vars),
       runs the transform, writes report. No DB writes.
  pnpm tsx scripts/migrate-from-nocodb.ts --apply --report .migration/<dir>/
    → Stubbed; exits with `not yet implemented` and a pointer to the
       design doc. Apply phase ships in a follow-up.

Tests
=====

tests/unit/dedup/migration-transform.test.ts (7 cases)
  Fixture-based regression. A frozen 12-row NocoDB snapshot covers
  every duplicate pattern in the design (§1.2). The test asserts:
    - 12 input rows → 7 unique clients (cluster math is right)
    - Patterns A / B / C / E auto-link
    - Pattern F (Etiennette Clamouze) does NOT auto-link
    - Every interest preserved as its own row even when clients merge
    - 8-stage → 9-stage enum mapping is correct per spec
    - Multi-yacht merge (Constanzo CALYPSO + Costanzo GEMINI under one
      client) — the design's signature win
    - Output is deterministic (run twice, identical)

Validation against real data
============================

Ran `pnpm tsx scripts/migrate-from-nocodb.ts --dry-run` against the
live NocoDB. Result on 252 Interests rows:
  - 237 clients (15 merged into 13 clusters)
  - 252 interests (one per source row)
  - 406 contacts, 52 addresses
  - 13 auto-linked clusters (every confirmed cluster from §1.2 audit)
  - 3 pairs flagged for review (Camazou, Zasso, one new)
  - 1 phone placeholder flagged

Total dedup test count: 57 (50 from P1 + 7 fixture tests).
Lint: clean. Tsc: clean for new files.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 14:50:01 +02:00
Matt Ciaccio
0fb7920db5 fix(auth/mobile): support LAN-IP access in dev + edge-to-edge auth bg 
- branded-auth-shell: split the background image into a separate
    fixed-positioned layer behind the layout. Previously the bg was on
    a min-h-screen container and iOS Safari left visible whitespace at
    the top/bottom when the URL bar showed/hid (the container's height
    didn't match the visual viewport). Now the bg pins to the actual
    visible viewport via `fixed inset-0`. min-h-[100dvh] also added
    so the layout layer matches.
  - auth client: derive baseURL from window.location.origin instead of
    NEXT_PUBLIC_APP_URL. Same dev build now works whether opened on
    localhost (Mac) or the LAN IP (iPhone on Wi-Fi).
  - auth server: dynamic trustedOrigins function that allows
    localhost / 127.x / 192.168.x / 10.x in dev (function form
    inspects the incoming request's Origin). Production stays locked
    to NEXT_PUBLIC_APP_URL.
  - new dev helper: scripts/dev-set-password.ts to set a user's
    better-auth password directly (bypasses the email-reset flow);
    used to bootstrap matt@letsbe.solutions for mobile testing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-01 16:21:59 +02:00
Matt Ciaccio
4c5334d471 sec: gate super-admin invite minting, OCR settings, and alert mutations 
Three findings from the branch security review:

1. HIGH — Privilege escalation via super-admin invite. POST
   /api/v1/admin/invitations was gated only by manage_users (held by the
   port-scoped director role). The body schema accepted isSuperAdmin
   from the request, createCrmInvite persisted it verbatim, and
   consumeCrmInvite copied it into userProfiles.isSuperAdmin — granting
   the new account cross-tenant access. Now the route rejects
   isSuperAdmin=true unless ctx.isSuperAdmin, and createCrmInvite
   requires invitedBy.isSuperAdmin as defense-in-depth.

2. HIGH — Receipt-image exfiltration via OCR settings. The route
   /api/v1/admin/ocr-settings (and the sibling /test) were wrapped only
   in withAuth — any port role including viewer could PUT a swapped
   provider apiKey + flip aiEnabled, redirecting every subsequent
   receipt scan to attacker infrastructure. Both are now wrapped in
   withPermission('admin','manage_settings',…) matching the sibling
   admin routes (ai-budget, settings).

3. MEDIUM — Cross-tenant alert IDOR. dismissAlert / acknowledgeAlert
   issued UPDATE … WHERE id=? with no portId predicate. Any
   authenticated user with a foreign alert UUID could mutate it. Both
   service functions now require portId and add it to the WHERE; the
   route handlers pass ctx.portId.

The dev-trigger-crm-invite script passes a synthetic super-admin caller
identity since it runs out-of-band.

The two public-form tests randomize their IP prefix per run so a fresh
test process doesn't collide with leftover redis sliding-window entries
from a prior run (publicForm limiter pexpires after 1h).

Two new regression test files cover the fixes (6 tests).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-29 02:27:01 +02:00
Matt Ciaccio
6eb0d3dc92 docs(ops): backup/restore + email deliverability runbooks 
Two new runbooks under docs/runbooks/ plus the automation scripts the
backup runbook references. Both are written so an operator who has only
the off-site backup credentials and the runbook can recover the system
unaided.

Backup/restore (Phase 4a):
- docs/runbooks/backup-and-restore.md — covers what gets backed up
  (Postgres / MinIO / .env+ENCRYPTION_KEY), schedule (hourly DB +
  hourly MinIO mirror, 7-day hourly + 30-day daily retention),
  cold-restore procedure with row-count verification, weekly drill
- scripts/backup/pg-backup.sh — pg_dump → gzip → optional GPG → mc
  upload, fails loud
- scripts/backup/minio-mirror.sh — incremental mc mirror, no --remove
  flag so accidental deletes on the live bucket can't cascade
- scripts/backup/restore.sh — interactive prod restore + --drill mode
  that runs against a sandbox DB and diffs row counts

Email deliverability (Phase 4b):
- docs/runbooks/email-deliverability.md — what the CRM sends, DNS
  records (SPF/DKIM/DMARC/MX), per-port override implications,
  diagnosis flow ("didn't arrive" → 4-step checklist starting with
  EMAIL_REDIRECT_TO), provider migration plan, realapi suite as the
  end-to-end probe

Tests still 778/778 vitest, tsc/lint clean — these phases are docs +
shell scripts, no code changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-28 20:10:30 +02:00
Matt Ciaccio
31fa3d08ec chore(cleanup): Phase 1 — gap closure across audit, alerts, soft-delete, perms 
Multi-area cleanup pass closing partial-implementation gaps surfaced by the
post-i18n audit. No behavior changes for happy-path users; closes real
correctness/security holes.

PR1a Public yacht-interest endpoint i18n. /api/public/interests now accepts
     phoneE164/phoneCountry, nationalityIso, address.{countryIso, subdivisionIso},
     and company.{incorporationCountryIso, incorporationSubdivisionIso}.
     Server-side parsePhone() fallback for legacy raw phone strings.

PR1b Alert rule registry trim. Two rule slots ('document.expiring_soon',
     'audit.suspicious_login') were registered but evaluators returned [].
     Both required schema/instrumentation that hadn't landed. Removed from
     the registry; comments record the dependencies needed to revive them.
     Effective rule count: 8 active.

PR1c vi.mock hoist + flake fix. Hoisted vi.mock calls to top-level in 5
     integration test files; webhook-delivery uses vi.hoisted for the
     queue-add ref. Vitest no longer warns about non-top-level mocks.
     Deflaked the 'short value' assertion in security-encryption.test.ts
     by switching plaintext from 'ab' to 'XY' (non-hex chars). 5/5 runs green.

PR1d Soft-delete reference audit. listClientOptions and listYachtsForOwner
     now filter by isNull(archivedAt). Berths use status (no archivedAt).

PR1e Permission-matrix audit script + report. scripts/audit-permissions.ts
     walks every src/app/api/v1/**/route.ts and reports handlers without a
     withPermission() wrapper. Initial run found 33 violations.
     - Allow-listed 17 with explicit reasons (self-data, admin, alerts,
       search, currency, ai, custom-fields — some marked TODO).
     - Wrapped 7 routes with concrete permissions: clients/options
       (clients:view), berths/options (berths:view), dashboard/*
       (reports:view_dashboard), analytics (reports:view_analytics).
     Audit report at docs/runbooks/permission-audit.md. Script exits
     non-zero on any unallow-listed violation so it can become a CI gate.

Vitest: 741 -> 741 (no new tests; existing suite covers the changes).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-28 18:48:22 +02:00
Matt Ciaccio
e8d61c91c4 feat(platform): residential module + admin UI + reliability fixes
All checks were successful
Build & Push Docker Images / lint (pull_request) Successful in 1m2s
Details
Build & Push Docker Images / build-and-push (pull_request) Has been skipped
Details 
Residential platform
- New schema: residentialClients, residentialInterests (separate from
  marina/yacht clients) with migration 0010
- Service layer with CRUD + audit + sockets + per-port portal toggle
- v1 + public API routes (/api/v1/residential/*, /api/public/residential-inquiries)
- List + detail pages with inline editing for clients and interests
- Per-user residentialAccess toggle on userPortRoles (migration 0011)
- Permission keys: residential_clients, residential_interests
- Sidebar nav + role form integration
- Smoke spec covering page loads, UI create flow, public endpoint

Admin & shared UI
- Admin → Forms (form templates CRUD) with validators + service
- Notification preferences page (in-app + email per type)
- Email composition + accounts list + threads view
- Branded auth shell shared across CRM + portal auth surfaces
- Inline editing extended to yacht/company/interest detail pages
- InlineTagEditor + per-entity tags endpoints (yachts, companies)
- Notes service polymorphic across clients/interests/yachts/companies
- Client list columns: yachtCount + companyCount badges
- Reservation file-download via presigned URL (replaces stale <a href>)

Route handler refactor
- Extracted yachts/companies/berths reservation handlers to sibling
  handlers.ts files (Next.js 15 route.ts only allows specific exports)

Reliability fixes
- apiFetch double-stringify bug fixed across 13 components
  (apiFetch already JSON.stringifies its body; passing a stringified
  body produced double-encoded JSON which failed zod validation)
- SocketProvider gated behind useSyncExternalStore-based mount check
  to avoid useSession() SSR crashes under React 19 + Next 15
- apiFetch falls back to URL-pathname → port-id resolution when the
  Zustand store hasn't hydrated yet (fresh contexts, e2e tests)
- CRM invite flow (schema, service, route, email, dev script)
- Dashboard route → [portSlug]/dashboard/page.tsx + redirect
- Document the dev-server restart-after-migration gotcha in CLAUDE.md

Tests
- 5-case residential smoke spec
- Integration test updates for new service signatures

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-27 21:54:32 +02:00
Matt Ciaccio
65b241805e test(portal): IMAP full-lifecycle activation E2E + dev probe helper 
New realapi spec walks the entire portal activation loop over real
network: invite via the admin endpoint → wait for the activation email
to land in the IMAP mailbox → extract the token from the body link →
activate the portal user via the public API → sign in with the new
password.

The match logic deliberately doesn't filter on the TO header — the
combination of EMAIL_REDIRECT_TO rewriting and +addressing made TO
matching brittle. Instead we discriminate by sender (noreply@…),
subject keyword, and body link pattern, which is unique enough to find
exactly the email this test triggered.

Companion script scripts/dev-imap-probe.ts dumps the most recent ~10
messages with from/to/subject/date — useful for debugging when an IMAP
match goes wrong.

Skips when IMAP_HOST / IMAP_USER / IMAP_PASS are absent so the suite
stays portable.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-27 15:40:28 +02:00
Matt Ciaccio
4441f1177f feat(portal): branded auth pages + legacy email styling + dev redirect override 
- New PortalAuthShell component: blurred Port Nimara overhead background +
  circular logo + white rounded card, used by /portal/login,
  /portal/activate, /portal/reset-password
- New email/templates/portal-auth.ts: table-based, responsive (max-width
  600px / width 100%), matching the existing legacy inquiry templates;
  replaces the inline templates that lived in portal-auth.service
- EMAIL_REDIRECT_TO env override: when set, sendEmail routes every
  outbound message to that address regardless of recipient and tags the
  subject with "[redirected from <original>]". Dev/test safety net only;
  unset in production
- Portal password minimum length 12 → 9 (service + both API routes +
  client-side form)
- Dev helper script scripts/dev-trigger-portal-invite.ts: seeds a portal
  user against the first port-nimara client and uses EMAIL_REDIRECT_TO
  as the stored email so the tester can sign in with the address that
  received the activation mail

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-27 15:04:21 +02:00