chore(launch-prep): hide unfinished report/import surfaces, defer big builds
Ship-what's-done prep ahead of the prod cutover (launch ~today): - Hide Financial + Marketing report cards from the reports landing (both were "Builder in development" placeholders gated on unbuilt data sources). Sales/Operational/Custom + templates/scheduling/ exports remain live. - Trim the Custom-report card copy to match the shipped basic builder (no group-by/filters yet; the builder page header was already honest). - Hide the Bulk Import mockup from search-nav-catalog + the admin sections browser; /admin/import is now unreachable from the UI. - Correct client-facing doc over-claims (waiting-list "next-in-line notification", Import) in features-list.md + new-system-feature-summary.md. - Un-stale BACKLOG.md (Documenso phases 2-7 confirmed shipped). - Log decisions + deferred work (full importer, full custom-builder, waiting-list, maintenance-log, paper-upload bug) to launch-readiness.md. Deferred-importer design spec added at docs/superpowers/specs/2026-06-01-bulk-import-design.md. Verified: tsc --noEmit clean, eslint clean on changed files, 1512/1519 vitest pass (7 failures are Redis-down, unrelated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -14,7 +14,16 @@ Documenso phases 2-7 stay back-burnered per user.
|
||||
|
||||
---
|
||||
|
||||
## A. Documenso build (deferred for later)
|
||||
## A. Documenso build (MOSTLY SHIPPED — see note)
|
||||
|
||||
> **Stale-doc fix (2026-06-01):** a feature-completeness sweep confirmed
|
||||
> the core of phases 2–7 has since shipped and is wired — cascading
|
||||
> "your turn" invites (Phase 2), custom doc upload-to-signing (Phase 3,
|
||||
> `custom-document-upload.service.ts` + `/api/v1/interests/[id]/upload-for-signing`),
|
||||
> the field-placement UI (Phase 4, `upload-for-signing-dialog.tsx`), and
|
||||
> Project Director user-linking (Phase 7). The integration is treated as
|
||||
> feature-complete. The phase table below is kept for history; re-verify
|
||||
> the Phase 5/6 polish line-items individually before relying on them.
|
||||
|
||||
**Source:** [`docs/documenso-build-plan.md`](./documenso-build-plan.md) — full phase plan with locked decisions (Q1–Q10).
|
||||
**Tracker delta:** [`docs/admin-ux-backlog.md`](./admin-ux-backlog.md) — what landed in Phase 1.
|
||||
|
||||
238
docs/deployment-plan.md
Normal file
238
docs/deployment-plan.md
Normal file
@@ -0,0 +1,238 @@
|
||||
# Production Deployment Plan — Port Nimara CRM
|
||||
|
||||
> **Status:** DRAFT · pre-deployment · 2026-05-31
|
||||
> **Target:** `https://crm.portnimara.com` on the PN Cloud server.
|
||||
> **Companion:** `docs/launch-readiness.md` (Initiative 5 — cutover).
|
||||
> Credentials live in `private/deployment-creds.md` (gitignored) — **never
|
||||
> put secrets in this file.**
|
||||
|
||||
## ⛔ Guardrails (non-negotiable)
|
||||
|
||||
1. **No change to anything on the prod server without Matt's explicit
|
||||
per-action approval.** Recon/reads are fine; every `sudo`, every file
|
||||
write, every `docker` mutation, every `certbot` run is approved
|
||||
individually before it runs.
|
||||
2. **Documenso is VITAL.** It has broken on past upgrades. Nothing touches
|
||||
the Documenso DB, volumes, or container until a verified backup +
|
||||
S3↔DB reconciliation exists AND the upgrade step is explicitly approved.
|
||||
3. Work one phase at a time; verify before moving on. Keep a rollback for
|
||||
each mutating step.
|
||||
|
||||
---
|
||||
|
||||
## Access (established 2026-05-31)
|
||||
|
||||
| What | Detail | Verified |
|
||||
| ---------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------ |
|
||||
| **Prod server (SSH)** | `45.142.177.246:22022`, user `stefan`, key `id_ed25519_2026` (macOS keychain) | ✅ connected, key auth |
|
||||
| **Gitea API** | `https://code.letsbe.solutions` as `matt` (admin) — reads build status, warnings, errors | ✅ v1.25.5, repo `letsbe/pn-new-crm` |
|
||||
| **Container registry** | `code.letsbe.solutions/letsbe/pn-new-crm/{crm-app,crm-worker}` | ✅ CI pushes `:latest` + `:<sha>` |
|
||||
|
||||
Notes:
|
||||
|
||||
- `stefan` is **unprivileged** (uid 1000, not in the `docker` group; `sudo`
|
||||
prompts for a password). Every `docker` / `nginx` / `certbot` / cert-read
|
||||
step needs `sudo` (root pass in `private/deployment-creds.md` — **VERIFY**;
|
||||
the per-server creds file had MOPC's pass by mistake).
|
||||
- Reading build logs: `GET /api/v1/repos/letsbe/pn-new-crm/actions/tasks`
|
||||
(run status) + per-job logs; latest `main` build is **success**.
|
||||
|
||||
---
|
||||
|
||||
## How builds reach prod
|
||||
|
||||
`git push origin main` → Gitea Actions `.gitea/workflows/build.yml`:
|
||||
|
||||
1. **lint** job: `pnpm lint` + `pnpm exec tsc --noEmit`.
|
||||
2. **build-and-push** job (main only): builds `Dockerfile` → `crm-app` and
|
||||
`Dockerfile.worker` → `crm-worker`, pushes `:latest` + `:<sha>` to the
|
||||
Gitea registry.
|
||||
|
||||
Prod **pulls** those images — it does not build. So a deploy is:
|
||||
push → wait for green CI → `docker compose pull` + `up -d` on the server.
|
||||
|
||||
---
|
||||
|
||||
## Prod stack (`docker-compose.prod.yml`)
|
||||
|
||||
| Service | Image | Notes |
|
||||
| ------------ | ---------------------------- | --------------------------------------------------------------- |
|
||||
| `postgres` | `postgres:16-alpine` | self-contained, volume `pgdata` |
|
||||
| `redis` | `redis:7-alpine` | self-contained, volume `redisdata` (BullMQ + socket.io adapter) |
|
||||
| `crm-app` | registry `crm-app:latest` | **host `7100` → container `3000`** |
|
||||
| `crm-worker` | registry `crm-worker:latest` | BullMQ worker |
|
||||
|
||||
- **Storage:** no MinIO service in the compose — the CRM uses **external
|
||||
MinIO** via `system_settings.storage_backend` + `getStorageBackend()`.
|
||||
The existing prod MinIO (`:9000`, `s3.conf` / `minio.conf` nginx vhosts)
|
||||
is the backend. Confirm bucket + keys (creds file §3).
|
||||
- **Decision needed:** does the CRM get its **own** Postgres (the compose
|
||||
default, isolated `pgdata`) or reuse an existing prod Postgres instance?
|
||||
Default = the compose's own Postgres (cleanest isolation). Confirm.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — `crm.portnimara.com` go-live
|
||||
|
||||
DNS already points `crm.portnimara.com` at the server. No `crm.portnimara`
|
||||
nginx vhost exists yet (fresh setup). Template: `portnimara_dev.conf`
|
||||
(reverse-proxy + Certbot pattern already in use on this box).
|
||||
|
||||
### Pre-flight (no approval needed — prep only)
|
||||
|
||||
- [ ] Assemble the prod `.env` for the CRM. Source of truth: `src/lib/env.ts`
|
||||
(Zod schema) + `.env.example`. Critical keys:
|
||||
- `APP_URL=https://crm.portnimara.com`
|
||||
- `DATABASE_URL` (compose Postgres), `REDIS_*`
|
||||
- storage / MinIO (endpoint, access/secret, bucket) — creds file §3
|
||||
- `DOCUMENSO_API_URL` (bare host, no `/api/v1`), `DOCUMENSO_API_VERSION`, API key
|
||||
- better-auth secret, `WEBSITE_INTAKE_SECRET`, SMTP/IMAP
|
||||
- **`EMAIL_REDIRECT_TO` MUST be unset in prod.**
|
||||
- [ ] Server can pull from the registry: `docker login code.letsbe.solutions`
|
||||
with a registry token (creds file §2 — generate a Gitea token; do
|
||||
**not** bake the account password into the server).
|
||||
|
||||
### Step 1 — nginx vhost (⚠ approval)
|
||||
|
||||
1. Create `/etc/nginx/sites-available/crm_portnimara.conf` modelled on
|
||||
`portnimara_dev.conf`: port-80 → 443 redirect + `.well-known/acme-challenge`
|
||||
location; port-443 server `proxy_pass http://127.0.0.1:7100` with the same
|
||||
header block (Host, X-Real-IP, CF-Connecting-IP, X-Forwarded-_, websocket
|
||||
`Upgrade`/`Connection` for socket.io), `client_max_body_size 64M`,
|
||||
`proxy_read_timeout 300`, buffering off. **HTTP-only first** (no `ssl\__`
|
||||
lines yet) so Certbot can complete the challenge.
|
||||
2. Symlink into `sites-enabled/`.
|
||||
3. `sudo nginx -t` — must pass. Then `sudo systemctl reload nginx`.
|
||||
|
||||
### Step 2 — TLS cert (⚠ approval)
|
||||
|
||||
- `sudo certbot --nginx -d crm.portnimara.com` — pulls + installs the cert,
|
||||
rewrites the vhost with the managed `ssl_certificate` lines + 80→443
|
||||
redirect. Re-run `sudo nginx -t` + reload.
|
||||
|
||||
### Step 3 — bring up the container (⚠ approval)
|
||||
|
||||
1. Place `docker-compose.prod.yml` + the prod `.env` in the deploy dir
|
||||
(e.g. `/opt/pn-crm` — confirm location).
|
||||
2. `sudo docker login code.letsbe.solutions` (registry token).
|
||||
3. `sudo docker compose -f docker-compose.prod.yml pull`.
|
||||
4. `sudo docker compose -f docker-compose.prod.yml up -d`.
|
||||
5. **Watch for errors:** `sudo docker compose logs -f crm-app crm-worker`.
|
||||
6. Apply schema: migrations via `psql` (per CLAUDE.md `db:migrate` is broken)
|
||||
or the app's push path — confirm the prod migration approach.
|
||||
7. Seed/bootstrap the port + admin user as needed.
|
||||
|
||||
### Verify
|
||||
|
||||
- [ ] `curl -fsS https://crm.portnimara.com/api/public/health` → `{status:"ok"...}`
|
||||
- [ ] Authenticated health w/ `X-Intake-Secret` → `{checks:{db,redis}}`
|
||||
- [ ] Login loads, branding renders, a berth list + a deal render.
|
||||
- [ ] socket.io realtime connects (websocket upgrade through nginx works).
|
||||
- [ ] No `42703` column errors (restart `crm-app` after any schema change).
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Documenso v1.13.1 → v2.x upgrade (VITAL — execute SOBER, heavily gated)
|
||||
|
||||
> **Do not execute while impaired.** This is the production signing system.
|
||||
> Every mutating step needs an explicit, sober go/no-go. The runbook below is
|
||||
> reference; the actual run is a scheduled session.
|
||||
|
||||
### Verified facts (2026-05-31 recon + research)
|
||||
|
||||
| Item | Value |
|
||||
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Current version | `documenso/documenso:v1.13.1` (Oct 2025 — last v1) |
|
||||
| Latest version | **`v2.11.0`** (May 2026). Path: 1.13.1 → 2.0.0 → … → 2.11.0 (major jump) |
|
||||
| Compose | `/root/docker-compose/documenso/docker-compose.yml` (project `documenso-production`, services `documenso` + `database`) |
|
||||
| DB | `postgres:15`, db `documenso_db`, user `admin`, vol `documenso-production_documenso-database` → `/var/lib/postgresql/data` |
|
||||
| App port | container `3000` → host `3020`; served at `https://signatures.portnimara.dev` (nginx `documenso.conf`, direct — **no Cloudflare**) |
|
||||
| Storage | external MinIO, bucket `signatures` @ `s3.portnimara.com`, region `eu-central-1` |
|
||||
| Signing cert | `/opt/documenso/certificate.p12` (+ passphrase in env) |
|
||||
|
||||
**Research conclusions (sources in chat):**
|
||||
|
||||
- **v1 API survives in v2** — _"API V1 is stable but deprecated; nothing breaks."_ So the CRM keeps working on v1 API; flip to v2 later. (Will be **explicitly re-tested against the clone in Phase 0** before committing.)
|
||||
- **Postgres 15 is v2's official DB** — no DB-engine upgrade needed.
|
||||
- **Env vars carry over unchanged**; only `NEXTAUTH_URL` is dropped in v2 (auth now derives from `NEXT_PUBLIC_WEBAPP_URL`, already set correctly) — harmless leftover.
|
||||
- Upgrade = pull new image + restart; `prisma migrate deploy` auto-runs all pending migrations on startup.
|
||||
- **Known migration-failure history** (issue #1880: NOT-NULL column added without backfill). 1.13.1 is past that one, but it's the failure pattern to expect — hence the clone dry-run.
|
||||
- The login bounce (non-`Secure` cookie / `NEXTAUTH_URL` quirk) is plausibly fixed in v2's reworked auth, but treat that as a hoped-for bonus, not the goal.
|
||||
|
||||
### Locked decisions (per Matt, 2026-05-31)
|
||||
|
||||
- Dry-run on a clone first: **yes**. Target **latest v2.11.0**, staged through v2.0.0.
|
||||
- **No-downtime caveat:** true zero-downtime is **not possible** (migrations run on restart). Goal = brief + pre-rehearsed: validate fully on the clone, pre-pull the image, then a fast prod cutover in a low-traffic window.
|
||||
- CRM stays on Documenso **v1 API** after upgrade.
|
||||
- Backups: `pg_dump` + cert + compose/env pulled to the Mac (`private/documenso-backups/`, gitignored) **and** a cold volume snapshot kept on-server for fastest rollback.
|
||||
- Privilege: root via `su` (stefan isn't in the docker group; sudo needs a password we don't have — root pass works for `su`).
|
||||
|
||||
### Phase 0 — Dry-run on a disposable clone (zero prod risk)
|
||||
|
||||
- [ ] `pg_dump -Fc documenso_db` (live, no downtime) → restore into a throwaway `postgres:15` + `documenso:v2.11.0` stack on a **different compose project + port**, with a copy of the signing cert.
|
||||
- [ ] Watch `prisma migrate deploy` run the full 1.13.1→2.11.0 chain. Confirm: all migrations succeed, app boots, **login works**, existing documents render.
|
||||
- [ ] **Re-test the CRM's v1 API calls** against the clone → expect 200s.
|
||||
- [ ] If a migration fails: capture it, fix forward (or decide a target version that's clean) BEFORE touching prod.
|
||||
|
||||
### Phase A — Prod backups (after Phase 0 passes; verified before any change)
|
||||
|
||||
- [ ] `pg_dump -Fc documenso_db` → pull to `private/documenso-backups/` on the Mac (off-box). Plus a plain SQL dump.
|
||||
- [ ] Cold volume snapshot: stop stack → `tar` `documenso-production_documenso-database` → keep on-server + copy off. (This is the gold rollback — Prisma migrations aren't reversible.)
|
||||
- [ ] Copy compose file + env + `/opt/documenso/{certificate.p12,private.key,certificate.crt}`.
|
||||
- [ ] **MinIO `signatures`**: read-only object inventory (`{key,size,lastModified,etag}`) + DB→storage-key mapping export (Document/DocumentData → storage key) so files can be re-matched if linkage breaks.
|
||||
- [ ] Test-restore the dump into a throwaway PG15; record SHA-256s.
|
||||
|
||||
### Phase B — Collation pre-fix (low risk; validate need on the clone first)
|
||||
|
||||
- [ ] `REFRESH COLLATION VERSION` on `documenso_db` (+ `template1`/`postgres`) + reindex, so the libc 2.36→2.41 mismatch can't interfere with migration index ops.
|
||||
|
||||
### Phase C — Prod upgrade (staged, pinned tags, low-traffic window)
|
||||
|
||||
- [ ] Pre-pull images. Edit compose: `v1.13.1 → v2.0.0` → `up -d` → watch migration logs → verify.
|
||||
- [ ] Then `v2.0.0 → v2.11.0` → verify. Keep `postgres:15`.
|
||||
|
||||
### Phase D — Verify
|
||||
|
||||
- [ ] Login works; an existing completed envelope's PDF resolves from MinIO; send a test envelope; **webhook reaches the CRM** (`X-Documenso-Secret`, idempotent `handleDocumentCompleted`); reminders/void work.
|
||||
- [ ] CRM unchanged (still v1 API).
|
||||
|
||||
### Phase E — Rollback (any failure)
|
||||
|
||||
- [ ] Revert image tag + restore the volume snapshot (and/or DB dump) → back to v1.13.1 exactly.
|
||||
|
||||
> Until Phase 0 passes AND a sober Phase A/C is explicitly approved step-by-step, **do not touch the Documenso container, DB, volumes, or `/opt/documenso`.**
|
||||
|
||||
---
|
||||
|
||||
## Open decisions / what I need from you
|
||||
|
||||
1. ✅ MinIO creds filled; Documenso DB creds filled (creds file §3/§4). Still need the Documenso **API token** + **webhook secret** (generate after login as `matt@portnimara.com`).
|
||||
2. **Verify the root/sudo password** (`IpMKQ0TW56ovv80` — confirmed it works for `su` to root; not stefan's sudo password).
|
||||
3. **CRM Postgres:** own (compose default) or reuse an existing instance?
|
||||
4. **Deploy dir** for the CRM on the server (`/opt/pn-crm`?).
|
||||
5. **Registry pull token** — Gitea token for `docker login` on the server.
|
||||
6. ✅ Documenso target = **v2.11.0**, staged, clone-validated first.
|
||||
7. **Maintenance window** for the (brief, unavoidable) Documenso restart downtime.
|
||||
8. **Off-box backup destination confirmed** = Mac `private/documenso-backups/` + on-server volume snapshot.
|
||||
|
||||
## Progress log
|
||||
|
||||
- 2026-05-31: Access established (SSH + Gitea API). Read-only recon done
|
||||
(nginx templates, prod compose, host port 7100). CRM deploy plan drafted.
|
||||
Documenso fully diagnosed read-only (v1.13.1, healthy app+DB, login issue =
|
||||
wrong email `@letsbe` vs `@portnimara.com` + a non-Secure-cookie quirk;
|
||||
5432 publicly exposed + brute-forced; libc collation mismatch). Researched
|
||||
v2 upgrade (v2.11.0 latest, PG15 ok, env vars carry over, v1 API survives).
|
||||
Upgrade runbook drafted. **No prod changes made; no backups taken.**
|
||||
- 2026-06-01: **Phase 0 dry-run PASSED (local, zero prod impact).** Read-only
|
||||
`pg_dump` of prod (3.5 MB — metadata only) → restored into a throwaway
|
||||
`postgres:15` → booted `documenso:v2.11.0` against it. Result: full
|
||||
v1.13.1→v2.11.0 chain applied cleanly (`All migrations have been
|
||||
successfully applied`, 140→157, none unfinished), app boots (home 302,
|
||||
signin 200, v2 api 200), and **v1 API still answers (400 not 404) → CRM
|
||||
safe**. Dump saved at `private/documenso-backups/` (off-box backup).
|
||||
Dry-run stack **torn down 2026-06-01** after the pass (`docker compose
|
||||
-p documenso-dryrun down -v` — containers + anonymous volume + network
|
||||
removed; restored clone gone, off-box dump retained). Compose file kept
|
||||
at `private/documenso-dryrun/docker-compose.yml` for a re-run. Prod
|
||||
still untouched.
|
||||
234
docs/features-list.md
Normal file
234
docs/features-list.md
Normal file
@@ -0,0 +1,234 @@
|
||||
# Port Nimara CRM — Feature List
|
||||
|
||||
A complete, purpose-built CRM for marina/port management: a single integrated workspace for sales, berths, documents, communications, and reporting, with the public website's berth feed and enquiry intake flowing directly into it. Multi-tenant by design — one branded instance per port.
|
||||
|
||||
> Scope note: this list covers the features ready for the beta launch. The new client portal, the tenancies module, and the new invoicing module are still being finalised and are not included here.
|
||||
|
||||
---
|
||||
|
||||
## Platform foundations
|
||||
|
||||
Apply across every feature area:
|
||||
|
||||
- **Purpose-built relational database (PostgreSQL)** modelled specifically for marina sales — fast on large data sets, rich relationships between entities (clients, companies, yachts, berths, deals, documents), and enforced data integrity.
|
||||
- **Real-time updates.** Edits, stage changes, file attachments, and completed signings propagate to every open window within a second.
|
||||
- **Per-port branding and configuration.** Each port has its own URL slug, logo, primary colour, default currency, timezone, and email templates, applied automatically to emails, PDFs, and the in-app shell.
|
||||
- **Granular role-based permissions.** Defined per resource (clients, berths, documents, expenses, reports, etc.) with separate view / create / edit / delete / export verbs. Per-user overrides on top of per-role definitions.
|
||||
- **Full audit trail.** Every meaningful change (who, what, before-and-after, when) recorded, retained 90 days, and searchable — surfaced in the activity feed, field-history popovers, and admin audit log.
|
||||
- **Backups and operational tooling.** Automatic daily database backups, weekly cleanup, configurable retention, and a built-in system-monitoring dashboard.
|
||||
- **Background job queue.** PDF generation, email sending, exports, webhook retries, and bounce polling run on a managed queue so the interface stays responsive.
|
||||
- **GDPR-ready.** One-click Article 15 data exports per client, automatic 30-day cleanup of export bundles, and a permissioned hard-delete flow for Article 17 requests.
|
||||
- **Pluggable file storage.** Object storage (S3-compatible) by default, with a one-command migration script to switch backends.
|
||||
|
||||
---
|
||||
|
||||
## 1. Sales pipeline
|
||||
|
||||
- **Kanban board** across seven canonical stages (Enquiry → Qualified → Nurturing → EOI → Reservation → Deposit Paid → Contract) with drag-and-drop, per-column counts, and completed-deal hiding.
|
||||
- **List view** with sorting, filtering, paging, card / table toggle, bulk actions, and saved views per user.
|
||||
- **Deal detail page** with tabs for overview, EOI, contract, reservation, documents, contact log, notes, and timeline. Every field is inline-editable in place.
|
||||
- **Multi-berth interests.** A single deal can attach multiple berths with three independent flags: which berth is primary, which are publicly "under offer", and which are included in the EOI bundle.
|
||||
- **Auto-advancing stages.** Deposits hitting their expected amount, EOI completion, contract signing, etc. move the deal forward automatically; staff can override.
|
||||
- **Pipeline rules engine.** Seven configurable triggers (EOI sent, EOI signed, deposit received, contract signed, deal archived, deal completed, berth unlinked), each with auto / suggest / off modes and a per-port target berth status. Admin-tunable.
|
||||
- **Outcomes.** Terminal outcomes (won, lost to another marina, lost unqualified, lost no response, cancelled) captured via an outcome dialog with required reason.
|
||||
- **Tags, notes, contact log, and activity timeline** on every deal.
|
||||
- **Saved views and recently-viewed.** Pin reusable filter+sort snapshots; recently-viewed items appear in the topbar.
|
||||
- **Lead scoring badge** and **qualification checklist.** Per-port qualifying criteria are admin-defined; each deal shows a checklist and derived score.
|
||||
- **Bulk actions.** Change stage, add/remove tags, archive — with confirmation dialogs and audit-logged outcomes.
|
||||
- **Pipeline summary on each client.** All open and historic deals roll up onto the client detail page.
|
||||
|
||||
---
|
||||
|
||||
## 2. Berths
|
||||
|
||||
- **Catalog with list and card views**, filterable by status, area, and dimensions; every field inline-editable on the detail page.
|
||||
- **Public berth feed** at `/api/public/berths` and `/api/public/berths/[mooringNumber]` for the marketing site; status computed with a clear precedence (Sold > Under Offer > Available), served from a 5-minute cache.
|
||||
- **Versioned per-berth PDFs.** Every upload creates a new version; the current version is live. Three-tier automatic parsing (form-fields → OCR → optional AI), with mooring-number mismatch flagging.
|
||||
- **Per-port brochures.** Multiple brochures per port with one enforced default; same upload + version flow as berth PDFs.
|
||||
- **Send-berth-PDF dialog.** Branded email composition that attaches the berth PDF (or a signed-URL link when over the size threshold).
|
||||
- **Berth recommender.** Pure-SQL ranking surfacing matching berths per deal via a four-tier ladder (A/B/C/D); Tier B uses heat scoring with admin-configurable weights.
|
||||
- **Demand heat scoring.** Per-berth demand intensity, shown on the dashboard widget and each berth's detail panel.
|
||||
- **Active interests popover.** Hover/tap any berth to see which deals are currently linked.
|
||||
- **Bulk price edit.** A sheet for updating prices across many berths at once.
|
||||
- **Bulk-add berths wizard** for onboarding inventory in batches.
|
||||
- **Catch-up wizard** to reconcile legacy state when migrating berth data.
|
||||
|
||||
---
|
||||
|
||||
## 3. Yachts
|
||||
|
||||
- **Polymorphic ownership.** A yacht can be owned by a client or a company; respected throughout search, documents, pipelines, and reports.
|
||||
- **Ownership history.** Every transfer recorded with date and parties; previous owners visible from the yacht detail.
|
||||
- **Yacht transfer dialog** for moving a yacht between owners (client → client, client → company, etc.) with audit trail.
|
||||
- **Inline editing** of all dimensions and identifiers; dimensions normalised and validated.
|
||||
- **Reusable yacht picker** — the same searchable picker appears when creating a deal, attaching a document, or filing under an entity.
|
||||
|
||||
---
|
||||
|
||||
## 4. Companies & memberships
|
||||
|
||||
- **Companies list and detail** with tabs for overview, members, owned yachts, and files.
|
||||
- **Members management.** Add/remove members with active/inactive state and roles; membership reach feeds the documents projection so a client sees relevant company files automatically.
|
||||
- **Polymorphic ownership.** Companies can own yachts and be the contractual party on a deal.
|
||||
- **Files tab** showing both directly-attached files and files reaching through related entities.
|
||||
|
||||
---
|
||||
|
||||
## 5. Clients
|
||||
|
||||
- **Single detail page** with tabs for overview, deals, yachts, companies, files, contact log, and notes.
|
||||
- **Inline editing everywhere** — name, addresses, phone numbers, emails, sales rep, communication preferences.
|
||||
- **Multi-channel contacts.** Multiple emails and phone numbers per client, with primary flagging and canonical phone normalisation for reliable search and matching.
|
||||
- **Audit-driven field history.** Per-field history icon shows who changed a value, when, and the previous value.
|
||||
- **Tags, notes, and contact log** via shared components for a consistent experience.
|
||||
- **Pipeline summary.** All a client's deals (open and closed) roll up onto the detail page.
|
||||
- **Smart archive / smart restore.** Archiving cascades related state intelligently; restore previews exactly what comes back.
|
||||
- **Hard-delete with bulk variant** behind a permission gate.
|
||||
- **GDPR Article 15 export button.** One click queues a ZIP bundle (JSON + readable HTML) and emails a signed download link; auto-deletes after 30 days.
|
||||
- **Dedup engine.** Surfaces probable duplicates and offers a merge flow that consolidates linked records, notes, files, and audit trail.
|
||||
- **Send-documents dialog** for branded multi-attachment sends from any client.
|
||||
|
||||
---
|
||||
|
||||
## 6. Documents hub
|
||||
|
||||
- **Folder tree** with nestable subfolders, drag-and-drop move, rename, and soft-rescue delete (children re-parent rather than disappear).
|
||||
- **System folders per entity type** — `Clients/`, `Companies/`, `Yachts/` — auto-populated with per-entity subfolders on first use.
|
||||
- **Auto-filing on signing.** When a signing envelope completes, the signed PDF lands in the correct entity folder automatically, based on who owns the deal.
|
||||
- **Aggregated view across relationships.** A client's files plus files attached to their companies and yachts, grouped under clear headings (Directly Attached / From Company / From Yacht / From Client), each group capped for skimmability.
|
||||
- **Rich file preview.** PDFs render inline; images preview at sensible sizes; everything else gets an icon, type label, and download.
|
||||
- **Upload-for-signing dialog.** Send any file straight into a signing flow from the hub.
|
||||
- **In-flight workflow tracker** — which envelopes are mid-signing across the aggregated reach.
|
||||
- **Permissions** scoped by role: separate `view` and `manage_folders` verbs; system folders immutable via API.
|
||||
- **Recent files** surfaced in the topbar and global search.
|
||||
|
||||
---
|
||||
|
||||
## 7. EOI generation & document signing
|
||||
|
||||
- **Two pathways from one model.** EOIs generated through document-signing templates (primary) or filled into the in-app EOI PDF directly; both share the same data context.
|
||||
- **Multi-berth EOI ranges.** Bundled berths render a compact range ("A1–A3, B5–B7") in the Berth Number field; the CRM shows the full set as chips. Catalogued merge tokens are enforced at template-creation time.
|
||||
- **Configurable signing order.** Parallel or sequential per port, with a tri-state default (use template default / always parallel / always sequential).
|
||||
- **Automation modes** per deal: manual, sequential auto (advances on each signature), or concurrent auto (everyone signs at once). Mode changes audit-logged.
|
||||
- **Idempotent webhook handling.** Retries don't double-write; status changes normalised across both supported API versions; 5-minute polling safety net for missed webhooks.
|
||||
- **Rejection reasons captured** when a signer declines.
|
||||
- **Reminders and voids** surfaced directly from the deal detail.
|
||||
- **Embedded signing card** for in-app signing where appropriate.
|
||||
- **External EOI upload.** Record an EOI signed outside the system (PDF + counterparty list).
|
||||
- **Webhook health card** in admin showing recent deliveries, failures, and a "test now" action.
|
||||
- **Per-port signing configuration** — provider instance, API key, signing order, redirect URL.
|
||||
|
||||
---
|
||||
|
||||
## 8. Email send-outs
|
||||
|
||||
- **Per-port branded templates.** Every transactional email (invites, signing notifications, residential and berth enquiries, contract comms, digests, etc.) shares one branded shell that applies the port's branding automatically.
|
||||
- **Configurable send-from accounts.** Per-port human send-from (e.g. `sales@portnimara.com`) and automation send-from (e.g. `noreply@portnimara.com`). SMTP/IMAP credentials encrypted at rest; APIs return only "is set" markers.
|
||||
- **Compose dialog** with rich body (markdown rendered safely with a strict allow-list), multi-attachment, and live preview.
|
||||
- **Smart attachment handling.** Files over a per-port size threshold ship as 24-hour signed-URL links instead of attaching.
|
||||
- **Send rate limit** (50 sends/user/hour) to protect deliverability.
|
||||
- **Email audit log.** Every send recorded with recipient list, body, attachments, and links; admin-browsable.
|
||||
- **Inbound bounce monitoring.** A scheduled job (every 15 minutes) reads non-delivery reports and matches them to the original send.
|
||||
- **Email threads** — replies to a CRM-originated email are threaded under the original.
|
||||
- **Tracked-link composer.** Per-recipient tracked links for open and click-through attribution.
|
||||
- **Per-port template overrides** from admin, without code changes.
|
||||
- **Notification digests.** Hourly digest assembled from each user's unread notifications above a threshold.
|
||||
|
||||
---
|
||||
|
||||
## 9. Reports
|
||||
|
||||
- **Sales report** with KPI strip (deals open, EOIs sent this month, deposits received, win rate, average days-in-stage, conversion by source, etc.), pipeline funnel, stage-velocity chart, source-conversion chart, rep leaderboard, deal-heat panel, win-rate-over-time line, and supporting detail tables. All filters (stage, lead category, outcome) apply live.
|
||||
- **Operational report** with an operational heatmap and signing-box plot for spotting signing/operations bottlenecks.
|
||||
- **Custom report builder (MVP).** Pick an entity, choose columns, pick a date range, and run. Four entities live at launch; more entities and column-level controls roll out incrementally.
|
||||
- **Save / load / save-as templates.** Any report configuration saved as a named template with an optional shareable link, re-runnable on demand.
|
||||
- **Scheduled runs.** Weekly, monthly, or quarterly cadences; runs on schedule and optionally emails recipients a branded PDF. Run history browsable in admin.
|
||||
- **PDF exports** server-side rendered with a branded cover page; CSV and Excel exports available client-side from every list.
|
||||
- **Status badges** for each scheduled run.
|
||||
- **Charts** combining standard bars/lines/pies with dedicated heatmap and funnel rendering.
|
||||
|
||||
---
|
||||
|
||||
## 10. Admin
|
||||
|
||||
- **Organised admin surface** grouping all settings into clear domains: Brand & Communication, Sales Workflow, Catalog, Identity & Access, Inbox & Data Quality, Integrations, and System & Observability.
|
||||
- **Permissions UI.** Browse roles, edit role definitions, browse users, and assign per-user overrides via a visual permission matrix.
|
||||
- **Settings registry.** A single, validated source of truth for every configurable setting, scoped per port.
|
||||
- **System monitoring dashboard.** Service health, queue depth, and reconcile state in one place.
|
||||
- **Port configuration** for adding new ports with their own branding, currency, timezone, and email background.
|
||||
- **Self-service customisation.** Tags, vocabularies, custom fields, and supplemental info-request forms that tenants can shape themselves, without engineering involvement.
|
||||
- **Onboarding checklist** to guide new ports through setup.
|
||||
|
||||
---
|
||||
|
||||
## 11. Search
|
||||
|
||||
- **Topbar search across every entity** — clients, residential clients, yachts, companies, deals, berths, invoices, expenses, documents, files, reminders, brochures, tags, plus navigation/settings deep-links.
|
||||
- **Multiple match strategies.** Full-text for documents, partial-word for names and titles, fuzzy trigram matching ("Jhon" finds "John"), canonical phone-number matching that ignores formatting, and direct ID lookup.
|
||||
- **Affinity ranking.** Recently-touched results are promoted.
|
||||
- **Cross-port super-admin pass.** Super-admins see other-port matches in a separate, clearly-labelled section.
|
||||
- **Permission-aware.** Viewers don't see results they couldn't open.
|
||||
- **Mobile search overlay** designed for thumb reach.
|
||||
- **Highlighted match terms** in each result.
|
||||
- **Admin search across the seven IA domains** — every admin page reachable from the topbar by keyword.
|
||||
|
||||
---
|
||||
|
||||
## 12. Activity feed & notifications
|
||||
|
||||
- **Dashboard activity widget** showing recent meaningful events across the port.
|
||||
- **Per-entity activity feed** on every client, deal, berth, yacht, and company detail page.
|
||||
- **Standardised verb vocabulary** — created, updated, archived, restored, merged, transferred, sent, signed, completed, rejected, voided, etc. Legacy events re-mapped to the current vocabulary.
|
||||
- **My reminders rail** on the dashboard surfacing due and overdue follow-ups.
|
||||
- **Reminders engine** with admin configuration (cadence, severity, recipients).
|
||||
- **Alert engine.** Rule-based alerts evaluated every 5 minutes; admins define rules, the engine generates notifications when they fire.
|
||||
- **In-app inbox** in the topbar.
|
||||
- **Hourly notification digest email** when unread items pass a threshold.
|
||||
|
||||
---
|
||||
|
||||
## 13. Analytics
|
||||
|
||||
- **Website-analytics dashboard** in the CRM: realtime visitors panel, world map, sessions list, session detail sheet, weekly heatmap, pageviews chart, top referrers / pages / devices, and per-metric detail shells.
|
||||
- **Per-port project linking** to a website analytics project — CRM outcome events (EOI sent, deposit received, etc.) cross-post so marketing and sales metrics share a timeline.
|
||||
- **Email-open pixel.** Branded sends include an open-tracking pixel; opens recorded against the original send and shown in the send audit log.
|
||||
|
||||
---
|
||||
|
||||
## 14. Mobile & responsive design
|
||||
|
||||
- **Dedicated mobile shell** on small viewports: mobile topbar, bottom tab bar, and a "more" sheet for overflow navigation.
|
||||
- **Card mode toggle on every list** — switch between table and card view; card view defaults on mobile.
|
||||
- **Mobile search overlay** designed for thumb reach.
|
||||
- **Responsive tab strips** that collapse intelligently.
|
||||
- **Touch-tuned form controls** — phone input, country picker, and timezone picker built for mobile keyboards.
|
||||
|
||||
---
|
||||
|
||||
## 15. Security & compliance
|
||||
|
||||
- **Authentication via `better-auth`** with session cookies; branded login, reset-password, and set-password surfaces.
|
||||
- **CRM invitations** via a token-based admin-driven invite flow.
|
||||
- **Granular RBAC.** Per-resource, per-action permissions applied at the service layer, not just the UI.
|
||||
- **Audit log everywhere.** All meaningful actions recorded with severity tier; 90-day retention configurable.
|
||||
- **GDPR Article 15 exports** (one-click bundle, signed download, 30-day cleanup) and Article 17 hard-delete with restore preview.
|
||||
- **PII masking at audit-write time.**
|
||||
- **Magic-byte PDF validation** on every upload path (in-server and presigned-PUT).
|
||||
- **Timing-safe webhook verification** for document-signing callbacks.
|
||||
- **Defense-in-depth port scoping** on every aggregated query — joins double-check `port_id`.
|
||||
- **30-second timeouts on object-storage calls** so a slow host can't stall the application.
|
||||
- **Per-port encryption-at-rest** for SMTP/IMAP credentials.
|
||||
- **Pre-commit hooks block accidental secret commits** (`.env` files including `.env.example`).
|
||||
|
||||
---
|
||||
|
||||
## 16. Multi-tenancy at port level
|
||||
|
||||
- **Per-port URL slug** — own URL prefix, brand, and configuration.
|
||||
- **Per-port branding** — logo, primary colour, default currency, timezone, branded email background.
|
||||
- **Per-port email templates** — every transactional template overridable per port from admin.
|
||||
- **Per-port signing configuration** — provider API version, API key, signing order, redirect URL.
|
||||
- **Per-port storage backend** — S3-compatible or filesystem, switchable via migration script.
|
||||
- **Per-port currency and timezone** flowing through the scheduler, dashboard timezone-drift banner, recommender deposit defaults, and every report.
|
||||
- **Per-port sales settings** — qualification criteria, pipeline rules, recommender weights, send-from accounts, and AI budgets, all scoped to the port.
|
||||
- **Cross-port super-admin search** — super-admins see other-port matches in a clearly-labelled secondary section; otherwise queries scope to the current port.
|
||||
@@ -471,6 +471,12 @@ Initiative 2's audit so we don't double-test.
|
||||
|
||||
**Status:** OPEN · High effort · Likely blocker for cutover
|
||||
|
||||
> **Infra cutover plan:** `docs/deployment-plan.md` — prod deploy of the CRM
|
||||
> to `crm.portnimara.com` (nginx + certbot + registry-image compose),
|
||||
> Gitea/CI access, and the Documenso backup + safe-upgrade procedure. Access
|
||||
> (SSH + Gitea API) established 2026-05-31; no prod changes without explicit
|
||||
> approval. Deployment creds in `private/deployment-creds.md` (gitignored).
|
||||
|
||||
User ask: "start pulling all existing prod data from the old system and
|
||||
connected systems (we'll have to backfill the EOIs by pulling them
|
||||
through MinIO — it's a fucking mess so I'll really need your help
|
||||
@@ -527,3 +533,71 @@ Cutover plan:
|
||||
- **Soft launch vs hard cutover?** Hard cutover is simpler operationally
|
||||
but risky; soft launch (parallel writes for a week) is safer but
|
||||
requires the old system to keep accepting writes for longer.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-01 — Feature-completeness sweep & launch-prep decisions
|
||||
|
||||
A read-only sweep (ahead of the ~same-day launch) checked the whole
|
||||
platform for half-built / stubbed surfaces beyond the known Reports
|
||||
gaps. It resolved two stale-doc contradictions: **Documenso signing
|
||||
phases 2–7 are fully built and wired** (`BACKLOG.md` §A is stale on
|
||||
this), and the **interest Contract/Reservation tabs are fully built**
|
||||
(not "coming soon" cards). Findings + decisions below.
|
||||
|
||||
**Decision (per Matt, 2026-06-01):** launch is ~today, so **ship what's
|
||||
done, hide what's not, defer the big builds** — do NOT revert to the old
|
||||
desktop-spreadsheet reports (a downgrade), and do NOT rush the
|
||||
unproven full builds onto a same-day prod launch.
|
||||
|
||||
### Shipped today (launch-prep, low-risk; SHIPPED)
|
||||
|
||||
- **Hid Financial + Marketing report cards** from the reports landing
|
||||
(`reports/page.tsx`) — both were "Builder in development" placeholders
|
||||
gated on unbuilt data sources (Init 1b/1c). The reports section ships
|
||||
with the working **Sales + Operational + Custom** reports + templates +
|
||||
scheduling + PDF/CSV/Excel exports. The basic Custom builder already
|
||||
covers the old desktop-report use case (entity + columns + date range +
|
||||
export) — parity-plus, not a regression.
|
||||
- **Trimmed the Custom-report card copy** so it stops promising
|
||||
group-by/filters/dimensions it doesn't yet have (the builder page
|
||||
header was already honest).
|
||||
- **Hid the Bulk Import mockup** from nav + search
|
||||
(`admin-sections-browser.tsx`, `search-nav-catalog.ts`). The static
|
||||
`/admin/import` mockup is now unreachable from the UI (route still
|
||||
resolves by direct URL).
|
||||
- **Corrected client-facing doc over-claims** in `features-list.md` +
|
||||
`new-system-feature-summary.md` (removed the waiting-list
|
||||
"next-in-line notification" claim — built but hidden; removed Import
|
||||
from the admin-pages list, 43→42).
|
||||
|
||||
### Deferred to post-launch (tracked here; none launch-blocking)
|
||||
|
||||
- **Full Bulk CSV/XLSX importer** — design APPROVED + spec written:
|
||||
`docs/superpowers/specs/2026-06-01-bulk-import-design.md` (generic
|
||||
engine + per-entity adapter registry; 7 entities; column-mapping,
|
||||
dry-run, dedup, per-batch undo). Cutover data migration runs through
|
||||
the existing CLI scripts (`import-berths-from-nocodb.ts` + the
|
||||
Initiative 5 migration scripts), so the UI importer is **not needed
|
||||
for launch**.
|
||||
- **Full Custom-report builder** — group-by + aggregates, sort,
|
||||
per-column filter rows (AND/OR), debounced live preview, the remaining
|
||||
6 of 10 entities, per-role PII column whitelist. Architecture decided
|
||||
(per-column expression map + generic Drizzle query composer); spec
|
||||
deferred. Basic builder ships as-is.
|
||||
- **Berth Waiting List** — un-hide the existing `WaitingListManager`
|
||||
(component + API already built) + build the availability-triggered
|
||||
next-in-line notification (today only a `notifyPref` column is stored;
|
||||
no sender exists). Component currently orphaned; berth tab removed.
|
||||
- **Berth Maintenance Log** — backend (API + service) exists; needs a UI
|
||||
tab mirroring the waiting-list manager.
|
||||
- **Contract/Reservation paper-upload misroute (BUG)** — the "Upload
|
||||
paper-signed copy" button on the contract/reservation tabs POSTs to the
|
||||
EOI-only endpoint (`/external-eoi`), filing a paper-signed
|
||||
contract/reservation as an EOI. Fix: add contract/reservation
|
||||
paper-upload endpoints + point `ExternalEoiUploadDialog` at the right
|
||||
one per docType. ("Mark as signed without file" already works for all
|
||||
three types.)
|
||||
- **Marketing + Financial reports** — remain unbuilt + now hidden; gated
|
||||
on Init 1b (website UTM/inquiry cutover) and Init 1c (invoices-module
|
||||
decision) respectively.
|
||||
|
||||
338
docs/new-system-feature-summary.md
Normal file
338
docs/new-system-feature-summary.md
Normal file
@@ -0,0 +1,338 @@
|
||||
# Port Nimara CRM — What's New & What's Improved
|
||||
|
||||
A client-friendly summary of the new Port Nimara CRM, framed against what the previous system provided. The new platform is a complete, purpose-built CRM that replaces a website + spreadsheet-style data store with a single integrated workspace for sales, berths, documents, communications, and reporting.
|
||||
|
||||
> Scope note: this summary covers the features that are ready for the beta launch. The new client portal, the tenancies module, and the new invoicing module are still being finalised and are intentionally not included here.
|
||||
|
||||
---
|
||||
|
||||
## At a glance
|
||||
|
||||
**Previously**, day-to-day sales work happened across three places: the public website (where enquiries landed), the back-end database tool (where data was inspected and edited), and a separate internal portal (where signing, expenses, and a handful of staff tools lived).
|
||||
|
||||
**Now**, all of that lives inside a single, branded CRM at `crm.portnimara.com`-style URLs (one per port). The website still publishes berths and accepts enquiries — but those enquiries flow into the CRM and are managed there, from first contact through deposit, contract, and signing.
|
||||
|
||||
The CRM is built on a dedicated relational database designed specifically for marina sales workflows, with real-time updates, role-based permissions, a full audit trail, and a clean modern interface that adapts to mobile.
|
||||
|
||||
---
|
||||
|
||||
## Platform-level upgrades
|
||||
|
||||
These improvements apply across every feature area:
|
||||
|
||||
- **Purpose-built database.** The system runs on a dedicated relational database (PostgreSQL) modelled specifically for marina sales. Compared with the previous spreadsheet-style data store, it's faster on large data sets, supports rich relationships between entities (clients, companies, yachts, berths, deals, documents), and enforces data integrity so duplicates and broken links don't slip through.
|
||||
- **Real-time updates.** When a colleague edits a deal, advances a stage, attaches a file, or completes a signing, every other open window updates within a second. No more "refresh to see what changed".
|
||||
- **Per-port branding and configuration.** Each port has its own URL slug, logo, primary colour, default currency, timezone, and email templates. Emails, PDFs, and the in-app shell all pick up the right brand automatically.
|
||||
- **Granular role-based permissions.** Roles are defined per resource (clients, berths, documents, expenses, reports, etc.) with separate view / create / edit / delete / export verbs. Admins can override permissions per user as well as per role.
|
||||
- **Full audit trail.** Every meaningful change (who, what, before-and-after, when) is recorded, retained for 90 days, and searchable. Used in the activity feed, the field-history popovers, and the admin audit log.
|
||||
- **Backups and operational tooling.** Automatic daily database backups, weekly cleanup, configurable retention windows, and a built-in system-monitoring dashboard for staff to verify the queue and integrations are healthy.
|
||||
- **Background job queue.** Heavy or slow work (PDF generation, email sending, exports, webhook retries, bounce polling) runs on a managed queue so the interface stays responsive and nothing is silently lost.
|
||||
- **GDPR-ready.** One-click Article 15 data exports per client, automatic 30-day cleanup of export bundles, and a permissioned hard-delete flow for Article 17 requests.
|
||||
- **Pluggable file storage.** Files live in object storage (S3-compatible) by default, with a one-command migration script to switch backends without rewriting any code.
|
||||
|
||||
---
|
||||
|
||||
## 1. Sales pipeline
|
||||
|
||||
A complete sales CRM where the team manages every deal from first enquiry to contract.
|
||||
|
||||
- **Kanban board** across seven canonical stages (Enquiry → Qualified → Nurturing → EOI → Reservation → Deposit Paid → Contract) with drag-and-drop, per-column counts, and completed-deal hiding.
|
||||
- **List view** with sorting, filtering, paging, card / table toggle, bulk actions, and saved views per user.
|
||||
- **Deal detail page** with tabs for overview, EOI, contract, reservation, documents, contact log, notes, and timeline. Every field is inline-editable in place — no separate edit modal to wade through.
|
||||
- **Multi-berth interests.** A single deal can attach multiple berths with three independent flags: which berth is the deal's primary, which are publicly "under offer", and which are included in the EOI bundle. The previous system stored at most a single berth link per enquiry.
|
||||
- **Auto-advancing stages.** Deposits hitting their expected amount, EOI completion, contract signing, etc. move the deal forward automatically; staff can intervene if the rules need overriding.
|
||||
- **Pipeline rules engine.** Seven configurable triggers (EOI sent, EOI signed, deposit received, contract signed, deal archived, deal completed, berth unlinked) each with auto / suggest / off modes and a per-port target berth status. Admins can tune the rules without engineering involvement.
|
||||
- **Outcomes.** Terminal outcomes (won, lost to another marina, lost unqualified, lost no response, cancelled) are captured via an outcome dialog with required reason capture.
|
||||
- **Tags, notes, contact log, and activity timeline** on every deal. Tags are inline-editable; notes use a single underlying engine shared across clients, deals, yachts, and companies.
|
||||
- **Saved views and recently-viewed.** Each user can pin reusable filter+sort snapshots; recently-viewed items appear in the topbar for quick return.
|
||||
- **Lead scoring badge** and **qualification checklist.** Per-port qualifying criteria are admin-defined; each deal shows a checklist and a derived score.
|
||||
- **Bulk actions.** Change stage, add/remove tags, archive — with confirmation dialogs and audit-logged outcomes.
|
||||
- **Pipeline summary on each client.** All a client's open and historic deals roll up onto their detail page.
|
||||
|
||||
_Previously, deal management happened directly inside the back-end data tool — no kanban, no stage workflow, no auto-advance, no tags, no notes per deal, no scoring, and no per-deal timeline._
|
||||
|
||||
---
|
||||
|
||||
## 2. Berths
|
||||
|
||||
Catalog, public-facing feed, recommender, demand signals, and rich per-berth artefacts.
|
||||
|
||||
- **Catalog with list and card views**, filterable by status, area, dimensions; every field inline-editable on the detail page.
|
||||
- **Public berth feed** at `/api/public/berths` and `/api/public/berths/[mooringNumber]` feeds the marketing site. Output mirrors the previous shape exactly so the website didn't need a rewrite; status is computed with a clear precedence (Sold > Under Offer > Available) and served from a 5-minute cache for fast page loads.
|
||||
- **Per-berth PDFs are versioned.** Every upload creates a new version; the current version is the live one. PDFs are parsed automatically through three tiers (form-fields → OCR → optional AI), and the system flags mismatches when the mooring number on the PDF doesn't match the berth.
|
||||
- **Per-port brochures.** Multiple brochures supported per port with one default enforced. Same upload + version flow as berth PDFs.
|
||||
- **Send-berth-PDF dialog.** Branded email composition that attaches the berth PDF (or shares a signed-URL link when the file is over the size threshold).
|
||||
- **Berth recommender.** A pure-SQL ranking that surfaces matching berths per deal via a four-tier ladder (A/B/C/D). Tier B uses heat scoring; weights are configurable in admin so the model can be tuned per port.
|
||||
- **Demand heat scoring.** Per-berth demand intensity, shown on the dashboard widget and on each berth's detail panel.
|
||||
- **Active interests popover.** Hover/tap any berth to see which deals are currently linked to it.
|
||||
- **Bulk price edit.** A sheet for updating prices across many berths at once.
|
||||
- **Bulk-add berths wizard** for onboarding new inventory in batches.
|
||||
- **Catch-up wizard** to reconcile legacy state when migrating berth data.
|
||||
|
||||
_Previously, berths were a flat list with a basic dimension filter on the public site. There was no recommender, no demand heat, no per-berth PDF versioning, no bulk price editor, and no internal berth detail page._
|
||||
|
||||
---
|
||||
|
||||
## 3. Yachts
|
||||
|
||||
First-class yacht records with proper ownership and history.
|
||||
|
||||
- **Polymorphic ownership.** A yacht can be owned by either a client (individual) or a company; the system models this correctly throughout — search, documents, pipelines, and reports all respect the discriminator.
|
||||
- **Ownership history.** Every transfer is recorded with date and parties; previous owners are visible from the yacht detail.
|
||||
- **Yacht transfer dialog** for moving a yacht between owners (client → client, client → company, etc.) with audit trail.
|
||||
- **Inline editing** of all dimensions and identifiers; dimensions are normalised and validated.
|
||||
- **Yacht picker reused everywhere** — when creating a deal, attaching a document, or filing under an entity, the same searchable picker appears.
|
||||
|
||||
_Previously, yachts were not stored as their own records — they were free-text fields on enquiry submissions._
|
||||
|
||||
---
|
||||
|
||||
## 4. Companies & memberships
|
||||
|
||||
First-class company entities with member relationships.
|
||||
|
||||
- **Companies list and detail** with tabs for overview, members, owned yachts, and files.
|
||||
- **Members management.** Add/remove members with active/inactive state and roles. Membership reach feeds into the documents projection (a client gets to see relevant company files automatically).
|
||||
- **Polymorphic ownership.** Companies can own yachts and be the contractual party on a deal, mirrored across the codebase rather than improvised per surface.
|
||||
- **Files tab** on company detail showing both directly-attached files and files reaching through related entities.
|
||||
|
||||
_Previously, companies did not exist as a separate concept; everything was attributed to a single named individual._
|
||||
|
||||
---
|
||||
|
||||
## 5. Clients
|
||||
|
||||
The detail page each contact deserves.
|
||||
|
||||
- **Single detail page** with tabs for overview, deals, yachts, companies, files, contact log, and notes.
|
||||
- **Inline editing everywhere.** Name, addresses, phone numbers, emails, sales rep, communication preferences — all editable in place via small inline fields.
|
||||
- **Multi-channel contacts.** Multiple emails and phone numbers per client, with primary flagging and canonical normalisation (phone numbers are normalised to a single international format for reliable search and matching).
|
||||
- **Audit-driven field history.** Click any field's history icon to see who changed it, when, and what the previous value was.
|
||||
- **Tags, notes, and contact log** — all the same shared components as elsewhere, so the experience is consistent.
|
||||
- **Pipeline summary.** All a client's deals — open and closed — roll up onto their detail page.
|
||||
- **Smart archive / smart restore.** Archive a client and the system handles cascading state (related deals, files) intelligently; restore previews exactly what will come back.
|
||||
- **Hard-delete with bulk variant** behind a permission gate, for genuine "remove from the system" requests.
|
||||
- **GDPR Article 15 export button.** One click queues a ZIP bundle (JSON + readable HTML) and emails the client a signed download link; the bundle auto-deletes after 30 days.
|
||||
- **Dedup engine.** The system surfaces probable duplicates and offers a merge flow that consolidates linked records, notes, files, and audit trail correctly.
|
||||
- **Send-documents dialog** for branded multi-attachment sends from any client.
|
||||
|
||||
_Previously, contact records were flat rows in the back-end tool — no detail page, no inline editing, no audit history, no GDPR export, no dedup, no per-client deal roll-up._
|
||||
|
||||
---
|
||||
|
||||
## 6. Documents hub
|
||||
|
||||
A nestable folder tree per port with intelligent auto-filing.
|
||||
|
||||
- **Tree of folders** with nestable subfolders, drag-and-drop move, rename, soft-rescue delete (children re-parent rather than disappear).
|
||||
- **System folders for each entity type** — `Clients/`, `Companies/`, `Yachts/` — auto-populated with per-entity subfolders the first time a record needs one.
|
||||
- **Auto-filing on signing.** When a Documenso envelope completes, the signed PDF lands in the right entity folder automatically based on who owns the deal — no manual filing needed.
|
||||
- **Aggregated view across relationships.** Open a client and you also see files attached to their companies and yachts, grouped under clear headings (Directly Attached / From Company / From Yacht / From Client). Each group is capped to keep the view skimmable; deeper drill-down is one click away.
|
||||
- **Rich file preview.** PDFs render inline; images preview at sensible sizes; everything else gets an icon, type label, and download.
|
||||
- **Upload for signing dialog.** Send any file straight into a Documenso signing flow without leaving the documents hub.
|
||||
- **In-flight workflow tracker** — see which envelopes are mid-signing across the same aggregated reach.
|
||||
- **Permissions** scoped by role: separate `view` and `manage_folders` verbs; system folders are immutable via API to keep the structure clean.
|
||||
- **Recent files** surface in the topbar and global search.
|
||||
|
||||
_Previously, file management lived in the separate internal portal as a flat S3 file browser with no folder tree, no auto-filing, no aggregated-by-entity view, and no signing-integration on individual files._
|
||||
|
||||
---
|
||||
|
||||
## 7. EOI generation & Documenso signing
|
||||
|
||||
Template-driven EOIs with multi-berth support and resilient signing.
|
||||
|
||||
- **Two pathways from one underlying model.** EOIs can be generated through Documenso templates (the primary path) or filled into the in-app EOI PDF directly. Both share the same data context, so any change to a deal is reflected identically.
|
||||
- **Multi-berth EOI ranges.** When an EOI bundles multiple berths, the document automatically renders a compact range ("A1–A3, B5–B7") in the Berth Number field, and the CRM UI shows the full set as chips. The catalogued merge tokens are enforced at template-creation time so a mistyped placeholder cannot silently slip into a generated document.
|
||||
- **Configurable signing order.** Parallel or sequential signing per port, with a tri-state default ("use template default / always parallel / always sequential").
|
||||
- **Automation modes** per deal: manual (staff sends each step), sequential auto (system advances on each signature), or concurrent auto (everyone signs at once). Mode changes are audit-logged.
|
||||
- **Idempotent webhook handling.** Documenso retries don't double-write; status changes are normalised across both supported API versions; the system polls every 5 minutes as a safety net if a webhook is missed.
|
||||
- **Rejection reasons captured** when a signer declines.
|
||||
- **Reminders and voids.** The CRM surfaces send-reminder and void-envelope actions directly from the deal detail.
|
||||
- **Embedded signing card** for clients to sign in-app where appropriate.
|
||||
- **External EOI upload.** Record an EOI that was signed outside the system (PDF upload + counterparty list) without breaking the rest of the deal flow.
|
||||
- **Webhook health card** in admin shows recent deliveries, failures, and a "test now" affordance.
|
||||
- **Per-port Documenso configuration.** Each port can target its own Documenso instance, API key, signing order, and redirect URL.
|
||||
|
||||
_Previously, signing was a Documenso embed hosted from the internal portal with token-based redirects, no multi-berth range support, no idempotent webhook handling, no automation modes, and no health diagnostics in the UI._
|
||||
|
||||
---
|
||||
|
||||
## 8. Email send-outs
|
||||
|
||||
Branded, audited, configurable outbound mail.
|
||||
|
||||
- **Per-port branded templates.** Every transactional email (invites, signing notifications, residential and berth enquiries, contract-related comms, digests, etc.) shares a single branded shell — port logo, blurred overhead background, consistent typography — that picks up the port's branding automatically.
|
||||
- **Configurable send-from accounts.** Each port can configure its human send-from (e.g. `sales@portnimara.com`) and its automation send-from (e.g. `noreply@portnimara.com`). SMTP/IMAP credentials are encrypted at rest; API endpoints return only "is set" markers, never the password.
|
||||
- **Compose dialog** with rich body (markdown rendered safely with a strict allow-list), multi-attachment, and live preview.
|
||||
- **Smart attachment handling.** Files over a configurable per-port size threshold ship as 24-hour signed-URL links instead of attaching directly, keeping email deliverable.
|
||||
- **Send rate limit** (50 sends/user/hour) to protect deliverability reputation.
|
||||
- **Email audit log.** Every send is recorded with recipient list, body, attachments, and links; admin can browse the full send log.
|
||||
- **Inbound bounce monitoring.** A scheduled job (every 15 minutes) reads non-delivery reports and matches them back to the original send so staff know a message bounced.
|
||||
- **Email threads** stitched together — replies to a CRM-originated email are threaded under the original.
|
||||
- **Tracked-link composer.** Generate per-recipient tracked links so opens and click-throughs can be attributed back.
|
||||
- **Per-port template overrides.** Admin can override any transactional template per port without touching code.
|
||||
- **Notification digests.** Hourly digest assembled from each user's unread notifications above a threshold.
|
||||
|
||||
_Previously, transactional email was sent via Gmail SMTP from string-template builders, with no per-port branding override, no send audit log, no bounce monitoring, no attachment-threshold logic, no rate limiting, and no per-template overrides without a redeploy._
|
||||
|
||||
---
|
||||
|
||||
## 9. Reports
|
||||
|
||||
Live Sales and Operational dashboards, plus a custom builder, scheduling, and exports.
|
||||
|
||||
- **Sales report** with KPI strip (deals open, EOIs sent this month, deposits received, win rate, average days-in-stage, conversion by source, etc.), pipeline funnel, stage-velocity chart, source-conversion chart, rep leaderboard, deal-heat panel, win-rate-over-time line, and supporting detail tables. Every filter (stage, lead category, outcome) applies live.
|
||||
- **Operational report** with an operational heatmap and signing-box plot — used to spot bottlenecks in the signing/operations pipeline.
|
||||
- **Custom report builder (MVP).** Pick an entity, choose columns, pick a date range, and run. Four entities are live at launch; additional entities and column-level controls roll out incrementally.
|
||||
- **Save / load / save-as templates.** Any report configuration can be saved as a named template with an optional shareable link, then re-run on demand.
|
||||
- **Scheduled runs.** Weekly, monthly, or quarterly cadences; system runs the report on schedule and (optionally) emails the recipients a branded PDF. Run history is browsable in admin.
|
||||
- **PDF exports** are server-side rendered with a branded cover page. CSV and Excel exports also available client-side from every list.
|
||||
- **Status badges** for each scheduled run so admin can see at a glance which schedules are healthy.
|
||||
- **Charts** use a mix of standard chart libraries — simple bars/lines/pies on top of a strong charting library, with heatmaps and funnels handled by a separate engine tuned for that purpose.
|
||||
|
||||
_Previously, there were no in-system reports. Staff exported NocoDB views to spreadsheets and built reporting by hand each time._
|
||||
|
||||
---
|
||||
|
||||
## 10. Admin
|
||||
|
||||
A purpose-built admin surface organised into seven domain groups.
|
||||
|
||||
- **Admin sections browser** that groups every admin page under: Brand & Communication, Sales Workflow, Catalog, Identity & Access, Inbox & Data Quality, Integrations, and System & Observability.
|
||||
- **42 dedicated admin pages** covering: AI usage caps, audit log, backups, berths, branding, brochures, custom fields, Documenso health, duplicates, email accounts, email templates, error log, forms, inquiries, invitations, monitoring, OCR, onboarding, pipeline rules, ports, "pulse" health indicators, qualification criteria, reminders, reports admin, residential stages, roles, sends log, settings, storage, tags, templates, users, vocabularies, webhooks, and website analytics.
|
||||
- **Permissions UI.** Browse roles, edit role definitions, browse users, and assign per-user overrides through a visual permission matrix.
|
||||
- **Settings registry.** A single source of truth for every configurable setting, with sections for email, Documenso, storage, pipeline auto-advance, AI providers, application URLs, operations toggles, residential partner integration, and more. Settings are per-port and validated.
|
||||
- **System monitoring dashboard.** Service health, queue depth, queue detail, reconcile state — all in one place.
|
||||
- **Port configuration** for adding new ports with their own branding, currency, timezone, and email background.
|
||||
- **Webhooks admin** for dispatching CRM events outward to external systems.
|
||||
- **Tags, vocabularies, and custom fields** that tenants can shape themselves without engineering involvement.
|
||||
- **Forms admin** for creating supplemental info-request forms (used in qualification, residential, etc.).
|
||||
- **Onboarding checklist and banner** to guide new ports through setup.
|
||||
|
||||
_Previously, "admin" meant opening the back-end data tool directly to edit rows, with no permissions model, no role assignments, no settings UI, no monitoring, and no onboarding flow._
|
||||
|
||||
---
|
||||
|
||||
## 11. Search
|
||||
|
||||
A fast, fuzzy, permission-aware global search.
|
||||
|
||||
- **Topbar search across every entity** — clients, residential clients, yachts, companies, deals, berths, invoices, expenses, documents, files, reminders, brochures, tags, plus navigation/settings deep-links.
|
||||
- **Multiple match strategies.** Full-text search for documents, partial-word matching for names and titles, fuzzy trigram matching so "Jhon" still finds "John", canonical phone-number matching that ignores formatting differences, and direct ID lookup for paste-a-record-id workflows.
|
||||
- **Affinity ranking.** Results you've recently touched are promoted, so "your John" appears above "some other John".
|
||||
- **Cross-port super-admin pass.** Super-admin users see other-port matches in a separate, clearly-labelled section.
|
||||
- **Permission-aware.** Viewers don't see search results they couldn't open.
|
||||
- **Mobile search overlay** designed for thumb reach.
|
||||
- **Highlighted match terms** so the relevant substring jumps out in each result.
|
||||
- **Admin search across the 7 IA domains** — every admin page is reachable from the topbar with a keyword.
|
||||
|
||||
_Previously, "search" meant filtering a single NocoDB table at a time. There was no global search, no cross-entity matching, no fuzzy matching, no affinity ranking, and no admin deep-link search._
|
||||
|
||||
---
|
||||
|
||||
## 12. Activity feed & notifications
|
||||
|
||||
A unified activity feed and a notification engine for both in-app and email.
|
||||
|
||||
- **Dashboard activity widget** shows recent meaningful events across the port.
|
||||
- **Per-entity activity feed** on every client, deal, berth, yacht, and company detail page.
|
||||
- **Standardised verb vocabulary** — created, updated, archived, restored, merged, transferred, sent, signed, completed, rejected, voided, and so on. Historical legacy-stage events are re-mapped to the current vocabulary so the timeline reads consistently.
|
||||
- **My reminders rail** on the dashboard surfaces due and overdue follow-ups.
|
||||
- **Reminders engine** with admin configuration (cadence, severity, recipients).
|
||||
- **Alert engine.** Rule-based alerts evaluated every 5 minutes — admins define the rules; the engine generates notifications when they fire.
|
||||
- **In-app inbox** in the topbar.
|
||||
- **Hourly notification digest email** when unread items pass a threshold.
|
||||
|
||||
_Previously, there was no in-system activity feed, no reminders engine, and no rule-based alerting._
|
||||
|
||||
---
|
||||
|
||||
## 13. Analytics
|
||||
|
||||
Website analytics, email-open tracking, and outcome events feeding into a privacy-respecting analytics platform.
|
||||
|
||||
- **Website-analytics dashboard** in the CRM with: realtime visitors panel, world map of visitors, sessions list, session detail sheet, weekly heatmap, pageviews chart, top referrers / pages / devices, and per-metric detail shells.
|
||||
- **Per-port project linking** to a Umami analytics project — outcome events from the CRM (EOI sent, deposit received, etc.) cross-post to the same project so marketing and sales metrics share a timeline.
|
||||
- **Email-open pixel.** Branded sends include a small open-tracking pixel; opens are recorded against the original send and surface in the send audit log.
|
||||
- **Admin → website-analytics** for configuring the link to the Umami project.
|
||||
|
||||
_Previously, website analytics lived only in the standalone analytics tool; there was no integration of marketing analytics into the sales surface._
|
||||
|
||||
---
|
||||
|
||||
## 14. Mobile & responsive design
|
||||
|
||||
Designed mobile-first; every list, sheet, and dialog is touch-friendly.
|
||||
|
||||
- **Dedicated mobile shell** when the viewport is small: a mobile topbar, bottom tab bar, and a "more" sheet for overflow navigation.
|
||||
- **Card mode toggle on every list.** Switch lists between table and card view; card view is the default on mobile.
|
||||
- **Mobile search overlay** designed for thumb reach.
|
||||
- **Responsive tab strips** that collapse intelligently.
|
||||
- **Touch-tuned form controls.** Phone input, country picker, and timezone picker are all built for mobile keyboards.
|
||||
|
||||
_Previously, the back-end data tool the team used was not designed for phone use; staff worked from a laptop by necessity._
|
||||
|
||||
---
|
||||
|
||||
## 15. Security & compliance
|
||||
|
||||
A defensive posture across the stack.
|
||||
|
||||
- **Authentication via `better-auth`** with session cookies; branded login, reset-password, and set-password surfaces.
|
||||
- **CRM invitations** with a token-based admin-driven invite flow.
|
||||
- **Granular RBAC.** Per-resource, per-action permissions — applied at the service layer, not just the UI.
|
||||
- **Audit log everywhere.** All meaningful actions recorded with severity tier; 90-day retention configurable.
|
||||
- **GDPR Article 15 exports** (one-click bundle, signed download, 30-day cleanup) and Article 17 hard-delete with restore preview.
|
||||
- **PII masking at audit-write time.** Old metadata still expires per retention; new metadata is masked before insertion.
|
||||
- **Magic-byte PDF validation** on every upload path (both in-server and presigned-PUT).
|
||||
- **Timing-safe webhook verification** for Documenso (no leaky string comparisons).
|
||||
- **Defense-in-depth port scoping** on every aggregated query — even joins double-check `port_id` so a cross-tenant leak would have to bypass multiple checks.
|
||||
- **30-second timeouts on object-storage calls** so a slow MinIO/S3 host can't stall the application.
|
||||
- **Per-port encryption-at-rest** for SMTP/IMAP credentials.
|
||||
- **Pre-commit hooks block accidental commits of secrets** (`.env` files including `.env.example`).
|
||||
|
||||
_Previously, the public website ran public forms straight into the data store with reCAPTCHA only; there was no audit log on website-originated changes, no permission model on the public surface, no GDPR-Article-15 export tooling, and no PDF content validation._
|
||||
|
||||
---
|
||||
|
||||
## 16. Multi-tenancy at port level
|
||||
|
||||
The platform is designed from the ground up for multiple ports.
|
||||
|
||||
- **Per-port URL slug.** Each port has its own URL prefix, brand, and configuration.
|
||||
- **Per-port branding** — logo, primary colour, default currency, timezone, branded email background.
|
||||
- **Per-port email templates** — every transactional template can be overridden per port from admin, without engineering involvement.
|
||||
- **Per-port Documenso configuration** — API version (v1 or v2), API key, signing order, redirect URL.
|
||||
- **Per-port storage backend** — choose S3-compatible or filesystem per port; switch with a single migration script.
|
||||
- **Per-port currency and timezone** flow through the scheduler, the dashboard's timezone-drift banner, the recommender's deposit defaults, and every report.
|
||||
- **Per-port sales settings** — qualification criteria, pipeline rules, recommender weights, send-from accounts, and AI budgets are all scoped to the port.
|
||||
- **Cross-port super-admin search** — super-admins see other-port matches in a clearly-labelled secondary section; otherwise all queries scope to the current port.
|
||||
|
||||
_Previously, the system was effectively single-tenant — a separate deployment would have been needed to onboard a second port._
|
||||
|
||||
---
|
||||
|
||||
## What's net-new (not present in the previous system at all)
|
||||
|
||||
- A full sales CRM with kanban, list, detail, inline editing, stages, outcomes, tags, notes, scoring, and qualification — for staff.
|
||||
- Yachts, companies, and memberships as first-class entities (the previous system had no concept of these).
|
||||
- A nestable documents hub with auto-filing and cross-relationship aggregation.
|
||||
- Reports — Sales and Operational dashboards plus a custom builder, with templates and scheduled runs.
|
||||
- Global cross-entity search with fuzzy matching and affinity ranking.
|
||||
- An activity feed, reminders, alert engine, and notification digest.
|
||||
- Per-port multi-tenancy (branding, configuration, currency, timezone, Documenso, storage).
|
||||
- Granular role-based permissions with per-user overrides.
|
||||
- A comprehensive audit log surfaced in the activity feed, field-history popovers, and admin audit log.
|
||||
- GDPR Article 15 export tooling and Article 17 hard-delete with restore preview.
|
||||
- Background job queue + scheduled cron jobs for reliability.
|
||||
- Real-time UI updates across every open session.
|
||||
- Mobile-first design with a dedicated mobile shell.
|
||||
- Website-analytics dashboard inside the CRM (with email-open tracking and event cross-posting).
|
||||
|
||||
## What stays similar but is improved
|
||||
|
||||
- **Berth catalog and public berth feed.** The data the marketing site sees is the same shape it always was, served from a faster, properly-cached endpoint backed by the new database. The internal side adds versioned per-berth PDFs, brochures, a recommender, and demand heat scoring.
|
||||
- **EOI generation and Documenso signing.** EOIs still flow through Documenso, but with multi-berth ranges, configurable signing order, automation modes, idempotent webhook handling, a 5-minute polling safety net, in-product reminders and voids, external-EOI upload, and a webhook health diagnostic.
|
||||
- **Transactional email.** Still SMTP-backed, but now with per-port branded templates, configurable send-from accounts, audited sends, bounce monitoring, attachment-threshold smart handling, and rate limits.
|
||||
- **Public enquiry intake.** The website still accepts enquiries, but they now land in a managed inbox in the CRM with deduping, owner assignment, and full audit, instead of becoming raw rows in the data store.
|
||||
168
docs/superpowers/specs/2026-06-01-bulk-import-design.md
Normal file
168
docs/superpowers/specs/2026-06-01-bulk-import-design.md
Normal file
@@ -0,0 +1,168 @@
|
||||
# Bulk CSV/XLSX Importer — Design Spec
|
||||
|
||||
> **Status:** Approved (2026-06-01) · ready for implementation plan
|
||||
> **Driver:** Replace the static `admin/import` mockup with a real
|
||||
> self-serve importer. Primary purpose: **one-time cutover migration**
|
||||
> of legacy NocoDB/portal data into the new CRM at launch.
|
||||
> **Tracker:** `docs/launch-readiness.md` · feature-completeness batch.
|
||||
|
||||
## Purpose & scope
|
||||
|
||||
A visual importer that ingests CSV/XLSX exports of the legacy system and
|
||||
loads them into the CRM with column-mapping, dry-run preview, dedup, and
|
||||
per-batch undo. Built for the cutover migration but engineered as a
|
||||
reusable engine (it can serve ongoing ops later without a rewrite).
|
||||
|
||||
**In scope — seven entities**, imported in dependency order so foreign
|
||||
keys resolve by natural key:
|
||||
|
||||
| # | Entity | Dedup match-key | FKs resolved by natural key |
|
||||
| --- | --------------- | ---------------------------------------------------------------- | --------------------------------------- |
|
||||
| 1 | Companies | `name` (case-insensitive) | — |
|
||||
| 2 | Clients | primary `email` → fallback canonical `phone` | — |
|
||||
| 3 | Yachts | `name` + owner (or HIN if present) | owner → client email / company name |
|
||||
| 4 | Berths | `mooringNumber` (canonical `^[A-Z]+\d+$`) | — |
|
||||
| 5 | Interests/deals | default **create-new** (flag likely dupes by client+berth+stage) | client → email, primary berth → mooring |
|
||||
| 6 | Tenancies | client + berth + `startDate` | client → email, berth → mooring |
|
||||
| 7 | Expenses | `date` + `amount` + `description` (or none) | — |
|
||||
|
||||
Berths are included for UI consistency even though
|
||||
`scripts/import-berths-from-nocodb.ts` already covers them via CLI.
|
||||
|
||||
**Non-goals (v1):** full pre-update snapshot/revert of _updated_ rows
|
||||
(undo covers inserts only); streaming multi-GB files (migration files
|
||||
are small); scheduling/automation of imports; importing attachments/PDFs
|
||||
(handled by the Initiative 5 MinIO backfill scripts, separate).
|
||||
|
||||
## Architecture — generic engine + per-entity adapter registry
|
||||
|
||||
One pipeline parameterised by a per-entity **adapter**, mirroring the
|
||||
existing `src/lib/reports/custom/registry.ts` and settings-registry
|
||||
patterns.
|
||||
|
||||
`src/lib/import/registry.ts` exports `IMPORT_ENTITY_KEYS` and
|
||||
`IMPORT_REGISTRY: Record<ImportEntityKey, ImportAdapter>`. Each adapter:
|
||||
|
||||
```ts
|
||||
interface ImportAdapter {
|
||||
key: ImportEntityKey;
|
||||
label: string;
|
||||
order: number; // dependency order (companies=1 … expenses=7)
|
||||
dependsOn: ImportEntityKey[];
|
||||
/** Target fields drive the column-mapping UI + zod validation. */
|
||||
targetFields: ImportField[]; // { key, label, required, type, zod }
|
||||
/** Natural key used for dedup + as the FK-resolution lookup value. */
|
||||
matchKey: (row: MappedRow) => string | null;
|
||||
/** Resolve FK ids by natural key against the live DB. Returns ids or a
|
||||
* per-field resolution error. */
|
||||
resolveForeignKeys: (row: MappedRow, ctx: ImportCtx) => Promise<FkResult>;
|
||||
/** Dedup lookup — find an existing row by matchKey within the port. */
|
||||
findExisting: (portId: string, matchKey: string) => Promise<{ id: string } | null>;
|
||||
/** Writes delegate to the EXISTING service helpers so audit logging,
|
||||
* validation, and polymorphic-ownership rules come for free. */
|
||||
insert: (row: ResolvedRow, ctx: ImportCtx) => Promise<{ id: string }>;
|
||||
update: (existingId: string, row: ResolvedRow, ctx: ImportCtx) => Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
Adding an entity = adding one adapter + registering it. No engine change.
|
||||
|
||||
## Pipeline (BullMQ `import` queue, concurrency 1)
|
||||
|
||||
The queue + worker already exist (`src/lib/queue/workers/import.ts` is
|
||||
currently a documented no-op). We replace the no-op body with the real
|
||||
processor and add a producer.
|
||||
|
||||
1. **Upload & parse.** Drag-drop CSV/XLSX → parse (papaparse for CSV;
|
||||
**ExcelJS already installed** for XLSX) → raw rows. The uploaded file
|
||||
is stored via `getStorageBackend()` under a temp prefix so the worker
|
||||
can re-read it; cleaned up after commit or on expiry.
|
||||
2. **Map columns.** Auto-suggest mappings by fuzzy header match to the
|
||||
adapter's `targetFields`; user overrides; **save mapping as a per-port
|
||||
template** (`import_mappings`) for re-runs.
|
||||
3. **Dry-run (no writes).** Per row: apply mapping → zod-validate →
|
||||
`resolveForeignKeys` → `findExisting` → classify as
|
||||
`will-insert | will-update | will-skip | error(line, reason)`. Surface
|
||||
counts + a sample of rows + a downloadable line-numbered error report.
|
||||
4. **Commit.** Producer enqueues the job; the worker streams rows applying
|
||||
the chosen **conflict policy** (`skip-matches` / `update-matches` /
|
||||
`error-on-match`) via the adapter's `insert`/`update`. Per-row try/catch
|
||||
so valid rows still land; every action recorded in `import_batch_rows`;
|
||||
`import_batches` updated with live progress + final counts.
|
||||
5. **History + Undo.** Admin list of batches (status, counts, error-report
|
||||
download). **Undo** deletes the rows a batch _inserted_, in reverse
|
||||
dependency order, refusing if any inserted row now has dependents
|
||||
created outside the batch. Updates are marked non-revertible in v1.
|
||||
|
||||
## Data model (3 new tables; no changes to entity tables)
|
||||
|
||||
- **`import_batches`** — `id, port_id, entity_type, filename, storage_key,
|
||||
status (uploaded|dry_run|committing|completed|failed|undone),
|
||||
total_rows, inserted, updated, skipped, errored, mapping_json,
|
||||
conflict_policy, created_by, created_at, completed_at`.
|
||||
- **`import_batch_rows`** — `id, batch_id, row_number, action
|
||||
(inserted|updated|skipped|errored), entity_id (nullable), error
|
||||
(nullable)`. Powers the error report + undo. Migration-scale volume is
|
||||
fine.
|
||||
- **`import_mappings`** — `id, port_id, entity_type, name, mapping_json,
|
||||
created_by, created_at`. Saved column mappings, reusable across runs.
|
||||
|
||||
Migration added via the project's `psql`-applied numbered migration flow;
|
||||
restart `next dev` after (prepared-statement cache caveat per CLAUDE.md).
|
||||
|
||||
## Validation, errors, conflict policy
|
||||
|
||||
- **Per-row zod** from each adapter's `targetFields`; failures collected
|
||||
with row number + field + message, never aborting the whole file.
|
||||
- **Downloadable error report** (CSV: row, field, message) from any
|
||||
dry-run or completed batch.
|
||||
- **Conflict policy** chosen per import, surfaced at the dry-run step
|
||||
(three distinct behaviours for a matched row):
|
||||
- `skip-matches` — insert new, leave matched rows untouched. Default;
|
||||
safe to re-run.
|
||||
- `update-matches` — insert new, overwrite matched rows with the file's
|
||||
values (correct earlier mistakes).
|
||||
- `error-on-match` — treat a match as a row error to review, importing
|
||||
nothing for it (strictest).
|
||||
|
||||
## UI
|
||||
|
||||
A 4-step wizard mirroring the existing **bulk-add-berths wizard**:
|
||||
|
||||
1. Pick entity (registry-driven, shown in dependency order with a hint) +
|
||||
upload file.
|
||||
2. Map columns (auto-suggested; load a saved mapping; save current).
|
||||
3. Dry-run preview — counts (new / update / skip / error), sample table,
|
||||
error-report download, pick conflict policy.
|
||||
4. Commit — progress bar (worker reports % via batch counts) → result
|
||||
summary with link to History.
|
||||
|
||||
Plus an **Import History** tab: batch list + status + counts + error
|
||||
report + **Undo**. Replaces the static mockup at
|
||||
`src/app/(dashboard)/[portSlug]/admin/import/page.tsx`.
|
||||
|
||||
## Permissions & tenancy
|
||||
|
||||
Gate behind a new `data.import` permission (admin-tier). Every query +
|
||||
write is `port_id`-scoped; FK resolution only matches within the port.
|
||||
|
||||
## Testing (TDD)
|
||||
|
||||
- **Per-adapter unit tests** (one suite each): column mapping, zod
|
||||
validation (valid + each failure mode), `matchKey`, `resolveForeignKeys`
|
||||
(hit / miss / ambiguous), `findExisting` dedup.
|
||||
- **Dry-run classifier integration test** on a seeded DB: a fixture file
|
||||
yielding one of each class (insert / update / skip / error).
|
||||
- **Commit worker integration test**: each conflict policy; partial-failure
|
||||
(valid rows land, errored rows reported); idempotent re-run.
|
||||
- **Undo test**: deletes inserted rows; refuses when an inserted row has an
|
||||
outside dependent.
|
||||
|
||||
## Decisions locked (defaults the user approved 2026-06-01)
|
||||
|
||||
- Rollback depth: **inserts-only undo**; updates non-revertible in v1.
|
||||
- Partial failure: **valid rows commit**, errors reported (not
|
||||
all-or-nothing).
|
||||
- Berths: **included** in the UI importer despite the existing CLI.
|
||||
- All seven entities in scope.
|
||||
- Purpose: one-time cutover migration (engine reusable for ongoing ops).
|
||||
@@ -1,15 +1,6 @@
|
||||
import Link from 'next/link';
|
||||
import type { Route } from 'next';
|
||||
import {
|
||||
BookOpen,
|
||||
Calendar,
|
||||
Clock,
|
||||
DollarSign,
|
||||
Layers,
|
||||
Megaphone,
|
||||
Sparkles,
|
||||
TrendingUp,
|
||||
} from 'lucide-react';
|
||||
import { BookOpen, Calendar, Clock, Layers, Sparkles, TrendingUp } from 'lucide-react';
|
||||
|
||||
import { PageHeader } from '@/components/shared/page-header';
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@/components/ui/card';
|
||||
@@ -41,19 +32,6 @@ const KIND_CARDS: KindCard[] = [
|
||||
'Rep leaderboards, win rates, average time-to-close, stalled deals, conversion funnel by stage.',
|
||||
icon: TrendingUp,
|
||||
},
|
||||
{
|
||||
href: 'financial',
|
||||
label: 'Financial',
|
||||
description: 'Revenue by month, deposits collected, AR aging, EOI to revenue conversion.',
|
||||
icon: DollarSign,
|
||||
},
|
||||
{
|
||||
href: 'marketing',
|
||||
label: 'Marketing & funnel',
|
||||
description:
|
||||
'Lead source ROI, inquiry-to-EOI conversion, attribution by campaign, lead reports.',
|
||||
icon: Megaphone,
|
||||
},
|
||||
{
|
||||
href: 'operational',
|
||||
label: 'Operational',
|
||||
@@ -65,7 +43,7 @@ const KIND_CARDS: KindCard[] = [
|
||||
href: 'custom',
|
||||
label: 'Custom report',
|
||||
description:
|
||||
'Build your own: pick an entity, choose columns and filters, group by any dimension, save as a template.',
|
||||
'Build your own: pick an entity, choose columns, set a date range, export to CSV, and save as a template.',
|
||||
icon: Sparkles,
|
||||
},
|
||||
];
|
||||
@@ -158,7 +136,7 @@ export default async function ReportsLandingPage({ params }: PageProps) {
|
||||
Compose a report
|
||||
</h2>
|
||||
<p className="text-xs text-muted-foreground/80">
|
||||
Four canonical categories plus an ad-hoc composer for anything else.
|
||||
Sales and operational dashboards, plus an ad-hoc composer for anything else.
|
||||
</p>
|
||||
</div>
|
||||
<div className="grid grid-cols-1 gap-4 sm:grid-cols-2 lg:grid-cols-3">
|
||||
|
||||
@@ -15,7 +15,6 @@ import {
|
||||
FilePen,
|
||||
FileSignature,
|
||||
FileText,
|
||||
FileUp,
|
||||
GitBranch,
|
||||
Home,
|
||||
Inbox,
|
||||
@@ -258,12 +257,6 @@ const GROUPS: AdminGroup[] = [
|
||||
description: 'Review queue of suspected duplicate clients flagged by the dedup engine.',
|
||||
icon: CopyCheck,
|
||||
},
|
||||
{
|
||||
href: 'import',
|
||||
label: 'Bulk Import',
|
||||
description: 'CSV-driven imports for clients, yachts, and tenancies.',
|
||||
icon: FileUp,
|
||||
},
|
||||
{
|
||||
href: 'berths',
|
||||
label: 'Berths admin',
|
||||
|
||||
@@ -352,12 +352,6 @@ export const NAV_CATALOG: NavCatalogEntry[] = [
|
||||
category: 'admin',
|
||||
keywords: ['dedup', 'duplicate clients', 'merge', 'review queue'],
|
||||
},
|
||||
{
|
||||
href: '/:portSlug/admin/import',
|
||||
label: 'Bulk import',
|
||||
category: 'admin',
|
||||
keywords: ['csv', 'import', 'bulk upload'],
|
||||
},
|
||||
{
|
||||
href: '/:portSlug/admin/sends',
|
||||
label: 'Send log',
|
||||
|
||||
Reference in New Issue
Block a user