pn-new-crm/docs/superpowers/specs/2026-06-01-legacy-data-migration-design.md

# Legacy → New CRM Data Migration — Design Spec

> **Status:** DRAFT (2026-06-01) · scope locked · awaiting stage-map sign-off
> **Goal:** Translate all live legacy data + reconnect documents/EOIs so the
> new CRM "picks up exactly where we left off."
> **Companion:** `docs/launch-readiness.md` Initiative 5 · `docs/deployment-plan.md`
> **Source snapshot:** read-only `pg_dump` of prod NocoDB at
> `private/nocodb-snapshot/` (gitignored), restored locally as `nocodb_legacy`.

## 1. Source landscape (verified 2026-06-01)

Legacy data is spread across these systems (portal has **no DB of its own**):

| System                                                                                                                    | What                                                                                                                                                                                                 | Migrate?                                            |
| ------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| **NocoDB "Port Nimara"** base `plplouets5zw1um`                                                                           | Interests (255), Berths (117), Residences (45), multi-berth junction `_nc_m2m_Berths_Interests` (83), Website subs (Interest 64 / Contact 50 / BerthEOI 1), Newsletter (69), reminder/alert settings | ✅                                                  |
| **NocoDB "Expenses"** base `p3hq2fxdevqcaq8`                                                                              | Expenses (165); `invoices` empty                                                                                                                                                                     | ✅                                                  |
| **MinIO bucket `client-portal`**                                                                                          | EOIs, berth PDFs, receipts, business cards, general files                                                                                                                                            | ✅ (Phase 2)                                        |
| **MinIO bucket `signatures`**                                                                                             | Documenso signed PDFs                                                                                                                                                                                | ✅ (Phase 2)                                        |
| **Documenso v1.13.1**                                                                                                     | Signing envelopes, linked per-deal by `documensoID`                                                                                                                                                  | ✅ (Phase 2)                                        |
| 9 other NocoDB bases (Customer_List, Registered Interest, Form Submissions, 2nd Residential, Image Uploads, EOI Queue, …) | Old imports/experiments/backups                                                                                                                                                                      | ❌ **excluded** — zero code refs; stale 7–14 months |
| Gmail (IMAP), Keycloak                                                                                                    | Email archive, portal auth                                                                                                                                                                           | ❌ out of scope (per Matt)                          |

**Authority for scope:** the live portal + website code reference table IDs in
**only** the two active bases above; the recency check confirms `Interests` is
the only actively-written table (last write 2026-05-21).

**Legacy has no Company entity** (everything is attributed to a person), so the
migration creates **clients + yachts (client-owned) + deals** — no companies.

## 2. Key linking facts

- **Client + yacht are inline on each Interests row** → extract + dedup.
- **`documensoID`** (e.g. `"82"`) on each deal → resolves to Documenso
  `Envelope.secondaryId = 'document_' || documensoID` (verified: deal
  `doc=114` → envelope `document_114`). The envelope's completed PDF = the
  signed EOI. (Prod Documenso = v1.13.1, 140 migrations — confirmed.)
- **`Berth Number`** (mooring, e.g. `D31`) + the `_nc_m2m_Berths_Interests`
  junction → multi-berth links.
- **Notes** = inline `Internal Notes` + `Extra Comments` (+ 5 rows in
  `nc_comments`).
- Dedup key for people: **lowercased email → fallback canonical phone**.

## 3. Phase 1 — NocoDB → new CRM (data)

Build against the local `nocodb_legacy` snapshot; idempotent; every new row
stamped with its `legacy_nocodb_id` (add a nullable column or a side mapping
table `migration_id_map(entity, legacy_id, new_id)`).

**Import order (FK-safe):** clients → yachts → interests → interest_berths →
notes → residential → expenses → website_submissions → settings.

### 3.1 Clients (from Interests, deduped)

Source fields → `clients`: `Full Name`→fullName (title-cased via the legacy
`normalizePersonName` rule), `Email Address`→primary email, `Phone Number`→
canonical phone, `Address`+`Place of Residence`→address/locality,
`Contact Method Preferred`→preferredContactMethod, `Source`→source,
`Lead Category`→(deal-level, see below). **Dedup:** group all 255 interests by
lowercased email (fallback canonical phone); one client per unique person,
N deals.

### 3.2 Yachts (from Interests)

`Yacht Name`→name (skip `TBC`/blank), `Length`/`Width`/`Depth`→dims. **Unit
note:** legacy stores strings like `"50ft"` — parse number + unit, convert ft→m
to match the berth/yacht numeric schema (store original string in a note if
ambiguous). Owner = the deduped client (polymorphic `client`).

### 3.3 Interests / deals

- **Stage:** map `Sales Process Level` (8) → new 7-stage pipeline — **see §4
  (needs sign-off).**
- `Lead Category` (General / Friends and Family)→leadCategory, `Source`→source.
- Statuses: `EOI Status`, `Deposit 10% Status`, `Contract Status`,
  `Contract Sent Status`, `Berth Info Sent Status` → drive stage + the new
  EOI/contract/deposit fields; `Deposit 10% Status='Received'` → a `payments`
  row (deposit) + auto-advance.
- Dates: `Date Added`/`Created At`→createdAt (DD-MM-YYYY → ISO; many are null —
  fall back to Documenso/earliest signal), `EOI Time Sent`, `Time LOI Sent`.
- `documensoID` → stored for Phase 2 EOI relink.
- **Outcome:** `Sales Process Level='Contract Signed'` + deposit/contract
  complete → won; otherwise open. (No explicit "lost" in legacy.)

### 3.4 interest_berths (multi-berth)

From `_nc_m2m_Berths_Interests` (83 links) → `interest_berths` via
`interestBerthsService`. `is_primary` = the `Berth Number` plain-text mooring
(or first link); `is_in_eoi_bundle` = true for signed/sent EOIs. Resolve berth
by mooring against the migrated 117 berths.

### 3.5 Notes

`Internal Notes` + `Extra Comments` (and `nc_comments`) → `interestNotes` via
`notes.service`, preserving original timestamps where present.

### 3.6 Residential

`Interests (Residences)` (45) → `residential_clients` + `residential_interests`
(dedup by email). The 2nd residential base (16 rows) is **excluded** (stale).

### 3.7 Expenses

`Expenses` base (165) → the expenses module. Map Time→date, Payer→payer,
Category→category, Price (string `"€1,234"`)→numeric+currency. Receipts linked
in Phase 2 (the `Receipts` images live in MinIO).

### 3.8 Website submissions + settings

Website Interest/Contact/BerthEOI subs → `website_submissions`. `reminder_settings`
/`alert_settings` → best-effort into `system_settings`.

## 4. Stage mapping (8 → 7) — NEEDS SIGN-OFF

Legacy `Sales Process Level` → new pipeline stage (proposed):

| Legacy                          | New stage                   |
| ------------------------------- | --------------------------- |
| General Qualified Interest      | `qualified`                 |
| Specific Qualified Interest     | `nurturing`                 |
| EOI and NDA Sent                | `eoi`                       |
| Signed EOI and NDA              | `eoi` (EOI signed)          |
| Made Reservation                | `reservation`               |
| Contract Negotiation            | `reservation` → `contract`? |
| Contract Negotiations Finalized | `contract`                  |
| Contract Signed                 | `contract` (won)            |

Open questions for Matt: (a) is "General Qualified Interest" really `qualified`
or should some map to `enquiry`? (b) does "Contract Negotiation" belong in
`reservation` or `contract`? (c) treat `Contract Signed` as a closed-won
outcome?

## 5. Phase 2 — documents & EOIs (MinIO inventoried 2026-06-01)

Documents live in **three** MinIO buckets (verified):

- **`client-portal`** (248 objects, 240 MB) — cleanly foldered: `Berth-PDFs/`
  (114, mooring in filename), `EOIs/` (95 signed EOIs foldered by client name),
  `Client Documents/` (6), `Legal/` (14), `expense-sheets/` (2),
  `client-emails/` (3 sent-email JSONs keyed `interest-<id>`).
- **`signatures`** (323) — Documenso's raw per-envelope store (many test dupes —
  secondary source).
- **`database`** — NocoDB's own attachment store at
  `database/nc/uploads/noco/plplouets5zw1um/mbs9hjauug4eseo/cjzx7y2h9sxwd0n/…`
  (field `cjzx7y2h9sxwd0n` = `EOI_Document`). **This is where the pre-Documenso
  ("before/aside") signed EOIs live**, as NocoDB attachments.

**EOI coverage — verified, no missing signed EOI.** Of 255 interests, 48 are
EOI-signed; every one resolves to a recoverable PDF:

1. **~38 via `documensoID`** → `Envelope.secondaryId='document_'||id` →
   completed PDF (+ curated copy in `client-portal/EOIs/<name>/`).
2. **~10 old LOI-process deals** (no documensoID, `LOI=Signing Complete`) →
   `EOI_Document` attachment in the **`database`** bucket.
3. **3 via explicit `S3_Documenso_Path`** → `client-portal/EOIs/`.

Backfill order per deal: prefer the curated `client-portal/EOIs/` copy → fall
back to Documenso (by secondaryId) → then the NocoDB `database` attachment. Each
→ store via `getStorageBackend()` → `files`+`documents` rows → `ensureEntityFolder`.
Still run a file↔deal reconciliation to flag orphan EOI files + confirm each
envelope PDF actually downloads.

4. **Berth PDFs:** `client-portal/Berth-PDFs/` (114) → `berth_pdf_versions`
   (mooring parsed from filename).
5. **Receipts / business cards:** NOT in `client-portal` — likely in `forms`/
   `images`/`directus` buckets (OpnForm uploads). Hunt only if wanted.
6. Unresolved → manual-review CSV.

### ⚠ Crossover gate — in-flight Documenso signings

Documenso currently holds **6 PENDING** (sent, awaiting signature) + **6 DRAFT**
envelopes (of 58 total; 46 COMPLETED). PENDING: Thomas Nemic (2026-02-04), Davy
Morée (2025-11-28), Matthew Ciaccio (2025-11-24), Ben Sturge (2025-10-11), Van
der Merwe (2025-10-02), Charles Davis (2025-08-22) — most stale/likely abandoned,
only one from 2026. **Before the Documenso upgrade/crossover, review these:** void
the dead ones, let any genuine one finish — don't strand an active signature.

## 6. Verification & reconcile

**Validated run (2026-06-01, `extract-nocodb.ts`):** 255 interests → **232
unique clients** (1.10×; 21 with >1 deal roll up correctly), 39 yachts, 84
deal↔berth links (12 multi-berth), 63 notes. Stages 8→7: qualified 171 · eoi 51
· nurturing 30 · reservation 2 · contract 1. **EOI coverage 48/48 resolvable.**
Signing state (Documenso-authoritative): signed 48 · **awaiting_signature 3**
(interests 581/633/639 → migrate as "awaiting" + keep envelope link + display
pending) · none 204. Duplicate review: 1 exact-name (Etiennette Clamouze ×2), 0
fuzzy. Residential 45→35. Expenses 165 (0 parse fails). Output →
`private/migration-output/` (gitignored).

**In-flight signing display:** the 3 `awaiting_signature` deals load with the
interest's EOI state = sent/awaiting + the Documenso envelope linked, so the new
CRM's webhook/poll completes them and the UI shows "Waiting for signatures."
Reconcile the 6 Documenso PENDING: 3 link to deals (in-flight above); 3 are
abandoned re-sends of already-signed deals → void-review before crossover.

Remaining: spot-check 5 deals end-to-end after load.

## 7. Deliverables (scripts/migration/)

- `probe-minio.ts` — bucket inventory (Phase 2 sizing; answers "are the
  business cards there?").
- `extract-nocodb.ts` — read the snapshot, emit normalized JSON per entity.
- `transform-load.ts` — dedup + map + load via service helpers, idempotent.
- `backfill-documents.ts` — Phase 2 EOI/PDF/receipt backfill.
- `reconcile.ts` — final report.

## 8. Decisions locked (2026-06-01)

- Scope = the 2 active bases only; 9 others excluded; email/Keycloak out.
- Extract via read-only pg_dump snapshot (done).
- No company entities (legacy has none).
- Idempotent, keyed on `legacy_nocodb_id`.