c4a41d5f5b8e3ec4d9f2c68ed18db55a4fa7a40c
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Matt Ciaccio
|
687a1f1c2f |
fix(audit-v3): platform-wide deferred-list cleanup (rounds 1-4)
Working through the audit-v2 deferred backlog. Each round was tested
(typecheck + 1168/1168 vitest) before moving on.
Round 1 — DB performance + AI cost visibility:
- Add missing FK indexes Postgres doesn't auto-create on
berth_reservations.{interest_id, contract_file_id},
documents.{file_id, signed_file_id}, document_events.signer_id,
document_templates.source_file_id, form_submissions.{form_template_id,
client_id}, document_sends.{brochure_id, brochure_version_id,
sent_by_user_id}. Without these, RESTRICT-checks on parent delete +
reverse-lookups walk the child tables fully. Migration 0037.
- AI worker now writes one ai_usage_ledger row per OpenAI call so admins
can audit spend per port/user/feature and future per-port budgets have
history to read from. Failure to write is logged-not-thrown so the
user-facing email draft is unaffected.
Round 2 — Boot-time + transport hardening:
- S3 backend verifies the bucket exists at startup (or auto-creates
when MINIO_AUTO_CREATE_BUCKET=true). A typo'd bucket name now
surfaces with a clear boot error instead of a vague Minio error
inside the first user-facing request.
- Documenso v1 placeFields: 3-attempt exponential-backoff retry on 5xx
+ network errors, fail-fast on 4xx. Stops one transient flake from
leaving a document with a partial field set.
- FilesystemBackend logs a structured warn-once at boot when the dev
HMAC fallback is in effect, so two processes started with different
BETTER_AUTH_SECRET values are observable (random 401s on file
downloads otherwise).
- Logger redact paths extended to cover *.headers.{authorization,
cookie}, *.config.headers.authorization, encrypted-credential blobs
(secretKeyEncrypted, smtpPassEncrypted, etc.), the Documenso
X-Documenso-Secret header, and 2-level nested forms.
Round 3 — UI feedback + permission gates:
- Storage admin migrate dialog: success toast with row count + error
toast on both dryRun and migrate mutations.
- Invoice detail Send + Record-payment buttons wrapped in
PermissionGate (invoices.send / invoices.record_payment); both
mutations now toast on success/error.
- Admin user list Edit button wrapped in PermissionGate(admin.manage_users).
- Scan-receipt page surfaces an amber warning when OCR fails so reps
know they can fill the form manually instead of staring at a stalled
spinner; the editable form now also opens on scanMutation.isError
/ uploadedFile, not only on success.
- Email threads list now renders skeleton rows during load + shared
EmptyState for the empty case (was a single "Loading…" line).
Round 4 — Service / route correctness:
- documentSends.sent_by_user_id was a free-text NOT NULL column with no
FK. Now nullable + FK to user(id) ON DELETE SET NULL so the audit row
survives a user being hard-deleted. Migration 0038 with a defensive
null-out for any orphan ids before attaching the constraint.
- Saved-views route: documented why withAuth alone is correct (the
service strictly filters by (portId, userId) — owner-only by design).
- Public-interests audit log: replaced "userId: null as unknown as
string" cast with userId: null; AuditLogParams already accepts null
for system-generated events.
- EOI in-app PDF fill: extracted setBerthRange() that, when the
AcroForm field is missing AND the context has a non-empty range
string, logs a structured warn so the deployment gap (live Documenso
template needs the field) is observable instead of silently dropping
the multi-berth range.
Test status: 1168/1168 vitest. tsc clean. Two new migrations
(0037/0038) need pnpm db:push (or migration apply) on the dev DB.
Deferred-doc updated with the remaining open items (bigger refactors).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Matt Ciaccio
|
83693dd993 |
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI
Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays
the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs,
brochures) without touching those domains yet.
New files:
- src/lib/storage/index.ts StorageBackend interface + per-process
factory keyed on system_settings.
- src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/
Wasabi/Tigris) wrapping the existing minio
JS client. Includes a healthCheck() used
by the admin "Test connection" button.
- src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a
mitigations baked in.
- src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock,
per-row resumable progress markers,
sha256 round-trip verification, atomic
storage_backend flip on success.
- scripts/migrate-storage.ts Thin CLI shim around runMigration().
- src/app/api/storage/[token]/route.ts
Filesystem proxy GET. Verifies HMAC,
enforces single-use replay protection
via Redis SET NX, streams via NextResponse
ReadableStream with explicit Content-Type
+ Content-Disposition. Node runtime only.
- src/app/api/v1/admin/storage/route.ts
GET status + POST connection test.
- src/app/api/v1/admin/storage/migrate/route.ts
Super-admin-only POST that runs the
exact same runMigration() as the CLI.
- src/app/(dashboard)/[portSlug]/admin/storage/page.tsx
Super-admin admin UI (current backend,
capacity stats, switch button with
dry-run, test connection, backup hint).
- src/components/admin/storage-admin-panel.tsx
Client component for the page above.
§14.9a critical mitigations implemented:
- Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$;
`..`, `.`, `//`, leading `/`, and overlength keys rejected.
- Realpath: storage root realpath'd at create time, every per-key
resolution checked against the realpath'd prefix.
- Storage root created (or chmod'd) to 0o700.
- Multi-node refusal: FilesystemBackend.create() throws when
MULTI_NODE_DEPLOYMENT=true.
- HMAC token: sha256-HMAC over the (key, expiry, nonce, filename,
content-type) payload. Verified with timingSafeEqual; bad sig,
expired, or invalid-key payloads all return 403.
- Single-use replay: token body cached in Redis SET NX EX 1800s.
- sha256 round-trip: copyAndVerify() re-fetches from the target after
put() and aborts the migration on any mismatch.
- Free-disk pre-flight: when migrating to filesystem, sums byte counts
via source.head() and aborts if free space < total * 1.2.
- pg_advisory_lock(0xc7000a01) prevents concurrent migrations.
- Resumable: per-row progress markers in _storage_migration_progress.
system_settings keys read by the factory (jsonb, no schema change):
storage_backend, storage_s3_endpoint, storage_s3_region,
storage_s3_bucket, storage_s3_access_key,
storage_s3_secret_key_encrypted, storage_s3_force_path_style,
storage_filesystem_root, storage_proxy_hmac_secret_encrypted.
Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage`
(./storage added to .gitignore).
Tests added (34 tests, all green):
- tests/unit/storage/filesystem-backend.test.ts — key validation
allow/reject matrix, realpath escape, 0o700 perms, multi-node
refusal, HMAC token sign/verify/tamper/expire/invalid-key.
- tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on
round-trip aborts the migration.
- tests/integration/storage/proxy-route.test.ts — happy path, wrong
HMAC secret, expired token, replay rejection.
Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is
intentionally empty. berth_pdf_versions and brochure_versions land in
Phase 6b and join the list there. Existing s3_key columns: only
gdpr_export_jobs.storage_key, already named correctly — no rename needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|