Files
pn-new-crm/src/components/admin/storage-admin-panel.tsx

246 lines
9.2 KiB
TypeScript
Raw Normal View History

feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs, brochures) without touching those domains yet. New files: - src/lib/storage/index.ts StorageBackend interface + per-process factory keyed on system_settings. - src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/ Wasabi/Tigris) wrapping the existing minio JS client. Includes a healthCheck() used by the admin "Test connection" button. - src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a mitigations baked in. - src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock, per-row resumable progress markers, sha256 round-trip verification, atomic storage_backend flip on success. - scripts/migrate-storage.ts Thin CLI shim around runMigration(). - src/app/api/storage/[token]/route.ts Filesystem proxy GET. Verifies HMAC, enforces single-use replay protection via Redis SET NX, streams via NextResponse ReadableStream with explicit Content-Type + Content-Disposition. Node runtime only. - src/app/api/v1/admin/storage/route.ts GET status + POST connection test. - src/app/api/v1/admin/storage/migrate/route.ts Super-admin-only POST that runs the exact same runMigration() as the CLI. - src/app/(dashboard)/[portSlug]/admin/storage/page.tsx Super-admin admin UI (current backend, capacity stats, switch button with dry-run, test connection, backup hint). - src/components/admin/storage-admin-panel.tsx Client component for the page above. §14.9a critical mitigations implemented: - Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$; `..`, `.`, `//`, leading `/`, and overlength keys rejected. - Realpath: storage root realpath'd at create time, every per-key resolution checked against the realpath'd prefix. - Storage root created (or chmod'd) to 0o700. - Multi-node refusal: FilesystemBackend.create() throws when MULTI_NODE_DEPLOYMENT=true. - HMAC token: sha256-HMAC over the (key, expiry, nonce, filename, content-type) payload. Verified with timingSafeEqual; bad sig, expired, or invalid-key payloads all return 403. - Single-use replay: token body cached in Redis SET NX EX 1800s. - sha256 round-trip: copyAndVerify() re-fetches from the target after put() and aborts the migration on any mismatch. - Free-disk pre-flight: when migrating to filesystem, sums byte counts via source.head() and aborts if free space < total * 1.2. - pg_advisory_lock(0xc7000a01) prevents concurrent migrations. - Resumable: per-row progress markers in _storage_migration_progress. system_settings keys read by the factory (jsonb, no schema change): storage_backend, storage_s3_endpoint, storage_s3_region, storage_s3_bucket, storage_s3_access_key, storage_s3_secret_key_encrypted, storage_s3_force_path_style, storage_filesystem_root, storage_proxy_hmac_secret_encrypted. Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage` (./storage added to .gitignore). Tests added (34 tests, all green): - tests/unit/storage/filesystem-backend.test.ts — key validation allow/reject matrix, realpath escape, 0o700 perms, multi-node refusal, HMAC token sign/verify/tamper/expire/invalid-key. - tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on round-trip aborts the migration. - tests/integration/storage/proxy-route.test.ts — happy path, wrong HMAC secret, expired token, replay rejection. Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is intentionally empty. berth_pdf_versions and brochure_versions land in Phase 6b and join the list there. Existing s3_key columns: only gdpr_export_jobs.storage_key, already named correctly — no rename needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
'use client';
import { useState } from 'react';
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
import { CheckCircle2, HardDrive, Loader2, RefreshCw, ServerCog, XCircle } from 'lucide-react';
fix(audit-v3): platform-wide deferred-list cleanup (rounds 1-4) Working through the audit-v2 deferred backlog. Each round was tested (typecheck + 1168/1168 vitest) before moving on. Round 1 — DB performance + AI cost visibility: - Add missing FK indexes Postgres doesn't auto-create on berth_reservations.{interest_id, contract_file_id}, documents.{file_id, signed_file_id}, document_events.signer_id, document_templates.source_file_id, form_submissions.{form_template_id, client_id}, document_sends.{brochure_id, brochure_version_id, sent_by_user_id}. Without these, RESTRICT-checks on parent delete + reverse-lookups walk the child tables fully. Migration 0037. - AI worker now writes one ai_usage_ledger row per OpenAI call so admins can audit spend per port/user/feature and future per-port budgets have history to read from. Failure to write is logged-not-thrown so the user-facing email draft is unaffected. Round 2 — Boot-time + transport hardening: - S3 backend verifies the bucket exists at startup (or auto-creates when MINIO_AUTO_CREATE_BUCKET=true). A typo'd bucket name now surfaces with a clear boot error instead of a vague Minio error inside the first user-facing request. - Documenso v1 placeFields: 3-attempt exponential-backoff retry on 5xx + network errors, fail-fast on 4xx. Stops one transient flake from leaving a document with a partial field set. - FilesystemBackend logs a structured warn-once at boot when the dev HMAC fallback is in effect, so two processes started with different BETTER_AUTH_SECRET values are observable (random 401s on file downloads otherwise). - Logger redact paths extended to cover *.headers.{authorization, cookie}, *.config.headers.authorization, encrypted-credential blobs (secretKeyEncrypted, smtpPassEncrypted, etc.), the Documenso X-Documenso-Secret header, and 2-level nested forms. Round 3 — UI feedback + permission gates: - Storage admin migrate dialog: success toast with row count + error toast on both dryRun and migrate mutations. - Invoice detail Send + Record-payment buttons wrapped in PermissionGate (invoices.send / invoices.record_payment); both mutations now toast on success/error. - Admin user list Edit button wrapped in PermissionGate(admin.manage_users). - Scan-receipt page surfaces an amber warning when OCR fails so reps know they can fill the form manually instead of staring at a stalled spinner; the editable form now also opens on scanMutation.isError / uploadedFile, not only on success. - Email threads list now renders skeleton rows during load + shared EmptyState for the empty case (was a single "Loading…" line). Round 4 — Service / route correctness: - documentSends.sent_by_user_id was a free-text NOT NULL column with no FK. Now nullable + FK to user(id) ON DELETE SET NULL so the audit row survives a user being hard-deleted. Migration 0038 with a defensive null-out for any orphan ids before attaching the constraint. - Saved-views route: documented why withAuth alone is correct (the service strictly filters by (portId, userId) — owner-only by design). - Public-interests audit log: replaced "userId: null as unknown as string" cast with userId: null; AuditLogParams already accepts null for system-generated events. - EOI in-app PDF fill: extracted setBerthRange() that, when the AcroForm field is missing AND the context has a non-empty range string, logs a structured warn so the deployment gap (live Documenso template needs the field) is observable instead of silently dropping the multi-berth range. Test status: 1168/1168 vitest. tsc clean. Two new migrations (0037/0038) need pnpm db:push (or migration apply) on the dev DB. Deferred-doc updated with the remaining open items (bigger refactors). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 12:49:53 +02:00
import { toast } from 'sonner';
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs, brochures) without touching those domains yet. New files: - src/lib/storage/index.ts StorageBackend interface + per-process factory keyed on system_settings. - src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/ Wasabi/Tigris) wrapping the existing minio JS client. Includes a healthCheck() used by the admin "Test connection" button. - src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a mitigations baked in. - src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock, per-row resumable progress markers, sha256 round-trip verification, atomic storage_backend flip on success. - scripts/migrate-storage.ts Thin CLI shim around runMigration(). - src/app/api/storage/[token]/route.ts Filesystem proxy GET. Verifies HMAC, enforces single-use replay protection via Redis SET NX, streams via NextResponse ReadableStream with explicit Content-Type + Content-Disposition. Node runtime only. - src/app/api/v1/admin/storage/route.ts GET status + POST connection test. - src/app/api/v1/admin/storage/migrate/route.ts Super-admin-only POST that runs the exact same runMigration() as the CLI. - src/app/(dashboard)/[portSlug]/admin/storage/page.tsx Super-admin admin UI (current backend, capacity stats, switch button with dry-run, test connection, backup hint). - src/components/admin/storage-admin-panel.tsx Client component for the page above. §14.9a critical mitigations implemented: - Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$; `..`, `.`, `//`, leading `/`, and overlength keys rejected. - Realpath: storage root realpath'd at create time, every per-key resolution checked against the realpath'd prefix. - Storage root created (or chmod'd) to 0o700. - Multi-node refusal: FilesystemBackend.create() throws when MULTI_NODE_DEPLOYMENT=true. - HMAC token: sha256-HMAC over the (key, expiry, nonce, filename, content-type) payload. Verified with timingSafeEqual; bad sig, expired, or invalid-key payloads all return 403. - Single-use replay: token body cached in Redis SET NX EX 1800s. - sha256 round-trip: copyAndVerify() re-fetches from the target after put() and aborts the migration on any mismatch. - Free-disk pre-flight: when migrating to filesystem, sums byte counts via source.head() and aborts if free space < total * 1.2. - pg_advisory_lock(0xc7000a01) prevents concurrent migrations. - Resumable: per-row progress markers in _storage_migration_progress. system_settings keys read by the factory (jsonb, no schema change): storage_backend, storage_s3_endpoint, storage_s3_region, storage_s3_bucket, storage_s3_access_key, storage_s3_secret_key_encrypted, storage_s3_force_path_style, storage_filesystem_root, storage_proxy_hmac_secret_encrypted. Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage` (./storage added to .gitignore). Tests added (34 tests, all green): - tests/unit/storage/filesystem-backend.test.ts — key validation allow/reject matrix, realpath escape, 0o700 perms, multi-node refusal, HMAC token sign/verify/tamper/expire/invalid-key. - tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on round-trip aborts the migration. - tests/integration/storage/proxy-route.test.ts — happy path, wrong HMAC secret, expired token, replay rejection. Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is intentionally empty. berth_pdf_versions and brochure_versions land in Phase 6b and join the list there. Existing s3_key columns: only gdpr_export_jobs.storage_key, already named correctly — no rename needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
import { PageHeader } from '@/components/shared/page-header';
import { Button } from '@/components/ui/button';
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@/components/ui/card';
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog';
import { apiFetch } from '@/lib/api/client';
type BackendName = 's3' | 'filesystem';
interface StorageStatus {
backend: BackendName;
fileCount: number;
totalBytes: number;
tablesTracked: string[];
}
interface MigrationResult {
rowsConsidered: number;
rowsMigrated: number;
rowsSkippedAlreadyDone: number;
totalBytes: number;
flipped: boolean;
dryRun: boolean;
}
export function StorageAdminPanel() {
const queryClient = useQueryClient();
const [confirmOpen, setConfirmOpen] = useState(false);
const [dryRun, setDryRun] = useState<MigrationResult | null>(null);
const [testResult, setTestResult] = useState<{ ok: boolean; error?: string } | null>(null);
const status = useQuery({
queryKey: ['admin', 'storage', 'status'],
queryFn: () => apiFetch<{ data: StorageStatus }>('/api/v1/admin/storage'),
});
const dryRunMutation = useMutation({
mutationFn: async (opts: { from: BackendName; to: BackendName }) =>
apiFetch<{ data: MigrationResult }>('/api/v1/admin/storage/migrate', {
method: 'POST',
body: JSON.stringify({ ...opts, dryRun: true }),
}),
onSuccess: (result) => {
setDryRun(result.data);
setConfirmOpen(true);
},
fix(audit-v3): platform-wide deferred-list cleanup (rounds 1-4) Working through the audit-v2 deferred backlog. Each round was tested (typecheck + 1168/1168 vitest) before moving on. Round 1 — DB performance + AI cost visibility: - Add missing FK indexes Postgres doesn't auto-create on berth_reservations.{interest_id, contract_file_id}, documents.{file_id, signed_file_id}, document_events.signer_id, document_templates.source_file_id, form_submissions.{form_template_id, client_id}, document_sends.{brochure_id, brochure_version_id, sent_by_user_id}. Without these, RESTRICT-checks on parent delete + reverse-lookups walk the child tables fully. Migration 0037. - AI worker now writes one ai_usage_ledger row per OpenAI call so admins can audit spend per port/user/feature and future per-port budgets have history to read from. Failure to write is logged-not-thrown so the user-facing email draft is unaffected. Round 2 — Boot-time + transport hardening: - S3 backend verifies the bucket exists at startup (or auto-creates when MINIO_AUTO_CREATE_BUCKET=true). A typo'd bucket name now surfaces with a clear boot error instead of a vague Minio error inside the first user-facing request. - Documenso v1 placeFields: 3-attempt exponential-backoff retry on 5xx + network errors, fail-fast on 4xx. Stops one transient flake from leaving a document with a partial field set. - FilesystemBackend logs a structured warn-once at boot when the dev HMAC fallback is in effect, so two processes started with different BETTER_AUTH_SECRET values are observable (random 401s on file downloads otherwise). - Logger redact paths extended to cover *.headers.{authorization, cookie}, *.config.headers.authorization, encrypted-credential blobs (secretKeyEncrypted, smtpPassEncrypted, etc.), the Documenso X-Documenso-Secret header, and 2-level nested forms. Round 3 — UI feedback + permission gates: - Storage admin migrate dialog: success toast with row count + error toast on both dryRun and migrate mutations. - Invoice detail Send + Record-payment buttons wrapped in PermissionGate (invoices.send / invoices.record_payment); both mutations now toast on success/error. - Admin user list Edit button wrapped in PermissionGate(admin.manage_users). - Scan-receipt page surfaces an amber warning when OCR fails so reps know they can fill the form manually instead of staring at a stalled spinner; the editable form now also opens on scanMutation.isError / uploadedFile, not only on success. - Email threads list now renders skeleton rows during load + shared EmptyState for the empty case (was a single "Loading…" line). Round 4 — Service / route correctness: - documentSends.sent_by_user_id was a free-text NOT NULL column with no FK. Now nullable + FK to user(id) ON DELETE SET NULL so the audit row survives a user being hard-deleted. Migration 0038 with a defensive null-out for any orphan ids before attaching the constraint. - Saved-views route: documented why withAuth alone is correct (the service strictly filters by (portId, userId) — owner-only by design). - Public-interests audit log: replaced "userId: null as unknown as string" cast with userId: null; AuditLogParams already accepts null for system-generated events. - EOI in-app PDF fill: extracted setBerthRange() that, when the AcroForm field is missing AND the context has a non-empty range string, logs a structured warn so the deployment gap (live Documenso template needs the field) is observable instead of silently dropping the multi-berth range. Test status: 1168/1168 vitest. tsc clean. Two new migrations (0037/0038) need pnpm db:push (or migration apply) on the dev DB. Deferred-doc updated with the remaining open items (bigger refactors). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 12:49:53 +02:00
onError: (e) =>
toast.error(e instanceof Error ? e.message : 'Storage migration dry-run failed'),
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs, brochures) without touching those domains yet. New files: - src/lib/storage/index.ts StorageBackend interface + per-process factory keyed on system_settings. - src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/ Wasabi/Tigris) wrapping the existing minio JS client. Includes a healthCheck() used by the admin "Test connection" button. - src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a mitigations baked in. - src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock, per-row resumable progress markers, sha256 round-trip verification, atomic storage_backend flip on success. - scripts/migrate-storage.ts Thin CLI shim around runMigration(). - src/app/api/storage/[token]/route.ts Filesystem proxy GET. Verifies HMAC, enforces single-use replay protection via Redis SET NX, streams via NextResponse ReadableStream with explicit Content-Type + Content-Disposition. Node runtime only. - src/app/api/v1/admin/storage/route.ts GET status + POST connection test. - src/app/api/v1/admin/storage/migrate/route.ts Super-admin-only POST that runs the exact same runMigration() as the CLI. - src/app/(dashboard)/[portSlug]/admin/storage/page.tsx Super-admin admin UI (current backend, capacity stats, switch button with dry-run, test connection, backup hint). - src/components/admin/storage-admin-panel.tsx Client component for the page above. §14.9a critical mitigations implemented: - Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$; `..`, `.`, `//`, leading `/`, and overlength keys rejected. - Realpath: storage root realpath'd at create time, every per-key resolution checked against the realpath'd prefix. - Storage root created (or chmod'd) to 0o700. - Multi-node refusal: FilesystemBackend.create() throws when MULTI_NODE_DEPLOYMENT=true. - HMAC token: sha256-HMAC over the (key, expiry, nonce, filename, content-type) payload. Verified with timingSafeEqual; bad sig, expired, or invalid-key payloads all return 403. - Single-use replay: token body cached in Redis SET NX EX 1800s. - sha256 round-trip: copyAndVerify() re-fetches from the target after put() and aborts the migration on any mismatch. - Free-disk pre-flight: when migrating to filesystem, sums byte counts via source.head() and aborts if free space < total * 1.2. - pg_advisory_lock(0xc7000a01) prevents concurrent migrations. - Resumable: per-row progress markers in _storage_migration_progress. system_settings keys read by the factory (jsonb, no schema change): storage_backend, storage_s3_endpoint, storage_s3_region, storage_s3_bucket, storage_s3_access_key, storage_s3_secret_key_encrypted, storage_s3_force_path_style, storage_filesystem_root, storage_proxy_hmac_secret_encrypted. Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage` (./storage added to .gitignore). Tests added (34 tests, all green): - tests/unit/storage/filesystem-backend.test.ts — key validation allow/reject matrix, realpath escape, 0o700 perms, multi-node refusal, HMAC token sign/verify/tamper/expire/invalid-key. - tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on round-trip aborts the migration. - tests/integration/storage/proxy-route.test.ts — happy path, wrong HMAC secret, expired token, replay rejection. Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is intentionally empty. berth_pdf_versions and brochure_versions land in Phase 6b and join the list there. Existing s3_key columns: only gdpr_export_jobs.storage_key, already named correctly — no rename needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
});
const migrateMutation = useMutation({
mutationFn: async (opts: { from: BackendName; to: BackendName }) =>
apiFetch<{ data: MigrationResult }>('/api/v1/admin/storage/migrate', {
method: 'POST',
body: JSON.stringify({ ...opts, dryRun: false }),
}),
fix(audit-v3): platform-wide deferred-list cleanup (rounds 1-4) Working through the audit-v2 deferred backlog. Each round was tested (typecheck + 1168/1168 vitest) before moving on. Round 1 — DB performance + AI cost visibility: - Add missing FK indexes Postgres doesn't auto-create on berth_reservations.{interest_id, contract_file_id}, documents.{file_id, signed_file_id}, document_events.signer_id, document_templates.source_file_id, form_submissions.{form_template_id, client_id}, document_sends.{brochure_id, brochure_version_id, sent_by_user_id}. Without these, RESTRICT-checks on parent delete + reverse-lookups walk the child tables fully. Migration 0037. - AI worker now writes one ai_usage_ledger row per OpenAI call so admins can audit spend per port/user/feature and future per-port budgets have history to read from. Failure to write is logged-not-thrown so the user-facing email draft is unaffected. Round 2 — Boot-time + transport hardening: - S3 backend verifies the bucket exists at startup (or auto-creates when MINIO_AUTO_CREATE_BUCKET=true). A typo'd bucket name now surfaces with a clear boot error instead of a vague Minio error inside the first user-facing request. - Documenso v1 placeFields: 3-attempt exponential-backoff retry on 5xx + network errors, fail-fast on 4xx. Stops one transient flake from leaving a document with a partial field set. - FilesystemBackend logs a structured warn-once at boot when the dev HMAC fallback is in effect, so two processes started with different BETTER_AUTH_SECRET values are observable (random 401s on file downloads otherwise). - Logger redact paths extended to cover *.headers.{authorization, cookie}, *.config.headers.authorization, encrypted-credential blobs (secretKeyEncrypted, smtpPassEncrypted, etc.), the Documenso X-Documenso-Secret header, and 2-level nested forms. Round 3 — UI feedback + permission gates: - Storage admin migrate dialog: success toast with row count + error toast on both dryRun and migrate mutations. - Invoice detail Send + Record-payment buttons wrapped in PermissionGate (invoices.send / invoices.record_payment); both mutations now toast on success/error. - Admin user list Edit button wrapped in PermissionGate(admin.manage_users). - Scan-receipt page surfaces an amber warning when OCR fails so reps know they can fill the form manually instead of staring at a stalled spinner; the editable form now also opens on scanMutation.isError / uploadedFile, not only on success. - Email threads list now renders skeleton rows during load + shared EmptyState for the empty case (was a single "Loading…" line). Round 4 — Service / route correctness: - documentSends.sent_by_user_id was a free-text NOT NULL column with no FK. Now nullable + FK to user(id) ON DELETE SET NULL so the audit row survives a user being hard-deleted. Migration 0038 with a defensive null-out for any orphan ids before attaching the constraint. - Saved-views route: documented why withAuth alone is correct (the service strictly filters by (portId, userId) — owner-only by design). - Public-interests audit log: replaced "userId: null as unknown as string" cast with userId: null; AuditLogParams already accepts null for system-generated events. - EOI in-app PDF fill: extracted setBerthRange() that, when the AcroForm field is missing AND the context has a non-empty range string, logs a structured warn so the deployment gap (live Documenso template needs the field) is observable instead of silently dropping the multi-berth range. Test status: 1168/1168 vitest. tsc clean. Two new migrations (0037/0038) need pnpm db:push (or migration apply) on the dev DB. Deferred-doc updated with the remaining open items (bigger refactors). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 12:49:53 +02:00
onSuccess: (result) => {
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs, brochures) without touching those domains yet. New files: - src/lib/storage/index.ts StorageBackend interface + per-process factory keyed on system_settings. - src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/ Wasabi/Tigris) wrapping the existing minio JS client. Includes a healthCheck() used by the admin "Test connection" button. - src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a mitigations baked in. - src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock, per-row resumable progress markers, sha256 round-trip verification, atomic storage_backend flip on success. - scripts/migrate-storage.ts Thin CLI shim around runMigration(). - src/app/api/storage/[token]/route.ts Filesystem proxy GET. Verifies HMAC, enforces single-use replay protection via Redis SET NX, streams via NextResponse ReadableStream with explicit Content-Type + Content-Disposition. Node runtime only. - src/app/api/v1/admin/storage/route.ts GET status + POST connection test. - src/app/api/v1/admin/storage/migrate/route.ts Super-admin-only POST that runs the exact same runMigration() as the CLI. - src/app/(dashboard)/[portSlug]/admin/storage/page.tsx Super-admin admin UI (current backend, capacity stats, switch button with dry-run, test connection, backup hint). - src/components/admin/storage-admin-panel.tsx Client component for the page above. §14.9a critical mitigations implemented: - Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$; `..`, `.`, `//`, leading `/`, and overlength keys rejected. - Realpath: storage root realpath'd at create time, every per-key resolution checked against the realpath'd prefix. - Storage root created (or chmod'd) to 0o700. - Multi-node refusal: FilesystemBackend.create() throws when MULTI_NODE_DEPLOYMENT=true. - HMAC token: sha256-HMAC over the (key, expiry, nonce, filename, content-type) payload. Verified with timingSafeEqual; bad sig, expired, or invalid-key payloads all return 403. - Single-use replay: token body cached in Redis SET NX EX 1800s. - sha256 round-trip: copyAndVerify() re-fetches from the target after put() and aborts the migration on any mismatch. - Free-disk pre-flight: when migrating to filesystem, sums byte counts via source.head() and aborts if free space < total * 1.2. - pg_advisory_lock(0xc7000a01) prevents concurrent migrations. - Resumable: per-row progress markers in _storage_migration_progress. system_settings keys read by the factory (jsonb, no schema change): storage_backend, storage_s3_endpoint, storage_s3_region, storage_s3_bucket, storage_s3_access_key, storage_s3_secret_key_encrypted, storage_s3_force_path_style, storage_filesystem_root, storage_proxy_hmac_secret_encrypted. Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage` (./storage added to .gitignore). Tests added (34 tests, all green): - tests/unit/storage/filesystem-backend.test.ts — key validation allow/reject matrix, realpath escape, 0o700 perms, multi-node refusal, HMAC token sign/verify/tamper/expire/invalid-key. - tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on round-trip aborts the migration. - tests/integration/storage/proxy-route.test.ts — happy path, wrong HMAC secret, expired token, replay rejection. Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is intentionally empty. berth_pdf_versions and brochure_versions land in Phase 6b and join the list there. Existing s3_key columns: only gdpr_export_jobs.storage_key, already named correctly — no rename needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
setConfirmOpen(false);
setDryRun(null);
fix(audit-v3): platform-wide deferred-list cleanup (rounds 1-4) Working through the audit-v2 deferred backlog. Each round was tested (typecheck + 1168/1168 vitest) before moving on. Round 1 — DB performance + AI cost visibility: - Add missing FK indexes Postgres doesn't auto-create on berth_reservations.{interest_id, contract_file_id}, documents.{file_id, signed_file_id}, document_events.signer_id, document_templates.source_file_id, form_submissions.{form_template_id, client_id}, document_sends.{brochure_id, brochure_version_id, sent_by_user_id}. Without these, RESTRICT-checks on parent delete + reverse-lookups walk the child tables fully. Migration 0037. - AI worker now writes one ai_usage_ledger row per OpenAI call so admins can audit spend per port/user/feature and future per-port budgets have history to read from. Failure to write is logged-not-thrown so the user-facing email draft is unaffected. Round 2 — Boot-time + transport hardening: - S3 backend verifies the bucket exists at startup (or auto-creates when MINIO_AUTO_CREATE_BUCKET=true). A typo'd bucket name now surfaces with a clear boot error instead of a vague Minio error inside the first user-facing request. - Documenso v1 placeFields: 3-attempt exponential-backoff retry on 5xx + network errors, fail-fast on 4xx. Stops one transient flake from leaving a document with a partial field set. - FilesystemBackend logs a structured warn-once at boot when the dev HMAC fallback is in effect, so two processes started with different BETTER_AUTH_SECRET values are observable (random 401s on file downloads otherwise). - Logger redact paths extended to cover *.headers.{authorization, cookie}, *.config.headers.authorization, encrypted-credential blobs (secretKeyEncrypted, smtpPassEncrypted, etc.), the Documenso X-Documenso-Secret header, and 2-level nested forms. Round 3 — UI feedback + permission gates: - Storage admin migrate dialog: success toast with row count + error toast on both dryRun and migrate mutations. - Invoice detail Send + Record-payment buttons wrapped in PermissionGate (invoices.send / invoices.record_payment); both mutations now toast on success/error. - Admin user list Edit button wrapped in PermissionGate(admin.manage_users). - Scan-receipt page surfaces an amber warning when OCR fails so reps know they can fill the form manually instead of staring at a stalled spinner; the editable form now also opens on scanMutation.isError / uploadedFile, not only on success. - Email threads list now renders skeleton rows during load + shared EmptyState for the empty case (was a single "Loading…" line). Round 4 — Service / route correctness: - documentSends.sent_by_user_id was a free-text NOT NULL column with no FK. Now nullable + FK to user(id) ON DELETE SET NULL so the audit row survives a user being hard-deleted. Migration 0038 with a defensive null-out for any orphan ids before attaching the constraint. - Saved-views route: documented why withAuth alone is correct (the service strictly filters by (portId, userId) — owner-only by design). - Public-interests audit log: replaced "userId: null as unknown as string" cast with userId: null; AuditLogParams already accepts null for system-generated events. - EOI in-app PDF fill: extracted setBerthRange() that, when the AcroForm field is missing AND the context has a non-empty range string, logs a structured warn so the deployment gap (live Documenso template needs the field) is observable instead of silently dropping the multi-berth range. Test status: 1168/1168 vitest. tsc clean. Two new migrations (0037/0038) need pnpm db:push (or migration apply) on the dev DB. Deferred-doc updated with the remaining open items (bigger refactors). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 12:49:53 +02:00
const copied = result.data.rowsMigrated ?? 0;
toast.success(`Storage migration complete (${copied} file${copied === 1 ? '' : 's'} copied)`);
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs, brochures) without touching those domains yet. New files: - src/lib/storage/index.ts StorageBackend interface + per-process factory keyed on system_settings. - src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/ Wasabi/Tigris) wrapping the existing minio JS client. Includes a healthCheck() used by the admin "Test connection" button. - src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a mitigations baked in. - src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock, per-row resumable progress markers, sha256 round-trip verification, atomic storage_backend flip on success. - scripts/migrate-storage.ts Thin CLI shim around runMigration(). - src/app/api/storage/[token]/route.ts Filesystem proxy GET. Verifies HMAC, enforces single-use replay protection via Redis SET NX, streams via NextResponse ReadableStream with explicit Content-Type + Content-Disposition. Node runtime only. - src/app/api/v1/admin/storage/route.ts GET status + POST connection test. - src/app/api/v1/admin/storage/migrate/route.ts Super-admin-only POST that runs the exact same runMigration() as the CLI. - src/app/(dashboard)/[portSlug]/admin/storage/page.tsx Super-admin admin UI (current backend, capacity stats, switch button with dry-run, test connection, backup hint). - src/components/admin/storage-admin-panel.tsx Client component for the page above. §14.9a critical mitigations implemented: - Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$; `..`, `.`, `//`, leading `/`, and overlength keys rejected. - Realpath: storage root realpath'd at create time, every per-key resolution checked against the realpath'd prefix. - Storage root created (or chmod'd) to 0o700. - Multi-node refusal: FilesystemBackend.create() throws when MULTI_NODE_DEPLOYMENT=true. - HMAC token: sha256-HMAC over the (key, expiry, nonce, filename, content-type) payload. Verified with timingSafeEqual; bad sig, expired, or invalid-key payloads all return 403. - Single-use replay: token body cached in Redis SET NX EX 1800s. - sha256 round-trip: copyAndVerify() re-fetches from the target after put() and aborts the migration on any mismatch. - Free-disk pre-flight: when migrating to filesystem, sums byte counts via source.head() and aborts if free space < total * 1.2. - pg_advisory_lock(0xc7000a01) prevents concurrent migrations. - Resumable: per-row progress markers in _storage_migration_progress. system_settings keys read by the factory (jsonb, no schema change): storage_backend, storage_s3_endpoint, storage_s3_region, storage_s3_bucket, storage_s3_access_key, storage_s3_secret_key_encrypted, storage_s3_force_path_style, storage_filesystem_root, storage_proxy_hmac_secret_encrypted. Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage` (./storage added to .gitignore). Tests added (34 tests, all green): - tests/unit/storage/filesystem-backend.test.ts — key validation allow/reject matrix, realpath escape, 0o700 perms, multi-node refusal, HMAC token sign/verify/tamper/expire/invalid-key. - tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on round-trip aborts the migration. - tests/integration/storage/proxy-route.test.ts — happy path, wrong HMAC secret, expired token, replay rejection. Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is intentionally empty. berth_pdf_versions and brochure_versions land in Phase 6b and join the list there. Existing s3_key columns: only gdpr_export_jobs.storage_key, already named correctly — no rename needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
queryClient.invalidateQueries({ queryKey: ['admin', 'storage', 'status'] });
},
fix(audit-v3): platform-wide deferred-list cleanup (rounds 1-4) Working through the audit-v2 deferred backlog. Each round was tested (typecheck + 1168/1168 vitest) before moving on. Round 1 — DB performance + AI cost visibility: - Add missing FK indexes Postgres doesn't auto-create on berth_reservations.{interest_id, contract_file_id}, documents.{file_id, signed_file_id}, document_events.signer_id, document_templates.source_file_id, form_submissions.{form_template_id, client_id}, document_sends.{brochure_id, brochure_version_id, sent_by_user_id}. Without these, RESTRICT-checks on parent delete + reverse-lookups walk the child tables fully. Migration 0037. - AI worker now writes one ai_usage_ledger row per OpenAI call so admins can audit spend per port/user/feature and future per-port budgets have history to read from. Failure to write is logged-not-thrown so the user-facing email draft is unaffected. Round 2 — Boot-time + transport hardening: - S3 backend verifies the bucket exists at startup (or auto-creates when MINIO_AUTO_CREATE_BUCKET=true). A typo'd bucket name now surfaces with a clear boot error instead of a vague Minio error inside the first user-facing request. - Documenso v1 placeFields: 3-attempt exponential-backoff retry on 5xx + network errors, fail-fast on 4xx. Stops one transient flake from leaving a document with a partial field set. - FilesystemBackend logs a structured warn-once at boot when the dev HMAC fallback is in effect, so two processes started with different BETTER_AUTH_SECRET values are observable (random 401s on file downloads otherwise). - Logger redact paths extended to cover *.headers.{authorization, cookie}, *.config.headers.authorization, encrypted-credential blobs (secretKeyEncrypted, smtpPassEncrypted, etc.), the Documenso X-Documenso-Secret header, and 2-level nested forms. Round 3 — UI feedback + permission gates: - Storage admin migrate dialog: success toast with row count + error toast on both dryRun and migrate mutations. - Invoice detail Send + Record-payment buttons wrapped in PermissionGate (invoices.send / invoices.record_payment); both mutations now toast on success/error. - Admin user list Edit button wrapped in PermissionGate(admin.manage_users). - Scan-receipt page surfaces an amber warning when OCR fails so reps know they can fill the form manually instead of staring at a stalled spinner; the editable form now also opens on scanMutation.isError / uploadedFile, not only on success. - Email threads list now renders skeleton rows during load + shared EmptyState for the empty case (was a single "Loading…" line). Round 4 — Service / route correctness: - documentSends.sent_by_user_id was a free-text NOT NULL column with no FK. Now nullable + FK to user(id) ON DELETE SET NULL so the audit row survives a user being hard-deleted. Migration 0038 with a defensive null-out for any orphan ids before attaching the constraint. - Saved-views route: documented why withAuth alone is correct (the service strictly filters by (portId, userId) — owner-only by design). - Public-interests audit log: replaced "userId: null as unknown as string" cast with userId: null; AuditLogParams already accepts null for system-generated events. - EOI in-app PDF fill: extracted setBerthRange() that, when the AcroForm field is missing AND the context has a non-empty range string, logs a structured warn so the deployment gap (live Documenso template needs the field) is observable instead of silently dropping the multi-berth range. Test status: 1168/1168 vitest. tsc clean. Two new migrations (0037/0038) need pnpm db:push (or migration apply) on the dev DB. Deferred-doc updated with the remaining open items (bigger refactors). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 12:49:53 +02:00
onError: (e) => toast.error(e instanceof Error ? e.message : 'Storage migration failed'),
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs, brochures) without touching those domains yet. New files: - src/lib/storage/index.ts StorageBackend interface + per-process factory keyed on system_settings. - src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/ Wasabi/Tigris) wrapping the existing minio JS client. Includes a healthCheck() used by the admin "Test connection" button. - src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a mitigations baked in. - src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock, per-row resumable progress markers, sha256 round-trip verification, atomic storage_backend flip on success. - scripts/migrate-storage.ts Thin CLI shim around runMigration(). - src/app/api/storage/[token]/route.ts Filesystem proxy GET. Verifies HMAC, enforces single-use replay protection via Redis SET NX, streams via NextResponse ReadableStream with explicit Content-Type + Content-Disposition. Node runtime only. - src/app/api/v1/admin/storage/route.ts GET status + POST connection test. - src/app/api/v1/admin/storage/migrate/route.ts Super-admin-only POST that runs the exact same runMigration() as the CLI. - src/app/(dashboard)/[portSlug]/admin/storage/page.tsx Super-admin admin UI (current backend, capacity stats, switch button with dry-run, test connection, backup hint). - src/components/admin/storage-admin-panel.tsx Client component for the page above. §14.9a critical mitigations implemented: - Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$; `..`, `.`, `//`, leading `/`, and overlength keys rejected. - Realpath: storage root realpath'd at create time, every per-key resolution checked against the realpath'd prefix. - Storage root created (or chmod'd) to 0o700. - Multi-node refusal: FilesystemBackend.create() throws when MULTI_NODE_DEPLOYMENT=true. - HMAC token: sha256-HMAC over the (key, expiry, nonce, filename, content-type) payload. Verified with timingSafeEqual; bad sig, expired, or invalid-key payloads all return 403. - Single-use replay: token body cached in Redis SET NX EX 1800s. - sha256 round-trip: copyAndVerify() re-fetches from the target after put() and aborts the migration on any mismatch. - Free-disk pre-flight: when migrating to filesystem, sums byte counts via source.head() and aborts if free space < total * 1.2. - pg_advisory_lock(0xc7000a01) prevents concurrent migrations. - Resumable: per-row progress markers in _storage_migration_progress. system_settings keys read by the factory (jsonb, no schema change): storage_backend, storage_s3_endpoint, storage_s3_region, storage_s3_bucket, storage_s3_access_key, storage_s3_secret_key_encrypted, storage_s3_force_path_style, storage_filesystem_root, storage_proxy_hmac_secret_encrypted. Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage` (./storage added to .gitignore). Tests added (34 tests, all green): - tests/unit/storage/filesystem-backend.test.ts — key validation allow/reject matrix, realpath escape, 0o700 perms, multi-node refusal, HMAC token sign/verify/tamper/expire/invalid-key. - tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on round-trip aborts the migration. - tests/integration/storage/proxy-route.test.ts — happy path, wrong HMAC secret, expired token, replay rejection. Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is intentionally empty. berth_pdf_versions and brochure_versions land in Phase 6b and join the list there. Existing s3_key columns: only gdpr_export_jobs.storage_key, already named correctly — no rename needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
});
const testMutation = useMutation({
mutationFn: async () =>
apiFetch<{ ok: boolean; error?: string }>('/api/v1/admin/storage', {
method: 'POST',
}),
onSuccess: (r) => setTestResult(r),
onError: (e: Error) => setTestResult({ ok: false, error: e.message }),
});
if (status.isLoading) {
return (
<div className="flex items-center gap-2 text-sm text-muted-foreground">
<Loader2 className="h-4 w-4 animate-spin" /> Loading storage status
</div>
);
}
if (status.isError || !status.data?.data) {
return <div className="text-sm text-destructive">Failed to load storage status.</div>;
}
const s = status.data.data;
const otherBackend: BackendName = s.backend === 's3' ? 'filesystem' : 's3';
return (
<div className="space-y-6">
<PageHeader
title="Storage Backend"
description="Where the CRM stores per-berth PDFs, brochures, GDPR exports, and other binary files."
/>
<div className="grid gap-4 lg:grid-cols-3">
<Card className="lg:col-span-2">
<CardHeader className="flex flex-row items-start gap-3 space-y-0 pb-2">
{s.backend === 's3' ? (
<ServerCog className="mt-0.5 h-5 w-5 text-muted-foreground" />
) : (
<HardDrive className="mt-0.5 h-5 w-5 text-muted-foreground" />
)}
<div>
<CardTitle className="text-base">Active backend: {s.backend}</CardTitle>
<CardDescription>
{s.backend === 's3'
? 'Files stored in an S3-compatible object store (MinIO, AWS S3, Backblaze B2, Cloudflare R2, Wasabi, Tigris).'
: 'Files stored on the local filesystem under storage_filesystem_root. Single-node deployments only.'}
</CardDescription>
</div>
</CardHeader>
<CardContent className="space-y-4">
<dl className="grid grid-cols-2 gap-3 text-sm">
<div>
<dt className="text-muted-foreground">Tracked tables</dt>
<dd>
{s.tablesTracked.length === 0
? '(none yet — Phase 6b)'
: s.tablesTracked.join(', ')}
</dd>
</div>
<div>
<dt className="text-muted-foreground">File count</dt>
<dd>{s.fileCount}</dd>
</div>
</dl>
<div className="flex flex-wrap gap-3">
<Button
variant="outline"
disabled={dryRunMutation.isPending}
onClick={() => dryRunMutation.mutate({ from: s.backend, to: otherBackend })}
>
{dryRunMutation.isPending && <Loader2 className="mr-2 h-4 w-4 animate-spin" />}
Switch to {otherBackend}
</Button>
{s.backend === 's3' && (
<Button
variant="outline"
onClick={() => testMutation.mutate()}
disabled={testMutation.isPending}
>
{testMutation.isPending && <Loader2 className="mr-2 h-4 w-4 animate-spin" />}
Test connection
</Button>
)}
<Button variant="ghost" onClick={() => status.refetch()} disabled={status.isFetching}>
<RefreshCw className="mr-2 h-4 w-4" /> Refresh
</Button>
</div>
{testResult && (
<div className="rounded-md border p-3 text-sm">
{testResult.ok ? (
<div className="flex items-center gap-2 text-emerald-600">
<CheckCircle2 className="h-4 w-4" /> Connection OK round-trip succeeded.
</div>
) : (
<div className="flex items-center gap-2 text-destructive">
<XCircle className="h-4 w-4" /> {testResult.error ?? 'Connection failed'}
</div>
)}
</div>
)}
</CardContent>
</Card>
<Card>
<CardHeader>
<CardTitle className="text-base">Backup notes</CardTitle>
</CardHeader>
<CardContent className="space-y-2 text-sm text-muted-foreground">
{s.backend === 's3' ? (
<p>
S3 mode: configure your provider&apos;s lifecycle / replication / versioning
policies as your primary backup. The CRM does not duplicate object storage in its
own backups.
</p>
) : (
<p>
Filesystem mode: include the storage root directory in your backup tool (restic,
borg, snapshots). It sits next to the database; the two should be backed up
together.
</p>
)}
<p className="pt-2 text-xs">
Filesystem mode refuses to start when MULTI_NODE_DEPLOYMENT=true. For multi-node
deployments, switch to an S3-compatible backend.
</p>
</CardContent>
</Card>
</div>
<Dialog open={confirmOpen} onOpenChange={setConfirmOpen}>
<DialogContent>
<DialogHeader>
<DialogTitle>Switch storage backend</DialogTitle>
<DialogDescription>
Move all tracked files from the current backend to the new backend, verify each file
via sha256, then atomically flip the active backend.
</DialogDescription>
</DialogHeader>
{dryRun && (
<div className="rounded-md border p-3 text-sm">
<dl className="grid grid-cols-2 gap-2">
<dt className="text-muted-foreground">Rows considered</dt>
<dd>{dryRun.rowsConsidered}</dd>
<dt className="text-muted-foreground">Already migrated (resumable)</dt>
<dd>{dryRun.rowsSkippedAlreadyDone}</dd>
<dt className="text-muted-foreground">Total bytes</dt>
<dd>{Math.round(dryRun.totalBytes / 1024)} KB</dd>
</dl>
</div>
)}
<DialogFooter>
<Button variant="outline" onClick={() => setConfirmOpen(false)}>
Cancel
</Button>
<Button
disabled={migrateMutation.isPending}
onClick={() => migrateMutation.mutate({ from: s.backend, to: otherBackend })}
>
{migrateMutation.isPending && <Loader2 className="mr-2 h-4 w-4 animate-spin" />}
Migrate now
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
</div>
);
}