Files
pn-new-crm/src/app/api/v1/admin/storage/route.ts

69 lines
2.1 KiB
TypeScript
Raw Normal View History

feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs, brochures) without touching those domains yet. New files: - src/lib/storage/index.ts StorageBackend interface + per-process factory keyed on system_settings. - src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/ Wasabi/Tigris) wrapping the existing minio JS client. Includes a healthCheck() used by the admin "Test connection" button. - src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a mitigations baked in. - src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock, per-row resumable progress markers, sha256 round-trip verification, atomic storage_backend flip on success. - scripts/migrate-storage.ts Thin CLI shim around runMigration(). - src/app/api/storage/[token]/route.ts Filesystem proxy GET. Verifies HMAC, enforces single-use replay protection via Redis SET NX, streams via NextResponse ReadableStream with explicit Content-Type + Content-Disposition. Node runtime only. - src/app/api/v1/admin/storage/route.ts GET status + POST connection test. - src/app/api/v1/admin/storage/migrate/route.ts Super-admin-only POST that runs the exact same runMigration() as the CLI. - src/app/(dashboard)/[portSlug]/admin/storage/page.tsx Super-admin admin UI (current backend, capacity stats, switch button with dry-run, test connection, backup hint). - src/components/admin/storage-admin-panel.tsx Client component for the page above. §14.9a critical mitigations implemented: - Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$; `..`, `.`, `//`, leading `/`, and overlength keys rejected. - Realpath: storage root realpath'd at create time, every per-key resolution checked against the realpath'd prefix. - Storage root created (or chmod'd) to 0o700. - Multi-node refusal: FilesystemBackend.create() throws when MULTI_NODE_DEPLOYMENT=true. - HMAC token: sha256-HMAC over the (key, expiry, nonce, filename, content-type) payload. Verified with timingSafeEqual; bad sig, expired, or invalid-key payloads all return 403. - Single-use replay: token body cached in Redis SET NX EX 1800s. - sha256 round-trip: copyAndVerify() re-fetches from the target after put() and aborts the migration on any mismatch. - Free-disk pre-flight: when migrating to filesystem, sums byte counts via source.head() and aborts if free space < total * 1.2. - pg_advisory_lock(0xc7000a01) prevents concurrent migrations. - Resumable: per-row progress markers in _storage_migration_progress. system_settings keys read by the factory (jsonb, no schema change): storage_backend, storage_s3_endpoint, storage_s3_region, storage_s3_bucket, storage_s3_access_key, storage_s3_secret_key_encrypted, storage_s3_force_path_style, storage_filesystem_root, storage_proxy_hmac_secret_encrypted. Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage` (./storage added to .gitignore). Tests added (34 tests, all green): - tests/unit/storage/filesystem-backend.test.ts — key validation allow/reject matrix, realpath escape, 0o700 perms, multi-node refusal, HMAC token sign/verify/tamper/expire/invalid-key. - tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on round-trip aborts the migration. - tests/integration/storage/proxy-route.test.ts — happy path, wrong HMAC secret, expired token, replay rejection. Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is intentionally empty. berth_pdf_versions and brochure_versions land in Phase 6b and join the list there. Existing s3_key columns: only gdpr_export_jobs.storage_key, already named correctly — no rename needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
/**
* Admin storage status + connection test. Super-admin only.
*
* GET /api/v1/admin/storage current backend + capacity stats
* POST /api/v1/admin/storage/test exercise list/put/get/delete on s3
*/
import { NextResponse } from 'next/server';
fix(audit-tier-2-routes): manual NextResponse.json error sweep + admin form banners Two final waves of error-surface hygiene closing the audit's MED §12 + HIGH §15 + HIGH §17 findings: * 50 route files swept (61 sites): manual NextResponse.json({error, status: 4xx|5xx}) early-returns replaced by typed throws + errorResponse(err) at the catch. - Super-admin gates (13 sites) use new requireSuperAdmin(ctx, action) helper from src/lib/api/helpers.ts so denials hit the audit log. - Path-param + body validation 400s become ValidationError throws. - 404s become NotFoundError or CodedError('NOT_FOUND') for AI feature-flag paths. - 11 manual 5xx returns now re-throw so error_events captures the request-id (the admin error inspector becomes usable from real incidents). - website-analytics 200-with-error anti-pattern flipped to 409 + UMAMI_NOT_CONFIGURED. 502 upstream paths use UMAMI_UPSTREAM_ERROR. - 11 sites intentionally preserved: storage/[token] anti-enumeration token-failure paths, webhook-secret 401, "Unknown port" 400 in public intake. * 7 admin forms (roles, users, ports, webhooks, custom-fields, document-templates, tags) gain a formatErrorBanner() helper from src/lib/api/toast-error.ts that builds a multi-line "Error code / Reference ID" banner — the rep can copy the request id when reporting a failed save. Banners get whitespace-pre-line so newlines render. Test status: 1168/1168 vitest, tsc clean. Refs: docs/audit-comprehensive-2026-05-05.md MED §12 (auditor-F Issue 1) + HIGH §15 (auditor-F Issue 2) + HIGH §17 (auditor-H Issue 2). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 20:36:59 +02:00
import { requireSuperAdmin, withAuth } from '@/lib/api/helpers';
import { errorResponse } from '@/lib/errors';
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs, brochures) without touching those domains yet. New files: - src/lib/storage/index.ts StorageBackend interface + per-process factory keyed on system_settings. - src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/ Wasabi/Tigris) wrapping the existing minio JS client. Includes a healthCheck() used by the admin "Test connection" button. - src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a mitigations baked in. - src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock, per-row resumable progress markers, sha256 round-trip verification, atomic storage_backend flip on success. - scripts/migrate-storage.ts Thin CLI shim around runMigration(). - src/app/api/storage/[token]/route.ts Filesystem proxy GET. Verifies HMAC, enforces single-use replay protection via Redis SET NX, streams via NextResponse ReadableStream with explicit Content-Type + Content-Disposition. Node runtime only. - src/app/api/v1/admin/storage/route.ts GET status + POST connection test. - src/app/api/v1/admin/storage/migrate/route.ts Super-admin-only POST that runs the exact same runMigration() as the CLI. - src/app/(dashboard)/[portSlug]/admin/storage/page.tsx Super-admin admin UI (current backend, capacity stats, switch button with dry-run, test connection, backup hint). - src/components/admin/storage-admin-panel.tsx Client component for the page above. §14.9a critical mitigations implemented: - Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$; `..`, `.`, `//`, leading `/`, and overlength keys rejected. - Realpath: storage root realpath'd at create time, every per-key resolution checked against the realpath'd prefix. - Storage root created (or chmod'd) to 0o700. - Multi-node refusal: FilesystemBackend.create() throws when MULTI_NODE_DEPLOYMENT=true. - HMAC token: sha256-HMAC over the (key, expiry, nonce, filename, content-type) payload. Verified with timingSafeEqual; bad sig, expired, or invalid-key payloads all return 403. - Single-use replay: token body cached in Redis SET NX EX 1800s. - sha256 round-trip: copyAndVerify() re-fetches from the target after put() and aborts the migration on any mismatch. - Free-disk pre-flight: when migrating to filesystem, sums byte counts via source.head() and aborts if free space < total * 1.2. - pg_advisory_lock(0xc7000a01) prevents concurrent migrations. - Resumable: per-row progress markers in _storage_migration_progress. system_settings keys read by the factory (jsonb, no schema change): storage_backend, storage_s3_endpoint, storage_s3_region, storage_s3_bucket, storage_s3_access_key, storage_s3_secret_key_encrypted, storage_s3_force_path_style, storage_filesystem_root, storage_proxy_hmac_secret_encrypted. Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage` (./storage added to .gitignore). Tests added (34 tests, all green): - tests/unit/storage/filesystem-backend.test.ts — key validation allow/reject matrix, realpath escape, 0o700 perms, multi-node refusal, HMAC token sign/verify/tamper/expire/invalid-key. - tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on round-trip aborts the migration. - tests/integration/storage/proxy-route.test.ts — happy path, wrong HMAC secret, expired token, replay rejection. Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is intentionally empty. berth_pdf_versions and brochure_versions land in Phase 6b and join the list there. Existing s3_key columns: only gdpr_export_jobs.storage_key, already named correctly — no rename needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
import { TABLES_WITH_STORAGE_KEYS } from '@/lib/storage/migrate';
import { getStorageBackend } from '@/lib/storage';
import { S3Backend } from '@/lib/storage/s3';
import { db } from '@/lib/db';
import { sql } from 'drizzle-orm';
export const runtime = 'nodejs';
export const GET = withAuth(async (_req, ctx) => {
try {
fix(audit-tier-2-routes): manual NextResponse.json error sweep + admin form banners Two final waves of error-surface hygiene closing the audit's MED §12 + HIGH §15 + HIGH §17 findings: * 50 route files swept (61 sites): manual NextResponse.json({error, status: 4xx|5xx}) early-returns replaced by typed throws + errorResponse(err) at the catch. - Super-admin gates (13 sites) use new requireSuperAdmin(ctx, action) helper from src/lib/api/helpers.ts so denials hit the audit log. - Path-param + body validation 400s become ValidationError throws. - 404s become NotFoundError or CodedError('NOT_FOUND') for AI feature-flag paths. - 11 manual 5xx returns now re-throw so error_events captures the request-id (the admin error inspector becomes usable from real incidents). - website-analytics 200-with-error anti-pattern flipped to 409 + UMAMI_NOT_CONFIGURED. 502 upstream paths use UMAMI_UPSTREAM_ERROR. - 11 sites intentionally preserved: storage/[token] anti-enumeration token-failure paths, webhook-secret 401, "Unknown port" 400 in public intake. * 7 admin forms (roles, users, ports, webhooks, custom-fields, document-templates, tags) gain a formatErrorBanner() helper from src/lib/api/toast-error.ts that builds a multi-line "Error code / Reference ID" banner — the rep can copy the request id when reporting a failed save. Banners get whitespace-pre-line so newlines render. Test status: 1168/1168 vitest, tsc clean. Refs: docs/audit-comprehensive-2026-05-05.md MED §12 (auditor-F Issue 1) + HIGH §15 (auditor-F Issue 2) + HIGH §17 (auditor-H Issue 2). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 20:36:59 +02:00
requireSuperAdmin(ctx, 'admin.storage.read');
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs, brochures) without touching those domains yet. New files: - src/lib/storage/index.ts StorageBackend interface + per-process factory keyed on system_settings. - src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/ Wasabi/Tigris) wrapping the existing minio JS client. Includes a healthCheck() used by the admin "Test connection" button. - src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a mitigations baked in. - src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock, per-row resumable progress markers, sha256 round-trip verification, atomic storage_backend flip on success. - scripts/migrate-storage.ts Thin CLI shim around runMigration(). - src/app/api/storage/[token]/route.ts Filesystem proxy GET. Verifies HMAC, enforces single-use replay protection via Redis SET NX, streams via NextResponse ReadableStream with explicit Content-Type + Content-Disposition. Node runtime only. - src/app/api/v1/admin/storage/route.ts GET status + POST connection test. - src/app/api/v1/admin/storage/migrate/route.ts Super-admin-only POST that runs the exact same runMigration() as the CLI. - src/app/(dashboard)/[portSlug]/admin/storage/page.tsx Super-admin admin UI (current backend, capacity stats, switch button with dry-run, test connection, backup hint). - src/components/admin/storage-admin-panel.tsx Client component for the page above. §14.9a critical mitigations implemented: - Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$; `..`, `.`, `//`, leading `/`, and overlength keys rejected. - Realpath: storage root realpath'd at create time, every per-key resolution checked against the realpath'd prefix. - Storage root created (or chmod'd) to 0o700. - Multi-node refusal: FilesystemBackend.create() throws when MULTI_NODE_DEPLOYMENT=true. - HMAC token: sha256-HMAC over the (key, expiry, nonce, filename, content-type) payload. Verified with timingSafeEqual; bad sig, expired, or invalid-key payloads all return 403. - Single-use replay: token body cached in Redis SET NX EX 1800s. - sha256 round-trip: copyAndVerify() re-fetches from the target after put() and aborts the migration on any mismatch. - Free-disk pre-flight: when migrating to filesystem, sums byte counts via source.head() and aborts if free space < total * 1.2. - pg_advisory_lock(0xc7000a01) prevents concurrent migrations. - Resumable: per-row progress markers in _storage_migration_progress. system_settings keys read by the factory (jsonb, no schema change): storage_backend, storage_s3_endpoint, storage_s3_region, storage_s3_bucket, storage_s3_access_key, storage_s3_secret_key_encrypted, storage_s3_force_path_style, storage_filesystem_root, storage_proxy_hmac_secret_encrypted. Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage` (./storage added to .gitignore). Tests added (34 tests, all green): - tests/unit/storage/filesystem-backend.test.ts — key validation allow/reject matrix, realpath escape, 0o700 perms, multi-node refusal, HMAC token sign/verify/tamper/expire/invalid-key. - tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on round-trip aborts the migration. - tests/integration/storage/proxy-route.test.ts — happy path, wrong HMAC secret, expired token, replay rejection. Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is intentionally empty. berth_pdf_versions and brochure_versions land in Phase 6b and join the list there. Existing s3_key columns: only gdpr_export_jobs.storage_key, already named correctly — no rename needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
const backend = await getStorageBackend();
// Aggregate row count + total bytes across every storage-bearing table.
let fileCount = 0;
const totalBytes = 0;
for (const tbl of TABLES_WITH_STORAGE_KEYS) {
const result = await db.execute(
sql.raw(
`SELECT COUNT(*)::bigint AS n FROM ${tbl.table} WHERE ${tbl.keyColumn} IS NOT NULL`,
),
);
const rows = (
Array.isArray(result) ? result : ((result as { rows?: unknown[] }).rows ?? [])
) as Array<{ n: number | string }>;
fileCount += Number(rows[0]?.n ?? 0);
}
return NextResponse.json({
data: {
backend: backend.name,
fileCount,
totalBytes,
tablesTracked: TABLES_WITH_STORAGE_KEYS.map((t) => t.table),
},
});
} catch (error) {
return errorResponse(error);
}
});
export const POST = withAuth(async (_req, ctx) => {
try {
fix(audit-tier-2-routes): manual NextResponse.json error sweep + admin form banners Two final waves of error-surface hygiene closing the audit's MED §12 + HIGH §15 + HIGH §17 findings: * 50 route files swept (61 sites): manual NextResponse.json({error, status: 4xx|5xx}) early-returns replaced by typed throws + errorResponse(err) at the catch. - Super-admin gates (13 sites) use new requireSuperAdmin(ctx, action) helper from src/lib/api/helpers.ts so denials hit the audit log. - Path-param + body validation 400s become ValidationError throws. - 404s become NotFoundError or CodedError('NOT_FOUND') for AI feature-flag paths. - 11 manual 5xx returns now re-throw so error_events captures the request-id (the admin error inspector becomes usable from real incidents). - website-analytics 200-with-error anti-pattern flipped to 409 + UMAMI_NOT_CONFIGURED. 502 upstream paths use UMAMI_UPSTREAM_ERROR. - 11 sites intentionally preserved: storage/[token] anti-enumeration token-failure paths, webhook-secret 401, "Unknown port" 400 in public intake. * 7 admin forms (roles, users, ports, webhooks, custom-fields, document-templates, tags) gain a formatErrorBanner() helper from src/lib/api/toast-error.ts that builds a multi-line "Error code / Reference ID" banner — the rep can copy the request id when reporting a failed save. Banners get whitespace-pre-line so newlines render. Test status: 1168/1168 vitest, tsc clean. Refs: docs/audit-comprehensive-2026-05-05.md MED §12 (auditor-F Issue 1) + HIGH §15 (auditor-F Issue 2) + HIGH §17 (auditor-H Issue 2). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 20:36:59 +02:00
requireSuperAdmin(ctx, 'admin.storage.test');
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs, brochures) without touching those domains yet. New files: - src/lib/storage/index.ts StorageBackend interface + per-process factory keyed on system_settings. - src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/ Wasabi/Tigris) wrapping the existing minio JS client. Includes a healthCheck() used by the admin "Test connection" button. - src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a mitigations baked in. - src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock, per-row resumable progress markers, sha256 round-trip verification, atomic storage_backend flip on success. - scripts/migrate-storage.ts Thin CLI shim around runMigration(). - src/app/api/storage/[token]/route.ts Filesystem proxy GET. Verifies HMAC, enforces single-use replay protection via Redis SET NX, streams via NextResponse ReadableStream with explicit Content-Type + Content-Disposition. Node runtime only. - src/app/api/v1/admin/storage/route.ts GET status + POST connection test. - src/app/api/v1/admin/storage/migrate/route.ts Super-admin-only POST that runs the exact same runMigration() as the CLI. - src/app/(dashboard)/[portSlug]/admin/storage/page.tsx Super-admin admin UI (current backend, capacity stats, switch button with dry-run, test connection, backup hint). - src/components/admin/storage-admin-panel.tsx Client component for the page above. §14.9a critical mitigations implemented: - Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$; `..`, `.`, `//`, leading `/`, and overlength keys rejected. - Realpath: storage root realpath'd at create time, every per-key resolution checked against the realpath'd prefix. - Storage root created (or chmod'd) to 0o700. - Multi-node refusal: FilesystemBackend.create() throws when MULTI_NODE_DEPLOYMENT=true. - HMAC token: sha256-HMAC over the (key, expiry, nonce, filename, content-type) payload. Verified with timingSafeEqual; bad sig, expired, or invalid-key payloads all return 403. - Single-use replay: token body cached in Redis SET NX EX 1800s. - sha256 round-trip: copyAndVerify() re-fetches from the target after put() and aborts the migration on any mismatch. - Free-disk pre-flight: when migrating to filesystem, sums byte counts via source.head() and aborts if free space < total * 1.2. - pg_advisory_lock(0xc7000a01) prevents concurrent migrations. - Resumable: per-row progress markers in _storage_migration_progress. system_settings keys read by the factory (jsonb, no schema change): storage_backend, storage_s3_endpoint, storage_s3_region, storage_s3_bucket, storage_s3_access_key, storage_s3_secret_key_encrypted, storage_s3_force_path_style, storage_filesystem_root, storage_proxy_hmac_secret_encrypted. Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage` (./storage added to .gitignore). Tests added (34 tests, all green): - tests/unit/storage/filesystem-backend.test.ts — key validation allow/reject matrix, realpath escape, 0o700 perms, multi-node refusal, HMAC token sign/verify/tamper/expire/invalid-key. - tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on round-trip aborts the migration. - tests/integration/storage/proxy-route.test.ts — happy path, wrong HMAC secret, expired token, replay rejection. Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is intentionally empty. berth_pdf_versions and brochure_versions land in Phase 6b and join the list there. Existing s3_key columns: only gdpr_export_jobs.storage_key, already named correctly — no rename needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
const backend = await getStorageBackend();
if (!(backend instanceof S3Backend)) {
return NextResponse.json(
{ ok: false, error: 'Test connection only available for S3 backend' },
{ status: 400 },
);
}
const result = await backend.healthCheck();
return NextResponse.json(result);
} catch (error) {
return errorResponse(error);
}
});