feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI
Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays
the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs,
brochures) without touching those domains yet.
New files:
- src/lib/storage/index.ts StorageBackend interface + per-process
factory keyed on system_settings.
- src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/
Wasabi/Tigris) wrapping the existing minio
JS client. Includes a healthCheck() used
by the admin "Test connection" button.
- src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a
mitigations baked in.
- src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock,
per-row resumable progress markers,
sha256 round-trip verification, atomic
storage_backend flip on success.
- scripts/migrate-storage.ts Thin CLI shim around runMigration().
- src/app/api/storage/[token]/route.ts
Filesystem proxy GET. Verifies HMAC,
enforces single-use replay protection
via Redis SET NX, streams via NextResponse
ReadableStream with explicit Content-Type
+ Content-Disposition. Node runtime only.
- src/app/api/v1/admin/storage/route.ts
GET status + POST connection test.
- src/app/api/v1/admin/storage/migrate/route.ts
Super-admin-only POST that runs the
exact same runMigration() as the CLI.
- src/app/(dashboard)/[portSlug]/admin/storage/page.tsx
Super-admin admin UI (current backend,
capacity stats, switch button with
dry-run, test connection, backup hint).
- src/components/admin/storage-admin-panel.tsx
Client component for the page above.
§14.9a critical mitigations implemented:
- Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$;
`..`, `.`, `//`, leading `/`, and overlength keys rejected.
- Realpath: storage root realpath'd at create time, every per-key
resolution checked against the realpath'd prefix.
- Storage root created (or chmod'd) to 0o700.
- Multi-node refusal: FilesystemBackend.create() throws when
MULTI_NODE_DEPLOYMENT=true.
- HMAC token: sha256-HMAC over the (key, expiry, nonce, filename,
content-type) payload. Verified with timingSafeEqual; bad sig,
expired, or invalid-key payloads all return 403.
- Single-use replay: token body cached in Redis SET NX EX 1800s.
- sha256 round-trip: copyAndVerify() re-fetches from the target after
put() and aborts the migration on any mismatch.
- Free-disk pre-flight: when migrating to filesystem, sums byte counts
via source.head() and aborts if free space < total * 1.2.
- pg_advisory_lock(0xc7000a01) prevents concurrent migrations.
- Resumable: per-row progress markers in _storage_migration_progress.
system_settings keys read by the factory (jsonb, no schema change):
storage_backend, storage_s3_endpoint, storage_s3_region,
storage_s3_bucket, storage_s3_access_key,
storage_s3_secret_key_encrypted, storage_s3_force_path_style,
storage_filesystem_root, storage_proxy_hmac_secret_encrypted.
Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage`
(./storage added to .gitignore).
Tests added (34 tests, all green):
- tests/unit/storage/filesystem-backend.test.ts — key validation
allow/reject matrix, realpath escape, 0o700 perms, multi-node
refusal, HMAC token sign/verify/tamper/expire/invalid-key.
- tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on
round-trip aborts the migration.
- tests/integration/storage/proxy-route.test.ts — happy path, wrong
HMAC secret, expired token, replay rejection.
Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is
intentionally empty. berth_pdf_versions and brochure_versions land in
Phase 6b and join the list there. Existing s3_key columns: only
gdpr_export_jobs.storage_key, already named correctly — no rename needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
|
|
|
/**
|
|
|
|
|
* Filesystem-backend download proxy.
|
|
|
|
|
*
|
|
|
|
|
* The `FilesystemBackend.presignDownload(...)` returns a CRM-internal URL of
|
|
|
|
|
* the form `/api/storage/<hmac-signed-token>`. This route verifies the HMAC,
|
|
|
|
|
* checks expiry, enforces single-use via a short Redis cache, then streams
|
|
|
|
|
* the file out with explicit `Content-Type` + `Content-Disposition`.
|
|
|
|
|
*
|
|
|
|
|
* §14.9a mitigations exercised here:
|
|
|
|
|
* - HMAC verification (timingSafeEqual via filesystem.verifyProxyToken)
|
|
|
|
|
* - expiry check (token includes `e` epoch seconds)
|
|
|
|
|
* - single-use replay protection via short Redis SET-NX
|
|
|
|
|
* - Node runtime only (no edge); explicit headers so Next.js doesn't try to
|
|
|
|
|
* process the bytes (no image optimization, no streaming transforms)
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
import { createReadStream } from 'node:fs';
|
|
|
|
|
import * as fs from 'node:fs/promises';
|
|
|
|
|
import { Readable } from 'node:stream';
|
|
|
|
|
|
|
|
|
|
import { NextRequest, NextResponse } from 'next/server';
|
|
|
|
|
|
|
|
|
|
import { logger } from '@/lib/logger';
|
|
|
|
|
import { redis } from '@/lib/redis';
|
|
|
|
|
import { FilesystemBackend, getStorageBackend } from '@/lib/storage';
|
|
|
|
|
import { verifyProxyToken } from '@/lib/storage/filesystem';
|
|
|
|
|
|
|
|
|
|
export const runtime = 'nodejs';
|
|
|
|
|
export const dynamic = 'force-dynamic';
|
|
|
|
|
|
2026-05-05 04:07:03 +02:00
|
|
|
// Replay-protection TTL must outlive the token itself, otherwise the
|
|
|
|
|
// dedup key expires and the same token can be redeemed twice. We pin it
|
|
|
|
|
// to the token's own expiry (clamped to a 25-day ceiling so a forged
|
|
|
|
|
// far-future token can't pollute Redis indefinitely). Send-out emails
|
|
|
|
|
// mint 24-hour tokens so the typical TTL is 24h + a small buffer.
|
|
|
|
|
const REPLAY_TTL_FLOOR_SECONDS = 60; // never below 60s (post-expiry tail).
|
|
|
|
|
const REPLAY_TTL_CEILING_SECONDS = 25 * 24 * 60 * 60; // 25 days.
|
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI
Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays
the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs,
brochures) without touching those domains yet.
New files:
- src/lib/storage/index.ts StorageBackend interface + per-process
factory keyed on system_settings.
- src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/
Wasabi/Tigris) wrapping the existing minio
JS client. Includes a healthCheck() used
by the admin "Test connection" button.
- src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a
mitigations baked in.
- src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock,
per-row resumable progress markers,
sha256 round-trip verification, atomic
storage_backend flip on success.
- scripts/migrate-storage.ts Thin CLI shim around runMigration().
- src/app/api/storage/[token]/route.ts
Filesystem proxy GET. Verifies HMAC,
enforces single-use replay protection
via Redis SET NX, streams via NextResponse
ReadableStream with explicit Content-Type
+ Content-Disposition. Node runtime only.
- src/app/api/v1/admin/storage/route.ts
GET status + POST connection test.
- src/app/api/v1/admin/storage/migrate/route.ts
Super-admin-only POST that runs the
exact same runMigration() as the CLI.
- src/app/(dashboard)/[portSlug]/admin/storage/page.tsx
Super-admin admin UI (current backend,
capacity stats, switch button with
dry-run, test connection, backup hint).
- src/components/admin/storage-admin-panel.tsx
Client component for the page above.
§14.9a critical mitigations implemented:
- Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$;
`..`, `.`, `//`, leading `/`, and overlength keys rejected.
- Realpath: storage root realpath'd at create time, every per-key
resolution checked against the realpath'd prefix.
- Storage root created (or chmod'd) to 0o700.
- Multi-node refusal: FilesystemBackend.create() throws when
MULTI_NODE_DEPLOYMENT=true.
- HMAC token: sha256-HMAC over the (key, expiry, nonce, filename,
content-type) payload. Verified with timingSafeEqual; bad sig,
expired, or invalid-key payloads all return 403.
- Single-use replay: token body cached in Redis SET NX EX 1800s.
- sha256 round-trip: copyAndVerify() re-fetches from the target after
put() and aborts the migration on any mismatch.
- Free-disk pre-flight: when migrating to filesystem, sums byte counts
via source.head() and aborts if free space < total * 1.2.
- pg_advisory_lock(0xc7000a01) prevents concurrent migrations.
- Resumable: per-row progress markers in _storage_migration_progress.
system_settings keys read by the factory (jsonb, no schema change):
storage_backend, storage_s3_endpoint, storage_s3_region,
storage_s3_bucket, storage_s3_access_key,
storage_s3_secret_key_encrypted, storage_s3_force_path_style,
storage_filesystem_root, storage_proxy_hmac_secret_encrypted.
Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage`
(./storage added to .gitignore).
Tests added (34 tests, all green):
- tests/unit/storage/filesystem-backend.test.ts — key validation
allow/reject matrix, realpath escape, 0o700 perms, multi-node
refusal, HMAC token sign/verify/tamper/expire/invalid-key.
- tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on
round-trip aborts the migration.
- tests/integration/storage/proxy-route.test.ts — happy path, wrong
HMAC secret, expired token, replay rejection.
Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is
intentionally empty. berth_pdf_versions and brochure_versions land in
Phase 6b and join the list there. Existing s3_key columns: only
gdpr_export_jobs.storage_key, already named correctly — no rename needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
|
|
|
|
|
|
|
|
export async function GET(
|
|
|
|
|
_req: NextRequest,
|
|
|
|
|
ctx: { params: Promise<{ token: string }> },
|
|
|
|
|
): Promise<NextResponse> {
|
|
|
|
|
const { token } = await ctx.params;
|
|
|
|
|
|
|
|
|
|
const backend = await getStorageBackend();
|
|
|
|
|
if (!(backend instanceof FilesystemBackend)) {
|
|
|
|
|
return NextResponse.json(
|
|
|
|
|
{ error: 'Storage proxy is only available in filesystem mode' },
|
|
|
|
|
{ status: 404 },
|
|
|
|
|
);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
const result = verifyProxyToken(token, backend.getHmacSecret());
|
|
|
|
|
if (!result.ok) {
|
|
|
|
|
logger.warn({ reason: result.reason }, 'Storage proxy token rejected');
|
|
|
|
|
return NextResponse.json({ error: 'Invalid or expired token' }, { status: 403 });
|
|
|
|
|
}
|
|
|
|
|
const { payload } = result;
|
|
|
|
|
|
2026-05-05 04:07:03 +02:00
|
|
|
// Single-use enforcement. SET NX with a TTL pinned to the token's own
|
|
|
|
|
// expiry so the dedup window never closes before the token does. Using
|
|
|
|
|
// the body half of the token as the dedup key (signature included
|
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI
Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays
the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs,
brochures) without touching those domains yet.
New files:
- src/lib/storage/index.ts StorageBackend interface + per-process
factory keyed on system_settings.
- src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/
Wasabi/Tigris) wrapping the existing minio
JS client. Includes a healthCheck() used
by the admin "Test connection" button.
- src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a
mitigations baked in.
- src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock,
per-row resumable progress markers,
sha256 round-trip verification, atomic
storage_backend flip on success.
- scripts/migrate-storage.ts Thin CLI shim around runMigration().
- src/app/api/storage/[token]/route.ts
Filesystem proxy GET. Verifies HMAC,
enforces single-use replay protection
via Redis SET NX, streams via NextResponse
ReadableStream with explicit Content-Type
+ Content-Disposition. Node runtime only.
- src/app/api/v1/admin/storage/route.ts
GET status + POST connection test.
- src/app/api/v1/admin/storage/migrate/route.ts
Super-admin-only POST that runs the
exact same runMigration() as the CLI.
- src/app/(dashboard)/[portSlug]/admin/storage/page.tsx
Super-admin admin UI (current backend,
capacity stats, switch button with
dry-run, test connection, backup hint).
- src/components/admin/storage-admin-panel.tsx
Client component for the page above.
§14.9a critical mitigations implemented:
- Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$;
`..`, `.`, `//`, leading `/`, and overlength keys rejected.
- Realpath: storage root realpath'd at create time, every per-key
resolution checked against the realpath'd prefix.
- Storage root created (or chmod'd) to 0o700.
- Multi-node refusal: FilesystemBackend.create() throws when
MULTI_NODE_DEPLOYMENT=true.
- HMAC token: sha256-HMAC over the (key, expiry, nonce, filename,
content-type) payload. Verified with timingSafeEqual; bad sig,
expired, or invalid-key payloads all return 403.
- Single-use replay: token body cached in Redis SET NX EX 1800s.
- sha256 round-trip: copyAndVerify() re-fetches from the target after
put() and aborts the migration on any mismatch.
- Free-disk pre-flight: when migrating to filesystem, sums byte counts
via source.head() and aborts if free space < total * 1.2.
- pg_advisory_lock(0xc7000a01) prevents concurrent migrations.
- Resumable: per-row progress markers in _storage_migration_progress.
system_settings keys read by the factory (jsonb, no schema change):
storage_backend, storage_s3_endpoint, storage_s3_region,
storage_s3_bucket, storage_s3_access_key,
storage_s3_secret_key_encrypted, storage_s3_force_path_style,
storage_filesystem_root, storage_proxy_hmac_secret_encrypted.
Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage`
(./storage added to .gitignore).
Tests added (34 tests, all green):
- tests/unit/storage/filesystem-backend.test.ts — key validation
allow/reject matrix, realpath escape, 0o700 perms, multi-node
refusal, HMAC token sign/verify/tamper/expire/invalid-key.
- tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on
round-trip aborts the migration.
- tests/integration/storage/proxy-route.test.ts — happy path, wrong
HMAC secret, expired token, replay rejection.
Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is
intentionally empty. berth_pdf_versions and brochure_versions land in
Phase 6b and join the list there. Existing s3_key columns: only
gdpr_export_jobs.storage_key, already named correctly — no rename needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
|
|
|
// would also work but body is enough — a reused token has the same body).
|
|
|
|
|
const replayKey = `storage:proxy:seen:${token.split('.')[0]}`;
|
2026-05-05 04:07:03 +02:00
|
|
|
const remainingSeconds = Math.max(
|
|
|
|
|
REPLAY_TTL_FLOOR_SECONDS,
|
|
|
|
|
Math.min(REPLAY_TTL_CEILING_SECONDS, payload.e - Math.floor(Date.now() / 1000) + 60),
|
|
|
|
|
);
|
|
|
|
|
const setOk = await redis.set(replayKey, '1', 'EX', remainingSeconds, 'NX');
|
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI
Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays
the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs,
brochures) without touching those domains yet.
New files:
- src/lib/storage/index.ts StorageBackend interface + per-process
factory keyed on system_settings.
- src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/
Wasabi/Tigris) wrapping the existing minio
JS client. Includes a healthCheck() used
by the admin "Test connection" button.
- src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a
mitigations baked in.
- src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock,
per-row resumable progress markers,
sha256 round-trip verification, atomic
storage_backend flip on success.
- scripts/migrate-storage.ts Thin CLI shim around runMigration().
- src/app/api/storage/[token]/route.ts
Filesystem proxy GET. Verifies HMAC,
enforces single-use replay protection
via Redis SET NX, streams via NextResponse
ReadableStream with explicit Content-Type
+ Content-Disposition. Node runtime only.
- src/app/api/v1/admin/storage/route.ts
GET status + POST connection test.
- src/app/api/v1/admin/storage/migrate/route.ts
Super-admin-only POST that runs the
exact same runMigration() as the CLI.
- src/app/(dashboard)/[portSlug]/admin/storage/page.tsx
Super-admin admin UI (current backend,
capacity stats, switch button with
dry-run, test connection, backup hint).
- src/components/admin/storage-admin-panel.tsx
Client component for the page above.
§14.9a critical mitigations implemented:
- Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$;
`..`, `.`, `//`, leading `/`, and overlength keys rejected.
- Realpath: storage root realpath'd at create time, every per-key
resolution checked against the realpath'd prefix.
- Storage root created (or chmod'd) to 0o700.
- Multi-node refusal: FilesystemBackend.create() throws when
MULTI_NODE_DEPLOYMENT=true.
- HMAC token: sha256-HMAC over the (key, expiry, nonce, filename,
content-type) payload. Verified with timingSafeEqual; bad sig,
expired, or invalid-key payloads all return 403.
- Single-use replay: token body cached in Redis SET NX EX 1800s.
- sha256 round-trip: copyAndVerify() re-fetches from the target after
put() and aborts the migration on any mismatch.
- Free-disk pre-flight: when migrating to filesystem, sums byte counts
via source.head() and aborts if free space < total * 1.2.
- pg_advisory_lock(0xc7000a01) prevents concurrent migrations.
- Resumable: per-row progress markers in _storage_migration_progress.
system_settings keys read by the factory (jsonb, no schema change):
storage_backend, storage_s3_endpoint, storage_s3_region,
storage_s3_bucket, storage_s3_access_key,
storage_s3_secret_key_encrypted, storage_s3_force_path_style,
storage_filesystem_root, storage_proxy_hmac_secret_encrypted.
Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage`
(./storage added to .gitignore).
Tests added (34 tests, all green):
- tests/unit/storage/filesystem-backend.test.ts — key validation
allow/reject matrix, realpath escape, 0o700 perms, multi-node
refusal, HMAC token sign/verify/tamper/expire/invalid-key.
- tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on
round-trip aborts the migration.
- tests/integration/storage/proxy-route.test.ts — happy path, wrong
HMAC secret, expired token, replay rejection.
Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is
intentionally empty. berth_pdf_versions and brochure_versions land in
Phase 6b and join the list there. Existing s3_key columns: only
gdpr_export_jobs.storage_key, already named correctly — no rename needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
|
|
|
if (setOk !== 'OK') {
|
|
|
|
|
logger.warn({ key: payload.k }, 'Storage proxy token replay rejected');
|
|
|
|
|
return NextResponse.json({ error: 'Token already used' }, { status: 403 });
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
let absolutePath: string;
|
|
|
|
|
try {
|
|
|
|
|
absolutePath = backend.resolveKeyForProxy(payload.k);
|
|
|
|
|
} catch (err) {
|
|
|
|
|
logger.warn({ err, key: payload.k }, 'Storage proxy key resolution failed');
|
|
|
|
|
return NextResponse.json({ error: 'Invalid key' }, { status: 400 });
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
let size: number;
|
|
|
|
|
try {
|
|
|
|
|
const stat = await fs.stat(absolutePath);
|
|
|
|
|
if (!stat.isFile()) {
|
|
|
|
|
return NextResponse.json({ error: 'Not found' }, { status: 404 });
|
|
|
|
|
}
|
|
|
|
|
size = stat.size;
|
|
|
|
|
} catch (err) {
|
|
|
|
|
const code = (err as NodeJS.ErrnoException).code;
|
|
|
|
|
if (code === 'ENOENT') {
|
|
|
|
|
return NextResponse.json({ error: 'Not found' }, { status: 404 });
|
|
|
|
|
}
|
|
|
|
|
throw err;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Convert the Node Readable into a Web ReadableStream for NextResponse.
|
|
|
|
|
const nodeStream = createReadStream(absolutePath);
|
|
|
|
|
const webStream = Readable.toWeb(nodeStream) as unknown as ReadableStream<Uint8Array>;
|
|
|
|
|
|
|
|
|
|
const headers = new Headers();
|
|
|
|
|
headers.set('Content-Type', payload.c ?? 'application/octet-stream');
|
|
|
|
|
headers.set('Content-Length', String(size));
|
|
|
|
|
if (payload.f) {
|
|
|
|
|
// RFC 5987 — quote the filename and provide a UTF-8 fallback.
|
|
|
|
|
const safe = payload.f.replace(/"/g, '');
|
|
|
|
|
headers.set(
|
|
|
|
|
'Content-Disposition',
|
|
|
|
|
`attachment; filename="${safe}"; filename*=UTF-8''${encodeURIComponent(payload.f)}`,
|
|
|
|
|
);
|
|
|
|
|
}
|
|
|
|
|
headers.set('Cache-Control', 'private, no-store');
|
|
|
|
|
headers.set('X-Content-Type-Options', 'nosniff');
|
|
|
|
|
|
|
|
|
|
return new NextResponse(webStream, { status: 200, headers });
|
|
|
|
|
}
|