feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI
Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays
the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs,
brochures) without touching those domains yet.
New files:
- src/lib/storage/index.ts StorageBackend interface + per-process
factory keyed on system_settings.
- src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/
Wasabi/Tigris) wrapping the existing minio
JS client. Includes a healthCheck() used
by the admin "Test connection" button.
- src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a
mitigations baked in.
- src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock,
per-row resumable progress markers,
sha256 round-trip verification, atomic
storage_backend flip on success.
- scripts/migrate-storage.ts Thin CLI shim around runMigration().
- src/app/api/storage/[token]/route.ts
Filesystem proxy GET. Verifies HMAC,
enforces single-use replay protection
via Redis SET NX, streams via NextResponse
ReadableStream with explicit Content-Type
+ Content-Disposition. Node runtime only.
- src/app/api/v1/admin/storage/route.ts
GET status + POST connection test.
- src/app/api/v1/admin/storage/migrate/route.ts
Super-admin-only POST that runs the
exact same runMigration() as the CLI.
- src/app/(dashboard)/[portSlug]/admin/storage/page.tsx
Super-admin admin UI (current backend,
capacity stats, switch button with
dry-run, test connection, backup hint).
- src/components/admin/storage-admin-panel.tsx
Client component for the page above.
§14.9a critical mitigations implemented:
- Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$;
`..`, `.`, `//`, leading `/`, and overlength keys rejected.
- Realpath: storage root realpath'd at create time, every per-key
resolution checked against the realpath'd prefix.
- Storage root created (or chmod'd) to 0o700.
- Multi-node refusal: FilesystemBackend.create() throws when
MULTI_NODE_DEPLOYMENT=true.
- HMAC token: sha256-HMAC over the (key, expiry, nonce, filename,
content-type) payload. Verified with timingSafeEqual; bad sig,
expired, or invalid-key payloads all return 403.
- Single-use replay: token body cached in Redis SET NX EX 1800s.
- sha256 round-trip: copyAndVerify() re-fetches from the target after
put() and aborts the migration on any mismatch.
- Free-disk pre-flight: when migrating to filesystem, sums byte counts
via source.head() and aborts if free space < total * 1.2.
- pg_advisory_lock(0xc7000a01) prevents concurrent migrations.
- Resumable: per-row progress markers in _storage_migration_progress.
system_settings keys read by the factory (jsonb, no schema change):
storage_backend, storage_s3_endpoint, storage_s3_region,
storage_s3_bucket, storage_s3_access_key,
storage_s3_secret_key_encrypted, storage_s3_force_path_style,
storage_filesystem_root, storage_proxy_hmac_secret_encrypted.
Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage`
(./storage added to .gitignore).
Tests added (34 tests, all green):
- tests/unit/storage/filesystem-backend.test.ts — key validation
allow/reject matrix, realpath escape, 0o700 perms, multi-node
refusal, HMAC token sign/verify/tamper/expire/invalid-key.
- tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on
round-trip aborts the migration.
- tests/integration/storage/proxy-route.test.ts — happy path, wrong
HMAC secret, expired token, replay rejection.
Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is
intentionally empty. berth_pdf_versions and brochure_versions land in
Phase 6b and join the list there. Existing s3_key columns: only
gdpr_export_jobs.storage_key, already named correctly — no rename needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
|
|
|
/**
|
|
|
|
|
* Pluggable storage backend (Phase 6a — see docs/berth-recommender-and-pdf-plan.md §4.7a).
|
|
|
|
|
*
|
|
|
|
|
* The CRM stores files (per-berth PDFs, brochures, GDPR exports, etc.) through a
|
|
|
|
|
* single `StorageBackend` abstraction. The deployment chooses between an
|
|
|
|
|
* S3-compatible store (MinIO / AWS S3 / Backblaze B2 / Cloudflare R2 / Wasabi /
|
|
|
|
|
* Tigris) and a local filesystem at runtime via `system_settings.storage_backend`.
|
|
|
|
|
*
|
|
|
|
|
* Callers should always import from this barrel — never from `s3.ts` or
|
|
|
|
|
* `filesystem.ts` directly — so the factory wiring stays the single source of
|
|
|
|
|
* truth.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
import { and, eq, isNull } from 'drizzle-orm';
|
|
|
|
|
|
|
|
|
|
import { db } from '@/lib/db';
|
|
|
|
|
import { systemSettings } from '@/lib/db/schema/system';
|
|
|
|
|
import { logger } from '@/lib/logger';
|
|
|
|
|
|
|
|
|
|
import { FilesystemBackend } from './filesystem';
|
|
|
|
|
import { S3Backend } from './s3';
|
|
|
|
|
|
|
|
|
|
export type StorageBackendName = 's3' | 'filesystem';
|
|
|
|
|
|
|
|
|
|
export interface PutOpts {
|
|
|
|
|
contentType: string;
|
|
|
|
|
/** Optional pre-computed sha256 hex — if absent, the backend computes one. */
|
|
|
|
|
sha256?: string;
|
|
|
|
|
/** Bytes (for streams that don't expose .length); used for capacity pre-flight. */
|
|
|
|
|
sizeBytes?: number;
|
|
|
|
|
/** Optional content-disposition for downloads (filesystem proxy only). */
|
|
|
|
|
contentDisposition?: string;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
export interface PresignOpts {
|
|
|
|
|
/** TTL in seconds. Default 900 (15min) per SECURITY-GUIDELINES §7.1. */
|
|
|
|
|
expirySeconds?: number;
|
|
|
|
|
contentType?: string;
|
|
|
|
|
/** Filename used in Content-Disposition for downloads. */
|
|
|
|
|
filename?: string;
|
fix(audit): non-Documenso backlog sweep — port-binding, NULLS NOT DISTINCT, custom merge tokens, company docs
Wave through the remaining audit-final-deferred items that aren't blocked
on the back-burnered Documenso work.
Multi-tenant isolation:
- Storage proxy ProxyTokenPayload gains optional `p` (port slug) claim;
verifier asserts `key.startsWith(${p}/)`. Defense-in-depth against a
buggy issuer in some future code path that mixes port scopes — every
storage key generated by generateStorageKey() already prefixes the
slug. document-sends opts in for 24h emailed download links; other
callers continue working unchanged via the optional field.
DB schema reconciliation:
- Migration 0047 rebuilds system_settings unique index with NULLS NOT
DISTINCT (Postgres 15+) so global settings (port_id IS NULL) are
uniquely keyed by `key` alone. Surfaced + dedupe'd 65 duplicate
(storage_backend, NULL) rows that had accumulated from race-prone
delete-then-insert patterns in ocr-config / settings / residential-
stages / ai-budget services. All four services converted to true
onConflictDoUpdate upserts so the race window is closed.
API uniformity:
- Response shape standardization: 16 routes converted from
`{ success: true }` to 204 No Content. CLAUDE.md documents the
convention (`{ data: <T> }` for content, 204 for empty mutations,
portal-auth retains `{ success: true }` for the frontend's auth chain).
- req.json() → parseBody() migration across 9 admin/CRM routes
(custom-fields, expenses/export ×3, currency convert,
search/recently-viewed, admin/duplicates, berths/pdf-{upload-url,
versions, parse-results}). Uniform 400 error shapes for
ZodError-flagged bodies.
Custom-fields merge tokens (shipped end-to-end):
- merge-fields.ts gains CUSTOM_MERGE_TOKEN_RE + helpers for the
`{{custom.<fieldName>}}` shape.
- document-templates validator accepts the dynamic shape alongside
the static catalog tokens.
- document-sends.service mergeCustomFieldValues resolver fetches
per-port custom_field_definitions for client/interest/berth contexts
and substitutes stored values keyed by `{{custom.fieldName}}`.
- custom-fields-manager amber banner updated to reflect that merge
tokens now expand (search index + entity-diff remain documented
design limitations).
/api/v1/files cross-entity filtering:
- Validator + listFiles + uploadFile accept companyId AND yachtId
alongside clientId. file-upload-zone propagates both.
- New CompanyFilesTab component mirrors ClientFilesTab; restored as a
visible Documents tab in company-tabs.tsx (was a hidden stub).
Inline TODOs:
- Reviewed remaining two TODOs (per-user reminder schedule, import
worker handlers). Both are placeholders for future feature surfaces,
not bugs — per-port digest works for every customer; nothing
currently enqueues import jobs (verified). Annotated in BACKLOG.
BACKLOG.md updated to reflect what landed and what's still pending
(Documenso-related items still bundled with the back-burnered phases).
Tests: 1185/1185 vitest, tsc clean.
2026-05-08 02:20:27 +02:00
|
|
|
/**
|
|
|
|
|
* Optional port slug to bind the token to. The filesystem proxy
|
|
|
|
|
* verifier asserts the storage key starts with `${portSlug}/` when
|
|
|
|
|
* present. S3 backend ignores this field (presigned S3 URLs carry
|
|
|
|
|
* their own signature scope). Pass it whenever the issuer is in a
|
|
|
|
|
* port-scoped request — `generateStorageKey()` already prefixes the
|
|
|
|
|
* slug, so this is the matching enforcement.
|
|
|
|
|
*/
|
|
|
|
|
portSlug?: string;
|
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI
Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays
the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs,
brochures) without touching those domains yet.
New files:
- src/lib/storage/index.ts StorageBackend interface + per-process
factory keyed on system_settings.
- src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/
Wasabi/Tigris) wrapping the existing minio
JS client. Includes a healthCheck() used
by the admin "Test connection" button.
- src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a
mitigations baked in.
- src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock,
per-row resumable progress markers,
sha256 round-trip verification, atomic
storage_backend flip on success.
- scripts/migrate-storage.ts Thin CLI shim around runMigration().
- src/app/api/storage/[token]/route.ts
Filesystem proxy GET. Verifies HMAC,
enforces single-use replay protection
via Redis SET NX, streams via NextResponse
ReadableStream with explicit Content-Type
+ Content-Disposition. Node runtime only.
- src/app/api/v1/admin/storage/route.ts
GET status + POST connection test.
- src/app/api/v1/admin/storage/migrate/route.ts
Super-admin-only POST that runs the
exact same runMigration() as the CLI.
- src/app/(dashboard)/[portSlug]/admin/storage/page.tsx
Super-admin admin UI (current backend,
capacity stats, switch button with
dry-run, test connection, backup hint).
- src/components/admin/storage-admin-panel.tsx
Client component for the page above.
§14.9a critical mitigations implemented:
- Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$;
`..`, `.`, `//`, leading `/`, and overlength keys rejected.
- Realpath: storage root realpath'd at create time, every per-key
resolution checked against the realpath'd prefix.
- Storage root created (or chmod'd) to 0o700.
- Multi-node refusal: FilesystemBackend.create() throws when
MULTI_NODE_DEPLOYMENT=true.
- HMAC token: sha256-HMAC over the (key, expiry, nonce, filename,
content-type) payload. Verified with timingSafeEqual; bad sig,
expired, or invalid-key payloads all return 403.
- Single-use replay: token body cached in Redis SET NX EX 1800s.
- sha256 round-trip: copyAndVerify() re-fetches from the target after
put() and aborts the migration on any mismatch.
- Free-disk pre-flight: when migrating to filesystem, sums byte counts
via source.head() and aborts if free space < total * 1.2.
- pg_advisory_lock(0xc7000a01) prevents concurrent migrations.
- Resumable: per-row progress markers in _storage_migration_progress.
system_settings keys read by the factory (jsonb, no schema change):
storage_backend, storage_s3_endpoint, storage_s3_region,
storage_s3_bucket, storage_s3_access_key,
storage_s3_secret_key_encrypted, storage_s3_force_path_style,
storage_filesystem_root, storage_proxy_hmac_secret_encrypted.
Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage`
(./storage added to .gitignore).
Tests added (34 tests, all green):
- tests/unit/storage/filesystem-backend.test.ts — key validation
allow/reject matrix, realpath escape, 0o700 perms, multi-node
refusal, HMAC token sign/verify/tamper/expire/invalid-key.
- tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on
round-trip aborts the migration.
- tests/integration/storage/proxy-route.test.ts — happy path, wrong
HMAC secret, expired token, replay rejection.
Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is
intentionally empty. berth_pdf_versions and brochure_versions land in
Phase 6b and join the list there. Existing s3_key columns: only
gdpr_export_jobs.storage_key, already named correctly — no rename needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
export interface StorageBackend {
|
|
|
|
|
/** Upload bytes. Returns the canonical key, size, and sha256 hex. */
|
|
|
|
|
put(
|
|
|
|
|
key: string,
|
|
|
|
|
body: Buffer | NodeJS.ReadableStream,
|
|
|
|
|
opts: PutOpts,
|
|
|
|
|
): Promise<{ key: string; sizeBytes: number; sha256: string }>;
|
|
|
|
|
|
|
|
|
|
/** Stream a file out. Throws NotFoundError if missing. */
|
|
|
|
|
get(key: string): Promise<NodeJS.ReadableStream>;
|
|
|
|
|
|
|
|
|
|
/** Existence + size check without reading the full body. Returns null when missing. */
|
|
|
|
|
head(key: string): Promise<{ sizeBytes: number; contentType: string } | null>;
|
|
|
|
|
|
|
|
|
|
/** Delete. Idempotent — missing keys must not throw. */
|
|
|
|
|
delete(key: string): Promise<void>;
|
|
|
|
|
|
|
|
|
|
/** Generate a short-lived URL the browser can PUT to. */
|
|
|
|
|
presignUpload(key: string, opts: PresignOpts): Promise<{ url: string; method: 'PUT' | 'POST' }>;
|
|
|
|
|
|
|
|
|
|
/** Generate a short-lived URL the browser can GET from. */
|
|
|
|
|
presignDownload(key: string, opts: PresignOpts): Promise<{ url: string; expiresAt: Date }>;
|
|
|
|
|
|
|
|
|
|
readonly name: StorageBackendName;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// ─── factory ────────────────────────────────────────────────────────────────
|
|
|
|
|
|
|
|
|
|
interface CachedFactory {
|
|
|
|
|
backend: StorageBackend;
|
|
|
|
|
/** Resolved at cache-time so we can re-fetch when settings change. */
|
|
|
|
|
configFingerprint: string;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
let cached: CachedFactory | null = null;
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Reset the per-process backend cache. Called after `system_settings` writes
|
|
|
|
|
* via the existing pub/sub invalidation hook, and exposed for tests.
|
|
|
|
|
*/
|
|
|
|
|
export function resetStorageBackendCache(): void {
|
|
|
|
|
cached = null;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
interface StorageConfigSnapshot {
|
|
|
|
|
backend: StorageBackendName;
|
|
|
|
|
s3?: {
|
|
|
|
|
endpoint?: string;
|
|
|
|
|
region?: string;
|
|
|
|
|
bucket?: string;
|
|
|
|
|
accessKey?: string;
|
|
|
|
|
secretKeyEncrypted?: string;
|
|
|
|
|
forcePathStyle?: boolean;
|
|
|
|
|
};
|
|
|
|
|
filesystem?: {
|
|
|
|
|
root?: string;
|
|
|
|
|
proxyHmacSecretEncrypted?: string;
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
async function readGlobalSetting<T = unknown>(key: string): Promise<T | null> {
|
|
|
|
|
const [row] = await db
|
|
|
|
|
.select()
|
|
|
|
|
.from(systemSettings)
|
|
|
|
|
.where(and(eq(systemSettings.key, key), isNull(systemSettings.portId)));
|
|
|
|
|
return (row?.value as T | undefined) ?? null;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
async function loadStorageConfig(): Promise<StorageConfigSnapshot> {
|
|
|
|
|
// Each setting key is a separate row. We read them in parallel.
|
|
|
|
|
const keys = [
|
|
|
|
|
'storage_backend',
|
|
|
|
|
'storage_s3_endpoint',
|
|
|
|
|
'storage_s3_region',
|
|
|
|
|
'storage_s3_bucket',
|
|
|
|
|
'storage_s3_access_key',
|
|
|
|
|
'storage_s3_secret_key_encrypted',
|
|
|
|
|
'storage_s3_force_path_style',
|
|
|
|
|
'storage_filesystem_root',
|
|
|
|
|
'storage_proxy_hmac_secret_encrypted',
|
|
|
|
|
] as const;
|
|
|
|
|
const [
|
|
|
|
|
backendRaw,
|
|
|
|
|
s3Endpoint,
|
|
|
|
|
s3Region,
|
|
|
|
|
s3Bucket,
|
|
|
|
|
s3AccessKey,
|
|
|
|
|
s3SecretKeyEncrypted,
|
|
|
|
|
s3ForcePathStyle,
|
|
|
|
|
fsRoot,
|
|
|
|
|
fsHmacSecretEncrypted,
|
|
|
|
|
] = await Promise.all(keys.map((k) => readGlobalSetting<unknown>(k)));
|
|
|
|
|
|
|
|
|
|
const backend: StorageBackendName = backendRaw === 'filesystem' ? 'filesystem' : 's3';
|
|
|
|
|
|
|
|
|
|
return {
|
|
|
|
|
backend,
|
|
|
|
|
s3: {
|
|
|
|
|
endpoint: typeof s3Endpoint === 'string' ? s3Endpoint : undefined,
|
|
|
|
|
region: typeof s3Region === 'string' ? s3Region : undefined,
|
|
|
|
|
bucket: typeof s3Bucket === 'string' ? s3Bucket : undefined,
|
|
|
|
|
accessKey: typeof s3AccessKey === 'string' ? s3AccessKey : undefined,
|
|
|
|
|
secretKeyEncrypted:
|
|
|
|
|
typeof s3SecretKeyEncrypted === 'string' ? s3SecretKeyEncrypted : undefined,
|
|
|
|
|
forcePathStyle:
|
|
|
|
|
typeof s3ForcePathStyle === 'boolean' ? s3ForcePathStyle : Boolean(s3ForcePathStyle),
|
|
|
|
|
},
|
|
|
|
|
filesystem: {
|
|
|
|
|
root: typeof fsRoot === 'string' ? fsRoot : undefined,
|
|
|
|
|
proxyHmacSecretEncrypted:
|
|
|
|
|
typeof fsHmacSecretEncrypted === 'string' ? fsHmacSecretEncrypted : undefined,
|
|
|
|
|
},
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
|
fix(audit): backlog sweep — partial archived indexes, custom-fields per-entity gate, polish
Wave through the 2026-05-07 backlog of small/concrete audit-final-deferred
items (deferring the Documenso Phases 2-7 build and items needing design
decisions or live external instances).
DB schema:
- Migration 0046 converts 5 composite (port_id, archived_at) indexes to
partial WHERE archived_at IS NULL — clients, interests, yachts, and
both residential tables. Smaller, faster planner choice for the
dominant list-query shape.
Multi-tenant isolation:
- document_sends now verifies recipient.interestId belongs to the port
before landing on the audit row (the surrounding clientId check was
already port-scoped; interestId pollution was the gap).
Routes / API:
- /api/v1/custom-fields/[entityId] requires entityType query param and
gates on the matching resource permission (clients/interests/berths/
yachts/companies). Fixes the cross-resource gap where a user with
clients.view could read company custom-field values.
- Admin user list trash button wrapped in PermissionGate (edit was
already gated; remove was not).
Service polish:
- berth-recommender accepts string-shaped JSONB booleans
('true'/'false') so admin UIs that wrap values as strings don't
silently fall through to defaults.
- expense-pdf renderReceiptHeader anchors all text positions to a
captured baseY rather than reading mutating doc.y after rect+stroke.
Headers no longer drift on the first receipt page after a soft page
break.
- berth-pdf apply: collect non-finite numeric coercion drops + warn-log
them so partial silent drops are observable (was invisible because
the no-fields-supplied check only fires when ALL drop).
- Storage cache fingerprint comment documenting the encrypted-secret
invariant + the explicit invalidation hook.
UI polish:
- invoice-detail typed: replaced two `any` casts with a proper
InvoiceDetailData / LineItem / LinkedExpense interface set.
- YachtForm now accepts initialOwner prop. Wired through:
- client-yachts-tab passes { type: 'client', id: clientId }
- interest-form passes { type: 'client', id: selectedClientId }
- Interest-form yacht picker now includes company-owned yachts where
the selected client is a member (fetches client.companies and feeds
YachtPicker an array filter). Plus an inline "Add new" button that
opens YachtForm pre-bound to the client.
- YachtPicker accepts ownerFilter as single OR array for "match any"
semantics.
BACKLOG.md updated with what landed vs what's still deferred (and why
each deferred item is genuinely larger than this push warrants).
Tests: 1185/1185 vitest, tsc clean.
2026-05-07 21:45:42 +02:00
|
|
|
/**
|
|
|
|
|
* The fingerprint includes encrypted-secret material because rotating the
|
|
|
|
|
* secret should invalidate the cached client. After a key rotation the
|
|
|
|
|
* settings-write hook calls `resetStorageBackendCache()` explicitly, so
|
|
|
|
|
* this comparison is a defense-in-depth backstop rather than the primary
|
|
|
|
|
* invalidation path. If you ever change `loadStorageConfig` to read
|
|
|
|
|
* additional sensitive material, make sure the rotation flow keeps
|
|
|
|
|
* resetting the cache — relying on fingerprint diff alone means the old
|
|
|
|
|
* client is held in memory until the next mismatch.
|
|
|
|
|
*/
|
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI
Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays
the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs,
brochures) without touching those domains yet.
New files:
- src/lib/storage/index.ts StorageBackend interface + per-process
factory keyed on system_settings.
- src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/
Wasabi/Tigris) wrapping the existing minio
JS client. Includes a healthCheck() used
by the admin "Test connection" button.
- src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a
mitigations baked in.
- src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock,
per-row resumable progress markers,
sha256 round-trip verification, atomic
storage_backend flip on success.
- scripts/migrate-storage.ts Thin CLI shim around runMigration().
- src/app/api/storage/[token]/route.ts
Filesystem proxy GET. Verifies HMAC,
enforces single-use replay protection
via Redis SET NX, streams via NextResponse
ReadableStream with explicit Content-Type
+ Content-Disposition. Node runtime only.
- src/app/api/v1/admin/storage/route.ts
GET status + POST connection test.
- src/app/api/v1/admin/storage/migrate/route.ts
Super-admin-only POST that runs the
exact same runMigration() as the CLI.
- src/app/(dashboard)/[portSlug]/admin/storage/page.tsx
Super-admin admin UI (current backend,
capacity stats, switch button with
dry-run, test connection, backup hint).
- src/components/admin/storage-admin-panel.tsx
Client component for the page above.
§14.9a critical mitigations implemented:
- Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$;
`..`, `.`, `//`, leading `/`, and overlength keys rejected.
- Realpath: storage root realpath'd at create time, every per-key
resolution checked against the realpath'd prefix.
- Storage root created (or chmod'd) to 0o700.
- Multi-node refusal: FilesystemBackend.create() throws when
MULTI_NODE_DEPLOYMENT=true.
- HMAC token: sha256-HMAC over the (key, expiry, nonce, filename,
content-type) payload. Verified with timingSafeEqual; bad sig,
expired, or invalid-key payloads all return 403.
- Single-use replay: token body cached in Redis SET NX EX 1800s.
- sha256 round-trip: copyAndVerify() re-fetches from the target after
put() and aborts the migration on any mismatch.
- Free-disk pre-flight: when migrating to filesystem, sums byte counts
via source.head() and aborts if free space < total * 1.2.
- pg_advisory_lock(0xc7000a01) prevents concurrent migrations.
- Resumable: per-row progress markers in _storage_migration_progress.
system_settings keys read by the factory (jsonb, no schema change):
storage_backend, storage_s3_endpoint, storage_s3_region,
storage_s3_bucket, storage_s3_access_key,
storage_s3_secret_key_encrypted, storage_s3_force_path_style,
storage_filesystem_root, storage_proxy_hmac_secret_encrypted.
Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage`
(./storage added to .gitignore).
Tests added (34 tests, all green):
- tests/unit/storage/filesystem-backend.test.ts — key validation
allow/reject matrix, realpath escape, 0o700 perms, multi-node
refusal, HMAC token sign/verify/tamper/expire/invalid-key.
- tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on
round-trip aborts the migration.
- tests/integration/storage/proxy-route.test.ts — happy path, wrong
HMAC secret, expired token, replay rejection.
Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is
intentionally empty. berth_pdf_versions and brochure_versions land in
Phase 6b and join the list there. Existing s3_key columns: only
gdpr_export_jobs.storage_key, already named correctly — no rename needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
|
|
|
function fingerprint(cfg: StorageConfigSnapshot): string {
|
|
|
|
|
return JSON.stringify(cfg);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Resolve the active backend. Caches per-process; the cache is invalidated by
|
|
|
|
|
* `resetStorageBackendCache()` (called when `system_settings.storage_backend`
|
|
|
|
|
* changes via the migration flow).
|
|
|
|
|
*/
|
|
|
|
|
export async function getStorageBackend(): Promise<StorageBackend> {
|
|
|
|
|
const cfg = await loadStorageConfig();
|
|
|
|
|
const fp = fingerprint(cfg);
|
|
|
|
|
if (cached && cached.configFingerprint === fp) {
|
|
|
|
|
return cached.backend;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
const backend = await buildBackend(cfg);
|
|
|
|
|
cached = { backend, configFingerprint: fp };
|
|
|
|
|
logger.info({ backend: backend.name }, 'Storage backend resolved');
|
|
|
|
|
return backend;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
async function buildBackend(cfg: StorageConfigSnapshot): Promise<StorageBackend> {
|
|
|
|
|
if (cfg.backend === 'filesystem') {
|
|
|
|
|
return FilesystemBackend.create({
|
|
|
|
|
root: cfg.filesystem?.root ?? './storage',
|
|
|
|
|
proxyHmacSecretEncrypted: cfg.filesystem?.proxyHmacSecretEncrypted ?? null,
|
|
|
|
|
});
|
|
|
|
|
}
|
|
|
|
|
return S3Backend.create({
|
|
|
|
|
endpoint: cfg.s3?.endpoint,
|
|
|
|
|
region: cfg.s3?.region,
|
|
|
|
|
bucket: cfg.s3?.bucket,
|
|
|
|
|
accessKey: cfg.s3?.accessKey,
|
|
|
|
|
secretKeyEncrypted: cfg.s3?.secretKeyEncrypted,
|
|
|
|
|
forcePathStyle: cfg.s3?.forcePathStyle,
|
|
|
|
|
});
|
|
|
|
|
}
|
|
|
|
|
|
fix(storage): route every file op through getStorageBackend()
Removes 12 direct minioClient.{put,get,remove}Object call sites that
bypassed the pluggable storage abstraction. Filesystem-mode deploys
(MULTI_NODE_DEPLOYMENT=false, storage_backend=filesystem) silently
broke at every site: GDPR export, invoice PDF, EOI generation, portal
download, file upload, folder create/rename/delete, signed PDF land,
maintenance cleanup, etc. Each site now resolves the active backend
and uses its put/get/delete + the new presignDownloadUrl() helper.
Folder marker objects in /files/folders/* keep the same on-the-wire
shape but route through the backend. A future refactor should move
folder bookkeeping to a DB-backed virtual-folder table (see audit
HIGH §3 follow-up note in the route file).
Sites left untouched: src/lib/services/system-monitoring.service.ts
and src/app/api/ready/route.ts use minioClient.bucketExists as an S3-
specific health probe — those are correctly mode-aware and stay.
Refs: docs/audit-comprehensive-2026-05-05.md HIGH §3 (auditor-D Issue 1)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 18:41:02 +02:00
|
|
|
// ─── url helpers ────────────────────────────────────────────────────────────
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Convenience wrapper that returns just the presigned download URL — the most
|
|
|
|
|
* common need at call sites that don't track expiry. Mirrors the legacy
|
|
|
|
|
* `getPresignedUrl(key)` helper in `@/lib/minio` but routes through the
|
|
|
|
|
* active backend so filesystem-mode deployments work too.
|
|
|
|
|
*/
|
|
|
|
|
export async function presignDownloadUrl(
|
|
|
|
|
key: string,
|
|
|
|
|
expirySeconds = 900,
|
|
|
|
|
filename?: string,
|
fix(audit): non-Documenso backlog sweep — port-binding, NULLS NOT DISTINCT, custom merge tokens, company docs
Wave through the remaining audit-final-deferred items that aren't blocked
on the back-burnered Documenso work.
Multi-tenant isolation:
- Storage proxy ProxyTokenPayload gains optional `p` (port slug) claim;
verifier asserts `key.startsWith(${p}/)`. Defense-in-depth against a
buggy issuer in some future code path that mixes port scopes — every
storage key generated by generateStorageKey() already prefixes the
slug. document-sends opts in for 24h emailed download links; other
callers continue working unchanged via the optional field.
DB schema reconciliation:
- Migration 0047 rebuilds system_settings unique index with NULLS NOT
DISTINCT (Postgres 15+) so global settings (port_id IS NULL) are
uniquely keyed by `key` alone. Surfaced + dedupe'd 65 duplicate
(storage_backend, NULL) rows that had accumulated from race-prone
delete-then-insert patterns in ocr-config / settings / residential-
stages / ai-budget services. All four services converted to true
onConflictDoUpdate upserts so the race window is closed.
API uniformity:
- Response shape standardization: 16 routes converted from
`{ success: true }` to 204 No Content. CLAUDE.md documents the
convention (`{ data: <T> }` for content, 204 for empty mutations,
portal-auth retains `{ success: true }` for the frontend's auth chain).
- req.json() → parseBody() migration across 9 admin/CRM routes
(custom-fields, expenses/export ×3, currency convert,
search/recently-viewed, admin/duplicates, berths/pdf-{upload-url,
versions, parse-results}). Uniform 400 error shapes for
ZodError-flagged bodies.
Custom-fields merge tokens (shipped end-to-end):
- merge-fields.ts gains CUSTOM_MERGE_TOKEN_RE + helpers for the
`{{custom.<fieldName>}}` shape.
- document-templates validator accepts the dynamic shape alongside
the static catalog tokens.
- document-sends.service mergeCustomFieldValues resolver fetches
per-port custom_field_definitions for client/interest/berth contexts
and substitutes stored values keyed by `{{custom.fieldName}}`.
- custom-fields-manager amber banner updated to reflect that merge
tokens now expand (search index + entity-diff remain documented
design limitations).
/api/v1/files cross-entity filtering:
- Validator + listFiles + uploadFile accept companyId AND yachtId
alongside clientId. file-upload-zone propagates both.
- New CompanyFilesTab component mirrors ClientFilesTab; restored as a
visible Documents tab in company-tabs.tsx (was a hidden stub).
Inline TODOs:
- Reviewed remaining two TODOs (per-user reminder schedule, import
worker handlers). Both are placeholders for future feature surfaces,
not bugs — per-port digest works for every customer; nothing
currently enqueues import jobs (verified). Annotated in BACKLOG.
BACKLOG.md updated to reflect what landed and what's still pending
(Documenso-related items still bundled with the back-burnered phases).
Tests: 1185/1185 vitest, tsc clean.
2026-05-08 02:20:27 +02:00
|
|
|
portSlug?: string,
|
fix(storage): route every file op through getStorageBackend()
Removes 12 direct minioClient.{put,get,remove}Object call sites that
bypassed the pluggable storage abstraction. Filesystem-mode deploys
(MULTI_NODE_DEPLOYMENT=false, storage_backend=filesystem) silently
broke at every site: GDPR export, invoice PDF, EOI generation, portal
download, file upload, folder create/rename/delete, signed PDF land,
maintenance cleanup, etc. Each site now resolves the active backend
and uses its put/get/delete + the new presignDownloadUrl() helper.
Folder marker objects in /files/folders/* keep the same on-the-wire
shape but route through the backend. A future refactor should move
folder bookkeeping to a DB-backed virtual-folder table (see audit
HIGH §3 follow-up note in the route file).
Sites left untouched: src/lib/services/system-monitoring.service.ts
and src/app/api/ready/route.ts use minioClient.bucketExists as an S3-
specific health probe — those are correctly mode-aware and stay.
Refs: docs/audit-comprehensive-2026-05-05.md HIGH §3 (auditor-D Issue 1)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 18:41:02 +02:00
|
|
|
): Promise<string> {
|
|
|
|
|
const backend = await getStorageBackend();
|
fix(audit): non-Documenso backlog sweep — port-binding, NULLS NOT DISTINCT, custom merge tokens, company docs
Wave through the remaining audit-final-deferred items that aren't blocked
on the back-burnered Documenso work.
Multi-tenant isolation:
- Storage proxy ProxyTokenPayload gains optional `p` (port slug) claim;
verifier asserts `key.startsWith(${p}/)`. Defense-in-depth against a
buggy issuer in some future code path that mixes port scopes — every
storage key generated by generateStorageKey() already prefixes the
slug. document-sends opts in for 24h emailed download links; other
callers continue working unchanged via the optional field.
DB schema reconciliation:
- Migration 0047 rebuilds system_settings unique index with NULLS NOT
DISTINCT (Postgres 15+) so global settings (port_id IS NULL) are
uniquely keyed by `key` alone. Surfaced + dedupe'd 65 duplicate
(storage_backend, NULL) rows that had accumulated from race-prone
delete-then-insert patterns in ocr-config / settings / residential-
stages / ai-budget services. All four services converted to true
onConflictDoUpdate upserts so the race window is closed.
API uniformity:
- Response shape standardization: 16 routes converted from
`{ success: true }` to 204 No Content. CLAUDE.md documents the
convention (`{ data: <T> }` for content, 204 for empty mutations,
portal-auth retains `{ success: true }` for the frontend's auth chain).
- req.json() → parseBody() migration across 9 admin/CRM routes
(custom-fields, expenses/export ×3, currency convert,
search/recently-viewed, admin/duplicates, berths/pdf-{upload-url,
versions, parse-results}). Uniform 400 error shapes for
ZodError-flagged bodies.
Custom-fields merge tokens (shipped end-to-end):
- merge-fields.ts gains CUSTOM_MERGE_TOKEN_RE + helpers for the
`{{custom.<fieldName>}}` shape.
- document-templates validator accepts the dynamic shape alongside
the static catalog tokens.
- document-sends.service mergeCustomFieldValues resolver fetches
per-port custom_field_definitions for client/interest/berth contexts
and substitutes stored values keyed by `{{custom.fieldName}}`.
- custom-fields-manager amber banner updated to reflect that merge
tokens now expand (search index + entity-diff remain documented
design limitations).
/api/v1/files cross-entity filtering:
- Validator + listFiles + uploadFile accept companyId AND yachtId
alongside clientId. file-upload-zone propagates both.
- New CompanyFilesTab component mirrors ClientFilesTab; restored as a
visible Documents tab in company-tabs.tsx (was a hidden stub).
Inline TODOs:
- Reviewed remaining two TODOs (per-user reminder schedule, import
worker handlers). Both are placeholders for future feature surfaces,
not bugs — per-port digest works for every customer; nothing
currently enqueues import jobs (verified). Annotated in BACKLOG.
BACKLOG.md updated to reflect what landed and what's still pending
(Documenso-related items still bundled with the back-burnered phases).
Tests: 1185/1185 vitest, tsc clean.
2026-05-08 02:20:27 +02:00
|
|
|
const { url } = await backend.presignDownload(key, { expirySeconds, filename, portSlug });
|
fix(storage): route every file op through getStorageBackend()
Removes 12 direct minioClient.{put,get,remove}Object call sites that
bypassed the pluggable storage abstraction. Filesystem-mode deploys
(MULTI_NODE_DEPLOYMENT=false, storage_backend=filesystem) silently
broke at every site: GDPR export, invoice PDF, EOI generation, portal
download, file upload, folder create/rename/delete, signed PDF land,
maintenance cleanup, etc. Each site now resolves the active backend
and uses its put/get/delete + the new presignDownloadUrl() helper.
Folder marker objects in /files/folders/* keep the same on-the-wire
shape but route through the backend. A future refactor should move
folder bookkeeping to a DB-backed virtual-folder table (see audit
HIGH §3 follow-up note in the route file).
Sites left untouched: src/lib/services/system-monitoring.service.ts
and src/app/api/ready/route.ts use minioClient.bucketExists as an S3-
specific health probe — those are correctly mode-aware and stay.
Refs: docs/audit-comprehensive-2026-05-05.md HIGH §3 (auditor-D Issue 1)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 18:41:02 +02:00
|
|
|
return url;
|
|
|
|
|
}
|
|
|
|
|
|
feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI
Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays
the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs,
brochures) without touching those domains yet.
New files:
- src/lib/storage/index.ts StorageBackend interface + per-process
factory keyed on system_settings.
- src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/
Wasabi/Tigris) wrapping the existing minio
JS client. Includes a healthCheck() used
by the admin "Test connection" button.
- src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a
mitigations baked in.
- src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock,
per-row resumable progress markers,
sha256 round-trip verification, atomic
storage_backend flip on success.
- scripts/migrate-storage.ts Thin CLI shim around runMigration().
- src/app/api/storage/[token]/route.ts
Filesystem proxy GET. Verifies HMAC,
enforces single-use replay protection
via Redis SET NX, streams via NextResponse
ReadableStream with explicit Content-Type
+ Content-Disposition. Node runtime only.
- src/app/api/v1/admin/storage/route.ts
GET status + POST connection test.
- src/app/api/v1/admin/storage/migrate/route.ts
Super-admin-only POST that runs the
exact same runMigration() as the CLI.
- src/app/(dashboard)/[portSlug]/admin/storage/page.tsx
Super-admin admin UI (current backend,
capacity stats, switch button with
dry-run, test connection, backup hint).
- src/components/admin/storage-admin-panel.tsx
Client component for the page above.
§14.9a critical mitigations implemented:
- Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$;
`..`, `.`, `//`, leading `/`, and overlength keys rejected.
- Realpath: storage root realpath'd at create time, every per-key
resolution checked against the realpath'd prefix.
- Storage root created (or chmod'd) to 0o700.
- Multi-node refusal: FilesystemBackend.create() throws when
MULTI_NODE_DEPLOYMENT=true.
- HMAC token: sha256-HMAC over the (key, expiry, nonce, filename,
content-type) payload. Verified with timingSafeEqual; bad sig,
expired, or invalid-key payloads all return 403.
- Single-use replay: token body cached in Redis SET NX EX 1800s.
- sha256 round-trip: copyAndVerify() re-fetches from the target after
put() and aborts the migration on any mismatch.
- Free-disk pre-flight: when migrating to filesystem, sums byte counts
via source.head() and aborts if free space < total * 1.2.
- pg_advisory_lock(0xc7000a01) prevents concurrent migrations.
- Resumable: per-row progress markers in _storage_migration_progress.
system_settings keys read by the factory (jsonb, no schema change):
storage_backend, storage_s3_endpoint, storage_s3_region,
storage_s3_bucket, storage_s3_access_key,
storage_s3_secret_key_encrypted, storage_s3_force_path_style,
storage_filesystem_root, storage_proxy_hmac_secret_encrypted.
Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage`
(./storage added to .gitignore).
Tests added (34 tests, all green):
- tests/unit/storage/filesystem-backend.test.ts — key validation
allow/reject matrix, realpath escape, 0o700 perms, multi-node
refusal, HMAC token sign/verify/tamper/expire/invalid-key.
- tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on
round-trip aborts the migration.
- tests/integration/storage/proxy-route.test.ts — happy path, wrong
HMAC secret, expired token, replay rejection.
Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is
intentionally empty. berth_pdf_versions and brochure_versions land in
Phase 6b and join the list there. Existing s3_key columns: only
gdpr_export_jobs.storage_key, already named correctly — no rename needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 03:15:59 +02:00
|
|
|
// ─── re-exports ─────────────────────────────────────────────────────────────
|
|
|
|
|
|
|
|
|
|
export { S3Backend } from './s3';
|
|
|
|
|
export { FilesystemBackend, validateStorageKey } from './filesystem';
|