feat(storage): pluggable s3-or-filesystem backend + migration CLI + admin UI
Phase 6a from docs/berth-recommender-and-pdf-plan.md §4.7a + §14.9a. Lays
the storage groundwork for Phase 6b/7 file-bearing schemas (per-berth PDFs,
brochures) without touching those domains yet.
New files:
- src/lib/storage/index.ts StorageBackend interface + per-process
factory keyed on system_settings.
- src/lib/storage/s3.ts S3-compatible backend (MinIO/AWS/B2/R2/
Wasabi/Tigris) wrapping the existing minio
JS client. Includes a healthCheck() used
by the admin "Test connection" button.
- src/lib/storage/filesystem.ts Local filesystem backend with all §14.9a
mitigations baked in.
- src/lib/storage/migrate.ts Shared migration core — pg_advisory_lock,
per-row resumable progress markers,
sha256 round-trip verification, atomic
storage_backend flip on success.
- scripts/migrate-storage.ts Thin CLI shim around runMigration().
- src/app/api/storage/[token]/route.ts
Filesystem proxy GET. Verifies HMAC,
enforces single-use replay protection
via Redis SET NX, streams via NextResponse
ReadableStream with explicit Content-Type
+ Content-Disposition. Node runtime only.
- src/app/api/v1/admin/storage/route.ts
GET status + POST connection test.
- src/app/api/v1/admin/storage/migrate/route.ts
Super-admin-only POST that runs the
exact same runMigration() as the CLI.
- src/app/(dashboard)/[portSlug]/admin/storage/page.tsx
Super-admin admin UI (current backend,
capacity stats, switch button with
dry-run, test connection, backup hint).
- src/components/admin/storage-admin-panel.tsx
Client component for the page above.
§14.9a critical mitigations implemented:
- Path-traversal: storage keys validated against ^[a-zA-Z0-9/_.-]+$;
`..`, `.`, `//`, leading `/`, and overlength keys rejected.
- Realpath: storage root realpath'd at create time, every per-key
resolution checked against the realpath'd prefix.
- Storage root created (or chmod'd) to 0o700.
- Multi-node refusal: FilesystemBackend.create() throws when
MULTI_NODE_DEPLOYMENT=true.
- HMAC token: sha256-HMAC over the (key, expiry, nonce, filename,
content-type) payload. Verified with timingSafeEqual; bad sig,
expired, or invalid-key payloads all return 403.
- Single-use replay: token body cached in Redis SET NX EX 1800s.
- sha256 round-trip: copyAndVerify() re-fetches from the target after
put() and aborts the migration on any mismatch.
- Free-disk pre-flight: when migrating to filesystem, sums byte counts
via source.head() and aborts if free space < total * 1.2.
- pg_advisory_lock(0xc7000a01) prevents concurrent migrations.
- Resumable: per-row progress markers in _storage_migration_progress.
system_settings keys read by the factory (jsonb, no schema change):
storage_backend, storage_s3_endpoint, storage_s3_region,
storage_s3_bucket, storage_s3_access_key,
storage_s3_secret_key_encrypted, storage_s3_force_path_style,
storage_filesystem_root, storage_proxy_hmac_secret_encrypted.
Defaults: storage_backend=`s3`, storage_filesystem_root=`./storage`
(./storage added to .gitignore).
Tests added (34 tests, all green):
- tests/unit/storage/filesystem-backend.test.ts — key validation
allow/reject matrix, realpath escape, 0o700 perms, multi-node
refusal, HMAC token sign/verify/tamper/expire/invalid-key.
- tests/unit/storage/copy-and-verify.test.ts — sha256 mismatch on
round-trip aborts the migration.
- tests/integration/storage/proxy-route.test.ts — happy path, wrong
HMAC secret, expired token, replay rejection.
Phase 6a ships zero file-bearing tables — TABLES_WITH_STORAGE_KEYS is
intentionally empty. berth_pdf_versions and brochure_versions land in
Phase 6b and join the list there. Existing s3_key columns: only
gdpr_export_jobs.storage_key, already named correctly — no rename needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
239
src/components/admin/storage-admin-panel.tsx
Normal file
239
src/components/admin/storage-admin-panel.tsx
Normal file
@@ -0,0 +1,239 @@
|
||||
'use client';
|
||||
|
||||
import { useState } from 'react';
|
||||
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query';
|
||||
import { CheckCircle2, HardDrive, Loader2, RefreshCw, ServerCog, XCircle } from 'lucide-react';
|
||||
|
||||
import { PageHeader } from '@/components/shared/page-header';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@/components/ui/card';
|
||||
import {
|
||||
Dialog,
|
||||
DialogContent,
|
||||
DialogDescription,
|
||||
DialogFooter,
|
||||
DialogHeader,
|
||||
DialogTitle,
|
||||
} from '@/components/ui/dialog';
|
||||
import { apiFetch } from '@/lib/api/client';
|
||||
|
||||
type BackendName = 's3' | 'filesystem';
|
||||
|
||||
interface StorageStatus {
|
||||
backend: BackendName;
|
||||
fileCount: number;
|
||||
totalBytes: number;
|
||||
tablesTracked: string[];
|
||||
}
|
||||
|
||||
interface MigrationResult {
|
||||
rowsConsidered: number;
|
||||
rowsMigrated: number;
|
||||
rowsSkippedAlreadyDone: number;
|
||||
totalBytes: number;
|
||||
flipped: boolean;
|
||||
dryRun: boolean;
|
||||
}
|
||||
|
||||
export function StorageAdminPanel() {
|
||||
const queryClient = useQueryClient();
|
||||
const [confirmOpen, setConfirmOpen] = useState(false);
|
||||
const [dryRun, setDryRun] = useState<MigrationResult | null>(null);
|
||||
const [testResult, setTestResult] = useState<{ ok: boolean; error?: string } | null>(null);
|
||||
|
||||
const status = useQuery({
|
||||
queryKey: ['admin', 'storage', 'status'],
|
||||
queryFn: () => apiFetch<{ data: StorageStatus }>('/api/v1/admin/storage'),
|
||||
});
|
||||
|
||||
const dryRunMutation = useMutation({
|
||||
mutationFn: async (opts: { from: BackendName; to: BackendName }) =>
|
||||
apiFetch<{ data: MigrationResult }>('/api/v1/admin/storage/migrate', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ ...opts, dryRun: true }),
|
||||
}),
|
||||
onSuccess: (result) => {
|
||||
setDryRun(result.data);
|
||||
setConfirmOpen(true);
|
||||
},
|
||||
});
|
||||
|
||||
const migrateMutation = useMutation({
|
||||
mutationFn: async (opts: { from: BackendName; to: BackendName }) =>
|
||||
apiFetch<{ data: MigrationResult }>('/api/v1/admin/storage/migrate', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ ...opts, dryRun: false }),
|
||||
}),
|
||||
onSuccess: () => {
|
||||
setConfirmOpen(false);
|
||||
setDryRun(null);
|
||||
queryClient.invalidateQueries({ queryKey: ['admin', 'storage', 'status'] });
|
||||
},
|
||||
});
|
||||
|
||||
const testMutation = useMutation({
|
||||
mutationFn: async () =>
|
||||
apiFetch<{ ok: boolean; error?: string }>('/api/v1/admin/storage', {
|
||||
method: 'POST',
|
||||
}),
|
||||
onSuccess: (r) => setTestResult(r),
|
||||
onError: (e: Error) => setTestResult({ ok: false, error: e.message }),
|
||||
});
|
||||
|
||||
if (status.isLoading) {
|
||||
return (
|
||||
<div className="flex items-center gap-2 text-sm text-muted-foreground">
|
||||
<Loader2 className="h-4 w-4 animate-spin" /> Loading storage status…
|
||||
</div>
|
||||
);
|
||||
}
|
||||
if (status.isError || !status.data?.data) {
|
||||
return <div className="text-sm text-destructive">Failed to load storage status.</div>;
|
||||
}
|
||||
const s = status.data.data;
|
||||
const otherBackend: BackendName = s.backend === 's3' ? 'filesystem' : 's3';
|
||||
|
||||
return (
|
||||
<div className="space-y-6">
|
||||
<PageHeader
|
||||
title="Storage Backend"
|
||||
description="Where the CRM stores per-berth PDFs, brochures, GDPR exports, and other binary files."
|
||||
/>
|
||||
|
||||
<div className="grid gap-4 lg:grid-cols-3">
|
||||
<Card className="lg:col-span-2">
|
||||
<CardHeader className="flex flex-row items-start gap-3 space-y-0 pb-2">
|
||||
{s.backend === 's3' ? (
|
||||
<ServerCog className="mt-0.5 h-5 w-5 text-muted-foreground" />
|
||||
) : (
|
||||
<HardDrive className="mt-0.5 h-5 w-5 text-muted-foreground" />
|
||||
)}
|
||||
<div>
|
||||
<CardTitle className="text-base">Active backend: {s.backend}</CardTitle>
|
||||
<CardDescription>
|
||||
{s.backend === 's3'
|
||||
? 'Files stored in an S3-compatible object store (MinIO, AWS S3, Backblaze B2, Cloudflare R2, Wasabi, Tigris).'
|
||||
: 'Files stored on the local filesystem under storage_filesystem_root. Single-node deployments only.'}
|
||||
</CardDescription>
|
||||
</div>
|
||||
</CardHeader>
|
||||
<CardContent className="space-y-4">
|
||||
<dl className="grid grid-cols-2 gap-3 text-sm">
|
||||
<div>
|
||||
<dt className="text-muted-foreground">Tracked tables</dt>
|
||||
<dd>
|
||||
{s.tablesTracked.length === 0
|
||||
? '(none yet — Phase 6b)'
|
||||
: s.tablesTracked.join(', ')}
|
||||
</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt className="text-muted-foreground">File count</dt>
|
||||
<dd>{s.fileCount}</dd>
|
||||
</div>
|
||||
</dl>
|
||||
|
||||
<div className="flex flex-wrap gap-3">
|
||||
<Button
|
||||
variant="outline"
|
||||
disabled={dryRunMutation.isPending}
|
||||
onClick={() => dryRunMutation.mutate({ from: s.backend, to: otherBackend })}
|
||||
>
|
||||
{dryRunMutation.isPending && <Loader2 className="mr-2 h-4 w-4 animate-spin" />}
|
||||
Switch to {otherBackend}
|
||||
</Button>
|
||||
{s.backend === 's3' && (
|
||||
<Button
|
||||
variant="outline"
|
||||
onClick={() => testMutation.mutate()}
|
||||
disabled={testMutation.isPending}
|
||||
>
|
||||
{testMutation.isPending && <Loader2 className="mr-2 h-4 w-4 animate-spin" />}
|
||||
Test connection
|
||||
</Button>
|
||||
)}
|
||||
<Button variant="ghost" onClick={() => status.refetch()} disabled={status.isFetching}>
|
||||
<RefreshCw className="mr-2 h-4 w-4" /> Refresh
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{testResult && (
|
||||
<div className="rounded-md border p-3 text-sm">
|
||||
{testResult.ok ? (
|
||||
<div className="flex items-center gap-2 text-emerald-600">
|
||||
<CheckCircle2 className="h-4 w-4" /> Connection OK — round-trip succeeded.
|
||||
</div>
|
||||
) : (
|
||||
<div className="flex items-center gap-2 text-destructive">
|
||||
<XCircle className="h-4 w-4" /> {testResult.error ?? 'Connection failed'}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
|
||||
<Card>
|
||||
<CardHeader>
|
||||
<CardTitle className="text-base">Backup notes</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent className="space-y-2 text-sm text-muted-foreground">
|
||||
{s.backend === 's3' ? (
|
||||
<p>
|
||||
S3 mode: configure your provider's lifecycle / replication / versioning
|
||||
policies as your primary backup. The CRM does not duplicate object storage in its
|
||||
own backups.
|
||||
</p>
|
||||
) : (
|
||||
<p>
|
||||
Filesystem mode: include the storage root directory in your backup tool (restic,
|
||||
borg, snapshots). It sits next to the database; the two should be backed up
|
||||
together.
|
||||
</p>
|
||||
)}
|
||||
<p className="pt-2 text-xs">
|
||||
Filesystem mode refuses to start when MULTI_NODE_DEPLOYMENT=true. For multi-node
|
||||
deployments, switch to an S3-compatible backend.
|
||||
</p>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
|
||||
<Dialog open={confirmOpen} onOpenChange={setConfirmOpen}>
|
||||
<DialogContent>
|
||||
<DialogHeader>
|
||||
<DialogTitle>Switch storage backend</DialogTitle>
|
||||
<DialogDescription>
|
||||
Move all tracked files from the current backend to the new backend, verify each file
|
||||
via sha256, then atomically flip the active backend.
|
||||
</DialogDescription>
|
||||
</DialogHeader>
|
||||
{dryRun && (
|
||||
<div className="rounded-md border p-3 text-sm">
|
||||
<dl className="grid grid-cols-2 gap-2">
|
||||
<dt className="text-muted-foreground">Rows considered</dt>
|
||||
<dd>{dryRun.rowsConsidered}</dd>
|
||||
<dt className="text-muted-foreground">Already migrated (resumable)</dt>
|
||||
<dd>{dryRun.rowsSkippedAlreadyDone}</dd>
|
||||
<dt className="text-muted-foreground">Total bytes</dt>
|
||||
<dd>{Math.round(dryRun.totalBytes / 1024)} KB</dd>
|
||||
</dl>
|
||||
</div>
|
||||
)}
|
||||
<DialogFooter>
|
||||
<Button variant="outline" onClick={() => setConfirmOpen(false)}>
|
||||
Cancel
|
||||
</Button>
|
||||
<Button
|
||||
disabled={migrateMutation.isPending}
|
||||
onClick={() => migrateMutation.mutate({ from: s.backend, to: otherBackend })}
|
||||
>
|
||||
{migrateMutation.isPending && <Loader2 className="mr-2 h-4 w-4 animate-spin" />}
|
||||
Migrate now
|
||||
</Button>
|
||||
</DialogFooter>
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
Reference in New Issue
Block a user