Files
pn-new-crm/src/app/api/v1/expenses/scan-receipt/route.ts

130 lines
4.7 KiB
TypeScript
Raw Normal View History

import { NextResponse } from 'next/server';
import { withAuth, withPermission, withRateLimit } from '@/lib/api/helpers';
fix(audit-tier-2-routes): manual NextResponse.json error sweep + admin form banners Two final waves of error-surface hygiene closing the audit's MED §12 + HIGH §15 + HIGH §17 findings: * 50 route files swept (61 sites): manual NextResponse.json({error, status: 4xx|5xx}) early-returns replaced by typed throws + errorResponse(err) at the catch. - Super-admin gates (13 sites) use new requireSuperAdmin(ctx, action) helper from src/lib/api/helpers.ts so denials hit the audit log. - Path-param + body validation 400s become ValidationError throws. - 404s become NotFoundError or CodedError('NOT_FOUND') for AI feature-flag paths. - 11 manual 5xx returns now re-throw so error_events captures the request-id (the admin error inspector becomes usable from real incidents). - website-analytics 200-with-error anti-pattern flipped to 409 + UMAMI_NOT_CONFIGURED. 502 upstream paths use UMAMI_UPSTREAM_ERROR. - 11 sites intentionally preserved: storage/[token] anti-enumeration token-failure paths, webhook-secret 401, "Unknown port" 400 in public intake. * 7 admin forms (roles, users, ports, webhooks, custom-fields, document-templates, tags) gain a formatErrorBanner() helper from src/lib/api/toast-error.ts that builds a multi-line "Error code / Reference ID" banner — the rep can copy the request id when reporting a failed save. Banners get whitespace-pre-line so newlines render. Test status: 1168/1168 vitest, tsc clean. Refs: docs/audit-comprehensive-2026-05-05.md MED §12 (auditor-F Issue 1) + HIGH §15 (auditor-F Issue 2) + HIGH §17 (auditor-H Issue 2). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 20:36:59 +02:00
import { errorResponse, ValidationError } from '@/lib/errors';
feat(phase-b): ship analytics dashboard, alerts, scanner PWA, dedup, audit view Phase B (Insights & Alerts) PR4-11 in one drop. Builds on the schema + service skeletons committed in PRs 1-3. PR4 Analytics dashboard — 4 chart types (funnel/timeline/breakdown/source), date-range picker (today/7d/30d/90d), CSV+PNG export per card. PR5 Alert rail UI + /alerts page — topbar bell w/ live count, dashboard right-rail, three-tab page (active/dismissed/resolved), socket-driven invalidation. Bell lazy-loads list on popover open to keep cold pages fast in non-dashboard routes. PR6 EOI queue tab on documents hub — filters to in-flight EOIs, count surfaces in tab label. PR7 Interests-by-berth tab on berth detail — replaces the stub. PR8 Expense duplicate detection — BullMQ job runs scan on create, yellow banner on detail w/ Merge / Not-a-duplicate, transactional merge consolidates receipts and archives the source. PR9 Receipt scanner PWA + multi-provider AI — port-scoped /scan route in its own (scanner) group with no dashboard chrome, dynamic per-port manifest, OpenAI + Claude provider abstraction, admin OCR settings page (port-level + super-admin global default w/ opt-in fallback), test-connection endpoint, manual-entry fallback when no key is configured. Verify form always shown before save — no ghost rows. PR10 Audit log read view — swap to tsvector full-text search on the existing GIN index, cursor pagination, filters for entity/action/user /date range, batched actor-email resolution. PR11 Real-API tests — opt-in receipt-ocr.spec (admin save+test, optional real-receipt parse via REALAPI_RECEIPT_FIXTURE) and alert-engine socket-fanout spec gated behind RUN_ALERT_ENGINE_REALAPI. Both skip cleanly without their gate envs so CI stays green. Test totals: vitest 690 -> 713, smoke 130 -> 138, realapi +2 opt-in. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 17:21:55 +02:00
import { logger } from '@/lib/logger';
import { getResolvedOcrConfig } from '@/lib/services/ocr-config.service';
import {
runOcr,
type ParsedReceipt,
OCR_FEATURE,
OCR_ESTIMATED_TOKENS,
} from '@/lib/services/ocr-providers';
import { checkBudget, recordAiUsage } from '@/lib/services/ai-budget.service';
feat(phase-b): ship analytics dashboard, alerts, scanner PWA, dedup, audit view Phase B (Insights & Alerts) PR4-11 in one drop. Builds on the schema + service skeletons committed in PRs 1-3. PR4 Analytics dashboard — 4 chart types (funnel/timeline/breakdown/source), date-range picker (today/7d/30d/90d), CSV+PNG export per card. PR5 Alert rail UI + /alerts page — topbar bell w/ live count, dashboard right-rail, three-tab page (active/dismissed/resolved), socket-driven invalidation. Bell lazy-loads list on popover open to keep cold pages fast in non-dashboard routes. PR6 EOI queue tab on documents hub — filters to in-flight EOIs, count surfaces in tab label. PR7 Interests-by-berth tab on berth detail — replaces the stub. PR8 Expense duplicate detection — BullMQ job runs scan on create, yellow banner on detail w/ Merge / Not-a-duplicate, transactional merge consolidates receipts and archives the source. PR9 Receipt scanner PWA + multi-provider AI — port-scoped /scan route in its own (scanner) group with no dashboard chrome, dynamic per-port manifest, OpenAI + Claude provider abstraction, admin OCR settings page (port-level + super-admin global default w/ opt-in fallback), test-connection endpoint, manual-entry fallback when no key is configured. Verify form always shown before save — no ghost rows. PR10 Audit log read view — swap to tsvector full-text search on the existing GIN index, cursor pagination, filters for entity/action/user /date range, batched actor-email resolution. PR11 Real-API tests — opt-in receipt-ocr.spec (admin save+test, optional real-receipt parse via REALAPI_RECEIPT_FIXTURE) and alert-engine socket-fanout spec gated behind RUN_ALERT_ENGINE_REALAPI. Both skip cleanly without their gate envs so CI stays green. Test totals: vitest 690 -> 713, smoke 130 -> 138, realapi +2 opt-in. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 17:21:55 +02:00
const EMPTY: ParsedReceipt = {
establishment: null,
date: null,
amount: null,
currency: null,
lineItems: [],
confidence: 0,
};
export const POST = withAuth(
withPermission(
'expenses',
'create',
withRateLimit('ocr', async (req, ctx) => {
try {
const formData = await req.formData();
const file = formData.get('file') as File | null;
fix(audit-tier-2-routes): manual NextResponse.json error sweep + admin form banners Two final waves of error-surface hygiene closing the audit's MED §12 + HIGH §15 + HIGH §17 findings: * 50 route files swept (61 sites): manual NextResponse.json({error, status: 4xx|5xx}) early-returns replaced by typed throws + errorResponse(err) at the catch. - Super-admin gates (13 sites) use new requireSuperAdmin(ctx, action) helper from src/lib/api/helpers.ts so denials hit the audit log. - Path-param + body validation 400s become ValidationError throws. - 404s become NotFoundError or CodedError('NOT_FOUND') for AI feature-flag paths. - 11 manual 5xx returns now re-throw so error_events captures the request-id (the admin error inspector becomes usable from real incidents). - website-analytics 200-with-error anti-pattern flipped to 409 + UMAMI_NOT_CONFIGURED. 502 upstream paths use UMAMI_UPSTREAM_ERROR. - 11 sites intentionally preserved: storage/[token] anti-enumeration token-failure paths, webhook-secret 401, "Unknown port" 400 in public intake. * 7 admin forms (roles, users, ports, webhooks, custom-fields, document-templates, tags) gain a formatErrorBanner() helper from src/lib/api/toast-error.ts that builds a multi-line "Error code / Reference ID" banner — the rep can copy the request id when reporting a failed save. Banners get whitespace-pre-line so newlines render. Test status: 1168/1168 vitest, tsc clean. Refs: docs/audit-comprehensive-2026-05-05.md MED §12 (auditor-F Issue 1) + HIGH §15 (auditor-F Issue 2) + HIGH §17 (auditor-H Issue 2). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 20:36:59 +02:00
if (!file) throw new ValidationError('A file is required');
fix(audit-tier-6): validation, perms, ops/infra, per-port webhook secret Final audit polish — closes the remaining LOW + MED items the previous tiers didn't reach: * Validation hardening: me.preferences uses .strict() + 8KB cap instead of unbounded .passthrough(); files.uploadFile gains magic-byte verification (jpeg/png/gif/webp/pdf/doc/xlsx); OCR scan endpoint enforces 10MB cap + magic-byte check on receipt images; port logoUrl + me.avatarUrl reject javascript:/data: schemes via a shared httpUrl refinement. * Permission gates: document-sends/{brochure,berth-pdf} now require email.send (was withAuth-only); document-sends/{preview,list} on email.view; ai/email-draft on email.send; documents/[id]/send uses send_for_signing (was create); expenses/export/parent-company flips from hard isSuperAdmin to expenses.export for parity; admin/users/options gated on reminders.assign_others (was withAuth). * Envelope hygiene: auth/set-password switches the third {message} variant to errorResponse + {data: {email}}; ai/email-draft wraps jobId in {data: {jobId}}. * UI polish: reports-list.handleDownload surfaces failures via toastError (was console-only). * Ops/infra: pin pnpm@10.33.2 across all three Dockerfiles + packageManager field in package.json; Dockerfile.worker re-orders user creation BEFORE pnpm install so node_modules / .cache dirs are worker-owned (fixes tesseract.js + sharp EACCES at first PDF parse); add Redis-ping HEALTHCHECK to the worker container. * Public health endpoint: returns full env+appUrl payload only when the caller presents X-Intake-Secret, otherwise a minimal {status} so generic uptime monitors still work but anonymous internet doesn't get deployment fingerprints. * Per-port Documenso webhook secret: new system_settings key + listDocumensoWebhookSecrets() helper. The webhook receiver iterates every configured per-port secret with timing-safe comparison + falls back to env, then forwards the resolved portId into handleDocumentExpired so two ports sharing a documensoId cannot cross-mutate. Deferred (handled in dedicated follow-up PRs): * Tier 5.1 — direct service tests for portal-auth / users / email-accounts / document-sends / sales-email-config. MED, large test-writing scope. * The {ok: true} → {data: null} envelope migration across alerts/expenses/admin-ocr-settings/storage routes. Mechanical but needs coordinated client + test updates. * CSP-nonce migration (drop unsafe-inline) — needs middleware-level nonce generation that the Next 15 router has to thread through. * Idempotency-Key header on Documenso createDocument. Requires schema column on documents to persist the key; deferred so it doesn't bundle a migration into this commit. * The 16 better-auth user_id FKs — separate dedicated migration with care (some columns are NOT NULL today and cascade decisions matter). * PermissionGate / Skeleton / EmptyState wraps across 5 admin lists (auditor-H §§36–37) and the residential-clients filter bar. Test status: 1175/1175 vitest, tsc clean. Refs: docs/audit-comprehensive-2026-05-05.md MED §§28,29,30 + LOW §§32–43 + HIGH §9 (Documenso secrets follow-up). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 21:03:31 +02:00
// Hard 10 MB cap — without this any authenticated rep could grief
// their own port's AI budget by sending arbitrarily large images
// and burning OCR tokens (auditor-E3 §28).
const MAX_OCR_BYTES = 10 * 1024 * 1024;
if (file.size > MAX_OCR_BYTES) {
throw new ValidationError('Receipt image is too large (10 MB max).');
}
const buffer = Buffer.from(await file.arrayBuffer());
const mimeType = file.type || 'image/jpeg';
fix(audit-tier-6): validation, perms, ops/infra, per-port webhook secret Final audit polish — closes the remaining LOW + MED items the previous tiers didn't reach: * Validation hardening: me.preferences uses .strict() + 8KB cap instead of unbounded .passthrough(); files.uploadFile gains magic-byte verification (jpeg/png/gif/webp/pdf/doc/xlsx); OCR scan endpoint enforces 10MB cap + magic-byte check on receipt images; port logoUrl + me.avatarUrl reject javascript:/data: schemes via a shared httpUrl refinement. * Permission gates: document-sends/{brochure,berth-pdf} now require email.send (was withAuth-only); document-sends/{preview,list} on email.view; ai/email-draft on email.send; documents/[id]/send uses send_for_signing (was create); expenses/export/parent-company flips from hard isSuperAdmin to expenses.export for parity; admin/users/options gated on reminders.assign_others (was withAuth). * Envelope hygiene: auth/set-password switches the third {message} variant to errorResponse + {data: {email}}; ai/email-draft wraps jobId in {data: {jobId}}. * UI polish: reports-list.handleDownload surfaces failures via toastError (was console-only). * Ops/infra: pin pnpm@10.33.2 across all three Dockerfiles + packageManager field in package.json; Dockerfile.worker re-orders user creation BEFORE pnpm install so node_modules / .cache dirs are worker-owned (fixes tesseract.js + sharp EACCES at first PDF parse); add Redis-ping HEALTHCHECK to the worker container. * Public health endpoint: returns full env+appUrl payload only when the caller presents X-Intake-Secret, otherwise a minimal {status} so generic uptime monitors still work but anonymous internet doesn't get deployment fingerprints. * Per-port Documenso webhook secret: new system_settings key + listDocumensoWebhookSecrets() helper. The webhook receiver iterates every configured per-port secret with timing-safe comparison + falls back to env, then forwards the resolved portId into handleDocumentExpired so two ports sharing a documensoId cannot cross-mutate. Deferred (handled in dedicated follow-up PRs): * Tier 5.1 — direct service tests for portal-auth / users / email-accounts / document-sends / sales-email-config. MED, large test-writing scope. * The {ok: true} → {data: null} envelope migration across alerts/expenses/admin-ocr-settings/storage routes. Mechanical but needs coordinated client + test updates. * CSP-nonce migration (drop unsafe-inline) — needs middleware-level nonce generation that the Next 15 router has to thread through. * Idempotency-Key header on Documenso createDocument. Requires schema column on documents to persist the key; deferred so it doesn't bundle a migration into this commit. * The 16 better-auth user_id FKs — separate dedicated migration with care (some columns are NOT NULL today and cascade decisions matter). * PermissionGate / Skeleton / EmptyState wraps across 5 admin lists (auditor-H §§36–37) and the residential-clients filter bar. Test status: 1175/1175 vitest, tsc clean. Refs: docs/audit-comprehensive-2026-05-05.md MED §§28,29,30 + LOW §§32–43 + HIGH §9 (Documenso secrets follow-up). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 21:03:31 +02:00
// Magic-byte gate so a forged Content-Type doesn't reach the OCR
// provider with arbitrary bytes.
const { bufferMatchesMime } = await import('@/lib/constants/file-validation');
const allowedOcrMimes = ['image/jpeg', 'image/png', 'image/webp'];
if (!allowedOcrMimes.includes(mimeType) || !bufferMatchesMime(buffer, mimeType)) {
throw new ValidationError('Unsupported receipt image type.');
}
const config = await getResolvedOcrConfig(ctx.portId);
// Tesseract.js (in-browser) is the default. The server only invokes
// an AI provider when (a) the port admin has flipped `aiEnabled` on
// and (b) a key resolves. Otherwise the client falls back to its
// local Tesseract result.
if (!config.aiEnabled) {
return NextResponse.json({
data: { parsed: EMPTY, source: 'manual', reason: 'ai-disabled' },
});
}
if (!config.apiKey) {
return NextResponse.json({
data: { parsed: EMPTY, source: 'manual', reason: 'no-ocr-configured' },
});
}
// Per-port budget gate - refuse the call before we spend tokens
// when the port has already hit its hard cap, or when the request
// would push it past the cap. Soft-cap warnings ride along on the
// success response so the UI can show a banner without blocking.
const budget = await checkBudget({
portId: ctx.portId,
estimatedTokens: OCR_ESTIMATED_TOKENS,
});
if (!budget.ok) {
return NextResponse.json({
data: {
parsed: EMPTY,
source: 'manual',
reason: 'budget-exceeded',
providerError: `AI budget reached (${budget.usedTokens}/${budget.capTokens} tokens this period).`,
},
});
}
try {
const result = await runOcr({
provider: config.provider,
model: config.model,
apiKey: config.apiKey,
imageBuffer: buffer,
mimeType,
});
await recordAiUsage({
portId: ctx.portId,
userId: ctx.userId,
feature: OCR_FEATURE,
provider: config.provider,
model: config.model,
inputTokens: result.usage.inputTokens,
outputTokens: result.usage.outputTokens,
requestId: result.usage.requestId,
});
return NextResponse.json({
data: {
parsed: result.parsed,
source: 'ai',
provider: config.provider,
model: config.model,
softCapWarning: budget.softCap,
},
});
} catch (err) {
logger.error({ err, provider: config.provider }, 'OCR provider call failed');
// Provider hiccup - degrade to manual entry rather than 500-ing.
return NextResponse.json({
data: {
parsed: EMPTY,
source: 'manual',
reason: 'provider-error',
providerError: err instanceof Error ? err.message.slice(0, 200) : 'Unknown error',
},
});
}
} catch (error) {
return errorResponse(error);
feat(phase-b): ship analytics dashboard, alerts, scanner PWA, dedup, audit view Phase B (Insights & Alerts) PR4-11 in one drop. Builds on the schema + service skeletons committed in PRs 1-3. PR4 Analytics dashboard — 4 chart types (funnel/timeline/breakdown/source), date-range picker (today/7d/30d/90d), CSV+PNG export per card. PR5 Alert rail UI + /alerts page — topbar bell w/ live count, dashboard right-rail, three-tab page (active/dismissed/resolved), socket-driven invalidation. Bell lazy-loads list on popover open to keep cold pages fast in non-dashboard routes. PR6 EOI queue tab on documents hub — filters to in-flight EOIs, count surfaces in tab label. PR7 Interests-by-berth tab on berth detail — replaces the stub. PR8 Expense duplicate detection — BullMQ job runs scan on create, yellow banner on detail w/ Merge / Not-a-duplicate, transactional merge consolidates receipts and archives the source. PR9 Receipt scanner PWA + multi-provider AI — port-scoped /scan route in its own (scanner) group with no dashboard chrome, dynamic per-port manifest, OpenAI + Claude provider abstraction, admin OCR settings page (port-level + super-admin global default w/ opt-in fallback), test-connection endpoint, manual-entry fallback when no key is configured. Verify form always shown before save — no ghost rows. PR10 Audit log read view — swap to tsvector full-text search on the existing GIN index, cursor pagination, filters for entity/action/user /date range, batched actor-email resolution. PR11 Real-API tests — opt-in receipt-ocr.spec (admin save+test, optional real-receipt parse via REALAPI_RECEIPT_FIXTURE) and alert-engine socket-fanout spec gated behind RUN_ALERT_ENGINE_REALAPI. Both skip cleanly without their gate envs so CI stays green. Test totals: vitest 690 -> 713, smoke 130 -> 138, realapi +2 opt-in. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 17:21:55 +02:00
}
}),
),
);