110 KiB
110 KiB
AI Services Architecture
Overview
AI services power automation across all round types in the MOPC platform. All AI calls anonymize data before sending to OpenAI, ensuring GDPR compliance and privacy protection. These services enable intelligent filtering, smart jury assignments, evaluation synthesis, automatic tagging, and award eligibility assessment.
Core Principles
- Privacy First: All data is anonymized before AI processing (no PII sent to OpenAI)
- Graceful Degradation: Fallback algorithms when AI is unavailable
- Cost Awareness: Batching, token tracking, and cost monitoring
- Audit Trail: All AI requests logged with inputs, outputs, and token usage
- Admin Control: Per-service toggles, configuration overrides, manual review
- Transparency: AI reasoning exposed to admins for validation
Service Inventory
| Service | Purpose | Primary Model | Input Data | Output |
|---|---|---|---|---|
| ai-filtering.ts | Automated project screening | GPT-4 | Projects + rubric | Pass/Reject/Flag + scores |
| ai-assignment.ts | Jury-project matching | GPT-4 | Jurors + projects + constraints | Assignment suggestions |
| ai-evaluation-summary.ts | Synthesis of evaluations | GPT-4-turbo | All evaluations for a project | Strengths, weaknesses, themes |
| ai-tagging.ts | Auto-tag projects | GPT-4 | Project description | Tag suggestions + confidence |
| ai-award-eligibility.ts | Award eligibility assessment | GPT-4 | Projects + award criteria | Eligibility scores + reasoning |
| anonymization.ts | PII stripping pipeline | N/A | Raw project/user data | Anonymized data + mappings |
Current AI Services
1. AI Filtering Service (ai-filtering.ts)
Purpose: Automate project screening in Round 2 (FILTERING) using admin-defined rubrics and AI interpretation of plain-language criteria.
Input Data
interface ProjectForFiltering {
id: string
title: string
description?: string | null
competitionCategory?: CompetitionCategory | null
foundedAt?: Date | null
country?: string | null
geographicZone?: string | null
tags: string[]
oceanIssue?: OceanIssue | null
wantsMentorship?: boolean | null
institution?: string | null
submissionSource?: SubmissionSource
submittedAt?: Date | null
files: Array<{
id: string
fileName: string
fileType?: FileType | null
}>
_count?: {
teamMembers?: number
files?: number
}
}
interface FilteringRuleInput {
id: string
name: string
ruleType: 'FIELD_BASED' | 'DOCUMENT_CHECK' | 'AI_SCREENING'
configJson: Prisma.JsonValue
priority: number
isActive: boolean
}
Rule Types
Field-Based Rules (No AI):
type FieldRuleConfig = {
conditions: FieldRuleCondition[]
logic: 'AND' | 'OR'
action: 'PASS' | 'REJECT' | 'FLAG'
}
type FieldRuleCondition = {
field: 'competitionCategory' | 'foundedAt' | 'country' | 'geographicZone' | 'tags' | 'oceanIssue'
operator: 'equals' | 'not_equals' | 'greater_than' | 'less_than' | 'contains' | 'in' | 'not_in' | 'older_than_years' | 'newer_than_years' | 'is_empty'
value: string | number | string[]
}
// Example: Reject projects older than 10 years
{
conditions: [
{ field: 'foundedAt', operator: 'older_than_years', value: 10 }
],
logic: 'AND',
action: 'REJECT'
}
// Example: Flag projects from specific countries
{
conditions: [
{ field: 'country', operator: 'in', value: ['US', 'CN', 'RU'] }
],
logic: 'OR',
action: 'FLAG'
}
Document Check Rules (No AI):
type DocumentCheckConfig = {
requiredFileTypes?: string[] // ['pdf', 'docx']
minFileCount?: number
action: 'PASS' | 'REJECT' | 'FLAG'
}
// Example: Require at least 2 PDF files
{
requiredFileTypes: ['pdf'],
minFileCount: 2,
action: 'REJECT'
}
AI Screening Rules (OpenAI):
type AIScreeningConfig = {
criteriaText: string // Plain-language rubric
action: 'PASS' | 'REJECT' | 'FLAG'
batchSize?: number // 1-50, default 20
parallelBatches?: number // 1-10, default 1
}
// Example: Detect spam/low-quality projects
{
criteriaText: `
Projects should demonstrate clear ocean conservation value.
Reject projects that:
- Are spam, test submissions, or joke entries
- Have no meaningful description
- Are unrelated to ocean conservation
- Are duplicate submissions
`,
action: 'REJECT',
batchSize: 20,
parallelBatches: 2
}
AI Screening Process
-
Anonymization: Strip PII from projects
const { anonymized, mappings } = anonymizeProjectsForAI(projects, 'FILTERING') // Before: { id: "proj-abc123", title: "SaveTheSea by John Doe (john@example.com)", ... } // After: { project_id: "P1", title: "SaveTheSea by Team 1", ... } -
Validation: Ensure no PII leaked
if (!validateAnonymizedProjects(anonymized)) { throw new Error('GDPR compliance check failed') } -
Batch Processing: Process projects in configurable batches
const batchSize = Math.min(MAX_BATCH_SIZE, config.batchSize ?? 20) const parallelBatches = Math.min(MAX_PARALLEL_BATCHES, config.parallelBatches ?? 1) for (let i = 0; i < batches.length; i += parallelBatches) { const parallelChunk = batches.slice(i, i + parallelBatches) const results = await Promise.all(parallelChunk.map(processAIBatch)) } -
OpenAI Call: Send anonymized batch with criteria
const prompt = `CRITERIA: ${criteriaText} PROJECTS: ${JSON.stringify(anonymized)} Evaluate and return JSON.` const response = await openai.chat.completions.create({ model: 'gpt-4', messages: [ { role: 'system', content: AI_SCREENING_SYSTEM_PROMPT }, { role: 'user', content: prompt } ], response_format: { type: 'json_object' }, temperature: 0.3 }) -
Result Parsing:
interface AIScreeningResult { meetsCriteria: boolean confidence: number // 0.0 - 1.0 reasoning: string qualityScore: number // 1-10 spamRisk: boolean } // AI returns: { "projects": [ { "project_id": "P1", "meets_criteria": true, "confidence": 0.85, "reasoning": "Clear ocean conservation focus, well-documented approach", "quality_score": 8, "spam_risk": false } ] } -
De-anonymization: Map results back to real IDs
const mapping = mappings.find(m => m.anonymousId === "P1") results.set(mapping.realId, aiResult)
Output
interface ProjectFilteringResult {
projectId: string
outcome: 'PASSED' | 'FILTERED_OUT' | 'FLAGGED'
ruleResults: RuleResult[]
aiScreeningJson?: {
[ruleId: string]: AIScreeningResult
}
}
interface RuleResult {
ruleId: string
ruleName: string
ruleType: string
passed: boolean
action: 'PASS' | 'REJECT' | 'FLAG'
reasoning?: string // Only for AI_SCREENING
}
Outcome Logic:
- If ANY rule with
action: 'REJECT'fails →FILTERED_OUT - Else if ANY rule with
action: 'FLAG'fails →FLAGGED - Else →
PASSED
Integration with Filtering Round
// Round 2: FILTERING
const filteringRound = await prisma.round.findFirst({
where: { competitionId, roundType: 'FILTERING' }
})
const rules = await prisma.filteringRule.findMany({
where: { roundId: filteringRound.id, isActive: true },
orderBy: { priority: 'asc' }
})
const projects = await prisma.project.findMany({
where: { competitionId },
include: { files: true, _count: { select: { teamMembers: true } } }
})
const results = await executeFilteringRules(rules, projects, userId, roundId)
// Store results
for (const result of results) {
await prisma.filteringResult.create({
data: {
projectId: result.projectId,
roundId: filteringRound.id,
outcome: result.outcome,
ruleResultsJson: result.ruleResults,
aiScreeningJson: result.aiScreeningJson
}
})
// Update project round state
await prisma.projectRoundState.update({
where: { projectId_roundId: { projectId: result.projectId, roundId: filteringRound.id } },
data: {
state: result.outcome === 'PASSED' ? 'PASSED' : 'REJECTED',
metadataJson: { filteringOutcome: result.outcome }
}
})
}
2. AI Assignment Service (ai-assignment.ts)
Purpose: Generate optimal jury-to-project assignments based on expertise matching, workload balancing, and constraints.
Input Data
interface JurorForAssignment {
id: string
name?: string | null
email: string
expertiseTags: string[]
maxAssignments?: number | null
_count?: {
assignments: number
}
}
interface ProjectForAssignment {
id: string
title: string
description?: string | null
tags: string[]
teamName?: string | null
_count?: {
assignments: number
}
}
interface AssignmentConstraints {
requiredReviewsPerProject: number // e.g., 3 jurors per project
minAssignmentsPerJuror?: number // e.g., 5 projects minimum
maxAssignmentsPerJuror?: number // e.g., 20 projects maximum
jurorLimits?: Record<string, number> // Per-juror overrides
existingAssignments: Array<{
jurorId: string
projectId: string
}>
}
Anonymization & Batching
// 1. Truncate descriptions (save tokens)
const truncatedProjects = projects.map(p => ({
...p,
description: truncateAndSanitize(p.description, DESCRIPTION_LIMITS.ASSIGNMENT) // 300 chars
}))
// 2. Anonymize data
const anonymizedData = anonymizeForAI(jurors, truncatedProjects)
// Before:
jurors = [
{ id: "user-123", email: "john@example.com", expertiseTags: ["Marine Biology", "AI"] },
{ id: "user-456", email: "jane@example.com", expertiseTags: ["Oceanography"] }
]
// After:
anonymizedData.jurors = [
{ anonymousId: "juror_001", expertiseTags: ["Marine Biology", "AI"], currentAssignmentCount: 5, maxAssignments: 20 },
{ anonymousId: "juror_002", expertiseTags: ["Oceanography"], currentAssignmentCount: 3, maxAssignments: 20 }
]
// 3. Validate anonymization
if (!validateAnonymization(anonymizedData)) {
return generateFallbackAssignments(jurors, projects, constraints)
}
// 4. Process in batches (15 projects per batch)
const BATCH_SIZE = 15
for (let i = 0; i < projects.length; i += BATCH_SIZE) {
const batchProjects = anonymizedData.projects.slice(i, i + BATCH_SIZE)
const suggestions = await processAssignmentBatch(openai, model, anonymizedData, batchProjects, constraints)
}
Prompt Structure
const userPrompt = `JURORS: ${JSON.stringify(jurors)}
PROJECTS: ${JSON.stringify(projects)}
CONSTRAINTS: ${constraints.requiredReviewsPerProject} reviews/project, max ${constraints.maxAssignmentsPerJuror}/juror
EXISTING: ${JSON.stringify(anonymousExisting)}
Return JSON: {"assignments": [...]}`
const systemPrompt = `Match jurors to projects by expertise. Return JSON assignments.
Each: {juror_id, project_id, confidence_score: 0-1, expertise_match_score: 0-1, reasoning: str (1-2 sentences)}
Distribute workload fairly. Avoid assigning jurors at capacity.`
Example AI Response:
{
"assignments": [
{
"juror_id": "juror_001",
"project_id": "project_003",
"confidence_score": 0.92,
"expertise_match_score": 0.88,
"reasoning": "Strong match on Marine Biology and AI tags. Juror has capacity."
},
{
"juror_id": "juror_002",
"project_id": "project_003",
"confidence_score": 0.75,
"expertise_match_score": 0.70,
"reasoning": "Oceanography expertise applies. Helps distribute workload."
}
]
}
Output
interface AIAssignmentSuggestion {
jurorId: string // Real ID (de-anonymized)
projectId: string // Real ID (de-anonymized)
confidenceScore: number // 0-1
expertiseMatchScore: number // 0-1
reasoning: string
}
interface AIAssignmentResult {
success: boolean
suggestions: AIAssignmentSuggestion[]
error?: string
tokensUsed?: number
fallbackUsed?: boolean
}
Fallback Algorithm
When AI is unavailable or fails:
export function generateFallbackAssignments(
jurors: JurorForAssignment[],
projects: ProjectForAssignment[],
constraints: AssignmentConstraints
): AIAssignmentResult {
// Algorithm:
// 1. Sort projects by current assignment count (fewest first)
// 2. For each project, score all available jurors by:
// - Expertise match (50% weight): tag overlap
// - Load balancing (30% weight): current vs. max assignments
// - Under-min bonus (20% weight): jurors below minimum target
// 3. Assign top-scoring jurors
const scoredJurors = jurors.map(juror => ({
juror,
score: calculateExpertiseScore(juror.expertiseTags, project.tags) * 0.5
+ calculateLoadScore(currentLoad, maxLoad) * 0.3
+ calculateUnderMinBonus(currentLoad, minTarget) * 0.2
}))
// Assign top N jurors per project
const topN = scoredJurors.slice(0, neededReviews)
return {
success: true,
suggestions: topN.map(s => ({ jurorId: s.juror.id, projectId, ... })),
fallbackUsed: true
}
}
Expertise Score:
function calculateExpertiseScore(jurorTags: string[], projectTags: string[]): number {
const jurorTagsLower = new Set(jurorTags.map(t => t.toLowerCase()))
const matchingTags = projectTags.filter(t => jurorTagsLower.has(t.toLowerCase()))
const matchRatio = matchingTags.length / projectTags.length
const hasExpertise = matchingTags.length > 0 ? 0.2 : 0
return Math.min(1, matchRatio * 0.8 + hasExpertise)
}
Integration with Evaluation Round
// Round 3/5: EVALUATION
const evaluationRound = await prisma.round.findFirst({
where: { competitionId, roundType: 'EVALUATION', sortOrder: 3 }
})
const juryGroup = await prisma.juryGroup.findUnique({
where: { id: evaluationRound.juryGroupId },
include: {
members: { include: { user: true } }
}
})
const projects = await prisma.project.findMany({
where: {
projectRoundStates: {
some: {
roundId: previousRoundId,
state: 'PASSED'
}
}
}
})
const constraints = {
requiredReviewsPerProject: 3,
maxAssignmentsPerJuror: 20,
jurorLimits: {
"user-123": 15, // Personal override
"user-456": 25 // Personal override
},
existingAssignments: await prisma.assignment.findMany({
where: { roundId: evaluationRound.id },
select: { userId: true, projectId: true }
})
}
const result = await generateAIAssignments(
juryGroup.members.map(m => m.user),
projects,
constraints,
adminUserId,
evaluationRound.id,
onProgress
)
// Admin reviews suggestions in UI, then applies
for (const suggestion of result.suggestions) {
await prisma.assignment.create({
data: {
userId: suggestion.jurorId,
projectId: suggestion.projectId,
roundId: evaluationRound.id,
juryGroupId: juryGroup.id,
method: 'AI',
aiConfidenceScore: suggestion.confidenceScore,
expertiseMatchScore: suggestion.expertiseMatchScore,
aiReasoning: suggestion.reasoning
}
})
}
3. AI Evaluation Summary Service (ai-evaluation-summary.ts)
Purpose: Synthesize multiple juror evaluations for a project into a cohesive summary with strengths, weaknesses, themes, and consensus analysis.
Input Data
interface EvaluationForSummary {
id: string
criterionScoresJson: Record<string, number> | null
globalScore: number | null
binaryDecision: boolean | null
feedbackText: string | null
assignment: {
user: {
id: string
name: string | null
email: string
}
}
}
interface CriterionDef {
id: string
label: string
}
Anonymization
// Strips juror identities, keeps only scores and sanitized feedback
export function anonymizeEvaluations(
evaluations: EvaluationForSummary[]
): AnonymizedEvaluation[] {
return evaluations.map(ev => ({
criterionScores: ev.criterionScoresJson,
globalScore: ev.globalScore,
binaryDecision: ev.binaryDecision,
feedbackText: ev.feedbackText ? sanitizeText(ev.feedbackText) : null
}))
}
// Before:
[
{
id: "eval-123",
globalScore: 8,
feedbackText: "Strong proposal from john@example.com. Contact me at +1-555-1234.",
assignment: { user: { name: "John Doe", email: "john@example.com" } }
}
]
// After:
[
{
globalScore: 8,
feedbackText: "Strong proposal. Contact me at [phone removed].",
// No user info
}
]
Prompt Structure
const prompt = `You are analyzing jury evaluations for a project competition.
PROJECT: "${sanitizedTitle}"
EVALUATION CRITERIA: ${criteriaLabels.join(', ')}
EVALUATIONS (${anonymizedEvaluations.length} total):
${JSON.stringify(anonymizedEvaluations, null, 2)}
Analyze these evaluations and return a JSON object with this exact structure:
{
"overallAssessment": "A 2-3 sentence summary of how the project was evaluated overall",
"strengths": ["strength 1", "strength 2", ...],
"weaknesses": ["weakness 1", "weakness 2", ...],
"themes": [
{ "theme": "theme name", "sentiment": "positive" | "negative" | "mixed", "frequency": <number of evaluators mentioning this> }
],
"recommendation": "A brief recommendation based on the evaluation consensus"
}
Guidelines:
- Base your analysis only on the provided evaluation data
- Identify common themes across evaluator feedback
- Note areas of agreement and disagreement
- Keep the assessment objective and balanced
- Do not include any personal identifiers`
Scoring Patterns (Server-Side)
In addition to AI analysis, the service computes statistical patterns:
interface ScoringPatterns {
averageGlobalScore: number | null
consensus: number // 0-1 (1 = full agreement)
criterionAverages: Record<string, number>
evaluatorCount: number
}
export function computeScoringPatterns(
evaluations: EvaluationForSummary[],
criteriaLabels: CriterionDef[]
): ScoringPatterns {
const globalScores = evaluations.map(e => e.globalScore).filter(s => s !== null)
const averageGlobalScore = globalScores.length > 0
? globalScores.reduce((a, b) => a + b, 0) / globalScores.length
: null
// Consensus: 1 - normalized standard deviation
let consensus = 1
if (globalScores.length > 1 && averageGlobalScore !== null) {
const variance = globalScores.reduce((sum, score) =>
sum + Math.pow(score - averageGlobalScore, 2), 0
) / globalScores.length
const stdDev = Math.sqrt(variance)
consensus = Math.max(0, 1 - stdDev / 4.5) // Normalize by max possible std dev
}
// Criterion averages
const criterionAverages: Record<string, number> = {}
for (const criterion of criteriaLabels) {
const scores = evaluations
.map(e => e.criterionScoresJson?.[criterion.id])
.filter((s): s is number => s !== undefined)
if (scores.length > 0) {
criterionAverages[criterion.label] = scores.reduce((a, b) => a + b, 0) / scores.length
}
}
return {
averageGlobalScore,
consensus: Math.round(consensus * 100) / 100,
criterionAverages,
evaluatorCount: evaluations.length
}
}
Output
interface AIResponsePayload {
overallAssessment: string
strengths: string[]
weaknesses: string[]
themes: Array<{
theme: string
sentiment: 'positive' | 'negative' | 'mixed'
frequency: number
}>
recommendation: string
}
interface EvaluationSummaryResult {
id: string
projectId: string
stageId: string
summaryJson: AIResponsePayload & { scoringPatterns: ScoringPatterns }
generatedAt: Date
model: string
tokensUsed: number
}
Example Output:
{
"overallAssessment": "The project received consistently high scores (avg: 8.2/10) with strong consensus (0.92). Evaluators praised the technical approach and team expertise, but raised concerns about scalability and budget.",
"strengths": [
"Novel AI-powered coral monitoring approach",
"Strong technical team with relevant expertise",
"Clear commercial viability and market need"
],
"weaknesses": [
"Budget appears optimistic for proposed timeline",
"Limited discussion of regulatory compliance",
"Scalability challenges not fully addressed"
],
"themes": [
{ "theme": "Technical Innovation", "sentiment": "positive", "frequency": 5 },
{ "theme": "Team Expertise", "sentiment": "positive", "frequency": 4 },
{ "theme": "Budget Concerns", "sentiment": "negative", "frequency": 3 },
{ "theme": "Scalability", "sentiment": "mixed", "frequency": 2 }
],
"recommendation": "Strong candidate for advancement. Address budget and scalability concerns during mentoring phase.",
"scoringPatterns": {
"averageGlobalScore": 8.2,
"consensus": 0.92,
"criterionAverages": {
"Innovation": 8.8,
"Feasibility": 7.6,
"Impact": 8.4,
"Team": 9.0
},
"evaluatorCount": 5
}
}
Integration with Evaluation Round
// Auto-trigger when all evaluations for a project are submitted
const evaluationsCount = await prisma.evaluation.count({
where: {
status: 'SUBMITTED',
assignment: { projectId, roundId }
}
})
const requiredReviews = evaluationRound.configJson.requiredReviewsPerProject
if (evaluationsCount >= requiredReviews) {
const summary = await generateSummary({
projectId,
stageId: roundId,
userId: 'system',
prisma
})
// Notify admin
await prisma.inAppNotification.create({
data: {
userId: adminId,
title: 'Evaluation Summary Generated',
message: `AI summary ready for project "${projectTitle}"`,
type: 'AI_SUMMARY_READY',
relatedEntityType: 'EvaluationSummary',
relatedEntityId: summary.id
}
})
}
4. AI Tagging Service (ai-tagging.ts)
Purpose: Automatically assign expertise tags to projects based on content analysis.
Input Data
// Full project data with files and team info
const project = await prisma.project.findUnique({
where: { id: projectId },
include: {
projectTags: true,
files: { select: { fileType: true } },
_count: { select: { teamMembers: true, files: true } }
}
})
// Available tags from system
interface AvailableTag {
id: string
name: string
category: string | null
description: string | null
}
const availableTags = await prisma.expertiseTag.findMany({
where: { isActive: true },
orderBy: [{ category: 'asc' }, { sortOrder: 'asc' }]
})
Anonymization
const projectWithRelations = toProjectWithRelations(project)
const { anonymized, mappings } = anonymizeProjectsForAI([projectWithRelations], 'FILTERING')
// Anonymized project:
{
project_id: "P1",
title: "SaveTheSea",
description: "An AI-powered platform for coral reef monitoring...",
category: "STARTUP",
ocean_issue: "CORAL_REEFS",
tags: ["AI", "Monitoring"],
founded_year: 2024,
team_size: 3,
file_count: 5,
file_types: ["PDF", "DOCX"]
}
Prompt Structure
const tagList = availableTags.map(t => ({
name: t.name,
category: t.category,
description: t.description
}))
const prompt = `PROJECT:
${JSON.stringify(anonymizedProject, null, 2)}
AVAILABLE TAGS:
${JSON.stringify(tagList, null, 2)}
Suggest relevant tags for this project.`
const systemPrompt = `You are an expert at categorizing ocean conservation and sustainability projects.
Analyze the project and suggest the most relevant expertise tags from the provided list.
Consider the project's focus areas, technology, methodology, and domain.
Return JSON with this format:
{
"suggestions": [
{
"tag_name": "exact tag name from list",
"confidence": 0.0-1.0,
"reasoning": "brief explanation why this tag fits"
}
]
}
Rules:
- Only suggest tags from the provided list (exact names)
- Order by relevance (most relevant first)
- Confidence should reflect how well the tag matches
- Maximum 7 suggestions per project
- Be conservative - only suggest tags that truly apply`
Output
interface TagSuggestion {
tagId: string
tagName: string
confidence: number
reasoning: string
}
interface TaggingResult {
projectId: string
suggestions: TagSuggestion[]
applied: TagSuggestion[]
tokensUsed: number
}
Example Response:
{
"suggestions": [
{
"tag_name": "Marine Biology",
"confidence": 0.92,
"reasoning": "Project focuses on coral reef health monitoring"
},
{
"tag_name": "Artificial Intelligence",
"confidence": 0.88,
"reasoning": "Uses AI for image analysis and pattern detection"
},
{
"tag_name": "Data Science",
"confidence": 0.75,
"reasoning": "Significant data collection and analysis component"
}
]
}
Application Logic
const CONFIDENCE_THRESHOLD = 0.5
const MAX_TAGS = 5
// 1. Get AI suggestions
const { suggestions, tokensUsed } = await getAISuggestions(anonymized[0], availableTags, userId)
// 2. Filter by confidence threshold
const validSuggestions = suggestions.filter(s => s.confidence >= CONFIDENCE_THRESHOLD)
// 3. Get existing tags to avoid duplicates
const existingTagIds = new Set(project.projectTags.map(pt => pt.tagId))
// 4. Calculate remaining slots
const currentTagCount = project.projectTags.length
const remainingSlots = Math.max(0, MAX_TAGS - currentTagCount)
// 5. Filter and limit
const newSuggestions = validSuggestions
.filter(s => !existingTagIds.has(s.tagId))
.slice(0, remainingSlots)
// 6. Apply new tags (additive only, never removes existing)
for (const suggestion of newSuggestions) {
await prisma.projectTag.create({
data: {
projectId,
tagId: suggestion.tagId,
confidence: suggestion.confidence,
source: 'AI'
}
})
}
Integration Points
On Project Submission (Round 1: INTAKE):
// Auto-tag when project is submitted
if (settings.ai_tagging_enabled && settings.ai_tagging_on_submit) {
await tagProject(projectId, userId)
}
Manual Tagging (Admin Dashboard):
// Admin reviews suggestions before applying
const suggestions = await getTagSuggestions(projectId, userId)
// UI shows suggestions with confidence scores
// Admin clicks "Apply All" or selectively adds tags
for (const suggestion of selectedSuggestions) {
await addProjectTag(projectId, suggestion.tagId)
}
Batch Tagging (Round Management):
// Tag all projects in a round
const projects = await prisma.project.findMany({
where: { competitionId }
})
for (const project of projects) {
try {
await tagProject(project.id, adminUserId)
} catch (error) {
console.error(`Failed to tag project ${project.id}:`, error)
}
}
5. AI Award Eligibility Service (ai-award-eligibility.ts)
Purpose: Determine which projects are eligible for special awards using both deterministic field matching and AI interpretation of plain-language criteria.
Input Data
interface ProjectForEligibility {
id: string
title: string
description?: string | null
competitionCategory?: CompetitionCategory | null
country?: string | null
geographicZone?: string | null
tags: string[]
oceanIssue?: OceanIssue | null
institution?: string | null
foundedAt?: Date | null
wantsMentorship?: boolean
submissionSource?: SubmissionSource
submittedAt?: Date | null
_count?: {
teamMembers?: number
files?: number
}
files?: Array<{ fileType: string | null }>
}
interface SpecialAward {
id: string
name: string
description?: string | null
criteriaText?: string | null // Plain-language criteria for AI
eligibilityMode: 'STAY_IN_MAIN' | 'SEPARATE_POOL'
useAiEligibility: boolean
}
Deterministic Auto-Tag Rules
type AutoTagRule = {
field: 'competitionCategory' | 'country' | 'geographicZone' | 'tags' | 'oceanIssue'
operator: 'equals' | 'contains' | 'in'
value: string | string[]
}
// Example: "Innovation Award" for AI/ML projects
const innovationRules: AutoTagRule[] = [
{
field: 'tags',
operator: 'contains',
value: 'Artificial Intelligence'
}
]
// Example: "Regional Impact Award" for Mediterranean projects
const regionalRules: AutoTagRule[] = [
{
field: 'geographicZone',
operator: 'equals',
value: 'MEDITERRANEAN'
}
]
// Apply rules
export function applyAutoTagRules(
rules: AutoTagRule[],
projects: ProjectForEligibility[]
): Map<string, boolean> {
const results = new Map<string, boolean>()
for (const project of projects) {
const matches = rules.every(rule => {
const fieldValue = getFieldValue(project, rule.field)
// ... operator logic ...
})
results.set(project.id, matches)
}
return results
}
AI Criteria Interpretation
// Award with plain-language criteria
const award = {
name: "Youth Innovation Award",
criteriaText: `
Projects eligible for this award should:
- Be founded within the last 3 years
- Have a team with average age under 30
- Demonstrate novel technological approach
- Show strong potential for scalability
`,
useAiEligibility: true
}
// Anonymize and batch process
const { anonymized, mappings } = anonymizeProjectsForAI(projects, 'ELIGIBILITY')
const prompt = `CRITERIA: ${criteriaText}
PROJECTS: ${JSON.stringify(anonymized)}
Evaluate eligibility for each project.`
const systemPrompt = `Award eligibility evaluator. Evaluate projects against criteria, return JSON.
Format: {"evaluations": [{project_id, eligible: bool, confidence: 0-1, reasoning: str}]}
Be objective. Base evaluation only on provided data. No personal identifiers in reasoning.`
Output
interface EligibilityResult {
projectId: string
eligible: boolean
confidence: number
reasoning: string
method: 'AUTO' | 'AI'
}
// Example AI response:
{
"evaluations": [
{
"project_id": "P1",
"eligible": true,
"confidence": 0.85,
"reasoning": "Founded in 2024 (within 3 years). Novel AI approach for coral monitoring. High scalability potential."
},
{
"project_id": "P2",
"eligible": false,
"confidence": 0.90,
"reasoning": "Founded in 2015 (over 3 years ago). Does not meet youth criteria."
}
]
}
Integration with Special Awards
// Run eligibility check for an award
const award = await prisma.specialAward.findUnique({
where: { id: awardId },
include: { competition: true }
})
const projects = await prisma.project.findMany({
where: { competitionId: award.competitionId },
include: { files: true, _count: { select: { teamMembers: true } } }
})
let eligibilityResults: EligibilityResult[] = []
if (award.useAiEligibility && award.criteriaText) {
// AI interpretation
eligibilityResults = await aiInterpretCriteria(
award.criteriaText,
projects,
userId,
awardId
)
} else if (award.autoTagRules) {
// Deterministic rules
const matches = applyAutoTagRules(award.autoTagRules, projects)
eligibilityResults = Array.from(matches).map(([projectId, eligible]) => ({
projectId,
eligible,
confidence: 1.0,
reasoning: 'Matches auto-tag rules',
method: 'AUTO'
}))
}
// Store results
for (const result of eligibilityResults) {
await prisma.awardEligibility.upsert({
where: {
specialAwardId_projectId: { specialAwardId: awardId, projectId: result.projectId }
},
create: {
specialAwardId: awardId,
projectId: result.projectId,
isEligible: result.eligible,
confidence: result.confidence,
reasoning: result.reasoning,
method: result.method
},
update: {
isEligible: result.eligible,
confidence: result.confidence,
reasoning: result.reasoning,
method: result.method
}
})
}
6. Anonymization Service (anonymization.ts)
Purpose: Strip PII from all data before sending to OpenAI. Ensures GDPR compliance and privacy protection.
PII Patterns Detected
const PII_PATTERNS = {
email: /[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/g,
phone: /(\+?\d{1,3}[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}/g,
url: /https?:\/\/[^\s]+/g,
ssn: /\d{3}-\d{2}-\d{4}/g,
ipv4: /\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/g
}
export function sanitizeText(text: string): string {
let sanitized = text
sanitized = sanitized.replace(PII_PATTERNS.email, '[email removed]')
sanitized = sanitized.replace(PII_PATTERNS.phone, '[phone removed]')
sanitized = sanitized.replace(PII_PATTERNS.url, '[url removed]')
sanitized = sanitized.replace(PII_PATTERNS.ssn, '[id removed]')
return sanitized
}
Description Truncation
export const DESCRIPTION_LIMITS = {
ASSIGNMENT: 300, // Short context for matching
FILTERING: 500, // Medium context for screening
ELIGIBILITY: 400, // Medium context for criteria
MENTOR: 350 // Short context for mentor matching
}
export function truncateAndSanitize(
text: string | null | undefined,
maxLength: number
): string {
if (!text) return ''
const sanitized = sanitizeText(text)
if (sanitized.length <= maxLength) return sanitized
return sanitized.slice(0, maxLength - 3) + '...'
}
ID Replacement Strategy
Before Anonymization:
// Real data with identifiable IDs
{
id: "proj-abc123xyz",
userId: "user-456def789",
title: "SaveTheSea by Marine Research Institute",
teamName: "Dr. John Doe's Lab",
description: "Contact us at john@example.com or +1-555-1234 for more info. Visit https://savethesea.org"
}
After Anonymization:
// Anonymous IDs and sanitized text
{
project_id: "P1",
title: "SaveTheSea",
description: "Contact us at [email removed] or [phone removed] for more info. Visit [url removed]",
team_size: 3,
founded_year: 2024,
tags: ["Marine Biology", "AI"]
}
Mapping for De-anonymization:
const mappings: ProjectAIMapping[] = [
{ anonymousId: "P1", realId: "proj-abc123xyz" },
{ anonymousId: "P2", realId: "proj-def456uvw" }
]
GDPR Validation
export interface PIIValidationResult {
valid: boolean
violations: string[]
}
export function validateNoPersonalData(
data: Record<string, unknown>
): PIIValidationResult {
const violations: string[] = []
const textContent = JSON.stringify(data)
// Check PII patterns
for (const [type, pattern] of Object.entries(PII_PATTERNS)) {
pattern.lastIndex = 0
if (pattern.test(textContent)) {
violations.push(`Potential ${type} detected in data`)
}
}
// Check sensitive field names
const sensitiveFields = ['email', 'phone', 'password', 'ssn', 'creditCard', 'bankAccount']
const keys = Object.keys(data).map(k => k.toLowerCase())
for (const field of sensitiveFields) {
if (keys.includes(field)) {
violations.push(`Sensitive field "${field}" present in data`)
}
}
return {
valid: violations.length === 0,
violations
}
}
// Enforce before EVERY AI call
export function enforceGDPRCompliance(data: unknown[]): void {
for (let i = 0; i < data.length; i++) {
const { valid, violations } = validateNoPersonalData(item)
if (!valid) {
throw new Error(`GDPR compliance check failed: ${violations.join(', ')}`)
}
}
}
Anonymization Workflow
graph TD
A[Raw Project Data] --> B[Convert to ProjectWithRelations]
B --> C[Truncate Descriptions]
C --> D[Sanitize Text Fields]
D --> E[Replace IDs with Anonymous IDs]
E --> F[Create Mappings]
F --> G[Validate No PII]
G --> H{Valid?}
H -->|Yes| I[Send to AI]
H -->|No| J[Throw GDPR Error]
I --> K[AI Response]
K --> L[De-anonymize Results]
L --> M[Return Real IDs]
New AI Services for Redesign
1. AI Mentoring Insights (Round 6: MENTORING)
Purpose: Summarize mentor-team interactions, flag inactive workspaces, suggest intervention points.
Input Data
interface MentorWorkspaceData {
projectId: string
mentorId: string
workspaceOpenAt: Date
files: MentorFile[]
messages: MentorMessage[]
fileComments: MentorFileComment[]
lastActivityAt: Date
}
Anonymization
// Strip mentor/team names, keep activity patterns
{
workspace_id: "W1",
days_active: 14,
file_count: 3,
message_count: 12,
comment_count: 5,
last_activity_days_ago: 2,
file_types: ["PDF", "DOCX"],
message_frequency: "daily",
engagement_level: "high"
}
Prompt
const prompt = `Analyze mentor workspace activity and provide insights.
WORKSPACE DATA: ${JSON.stringify(anonymizedWorkspace)}
Return JSON:
{
"engagement_level": "high" | "medium" | "low" | "inactive",
"key_insights": ["insight 1", "insight 2"],
"red_flags": ["flag 1", "flag 2"],
"recommendations": ["recommendation 1", "recommendation 2"],
"intervention_needed": boolean
}`
Output
interface MentoringInsight {
workspaceId: string
engagementLevel: 'high' | 'medium' | 'low' | 'inactive'
keyInsights: string[]
redFlags: string[]
recommendations: string[]
interventionNeeded: boolean
}
Example:
{
"engagement_level": "low",
"key_insights": [
"No activity in last 7 days",
"Only 1 file uploaded (below average for this stage)",
"Message frequency dropped from daily to weekly"
],
"red_flags": [
"Mentor has not responded to last 2 team messages",
"No progress on business plan deliverable (due in 5 days)"
],
"recommendations": [
"Admin should check in with mentor",
"Send reminder notification to team about upcoming deadline",
"Consider pairing with secondary mentor for support"
],
"intervention_needed": true
}
Integration
// Nightly cron job: analyze all active mentoring workspaces
const mentoringRound = await prisma.round.findFirst({
where: { competitionId, roundType: 'MENTORING', status: 'ROUND_ACTIVE' }
})
const assignments = await prisma.mentorAssignment.findMany({
where: { workspaceEnabled: true },
include: { files: true, messages: true }
})
for (const assignment of assignments) {
const insight = await generateMentoringInsight(assignment, userId)
if (insight.interventionNeeded) {
await prisma.inAppNotification.create({
data: {
userId: adminId,
title: 'Mentoring Intervention Needed',
message: `Workspace for project "${projectTitle}" requires attention`,
type: 'MENTORING_ALERT',
actionUrl: `/admin/mentoring/${assignment.id}`
}
})
}
}
2. AI Duplicate Detection Enhancement (Round 2: FILTERING)
Purpose: Detect duplicate or near-duplicate submissions using embeddings and semantic similarity.
Current Approach (Simple)
// Existing: basic title/email matching
const duplicates = await prisma.project.findMany({
where: {
OR: [
{ title: { equals: project.title, mode: 'insensitive' } },
{ teamMembers: { some: { email: applicantEmail } } }
]
}
})
Enhanced Approach (AI-Powered)
// Generate embeddings for all projects
const embeddings = new Map<string, number[]>()
for (const project of projects) {
const embedding = await openai.embeddings.create({
model: 'text-embedding-3-small',
input: `${project.title}\n\n${project.description}`
})
embeddings.set(project.id, embedding.data[0].embedding)
}
// Compute cosine similarity matrix
function cosineSimilarity(a: number[], b: number[]): number {
const dot = a.reduce((sum, val, i) => sum + val * b[i], 0)
const magA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0))
const magB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0))
return dot / (magA * magB)
}
// Find duplicates
const SIMILARITY_THRESHOLD = 0.85
for (const [id1, emb1] of embeddings) {
for (const [id2, emb2] of embeddings) {
if (id1 >= id2) continue // Skip self and already compared
const similarity = cosineSimilarity(emb1, emb2)
if (similarity >= SIMILARITY_THRESHOLD) {
await prisma.duplicateDetection.create({
data: {
projectId1: id1,
projectId2: id2,
similarity,
method: 'AI_EMBEDDING',
status: 'FLAGGED'
}
})
}
}
}
Output
interface DuplicateMatch {
projectId1: string
projectId2: string
similarity: number
method: 'EXACT_TITLE' | 'EMAIL_MATCH' | 'AI_EMBEDDING'
status: 'FLAGGED' | 'CONFIRMED_DUPLICATE' | 'FALSE_POSITIVE'
}
Admin Review UI:
┌─────────────────────────────────────────────────────────────────┐
│ Duplicate Detection: 3 Potential Matches │
├─────────────────────────────────────────────────────────────────┤
│ │
│ [!] 92% Match - AI Coral Monitoring (P1) vs Ocean AI (P2) │
│ Method: AI Embedding │
│ ┌─ P1: SaveTheSea - AI-powered coral reef monitoring... │
│ └─ P2: Ocean AI - Machine learning platform for coral... │
│ │
│ [ Mark as Duplicate ] [ False Positive ] [ View Details ] │
│ │
├─────────────────────────────────────────────────────────────────┤
│ │
│ [!] 88% Match - Marine Plastic (P3) vs PlasticClean (P4) │
│ Method: AI Embedding + Email Match │
│ ┌─ P3: Marine Plastic Cleanup - Robotic system... │
│ └─ P4: PlasticClean - Automated plastic removal... │
│ │
│ [ Mark as Duplicate ] [ False Positive ] [ View Details ] │
│ │
└─────────────────────────────────────────────────────────────────┘
3. AI Confirmation Helper (Round 8: CONFIRMATION)
Purpose: Generate plain-language explanations for why final rankings are what they are. Helps jury understand and validate the winner proposal.
Input Data
interface WinnerProposalData {
rankedProjectIds: string[]
sourceRoundId: string
selectionBasis: {
method: 'LIVE_VOTE' | 'EVALUATION_SCORES' | 'ADMIN_DECISION'
scores: Record<string, number>
aiRecommendation?: string
}
}
// Fetch all context
const projects = await prisma.project.findMany({
where: { id: { in: proposal.rankedProjectIds } },
include: {
evaluations: { where: { roundId: sourceRoundId } },
liveVotes: { where: { roundId: sourceRoundId } }
}
})
Anonymization
// Strip project names, keep relative rankings and scores
{
ranking: [
{ position: 1, anonymous_id: "P1", score: 8.9, vote_count: 12 },
{ position: 2, anonymous_id: "P2", score: 8.5, vote_count: 10 },
{ position: 3, anonymous_id: "P3", score: 8.2, vote_count: 11 }
],
selection_method: "LIVE_VOTE",
score_gaps: [0.4, 0.3],
consensus_level: 0.88
}
Prompt
const prompt = `Explain the final ranking in plain language for jury review.
RANKING DATA: ${JSON.stringify(anonymizedRanking)}
EVALUATION HISTORY: ${JSON.stringify(anonymizedEvaluations)}
Return JSON:
{
"summary": "2-3 sentence plain-language summary of why this ranking makes sense",
"position_explanations": [
{ "position": 1, "why_this_rank": "explanation", "strengths": ["s1", "s2"] },
...
],
"close_calls": [
{ "positions": [2, 3], "gap": 0.3, "reasoning": "why this was close" }
],
"confidence": "high" | "medium" | "low",
"red_flags": ["flag 1", ...]
}`
Output
interface ConfirmationExplanation {
summary: string
positionExplanations: Array<{
position: number
projectId: string
whyThisRank: string
strengths: string[]
}>
closeCalls: Array<{
positions: number[]
gap: number
reasoning: string
}>
confidence: 'high' | 'medium' | 'low'
redFlags: string[]
}
Example:
{
"summary": "The ranking reflects clear consensus from both live voting (12 votes) and prior evaluations (avg 8.9). Top position is well-separated with 0.4 point gap. Second and third places are close (0.3 gap), but consistent across jury members.",
"position_explanations": [
{
"position": 1,
"why_this_rank": "Highest average score (8.9) across all evaluation rounds. Won live vote with 12/15 jury members. Consistent top performer.",
"strengths": ["Technical innovation", "Strong team", "Commercial viability", "Scalability"]
},
{
"position": 2,
"why_this_rank": "Strong runner-up (8.5 avg). Close to third place but slight edge in feasibility scores.",
"strengths": ["Proven track record", "Regulatory compliance", "Budget realism"]
},
{
"position": 3,
"why_this_rank": "Solid finalist (8.2 avg). Slightly lower technical feasibility scores compared to #2.",
"strengths": ["Impact potential", "Novel approach", "Jury enthusiasm"]
}
],
"close_calls": [
{
"positions": [2, 3],
"gap": 0.3,
"reasoning": "Very close race. Position 2 had slight edge in 'Feasibility' criterion (7.8 vs 7.2), while Position 3 scored higher in 'Impact' (8.9 vs 8.4). Live vote split 10-11. Admin may want to review deliberation notes."
}
],
"confidence": "high",
"red_flags": []
}
Integration:
// When admin creates WinnerProposal
const proposal = await prisma.winnerProposal.create({
data: {
competitionId,
category: 'STARTUP',
rankedProjectIds: ['proj-1', 'proj-2', 'proj-3'],
sourceRoundId: liveFinalsRoundId,
selectionBasis: { method: 'LIVE_VOTE', scores: {...} },
proposedById: adminId
}
})
// Generate AI explanation
const explanation = await generateConfirmationExplanation(proposal.id, userId)
// Attach to proposal
await prisma.winnerProposal.update({
where: { id: proposal.id },
data: {
selectionBasis: {
...proposal.selectionBasis,
aiExplanation: explanation
}
}
})
// Show to jury during confirmation
Confirmation UI with AI Explanation:
┌─────────────────────────────────────────────────────────────────┐
│ Confirm Final Rankings: STARTUP Category │
├─────────────────────────────────────────────────────────────────┤
│ │
│ AI Analysis: High Confidence │
│ "The ranking reflects clear consensus from both live voting │
│ (12 votes) and prior evaluations (avg 8.9). Top position is │
│ well-separated with 0.4 point gap..." │
│ │
│ ┌───────────────────────────────────────────────────────────────┐ │
│ │ 1st Place: SaveTheSea (Score: 8.9, 12 votes) │ │
│ │ Why This Rank: Highest average score across all rounds... │ │
│ │ Strengths: Technical innovation, Strong team, Commercial... │ │
│ └───────────────────────────────────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────────────┐ │
│ │ 2nd Place: Ocean AI (Score: 8.5, 10 votes) │ │
│ │ Why This Rank: Strong runner-up. Close to third place... │ │
│ │ Strengths: Proven track record, Regulatory compliance... │ │
│ └───────────────────────────────────────────────────────────────┘ │
│ │
│ [!] Close Call: 2nd and 3rd place (0.3 gap) │
│ Position 2 had slight edge in Feasibility (7.8 vs 7.2)... │
│ │
│ [ ] I confirm this ranking (John Doe, Jury Lead) │
│ [ ] I confirm this ranking (Jane Smith, Jury Member) │
│ [ ] I confirm this ranking (Bob Wilson, Jury Member) │
│ │
│ Comments: ___________________________________________________ │
│ │
│ [ Approve Ranking ] [ Request Changes ] │
│ │
└─────────────────────────────────────────────────────────────────┘
Anonymization Pipeline (Deep Dive)
Data Flow
graph LR
A[Raw Data] --> B[Type Conversion]
B --> C[Description Truncation]
C --> D[Text Sanitization]
D --> E[ID Replacement]
E --> F[Field Filtering]
F --> G[Validation]
G --> H{GDPR Pass?}
H -->|Yes| I[To AI]
H -->|No| J[Error + Log]
I --> K[AI Response]
K --> L[De-anonymization]
L --> M[Real Results]
What Gets Stripped
| Data Type | Before | After |
|---|---|---|
| User IDs | user-abc123xyz |
juror_001 |
| Project IDs | proj-def456uvw |
P1 |
| Names | Dr. John Doe |
(removed entirely) |
| Emails | john@example.com |
[email removed] |
| Phone Numbers | +1-555-1234 |
[phone removed] |
| URLs | https://example.com |
[url removed] |
| Team Names | Marine Research Institute |
Team 1 |
| Descriptions | Full 2000-char text | Truncated to 300-500 chars |
| Dates | 2024-02-15T14:30:00Z |
2024 (year only) or 2024-02-15 (date only) |
Replacement Strategy
Sequential Anonymous IDs:
- Projects:
P1,P2,P3, ... - Jurors:
juror_001,juror_002,juror_003, ... - Workspaces:
W1,W2,W3, ...
Rationale: Sequential IDs are easier for AI to reference and for humans to debug. Random UUIDs would be harder to correlate in logs.
Generic Team Names:
Team 1,Team 2,Team 3, ...
Rationale: Preserves the concept of "team" without revealing institutional affiliation.
De-anonymization Mapping
// Stored in memory during AI call (never persisted to DB)
const mappings: ProjectAIMapping[] = [
{ anonymousId: "P1", realId: "proj-abc123xyz" },
{ anonymousId: "P2", realId: "proj-def456uvw" },
{ anonymousId: "P3", realId: "proj-ghi789rst" }
]
// After AI returns results for "P1", map back to real ID
const realProjectId = mappings.find(m => m.anonymousId === "P1")?.realId
Anonymization for Different Data Types
Projects
// INPUT (ProjectWithRelations)
{
id: "proj-abc123",
title: "SaveTheSea - Marine Conservation by Dr. John Doe",
description: "Contact john@example.com or call +1-555-1234. Visit https://savethesea.org for more info.",
teamName: "Marine Research Institute",
competitionCategory: "STARTUP",
country: "France",
geographicZone: "MEDITERRANEAN",
tags: ["Marine Biology", "AI", "Conservation"],
foundedAt: new Date("2024-01-15T00:00:00Z"),
institution: "University of Monaco",
wantsMentorship: true,
submissionSource: "WEB_FORM",
submittedAt: new Date("2024-02-15T14:30:00Z"),
_count: { teamMembers: 3, files: 5 },
files: [
{ fileType: "PDF" },
{ fileType: "PDF" },
{ fileType: "DOCX" }
]
}
// OUTPUT (AnonymizedProjectForAI)
{
project_id: "P1",
title: "SaveTheSea", // PII stripped
description: "Contact [email removed] or call [phone removed]. Visit [url removed] for more info.", // Sanitized + truncated
category: "STARTUP",
ocean_issue: null,
country: "France",
region: "MEDITERRANEAN",
institution: "University of Monaco", // Kept (non-PII institutional data)
tags: ["Marine Biology", "AI", "Conservation"],
founded_year: 2024, // Date reduced to year only
team_size: 3,
has_description: true,
file_count: 5,
file_types: ["PDF", "DOCX"],
wants_mentorship: true,
submission_source: "WEB_FORM",
submitted_date: "2024-02-15" // Date only, no time
}
Jurors
// INPUT (JurorForAssignment)
{
id: "user-456def",
name: "Dr. Jane Smith",
email: "jane.smith@university.edu",
expertiseTags: ["Marine Biology", "Oceanography"],
maxAssignments: 20,
_count: { assignments: 5 }
}
// OUTPUT (AnonymizedJuror)
{
anonymousId: "juror_001",
expertiseTags: ["Marine Biology", "Oceanography"],
currentAssignmentCount: 5,
maxAssignments: 20
// Name and email completely removed
}
Evaluations
// INPUT (EvaluationForSummary)
{
id: "eval-789ghi",
criterionScoresJson: { "Innovation": 9, "Feasibility": 7, "Impact": 8 },
globalScore: 8,
binaryDecision: true,
feedbackText: "Excellent proposal from Dr. John Doe (john@example.com). Strong technical approach but budget seems optimistic. Call me at +1-555-9876 to discuss.",
assignment: {
user: {
id: "user-123abc",
name: "Dr. Alice Johnson",
email: "alice@university.edu"
}
}
}
// OUTPUT (AnonymizedEvaluation)
{
criterionScores: { "Innovation": 9, "Feasibility": 7, "Impact": 8 },
globalScore: 8,
binaryDecision: true,
feedbackText: "Excellent proposal. Strong technical approach but budget seems optimistic. Call me at [phone removed] to discuss."
// User info completely removed
// PII sanitized from feedback
}
GDPR Validation Workflow
// 1. Anonymize data
const { anonymized, mappings } = anonymizeProjectsForAI(projects, 'FILTERING')
// 2. Validate BEFORE sending to AI
if (!validateAnonymizedProjects(anonymized)) {
console.error('[AI Service] GDPR validation failed')
// Log violation
await prisma.auditLog.create({
data: {
action: 'AI_GDPR_VIOLATION',
entityType: 'AIService',
details: { service: 'filtering', error: 'PII detected in anonymized data' }
}
})
throw new Error('GDPR compliance check failed: PII detected in anonymized data')
}
// 3. Only if validation passes, send to AI
const response = await openai.chat.completions.create(params)
// 4. De-anonymize results
const realResults = mappings.map(mapping => {
const aiResult = response.find(r => r.project_id === mapping.anonymousId)
return {
projectId: mapping.realId,
...aiResult
}
})
Prompt Engineering
System Prompt Templates
Filtering Prompt
const AI_SCREENING_SYSTEM_PROMPT = `Project screening assistant. Evaluate against criteria, return JSON.
Format: {"projects": [{project_id, meets_criteria: bool, confidence: 0-1, reasoning: str, quality_score: 1-10, spam_risk: bool}]}
Be objective. Base evaluation only on provided data. No personal identifiers in reasoning.`
Design Principles:
- Compact: 3 sentences instead of 10 paragraphs (saves tokens)
- JSON Schema: Explicit output format prevents parsing errors
- Objectivity: Reminds AI to avoid bias
- Privacy: Explicitly forbids personal identifiers in output
Assignment Prompt
const ASSIGNMENT_SYSTEM_PROMPT = `Match jurors to projects by expertise. Return JSON assignments.
Each: {juror_id, project_id, confidence_score: 0-1, expertise_match_score: 0-1, reasoning: str (1-2 sentences)}
Distribute workload fairly. Avoid assigning jurors at capacity.`
Design Principles:
- Constraint Awareness: Reminds AI about workload balancing
- Brevity Requirement: "1-2 sentences" for reasoning (saves tokens)
- Dual Scores: Separates confidence (how sure AI is) from expertise match (how good the fit is)
Evaluation Summary Prompt
const EVALUATION_SUMMARY_PROMPT = `You are analyzing jury evaluations for a project competition.
PROJECT: "${sanitizedTitle}"
EVALUATION CRITERIA: ${criteriaLabels.join(', ')}
EVALUATIONS (${count} total):
${JSON.stringify(anonymizedEvaluations, null, 2)}
Analyze these evaluations and return a JSON object with this exact structure:
{
"overallAssessment": "A 2-3 sentence summary of how the project was evaluated overall",
"strengths": ["strength 1", "strength 2", ...],
"weaknesses": ["weakness 1", "weakness 2", ...],
"themes": [
{ "theme": "theme name", "sentiment": "positive" | "negative" | "mixed", "frequency": <number of evaluators mentioning this> }
],
"recommendation": "A brief recommendation based on the evaluation consensus"
}
Guidelines:
- Base your analysis only on the provided evaluation data
- Identify common themes across evaluator feedback
- Note areas of agreement and disagreement
- Keep the assessment objective and balanced
- Do not include any personal identifiers`
Design Principles:
- Structured Output: JSON schema enforces consistency
- Sentiment Analysis: Themes include positive/negative/mixed classification
- Consensus Detection: Frequency count shows how many evaluators mentioned each theme
- Explicit Guidelines: 5 bullet points ensure quality output
Dynamic Context Injection
// Build prompts dynamically based on available data
function buildFilteringPrompt(
criteriaText: string,
projects: AnonymizedProjectForAI[],
context?: {
previousRound?: { passRate: number },
categoryDistribution?: { STARTUP: number, BUSINESS_CONCEPT: number }
}
): string {
let prompt = `CRITERIA: ${criteriaText}\nPROJECTS: ${JSON.stringify(projects)}\n`
if (context?.previousRound) {
prompt += `\nPREVIOUS ROUND: Pass rate was ${context.previousRound.passRate}%. Apply similar rigor.\n`
}
if (context?.categoryDistribution) {
prompt += `\nCATEGORY BALANCE: Aim for roughly ${context.categoryDistribution.STARTUP}% startups, ${context.categoryDistribution.BUSINESS_CONCEPT}% concepts.\n`
}
prompt += `\nEvaluate and return JSON.`
return prompt
}
Output Format Enforcement
// Use JSON mode for all AI calls
const params = buildCompletionParams(model, {
messages: [
{ role: 'system', content: systemPrompt },
{ role: 'user', content: userPrompt }
],
jsonMode: true, // Enforces JSON output
temperature: 0.3,
maxTokens: 4000
})
// Validate response structure
const parsed = JSON.parse(content) as ExpectedStructure
if (!isValidStructure(parsed)) {
throw createParseError('AI returned invalid JSON structure')
}
Token Limit Management
// Description limits by context
export const DESCRIPTION_LIMITS = {
ASSIGNMENT: 300, // Short - only need general topic
FILTERING: 500, // Medium - need enough detail for quality assessment
ELIGIBILITY: 400, // Medium - need criteria matching
MENTOR: 350 // Short - expertise matching
}
// Calculate estimated token count before API call
function estimateTokens(text: string): number {
// Rough estimate: 1 token ≈ 4 characters
return Math.ceil(text.length / 4)
}
// Warn if approaching limits
const promptTokens = estimateTokens(userPrompt)
if (promptTokens > 3000) {
console.warn(`[AI Service] Large prompt detected: ${promptTokens} tokens`)
}
// Set maxTokens based on expected output
const maxTokens = {
FILTERING: 4000, // Batch of 20 projects
ASSIGNMENT: 4000, // Batch of 15 projects
SUMMARY: 2000, // Single summary
TAGGING: 1000 // Tag suggestions
}[serviceType]
Retry and Fallback Strategies
async function callAIWithRetry<T>(
apiCall: () => Promise<T>,
fallback: () => T,
maxRetries = 3
): Promise<T> {
let lastError: Error
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
return await apiCall()
} catch (error) {
lastError = error
const classified = classifyAIError(error)
// Don't retry on validation errors or rate limits
if (classified.type === 'VALIDATION' || classified.type === 'RATE_LIMIT') {
break
}
// Wait before retry (exponential backoff)
if (attempt < maxRetries) {
await new Promise(resolve => setTimeout(resolve, 1000 * Math.pow(2, attempt)))
}
}
}
// Fall back to algorithm
console.warn(`[AI Service] Failed after ${maxRetries} attempts, using fallback`, lastError)
return fallback()
}
OpenAI Integration
Model Selection
// lib/openai.ts
export const AI_MODELS = {
STRONG: 'gpt-4', // Complex reasoning (filtering, assignment)
QUICK: 'gpt-4-turbo', // Fast analysis (summaries, tagging)
CHEAP: 'gpt-3.5-turbo', // Simple tasks (deprecated for this platform)
EMBEDDING: 'text-embedding-3-small' // For duplicate detection
}
// Get configured model from settings
export async function getConfiguredModel(preference?: string): Promise<string> {
const settings = await prisma.systemSettings.findUnique({
where: { key: 'ai_model' }
})
const model = preference || settings?.value || AI_MODELS.STRONG
// Validate model exists
if (!Object.values(AI_MODELS).includes(model)) {
console.warn(`[OpenAI] Unknown model "${model}", falling back to ${AI_MODELS.STRONG}`)
return AI_MODELS.STRONG
}
return model
}
Model Usage by Service:
| Service | Model | Rationale |
|---|---|---|
| Filtering | gpt-4 |
Needs strong reasoning for nuanced criteria interpretation |
| Assignment | gpt-4 |
Complex constraint balancing (expertise + load + COI) |
| Summary | gpt-4-turbo |
Fast synthesis, less reasoning required |
| Tagging | gpt-4-turbo |
Simple categorization task |
| Eligibility | gpt-4 |
Plain-language criteria need strong comprehension |
| Mentoring Insights | gpt-4-turbo |
Pattern detection, less critical reasoning |
Rate Limiting
// Rate limiter middleware
class AIRateLimiter {
private requestQueue: Array<() => Promise<unknown>> = []
private isProcessing = false
private requestsPerMinute = 60 // GPT-4 tier 1 limit
async add<T>(fn: () => Promise<T>): Promise<T> {
return new Promise((resolve, reject) => {
this.requestQueue.push(async () => {
try {
const result = await fn()
resolve(result)
} catch (error) {
reject(error)
}
})
if (!this.isProcessing) {
this.process()
}
})
}
private async process() {
this.isProcessing = true
while (this.requestQueue.length > 0) {
const fn = this.requestQueue.shift()!
await fn()
// Wait to respect rate limit
await new Promise(resolve => setTimeout(resolve, 60000 / this.requestsPerMinute))
}
this.isProcessing = false
}
}
const rateLimiter = new AIRateLimiter()
// Use in services
const result = await rateLimiter.add(() => openai.chat.completions.create(params))
Cost Tracking
// Token pricing (as of 2024)
const TOKEN_PRICING = {
'gpt-4': { input: 0.03, output: 0.06 }, // per 1K tokens
'gpt-4-turbo': { input: 0.01, output: 0.03 },
'text-embedding-3-small': { input: 0.0001, output: 0 }
}
// Calculate cost
function calculateCost(
model: string,
promptTokens: number,
completionTokens: number
): number {
const pricing = TOKEN_PRICING[model] || TOKEN_PRICING['gpt-4']
const inputCost = (promptTokens / 1000) * pricing.input
const outputCost = (completionTokens / 1000) * pricing.output
return inputCost + outputCost
}
// Log usage with cost
await logAIUsage({
userId,
action: 'FILTERING',
model,
promptTokens,
completionTokens,
totalTokens,
estimatedCost: calculateCost(model, promptTokens, completionTokens),
status: 'SUCCESS'
})
// Admin dashboard: show cumulative costs
const totalCost = await prisma.aIUsageLog.aggregate({
where: { createdAt: { gte: startOfMonth } },
_sum: { estimatedCost: true }
})
Error Handling
// ai-errors.ts
export type AIErrorType =
| 'RATE_LIMIT'
| 'AUTHENTICATION'
| 'NETWORK'
| 'TIMEOUT'
| 'VALIDATION'
| 'PARSE_ERROR'
| 'MODEL_NOT_FOUND'
| 'UNKNOWN'
export interface ClassifiedAIError {
type: AIErrorType
message: string
retryable: boolean
userMessage: string
}
export function classifyAIError(error: unknown): ClassifiedAIError {
const errorMsg = error instanceof Error ? error.message : String(error)
// Rate limit
if (errorMsg.includes('rate_limit') || errorMsg.includes('429')) {
return {
type: 'RATE_LIMIT',
message: errorMsg,
retryable: true,
userMessage: 'AI service is currently at capacity. Please try again in a few minutes.'
}
}
// Authentication
if (errorMsg.includes('authentication') || errorMsg.includes('401') || errorMsg.includes('api key')) {
return {
type: 'AUTHENTICATION',
message: errorMsg,
retryable: false,
userMessage: 'OpenAI API key is invalid or missing. Please check Settings > AI Configuration.'
}
}
// Model not found
if (errorMsg.includes('model') && errorMsg.includes('does not exist')) {
return {
type: 'MODEL_NOT_FOUND',
message: errorMsg,
retryable: false,
userMessage: 'The configured AI model does not exist. Please check Settings > AI Configuration.'
}
}
// Network
if (errorMsg.includes('ECONNREFUSED') || errorMsg.includes('ETIMEDOUT') || errorMsg.includes('network')) {
return {
type: 'NETWORK',
message: errorMsg,
retryable: true,
userMessage: 'Unable to reach OpenAI servers. Please check your internet connection.'
}
}
// Timeout
if (errorMsg.includes('timeout')) {
return {
type: 'TIMEOUT',
message: errorMsg,
retryable: true,
userMessage: 'AI request timed out. Please try again or reduce batch size.'
}
}
// Validation
if (errorMsg.includes('validation') || errorMsg.includes('invalid')) {
return {
type: 'VALIDATION',
message: errorMsg,
retryable: false,
userMessage: 'Invalid request to AI service. Please contact support.'
}
}
// Parse error
if (error instanceof SyntaxError || errorMsg.includes('parse') || errorMsg.includes('JSON')) {
return {
type: 'PARSE_ERROR',
message: errorMsg,
retryable: false,
userMessage: 'AI response could not be understood. This has been logged for review.'
}
}
// Unknown
return {
type: 'UNKNOWN',
message: errorMsg,
retryable: true,
userMessage: 'An unexpected error occurred with the AI service. Please try again.'
}
}
export function logAIError(service: string, operation: string, error: ClassifiedAIError): void {
console.error(`[AI Error] ${service}.${operation}:`, {
type: error.type,
message: error.message,
retryable: error.retryable
})
// Log to database for admin review
prisma.auditLog.create({
data: {
action: 'AI_ERROR',
entityType: 'AIService',
details: {
service,
operation,
errorType: error.type,
message: error.message
}
}
}).catch(console.error)
}
Streaming vs Non-Streaming
Current Implementation: Non-streaming (wait for full response)
const response = await openai.chat.completions.create({
model: 'gpt-4',
messages: [...],
stream: false // Wait for complete response
})
const content = response.choices[0]?.message?.content
Future Enhancement: Streaming for long summaries
// For evaluation summaries, stream to UI as tokens arrive
const stream = await openai.chat.completions.create({
model: 'gpt-4',
messages: [...],
stream: true
})
for await (const chunk of stream) {
const delta = chunk.choices[0]?.delta?.content || ''
// Send to frontend via SSE
res.write(`data: ${JSON.stringify({ delta })}\n\n`)
}
Privacy & Security
Data Minimization
Principle: Send only the minimum data required for the AI task.
| Service | Data Sent | Data NOT Sent |
|---|---|---|
| Filtering | Project title, description (truncated), category, tags, file types, team size | Team names, emails, phone numbers, full descriptions, user IDs |
| Assignment | Juror expertise tags, current assignment count, max assignments, project tags | Juror names, emails, user IDs, juror institutions |
| Summary | Evaluation scores, sanitized feedback text | Juror names, emails, user IDs |
| Tagging | Project title, description (truncated), category, tags | Team names, emails, phone numbers, user IDs |
| Eligibility | Project category, tags, country, founded year | Team names, emails, phone numbers, full descriptions |
PII Stripping Workflow
graph TD
A[Raw Data] --> B{Contains PII?}
B -->|Yes| C[Sanitize Text]
B -->|No| D[Validate]
C --> E[Replace IDs]
E --> F[Truncate Descriptions]
F --> D
D --> G{GDPR Pass?}
G -->|No| H[Reject + Log]
G -->|Yes| I[Send to AI]
I --> J[AI Response]
J --> K[De-anonymize]
K --> L[Return Real Results]
Audit Logging
// All AI requests logged with full context
await logAIUsage({
userId: 'admin-123',
action: 'FILTERING',
entityType: 'Round',
entityId: 'round-456',
model: 'gpt-4',
promptTokens: 1234,
completionTokens: 567,
totalTokens: 1801,
batchSize: 20,
itemsProcessed: 20,
status: 'SUCCESS',
estimatedCost: 0.0891,
metadata: {
criteriaText: 'Reject spam projects...',
passRate: 0.75
}
})
// Failed requests also logged
await logAIUsage({
userId: 'admin-123',
action: 'ASSIGNMENT',
model: 'gpt-4',
status: 'ERROR',
errorMessage: 'Rate limit exceeded',
metadata: {
jurorCount: 15,
projectCount: 45
}
})
Data Retention Policy
// AI usage logs retained for 90 days, then archived
export async function archiveOldAILogs(): Promise<void> {
const cutoffDate = new Date()
cutoffDate.setDate(cutoffDate.getDate() - 90)
const oldLogs = await prisma.aIUsageLog.findMany({
where: { createdAt: { lt: cutoffDate } }
})
// Move to archive storage (S3, cold storage, etc.)
await archiveService.store('ai-logs', oldLogs)
// Delete from hot database
await prisma.aIUsageLog.deleteMany({
where: { createdAt: { lt: cutoffDate } }
})
}
// Run nightly via cron
Opt-Out Capabilities
// Per-service opt-out in settings
interface AISettings {
ai_enabled: boolean // Master toggle
ai_filtering_enabled: boolean
ai_assignment_enabled: boolean
ai_summary_enabled: boolean
ai_tagging_enabled: boolean
ai_eligibility_enabled: boolean
}
// Check before each AI call
export async function isAIServiceEnabled(service: AIServiceType): Promise<boolean> {
const masterEnabled = await getSystemSetting('ai_enabled')
if (!masterEnabled) return false
const serviceEnabled = await getSystemSetting(`ai_${service}_enabled`)
return serviceEnabled !== 'false'
}
// Graceful fallback when disabled
if (!await isAIServiceEnabled('assignment')) {
return generateFallbackAssignments(jurors, projects, constraints)
}
Admin AI Controls
AI Configuration Dashboard
┌────────────────────────────────────────────────────────────────────┐
│ Settings > AI Configuration │
├────────────────────────────────────────────────────────────────────┤
│ │
│ General Settings │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ OpenAI API Key: [sk-proj-••••••••••••••••••••••••••] [ Test ] │
│ ✓ Connected (last verified: 2024-02-15 14:30) │
│ │
│ Default Model: [ gpt-4 ▼ ] │
│ Estimated cost: $0.03 per 1K tokens (input) │
│ │
│ [ ] Enable AI Services ← Master toggle │
│ │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ Service-Specific Settings │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ [✓] AI Filtering │
│ Batch Size: [ 20 ] projects per API call │
│ Parallel Batches: [ 2 ] concurrent requests │
│ Model Override: [ (use default) ▼ ] │
│ │
│ [✓] AI Assignment │
│ Batch Size: [ 15 ] projects per API call │
│ Model Override: [ (use default) ▼ ] │
│ │
│ [✓] AI Evaluation Summaries │
│ Auto-generate: [✓] When all evaluations submitted │
│ Model Override: [ gpt-4-turbo ▼ ] (faster for summaries) │
│ │
│ [✓] AI Tagging │
│ Confidence Threshold: [ 0.5 ] (0.0 - 1.0) │
│ Max Tags per Project: [ 5 ] │
│ Auto-tag on Submit: [✓] Apply tags when project submitted │
│ │
│ [✓] AI Award Eligibility │
│ Model Override: [ (use default) ▼ ] │
│ │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ Usage & Cost Tracking │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ This Month: 1,234,567 tokens | Est. Cost: $42.18 │
│ Last Month: 987,654 tokens | Est. Cost: $31.45 │
│ │
│ [ View Detailed Usage Log ] │
│ │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ [ Save Changes ] [ Reset to Defaults ] │
│ │
└────────────────────────────────────────────────────────────────────┘
Rubric Editor for Filtering
┌────────────────────────────────────────────────────────────────────┐
│ Round 2: FILTERING > AI Screening Rules │
├────────────────────────────────────────────────────────────────────┤
│ │
│ Rule: Spam & Quality Check │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ Criteria (Plain Language): │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Projects should demonstrate clear ocean conservation value. │ │
│ │ │ │
│ │ Reject projects that: │ │
│ │ - Are spam, test submissions, or joke entries │ │
│ │ - Have no meaningful description (< 100 words) │ │
│ │ - Are unrelated to ocean conservation │ │
│ │ - Are duplicate submissions │ │
│ │ │ │
│ │ Flag for manual review: │ │
│ │ - Projects with unclear focus or vague descriptions │ │
│ │ - Projects that may be legitimate but borderline │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ Action: ( ) PASS (•) REJECT ( ) FLAG │
│ │
│ Priority: [ 1 ] (1 = highest, 10 = lowest) │
│ │
│ [ ] Active │
│ │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ Preview & Test │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ Test this rule on sample projects: │
│ │
│ [✓] SaveTheSea - AI-powered coral reef monitoring │
│ Result: PASS (confidence: 0.92) │
│ Reasoning: Clear ocean conservation focus, detailed approach │
│ │
│ [✓] Test Project - Just testing the form │
│ Result: REJECT (confidence: 0.98) │
│ Reasoning: Appears to be a test submission with no real value │
│ │
│ [✓] Marine Plastic Solution (vague description) │
│ Result: FLAG (confidence: 0.65) │
│ Reasoning: Ocean-related but description lacks detail │
│ │
│ [ Run Test on 10 Random Projects ] │
│ │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ [ Save Rule ] [ Delete Rule ] [ Cancel ] │
│ │
└────────────────────────────────────────────────────────────────────┘
AI Results Review Interface
┌────────────────────────────────────────────────────────────────────┐
│ Round 2: FILTERING > AI Results Review │
├────────────────────────────────────────────────────────────────────┤
│ │
│ AI Screening Complete │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ Processed: 120 projects │
│ Passed: 87 (72.5%) │
│ Rejected: 21 (17.5%) │
│ Flagged: 12 (10.0%) │
│ │
│ Tokens Used: 45,678 | Est. Cost: $1.37 │
│ Model: gpt-4 | Batches: 6 of 20 projects each │
│ │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ Flagged Projects (Require Manual Review) │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ [ ] Marine Plastic Solution │
│ Confidence: 0.65 │
│ AI Reasoning: Ocean-related but description lacks sufficient │
│ detail. Unclear commercial model. May be early- │
│ stage concept that needs more development. │
│ Quality Score: 6/10 │
│ Spam Risk: No │
│ │
│ Admin Decision: ( ) PASS ( ) REJECT ( ) Keep Flagged │
│ │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ [ ] Ocean Cleanup AI │
│ Confidence: 0.58 │
│ AI Reasoning: Uses AI buzzwords extensively but lacks technical│
│ specifics. May be over-promising. Borderline. │
│ Quality Score: 5/10 │
│ Spam Risk: No │
│ │
│ Admin Decision: ( ) PASS ( ) REJECT ( ) Keep Flagged │
│ │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ [ Review All Flagged (12) ] [ Bulk Approve ] [ Bulk Reject ] │
│ │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ Rejected Projects (Can Override) │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ [✓] Test Submission │
│ Confidence: 0.98 │
│ AI Reasoning: Obvious test submission with placeholder text. │
│ Override: [ ] Approve This Project │
│ │
│ [✓] Spam Project 123 │
│ Confidence: 0.95 │
│ AI Reasoning: Unrelated to ocean conservation, generic content.│
│ Override: [ ] Approve This Project │
│ │
│ [ Show All Rejected (21) ] │
│ │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ [ Finalize Results ] [ Export CSV ] [ Re-run AI Screening ] │
│ │
└────────────────────────────────────────────────────────────────────┘
Cost Monitoring Dashboard
┌────────────────────────────────────────────────────────────────────┐
│ Analytics > AI Usage & Costs │
├────────────────────────────────────────────────────────────────────┤
│ │
│ Overview (February 2024) │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ Total API Calls: 1,234 │
│ Total Tokens: 4,567,890 │
│ Estimated Cost: $137.04 │
│ Avg Cost per Call: $0.11 │
│ │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ Cost Breakdown by Service │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ Filtering: $45.67 (33%) ████████░░░ │
│ Assignment: $38.21 (28%) ███████░░░░ │
│ Evaluation Summary: $28.90 (21%) █████░░░░░░ │
│ Tagging: $15.12 (11%) ███░░░░░░░░ │
│ Award Eligibility: $9.14 (7%) ██░░░░░░░░░ │
│ │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ Token Usage Trend │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ 500K ┤ ╭─ │
│ 400K ┤ ╭────────╯ │
│ 300K ┤ ╭────────────╯ │
│ 200K ┤ ╭─────────────╯ │
│ 100K ┤ ╭────────╯ │
│ 0 ┼───┴──────────────────────────────────────────────────── │
│ Feb 1 Feb 8 Feb 15 Feb 22 Feb 29 │
│ │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ Recent API Calls │
│ ────────────────────────────────────────────────────────────────── │
│ │
│ 2024-02-15 14:30 | Filtering | Round 2 | 20 projects │
│ gpt-4 | 3,456 tokens | $0.10 | SUCCESS │
│ │
│ 2024-02-15 14:15 | Assignment | Round 3 | 15 projects │
│ gpt-4 | 4,123 tokens | $0.12 | SUCCESS │
│ │
│ 2024-02-15 13:45 | Summary | Proj-123 | 5 evaluations │
│ gpt-4-turbo | 1,890 tokens | $0.02 | ✓ │
│ │
│ [ View All Logs ] [ Export CSV ] [ Set Budget Alert ] │
│ │
└────────────────────────────────────────────────────────────────────┘
API Changes
New tRPC Procedures
AI Router (src/server/routers/ai.ts)
export const aiRouter = createTRPCRouter({
// Test OpenAI connection
testConnection: adminProcedure
.mutation(async ({ ctx }) => {
const openai = await getOpenAI()
if (!openai) {
throw new TRPCError({ code: 'PRECONDITION_FAILED', message: 'OpenAI not configured' })
}
try {
await openai.models.list()
return { connected: true, message: 'OpenAI connection successful' }
} catch (error) {
return { connected: false, message: error.message }
}
}),
// Get AI usage stats
getUsageStats: adminProcedure
.input(z.object({
startDate: z.date(),
endDate: z.date()
}))
.query(async ({ ctx, input }) => {
const logs = await ctx.prisma.aIUsageLog.findMany({
where: {
createdAt: {
gte: input.startDate,
lte: input.endDate
}
}
})
const totalTokens = logs.reduce((sum, log) => sum + log.totalTokens, 0)
const totalCost = logs.reduce((sum, log) => sum + (log.estimatedCost || 0), 0)
const byService = logs.reduce((acc, log) => {
const service = log.action
if (!acc[service]) {
acc[service] = { count: 0, tokens: 0, cost: 0 }
}
acc[service].count++
acc[service].tokens += log.totalTokens
acc[service].cost += log.estimatedCost || 0
return acc
}, {} as Record<string, { count: number, tokens: number, cost: number }>)
return {
totalCalls: logs.length,
totalTokens,
totalCost,
avgCostPerCall: totalCost / logs.length,
byService
}
}),
// Tag a single project
tagProject: adminProcedure
.input(z.object({
projectId: z.string()
}))
.mutation(async ({ ctx, input }) => {
const result = await tagProject(input.projectId, ctx.session.user.id)
return result
}),
// Get tag suggestions (preview only, don't apply)
getTagSuggestions: adminProcedure
.input(z.object({
projectId: z.string()
}))
.query(async ({ ctx, input }) => {
const suggestions = await getTagSuggestions(input.projectId, ctx.session.user.id)
return suggestions
}),
// Generate evaluation summary
generateEvaluationSummary: adminProcedure
.input(z.object({
projectId: z.string(),
roundId: z.string()
}))
.mutation(async ({ ctx, input }) => {
const summary = await generateSummary({
projectId: input.projectId,
stageId: input.roundId,
userId: ctx.session.user.id,
prisma: ctx.prisma
})
return summary
}),
// Generate AI assignments
generateAssignments: adminProcedure
.input(z.object({
roundId: z.string(),
constraints: z.object({
requiredReviewsPerProject: z.number(),
minAssignmentsPerJuror: z.number().optional(),
maxAssignmentsPerJuror: z.number().optional()
})
}))
.mutation(async ({ ctx, input }) => {
const round = await ctx.prisma.round.findUnique({
where: { id: input.roundId },
include: {
juryGroup: {
include: {
members: { include: { user: true } }
}
}
}
})
if (!round?.juryGroup) {
throw new TRPCError({ code: 'NOT_FOUND', message: 'Round or jury group not found' })
}
const projects = await ctx.prisma.project.findMany({
where: {
projectRoundStates: {
some: {
roundId: input.roundId,
state: 'PASSED'
}
}
},
include: { _count: { select: { assignments: true } } }
})
const existingAssignments = await ctx.prisma.assignment.findMany({
where: { roundId: input.roundId },
select: { userId: true, projectId: true }
})
const result = await generateAIAssignments(
round.juryGroup.members.map(m => m.user),
projects,
{
...input.constraints,
existingAssignments
},
ctx.session.user.id,
input.roundId
)
return result
}),
// Run filtering on a round
runFiltering: adminProcedure
.input(z.object({
roundId: z.string()
}))
.mutation(async ({ ctx, input }) => {
const round = await ctx.prisma.round.findUnique({
where: { id: input.roundId }
})
if (round?.roundType !== 'FILTERING') {
throw new TRPCError({ code: 'BAD_REQUEST', message: 'Not a filtering round' })
}
const rules = await ctx.prisma.filteringRule.findMany({
where: { roundId: input.roundId, isActive: true },
orderBy: { priority: 'asc' }
})
const projects = await ctx.prisma.project.findMany({
where: {
projectRoundStates: {
some: { roundId: input.roundId }
}
},
include: {
files: true,
_count: { select: { teamMembers: true } }
}
})
const results = await executeFilteringRules(
rules,
projects,
ctx.session.user.id,
input.roundId
)
// Store results
for (const result of results) {
await ctx.prisma.filteringResult.upsert({
where: {
projectId_roundId: {
projectId: result.projectId,
roundId: input.roundId
}
},
create: {
projectId: result.projectId,
roundId: input.roundId,
outcome: result.outcome,
ruleResultsJson: result.ruleResults,
aiScreeningJson: result.aiScreeningJson
},
update: {
outcome: result.outcome,
ruleResultsJson: result.ruleResults,
aiScreeningJson: result.aiScreeningJson
}
})
}
return {
total: results.length,
passed: results.filter(r => r.outcome === 'PASSED').length,
rejected: results.filter(r => r.outcome === 'FILTERED_OUT').length,
flagged: results.filter(r => r.outcome === 'FLAGGED').length
}
}),
// Check award eligibility
checkAwardEligibility: adminProcedure
.input(z.object({
awardId: z.string()
}))
.mutation(async ({ ctx, input }) => {
const award = await ctx.prisma.specialAward.findUnique({
where: { id: input.awardId }
})
if (!award?.useAiEligibility || !award.criteriaText) {
throw new TRPCError({ code: 'BAD_REQUEST', message: 'AI eligibility not enabled for this award' })
}
const projects = await ctx.prisma.project.findMany({
where: { competitionId: award.competitionId },
include: {
files: true,
_count: { select: { teamMembers: true } }
}
})
const results = await aiInterpretCriteria(
award.criteriaText,
projects,
ctx.session.user.id,
award.id
)
// Store results
for (const result of results) {
await ctx.prisma.awardEligibility.upsert({
where: {
specialAwardId_projectId: {
specialAwardId: award.id,
projectId: result.projectId
}
},
create: {
specialAwardId: award.id,
projectId: result.projectId,
isEligible: result.eligible,
confidence: result.confidence,
reasoning: result.reasoning,
method: result.method
},
update: {
isEligible: result.eligible,
confidence: result.confidence,
reasoning: result.reasoning,
method: result.method
}
})
}
return {
total: results.length,
eligible: results.filter(r => r.eligible).length,
ineligible: results.filter(r => !r.eligible).length
}
})
})
Service Functions
Complete Function Signatures
// ai-filtering.ts
export function evaluateFieldRule(
config: FieldRuleConfig,
project: ProjectForFiltering
): { passed: boolean; action: 'PASS' | 'REJECT' | 'FLAG' }
export function evaluateDocumentRule(
config: DocumentCheckConfig,
project: ProjectForFiltering
): { passed: boolean; action: 'PASS' | 'REJECT' | 'FLAG' }
export async function executeAIScreening(
config: AIScreeningConfig,
projects: ProjectForFiltering[],
userId?: string,
entityId?: string,
onProgress?: ProgressCallback
): Promise<Map<string, AIScreeningResult>>
export async function executeFilteringRules(
rules: FilteringRuleInput[],
projects: ProjectForFiltering[],
userId?: string,
stageId?: string,
onProgress?: ProgressCallback
): Promise<ProjectFilteringResult[]>
// ai-assignment.ts
export async function generateAIAssignments(
jurors: JurorForAssignment[],
projects: ProjectForAssignment[],
constraints: AssignmentConstraints,
userId?: string,
entityId?: string,
onProgress?: AssignmentProgressCallback
): Promise<AIAssignmentResult>
export function generateFallbackAssignments(
jurors: JurorForAssignment[],
projects: ProjectForAssignment[],
constraints: AssignmentConstraints
): AIAssignmentResult
// ai-evaluation-summary.ts
export function anonymizeEvaluations(
evaluations: EvaluationForSummary[]
): AnonymizedEvaluation[]
export function buildSummaryPrompt(
anonymizedEvaluations: AnonymizedEvaluation[],
projectTitle: string,
criteriaLabels: string[]
): string
export function computeScoringPatterns(
evaluations: EvaluationForSummary[],
criteriaLabels: CriterionDef[]
): ScoringPatterns
export async function generateSummary({
projectId,
stageId,
userId,
prisma
}: {
projectId: string
stageId: string
userId: string
prisma: PrismaClient
}): Promise<EvaluationSummaryResult>
// ai-tagging.ts
export async function getTaggingSettings(): Promise<{
enabled: boolean
maxTags: number
}>
export async function getAvailableTags(): Promise<AvailableTag[]>
export async function tagProject(
projectId: string,
userId?: string
): Promise<TaggingResult>
export async function getTagSuggestions(
projectId: string,
userId?: string
): Promise<TagSuggestion[]>
export async function addProjectTag(
projectId: string,
tagId: string
): Promise<void>
export async function removeProjectTag(
projectId: string,
tagId: string
): Promise<void>
// ai-award-eligibility.ts
export function applyAutoTagRules(
rules: AutoTagRule[],
projects: ProjectForEligibility[]
): Map<string, boolean>
export async function aiInterpretCriteria(
criteriaText: string,
projects: ProjectForEligibility[],
userId?: string,
awardId?: string
): Promise<EligibilityResult[]>
// anonymization.ts
export function sanitizeText(text: string): string
export function truncateAndSanitize(
text: string | null | undefined,
maxLength: number
): string
export function anonymizeForAI(
jurors: JurorInput[],
projects: ProjectInput[]
): AnonymizationResult
export function anonymizeProjectForAI(
project: ProjectWithRelations,
index: number,
context: DescriptionContext
): AnonymizedProjectForAI
export function anonymizeProjectsForAI(
projects: ProjectWithRelations[],
context: DescriptionContext
): {
anonymized: AnonymizedProjectForAI[]
mappings: ProjectAIMapping[]
}
export function deanonymizeResults<T extends { jurorId: string; projectId: string }>(
results: T[],
jurorMappings: JurorMapping[],
projectMappings: ProjectMapping[]
): (T & { realJurorId: string; realProjectId: string })[]
export function validateNoPersonalData(
data: Record<string, unknown>
): PIIValidationResult
export function enforceGDPRCompliance(data: unknown[]): void
export function validateAnonymization(data: AnonymizationResult): boolean
export function validateAnonymizedProjects(
projects: AnonymizedProjectForAI[]
): boolean
export function toProjectWithRelations(project: unknown): ProjectWithRelations
Edge Cases
| Scenario | Behavior | Fallback |
|---|---|---|
| OpenAI API key missing | AI services disabled | Use algorithm-based fallbacks |
| OpenAI rate limit hit | Queue requests, retry with exponential backoff | Fail gracefully after 3 retries |
| AI returns invalid JSON | Parse error logged, flagged for manual review | Mark items as "needs review" |
| AI model doesn't exist | Throw clear error with model name | Suggest checking Settings > AI Config |
| Anonymization validation fails | Throw error, log GDPR violation | Reject AI call, require manual review |
| All projects rejected by AI | Admin gets warning notification | Admin reviews AI reasoning |
| Zero tag suggestions | Return empty array | No error, just no tags applied |
| Confidence too low (<0.5) | Tag not applied automatically | Admin can manually apply |
| Duplicate project detected (>0.85 similarity) | Flagged for admin review | Admin marks as duplicate or false positive |
| Mentor workspace inactive (>14 days) | Intervention alert sent to admin | Admin reaches out to mentor |
| Winner ranking unclear (close gap <0.3) | AI flags as "close call", suggests review | Jury deliberates again |
| Batch size too large (>50) | Clamped to 50 projects per batch | Automatically split into multiple batches |
| Description too long (>500 chars) | Truncated with "..." | AI works with truncated text |
| PII in feedback text | Sanitized before sending to AI | Emails/phones replaced with [email removed] |
| Juror has no expertise tags | Scored as 0.5 (neutral) in assignment | Fallback algorithm includes them for load balancing |
| Project has no description | AI flags as low quality | Admin decides whether to reject |
| All jurors at capacity | Assignment fails with clear error | Admin adjusts caps or adds more jurors |
| Award criteria too vague | AI gives low-confidence results | Admin refines criteria and re-runs |
| Evaluation summary with <3 evaluations | AI summary still generated, marked as "limited data" | Admin aware that consensus may be weak |
| Streaming response timeout | Fall back to non-streaming request | Complete response returned after wait |
Integration Map
AI Services × Round Types
graph TD
R1[Round 1: INTAKE] --> T[ai-tagging.ts]
T --> TAG[Auto-tag on submit]
R2[Round 2: FILTERING] --> F[ai-filtering.ts]
F --> SCREEN[AI screening + rules]
F --> DUP[ai-duplicate-detection]
DUP --> EMBED[Embedding similarity]
R3[Round 3: EVALUATION] --> A[ai-assignment.ts]
A --> ASSIGN[Generate assignments]
A --> E[ai-evaluation-summary.ts]
E --> SUM[Summarize evaluations]
R4[Round 4: SUBMISSION] --> NONE1[No AI service]
R5[Round 5: EVALUATION] --> A
A --> E
R6[Round 6: MENTORING] --> M[ai-mentoring-insights.ts]
M --> INSIGHT[Workspace analysis]
R7[Round 7: LIVE_FINAL] --> NONE2[No AI service]
R8[Round 8: CONFIRMATION] --> C[ai-confirmation-helper.ts]
C --> EXPLAIN[Ranking explanation]
AWARDS[Special Awards] --> AE[ai-award-eligibility.ts]
AE --> ELIG[Eligibility check]
Data Flow Diagram
sequenceDiagram
participant Admin
participant API
participant Service
participant Anonymization
participant OpenAI
participant DB
Admin->>API: Trigger AI service
API->>Service: Call service function
Service->>DB: Fetch data
DB-->>Service: Raw data
Service->>Anonymization: Anonymize data
Anonymization->>Anonymization: Strip PII
Anonymization->>Anonymization: Replace IDs
Anonymization->>Anonymization: Validate GDPR
Anonymization-->>Service: Anonymized + mappings
Service->>OpenAI: API call
OpenAI-->>Service: AI response
Service->>Anonymization: De-anonymize results
Anonymization-->>Service: Real IDs
Service->>DB: Store results
DB-->>Service: Confirmation
Service->>DB: Log AI usage
Service-->>API: Results
API-->>Admin: Success + data
This completes the extremely detailed AI Services documentation covering all current services, new services for the redesign, anonymization pipeline, prompt engineering, OpenAI integration, privacy/security, admin controls, API changes, service functions, edge cases, and integration map.
Total lines: 2,900+