# AI Services Architecture ## Overview AI services power automation across all round types in the MOPC platform. All AI calls anonymize data before sending to OpenAI, ensuring GDPR compliance and privacy protection. These services enable intelligent filtering, smart jury assignments, evaluation synthesis, automatic tagging, and award eligibility assessment. ### Core Principles 1. **Privacy First**: All data is anonymized before AI processing (no PII sent to OpenAI) 2. **Graceful Degradation**: Fallback algorithms when AI is unavailable 3. **Cost Awareness**: Batching, token tracking, and cost monitoring 4. **Audit Trail**: All AI requests logged with inputs, outputs, and token usage 5. **Admin Control**: Per-service toggles, configuration overrides, manual review 6. **Transparency**: AI reasoning exposed to admins for validation ### Service Inventory | Service | Purpose | Primary Model | Input Data | Output | |---------|---------|---------------|------------|--------| | **ai-filtering.ts** | Automated project screening | GPT-4 | Projects + rubric | Pass/Reject/Flag + scores | | **ai-assignment.ts** | Jury-project matching | GPT-4 | Jurors + projects + constraints | Assignment suggestions | | **ai-evaluation-summary.ts** | Synthesis of evaluations | GPT-4-turbo | All evaluations for a project | Strengths, weaknesses, themes | | **ai-tagging.ts** | Auto-tag projects | GPT-4 | Project description | Tag suggestions + confidence | | **ai-award-eligibility.ts** | Award eligibility assessment | GPT-4 | Projects + award criteria | Eligibility scores + reasoning | | **anonymization.ts** | PII stripping pipeline | N/A | Raw project/user data | Anonymized data + mappings | --- ## Current AI Services ### 1. AI Filtering Service (`ai-filtering.ts`) **Purpose**: Automate project screening in Round 2 (FILTERING) using admin-defined rubrics and AI interpretation of plain-language criteria. #### Input Data ```typescript interface ProjectForFiltering { id: string title: string description?: string | null competitionCategory?: CompetitionCategory | null foundedAt?: Date | null country?: string | null geographicZone?: string | null tags: string[] oceanIssue?: OceanIssue | null wantsMentorship?: boolean | null institution?: string | null submissionSource?: SubmissionSource submittedAt?: Date | null files: Array<{ id: string fileName: string fileType?: FileType | null }> _count?: { teamMembers?: number files?: number } } interface FilteringRuleInput { id: string name: string ruleType: 'FIELD_BASED' | 'DOCUMENT_CHECK' | 'AI_SCREENING' configJson: Prisma.JsonValue priority: number isActive: boolean } ``` #### Rule Types **Field-Based Rules** (No AI): ```typescript type FieldRuleConfig = { conditions: FieldRuleCondition[] logic: 'AND' | 'OR' action: 'PASS' | 'REJECT' | 'FLAG' } type FieldRuleCondition = { field: 'competitionCategory' | 'foundedAt' | 'country' | 'geographicZone' | 'tags' | 'oceanIssue' operator: 'equals' | 'not_equals' | 'greater_than' | 'less_than' | 'contains' | 'in' | 'not_in' | 'older_than_years' | 'newer_than_years' | 'is_empty' value: string | number | string[] } // Example: Reject projects older than 10 years { conditions: [ { field: 'foundedAt', operator: 'older_than_years', value: 10 } ], logic: 'AND', action: 'REJECT' } // Example: Flag projects from specific countries { conditions: [ { field: 'country', operator: 'in', value: ['US', 'CN', 'RU'] } ], logic: 'OR', action: 'FLAG' } ``` **Document Check Rules** (No AI): ```typescript type DocumentCheckConfig = { requiredFileTypes?: string[] // ['pdf', 'docx'] minFileCount?: number action: 'PASS' | 'REJECT' | 'FLAG' } // Example: Require at least 2 PDF files { requiredFileTypes: ['pdf'], minFileCount: 2, action: 'REJECT' } ``` **AI Screening Rules** (OpenAI): ```typescript type AIScreeningConfig = { criteriaText: string // Plain-language rubric action: 'PASS' | 'REJECT' | 'FLAG' batchSize?: number // 1-50, default 20 parallelBatches?: number // 1-10, default 1 } // Example: Detect spam/low-quality projects { criteriaText: ` Projects should demonstrate clear ocean conservation value. Reject projects that: - Are spam, test submissions, or joke entries - Have no meaningful description - Are unrelated to ocean conservation - Are duplicate submissions `, action: 'REJECT', batchSize: 20, parallelBatches: 2 } ``` #### AI Screening Process 1. **Anonymization**: Strip PII from projects ```typescript const { anonymized, mappings } = anonymizeProjectsForAI(projects, 'FILTERING') // Before: { id: "proj-abc123", title: "SaveTheSea by John Doe (john@example.com)", ... } // After: { project_id: "P1", title: "SaveTheSea by Team 1", ... } ``` 2. **Validation**: Ensure no PII leaked ```typescript if (!validateAnonymizedProjects(anonymized)) { throw new Error('GDPR compliance check failed') } ``` 3. **Batch Processing**: Process projects in configurable batches ```typescript const batchSize = Math.min(MAX_BATCH_SIZE, config.batchSize ?? 20) const parallelBatches = Math.min(MAX_PARALLEL_BATCHES, config.parallelBatches ?? 1) for (let i = 0; i < batches.length; i += parallelBatches) { const parallelChunk = batches.slice(i, i + parallelBatches) const results = await Promise.all(parallelChunk.map(processAIBatch)) } ``` 4. **OpenAI Call**: Send anonymized batch with criteria ```typescript const prompt = `CRITERIA: ${criteriaText} PROJECTS: ${JSON.stringify(anonymized)} Evaluate and return JSON.` const response = await openai.chat.completions.create({ model: 'gpt-4', messages: [ { role: 'system', content: AI_SCREENING_SYSTEM_PROMPT }, { role: 'user', content: prompt } ], response_format: { type: 'json_object' }, temperature: 0.3 }) ``` 5. **Result Parsing**: ```typescript interface AIScreeningResult { meetsCriteria: boolean confidence: number // 0.0 - 1.0 reasoning: string qualityScore: number // 1-10 spamRisk: boolean } // AI returns: { "projects": [ { "project_id": "P1", "meets_criteria": true, "confidence": 0.85, "reasoning": "Clear ocean conservation focus, well-documented approach", "quality_score": 8, "spam_risk": false } ] } ``` 6. **De-anonymization**: Map results back to real IDs ```typescript const mapping = mappings.find(m => m.anonymousId === "P1") results.set(mapping.realId, aiResult) ``` #### Output ```typescript interface ProjectFilteringResult { projectId: string outcome: 'PASSED' | 'FILTERED_OUT' | 'FLAGGED' ruleResults: RuleResult[] aiScreeningJson?: { [ruleId: string]: AIScreeningResult } } interface RuleResult { ruleId: string ruleName: string ruleType: string passed: boolean action: 'PASS' | 'REJECT' | 'FLAG' reasoning?: string // Only for AI_SCREENING } ``` **Outcome Logic**: - If ANY rule with `action: 'REJECT'` fails → `FILTERED_OUT` - Else if ANY rule with `action: 'FLAG'` fails → `FLAGGED` - Else → `PASSED` #### Integration with Filtering Round ```typescript // Round 2: FILTERING const filteringRound = await prisma.round.findFirst({ where: { competitionId, roundType: 'FILTERING' } }) const rules = await prisma.filteringRule.findMany({ where: { roundId: filteringRound.id, isActive: true }, orderBy: { priority: 'asc' } }) const projects = await prisma.project.findMany({ where: { competitionId }, include: { files: true, _count: { select: { teamMembers: true } } } }) const results = await executeFilteringRules(rules, projects, userId, roundId) // Store results for (const result of results) { await prisma.filteringResult.create({ data: { projectId: result.projectId, roundId: filteringRound.id, outcome: result.outcome, ruleResultsJson: result.ruleResults, aiScreeningJson: result.aiScreeningJson } }) // Update project round state await prisma.projectRoundState.update({ where: { projectId_roundId: { projectId: result.projectId, roundId: filteringRound.id } }, data: { state: result.outcome === 'PASSED' ? 'PASSED' : 'REJECTED', metadataJson: { filteringOutcome: result.outcome } } }) } ``` --- ### 2. AI Assignment Service (`ai-assignment.ts`) **Purpose**: Generate optimal jury-to-project assignments based on expertise matching, workload balancing, and constraints. #### Input Data ```typescript interface JurorForAssignment { id: string name?: string | null email: string expertiseTags: string[] maxAssignments?: number | null _count?: { assignments: number } } interface ProjectForAssignment { id: string title: string description?: string | null tags: string[] teamName?: string | null _count?: { assignments: number } } interface AssignmentConstraints { requiredReviewsPerProject: number // e.g., 3 jurors per project minAssignmentsPerJuror?: number // e.g., 5 projects minimum maxAssignmentsPerJuror?: number // e.g., 20 projects maximum jurorLimits?: Record // Per-juror overrides existingAssignments: Array<{ jurorId: string projectId: string }> } ``` #### Anonymization & Batching ```typescript // 1. Truncate descriptions (save tokens) const truncatedProjects = projects.map(p => ({ ...p, description: truncateAndSanitize(p.description, DESCRIPTION_LIMITS.ASSIGNMENT) // 300 chars })) // 2. Anonymize data const anonymizedData = anonymizeForAI(jurors, truncatedProjects) // Before: jurors = [ { id: "user-123", email: "john@example.com", expertiseTags: ["Marine Biology", "AI"] }, { id: "user-456", email: "jane@example.com", expertiseTags: ["Oceanography"] } ] // After: anonymizedData.jurors = [ { anonymousId: "juror_001", expertiseTags: ["Marine Biology", "AI"], currentAssignmentCount: 5, maxAssignments: 20 }, { anonymousId: "juror_002", expertiseTags: ["Oceanography"], currentAssignmentCount: 3, maxAssignments: 20 } ] // 3. Validate anonymization if (!validateAnonymization(anonymizedData)) { return generateFallbackAssignments(jurors, projects, constraints) } // 4. Process in batches (15 projects per batch) const BATCH_SIZE = 15 for (let i = 0; i < projects.length; i += BATCH_SIZE) { const batchProjects = anonymizedData.projects.slice(i, i + BATCH_SIZE) const suggestions = await processAssignmentBatch(openai, model, anonymizedData, batchProjects, constraints) } ``` #### Prompt Structure ```typescript const userPrompt = `JURORS: ${JSON.stringify(jurors)} PROJECTS: ${JSON.stringify(projects)} CONSTRAINTS: ${constraints.requiredReviewsPerProject} reviews/project, max ${constraints.maxAssignmentsPerJuror}/juror EXISTING: ${JSON.stringify(anonymousExisting)} Return JSON: {"assignments": [...]}` const systemPrompt = `Match jurors to projects by expertise. Return JSON assignments. Each: {juror_id, project_id, confidence_score: 0-1, expertise_match_score: 0-1, reasoning: str (1-2 sentences)} Distribute workload fairly. Avoid assigning jurors at capacity.` ``` **Example AI Response**: ```json { "assignments": [ { "juror_id": "juror_001", "project_id": "project_003", "confidence_score": 0.92, "expertise_match_score": 0.88, "reasoning": "Strong match on Marine Biology and AI tags. Juror has capacity." }, { "juror_id": "juror_002", "project_id": "project_003", "confidence_score": 0.75, "expertise_match_score": 0.70, "reasoning": "Oceanography expertise applies. Helps distribute workload." } ] } ``` #### Output ```typescript interface AIAssignmentSuggestion { jurorId: string // Real ID (de-anonymized) projectId: string // Real ID (de-anonymized) confidenceScore: number // 0-1 expertiseMatchScore: number // 0-1 reasoning: string } interface AIAssignmentResult { success: boolean suggestions: AIAssignmentSuggestion[] error?: string tokensUsed?: number fallbackUsed?: boolean } ``` #### Fallback Algorithm When AI is unavailable or fails: ```typescript export function generateFallbackAssignments( jurors: JurorForAssignment[], projects: ProjectForAssignment[], constraints: AssignmentConstraints ): AIAssignmentResult { // Algorithm: // 1. Sort projects by current assignment count (fewest first) // 2. For each project, score all available jurors by: // - Expertise match (50% weight): tag overlap // - Load balancing (30% weight): current vs. max assignments // - Under-min bonus (20% weight): jurors below minimum target // 3. Assign top-scoring jurors const scoredJurors = jurors.map(juror => ({ juror, score: calculateExpertiseScore(juror.expertiseTags, project.tags) * 0.5 + calculateLoadScore(currentLoad, maxLoad) * 0.3 + calculateUnderMinBonus(currentLoad, minTarget) * 0.2 })) // Assign top N jurors per project const topN = scoredJurors.slice(0, neededReviews) return { success: true, suggestions: topN.map(s => ({ jurorId: s.juror.id, projectId, ... })), fallbackUsed: true } } ``` **Expertise Score**: ```typescript function calculateExpertiseScore(jurorTags: string[], projectTags: string[]): number { const jurorTagsLower = new Set(jurorTags.map(t => t.toLowerCase())) const matchingTags = projectTags.filter(t => jurorTagsLower.has(t.toLowerCase())) const matchRatio = matchingTags.length / projectTags.length const hasExpertise = matchingTags.length > 0 ? 0.2 : 0 return Math.min(1, matchRatio * 0.8 + hasExpertise) } ``` #### Integration with Evaluation Round ```typescript // Round 3/5: EVALUATION const evaluationRound = await prisma.round.findFirst({ where: { competitionId, roundType: 'EVALUATION', sortOrder: 3 } }) const juryGroup = await prisma.juryGroup.findUnique({ where: { id: evaluationRound.juryGroupId }, include: { members: { include: { user: true } } } }) const projects = await prisma.project.findMany({ where: { projectRoundStates: { some: { roundId: previousRoundId, state: 'PASSED' } } } }) const constraints = { requiredReviewsPerProject: 3, maxAssignmentsPerJuror: 20, jurorLimits: { "user-123": 15, // Personal override "user-456": 25 // Personal override }, existingAssignments: await prisma.assignment.findMany({ where: { roundId: evaluationRound.id }, select: { userId: true, projectId: true } }) } const result = await generateAIAssignments( juryGroup.members.map(m => m.user), projects, constraints, adminUserId, evaluationRound.id, onProgress ) // Admin reviews suggestions in UI, then applies for (const suggestion of result.suggestions) { await prisma.assignment.create({ data: { userId: suggestion.jurorId, projectId: suggestion.projectId, roundId: evaluationRound.id, juryGroupId: juryGroup.id, method: 'AI', aiConfidenceScore: suggestion.confidenceScore, expertiseMatchScore: suggestion.expertiseMatchScore, aiReasoning: suggestion.reasoning } }) } ``` --- ### 3. AI Evaluation Summary Service (`ai-evaluation-summary.ts`) **Purpose**: Synthesize multiple juror evaluations for a project into a cohesive summary with strengths, weaknesses, themes, and consensus analysis. #### Input Data ```typescript interface EvaluationForSummary { id: string criterionScoresJson: Record | null globalScore: number | null binaryDecision: boolean | null feedbackText: string | null assignment: { user: { id: string name: string | null email: string } } } interface CriterionDef { id: string label: string } ``` #### Anonymization ```typescript // Strips juror identities, keeps only scores and sanitized feedback export function anonymizeEvaluations( evaluations: EvaluationForSummary[] ): AnonymizedEvaluation[] { return evaluations.map(ev => ({ criterionScores: ev.criterionScoresJson, globalScore: ev.globalScore, binaryDecision: ev.binaryDecision, feedbackText: ev.feedbackText ? sanitizeText(ev.feedbackText) : null })) } // Before: [ { id: "eval-123", globalScore: 8, feedbackText: "Strong proposal from john@example.com. Contact me at +1-555-1234.", assignment: { user: { name: "John Doe", email: "john@example.com" } } } ] // After: [ { globalScore: 8, feedbackText: "Strong proposal. Contact me at [phone removed].", // No user info } ] ``` #### Prompt Structure ```typescript const prompt = `You are analyzing jury evaluations for a project competition. PROJECT: "${sanitizedTitle}" EVALUATION CRITERIA: ${criteriaLabels.join(', ')} EVALUATIONS (${anonymizedEvaluations.length} total): ${JSON.stringify(anonymizedEvaluations, null, 2)} Analyze these evaluations and return a JSON object with this exact structure: { "overallAssessment": "A 2-3 sentence summary of how the project was evaluated overall", "strengths": ["strength 1", "strength 2", ...], "weaknesses": ["weakness 1", "weakness 2", ...], "themes": [ { "theme": "theme name", "sentiment": "positive" | "negative" | "mixed", "frequency": } ], "recommendation": "A brief recommendation based on the evaluation consensus" } Guidelines: - Base your analysis only on the provided evaluation data - Identify common themes across evaluator feedback - Note areas of agreement and disagreement - Keep the assessment objective and balanced - Do not include any personal identifiers` ``` #### Scoring Patterns (Server-Side) In addition to AI analysis, the service computes statistical patterns: ```typescript interface ScoringPatterns { averageGlobalScore: number | null consensus: number // 0-1 (1 = full agreement) criterionAverages: Record evaluatorCount: number } export function computeScoringPatterns( evaluations: EvaluationForSummary[], criteriaLabels: CriterionDef[] ): ScoringPatterns { const globalScores = evaluations.map(e => e.globalScore).filter(s => s !== null) const averageGlobalScore = globalScores.length > 0 ? globalScores.reduce((a, b) => a + b, 0) / globalScores.length : null // Consensus: 1 - normalized standard deviation let consensus = 1 if (globalScores.length > 1 && averageGlobalScore !== null) { const variance = globalScores.reduce((sum, score) => sum + Math.pow(score - averageGlobalScore, 2), 0 ) / globalScores.length const stdDev = Math.sqrt(variance) consensus = Math.max(0, 1 - stdDev / 4.5) // Normalize by max possible std dev } // Criterion averages const criterionAverages: Record = {} for (const criterion of criteriaLabels) { const scores = evaluations .map(e => e.criterionScoresJson?.[criterion.id]) .filter((s): s is number => s !== undefined) if (scores.length > 0) { criterionAverages[criterion.label] = scores.reduce((a, b) => a + b, 0) / scores.length } } return { averageGlobalScore, consensus: Math.round(consensus * 100) / 100, criterionAverages, evaluatorCount: evaluations.length } } ``` #### Output ```typescript interface AIResponsePayload { overallAssessment: string strengths: string[] weaknesses: string[] themes: Array<{ theme: string sentiment: 'positive' | 'negative' | 'mixed' frequency: number }> recommendation: string } interface EvaluationSummaryResult { id: string projectId: string stageId: string summaryJson: AIResponsePayload & { scoringPatterns: ScoringPatterns } generatedAt: Date model: string tokensUsed: number } ``` **Example Output**: ```json { "overallAssessment": "The project received consistently high scores (avg: 8.2/10) with strong consensus (0.92). Evaluators praised the technical approach and team expertise, but raised concerns about scalability and budget.", "strengths": [ "Novel AI-powered coral monitoring approach", "Strong technical team with relevant expertise", "Clear commercial viability and market need" ], "weaknesses": [ "Budget appears optimistic for proposed timeline", "Limited discussion of regulatory compliance", "Scalability challenges not fully addressed" ], "themes": [ { "theme": "Technical Innovation", "sentiment": "positive", "frequency": 5 }, { "theme": "Team Expertise", "sentiment": "positive", "frequency": 4 }, { "theme": "Budget Concerns", "sentiment": "negative", "frequency": 3 }, { "theme": "Scalability", "sentiment": "mixed", "frequency": 2 } ], "recommendation": "Strong candidate for advancement. Address budget and scalability concerns during mentoring phase.", "scoringPatterns": { "averageGlobalScore": 8.2, "consensus": 0.92, "criterionAverages": { "Innovation": 8.8, "Feasibility": 7.6, "Impact": 8.4, "Team": 9.0 }, "evaluatorCount": 5 } } ``` #### Integration with Evaluation Round ```typescript // Auto-trigger when all evaluations for a project are submitted const evaluationsCount = await prisma.evaluation.count({ where: { status: 'SUBMITTED', assignment: { projectId, roundId } } }) const requiredReviews = evaluationRound.configJson.requiredReviewsPerProject if (evaluationsCount >= requiredReviews) { const summary = await generateSummary({ projectId, stageId: roundId, userId: 'system', prisma }) // Notify admin await prisma.inAppNotification.create({ data: { userId: adminId, title: 'Evaluation Summary Generated', message: `AI summary ready for project "${projectTitle}"`, type: 'AI_SUMMARY_READY', relatedEntityType: 'EvaluationSummary', relatedEntityId: summary.id } }) } ``` --- ### 4. AI Tagging Service (`ai-tagging.ts`) **Purpose**: Automatically assign expertise tags to projects based on content analysis. #### Input Data ```typescript // Full project data with files and team info const project = await prisma.project.findUnique({ where: { id: projectId }, include: { projectTags: true, files: { select: { fileType: true } }, _count: { select: { teamMembers: true, files: true } } } }) // Available tags from system interface AvailableTag { id: string name: string category: string | null description: string | null } const availableTags = await prisma.expertiseTag.findMany({ where: { isActive: true }, orderBy: [{ category: 'asc' }, { sortOrder: 'asc' }] }) ``` #### Anonymization ```typescript const projectWithRelations = toProjectWithRelations(project) const { anonymized, mappings } = anonymizeProjectsForAI([projectWithRelations], 'FILTERING') // Anonymized project: { project_id: "P1", title: "SaveTheSea", description: "An AI-powered platform for coral reef monitoring...", category: "STARTUP", ocean_issue: "CORAL_REEFS", tags: ["AI", "Monitoring"], founded_year: 2024, team_size: 3, file_count: 5, file_types: ["PDF", "DOCX"] } ``` #### Prompt Structure ```typescript const tagList = availableTags.map(t => ({ name: t.name, category: t.category, description: t.description })) const prompt = `PROJECT: ${JSON.stringify(anonymizedProject, null, 2)} AVAILABLE TAGS: ${JSON.stringify(tagList, null, 2)} Suggest relevant tags for this project.` const systemPrompt = `You are an expert at categorizing ocean conservation and sustainability projects. Analyze the project and suggest the most relevant expertise tags from the provided list. Consider the project's focus areas, technology, methodology, and domain. Return JSON with this format: { "suggestions": [ { "tag_name": "exact tag name from list", "confidence": 0.0-1.0, "reasoning": "brief explanation why this tag fits" } ] } Rules: - Only suggest tags from the provided list (exact names) - Order by relevance (most relevant first) - Confidence should reflect how well the tag matches - Maximum 7 suggestions per project - Be conservative - only suggest tags that truly apply` ``` #### Output ```typescript interface TagSuggestion { tagId: string tagName: string confidence: number reasoning: string } interface TaggingResult { projectId: string suggestions: TagSuggestion[] applied: TagSuggestion[] tokensUsed: number } ``` **Example Response**: ```json { "suggestions": [ { "tag_name": "Marine Biology", "confidence": 0.92, "reasoning": "Project focuses on coral reef health monitoring" }, { "tag_name": "Artificial Intelligence", "confidence": 0.88, "reasoning": "Uses AI for image analysis and pattern detection" }, { "tag_name": "Data Science", "confidence": 0.75, "reasoning": "Significant data collection and analysis component" } ] } ``` #### Application Logic ```typescript const CONFIDENCE_THRESHOLD = 0.5 const MAX_TAGS = 5 // 1. Get AI suggestions const { suggestions, tokensUsed } = await getAISuggestions(anonymized[0], availableTags, userId) // 2. Filter by confidence threshold const validSuggestions = suggestions.filter(s => s.confidence >= CONFIDENCE_THRESHOLD) // 3. Get existing tags to avoid duplicates const existingTagIds = new Set(project.projectTags.map(pt => pt.tagId)) // 4. Calculate remaining slots const currentTagCount = project.projectTags.length const remainingSlots = Math.max(0, MAX_TAGS - currentTagCount) // 5. Filter and limit const newSuggestions = validSuggestions .filter(s => !existingTagIds.has(s.tagId)) .slice(0, remainingSlots) // 6. Apply new tags (additive only, never removes existing) for (const suggestion of newSuggestions) { await prisma.projectTag.create({ data: { projectId, tagId: suggestion.tagId, confidence: suggestion.confidence, source: 'AI' } }) } ``` #### Integration Points **On Project Submission** (Round 1: INTAKE): ```typescript // Auto-tag when project is submitted if (settings.ai_tagging_enabled && settings.ai_tagging_on_submit) { await tagProject(projectId, userId) } ``` **Manual Tagging** (Admin Dashboard): ```typescript // Admin reviews suggestions before applying const suggestions = await getTagSuggestions(projectId, userId) // UI shows suggestions with confidence scores // Admin clicks "Apply All" or selectively adds tags for (const suggestion of selectedSuggestions) { await addProjectTag(projectId, suggestion.tagId) } ``` **Batch Tagging** (Round Management): ```typescript // Tag all projects in a round const projects = await prisma.project.findMany({ where: { competitionId } }) for (const project of projects) { try { await tagProject(project.id, adminUserId) } catch (error) { console.error(`Failed to tag project ${project.id}:`, error) } } ``` --- ### 5. AI Award Eligibility Service (`ai-award-eligibility.ts`) **Purpose**: Determine which projects are eligible for special awards using both deterministic field matching and AI interpretation of plain-language criteria. #### Input Data ```typescript interface ProjectForEligibility { id: string title: string description?: string | null competitionCategory?: CompetitionCategory | null country?: string | null geographicZone?: string | null tags: string[] oceanIssue?: OceanIssue | null institution?: string | null foundedAt?: Date | null wantsMentorship?: boolean submissionSource?: SubmissionSource submittedAt?: Date | null _count?: { teamMembers?: number files?: number } files?: Array<{ fileType: string | null }> } interface SpecialAward { id: string name: string description?: string | null criteriaText?: string | null // Plain-language criteria for AI eligibilityMode: 'STAY_IN_MAIN' | 'SEPARATE_POOL' useAiEligibility: boolean } ``` #### Deterministic Auto-Tag Rules ```typescript type AutoTagRule = { field: 'competitionCategory' | 'country' | 'geographicZone' | 'tags' | 'oceanIssue' operator: 'equals' | 'contains' | 'in' value: string | string[] } // Example: "Innovation Award" for AI/ML projects const innovationRules: AutoTagRule[] = [ { field: 'tags', operator: 'contains', value: 'Artificial Intelligence' } ] // Example: "Regional Impact Award" for Mediterranean projects const regionalRules: AutoTagRule[] = [ { field: 'geographicZone', operator: 'equals', value: 'MEDITERRANEAN' } ] // Apply rules export function applyAutoTagRules( rules: AutoTagRule[], projects: ProjectForEligibility[] ): Map { const results = new Map() for (const project of projects) { const matches = rules.every(rule => { const fieldValue = getFieldValue(project, rule.field) // ... operator logic ... }) results.set(project.id, matches) } return results } ``` #### AI Criteria Interpretation ```typescript // Award with plain-language criteria const award = { name: "Youth Innovation Award", criteriaText: ` Projects eligible for this award should: - Be founded within the last 3 years - Have a team with average age under 30 - Demonstrate novel technological approach - Show strong potential for scalability `, useAiEligibility: true } // Anonymize and batch process const { anonymized, mappings } = anonymizeProjectsForAI(projects, 'ELIGIBILITY') const prompt = `CRITERIA: ${criteriaText} PROJECTS: ${JSON.stringify(anonymized)} Evaluate eligibility for each project.` const systemPrompt = `Award eligibility evaluator. Evaluate projects against criteria, return JSON. Format: {"evaluations": [{project_id, eligible: bool, confidence: 0-1, reasoning: str}]} Be objective. Base evaluation only on provided data. No personal identifiers in reasoning.` ``` #### Output ```typescript interface EligibilityResult { projectId: string eligible: boolean confidence: number reasoning: string method: 'AUTO' | 'AI' } // Example AI response: { "evaluations": [ { "project_id": "P1", "eligible": true, "confidence": 0.85, "reasoning": "Founded in 2024 (within 3 years). Novel AI approach for coral monitoring. High scalability potential." }, { "project_id": "P2", "eligible": false, "confidence": 0.90, "reasoning": "Founded in 2015 (over 3 years ago). Does not meet youth criteria." } ] } ``` #### Integration with Special Awards ```typescript // Run eligibility check for an award const award = await prisma.specialAward.findUnique({ where: { id: awardId }, include: { competition: true } }) const projects = await prisma.project.findMany({ where: { competitionId: award.competitionId }, include: { files: true, _count: { select: { teamMembers: true } } } }) let eligibilityResults: EligibilityResult[] = [] if (award.useAiEligibility && award.criteriaText) { // AI interpretation eligibilityResults = await aiInterpretCriteria( award.criteriaText, projects, userId, awardId ) } else if (award.autoTagRules) { // Deterministic rules const matches = applyAutoTagRules(award.autoTagRules, projects) eligibilityResults = Array.from(matches).map(([projectId, eligible]) => ({ projectId, eligible, confidence: 1.0, reasoning: 'Matches auto-tag rules', method: 'AUTO' })) } // Store results for (const result of eligibilityResults) { await prisma.awardEligibility.upsert({ where: { specialAwardId_projectId: { specialAwardId: awardId, projectId: result.projectId } }, create: { specialAwardId: awardId, projectId: result.projectId, isEligible: result.eligible, confidence: result.confidence, reasoning: result.reasoning, method: result.method }, update: { isEligible: result.eligible, confidence: result.confidence, reasoning: result.reasoning, method: result.method } }) } ``` --- ### 6. Anonymization Service (`anonymization.ts`) **Purpose**: Strip PII from all data before sending to OpenAI. Ensures GDPR compliance and privacy protection. #### PII Patterns Detected ```typescript const PII_PATTERNS = { email: /[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/g, phone: /(\+?\d{1,3}[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}/g, url: /https?:\/\/[^\s]+/g, ssn: /\d{3}-\d{2}-\d{4}/g, ipv4: /\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/g } export function sanitizeText(text: string): string { let sanitized = text sanitized = sanitized.replace(PII_PATTERNS.email, '[email removed]') sanitized = sanitized.replace(PII_PATTERNS.phone, '[phone removed]') sanitized = sanitized.replace(PII_PATTERNS.url, '[url removed]') sanitized = sanitized.replace(PII_PATTERNS.ssn, '[id removed]') return sanitized } ``` #### Description Truncation ```typescript export const DESCRIPTION_LIMITS = { ASSIGNMENT: 300, // Short context for matching FILTERING: 500, // Medium context for screening ELIGIBILITY: 400, // Medium context for criteria MENTOR: 350 // Short context for mentor matching } export function truncateAndSanitize( text: string | null | undefined, maxLength: number ): string { if (!text) return '' const sanitized = sanitizeText(text) if (sanitized.length <= maxLength) return sanitized return sanitized.slice(0, maxLength - 3) + '...' } ``` #### ID Replacement Strategy **Before Anonymization**: ```typescript // Real data with identifiable IDs { id: "proj-abc123xyz", userId: "user-456def789", title: "SaveTheSea by Marine Research Institute", teamName: "Dr. John Doe's Lab", description: "Contact us at john@example.com or +1-555-1234 for more info. Visit https://savethesea.org" } ``` **After Anonymization**: ```typescript // Anonymous IDs and sanitized text { project_id: "P1", title: "SaveTheSea", description: "Contact us at [email removed] or [phone removed] for more info. Visit [url removed]", team_size: 3, founded_year: 2024, tags: ["Marine Biology", "AI"] } ``` **Mapping for De-anonymization**: ```typescript const mappings: ProjectAIMapping[] = [ { anonymousId: "P1", realId: "proj-abc123xyz" }, { anonymousId: "P2", realId: "proj-def456uvw" } ] ``` #### GDPR Validation ```typescript export interface PIIValidationResult { valid: boolean violations: string[] } export function validateNoPersonalData( data: Record ): PIIValidationResult { const violations: string[] = [] const textContent = JSON.stringify(data) // Check PII patterns for (const [type, pattern] of Object.entries(PII_PATTERNS)) { pattern.lastIndex = 0 if (pattern.test(textContent)) { violations.push(`Potential ${type} detected in data`) } } // Check sensitive field names const sensitiveFields = ['email', 'phone', 'password', 'ssn', 'creditCard', 'bankAccount'] const keys = Object.keys(data).map(k => k.toLowerCase()) for (const field of sensitiveFields) { if (keys.includes(field)) { violations.push(`Sensitive field "${field}" present in data`) } } return { valid: violations.length === 0, violations } } // Enforce before EVERY AI call export function enforceGDPRCompliance(data: unknown[]): void { for (let i = 0; i < data.length; i++) { const { valid, violations } = validateNoPersonalData(item) if (!valid) { throw new Error(`GDPR compliance check failed: ${violations.join(', ')}`) } } } ``` #### Anonymization Workflow ```mermaid graph TD A[Raw Project Data] --> B[Convert to ProjectWithRelations] B --> C[Truncate Descriptions] C --> D[Sanitize Text Fields] D --> E[Replace IDs with Anonymous IDs] E --> F[Create Mappings] F --> G[Validate No PII] G --> H{Valid?} H -->|Yes| I[Send to AI] H -->|No| J[Throw GDPR Error] I --> K[AI Response] K --> L[De-anonymize Results] L --> M[Return Real IDs] ``` --- ## New AI Services for Redesign ### 1. AI Mentoring Insights (Round 6: MENTORING) **Purpose**: Summarize mentor-team interactions, flag inactive workspaces, suggest intervention points. #### Input Data ```typescript interface MentorWorkspaceData { projectId: string mentorId: string workspaceOpenAt: Date files: MentorFile[] messages: MentorMessage[] fileComments: MentorFileComment[] lastActivityAt: Date } ``` #### Anonymization ```typescript // Strip mentor/team names, keep activity patterns { workspace_id: "W1", days_active: 14, file_count: 3, message_count: 12, comment_count: 5, last_activity_days_ago: 2, file_types: ["PDF", "DOCX"], message_frequency: "daily", engagement_level: "high" } ``` #### Prompt ```typescript const prompt = `Analyze mentor workspace activity and provide insights. WORKSPACE DATA: ${JSON.stringify(anonymizedWorkspace)} Return JSON: { "engagement_level": "high" | "medium" | "low" | "inactive", "key_insights": ["insight 1", "insight 2"], "red_flags": ["flag 1", "flag 2"], "recommendations": ["recommendation 1", "recommendation 2"], "intervention_needed": boolean }` ``` #### Output ```typescript interface MentoringInsight { workspaceId: string engagementLevel: 'high' | 'medium' | 'low' | 'inactive' keyInsights: string[] redFlags: string[] recommendations: string[] interventionNeeded: boolean } ``` **Example**: ```json { "engagement_level": "low", "key_insights": [ "No activity in last 7 days", "Only 1 file uploaded (below average for this stage)", "Message frequency dropped from daily to weekly" ], "red_flags": [ "Mentor has not responded to last 2 team messages", "No progress on business plan deliverable (due in 5 days)" ], "recommendations": [ "Admin should check in with mentor", "Send reminder notification to team about upcoming deadline", "Consider pairing with secondary mentor for support" ], "intervention_needed": true } ``` #### Integration ```typescript // Nightly cron job: analyze all active mentoring workspaces const mentoringRound = await prisma.round.findFirst({ where: { competitionId, roundType: 'MENTORING', status: 'ROUND_ACTIVE' } }) const assignments = await prisma.mentorAssignment.findMany({ where: { workspaceEnabled: true }, include: { files: true, messages: true } }) for (const assignment of assignments) { const insight = await generateMentoringInsight(assignment, userId) if (insight.interventionNeeded) { await prisma.inAppNotification.create({ data: { userId: adminId, title: 'Mentoring Intervention Needed', message: `Workspace for project "${projectTitle}" requires attention`, type: 'MENTORING_ALERT', actionUrl: `/admin/mentoring/${assignment.id}` } }) } } ``` --- ### 2. AI Duplicate Detection Enhancement (Round 2: FILTERING) **Purpose**: Detect duplicate or near-duplicate submissions using embeddings and semantic similarity. #### Current Approach (Simple) ```typescript // Existing: basic title/email matching const duplicates = await prisma.project.findMany({ where: { OR: [ { title: { equals: project.title, mode: 'insensitive' } }, { teamMembers: { some: { email: applicantEmail } } } ] } }) ``` #### Enhanced Approach (AI-Powered) ```typescript // Generate embeddings for all projects const embeddings = new Map() for (const project of projects) { const embedding = await openai.embeddings.create({ model: 'text-embedding-3-small', input: `${project.title}\n\n${project.description}` }) embeddings.set(project.id, embedding.data[0].embedding) } // Compute cosine similarity matrix function cosineSimilarity(a: number[], b: number[]): number { const dot = a.reduce((sum, val, i) => sum + val * b[i], 0) const magA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0)) const magB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0)) return dot / (magA * magB) } // Find duplicates const SIMILARITY_THRESHOLD = 0.85 for (const [id1, emb1] of embeddings) { for (const [id2, emb2] of embeddings) { if (id1 >= id2) continue // Skip self and already compared const similarity = cosineSimilarity(emb1, emb2) if (similarity >= SIMILARITY_THRESHOLD) { await prisma.duplicateDetection.create({ data: { projectId1: id1, projectId2: id2, similarity, method: 'AI_EMBEDDING', status: 'FLAGGED' } }) } } } ``` #### Output ```typescript interface DuplicateMatch { projectId1: string projectId2: string similarity: number method: 'EXACT_TITLE' | 'EMAIL_MATCH' | 'AI_EMBEDDING' status: 'FLAGGED' | 'CONFIRMED_DUPLICATE' | 'FALSE_POSITIVE' } ``` **Admin Review UI**: ``` ┌─────────────────────────────────────────────────────────────────┐ │ Duplicate Detection: 3 Potential Matches │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ [!] 92% Match - AI Coral Monitoring (P1) vs Ocean AI (P2) │ │ Method: AI Embedding │ │ ┌─ P1: SaveTheSea - AI-powered coral reef monitoring... │ │ └─ P2: Ocean AI - Machine learning platform for coral... │ │ │ │ [ Mark as Duplicate ] [ False Positive ] [ View Details ] │ │ │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ [!] 88% Match - Marine Plastic (P3) vs PlasticClean (P4) │ │ Method: AI Embedding + Email Match │ │ ┌─ P3: Marine Plastic Cleanup - Robotic system... │ │ └─ P4: PlasticClean - Automated plastic removal... │ │ │ │ [ Mark as Duplicate ] [ False Positive ] [ View Details ] │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` --- ### 3. AI Confirmation Helper (Round 8: CONFIRMATION) **Purpose**: Generate plain-language explanations for why final rankings are what they are. Helps jury understand and validate the winner proposal. #### Input Data ```typescript interface WinnerProposalData { rankedProjectIds: string[] sourceRoundId: string selectionBasis: { method: 'LIVE_VOTE' | 'EVALUATION_SCORES' | 'ADMIN_DECISION' scores: Record aiRecommendation?: string } } // Fetch all context const projects = await prisma.project.findMany({ where: { id: { in: proposal.rankedProjectIds } }, include: { evaluations: { where: { roundId: sourceRoundId } }, liveVotes: { where: { roundId: sourceRoundId } } } }) ``` #### Anonymization ```typescript // Strip project names, keep relative rankings and scores { ranking: [ { position: 1, anonymous_id: "P1", score: 8.9, vote_count: 12 }, { position: 2, anonymous_id: "P2", score: 8.5, vote_count: 10 }, { position: 3, anonymous_id: "P3", score: 8.2, vote_count: 11 } ], selection_method: "LIVE_VOTE", score_gaps: [0.4, 0.3], consensus_level: 0.88 } ``` #### Prompt ```typescript const prompt = `Explain the final ranking in plain language for jury review. RANKING DATA: ${JSON.stringify(anonymizedRanking)} EVALUATION HISTORY: ${JSON.stringify(anonymizedEvaluations)} Return JSON: { "summary": "2-3 sentence plain-language summary of why this ranking makes sense", "position_explanations": [ { "position": 1, "why_this_rank": "explanation", "strengths": ["s1", "s2"] }, ... ], "close_calls": [ { "positions": [2, 3], "gap": 0.3, "reasoning": "why this was close" } ], "confidence": "high" | "medium" | "low", "red_flags": ["flag 1", ...] }` ``` #### Output ```typescript interface ConfirmationExplanation { summary: string positionExplanations: Array<{ position: number projectId: string whyThisRank: string strengths: string[] }> closeCalls: Array<{ positions: number[] gap: number reasoning: string }> confidence: 'high' | 'medium' | 'low' redFlags: string[] } ``` **Example**: ```json { "summary": "The ranking reflects clear consensus from both live voting (12 votes) and prior evaluations (avg 8.9). Top position is well-separated with 0.4 point gap. Second and third places are close (0.3 gap), but consistent across jury members.", "position_explanations": [ { "position": 1, "why_this_rank": "Highest average score (8.9) across all evaluation rounds. Won live vote with 12/15 jury members. Consistent top performer.", "strengths": ["Technical innovation", "Strong team", "Commercial viability", "Scalability"] }, { "position": 2, "why_this_rank": "Strong runner-up (8.5 avg). Close to third place but slight edge in feasibility scores.", "strengths": ["Proven track record", "Regulatory compliance", "Budget realism"] }, { "position": 3, "why_this_rank": "Solid finalist (8.2 avg). Slightly lower technical feasibility scores compared to #2.", "strengths": ["Impact potential", "Novel approach", "Jury enthusiasm"] } ], "close_calls": [ { "positions": [2, 3], "gap": 0.3, "reasoning": "Very close race. Position 2 had slight edge in 'Feasibility' criterion (7.8 vs 7.2), while Position 3 scored higher in 'Impact' (8.9 vs 8.4). Live vote split 10-11. Admin may want to review deliberation notes." } ], "confidence": "high", "red_flags": [] } ``` **Integration**: ```typescript // When admin creates WinnerProposal const proposal = await prisma.winnerProposal.create({ data: { competitionId, category: 'STARTUP', rankedProjectIds: ['proj-1', 'proj-2', 'proj-3'], sourceRoundId: liveFinalsRoundId, selectionBasis: { method: 'LIVE_VOTE', scores: {...} }, proposedById: adminId } }) // Generate AI explanation const explanation = await generateConfirmationExplanation(proposal.id, userId) // Attach to proposal await prisma.winnerProposal.update({ where: { id: proposal.id }, data: { selectionBasis: { ...proposal.selectionBasis, aiExplanation: explanation } } }) // Show to jury during confirmation ``` **Confirmation UI with AI Explanation**: ``` ┌─────────────────────────────────────────────────────────────────┐ │ Confirm Final Rankings: STARTUP Category │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ AI Analysis: High Confidence │ │ "The ranking reflects clear consensus from both live voting │ │ (12 votes) and prior evaluations (avg 8.9). Top position is │ │ well-separated with 0.4 point gap..." │ │ │ │ ┌───────────────────────────────────────────────────────────────┐ │ │ │ 1st Place: SaveTheSea (Score: 8.9, 12 votes) │ │ │ │ Why This Rank: Highest average score across all rounds... │ │ │ │ Strengths: Technical innovation, Strong team, Commercial... │ │ │ └───────────────────────────────────────────────────────────────┘ │ │ │ │ ┌───────────────────────────────────────────────────────────────┐ │ │ │ 2nd Place: Ocean AI (Score: 8.5, 10 votes) │ │ │ │ Why This Rank: Strong runner-up. Close to third place... │ │ │ │ Strengths: Proven track record, Regulatory compliance... │ │ │ └───────────────────────────────────────────────────────────────┘ │ │ │ │ [!] Close Call: 2nd and 3rd place (0.3 gap) │ │ Position 2 had slight edge in Feasibility (7.8 vs 7.2)... │ │ │ │ [ ] I confirm this ranking (John Doe, Jury Lead) │ │ [ ] I confirm this ranking (Jane Smith, Jury Member) │ │ [ ] I confirm this ranking (Bob Wilson, Jury Member) │ │ │ │ Comments: ___________________________________________________ │ │ │ │ [ Approve Ranking ] [ Request Changes ] │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` --- ## Anonymization Pipeline (Deep Dive) ### Data Flow ```mermaid graph LR A[Raw Data] --> B[Type Conversion] B --> C[Description Truncation] C --> D[Text Sanitization] D --> E[ID Replacement] E --> F[Field Filtering] F --> G[Validation] G --> H{GDPR Pass?} H -->|Yes| I[To AI] H -->|No| J[Error + Log] I --> K[AI Response] K --> L[De-anonymization] L --> M[Real Results] ``` ### What Gets Stripped | Data Type | Before | After | |-----------|--------|-------| | **User IDs** | `user-abc123xyz` | `juror_001` | | **Project IDs** | `proj-def456uvw` | `P1` | | **Names** | `Dr. John Doe` | (removed entirely) | | **Emails** | `john@example.com` | `[email removed]` | | **Phone Numbers** | `+1-555-1234` | `[phone removed]` | | **URLs** | `https://example.com` | `[url removed]` | | **Team Names** | `Marine Research Institute` | `Team 1` | | **Descriptions** | Full 2000-char text | Truncated to 300-500 chars | | **Dates** | `2024-02-15T14:30:00Z` | `2024` (year only) or `2024-02-15` (date only) | ### Replacement Strategy **Sequential Anonymous IDs**: - Projects: `P1`, `P2`, `P3`, ... - Jurors: `juror_001`, `juror_002`, `juror_003`, ... - Workspaces: `W1`, `W2`, `W3`, ... **Rationale**: Sequential IDs are easier for AI to reference and for humans to debug. Random UUIDs would be harder to correlate in logs. **Generic Team Names**: - `Team 1`, `Team 2`, `Team 3`, ... **Rationale**: Preserves the concept of "team" without revealing institutional affiliation. ### De-anonymization Mapping ```typescript // Stored in memory during AI call (never persisted to DB) const mappings: ProjectAIMapping[] = [ { anonymousId: "P1", realId: "proj-abc123xyz" }, { anonymousId: "P2", realId: "proj-def456uvw" }, { anonymousId: "P3", realId: "proj-ghi789rst" } ] // After AI returns results for "P1", map back to real ID const realProjectId = mappings.find(m => m.anonymousId === "P1")?.realId ``` ### Anonymization for Different Data Types #### Projects ```typescript // INPUT (ProjectWithRelations) { id: "proj-abc123", title: "SaveTheSea - Marine Conservation by Dr. John Doe", description: "Contact john@example.com or call +1-555-1234. Visit https://savethesea.org for more info.", teamName: "Marine Research Institute", competitionCategory: "STARTUP", country: "France", geographicZone: "MEDITERRANEAN", tags: ["Marine Biology", "AI", "Conservation"], foundedAt: new Date("2024-01-15T00:00:00Z"), institution: "University of Monaco", wantsMentorship: true, submissionSource: "WEB_FORM", submittedAt: new Date("2024-02-15T14:30:00Z"), _count: { teamMembers: 3, files: 5 }, files: [ { fileType: "PDF" }, { fileType: "PDF" }, { fileType: "DOCX" } ] } // OUTPUT (AnonymizedProjectForAI) { project_id: "P1", title: "SaveTheSea", // PII stripped description: "Contact [email removed] or call [phone removed]. Visit [url removed] for more info.", // Sanitized + truncated category: "STARTUP", ocean_issue: null, country: "France", region: "MEDITERRANEAN", institution: "University of Monaco", // Kept (non-PII institutional data) tags: ["Marine Biology", "AI", "Conservation"], founded_year: 2024, // Date reduced to year only team_size: 3, has_description: true, file_count: 5, file_types: ["PDF", "DOCX"], wants_mentorship: true, submission_source: "WEB_FORM", submitted_date: "2024-02-15" // Date only, no time } ``` #### Jurors ```typescript // INPUT (JurorForAssignment) { id: "user-456def", name: "Dr. Jane Smith", email: "jane.smith@university.edu", expertiseTags: ["Marine Biology", "Oceanography"], maxAssignments: 20, _count: { assignments: 5 } } // OUTPUT (AnonymizedJuror) { anonymousId: "juror_001", expertiseTags: ["Marine Biology", "Oceanography"], currentAssignmentCount: 5, maxAssignments: 20 // Name and email completely removed } ``` #### Evaluations ```typescript // INPUT (EvaluationForSummary) { id: "eval-789ghi", criterionScoresJson: { "Innovation": 9, "Feasibility": 7, "Impact": 8 }, globalScore: 8, binaryDecision: true, feedbackText: "Excellent proposal from Dr. John Doe (john@example.com). Strong technical approach but budget seems optimistic. Call me at +1-555-9876 to discuss.", assignment: { user: { id: "user-123abc", name: "Dr. Alice Johnson", email: "alice@university.edu" } } } // OUTPUT (AnonymizedEvaluation) { criterionScores: { "Innovation": 9, "Feasibility": 7, "Impact": 8 }, globalScore: 8, binaryDecision: true, feedbackText: "Excellent proposal. Strong technical approach but budget seems optimistic. Call me at [phone removed] to discuss." // User info completely removed // PII sanitized from feedback } ``` ### GDPR Validation Workflow ```typescript // 1. Anonymize data const { anonymized, mappings } = anonymizeProjectsForAI(projects, 'FILTERING') // 2. Validate BEFORE sending to AI if (!validateAnonymizedProjects(anonymized)) { console.error('[AI Service] GDPR validation failed') // Log violation await prisma.auditLog.create({ data: { action: 'AI_GDPR_VIOLATION', entityType: 'AIService', details: { service: 'filtering', error: 'PII detected in anonymized data' } } }) throw new Error('GDPR compliance check failed: PII detected in anonymized data') } // 3. Only if validation passes, send to AI const response = await openai.chat.completions.create(params) // 4. De-anonymize results const realResults = mappings.map(mapping => { const aiResult = response.find(r => r.project_id === mapping.anonymousId) return { projectId: mapping.realId, ...aiResult } }) ``` --- ## Prompt Engineering ### System Prompt Templates #### Filtering Prompt ```typescript const AI_SCREENING_SYSTEM_PROMPT = `Project screening assistant. Evaluate against criteria, return JSON. Format: {"projects": [{project_id, meets_criteria: bool, confidence: 0-1, reasoning: str, quality_score: 1-10, spam_risk: bool}]} Be objective. Base evaluation only on provided data. No personal identifiers in reasoning.` ``` **Design Principles**: - **Compact**: 3 sentences instead of 10 paragraphs (saves tokens) - **JSON Schema**: Explicit output format prevents parsing errors - **Objectivity**: Reminds AI to avoid bias - **Privacy**: Explicitly forbids personal identifiers in output #### Assignment Prompt ```typescript const ASSIGNMENT_SYSTEM_PROMPT = `Match jurors to projects by expertise. Return JSON assignments. Each: {juror_id, project_id, confidence_score: 0-1, expertise_match_score: 0-1, reasoning: str (1-2 sentences)} Distribute workload fairly. Avoid assigning jurors at capacity.` ``` **Design Principles**: - **Constraint Awareness**: Reminds AI about workload balancing - **Brevity Requirement**: "1-2 sentences" for reasoning (saves tokens) - **Dual Scores**: Separates confidence (how sure AI is) from expertise match (how good the fit is) #### Evaluation Summary Prompt ```typescript const EVALUATION_SUMMARY_PROMPT = `You are analyzing jury evaluations for a project competition. PROJECT: "${sanitizedTitle}" EVALUATION CRITERIA: ${criteriaLabels.join(', ')} EVALUATIONS (${count} total): ${JSON.stringify(anonymizedEvaluations, null, 2)} Analyze these evaluations and return a JSON object with this exact structure: { "overallAssessment": "A 2-3 sentence summary of how the project was evaluated overall", "strengths": ["strength 1", "strength 2", ...], "weaknesses": ["weakness 1", "weakness 2", ...], "themes": [ { "theme": "theme name", "sentiment": "positive" | "negative" | "mixed", "frequency": } ], "recommendation": "A brief recommendation based on the evaluation consensus" } Guidelines: - Base your analysis only on the provided evaluation data - Identify common themes across evaluator feedback - Note areas of agreement and disagreement - Keep the assessment objective and balanced - Do not include any personal identifiers` ``` **Design Principles**: - **Structured Output**: JSON schema enforces consistency - **Sentiment Analysis**: Themes include positive/negative/mixed classification - **Consensus Detection**: Frequency count shows how many evaluators mentioned each theme - **Explicit Guidelines**: 5 bullet points ensure quality output ### Dynamic Context Injection ```typescript // Build prompts dynamically based on available data function buildFilteringPrompt( criteriaText: string, projects: AnonymizedProjectForAI[], context?: { previousRound?: { passRate: number }, categoryDistribution?: { STARTUP: number, BUSINESS_CONCEPT: number } } ): string { let prompt = `CRITERIA: ${criteriaText}\nPROJECTS: ${JSON.stringify(projects)}\n` if (context?.previousRound) { prompt += `\nPREVIOUS ROUND: Pass rate was ${context.previousRound.passRate}%. Apply similar rigor.\n` } if (context?.categoryDistribution) { prompt += `\nCATEGORY BALANCE: Aim for roughly ${context.categoryDistribution.STARTUP}% startups, ${context.categoryDistribution.BUSINESS_CONCEPT}% concepts.\n` } prompt += `\nEvaluate and return JSON.` return prompt } ``` ### Output Format Enforcement ```typescript // Use JSON mode for all AI calls const params = buildCompletionParams(model, { messages: [ { role: 'system', content: systemPrompt }, { role: 'user', content: userPrompt } ], jsonMode: true, // Enforces JSON output temperature: 0.3, maxTokens: 4000 }) // Validate response structure const parsed = JSON.parse(content) as ExpectedStructure if (!isValidStructure(parsed)) { throw createParseError('AI returned invalid JSON structure') } ``` ### Token Limit Management ```typescript // Description limits by context export const DESCRIPTION_LIMITS = { ASSIGNMENT: 300, // Short - only need general topic FILTERING: 500, // Medium - need enough detail for quality assessment ELIGIBILITY: 400, // Medium - need criteria matching MENTOR: 350 // Short - expertise matching } // Calculate estimated token count before API call function estimateTokens(text: string): number { // Rough estimate: 1 token ≈ 4 characters return Math.ceil(text.length / 4) } // Warn if approaching limits const promptTokens = estimateTokens(userPrompt) if (promptTokens > 3000) { console.warn(`[AI Service] Large prompt detected: ${promptTokens} tokens`) } // Set maxTokens based on expected output const maxTokens = { FILTERING: 4000, // Batch of 20 projects ASSIGNMENT: 4000, // Batch of 15 projects SUMMARY: 2000, // Single summary TAGGING: 1000 // Tag suggestions }[serviceType] ``` ### Retry and Fallback Strategies ```typescript async function callAIWithRetry( apiCall: () => Promise, fallback: () => T, maxRetries = 3 ): Promise { let lastError: Error for (let attempt = 1; attempt <= maxRetries; attempt++) { try { return await apiCall() } catch (error) { lastError = error const classified = classifyAIError(error) // Don't retry on validation errors or rate limits if (classified.type === 'VALIDATION' || classified.type === 'RATE_LIMIT') { break } // Wait before retry (exponential backoff) if (attempt < maxRetries) { await new Promise(resolve => setTimeout(resolve, 1000 * Math.pow(2, attempt))) } } } // Fall back to algorithm console.warn(`[AI Service] Failed after ${maxRetries} attempts, using fallback`, lastError) return fallback() } ``` --- ## OpenAI Integration ### Model Selection ```typescript // lib/openai.ts export const AI_MODELS = { STRONG: 'gpt-4', // Complex reasoning (filtering, assignment) QUICK: 'gpt-4-turbo', // Fast analysis (summaries, tagging) CHEAP: 'gpt-3.5-turbo', // Simple tasks (deprecated for this platform) EMBEDDING: 'text-embedding-3-small' // For duplicate detection } // Get configured model from settings export async function getConfiguredModel(preference?: string): Promise { const settings = await prisma.systemSettings.findUnique({ where: { key: 'ai_model' } }) const model = preference || settings?.value || AI_MODELS.STRONG // Validate model exists if (!Object.values(AI_MODELS).includes(model)) { console.warn(`[OpenAI] Unknown model "${model}", falling back to ${AI_MODELS.STRONG}`) return AI_MODELS.STRONG } return model } ``` **Model Usage by Service**: | Service | Model | Rationale | |---------|-------|-----------| | Filtering | `gpt-4` | Needs strong reasoning for nuanced criteria interpretation | | Assignment | `gpt-4` | Complex constraint balancing (expertise + load + COI) | | Summary | `gpt-4-turbo` | Fast synthesis, less reasoning required | | Tagging | `gpt-4-turbo` | Simple categorization task | | Eligibility | `gpt-4` | Plain-language criteria need strong comprehension | | Mentoring Insights | `gpt-4-turbo` | Pattern detection, less critical reasoning | ### Rate Limiting ```typescript // Rate limiter middleware class AIRateLimiter { private requestQueue: Array<() => Promise> = [] private isProcessing = false private requestsPerMinute = 60 // GPT-4 tier 1 limit async add(fn: () => Promise): Promise { return new Promise((resolve, reject) => { this.requestQueue.push(async () => { try { const result = await fn() resolve(result) } catch (error) { reject(error) } }) if (!this.isProcessing) { this.process() } }) } private async process() { this.isProcessing = true while (this.requestQueue.length > 0) { const fn = this.requestQueue.shift()! await fn() // Wait to respect rate limit await new Promise(resolve => setTimeout(resolve, 60000 / this.requestsPerMinute)) } this.isProcessing = false } } const rateLimiter = new AIRateLimiter() // Use in services const result = await rateLimiter.add(() => openai.chat.completions.create(params)) ``` ### Cost Tracking ```typescript // Token pricing (as of 2024) const TOKEN_PRICING = { 'gpt-4': { input: 0.03, output: 0.06 }, // per 1K tokens 'gpt-4-turbo': { input: 0.01, output: 0.03 }, 'text-embedding-3-small': { input: 0.0001, output: 0 } } // Calculate cost function calculateCost( model: string, promptTokens: number, completionTokens: number ): number { const pricing = TOKEN_PRICING[model] || TOKEN_PRICING['gpt-4'] const inputCost = (promptTokens / 1000) * pricing.input const outputCost = (completionTokens / 1000) * pricing.output return inputCost + outputCost } // Log usage with cost await logAIUsage({ userId, action: 'FILTERING', model, promptTokens, completionTokens, totalTokens, estimatedCost: calculateCost(model, promptTokens, completionTokens), status: 'SUCCESS' }) // Admin dashboard: show cumulative costs const totalCost = await prisma.aIUsageLog.aggregate({ where: { createdAt: { gte: startOfMonth } }, _sum: { estimatedCost: true } }) ``` ### Error Handling ```typescript // ai-errors.ts export type AIErrorType = | 'RATE_LIMIT' | 'AUTHENTICATION' | 'NETWORK' | 'TIMEOUT' | 'VALIDATION' | 'PARSE_ERROR' | 'MODEL_NOT_FOUND' | 'UNKNOWN' export interface ClassifiedAIError { type: AIErrorType message: string retryable: boolean userMessage: string } export function classifyAIError(error: unknown): ClassifiedAIError { const errorMsg = error instanceof Error ? error.message : String(error) // Rate limit if (errorMsg.includes('rate_limit') || errorMsg.includes('429')) { return { type: 'RATE_LIMIT', message: errorMsg, retryable: true, userMessage: 'AI service is currently at capacity. Please try again in a few minutes.' } } // Authentication if (errorMsg.includes('authentication') || errorMsg.includes('401') || errorMsg.includes('api key')) { return { type: 'AUTHENTICATION', message: errorMsg, retryable: false, userMessage: 'OpenAI API key is invalid or missing. Please check Settings > AI Configuration.' } } // Model not found if (errorMsg.includes('model') && errorMsg.includes('does not exist')) { return { type: 'MODEL_NOT_FOUND', message: errorMsg, retryable: false, userMessage: 'The configured AI model does not exist. Please check Settings > AI Configuration.' } } // Network if (errorMsg.includes('ECONNREFUSED') || errorMsg.includes('ETIMEDOUT') || errorMsg.includes('network')) { return { type: 'NETWORK', message: errorMsg, retryable: true, userMessage: 'Unable to reach OpenAI servers. Please check your internet connection.' } } // Timeout if (errorMsg.includes('timeout')) { return { type: 'TIMEOUT', message: errorMsg, retryable: true, userMessage: 'AI request timed out. Please try again or reduce batch size.' } } // Validation if (errorMsg.includes('validation') || errorMsg.includes('invalid')) { return { type: 'VALIDATION', message: errorMsg, retryable: false, userMessage: 'Invalid request to AI service. Please contact support.' } } // Parse error if (error instanceof SyntaxError || errorMsg.includes('parse') || errorMsg.includes('JSON')) { return { type: 'PARSE_ERROR', message: errorMsg, retryable: false, userMessage: 'AI response could not be understood. This has been logged for review.' } } // Unknown return { type: 'UNKNOWN', message: errorMsg, retryable: true, userMessage: 'An unexpected error occurred with the AI service. Please try again.' } } export function logAIError(service: string, operation: string, error: ClassifiedAIError): void { console.error(`[AI Error] ${service}.${operation}:`, { type: error.type, message: error.message, retryable: error.retryable }) // Log to database for admin review prisma.auditLog.create({ data: { action: 'AI_ERROR', entityType: 'AIService', details: { service, operation, errorType: error.type, message: error.message } } }).catch(console.error) } ``` ### Streaming vs Non-Streaming **Current Implementation**: Non-streaming (wait for full response) ```typescript const response = await openai.chat.completions.create({ model: 'gpt-4', messages: [...], stream: false // Wait for complete response }) const content = response.choices[0]?.message?.content ``` **Future Enhancement**: Streaming for long summaries ```typescript // For evaluation summaries, stream to UI as tokens arrive const stream = await openai.chat.completions.create({ model: 'gpt-4', messages: [...], stream: true }) for await (const chunk of stream) { const delta = chunk.choices[0]?.delta?.content || '' // Send to frontend via SSE res.write(`data: ${JSON.stringify({ delta })}\n\n`) } ``` --- ## Privacy & Security ### Data Minimization **Principle**: Send only the minimum data required for the AI task. | Service | Data Sent | Data NOT Sent | |---------|-----------|---------------| | **Filtering** | Project title, description (truncated), category, tags, file types, team size | Team names, emails, phone numbers, full descriptions, user IDs | | **Assignment** | Juror expertise tags, current assignment count, max assignments, project tags | Juror names, emails, user IDs, juror institutions | | **Summary** | Evaluation scores, sanitized feedback text | Juror names, emails, user IDs | | **Tagging** | Project title, description (truncated), category, tags | Team names, emails, phone numbers, user IDs | | **Eligibility** | Project category, tags, country, founded year | Team names, emails, phone numbers, full descriptions | ### PII Stripping Workflow ```mermaid graph TD A[Raw Data] --> B{Contains PII?} B -->|Yes| C[Sanitize Text] B -->|No| D[Validate] C --> E[Replace IDs] E --> F[Truncate Descriptions] F --> D D --> G{GDPR Pass?} G -->|No| H[Reject + Log] G -->|Yes| I[Send to AI] I --> J[AI Response] J --> K[De-anonymize] K --> L[Return Real Results] ``` ### Audit Logging ```typescript // All AI requests logged with full context await logAIUsage({ userId: 'admin-123', action: 'FILTERING', entityType: 'Round', entityId: 'round-456', model: 'gpt-4', promptTokens: 1234, completionTokens: 567, totalTokens: 1801, batchSize: 20, itemsProcessed: 20, status: 'SUCCESS', estimatedCost: 0.0891, metadata: { criteriaText: 'Reject spam projects...', passRate: 0.75 } }) // Failed requests also logged await logAIUsage({ userId: 'admin-123', action: 'ASSIGNMENT', model: 'gpt-4', status: 'ERROR', errorMessage: 'Rate limit exceeded', metadata: { jurorCount: 15, projectCount: 45 } }) ``` ### Data Retention Policy ```typescript // AI usage logs retained for 90 days, then archived export async function archiveOldAILogs(): Promise { const cutoffDate = new Date() cutoffDate.setDate(cutoffDate.getDate() - 90) const oldLogs = await prisma.aIUsageLog.findMany({ where: { createdAt: { lt: cutoffDate } } }) // Move to archive storage (S3, cold storage, etc.) await archiveService.store('ai-logs', oldLogs) // Delete from hot database await prisma.aIUsageLog.deleteMany({ where: { createdAt: { lt: cutoffDate } } }) } // Run nightly via cron ``` ### Opt-Out Capabilities ```typescript // Per-service opt-out in settings interface AISettings { ai_enabled: boolean // Master toggle ai_filtering_enabled: boolean ai_assignment_enabled: boolean ai_summary_enabled: boolean ai_tagging_enabled: boolean ai_eligibility_enabled: boolean } // Check before each AI call export async function isAIServiceEnabled(service: AIServiceType): Promise { const masterEnabled = await getSystemSetting('ai_enabled') if (!masterEnabled) return false const serviceEnabled = await getSystemSetting(`ai_${service}_enabled`) return serviceEnabled !== 'false' } // Graceful fallback when disabled if (!await isAIServiceEnabled('assignment')) { return generateFallbackAssignments(jurors, projects, constraints) } ``` --- ## Admin AI Controls ### AI Configuration Dashboard ``` ┌────────────────────────────────────────────────────────────────────┐ │ Settings > AI Configuration │ ├────────────────────────────────────────────────────────────────────┤ │ │ │ General Settings │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ OpenAI API Key: [sk-proj-••••••••••••••••••••••••••] [ Test ] │ │ ✓ Connected (last verified: 2024-02-15 14:30) │ │ │ │ Default Model: [ gpt-4 ▼ ] │ │ Estimated cost: $0.03 per 1K tokens (input) │ │ │ │ [ ] Enable AI Services ← Master toggle │ │ │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ Service-Specific Settings │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ [✓] AI Filtering │ │ Batch Size: [ 20 ] projects per API call │ │ Parallel Batches: [ 2 ] concurrent requests │ │ Model Override: [ (use default) ▼ ] │ │ │ │ [✓] AI Assignment │ │ Batch Size: [ 15 ] projects per API call │ │ Model Override: [ (use default) ▼ ] │ │ │ │ [✓] AI Evaluation Summaries │ │ Auto-generate: [✓] When all evaluations submitted │ │ Model Override: [ gpt-4-turbo ▼ ] (faster for summaries) │ │ │ │ [✓] AI Tagging │ │ Confidence Threshold: [ 0.5 ] (0.0 - 1.0) │ │ Max Tags per Project: [ 5 ] │ │ Auto-tag on Submit: [✓] Apply tags when project submitted │ │ │ │ [✓] AI Award Eligibility │ │ Model Override: [ (use default) ▼ ] │ │ │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ Usage & Cost Tracking │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ This Month: 1,234,567 tokens | Est. Cost: $42.18 │ │ Last Month: 987,654 tokens | Est. Cost: $31.45 │ │ │ │ [ View Detailed Usage Log ] │ │ │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ [ Save Changes ] [ Reset to Defaults ] │ │ │ └────────────────────────────────────────────────────────────────────┘ ``` ### Rubric Editor for Filtering ``` ┌────────────────────────────────────────────────────────────────────┐ │ Round 2: FILTERING > AI Screening Rules │ ├────────────────────────────────────────────────────────────────────┤ │ │ │ Rule: Spam & Quality Check │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ Criteria (Plain Language): │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ Projects should demonstrate clear ocean conservation value. │ │ │ │ │ │ │ │ Reject projects that: │ │ │ │ - Are spam, test submissions, or joke entries │ │ │ │ - Have no meaningful description (< 100 words) │ │ │ │ - Are unrelated to ocean conservation │ │ │ │ - Are duplicate submissions │ │ │ │ │ │ │ │ Flag for manual review: │ │ │ │ - Projects with unclear focus or vague descriptions │ │ │ │ - Projects that may be legitimate but borderline │ │ │ └──────────────────────────────────────────────────────────────┘ │ │ │ │ Action: ( ) PASS (•) REJECT ( ) FLAG │ │ │ │ Priority: [ 1 ] (1 = highest, 10 = lowest) │ │ │ │ [ ] Active │ │ │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ Preview & Test │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ Test this rule on sample projects: │ │ │ │ [✓] SaveTheSea - AI-powered coral reef monitoring │ │ Result: PASS (confidence: 0.92) │ │ Reasoning: Clear ocean conservation focus, detailed approach │ │ │ │ [✓] Test Project - Just testing the form │ │ Result: REJECT (confidence: 0.98) │ │ Reasoning: Appears to be a test submission with no real value │ │ │ │ [✓] Marine Plastic Solution (vague description) │ │ Result: FLAG (confidence: 0.65) │ │ Reasoning: Ocean-related but description lacks detail │ │ │ │ [ Run Test on 10 Random Projects ] │ │ │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ [ Save Rule ] [ Delete Rule ] [ Cancel ] │ │ │ └────────────────────────────────────────────────────────────────────┘ ``` ### AI Results Review Interface ``` ┌────────────────────────────────────────────────────────────────────┐ │ Round 2: FILTERING > AI Results Review │ ├────────────────────────────────────────────────────────────────────┤ │ │ │ AI Screening Complete │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ Processed: 120 projects │ │ Passed: 87 (72.5%) │ │ Rejected: 21 (17.5%) │ │ Flagged: 12 (10.0%) │ │ │ │ Tokens Used: 45,678 | Est. Cost: $1.37 │ │ Model: gpt-4 | Batches: 6 of 20 projects each │ │ │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ Flagged Projects (Require Manual Review) │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ [ ] Marine Plastic Solution │ │ Confidence: 0.65 │ │ AI Reasoning: Ocean-related but description lacks sufficient │ │ detail. Unclear commercial model. May be early- │ │ stage concept that needs more development. │ │ Quality Score: 6/10 │ │ Spam Risk: No │ │ │ │ Admin Decision: ( ) PASS ( ) REJECT ( ) Keep Flagged │ │ │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ [ ] Ocean Cleanup AI │ │ Confidence: 0.58 │ │ AI Reasoning: Uses AI buzzwords extensively but lacks technical│ │ specifics. May be over-promising. Borderline. │ │ Quality Score: 5/10 │ │ Spam Risk: No │ │ │ │ Admin Decision: ( ) PASS ( ) REJECT ( ) Keep Flagged │ │ │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ [ Review All Flagged (12) ] [ Bulk Approve ] [ Bulk Reject ] │ │ │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ Rejected Projects (Can Override) │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ [✓] Test Submission │ │ Confidence: 0.98 │ │ AI Reasoning: Obvious test submission with placeholder text. │ │ Override: [ ] Approve This Project │ │ │ │ [✓] Spam Project 123 │ │ Confidence: 0.95 │ │ AI Reasoning: Unrelated to ocean conservation, generic content.│ │ Override: [ ] Approve This Project │ │ │ │ [ Show All Rejected (21) ] │ │ │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ [ Finalize Results ] [ Export CSV ] [ Re-run AI Screening ] │ │ │ └────────────────────────────────────────────────────────────────────┘ ``` ### Cost Monitoring Dashboard ``` ┌────────────────────────────────────────────────────────────────────┐ │ Analytics > AI Usage & Costs │ ├────────────────────────────────────────────────────────────────────┤ │ │ │ Overview (February 2024) │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ Total API Calls: 1,234 │ │ Total Tokens: 4,567,890 │ │ Estimated Cost: $137.04 │ │ Avg Cost per Call: $0.11 │ │ │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ Cost Breakdown by Service │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ Filtering: $45.67 (33%) ████████░░░ │ │ Assignment: $38.21 (28%) ███████░░░░ │ │ Evaluation Summary: $28.90 (21%) █████░░░░░░ │ │ Tagging: $15.12 (11%) ███░░░░░░░░ │ │ Award Eligibility: $9.14 (7%) ██░░░░░░░░░ │ │ │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ Token Usage Trend │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ 500K ┤ ╭─ │ │ 400K ┤ ╭────────╯ │ │ 300K ┤ ╭────────────╯ │ │ 200K ┤ ╭─────────────╯ │ │ 100K ┤ ╭────────╯ │ │ 0 ┼───┴──────────────────────────────────────────────────── │ │ Feb 1 Feb 8 Feb 15 Feb 22 Feb 29 │ │ │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ Recent API Calls │ │ ────────────────────────────────────────────────────────────────── │ │ │ │ 2024-02-15 14:30 | Filtering | Round 2 | 20 projects │ │ gpt-4 | 3,456 tokens | $0.10 | SUCCESS │ │ │ │ 2024-02-15 14:15 | Assignment | Round 3 | 15 projects │ │ gpt-4 | 4,123 tokens | $0.12 | SUCCESS │ │ │ │ 2024-02-15 13:45 | Summary | Proj-123 | 5 evaluations │ │ gpt-4-turbo | 1,890 tokens | $0.02 | ✓ │ │ │ │ [ View All Logs ] [ Export CSV ] [ Set Budget Alert ] │ │ │ └────────────────────────────────────────────────────────────────────┘ ``` --- ## API Changes ### New tRPC Procedures #### AI Router (`src/server/routers/ai.ts`) ```typescript export const aiRouter = createTRPCRouter({ // Test OpenAI connection testConnection: adminProcedure .mutation(async ({ ctx }) => { const openai = await getOpenAI() if (!openai) { throw new TRPCError({ code: 'PRECONDITION_FAILED', message: 'OpenAI not configured' }) } try { await openai.models.list() return { connected: true, message: 'OpenAI connection successful' } } catch (error) { return { connected: false, message: error.message } } }), // Get AI usage stats getUsageStats: adminProcedure .input(z.object({ startDate: z.date(), endDate: z.date() })) .query(async ({ ctx, input }) => { const logs = await ctx.prisma.aIUsageLog.findMany({ where: { createdAt: { gte: input.startDate, lte: input.endDate } } }) const totalTokens = logs.reduce((sum, log) => sum + log.totalTokens, 0) const totalCost = logs.reduce((sum, log) => sum + (log.estimatedCost || 0), 0) const byService = logs.reduce((acc, log) => { const service = log.action if (!acc[service]) { acc[service] = { count: 0, tokens: 0, cost: 0 } } acc[service].count++ acc[service].tokens += log.totalTokens acc[service].cost += log.estimatedCost || 0 return acc }, {} as Record) return { totalCalls: logs.length, totalTokens, totalCost, avgCostPerCall: totalCost / logs.length, byService } }), // Tag a single project tagProject: adminProcedure .input(z.object({ projectId: z.string() })) .mutation(async ({ ctx, input }) => { const result = await tagProject(input.projectId, ctx.session.user.id) return result }), // Get tag suggestions (preview only, don't apply) getTagSuggestions: adminProcedure .input(z.object({ projectId: z.string() })) .query(async ({ ctx, input }) => { const suggestions = await getTagSuggestions(input.projectId, ctx.session.user.id) return suggestions }), // Generate evaluation summary generateEvaluationSummary: adminProcedure .input(z.object({ projectId: z.string(), roundId: z.string() })) .mutation(async ({ ctx, input }) => { const summary = await generateSummary({ projectId: input.projectId, stageId: input.roundId, userId: ctx.session.user.id, prisma: ctx.prisma }) return summary }), // Generate AI assignments generateAssignments: adminProcedure .input(z.object({ roundId: z.string(), constraints: z.object({ requiredReviewsPerProject: z.number(), minAssignmentsPerJuror: z.number().optional(), maxAssignmentsPerJuror: z.number().optional() }) })) .mutation(async ({ ctx, input }) => { const round = await ctx.prisma.round.findUnique({ where: { id: input.roundId }, include: { juryGroup: { include: { members: { include: { user: true } } } } } }) if (!round?.juryGroup) { throw new TRPCError({ code: 'NOT_FOUND', message: 'Round or jury group not found' }) } const projects = await ctx.prisma.project.findMany({ where: { projectRoundStates: { some: { roundId: input.roundId, state: 'PASSED' } } }, include: { _count: { select: { assignments: true } } } }) const existingAssignments = await ctx.prisma.assignment.findMany({ where: { roundId: input.roundId }, select: { userId: true, projectId: true } }) const result = await generateAIAssignments( round.juryGroup.members.map(m => m.user), projects, { ...input.constraints, existingAssignments }, ctx.session.user.id, input.roundId ) return result }), // Run filtering on a round runFiltering: adminProcedure .input(z.object({ roundId: z.string() })) .mutation(async ({ ctx, input }) => { const round = await ctx.prisma.round.findUnique({ where: { id: input.roundId } }) if (round?.roundType !== 'FILTERING') { throw new TRPCError({ code: 'BAD_REQUEST', message: 'Not a filtering round' }) } const rules = await ctx.prisma.filteringRule.findMany({ where: { roundId: input.roundId, isActive: true }, orderBy: { priority: 'asc' } }) const projects = await ctx.prisma.project.findMany({ where: { projectRoundStates: { some: { roundId: input.roundId } } }, include: { files: true, _count: { select: { teamMembers: true } } } }) const results = await executeFilteringRules( rules, projects, ctx.session.user.id, input.roundId ) // Store results for (const result of results) { await ctx.prisma.filteringResult.upsert({ where: { projectId_roundId: { projectId: result.projectId, roundId: input.roundId } }, create: { projectId: result.projectId, roundId: input.roundId, outcome: result.outcome, ruleResultsJson: result.ruleResults, aiScreeningJson: result.aiScreeningJson }, update: { outcome: result.outcome, ruleResultsJson: result.ruleResults, aiScreeningJson: result.aiScreeningJson } }) } return { total: results.length, passed: results.filter(r => r.outcome === 'PASSED').length, rejected: results.filter(r => r.outcome === 'FILTERED_OUT').length, flagged: results.filter(r => r.outcome === 'FLAGGED').length } }), // Check award eligibility checkAwardEligibility: adminProcedure .input(z.object({ awardId: z.string() })) .mutation(async ({ ctx, input }) => { const award = await ctx.prisma.specialAward.findUnique({ where: { id: input.awardId } }) if (!award?.useAiEligibility || !award.criteriaText) { throw new TRPCError({ code: 'BAD_REQUEST', message: 'AI eligibility not enabled for this award' }) } const projects = await ctx.prisma.project.findMany({ where: { competitionId: award.competitionId }, include: { files: true, _count: { select: { teamMembers: true } } } }) const results = await aiInterpretCriteria( award.criteriaText, projects, ctx.session.user.id, award.id ) // Store results for (const result of results) { await ctx.prisma.awardEligibility.upsert({ where: { specialAwardId_projectId: { specialAwardId: award.id, projectId: result.projectId } }, create: { specialAwardId: award.id, projectId: result.projectId, isEligible: result.eligible, confidence: result.confidence, reasoning: result.reasoning, method: result.method }, update: { isEligible: result.eligible, confidence: result.confidence, reasoning: result.reasoning, method: result.method } }) } return { total: results.length, eligible: results.filter(r => r.eligible).length, ineligible: results.filter(r => !r.eligible).length } }) }) ``` --- ## Service Functions ### Complete Function Signatures ```typescript // ai-filtering.ts export function evaluateFieldRule( config: FieldRuleConfig, project: ProjectForFiltering ): { passed: boolean; action: 'PASS' | 'REJECT' | 'FLAG' } export function evaluateDocumentRule( config: DocumentCheckConfig, project: ProjectForFiltering ): { passed: boolean; action: 'PASS' | 'REJECT' | 'FLAG' } export async function executeAIScreening( config: AIScreeningConfig, projects: ProjectForFiltering[], userId?: string, entityId?: string, onProgress?: ProgressCallback ): Promise> export async function executeFilteringRules( rules: FilteringRuleInput[], projects: ProjectForFiltering[], userId?: string, stageId?: string, onProgress?: ProgressCallback ): Promise // ai-assignment.ts export async function generateAIAssignments( jurors: JurorForAssignment[], projects: ProjectForAssignment[], constraints: AssignmentConstraints, userId?: string, entityId?: string, onProgress?: AssignmentProgressCallback ): Promise export function generateFallbackAssignments( jurors: JurorForAssignment[], projects: ProjectForAssignment[], constraints: AssignmentConstraints ): AIAssignmentResult // ai-evaluation-summary.ts export function anonymizeEvaluations( evaluations: EvaluationForSummary[] ): AnonymizedEvaluation[] export function buildSummaryPrompt( anonymizedEvaluations: AnonymizedEvaluation[], projectTitle: string, criteriaLabels: string[] ): string export function computeScoringPatterns( evaluations: EvaluationForSummary[], criteriaLabels: CriterionDef[] ): ScoringPatterns export async function generateSummary({ projectId, stageId, userId, prisma }: { projectId: string stageId: string userId: string prisma: PrismaClient }): Promise // ai-tagging.ts export async function getTaggingSettings(): Promise<{ enabled: boolean maxTags: number }> export async function getAvailableTags(): Promise export async function tagProject( projectId: string, userId?: string ): Promise export async function getTagSuggestions( projectId: string, userId?: string ): Promise export async function addProjectTag( projectId: string, tagId: string ): Promise export async function removeProjectTag( projectId: string, tagId: string ): Promise // ai-award-eligibility.ts export function applyAutoTagRules( rules: AutoTagRule[], projects: ProjectForEligibility[] ): Map export async function aiInterpretCriteria( criteriaText: string, projects: ProjectForEligibility[], userId?: string, awardId?: string ): Promise // anonymization.ts export function sanitizeText(text: string): string export function truncateAndSanitize( text: string | null | undefined, maxLength: number ): string export function anonymizeForAI( jurors: JurorInput[], projects: ProjectInput[] ): AnonymizationResult export function anonymizeProjectForAI( project: ProjectWithRelations, index: number, context: DescriptionContext ): AnonymizedProjectForAI export function anonymizeProjectsForAI( projects: ProjectWithRelations[], context: DescriptionContext ): { anonymized: AnonymizedProjectForAI[] mappings: ProjectAIMapping[] } export function deanonymizeResults( results: T[], jurorMappings: JurorMapping[], projectMappings: ProjectMapping[] ): (T & { realJurorId: string; realProjectId: string })[] export function validateNoPersonalData( data: Record ): PIIValidationResult export function enforceGDPRCompliance(data: unknown[]): void export function validateAnonymization(data: AnonymizationResult): boolean export function validateAnonymizedProjects( projects: AnonymizedProjectForAI[] ): boolean export function toProjectWithRelations(project: unknown): ProjectWithRelations ``` --- ## Edge Cases | Scenario | Behavior | Fallback | |----------|----------|----------| | **OpenAI API key missing** | AI services disabled | Use algorithm-based fallbacks | | **OpenAI rate limit hit** | Queue requests, retry with exponential backoff | Fail gracefully after 3 retries | | **AI returns invalid JSON** | Parse error logged, flagged for manual review | Mark items as "needs review" | | **AI model doesn't exist** | Throw clear error with model name | Suggest checking Settings > AI Config | | **Anonymization validation fails** | Throw error, log GDPR violation | Reject AI call, require manual review | | **All projects rejected by AI** | Admin gets warning notification | Admin reviews AI reasoning | | **Zero tag suggestions** | Return empty array | No error, just no tags applied | | **Confidence too low (<0.5)** | Tag not applied automatically | Admin can manually apply | | **Duplicate project detected (>0.85 similarity)** | Flagged for admin review | Admin marks as duplicate or false positive | | **Mentor workspace inactive (>14 days)** | Intervention alert sent to admin | Admin reaches out to mentor | | **Winner ranking unclear (close gap <0.3)** | AI flags as "close call", suggests review | Jury deliberates again | | **Batch size too large (>50)** | Clamped to 50 projects per batch | Automatically split into multiple batches | | **Description too long (>500 chars)** | Truncated with "..." | AI works with truncated text | | **PII in feedback text** | Sanitized before sending to AI | Emails/phones replaced with `[email removed]` | | **Juror has no expertise tags** | Scored as 0.5 (neutral) in assignment | Fallback algorithm includes them for load balancing | | **Project has no description** | AI flags as low quality | Admin decides whether to reject | | **All jurors at capacity** | Assignment fails with clear error | Admin adjusts caps or adds more jurors | | **Award criteria too vague** | AI gives low-confidence results | Admin refines criteria and re-runs | | **Evaluation summary with <3 evaluations** | AI summary still generated, marked as "limited data" | Admin aware that consensus may be weak | | **Streaming response timeout** | Fall back to non-streaming request | Complete response returned after wait | --- ## Integration Map ### AI Services × Round Types ```mermaid graph TD R1[Round 1: INTAKE] --> T[ai-tagging.ts] T --> TAG[Auto-tag on submit] R2[Round 2: FILTERING] --> F[ai-filtering.ts] F --> SCREEN[AI screening + rules] F --> DUP[ai-duplicate-detection] DUP --> EMBED[Embedding similarity] R3[Round 3: EVALUATION] --> A[ai-assignment.ts] A --> ASSIGN[Generate assignments] A --> E[ai-evaluation-summary.ts] E --> SUM[Summarize evaluations] R4[Round 4: SUBMISSION] --> NONE1[No AI service] R5[Round 5: EVALUATION] --> A A --> E R6[Round 6: MENTORING] --> M[ai-mentoring-insights.ts] M --> INSIGHT[Workspace analysis] R7[Round 7: LIVE_FINAL] --> NONE2[No AI service] R8[Round 8: CONFIRMATION] --> C[ai-confirmation-helper.ts] C --> EXPLAIN[Ranking explanation] AWARDS[Special Awards] --> AE[ai-award-eligibility.ts] AE --> ELIG[Eligibility check] ``` ### Data Flow Diagram ```mermaid sequenceDiagram participant Admin participant API participant Service participant Anonymization participant OpenAI participant DB Admin->>API: Trigger AI service API->>Service: Call service function Service->>DB: Fetch data DB-->>Service: Raw data Service->>Anonymization: Anonymize data Anonymization->>Anonymization: Strip PII Anonymization->>Anonymization: Replace IDs Anonymization->>Anonymization: Validate GDPR Anonymization-->>Service: Anonymized + mappings Service->>OpenAI: API call OpenAI-->>Service: AI response Service->>Anonymization: De-anonymize results Anonymization-->>Service: Real IDs Service->>DB: Store results DB-->>Service: Confirmation Service->>DB: Log AI usage Service-->>API: Results API-->>Admin: Success + data ``` --- This completes the extremely detailed AI Services documentation covering all current services, new services for the redesign, anonymization pipeline, prompt engineering, OpenAI integration, privacy/security, admin controls, API changes, service functions, edge cases, and integration map. Total lines: 2,900+