3385 lines
110 KiB
Markdown
3385 lines
110 KiB
Markdown
# AI Services Architecture
|
||
|
||
## Overview
|
||
|
||
AI services power automation across all round types in the MOPC platform. All AI calls anonymize data before sending to OpenAI, ensuring GDPR compliance and privacy protection. These services enable intelligent filtering, smart jury assignments, evaluation synthesis, automatic tagging, and award eligibility assessment.
|
||
|
||
### Core Principles
|
||
|
||
1. **Privacy First**: All data is anonymized before AI processing (no PII sent to OpenAI)
|
||
2. **Graceful Degradation**: Fallback algorithms when AI is unavailable
|
||
3. **Cost Awareness**: Batching, token tracking, and cost monitoring
|
||
4. **Audit Trail**: All AI requests logged with inputs, outputs, and token usage
|
||
5. **Admin Control**: Per-service toggles, configuration overrides, manual review
|
||
6. **Transparency**: AI reasoning exposed to admins for validation
|
||
|
||
### Service Inventory
|
||
|
||
| Service | Purpose | Primary Model | Input Data | Output |
|
||
|---------|---------|---------------|------------|--------|
|
||
| **ai-filtering.ts** | Automated project screening | GPT-4 | Projects + rubric | Pass/Reject/Flag + scores |
|
||
| **ai-assignment.ts** | Jury-project matching | GPT-4 | Jurors + projects + constraints | Assignment suggestions |
|
||
| **ai-evaluation-summary.ts** | Synthesis of evaluations | GPT-4-turbo | All evaluations for a project | Strengths, weaknesses, themes |
|
||
| **ai-tagging.ts** | Auto-tag projects | GPT-4 | Project description | Tag suggestions + confidence |
|
||
| **ai-award-eligibility.ts** | Award eligibility assessment | GPT-4 | Projects + award criteria | Eligibility scores + reasoning |
|
||
| **anonymization.ts** | PII stripping pipeline | N/A | Raw project/user data | Anonymized data + mappings |
|
||
|
||
---
|
||
|
||
## Current AI Services
|
||
|
||
### 1. AI Filtering Service (`ai-filtering.ts`)
|
||
|
||
**Purpose**: Automate project screening in Round 2 (FILTERING) using admin-defined rubrics and AI interpretation of plain-language criteria.
|
||
|
||
#### Input Data
|
||
|
||
```typescript
|
||
interface ProjectForFiltering {
|
||
id: string
|
||
title: string
|
||
description?: string | null
|
||
competitionCategory?: CompetitionCategory | null
|
||
foundedAt?: Date | null
|
||
country?: string | null
|
||
geographicZone?: string | null
|
||
tags: string[]
|
||
oceanIssue?: OceanIssue | null
|
||
wantsMentorship?: boolean | null
|
||
institution?: string | null
|
||
submissionSource?: SubmissionSource
|
||
submittedAt?: Date | null
|
||
files: Array<{
|
||
id: string
|
||
fileName: string
|
||
fileType?: FileType | null
|
||
}>
|
||
_count?: {
|
||
teamMembers?: number
|
||
files?: number
|
||
}
|
||
}
|
||
|
||
interface FilteringRuleInput {
|
||
id: string
|
||
name: string
|
||
ruleType: 'FIELD_BASED' | 'DOCUMENT_CHECK' | 'AI_SCREENING'
|
||
configJson: Prisma.JsonValue
|
||
priority: number
|
||
isActive: boolean
|
||
}
|
||
```
|
||
|
||
#### Rule Types
|
||
|
||
**Field-Based Rules** (No AI):
|
||
```typescript
|
||
type FieldRuleConfig = {
|
||
conditions: FieldRuleCondition[]
|
||
logic: 'AND' | 'OR'
|
||
action: 'PASS' | 'REJECT' | 'FLAG'
|
||
}
|
||
|
||
type FieldRuleCondition = {
|
||
field: 'competitionCategory' | 'foundedAt' | 'country' | 'geographicZone' | 'tags' | 'oceanIssue'
|
||
operator: 'equals' | 'not_equals' | 'greater_than' | 'less_than' | 'contains' | 'in' | 'not_in' | 'older_than_years' | 'newer_than_years' | 'is_empty'
|
||
value: string | number | string[]
|
||
}
|
||
|
||
// Example: Reject projects older than 10 years
|
||
{
|
||
conditions: [
|
||
{ field: 'foundedAt', operator: 'older_than_years', value: 10 }
|
||
],
|
||
logic: 'AND',
|
||
action: 'REJECT'
|
||
}
|
||
|
||
// Example: Flag projects from specific countries
|
||
{
|
||
conditions: [
|
||
{ field: 'country', operator: 'in', value: ['US', 'CN', 'RU'] }
|
||
],
|
||
logic: 'OR',
|
||
action: 'FLAG'
|
||
}
|
||
```
|
||
|
||
**Document Check Rules** (No AI):
|
||
```typescript
|
||
type DocumentCheckConfig = {
|
||
requiredFileTypes?: string[] // ['pdf', 'docx']
|
||
minFileCount?: number
|
||
action: 'PASS' | 'REJECT' | 'FLAG'
|
||
}
|
||
|
||
// Example: Require at least 2 PDF files
|
||
{
|
||
requiredFileTypes: ['pdf'],
|
||
minFileCount: 2,
|
||
action: 'REJECT'
|
||
}
|
||
```
|
||
|
||
**AI Screening Rules** (OpenAI):
|
||
```typescript
|
||
type AIScreeningConfig = {
|
||
criteriaText: string // Plain-language rubric
|
||
action: 'PASS' | 'REJECT' | 'FLAG'
|
||
batchSize?: number // 1-50, default 20
|
||
parallelBatches?: number // 1-10, default 1
|
||
}
|
||
|
||
// Example: Detect spam/low-quality projects
|
||
{
|
||
criteriaText: `
|
||
Projects should demonstrate clear ocean conservation value.
|
||
Reject projects that:
|
||
- Are spam, test submissions, or joke entries
|
||
- Have no meaningful description
|
||
- Are unrelated to ocean conservation
|
||
- Are duplicate submissions
|
||
`,
|
||
action: 'REJECT',
|
||
batchSize: 20,
|
||
parallelBatches: 2
|
||
}
|
||
```
|
||
|
||
#### AI Screening Process
|
||
|
||
1. **Anonymization**: Strip PII from projects
|
||
```typescript
|
||
const { anonymized, mappings } = anonymizeProjectsForAI(projects, 'FILTERING')
|
||
|
||
// Before: { id: "proj-abc123", title: "SaveTheSea by John Doe (john@example.com)", ... }
|
||
// After: { project_id: "P1", title: "SaveTheSea by Team 1", ... }
|
||
```
|
||
|
||
2. **Validation**: Ensure no PII leaked
|
||
```typescript
|
||
if (!validateAnonymizedProjects(anonymized)) {
|
||
throw new Error('GDPR compliance check failed')
|
||
}
|
||
```
|
||
|
||
3. **Batch Processing**: Process projects in configurable batches
|
||
```typescript
|
||
const batchSize = Math.min(MAX_BATCH_SIZE, config.batchSize ?? 20)
|
||
const parallelBatches = Math.min(MAX_PARALLEL_BATCHES, config.parallelBatches ?? 1)
|
||
|
||
for (let i = 0; i < batches.length; i += parallelBatches) {
|
||
const parallelChunk = batches.slice(i, i + parallelBatches)
|
||
const results = await Promise.all(parallelChunk.map(processAIBatch))
|
||
}
|
||
```
|
||
|
||
4. **OpenAI Call**: Send anonymized batch with criteria
|
||
```typescript
|
||
const prompt = `CRITERIA: ${criteriaText}
|
||
PROJECTS: ${JSON.stringify(anonymized)}
|
||
Evaluate and return JSON.`
|
||
|
||
const response = await openai.chat.completions.create({
|
||
model: 'gpt-4',
|
||
messages: [
|
||
{ role: 'system', content: AI_SCREENING_SYSTEM_PROMPT },
|
||
{ role: 'user', content: prompt }
|
||
],
|
||
response_format: { type: 'json_object' },
|
||
temperature: 0.3
|
||
})
|
||
```
|
||
|
||
5. **Result Parsing**:
|
||
```typescript
|
||
interface AIScreeningResult {
|
||
meetsCriteria: boolean
|
||
confidence: number // 0.0 - 1.0
|
||
reasoning: string
|
||
qualityScore: number // 1-10
|
||
spamRisk: boolean
|
||
}
|
||
|
||
// AI returns:
|
||
{
|
||
"projects": [
|
||
{
|
||
"project_id": "P1",
|
||
"meets_criteria": true,
|
||
"confidence": 0.85,
|
||
"reasoning": "Clear ocean conservation focus, well-documented approach",
|
||
"quality_score": 8,
|
||
"spam_risk": false
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
6. **De-anonymization**: Map results back to real IDs
|
||
```typescript
|
||
const mapping = mappings.find(m => m.anonymousId === "P1")
|
||
results.set(mapping.realId, aiResult)
|
||
```
|
||
|
||
#### Output
|
||
|
||
```typescript
|
||
interface ProjectFilteringResult {
|
||
projectId: string
|
||
outcome: 'PASSED' | 'FILTERED_OUT' | 'FLAGGED'
|
||
ruleResults: RuleResult[]
|
||
aiScreeningJson?: {
|
||
[ruleId: string]: AIScreeningResult
|
||
}
|
||
}
|
||
|
||
interface RuleResult {
|
||
ruleId: string
|
||
ruleName: string
|
||
ruleType: string
|
||
passed: boolean
|
||
action: 'PASS' | 'REJECT' | 'FLAG'
|
||
reasoning?: string // Only for AI_SCREENING
|
||
}
|
||
```
|
||
|
||
**Outcome Logic**:
|
||
- If ANY rule with `action: 'REJECT'` fails → `FILTERED_OUT`
|
||
- Else if ANY rule with `action: 'FLAG'` fails → `FLAGGED`
|
||
- Else → `PASSED`
|
||
|
||
#### Integration with Filtering Round
|
||
|
||
```typescript
|
||
// Round 2: FILTERING
|
||
const filteringRound = await prisma.round.findFirst({
|
||
where: { competitionId, roundType: 'FILTERING' }
|
||
})
|
||
|
||
const rules = await prisma.filteringRule.findMany({
|
||
where: { roundId: filteringRound.id, isActive: true },
|
||
orderBy: { priority: 'asc' }
|
||
})
|
||
|
||
const projects = await prisma.project.findMany({
|
||
where: { competitionId },
|
||
include: { files: true, _count: { select: { teamMembers: true } } }
|
||
})
|
||
|
||
const results = await executeFilteringRules(rules, projects, userId, roundId)
|
||
|
||
// Store results
|
||
for (const result of results) {
|
||
await prisma.filteringResult.create({
|
||
data: {
|
||
projectId: result.projectId,
|
||
roundId: filteringRound.id,
|
||
outcome: result.outcome,
|
||
ruleResultsJson: result.ruleResults,
|
||
aiScreeningJson: result.aiScreeningJson
|
||
}
|
||
})
|
||
|
||
// Update project round state
|
||
await prisma.projectRoundState.update({
|
||
where: { projectId_roundId: { projectId: result.projectId, roundId: filteringRound.id } },
|
||
data: {
|
||
state: result.outcome === 'PASSED' ? 'PASSED' : 'REJECTED',
|
||
metadataJson: { filteringOutcome: result.outcome }
|
||
}
|
||
})
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### 2. AI Assignment Service (`ai-assignment.ts`)
|
||
|
||
**Purpose**: Generate optimal jury-to-project assignments based on expertise matching, workload balancing, and constraints.
|
||
|
||
#### Input Data
|
||
|
||
```typescript
|
||
interface JurorForAssignment {
|
||
id: string
|
||
name?: string | null
|
||
email: string
|
||
expertiseTags: string[]
|
||
maxAssignments?: number | null
|
||
_count?: {
|
||
assignments: number
|
||
}
|
||
}
|
||
|
||
interface ProjectForAssignment {
|
||
id: string
|
||
title: string
|
||
description?: string | null
|
||
tags: string[]
|
||
teamName?: string | null
|
||
_count?: {
|
||
assignments: number
|
||
}
|
||
}
|
||
|
||
interface AssignmentConstraints {
|
||
requiredReviewsPerProject: number // e.g., 3 jurors per project
|
||
minAssignmentsPerJuror?: number // e.g., 5 projects minimum
|
||
maxAssignmentsPerJuror?: number // e.g., 20 projects maximum
|
||
jurorLimits?: Record<string, number> // Per-juror overrides
|
||
existingAssignments: Array<{
|
||
jurorId: string
|
||
projectId: string
|
||
}>
|
||
}
|
||
```
|
||
|
||
#### Anonymization & Batching
|
||
|
||
```typescript
|
||
// 1. Truncate descriptions (save tokens)
|
||
const truncatedProjects = projects.map(p => ({
|
||
...p,
|
||
description: truncateAndSanitize(p.description, DESCRIPTION_LIMITS.ASSIGNMENT) // 300 chars
|
||
}))
|
||
|
||
// 2. Anonymize data
|
||
const anonymizedData = anonymizeForAI(jurors, truncatedProjects)
|
||
|
||
// Before:
|
||
jurors = [
|
||
{ id: "user-123", email: "john@example.com", expertiseTags: ["Marine Biology", "AI"] },
|
||
{ id: "user-456", email: "jane@example.com", expertiseTags: ["Oceanography"] }
|
||
]
|
||
|
||
// After:
|
||
anonymizedData.jurors = [
|
||
{ anonymousId: "juror_001", expertiseTags: ["Marine Biology", "AI"], currentAssignmentCount: 5, maxAssignments: 20 },
|
||
{ anonymousId: "juror_002", expertiseTags: ["Oceanography"], currentAssignmentCount: 3, maxAssignments: 20 }
|
||
]
|
||
|
||
// 3. Validate anonymization
|
||
if (!validateAnonymization(anonymizedData)) {
|
||
return generateFallbackAssignments(jurors, projects, constraints)
|
||
}
|
||
|
||
// 4. Process in batches (15 projects per batch)
|
||
const BATCH_SIZE = 15
|
||
for (let i = 0; i < projects.length; i += BATCH_SIZE) {
|
||
const batchProjects = anonymizedData.projects.slice(i, i + BATCH_SIZE)
|
||
const suggestions = await processAssignmentBatch(openai, model, anonymizedData, batchProjects, constraints)
|
||
}
|
||
```
|
||
|
||
#### Prompt Structure
|
||
|
||
```typescript
|
||
const userPrompt = `JURORS: ${JSON.stringify(jurors)}
|
||
PROJECTS: ${JSON.stringify(projects)}
|
||
CONSTRAINTS: ${constraints.requiredReviewsPerProject} reviews/project, max ${constraints.maxAssignmentsPerJuror}/juror
|
||
EXISTING: ${JSON.stringify(anonymousExisting)}
|
||
Return JSON: {"assignments": [...]}`
|
||
|
||
const systemPrompt = `Match jurors to projects by expertise. Return JSON assignments.
|
||
Each: {juror_id, project_id, confidence_score: 0-1, expertise_match_score: 0-1, reasoning: str (1-2 sentences)}
|
||
Distribute workload fairly. Avoid assigning jurors at capacity.`
|
||
```
|
||
|
||
**Example AI Response**:
|
||
```json
|
||
{
|
||
"assignments": [
|
||
{
|
||
"juror_id": "juror_001",
|
||
"project_id": "project_003",
|
||
"confidence_score": 0.92,
|
||
"expertise_match_score": 0.88,
|
||
"reasoning": "Strong match on Marine Biology and AI tags. Juror has capacity."
|
||
},
|
||
{
|
||
"juror_id": "juror_002",
|
||
"project_id": "project_003",
|
||
"confidence_score": 0.75,
|
||
"expertise_match_score": 0.70,
|
||
"reasoning": "Oceanography expertise applies. Helps distribute workload."
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
#### Output
|
||
|
||
```typescript
|
||
interface AIAssignmentSuggestion {
|
||
jurorId: string // Real ID (de-anonymized)
|
||
projectId: string // Real ID (de-anonymized)
|
||
confidenceScore: number // 0-1
|
||
expertiseMatchScore: number // 0-1
|
||
reasoning: string
|
||
}
|
||
|
||
interface AIAssignmentResult {
|
||
success: boolean
|
||
suggestions: AIAssignmentSuggestion[]
|
||
error?: string
|
||
tokensUsed?: number
|
||
fallbackUsed?: boolean
|
||
}
|
||
```
|
||
|
||
#### Fallback Algorithm
|
||
|
||
When AI is unavailable or fails:
|
||
|
||
```typescript
|
||
export function generateFallbackAssignments(
|
||
jurors: JurorForAssignment[],
|
||
projects: ProjectForAssignment[],
|
||
constraints: AssignmentConstraints
|
||
): AIAssignmentResult {
|
||
// Algorithm:
|
||
// 1. Sort projects by current assignment count (fewest first)
|
||
// 2. For each project, score all available jurors by:
|
||
// - Expertise match (50% weight): tag overlap
|
||
// - Load balancing (30% weight): current vs. max assignments
|
||
// - Under-min bonus (20% weight): jurors below minimum target
|
||
// 3. Assign top-scoring jurors
|
||
|
||
const scoredJurors = jurors.map(juror => ({
|
||
juror,
|
||
score: calculateExpertiseScore(juror.expertiseTags, project.tags) * 0.5
|
||
+ calculateLoadScore(currentLoad, maxLoad) * 0.3
|
||
+ calculateUnderMinBonus(currentLoad, minTarget) * 0.2
|
||
}))
|
||
|
||
// Assign top N jurors per project
|
||
const topN = scoredJurors.slice(0, neededReviews)
|
||
|
||
return {
|
||
success: true,
|
||
suggestions: topN.map(s => ({ jurorId: s.juror.id, projectId, ... })),
|
||
fallbackUsed: true
|
||
}
|
||
}
|
||
```
|
||
|
||
**Expertise Score**:
|
||
```typescript
|
||
function calculateExpertiseScore(jurorTags: string[], projectTags: string[]): number {
|
||
const jurorTagsLower = new Set(jurorTags.map(t => t.toLowerCase()))
|
||
const matchingTags = projectTags.filter(t => jurorTagsLower.has(t.toLowerCase()))
|
||
|
||
const matchRatio = matchingTags.length / projectTags.length
|
||
const hasExpertise = matchingTags.length > 0 ? 0.2 : 0
|
||
|
||
return Math.min(1, matchRatio * 0.8 + hasExpertise)
|
||
}
|
||
```
|
||
|
||
#### Integration with Evaluation Round
|
||
|
||
```typescript
|
||
// Round 3/5: EVALUATION
|
||
const evaluationRound = await prisma.round.findFirst({
|
||
where: { competitionId, roundType: 'EVALUATION', sortOrder: 3 }
|
||
})
|
||
|
||
const juryGroup = await prisma.juryGroup.findUnique({
|
||
where: { id: evaluationRound.juryGroupId },
|
||
include: {
|
||
members: { include: { user: true } }
|
||
}
|
||
})
|
||
|
||
const projects = await prisma.project.findMany({
|
||
where: {
|
||
projectRoundStates: {
|
||
some: {
|
||
roundId: previousRoundId,
|
||
state: 'PASSED'
|
||
}
|
||
}
|
||
}
|
||
})
|
||
|
||
const constraints = {
|
||
requiredReviewsPerProject: 3,
|
||
maxAssignmentsPerJuror: 20,
|
||
jurorLimits: {
|
||
"user-123": 15, // Personal override
|
||
"user-456": 25 // Personal override
|
||
},
|
||
existingAssignments: await prisma.assignment.findMany({
|
||
where: { roundId: evaluationRound.id },
|
||
select: { userId: true, projectId: true }
|
||
})
|
||
}
|
||
|
||
const result = await generateAIAssignments(
|
||
juryGroup.members.map(m => m.user),
|
||
projects,
|
||
constraints,
|
||
adminUserId,
|
||
evaluationRound.id,
|
||
onProgress
|
||
)
|
||
|
||
// Admin reviews suggestions in UI, then applies
|
||
for (const suggestion of result.suggestions) {
|
||
await prisma.assignment.create({
|
||
data: {
|
||
userId: suggestion.jurorId,
|
||
projectId: suggestion.projectId,
|
||
roundId: evaluationRound.id,
|
||
juryGroupId: juryGroup.id,
|
||
method: 'AI',
|
||
aiConfidenceScore: suggestion.confidenceScore,
|
||
expertiseMatchScore: suggestion.expertiseMatchScore,
|
||
aiReasoning: suggestion.reasoning
|
||
}
|
||
})
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### 3. AI Evaluation Summary Service (`ai-evaluation-summary.ts`)
|
||
|
||
**Purpose**: Synthesize multiple juror evaluations for a project into a cohesive summary with strengths, weaknesses, themes, and consensus analysis.
|
||
|
||
#### Input Data
|
||
|
||
```typescript
|
||
interface EvaluationForSummary {
|
||
id: string
|
||
criterionScoresJson: Record<string, number> | null
|
||
globalScore: number | null
|
||
binaryDecision: boolean | null
|
||
feedbackText: string | null
|
||
assignment: {
|
||
user: {
|
||
id: string
|
||
name: string | null
|
||
email: string
|
||
}
|
||
}
|
||
}
|
||
|
||
interface CriterionDef {
|
||
id: string
|
||
label: string
|
||
}
|
||
```
|
||
|
||
#### Anonymization
|
||
|
||
```typescript
|
||
// Strips juror identities, keeps only scores and sanitized feedback
|
||
export function anonymizeEvaluations(
|
||
evaluations: EvaluationForSummary[]
|
||
): AnonymizedEvaluation[] {
|
||
return evaluations.map(ev => ({
|
||
criterionScores: ev.criterionScoresJson,
|
||
globalScore: ev.globalScore,
|
||
binaryDecision: ev.binaryDecision,
|
||
feedbackText: ev.feedbackText ? sanitizeText(ev.feedbackText) : null
|
||
}))
|
||
}
|
||
|
||
// Before:
|
||
[
|
||
{
|
||
id: "eval-123",
|
||
globalScore: 8,
|
||
feedbackText: "Strong proposal from john@example.com. Contact me at +1-555-1234.",
|
||
assignment: { user: { name: "John Doe", email: "john@example.com" } }
|
||
}
|
||
]
|
||
|
||
// After:
|
||
[
|
||
{
|
||
globalScore: 8,
|
||
feedbackText: "Strong proposal. Contact me at [phone removed].",
|
||
// No user info
|
||
}
|
||
]
|
||
```
|
||
|
||
#### Prompt Structure
|
||
|
||
```typescript
|
||
const prompt = `You are analyzing jury evaluations for a project competition.
|
||
|
||
PROJECT: "${sanitizedTitle}"
|
||
|
||
EVALUATION CRITERIA: ${criteriaLabels.join(', ')}
|
||
|
||
EVALUATIONS (${anonymizedEvaluations.length} total):
|
||
${JSON.stringify(anonymizedEvaluations, null, 2)}
|
||
|
||
Analyze these evaluations and return a JSON object with this exact structure:
|
||
{
|
||
"overallAssessment": "A 2-3 sentence summary of how the project was evaluated overall",
|
||
"strengths": ["strength 1", "strength 2", ...],
|
||
"weaknesses": ["weakness 1", "weakness 2", ...],
|
||
"themes": [
|
||
{ "theme": "theme name", "sentiment": "positive" | "negative" | "mixed", "frequency": <number of evaluators mentioning this> }
|
||
],
|
||
"recommendation": "A brief recommendation based on the evaluation consensus"
|
||
}
|
||
|
||
Guidelines:
|
||
- Base your analysis only on the provided evaluation data
|
||
- Identify common themes across evaluator feedback
|
||
- Note areas of agreement and disagreement
|
||
- Keep the assessment objective and balanced
|
||
- Do not include any personal identifiers`
|
||
```
|
||
|
||
#### Scoring Patterns (Server-Side)
|
||
|
||
In addition to AI analysis, the service computes statistical patterns:
|
||
|
||
```typescript
|
||
interface ScoringPatterns {
|
||
averageGlobalScore: number | null
|
||
consensus: number // 0-1 (1 = full agreement)
|
||
criterionAverages: Record<string, number>
|
||
evaluatorCount: number
|
||
}
|
||
|
||
export function computeScoringPatterns(
|
||
evaluations: EvaluationForSummary[],
|
||
criteriaLabels: CriterionDef[]
|
||
): ScoringPatterns {
|
||
const globalScores = evaluations.map(e => e.globalScore).filter(s => s !== null)
|
||
|
||
const averageGlobalScore = globalScores.length > 0
|
||
? globalScores.reduce((a, b) => a + b, 0) / globalScores.length
|
||
: null
|
||
|
||
// Consensus: 1 - normalized standard deviation
|
||
let consensus = 1
|
||
if (globalScores.length > 1 && averageGlobalScore !== null) {
|
||
const variance = globalScores.reduce((sum, score) =>
|
||
sum + Math.pow(score - averageGlobalScore, 2), 0
|
||
) / globalScores.length
|
||
const stdDev = Math.sqrt(variance)
|
||
consensus = Math.max(0, 1 - stdDev / 4.5) // Normalize by max possible std dev
|
||
}
|
||
|
||
// Criterion averages
|
||
const criterionAverages: Record<string, number> = {}
|
||
for (const criterion of criteriaLabels) {
|
||
const scores = evaluations
|
||
.map(e => e.criterionScoresJson?.[criterion.id])
|
||
.filter((s): s is number => s !== undefined)
|
||
if (scores.length > 0) {
|
||
criterionAverages[criterion.label] = scores.reduce((a, b) => a + b, 0) / scores.length
|
||
}
|
||
}
|
||
|
||
return {
|
||
averageGlobalScore,
|
||
consensus: Math.round(consensus * 100) / 100,
|
||
criterionAverages,
|
||
evaluatorCount: evaluations.length
|
||
}
|
||
}
|
||
```
|
||
|
||
#### Output
|
||
|
||
```typescript
|
||
interface AIResponsePayload {
|
||
overallAssessment: string
|
||
strengths: string[]
|
||
weaknesses: string[]
|
||
themes: Array<{
|
||
theme: string
|
||
sentiment: 'positive' | 'negative' | 'mixed'
|
||
frequency: number
|
||
}>
|
||
recommendation: string
|
||
}
|
||
|
||
interface EvaluationSummaryResult {
|
||
id: string
|
||
projectId: string
|
||
stageId: string
|
||
summaryJson: AIResponsePayload & { scoringPatterns: ScoringPatterns }
|
||
generatedAt: Date
|
||
model: string
|
||
tokensUsed: number
|
||
}
|
||
```
|
||
|
||
**Example Output**:
|
||
```json
|
||
{
|
||
"overallAssessment": "The project received consistently high scores (avg: 8.2/10) with strong consensus (0.92). Evaluators praised the technical approach and team expertise, but raised concerns about scalability and budget.",
|
||
"strengths": [
|
||
"Novel AI-powered coral monitoring approach",
|
||
"Strong technical team with relevant expertise",
|
||
"Clear commercial viability and market need"
|
||
],
|
||
"weaknesses": [
|
||
"Budget appears optimistic for proposed timeline",
|
||
"Limited discussion of regulatory compliance",
|
||
"Scalability challenges not fully addressed"
|
||
],
|
||
"themes": [
|
||
{ "theme": "Technical Innovation", "sentiment": "positive", "frequency": 5 },
|
||
{ "theme": "Team Expertise", "sentiment": "positive", "frequency": 4 },
|
||
{ "theme": "Budget Concerns", "sentiment": "negative", "frequency": 3 },
|
||
{ "theme": "Scalability", "sentiment": "mixed", "frequency": 2 }
|
||
],
|
||
"recommendation": "Strong candidate for advancement. Address budget and scalability concerns during mentoring phase.",
|
||
"scoringPatterns": {
|
||
"averageGlobalScore": 8.2,
|
||
"consensus": 0.92,
|
||
"criterionAverages": {
|
||
"Innovation": 8.8,
|
||
"Feasibility": 7.6,
|
||
"Impact": 8.4,
|
||
"Team": 9.0
|
||
},
|
||
"evaluatorCount": 5
|
||
}
|
||
}
|
||
```
|
||
|
||
#### Integration with Evaluation Round
|
||
|
||
```typescript
|
||
// Auto-trigger when all evaluations for a project are submitted
|
||
const evaluationsCount = await prisma.evaluation.count({
|
||
where: {
|
||
status: 'SUBMITTED',
|
||
assignment: { projectId, roundId }
|
||
}
|
||
})
|
||
|
||
const requiredReviews = evaluationRound.configJson.requiredReviewsPerProject
|
||
|
||
if (evaluationsCount >= requiredReviews) {
|
||
const summary = await generateSummary({
|
||
projectId,
|
||
stageId: roundId,
|
||
userId: 'system',
|
||
prisma
|
||
})
|
||
|
||
// Notify admin
|
||
await prisma.inAppNotification.create({
|
||
data: {
|
||
userId: adminId,
|
||
title: 'Evaluation Summary Generated',
|
||
message: `AI summary ready for project "${projectTitle}"`,
|
||
type: 'AI_SUMMARY_READY',
|
||
relatedEntityType: 'EvaluationSummary',
|
||
relatedEntityId: summary.id
|
||
}
|
||
})
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### 4. AI Tagging Service (`ai-tagging.ts`)
|
||
|
||
**Purpose**: Automatically assign expertise tags to projects based on content analysis.
|
||
|
||
#### Input Data
|
||
|
||
```typescript
|
||
// Full project data with files and team info
|
||
const project = await prisma.project.findUnique({
|
||
where: { id: projectId },
|
||
include: {
|
||
projectTags: true,
|
||
files: { select: { fileType: true } },
|
||
_count: { select: { teamMembers: true, files: true } }
|
||
}
|
||
})
|
||
|
||
// Available tags from system
|
||
interface AvailableTag {
|
||
id: string
|
||
name: string
|
||
category: string | null
|
||
description: string | null
|
||
}
|
||
|
||
const availableTags = await prisma.expertiseTag.findMany({
|
||
where: { isActive: true },
|
||
orderBy: [{ category: 'asc' }, { sortOrder: 'asc' }]
|
||
})
|
||
```
|
||
|
||
#### Anonymization
|
||
|
||
```typescript
|
||
const projectWithRelations = toProjectWithRelations(project)
|
||
const { anonymized, mappings } = anonymizeProjectsForAI([projectWithRelations], 'FILTERING')
|
||
|
||
// Anonymized project:
|
||
{
|
||
project_id: "P1",
|
||
title: "SaveTheSea",
|
||
description: "An AI-powered platform for coral reef monitoring...",
|
||
category: "STARTUP",
|
||
ocean_issue: "CORAL_REEFS",
|
||
tags: ["AI", "Monitoring"],
|
||
founded_year: 2024,
|
||
team_size: 3,
|
||
file_count: 5,
|
||
file_types: ["PDF", "DOCX"]
|
||
}
|
||
```
|
||
|
||
#### Prompt Structure
|
||
|
||
```typescript
|
||
const tagList = availableTags.map(t => ({
|
||
name: t.name,
|
||
category: t.category,
|
||
description: t.description
|
||
}))
|
||
|
||
const prompt = `PROJECT:
|
||
${JSON.stringify(anonymizedProject, null, 2)}
|
||
|
||
AVAILABLE TAGS:
|
||
${JSON.stringify(tagList, null, 2)}
|
||
|
||
Suggest relevant tags for this project.`
|
||
|
||
const systemPrompt = `You are an expert at categorizing ocean conservation and sustainability projects.
|
||
|
||
Analyze the project and suggest the most relevant expertise tags from the provided list.
|
||
Consider the project's focus areas, technology, methodology, and domain.
|
||
|
||
Return JSON with this format:
|
||
{
|
||
"suggestions": [
|
||
{
|
||
"tag_name": "exact tag name from list",
|
||
"confidence": 0.0-1.0,
|
||
"reasoning": "brief explanation why this tag fits"
|
||
}
|
||
]
|
||
}
|
||
|
||
Rules:
|
||
- Only suggest tags from the provided list (exact names)
|
||
- Order by relevance (most relevant first)
|
||
- Confidence should reflect how well the tag matches
|
||
- Maximum 7 suggestions per project
|
||
- Be conservative - only suggest tags that truly apply`
|
||
```
|
||
|
||
#### Output
|
||
|
||
```typescript
|
||
interface TagSuggestion {
|
||
tagId: string
|
||
tagName: string
|
||
confidence: number
|
||
reasoning: string
|
||
}
|
||
|
||
interface TaggingResult {
|
||
projectId: string
|
||
suggestions: TagSuggestion[]
|
||
applied: TagSuggestion[]
|
||
tokensUsed: number
|
||
}
|
||
```
|
||
|
||
**Example Response**:
|
||
```json
|
||
{
|
||
"suggestions": [
|
||
{
|
||
"tag_name": "Marine Biology",
|
||
"confidence": 0.92,
|
||
"reasoning": "Project focuses on coral reef health monitoring"
|
||
},
|
||
{
|
||
"tag_name": "Artificial Intelligence",
|
||
"confidence": 0.88,
|
||
"reasoning": "Uses AI for image analysis and pattern detection"
|
||
},
|
||
{
|
||
"tag_name": "Data Science",
|
||
"confidence": 0.75,
|
||
"reasoning": "Significant data collection and analysis component"
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
#### Application Logic
|
||
|
||
```typescript
|
||
const CONFIDENCE_THRESHOLD = 0.5
|
||
const MAX_TAGS = 5
|
||
|
||
// 1. Get AI suggestions
|
||
const { suggestions, tokensUsed } = await getAISuggestions(anonymized[0], availableTags, userId)
|
||
|
||
// 2. Filter by confidence threshold
|
||
const validSuggestions = suggestions.filter(s => s.confidence >= CONFIDENCE_THRESHOLD)
|
||
|
||
// 3. Get existing tags to avoid duplicates
|
||
const existingTagIds = new Set(project.projectTags.map(pt => pt.tagId))
|
||
|
||
// 4. Calculate remaining slots
|
||
const currentTagCount = project.projectTags.length
|
||
const remainingSlots = Math.max(0, MAX_TAGS - currentTagCount)
|
||
|
||
// 5. Filter and limit
|
||
const newSuggestions = validSuggestions
|
||
.filter(s => !existingTagIds.has(s.tagId))
|
||
.slice(0, remainingSlots)
|
||
|
||
// 6. Apply new tags (additive only, never removes existing)
|
||
for (const suggestion of newSuggestions) {
|
||
await prisma.projectTag.create({
|
||
data: {
|
||
projectId,
|
||
tagId: suggestion.tagId,
|
||
confidence: suggestion.confidence,
|
||
source: 'AI'
|
||
}
|
||
})
|
||
}
|
||
```
|
||
|
||
#### Integration Points
|
||
|
||
**On Project Submission** (Round 1: INTAKE):
|
||
```typescript
|
||
// Auto-tag when project is submitted
|
||
if (settings.ai_tagging_enabled && settings.ai_tagging_on_submit) {
|
||
await tagProject(projectId, userId)
|
||
}
|
||
```
|
||
|
||
**Manual Tagging** (Admin Dashboard):
|
||
```typescript
|
||
// Admin reviews suggestions before applying
|
||
const suggestions = await getTagSuggestions(projectId, userId)
|
||
|
||
// UI shows suggestions with confidence scores
|
||
// Admin clicks "Apply All" or selectively adds tags
|
||
for (const suggestion of selectedSuggestions) {
|
||
await addProjectTag(projectId, suggestion.tagId)
|
||
}
|
||
```
|
||
|
||
**Batch Tagging** (Round Management):
|
||
```typescript
|
||
// Tag all projects in a round
|
||
const projects = await prisma.project.findMany({
|
||
where: { competitionId }
|
||
})
|
||
|
||
for (const project of projects) {
|
||
try {
|
||
await tagProject(project.id, adminUserId)
|
||
} catch (error) {
|
||
console.error(`Failed to tag project ${project.id}:`, error)
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### 5. AI Award Eligibility Service (`ai-award-eligibility.ts`)
|
||
|
||
**Purpose**: Determine which projects are eligible for special awards using both deterministic field matching and AI interpretation of plain-language criteria.
|
||
|
||
#### Input Data
|
||
|
||
```typescript
|
||
interface ProjectForEligibility {
|
||
id: string
|
||
title: string
|
||
description?: string | null
|
||
competitionCategory?: CompetitionCategory | null
|
||
country?: string | null
|
||
geographicZone?: string | null
|
||
tags: string[]
|
||
oceanIssue?: OceanIssue | null
|
||
institution?: string | null
|
||
foundedAt?: Date | null
|
||
wantsMentorship?: boolean
|
||
submissionSource?: SubmissionSource
|
||
submittedAt?: Date | null
|
||
_count?: {
|
||
teamMembers?: number
|
||
files?: number
|
||
}
|
||
files?: Array<{ fileType: string | null }>
|
||
}
|
||
|
||
interface SpecialAward {
|
||
id: string
|
||
name: string
|
||
description?: string | null
|
||
criteriaText?: string | null // Plain-language criteria for AI
|
||
eligibilityMode: 'STAY_IN_MAIN' | 'SEPARATE_POOL'
|
||
useAiEligibility: boolean
|
||
}
|
||
```
|
||
|
||
#### Deterministic Auto-Tag Rules
|
||
|
||
```typescript
|
||
type AutoTagRule = {
|
||
field: 'competitionCategory' | 'country' | 'geographicZone' | 'tags' | 'oceanIssue'
|
||
operator: 'equals' | 'contains' | 'in'
|
||
value: string | string[]
|
||
}
|
||
|
||
// Example: "Innovation Award" for AI/ML projects
|
||
const innovationRules: AutoTagRule[] = [
|
||
{
|
||
field: 'tags',
|
||
operator: 'contains',
|
||
value: 'Artificial Intelligence'
|
||
}
|
||
]
|
||
|
||
// Example: "Regional Impact Award" for Mediterranean projects
|
||
const regionalRules: AutoTagRule[] = [
|
||
{
|
||
field: 'geographicZone',
|
||
operator: 'equals',
|
||
value: 'MEDITERRANEAN'
|
||
}
|
||
]
|
||
|
||
// Apply rules
|
||
export function applyAutoTagRules(
|
||
rules: AutoTagRule[],
|
||
projects: ProjectForEligibility[]
|
||
): Map<string, boolean> {
|
||
const results = new Map<string, boolean>()
|
||
|
||
for (const project of projects) {
|
||
const matches = rules.every(rule => {
|
||
const fieldValue = getFieldValue(project, rule.field)
|
||
// ... operator logic ...
|
||
})
|
||
results.set(project.id, matches)
|
||
}
|
||
|
||
return results
|
||
}
|
||
```
|
||
|
||
#### AI Criteria Interpretation
|
||
|
||
```typescript
|
||
// Award with plain-language criteria
|
||
const award = {
|
||
name: "Youth Innovation Award",
|
||
criteriaText: `
|
||
Projects eligible for this award should:
|
||
- Be founded within the last 3 years
|
||
- Have a team with average age under 30
|
||
- Demonstrate novel technological approach
|
||
- Show strong potential for scalability
|
||
`,
|
||
useAiEligibility: true
|
||
}
|
||
|
||
// Anonymize and batch process
|
||
const { anonymized, mappings } = anonymizeProjectsForAI(projects, 'ELIGIBILITY')
|
||
|
||
const prompt = `CRITERIA: ${criteriaText}
|
||
PROJECTS: ${JSON.stringify(anonymized)}
|
||
Evaluate eligibility for each project.`
|
||
|
||
const systemPrompt = `Award eligibility evaluator. Evaluate projects against criteria, return JSON.
|
||
Format: {"evaluations": [{project_id, eligible: bool, confidence: 0-1, reasoning: str}]}
|
||
Be objective. Base evaluation only on provided data. No personal identifiers in reasoning.`
|
||
```
|
||
|
||
#### Output
|
||
|
||
```typescript
|
||
interface EligibilityResult {
|
||
projectId: string
|
||
eligible: boolean
|
||
confidence: number
|
||
reasoning: string
|
||
method: 'AUTO' | 'AI'
|
||
}
|
||
|
||
// Example AI response:
|
||
{
|
||
"evaluations": [
|
||
{
|
||
"project_id": "P1",
|
||
"eligible": true,
|
||
"confidence": 0.85,
|
||
"reasoning": "Founded in 2024 (within 3 years). Novel AI approach for coral monitoring. High scalability potential."
|
||
},
|
||
{
|
||
"project_id": "P2",
|
||
"eligible": false,
|
||
"confidence": 0.90,
|
||
"reasoning": "Founded in 2015 (over 3 years ago). Does not meet youth criteria."
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
#### Integration with Special Awards
|
||
|
||
```typescript
|
||
// Run eligibility check for an award
|
||
const award = await prisma.specialAward.findUnique({
|
||
where: { id: awardId },
|
||
include: { competition: true }
|
||
})
|
||
|
||
const projects = await prisma.project.findMany({
|
||
where: { competitionId: award.competitionId },
|
||
include: { files: true, _count: { select: { teamMembers: true } } }
|
||
})
|
||
|
||
let eligibilityResults: EligibilityResult[] = []
|
||
|
||
if (award.useAiEligibility && award.criteriaText) {
|
||
// AI interpretation
|
||
eligibilityResults = await aiInterpretCriteria(
|
||
award.criteriaText,
|
||
projects,
|
||
userId,
|
||
awardId
|
||
)
|
||
} else if (award.autoTagRules) {
|
||
// Deterministic rules
|
||
const matches = applyAutoTagRules(award.autoTagRules, projects)
|
||
eligibilityResults = Array.from(matches).map(([projectId, eligible]) => ({
|
||
projectId,
|
||
eligible,
|
||
confidence: 1.0,
|
||
reasoning: 'Matches auto-tag rules',
|
||
method: 'AUTO'
|
||
}))
|
||
}
|
||
|
||
// Store results
|
||
for (const result of eligibilityResults) {
|
||
await prisma.awardEligibility.upsert({
|
||
where: {
|
||
specialAwardId_projectId: { specialAwardId: awardId, projectId: result.projectId }
|
||
},
|
||
create: {
|
||
specialAwardId: awardId,
|
||
projectId: result.projectId,
|
||
isEligible: result.eligible,
|
||
confidence: result.confidence,
|
||
reasoning: result.reasoning,
|
||
method: result.method
|
||
},
|
||
update: {
|
||
isEligible: result.eligible,
|
||
confidence: result.confidence,
|
||
reasoning: result.reasoning,
|
||
method: result.method
|
||
}
|
||
})
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### 6. Anonymization Service (`anonymization.ts`)
|
||
|
||
**Purpose**: Strip PII from all data before sending to OpenAI. Ensures GDPR compliance and privacy protection.
|
||
|
||
#### PII Patterns Detected
|
||
|
||
```typescript
|
||
const PII_PATTERNS = {
|
||
email: /[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/g,
|
||
phone: /(\+?\d{1,3}[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}/g,
|
||
url: /https?:\/\/[^\s]+/g,
|
||
ssn: /\d{3}-\d{2}-\d{4}/g,
|
||
ipv4: /\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/g
|
||
}
|
||
|
||
export function sanitizeText(text: string): string {
|
||
let sanitized = text
|
||
sanitized = sanitized.replace(PII_PATTERNS.email, '[email removed]')
|
||
sanitized = sanitized.replace(PII_PATTERNS.phone, '[phone removed]')
|
||
sanitized = sanitized.replace(PII_PATTERNS.url, '[url removed]')
|
||
sanitized = sanitized.replace(PII_PATTERNS.ssn, '[id removed]')
|
||
return sanitized
|
||
}
|
||
```
|
||
|
||
#### Description Truncation
|
||
|
||
```typescript
|
||
export const DESCRIPTION_LIMITS = {
|
||
ASSIGNMENT: 300, // Short context for matching
|
||
FILTERING: 500, // Medium context for screening
|
||
ELIGIBILITY: 400, // Medium context for criteria
|
||
MENTOR: 350 // Short context for mentor matching
|
||
}
|
||
|
||
export function truncateAndSanitize(
|
||
text: string | null | undefined,
|
||
maxLength: number
|
||
): string {
|
||
if (!text) return ''
|
||
const sanitized = sanitizeText(text)
|
||
if (sanitized.length <= maxLength) return sanitized
|
||
return sanitized.slice(0, maxLength - 3) + '...'
|
||
}
|
||
```
|
||
|
||
#### ID Replacement Strategy
|
||
|
||
**Before Anonymization**:
|
||
```typescript
|
||
// Real data with identifiable IDs
|
||
{
|
||
id: "proj-abc123xyz",
|
||
userId: "user-456def789",
|
||
title: "SaveTheSea by Marine Research Institute",
|
||
teamName: "Dr. John Doe's Lab",
|
||
description: "Contact us at john@example.com or +1-555-1234 for more info. Visit https://savethesea.org"
|
||
}
|
||
```
|
||
|
||
**After Anonymization**:
|
||
```typescript
|
||
// Anonymous IDs and sanitized text
|
||
{
|
||
project_id: "P1",
|
||
title: "SaveTheSea",
|
||
description: "Contact us at [email removed] or [phone removed] for more info. Visit [url removed]",
|
||
team_size: 3,
|
||
founded_year: 2024,
|
||
tags: ["Marine Biology", "AI"]
|
||
}
|
||
```
|
||
|
||
**Mapping for De-anonymization**:
|
||
```typescript
|
||
const mappings: ProjectAIMapping[] = [
|
||
{ anonymousId: "P1", realId: "proj-abc123xyz" },
|
||
{ anonymousId: "P2", realId: "proj-def456uvw" }
|
||
]
|
||
```
|
||
|
||
#### GDPR Validation
|
||
|
||
```typescript
|
||
export interface PIIValidationResult {
|
||
valid: boolean
|
||
violations: string[]
|
||
}
|
||
|
||
export function validateNoPersonalData(
|
||
data: Record<string, unknown>
|
||
): PIIValidationResult {
|
||
const violations: string[] = []
|
||
const textContent = JSON.stringify(data)
|
||
|
||
// Check PII patterns
|
||
for (const [type, pattern] of Object.entries(PII_PATTERNS)) {
|
||
pattern.lastIndex = 0
|
||
if (pattern.test(textContent)) {
|
||
violations.push(`Potential ${type} detected in data`)
|
||
}
|
||
}
|
||
|
||
// Check sensitive field names
|
||
const sensitiveFields = ['email', 'phone', 'password', 'ssn', 'creditCard', 'bankAccount']
|
||
const keys = Object.keys(data).map(k => k.toLowerCase())
|
||
for (const field of sensitiveFields) {
|
||
if (keys.includes(field)) {
|
||
violations.push(`Sensitive field "${field}" present in data`)
|
||
}
|
||
}
|
||
|
||
return {
|
||
valid: violations.length === 0,
|
||
violations
|
||
}
|
||
}
|
||
|
||
// Enforce before EVERY AI call
|
||
export function enforceGDPRCompliance(data: unknown[]): void {
|
||
for (let i = 0; i < data.length; i++) {
|
||
const { valid, violations } = validateNoPersonalData(item)
|
||
if (!valid) {
|
||
throw new Error(`GDPR compliance check failed: ${violations.join(', ')}`)
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
#### Anonymization Workflow
|
||
|
||
```mermaid
|
||
graph TD
|
||
A[Raw Project Data] --> B[Convert to ProjectWithRelations]
|
||
B --> C[Truncate Descriptions]
|
||
C --> D[Sanitize Text Fields]
|
||
D --> E[Replace IDs with Anonymous IDs]
|
||
E --> F[Create Mappings]
|
||
F --> G[Validate No PII]
|
||
G --> H{Valid?}
|
||
H -->|Yes| I[Send to AI]
|
||
H -->|No| J[Throw GDPR Error]
|
||
I --> K[AI Response]
|
||
K --> L[De-anonymize Results]
|
||
L --> M[Return Real IDs]
|
||
```
|
||
|
||
---
|
||
|
||
## New AI Services for Redesign
|
||
|
||
### 1. AI Mentoring Insights (Round 6: MENTORING)
|
||
|
||
**Purpose**: Summarize mentor-team interactions, flag inactive workspaces, suggest intervention points.
|
||
|
||
#### Input Data
|
||
|
||
```typescript
|
||
interface MentorWorkspaceData {
|
||
projectId: string
|
||
mentorId: string
|
||
workspaceOpenAt: Date
|
||
files: MentorFile[]
|
||
messages: MentorMessage[]
|
||
fileComments: MentorFileComment[]
|
||
lastActivityAt: Date
|
||
}
|
||
```
|
||
|
||
#### Anonymization
|
||
|
||
```typescript
|
||
// Strip mentor/team names, keep activity patterns
|
||
{
|
||
workspace_id: "W1",
|
||
days_active: 14,
|
||
file_count: 3,
|
||
message_count: 12,
|
||
comment_count: 5,
|
||
last_activity_days_ago: 2,
|
||
file_types: ["PDF", "DOCX"],
|
||
message_frequency: "daily",
|
||
engagement_level: "high"
|
||
}
|
||
```
|
||
|
||
#### Prompt
|
||
|
||
```typescript
|
||
const prompt = `Analyze mentor workspace activity and provide insights.
|
||
|
||
WORKSPACE DATA: ${JSON.stringify(anonymizedWorkspace)}
|
||
|
||
Return JSON:
|
||
{
|
||
"engagement_level": "high" | "medium" | "low" | "inactive",
|
||
"key_insights": ["insight 1", "insight 2"],
|
||
"red_flags": ["flag 1", "flag 2"],
|
||
"recommendations": ["recommendation 1", "recommendation 2"],
|
||
"intervention_needed": boolean
|
||
}`
|
||
```
|
||
|
||
#### Output
|
||
|
||
```typescript
|
||
interface MentoringInsight {
|
||
workspaceId: string
|
||
engagementLevel: 'high' | 'medium' | 'low' | 'inactive'
|
||
keyInsights: string[]
|
||
redFlags: string[]
|
||
recommendations: string[]
|
||
interventionNeeded: boolean
|
||
}
|
||
```
|
||
|
||
**Example**:
|
||
```json
|
||
{
|
||
"engagement_level": "low",
|
||
"key_insights": [
|
||
"No activity in last 7 days",
|
||
"Only 1 file uploaded (below average for this stage)",
|
||
"Message frequency dropped from daily to weekly"
|
||
],
|
||
"red_flags": [
|
||
"Mentor has not responded to last 2 team messages",
|
||
"No progress on business plan deliverable (due in 5 days)"
|
||
],
|
||
"recommendations": [
|
||
"Admin should check in with mentor",
|
||
"Send reminder notification to team about upcoming deadline",
|
||
"Consider pairing with secondary mentor for support"
|
||
],
|
||
"intervention_needed": true
|
||
}
|
||
```
|
||
|
||
#### Integration
|
||
|
||
```typescript
|
||
// Nightly cron job: analyze all active mentoring workspaces
|
||
const mentoringRound = await prisma.round.findFirst({
|
||
where: { competitionId, roundType: 'MENTORING', status: 'ROUND_ACTIVE' }
|
||
})
|
||
|
||
const assignments = await prisma.mentorAssignment.findMany({
|
||
where: { workspaceEnabled: true },
|
||
include: { files: true, messages: true }
|
||
})
|
||
|
||
for (const assignment of assignments) {
|
||
const insight = await generateMentoringInsight(assignment, userId)
|
||
|
||
if (insight.interventionNeeded) {
|
||
await prisma.inAppNotification.create({
|
||
data: {
|
||
userId: adminId,
|
||
title: 'Mentoring Intervention Needed',
|
||
message: `Workspace for project "${projectTitle}" requires attention`,
|
||
type: 'MENTORING_ALERT',
|
||
actionUrl: `/admin/mentoring/${assignment.id}`
|
||
}
|
||
})
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### 2. AI Duplicate Detection Enhancement (Round 2: FILTERING)
|
||
|
||
**Purpose**: Detect duplicate or near-duplicate submissions using embeddings and semantic similarity.
|
||
|
||
#### Current Approach (Simple)
|
||
|
||
```typescript
|
||
// Existing: basic title/email matching
|
||
const duplicates = await prisma.project.findMany({
|
||
where: {
|
||
OR: [
|
||
{ title: { equals: project.title, mode: 'insensitive' } },
|
||
{ teamMembers: { some: { email: applicantEmail } } }
|
||
]
|
||
}
|
||
})
|
||
```
|
||
|
||
#### Enhanced Approach (AI-Powered)
|
||
|
||
```typescript
|
||
// Generate embeddings for all projects
|
||
const embeddings = new Map<string, number[]>()
|
||
|
||
for (const project of projects) {
|
||
const embedding = await openai.embeddings.create({
|
||
model: 'text-embedding-3-small',
|
||
input: `${project.title}\n\n${project.description}`
|
||
})
|
||
embeddings.set(project.id, embedding.data[0].embedding)
|
||
}
|
||
|
||
// Compute cosine similarity matrix
|
||
function cosineSimilarity(a: number[], b: number[]): number {
|
||
const dot = a.reduce((sum, val, i) => sum + val * b[i], 0)
|
||
const magA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0))
|
||
const magB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0))
|
||
return dot / (magA * magB)
|
||
}
|
||
|
||
// Find duplicates
|
||
const SIMILARITY_THRESHOLD = 0.85
|
||
|
||
for (const [id1, emb1] of embeddings) {
|
||
for (const [id2, emb2] of embeddings) {
|
||
if (id1 >= id2) continue // Skip self and already compared
|
||
|
||
const similarity = cosineSimilarity(emb1, emb2)
|
||
if (similarity >= SIMILARITY_THRESHOLD) {
|
||
await prisma.duplicateDetection.create({
|
||
data: {
|
||
projectId1: id1,
|
||
projectId2: id2,
|
||
similarity,
|
||
method: 'AI_EMBEDDING',
|
||
status: 'FLAGGED'
|
||
}
|
||
})
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
#### Output
|
||
|
||
```typescript
|
||
interface DuplicateMatch {
|
||
projectId1: string
|
||
projectId2: string
|
||
similarity: number
|
||
method: 'EXACT_TITLE' | 'EMAIL_MATCH' | 'AI_EMBEDDING'
|
||
status: 'FLAGGED' | 'CONFIRMED_DUPLICATE' | 'FALSE_POSITIVE'
|
||
}
|
||
```
|
||
|
||
**Admin Review UI**:
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ Duplicate Detection: 3 Potential Matches │
|
||
├─────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ [!] 92% Match - AI Coral Monitoring (P1) vs Ocean AI (P2) │
|
||
│ Method: AI Embedding │
|
||
│ ┌─ P1: SaveTheSea - AI-powered coral reef monitoring... │
|
||
│ └─ P2: Ocean AI - Machine learning platform for coral... │
|
||
│ │
|
||
│ [ Mark as Duplicate ] [ False Positive ] [ View Details ] │
|
||
│ │
|
||
├─────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ [!] 88% Match - Marine Plastic (P3) vs PlasticClean (P4) │
|
||
│ Method: AI Embedding + Email Match │
|
||
│ ┌─ P3: Marine Plastic Cleanup - Robotic system... │
|
||
│ └─ P4: PlasticClean - Automated plastic removal... │
|
||
│ │
|
||
│ [ Mark as Duplicate ] [ False Positive ] [ View Details ] │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
### 3. AI Confirmation Helper (Round 8: CONFIRMATION)
|
||
|
||
**Purpose**: Generate plain-language explanations for why final rankings are what they are. Helps jury understand and validate the winner proposal.
|
||
|
||
#### Input Data
|
||
|
||
```typescript
|
||
interface WinnerProposalData {
|
||
rankedProjectIds: string[]
|
||
sourceRoundId: string
|
||
selectionBasis: {
|
||
method: 'LIVE_VOTE' | 'EVALUATION_SCORES' | 'ADMIN_DECISION'
|
||
scores: Record<string, number>
|
||
aiRecommendation?: string
|
||
}
|
||
}
|
||
|
||
// Fetch all context
|
||
const projects = await prisma.project.findMany({
|
||
where: { id: { in: proposal.rankedProjectIds } },
|
||
include: {
|
||
evaluations: { where: { roundId: sourceRoundId } },
|
||
liveVotes: { where: { roundId: sourceRoundId } }
|
||
}
|
||
})
|
||
```
|
||
|
||
#### Anonymization
|
||
|
||
```typescript
|
||
// Strip project names, keep relative rankings and scores
|
||
{
|
||
ranking: [
|
||
{ position: 1, anonymous_id: "P1", score: 8.9, vote_count: 12 },
|
||
{ position: 2, anonymous_id: "P2", score: 8.5, vote_count: 10 },
|
||
{ position: 3, anonymous_id: "P3", score: 8.2, vote_count: 11 }
|
||
],
|
||
selection_method: "LIVE_VOTE",
|
||
score_gaps: [0.4, 0.3],
|
||
consensus_level: 0.88
|
||
}
|
||
```
|
||
|
||
#### Prompt
|
||
|
||
```typescript
|
||
const prompt = `Explain the final ranking in plain language for jury review.
|
||
|
||
RANKING DATA: ${JSON.stringify(anonymizedRanking)}
|
||
|
||
EVALUATION HISTORY: ${JSON.stringify(anonymizedEvaluations)}
|
||
|
||
Return JSON:
|
||
{
|
||
"summary": "2-3 sentence plain-language summary of why this ranking makes sense",
|
||
"position_explanations": [
|
||
{ "position": 1, "why_this_rank": "explanation", "strengths": ["s1", "s2"] },
|
||
...
|
||
],
|
||
"close_calls": [
|
||
{ "positions": [2, 3], "gap": 0.3, "reasoning": "why this was close" }
|
||
],
|
||
"confidence": "high" | "medium" | "low",
|
||
"red_flags": ["flag 1", ...]
|
||
}`
|
||
```
|
||
|
||
#### Output
|
||
|
||
```typescript
|
||
interface ConfirmationExplanation {
|
||
summary: string
|
||
positionExplanations: Array<{
|
||
position: number
|
||
projectId: string
|
||
whyThisRank: string
|
||
strengths: string[]
|
||
}>
|
||
closeCalls: Array<{
|
||
positions: number[]
|
||
gap: number
|
||
reasoning: string
|
||
}>
|
||
confidence: 'high' | 'medium' | 'low'
|
||
redFlags: string[]
|
||
}
|
||
```
|
||
|
||
**Example**:
|
||
```json
|
||
{
|
||
"summary": "The ranking reflects clear consensus from both live voting (12 votes) and prior evaluations (avg 8.9). Top position is well-separated with 0.4 point gap. Second and third places are close (0.3 gap), but consistent across jury members.",
|
||
"position_explanations": [
|
||
{
|
||
"position": 1,
|
||
"why_this_rank": "Highest average score (8.9) across all evaluation rounds. Won live vote with 12/15 jury members. Consistent top performer.",
|
||
"strengths": ["Technical innovation", "Strong team", "Commercial viability", "Scalability"]
|
||
},
|
||
{
|
||
"position": 2,
|
||
"why_this_rank": "Strong runner-up (8.5 avg). Close to third place but slight edge in feasibility scores.",
|
||
"strengths": ["Proven track record", "Regulatory compliance", "Budget realism"]
|
||
},
|
||
{
|
||
"position": 3,
|
||
"why_this_rank": "Solid finalist (8.2 avg). Slightly lower technical feasibility scores compared to #2.",
|
||
"strengths": ["Impact potential", "Novel approach", "Jury enthusiasm"]
|
||
}
|
||
],
|
||
"close_calls": [
|
||
{
|
||
"positions": [2, 3],
|
||
"gap": 0.3,
|
||
"reasoning": "Very close race. Position 2 had slight edge in 'Feasibility' criterion (7.8 vs 7.2), while Position 3 scored higher in 'Impact' (8.9 vs 8.4). Live vote split 10-11. Admin may want to review deliberation notes."
|
||
}
|
||
],
|
||
"confidence": "high",
|
||
"red_flags": []
|
||
}
|
||
```
|
||
|
||
**Integration**:
|
||
```typescript
|
||
// When admin creates WinnerProposal
|
||
const proposal = await prisma.winnerProposal.create({
|
||
data: {
|
||
competitionId,
|
||
category: 'STARTUP',
|
||
rankedProjectIds: ['proj-1', 'proj-2', 'proj-3'],
|
||
sourceRoundId: liveFinalsRoundId,
|
||
selectionBasis: { method: 'LIVE_VOTE', scores: {...} },
|
||
proposedById: adminId
|
||
}
|
||
})
|
||
|
||
// Generate AI explanation
|
||
const explanation = await generateConfirmationExplanation(proposal.id, userId)
|
||
|
||
// Attach to proposal
|
||
await prisma.winnerProposal.update({
|
||
where: { id: proposal.id },
|
||
data: {
|
||
selectionBasis: {
|
||
...proposal.selectionBasis,
|
||
aiExplanation: explanation
|
||
}
|
||
}
|
||
})
|
||
|
||
// Show to jury during confirmation
|
||
```
|
||
|
||
**Confirmation UI with AI Explanation**:
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ Confirm Final Rankings: STARTUP Category │
|
||
├─────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ AI Analysis: High Confidence │
|
||
│ "The ranking reflects clear consensus from both live voting │
|
||
│ (12 votes) and prior evaluations (avg 8.9). Top position is │
|
||
│ well-separated with 0.4 point gap..." │
|
||
│ │
|
||
│ ┌───────────────────────────────────────────────────────────────┐ │
|
||
│ │ 1st Place: SaveTheSea (Score: 8.9, 12 votes) │ │
|
||
│ │ Why This Rank: Highest average score across all rounds... │ │
|
||
│ │ Strengths: Technical innovation, Strong team, Commercial... │ │
|
||
│ └───────────────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
│ ┌───────────────────────────────────────────────────────────────┐ │
|
||
│ │ 2nd Place: Ocean AI (Score: 8.5, 10 votes) │ │
|
||
│ │ Why This Rank: Strong runner-up. Close to third place... │ │
|
||
│ │ Strengths: Proven track record, Regulatory compliance... │ │
|
||
│ └───────────────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
│ [!] Close Call: 2nd and 3rd place (0.3 gap) │
|
||
│ Position 2 had slight edge in Feasibility (7.8 vs 7.2)... │
|
||
│ │
|
||
│ [ ] I confirm this ranking (John Doe, Jury Lead) │
|
||
│ [ ] I confirm this ranking (Jane Smith, Jury Member) │
|
||
│ [ ] I confirm this ranking (Bob Wilson, Jury Member) │
|
||
│ │
|
||
│ Comments: ___________________________________________________ │
|
||
│ │
|
||
│ [ Approve Ranking ] [ Request Changes ] │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Anonymization Pipeline (Deep Dive)
|
||
|
||
### Data Flow
|
||
|
||
```mermaid
|
||
graph LR
|
||
A[Raw Data] --> B[Type Conversion]
|
||
B --> C[Description Truncation]
|
||
C --> D[Text Sanitization]
|
||
D --> E[ID Replacement]
|
||
E --> F[Field Filtering]
|
||
F --> G[Validation]
|
||
G --> H{GDPR Pass?}
|
||
H -->|Yes| I[To AI]
|
||
H -->|No| J[Error + Log]
|
||
I --> K[AI Response]
|
||
K --> L[De-anonymization]
|
||
L --> M[Real Results]
|
||
```
|
||
|
||
### What Gets Stripped
|
||
|
||
| Data Type | Before | After |
|
||
|-----------|--------|-------|
|
||
| **User IDs** | `user-abc123xyz` | `juror_001` |
|
||
| **Project IDs** | `proj-def456uvw` | `P1` |
|
||
| **Names** | `Dr. John Doe` | (removed entirely) |
|
||
| **Emails** | `john@example.com` | `[email removed]` |
|
||
| **Phone Numbers** | `+1-555-1234` | `[phone removed]` |
|
||
| **URLs** | `https://example.com` | `[url removed]` |
|
||
| **Team Names** | `Marine Research Institute` | `Team 1` |
|
||
| **Descriptions** | Full 2000-char text | Truncated to 300-500 chars |
|
||
| **Dates** | `2024-02-15T14:30:00Z` | `2024` (year only) or `2024-02-15` (date only) |
|
||
|
||
### Replacement Strategy
|
||
|
||
**Sequential Anonymous IDs**:
|
||
- Projects: `P1`, `P2`, `P3`, ...
|
||
- Jurors: `juror_001`, `juror_002`, `juror_003`, ...
|
||
- Workspaces: `W1`, `W2`, `W3`, ...
|
||
|
||
**Rationale**: Sequential IDs are easier for AI to reference and for humans to debug. Random UUIDs would be harder to correlate in logs.
|
||
|
||
**Generic Team Names**:
|
||
- `Team 1`, `Team 2`, `Team 3`, ...
|
||
|
||
**Rationale**: Preserves the concept of "team" without revealing institutional affiliation.
|
||
|
||
### De-anonymization Mapping
|
||
|
||
```typescript
|
||
// Stored in memory during AI call (never persisted to DB)
|
||
const mappings: ProjectAIMapping[] = [
|
||
{ anonymousId: "P1", realId: "proj-abc123xyz" },
|
||
{ anonymousId: "P2", realId: "proj-def456uvw" },
|
||
{ anonymousId: "P3", realId: "proj-ghi789rst" }
|
||
]
|
||
|
||
// After AI returns results for "P1", map back to real ID
|
||
const realProjectId = mappings.find(m => m.anonymousId === "P1")?.realId
|
||
```
|
||
|
||
### Anonymization for Different Data Types
|
||
|
||
#### Projects
|
||
|
||
```typescript
|
||
// INPUT (ProjectWithRelations)
|
||
{
|
||
id: "proj-abc123",
|
||
title: "SaveTheSea - Marine Conservation by Dr. John Doe",
|
||
description: "Contact john@example.com or call +1-555-1234. Visit https://savethesea.org for more info.",
|
||
teamName: "Marine Research Institute",
|
||
competitionCategory: "STARTUP",
|
||
country: "France",
|
||
geographicZone: "MEDITERRANEAN",
|
||
tags: ["Marine Biology", "AI", "Conservation"],
|
||
foundedAt: new Date("2024-01-15T00:00:00Z"),
|
||
institution: "University of Monaco",
|
||
wantsMentorship: true,
|
||
submissionSource: "WEB_FORM",
|
||
submittedAt: new Date("2024-02-15T14:30:00Z"),
|
||
_count: { teamMembers: 3, files: 5 },
|
||
files: [
|
||
{ fileType: "PDF" },
|
||
{ fileType: "PDF" },
|
||
{ fileType: "DOCX" }
|
||
]
|
||
}
|
||
|
||
// OUTPUT (AnonymizedProjectForAI)
|
||
{
|
||
project_id: "P1",
|
||
title: "SaveTheSea", // PII stripped
|
||
description: "Contact [email removed] or call [phone removed]. Visit [url removed] for more info.", // Sanitized + truncated
|
||
category: "STARTUP",
|
||
ocean_issue: null,
|
||
country: "France",
|
||
region: "MEDITERRANEAN",
|
||
institution: "University of Monaco", // Kept (non-PII institutional data)
|
||
tags: ["Marine Biology", "AI", "Conservation"],
|
||
founded_year: 2024, // Date reduced to year only
|
||
team_size: 3,
|
||
has_description: true,
|
||
file_count: 5,
|
||
file_types: ["PDF", "DOCX"],
|
||
wants_mentorship: true,
|
||
submission_source: "WEB_FORM",
|
||
submitted_date: "2024-02-15" // Date only, no time
|
||
}
|
||
```
|
||
|
||
#### Jurors
|
||
|
||
```typescript
|
||
// INPUT (JurorForAssignment)
|
||
{
|
||
id: "user-456def",
|
||
name: "Dr. Jane Smith",
|
||
email: "jane.smith@university.edu",
|
||
expertiseTags: ["Marine Biology", "Oceanography"],
|
||
maxAssignments: 20,
|
||
_count: { assignments: 5 }
|
||
}
|
||
|
||
// OUTPUT (AnonymizedJuror)
|
||
{
|
||
anonymousId: "juror_001",
|
||
expertiseTags: ["Marine Biology", "Oceanography"],
|
||
currentAssignmentCount: 5,
|
||
maxAssignments: 20
|
||
// Name and email completely removed
|
||
}
|
||
```
|
||
|
||
#### Evaluations
|
||
|
||
```typescript
|
||
// INPUT (EvaluationForSummary)
|
||
{
|
||
id: "eval-789ghi",
|
||
criterionScoresJson: { "Innovation": 9, "Feasibility": 7, "Impact": 8 },
|
||
globalScore: 8,
|
||
binaryDecision: true,
|
||
feedbackText: "Excellent proposal from Dr. John Doe (john@example.com). Strong technical approach but budget seems optimistic. Call me at +1-555-9876 to discuss.",
|
||
assignment: {
|
||
user: {
|
||
id: "user-123abc",
|
||
name: "Dr. Alice Johnson",
|
||
email: "alice@university.edu"
|
||
}
|
||
}
|
||
}
|
||
|
||
// OUTPUT (AnonymizedEvaluation)
|
||
{
|
||
criterionScores: { "Innovation": 9, "Feasibility": 7, "Impact": 8 },
|
||
globalScore: 8,
|
||
binaryDecision: true,
|
||
feedbackText: "Excellent proposal. Strong technical approach but budget seems optimistic. Call me at [phone removed] to discuss."
|
||
// User info completely removed
|
||
// PII sanitized from feedback
|
||
}
|
||
```
|
||
|
||
### GDPR Validation Workflow
|
||
|
||
```typescript
|
||
// 1. Anonymize data
|
||
const { anonymized, mappings } = anonymizeProjectsForAI(projects, 'FILTERING')
|
||
|
||
// 2. Validate BEFORE sending to AI
|
||
if (!validateAnonymizedProjects(anonymized)) {
|
||
console.error('[AI Service] GDPR validation failed')
|
||
|
||
// Log violation
|
||
await prisma.auditLog.create({
|
||
data: {
|
||
action: 'AI_GDPR_VIOLATION',
|
||
entityType: 'AIService',
|
||
details: { service: 'filtering', error: 'PII detected in anonymized data' }
|
||
}
|
||
})
|
||
|
||
throw new Error('GDPR compliance check failed: PII detected in anonymized data')
|
||
}
|
||
|
||
// 3. Only if validation passes, send to AI
|
||
const response = await openai.chat.completions.create(params)
|
||
|
||
// 4. De-anonymize results
|
||
const realResults = mappings.map(mapping => {
|
||
const aiResult = response.find(r => r.project_id === mapping.anonymousId)
|
||
return {
|
||
projectId: mapping.realId,
|
||
...aiResult
|
||
}
|
||
})
|
||
```
|
||
|
||
---
|
||
|
||
## Prompt Engineering
|
||
|
||
### System Prompt Templates
|
||
|
||
#### Filtering Prompt
|
||
|
||
```typescript
|
||
const AI_SCREENING_SYSTEM_PROMPT = `Project screening assistant. Evaluate against criteria, return JSON.
|
||
Format: {"projects": [{project_id, meets_criteria: bool, confidence: 0-1, reasoning: str, quality_score: 1-10, spam_risk: bool}]}
|
||
Be objective. Base evaluation only on provided data. No personal identifiers in reasoning.`
|
||
```
|
||
|
||
**Design Principles**:
|
||
- **Compact**: 3 sentences instead of 10 paragraphs (saves tokens)
|
||
- **JSON Schema**: Explicit output format prevents parsing errors
|
||
- **Objectivity**: Reminds AI to avoid bias
|
||
- **Privacy**: Explicitly forbids personal identifiers in output
|
||
|
||
#### Assignment Prompt
|
||
|
||
```typescript
|
||
const ASSIGNMENT_SYSTEM_PROMPT = `Match jurors to projects by expertise. Return JSON assignments.
|
||
Each: {juror_id, project_id, confidence_score: 0-1, expertise_match_score: 0-1, reasoning: str (1-2 sentences)}
|
||
Distribute workload fairly. Avoid assigning jurors at capacity.`
|
||
```
|
||
|
||
**Design Principles**:
|
||
- **Constraint Awareness**: Reminds AI about workload balancing
|
||
- **Brevity Requirement**: "1-2 sentences" for reasoning (saves tokens)
|
||
- **Dual Scores**: Separates confidence (how sure AI is) from expertise match (how good the fit is)
|
||
|
||
#### Evaluation Summary Prompt
|
||
|
||
```typescript
|
||
const EVALUATION_SUMMARY_PROMPT = `You are analyzing jury evaluations for a project competition.
|
||
|
||
PROJECT: "${sanitizedTitle}"
|
||
|
||
EVALUATION CRITERIA: ${criteriaLabels.join(', ')}
|
||
|
||
EVALUATIONS (${count} total):
|
||
${JSON.stringify(anonymizedEvaluations, null, 2)}
|
||
|
||
Analyze these evaluations and return a JSON object with this exact structure:
|
||
{
|
||
"overallAssessment": "A 2-3 sentence summary of how the project was evaluated overall",
|
||
"strengths": ["strength 1", "strength 2", ...],
|
||
"weaknesses": ["weakness 1", "weakness 2", ...],
|
||
"themes": [
|
||
{ "theme": "theme name", "sentiment": "positive" | "negative" | "mixed", "frequency": <number of evaluators mentioning this> }
|
||
],
|
||
"recommendation": "A brief recommendation based on the evaluation consensus"
|
||
}
|
||
|
||
Guidelines:
|
||
- Base your analysis only on the provided evaluation data
|
||
- Identify common themes across evaluator feedback
|
||
- Note areas of agreement and disagreement
|
||
- Keep the assessment objective and balanced
|
||
- Do not include any personal identifiers`
|
||
```
|
||
|
||
**Design Principles**:
|
||
- **Structured Output**: JSON schema enforces consistency
|
||
- **Sentiment Analysis**: Themes include positive/negative/mixed classification
|
||
- **Consensus Detection**: Frequency count shows how many evaluators mentioned each theme
|
||
- **Explicit Guidelines**: 5 bullet points ensure quality output
|
||
|
||
### Dynamic Context Injection
|
||
|
||
```typescript
|
||
// Build prompts dynamically based on available data
|
||
function buildFilteringPrompt(
|
||
criteriaText: string,
|
||
projects: AnonymizedProjectForAI[],
|
||
context?: {
|
||
previousRound?: { passRate: number },
|
||
categoryDistribution?: { STARTUP: number, BUSINESS_CONCEPT: number }
|
||
}
|
||
): string {
|
||
let prompt = `CRITERIA: ${criteriaText}\nPROJECTS: ${JSON.stringify(projects)}\n`
|
||
|
||
if (context?.previousRound) {
|
||
prompt += `\nPREVIOUS ROUND: Pass rate was ${context.previousRound.passRate}%. Apply similar rigor.\n`
|
||
}
|
||
|
||
if (context?.categoryDistribution) {
|
||
prompt += `\nCATEGORY BALANCE: Aim for roughly ${context.categoryDistribution.STARTUP}% startups, ${context.categoryDistribution.BUSINESS_CONCEPT}% concepts.\n`
|
||
}
|
||
|
||
prompt += `\nEvaluate and return JSON.`
|
||
return prompt
|
||
}
|
||
```
|
||
|
||
### Output Format Enforcement
|
||
|
||
```typescript
|
||
// Use JSON mode for all AI calls
|
||
const params = buildCompletionParams(model, {
|
||
messages: [
|
||
{ role: 'system', content: systemPrompt },
|
||
{ role: 'user', content: userPrompt }
|
||
],
|
||
jsonMode: true, // Enforces JSON output
|
||
temperature: 0.3,
|
||
maxTokens: 4000
|
||
})
|
||
|
||
// Validate response structure
|
||
const parsed = JSON.parse(content) as ExpectedStructure
|
||
|
||
if (!isValidStructure(parsed)) {
|
||
throw createParseError('AI returned invalid JSON structure')
|
||
}
|
||
```
|
||
|
||
### Token Limit Management
|
||
|
||
```typescript
|
||
// Description limits by context
|
||
export const DESCRIPTION_LIMITS = {
|
||
ASSIGNMENT: 300, // Short - only need general topic
|
||
FILTERING: 500, // Medium - need enough detail for quality assessment
|
||
ELIGIBILITY: 400, // Medium - need criteria matching
|
||
MENTOR: 350 // Short - expertise matching
|
||
}
|
||
|
||
// Calculate estimated token count before API call
|
||
function estimateTokens(text: string): number {
|
||
// Rough estimate: 1 token ≈ 4 characters
|
||
return Math.ceil(text.length / 4)
|
||
}
|
||
|
||
// Warn if approaching limits
|
||
const promptTokens = estimateTokens(userPrompt)
|
||
if (promptTokens > 3000) {
|
||
console.warn(`[AI Service] Large prompt detected: ${promptTokens} tokens`)
|
||
}
|
||
|
||
// Set maxTokens based on expected output
|
||
const maxTokens = {
|
||
FILTERING: 4000, // Batch of 20 projects
|
||
ASSIGNMENT: 4000, // Batch of 15 projects
|
||
SUMMARY: 2000, // Single summary
|
||
TAGGING: 1000 // Tag suggestions
|
||
}[serviceType]
|
||
```
|
||
|
||
### Retry and Fallback Strategies
|
||
|
||
```typescript
|
||
async function callAIWithRetry<T>(
|
||
apiCall: () => Promise<T>,
|
||
fallback: () => T,
|
||
maxRetries = 3
|
||
): Promise<T> {
|
||
let lastError: Error
|
||
|
||
for (let attempt = 1; attempt <= maxRetries; attempt++) {
|
||
try {
|
||
return await apiCall()
|
||
} catch (error) {
|
||
lastError = error
|
||
|
||
const classified = classifyAIError(error)
|
||
|
||
// Don't retry on validation errors or rate limits
|
||
if (classified.type === 'VALIDATION' || classified.type === 'RATE_LIMIT') {
|
||
break
|
||
}
|
||
|
||
// Wait before retry (exponential backoff)
|
||
if (attempt < maxRetries) {
|
||
await new Promise(resolve => setTimeout(resolve, 1000 * Math.pow(2, attempt)))
|
||
}
|
||
}
|
||
}
|
||
|
||
// Fall back to algorithm
|
||
console.warn(`[AI Service] Failed after ${maxRetries} attempts, using fallback`, lastError)
|
||
return fallback()
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## OpenAI Integration
|
||
|
||
### Model Selection
|
||
|
||
```typescript
|
||
// lib/openai.ts
|
||
export const AI_MODELS = {
|
||
STRONG: 'gpt-4', // Complex reasoning (filtering, assignment)
|
||
QUICK: 'gpt-4-turbo', // Fast analysis (summaries, tagging)
|
||
CHEAP: 'gpt-3.5-turbo', // Simple tasks (deprecated for this platform)
|
||
EMBEDDING: 'text-embedding-3-small' // For duplicate detection
|
||
}
|
||
|
||
// Get configured model from settings
|
||
export async function getConfiguredModel(preference?: string): Promise<string> {
|
||
const settings = await prisma.systemSettings.findUnique({
|
||
where: { key: 'ai_model' }
|
||
})
|
||
|
||
const model = preference || settings?.value || AI_MODELS.STRONG
|
||
|
||
// Validate model exists
|
||
if (!Object.values(AI_MODELS).includes(model)) {
|
||
console.warn(`[OpenAI] Unknown model "${model}", falling back to ${AI_MODELS.STRONG}`)
|
||
return AI_MODELS.STRONG
|
||
}
|
||
|
||
return model
|
||
}
|
||
```
|
||
|
||
**Model Usage by Service**:
|
||
| Service | Model | Rationale |
|
||
|---------|-------|-----------|
|
||
| Filtering | `gpt-4` | Needs strong reasoning for nuanced criteria interpretation |
|
||
| Assignment | `gpt-4` | Complex constraint balancing (expertise + load + COI) |
|
||
| Summary | `gpt-4-turbo` | Fast synthesis, less reasoning required |
|
||
| Tagging | `gpt-4-turbo` | Simple categorization task |
|
||
| Eligibility | `gpt-4` | Plain-language criteria need strong comprehension |
|
||
| Mentoring Insights | `gpt-4-turbo` | Pattern detection, less critical reasoning |
|
||
|
||
### Rate Limiting
|
||
|
||
```typescript
|
||
// Rate limiter middleware
|
||
class AIRateLimiter {
|
||
private requestQueue: Array<() => Promise<unknown>> = []
|
||
private isProcessing = false
|
||
private requestsPerMinute = 60 // GPT-4 tier 1 limit
|
||
|
||
async add<T>(fn: () => Promise<T>): Promise<T> {
|
||
return new Promise((resolve, reject) => {
|
||
this.requestQueue.push(async () => {
|
||
try {
|
||
const result = await fn()
|
||
resolve(result)
|
||
} catch (error) {
|
||
reject(error)
|
||
}
|
||
})
|
||
|
||
if (!this.isProcessing) {
|
||
this.process()
|
||
}
|
||
})
|
||
}
|
||
|
||
private async process() {
|
||
this.isProcessing = true
|
||
|
||
while (this.requestQueue.length > 0) {
|
||
const fn = this.requestQueue.shift()!
|
||
await fn()
|
||
|
||
// Wait to respect rate limit
|
||
await new Promise(resolve => setTimeout(resolve, 60000 / this.requestsPerMinute))
|
||
}
|
||
|
||
this.isProcessing = false
|
||
}
|
||
}
|
||
|
||
const rateLimiter = new AIRateLimiter()
|
||
|
||
// Use in services
|
||
const result = await rateLimiter.add(() => openai.chat.completions.create(params))
|
||
```
|
||
|
||
### Cost Tracking
|
||
|
||
```typescript
|
||
// Token pricing (as of 2024)
|
||
const TOKEN_PRICING = {
|
||
'gpt-4': { input: 0.03, output: 0.06 }, // per 1K tokens
|
||
'gpt-4-turbo': { input: 0.01, output: 0.03 },
|
||
'text-embedding-3-small': { input: 0.0001, output: 0 }
|
||
}
|
||
|
||
// Calculate cost
|
||
function calculateCost(
|
||
model: string,
|
||
promptTokens: number,
|
||
completionTokens: number
|
||
): number {
|
||
const pricing = TOKEN_PRICING[model] || TOKEN_PRICING['gpt-4']
|
||
|
||
const inputCost = (promptTokens / 1000) * pricing.input
|
||
const outputCost = (completionTokens / 1000) * pricing.output
|
||
|
||
return inputCost + outputCost
|
||
}
|
||
|
||
// Log usage with cost
|
||
await logAIUsage({
|
||
userId,
|
||
action: 'FILTERING',
|
||
model,
|
||
promptTokens,
|
||
completionTokens,
|
||
totalTokens,
|
||
estimatedCost: calculateCost(model, promptTokens, completionTokens),
|
||
status: 'SUCCESS'
|
||
})
|
||
|
||
// Admin dashboard: show cumulative costs
|
||
const totalCost = await prisma.aIUsageLog.aggregate({
|
||
where: { createdAt: { gte: startOfMonth } },
|
||
_sum: { estimatedCost: true }
|
||
})
|
||
```
|
||
|
||
### Error Handling
|
||
|
||
```typescript
|
||
// ai-errors.ts
|
||
export type AIErrorType =
|
||
| 'RATE_LIMIT'
|
||
| 'AUTHENTICATION'
|
||
| 'NETWORK'
|
||
| 'TIMEOUT'
|
||
| 'VALIDATION'
|
||
| 'PARSE_ERROR'
|
||
| 'MODEL_NOT_FOUND'
|
||
| 'UNKNOWN'
|
||
|
||
export interface ClassifiedAIError {
|
||
type: AIErrorType
|
||
message: string
|
||
retryable: boolean
|
||
userMessage: string
|
||
}
|
||
|
||
export function classifyAIError(error: unknown): ClassifiedAIError {
|
||
const errorMsg = error instanceof Error ? error.message : String(error)
|
||
|
||
// Rate limit
|
||
if (errorMsg.includes('rate_limit') || errorMsg.includes('429')) {
|
||
return {
|
||
type: 'RATE_LIMIT',
|
||
message: errorMsg,
|
||
retryable: true,
|
||
userMessage: 'AI service is currently at capacity. Please try again in a few minutes.'
|
||
}
|
||
}
|
||
|
||
// Authentication
|
||
if (errorMsg.includes('authentication') || errorMsg.includes('401') || errorMsg.includes('api key')) {
|
||
return {
|
||
type: 'AUTHENTICATION',
|
||
message: errorMsg,
|
||
retryable: false,
|
||
userMessage: 'OpenAI API key is invalid or missing. Please check Settings > AI Configuration.'
|
||
}
|
||
}
|
||
|
||
// Model not found
|
||
if (errorMsg.includes('model') && errorMsg.includes('does not exist')) {
|
||
return {
|
||
type: 'MODEL_NOT_FOUND',
|
||
message: errorMsg,
|
||
retryable: false,
|
||
userMessage: 'The configured AI model does not exist. Please check Settings > AI Configuration.'
|
||
}
|
||
}
|
||
|
||
// Network
|
||
if (errorMsg.includes('ECONNREFUSED') || errorMsg.includes('ETIMEDOUT') || errorMsg.includes('network')) {
|
||
return {
|
||
type: 'NETWORK',
|
||
message: errorMsg,
|
||
retryable: true,
|
||
userMessage: 'Unable to reach OpenAI servers. Please check your internet connection.'
|
||
}
|
||
}
|
||
|
||
// Timeout
|
||
if (errorMsg.includes('timeout')) {
|
||
return {
|
||
type: 'TIMEOUT',
|
||
message: errorMsg,
|
||
retryable: true,
|
||
userMessage: 'AI request timed out. Please try again or reduce batch size.'
|
||
}
|
||
}
|
||
|
||
// Validation
|
||
if (errorMsg.includes('validation') || errorMsg.includes('invalid')) {
|
||
return {
|
||
type: 'VALIDATION',
|
||
message: errorMsg,
|
||
retryable: false,
|
||
userMessage: 'Invalid request to AI service. Please contact support.'
|
||
}
|
||
}
|
||
|
||
// Parse error
|
||
if (error instanceof SyntaxError || errorMsg.includes('parse') || errorMsg.includes('JSON')) {
|
||
return {
|
||
type: 'PARSE_ERROR',
|
||
message: errorMsg,
|
||
retryable: false,
|
||
userMessage: 'AI response could not be understood. This has been logged for review.'
|
||
}
|
||
}
|
||
|
||
// Unknown
|
||
return {
|
||
type: 'UNKNOWN',
|
||
message: errorMsg,
|
||
retryable: true,
|
||
userMessage: 'An unexpected error occurred with the AI service. Please try again.'
|
||
}
|
||
}
|
||
|
||
export function logAIError(service: string, operation: string, error: ClassifiedAIError): void {
|
||
console.error(`[AI Error] ${service}.${operation}:`, {
|
||
type: error.type,
|
||
message: error.message,
|
||
retryable: error.retryable
|
||
})
|
||
|
||
// Log to database for admin review
|
||
prisma.auditLog.create({
|
||
data: {
|
||
action: 'AI_ERROR',
|
||
entityType: 'AIService',
|
||
details: {
|
||
service,
|
||
operation,
|
||
errorType: error.type,
|
||
message: error.message
|
||
}
|
||
}
|
||
}).catch(console.error)
|
||
}
|
||
```
|
||
|
||
### Streaming vs Non-Streaming
|
||
|
||
**Current Implementation**: Non-streaming (wait for full response)
|
||
|
||
```typescript
|
||
const response = await openai.chat.completions.create({
|
||
model: 'gpt-4',
|
||
messages: [...],
|
||
stream: false // Wait for complete response
|
||
})
|
||
|
||
const content = response.choices[0]?.message?.content
|
||
```
|
||
|
||
**Future Enhancement**: Streaming for long summaries
|
||
|
||
```typescript
|
||
// For evaluation summaries, stream to UI as tokens arrive
|
||
const stream = await openai.chat.completions.create({
|
||
model: 'gpt-4',
|
||
messages: [...],
|
||
stream: true
|
||
})
|
||
|
||
for await (const chunk of stream) {
|
||
const delta = chunk.choices[0]?.delta?.content || ''
|
||
|
||
// Send to frontend via SSE
|
||
res.write(`data: ${JSON.stringify({ delta })}\n\n`)
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Privacy & Security
|
||
|
||
### Data Minimization
|
||
|
||
**Principle**: Send only the minimum data required for the AI task.
|
||
|
||
| Service | Data Sent | Data NOT Sent |
|
||
|---------|-----------|---------------|
|
||
| **Filtering** | Project title, description (truncated), category, tags, file types, team size | Team names, emails, phone numbers, full descriptions, user IDs |
|
||
| **Assignment** | Juror expertise tags, current assignment count, max assignments, project tags | Juror names, emails, user IDs, juror institutions |
|
||
| **Summary** | Evaluation scores, sanitized feedback text | Juror names, emails, user IDs |
|
||
| **Tagging** | Project title, description (truncated), category, tags | Team names, emails, phone numbers, user IDs |
|
||
| **Eligibility** | Project category, tags, country, founded year | Team names, emails, phone numbers, full descriptions |
|
||
|
||
### PII Stripping Workflow
|
||
|
||
```mermaid
|
||
graph TD
|
||
A[Raw Data] --> B{Contains PII?}
|
||
B -->|Yes| C[Sanitize Text]
|
||
B -->|No| D[Validate]
|
||
C --> E[Replace IDs]
|
||
E --> F[Truncate Descriptions]
|
||
F --> D
|
||
D --> G{GDPR Pass?}
|
||
G -->|No| H[Reject + Log]
|
||
G -->|Yes| I[Send to AI]
|
||
I --> J[AI Response]
|
||
J --> K[De-anonymize]
|
||
K --> L[Return Real Results]
|
||
```
|
||
|
||
### Audit Logging
|
||
|
||
```typescript
|
||
// All AI requests logged with full context
|
||
await logAIUsage({
|
||
userId: 'admin-123',
|
||
action: 'FILTERING',
|
||
entityType: 'Round',
|
||
entityId: 'round-456',
|
||
model: 'gpt-4',
|
||
promptTokens: 1234,
|
||
completionTokens: 567,
|
||
totalTokens: 1801,
|
||
batchSize: 20,
|
||
itemsProcessed: 20,
|
||
status: 'SUCCESS',
|
||
estimatedCost: 0.0891,
|
||
metadata: {
|
||
criteriaText: 'Reject spam projects...',
|
||
passRate: 0.75
|
||
}
|
||
})
|
||
|
||
// Failed requests also logged
|
||
await logAIUsage({
|
||
userId: 'admin-123',
|
||
action: 'ASSIGNMENT',
|
||
model: 'gpt-4',
|
||
status: 'ERROR',
|
||
errorMessage: 'Rate limit exceeded',
|
||
metadata: {
|
||
jurorCount: 15,
|
||
projectCount: 45
|
||
}
|
||
})
|
||
```
|
||
|
||
### Data Retention Policy
|
||
|
||
```typescript
|
||
// AI usage logs retained for 90 days, then archived
|
||
export async function archiveOldAILogs(): Promise<void> {
|
||
const cutoffDate = new Date()
|
||
cutoffDate.setDate(cutoffDate.getDate() - 90)
|
||
|
||
const oldLogs = await prisma.aIUsageLog.findMany({
|
||
where: { createdAt: { lt: cutoffDate } }
|
||
})
|
||
|
||
// Move to archive storage (S3, cold storage, etc.)
|
||
await archiveService.store('ai-logs', oldLogs)
|
||
|
||
// Delete from hot database
|
||
await prisma.aIUsageLog.deleteMany({
|
||
where: { createdAt: { lt: cutoffDate } }
|
||
})
|
||
}
|
||
|
||
// Run nightly via cron
|
||
```
|
||
|
||
### Opt-Out Capabilities
|
||
|
||
```typescript
|
||
// Per-service opt-out in settings
|
||
interface AISettings {
|
||
ai_enabled: boolean // Master toggle
|
||
ai_filtering_enabled: boolean
|
||
ai_assignment_enabled: boolean
|
||
ai_summary_enabled: boolean
|
||
ai_tagging_enabled: boolean
|
||
ai_eligibility_enabled: boolean
|
||
}
|
||
|
||
// Check before each AI call
|
||
export async function isAIServiceEnabled(service: AIServiceType): Promise<boolean> {
|
||
const masterEnabled = await getSystemSetting('ai_enabled')
|
||
if (!masterEnabled) return false
|
||
|
||
const serviceEnabled = await getSystemSetting(`ai_${service}_enabled`)
|
||
return serviceEnabled !== 'false'
|
||
}
|
||
|
||
// Graceful fallback when disabled
|
||
if (!await isAIServiceEnabled('assignment')) {
|
||
return generateFallbackAssignments(jurors, projects, constraints)
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Admin AI Controls
|
||
|
||
### AI Configuration Dashboard
|
||
|
||
```
|
||
┌────────────────────────────────────────────────────────────────────┐
|
||
│ Settings > AI Configuration │
|
||
├────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ General Settings │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ OpenAI API Key: [sk-proj-••••••••••••••••••••••••••] [ Test ] │
|
||
│ ✓ Connected (last verified: 2024-02-15 14:30) │
|
||
│ │
|
||
│ Default Model: [ gpt-4 ▼ ] │
|
||
│ Estimated cost: $0.03 per 1K tokens (input) │
|
||
│ │
|
||
│ [ ] Enable AI Services ← Master toggle │
|
||
│ │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ Service-Specific Settings │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ [✓] AI Filtering │
|
||
│ Batch Size: [ 20 ] projects per API call │
|
||
│ Parallel Batches: [ 2 ] concurrent requests │
|
||
│ Model Override: [ (use default) ▼ ] │
|
||
│ │
|
||
│ [✓] AI Assignment │
|
||
│ Batch Size: [ 15 ] projects per API call │
|
||
│ Model Override: [ (use default) ▼ ] │
|
||
│ │
|
||
│ [✓] AI Evaluation Summaries │
|
||
│ Auto-generate: [✓] When all evaluations submitted │
|
||
│ Model Override: [ gpt-4-turbo ▼ ] (faster for summaries) │
|
||
│ │
|
||
│ [✓] AI Tagging │
|
||
│ Confidence Threshold: [ 0.5 ] (0.0 - 1.0) │
|
||
│ Max Tags per Project: [ 5 ] │
|
||
│ Auto-tag on Submit: [✓] Apply tags when project submitted │
|
||
│ │
|
||
│ [✓] AI Award Eligibility │
|
||
│ Model Override: [ (use default) ▼ ] │
|
||
│ │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ Usage & Cost Tracking │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ This Month: 1,234,567 tokens | Est. Cost: $42.18 │
|
||
│ Last Month: 987,654 tokens | Est. Cost: $31.45 │
|
||
│ │
|
||
│ [ View Detailed Usage Log ] │
|
||
│ │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ [ Save Changes ] [ Reset to Defaults ] │
|
||
│ │
|
||
└────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### Rubric Editor for Filtering
|
||
|
||
```
|
||
┌────────────────────────────────────────────────────────────────────┐
|
||
│ Round 2: FILTERING > AI Screening Rules │
|
||
├────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ Rule: Spam & Quality Check │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ Criteria (Plain Language): │
|
||
│ ┌──────────────────────────────────────────────────────────────┐ │
|
||
│ │ Projects should demonstrate clear ocean conservation value. │ │
|
||
│ │ │ │
|
||
│ │ Reject projects that: │ │
|
||
│ │ - Are spam, test submissions, or joke entries │ │
|
||
│ │ - Have no meaningful description (< 100 words) │ │
|
||
│ │ - Are unrelated to ocean conservation │ │
|
||
│ │ - Are duplicate submissions │ │
|
||
│ │ │ │
|
||
│ │ Flag for manual review: │ │
|
||
│ │ - Projects with unclear focus or vague descriptions │ │
|
||
│ │ - Projects that may be legitimate but borderline │ │
|
||
│ └──────────────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
│ Action: ( ) PASS (•) REJECT ( ) FLAG │
|
||
│ │
|
||
│ Priority: [ 1 ] (1 = highest, 10 = lowest) │
|
||
│ │
|
||
│ [ ] Active │
|
||
│ │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ Preview & Test │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ Test this rule on sample projects: │
|
||
│ │
|
||
│ [✓] SaveTheSea - AI-powered coral reef monitoring │
|
||
│ Result: PASS (confidence: 0.92) │
|
||
│ Reasoning: Clear ocean conservation focus, detailed approach │
|
||
│ │
|
||
│ [✓] Test Project - Just testing the form │
|
||
│ Result: REJECT (confidence: 0.98) │
|
||
│ Reasoning: Appears to be a test submission with no real value │
|
||
│ │
|
||
│ [✓] Marine Plastic Solution (vague description) │
|
||
│ Result: FLAG (confidence: 0.65) │
|
||
│ Reasoning: Ocean-related but description lacks detail │
|
||
│ │
|
||
│ [ Run Test on 10 Random Projects ] │
|
||
│ │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ [ Save Rule ] [ Delete Rule ] [ Cancel ] │
|
||
│ │
|
||
└────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### AI Results Review Interface
|
||
|
||
```
|
||
┌────────────────────────────────────────────────────────────────────┐
|
||
│ Round 2: FILTERING > AI Results Review │
|
||
├────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ AI Screening Complete │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ Processed: 120 projects │
|
||
│ Passed: 87 (72.5%) │
|
||
│ Rejected: 21 (17.5%) │
|
||
│ Flagged: 12 (10.0%) │
|
||
│ │
|
||
│ Tokens Used: 45,678 | Est. Cost: $1.37 │
|
||
│ Model: gpt-4 | Batches: 6 of 20 projects each │
|
||
│ │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ Flagged Projects (Require Manual Review) │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ [ ] Marine Plastic Solution │
|
||
│ Confidence: 0.65 │
|
||
│ AI Reasoning: Ocean-related but description lacks sufficient │
|
||
│ detail. Unclear commercial model. May be early- │
|
||
│ stage concept that needs more development. │
|
||
│ Quality Score: 6/10 │
|
||
│ Spam Risk: No │
|
||
│ │
|
||
│ Admin Decision: ( ) PASS ( ) REJECT ( ) Keep Flagged │
|
||
│ │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ [ ] Ocean Cleanup AI │
|
||
│ Confidence: 0.58 │
|
||
│ AI Reasoning: Uses AI buzzwords extensively but lacks technical│
|
||
│ specifics. May be over-promising. Borderline. │
|
||
│ Quality Score: 5/10 │
|
||
│ Spam Risk: No │
|
||
│ │
|
||
│ Admin Decision: ( ) PASS ( ) REJECT ( ) Keep Flagged │
|
||
│ │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ [ Review All Flagged (12) ] [ Bulk Approve ] [ Bulk Reject ] │
|
||
│ │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ Rejected Projects (Can Override) │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ [✓] Test Submission │
|
||
│ Confidence: 0.98 │
|
||
│ AI Reasoning: Obvious test submission with placeholder text. │
|
||
│ Override: [ ] Approve This Project │
|
||
│ │
|
||
│ [✓] Spam Project 123 │
|
||
│ Confidence: 0.95 │
|
||
│ AI Reasoning: Unrelated to ocean conservation, generic content.│
|
||
│ Override: [ ] Approve This Project │
|
||
│ │
|
||
│ [ Show All Rejected (21) ] │
|
||
│ │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ [ Finalize Results ] [ Export CSV ] [ Re-run AI Screening ] │
|
||
│ │
|
||
└────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### Cost Monitoring Dashboard
|
||
|
||
```
|
||
┌────────────────────────────────────────────────────────────────────┐
|
||
│ Analytics > AI Usage & Costs │
|
||
├────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ Overview (February 2024) │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ Total API Calls: 1,234 │
|
||
│ Total Tokens: 4,567,890 │
|
||
│ Estimated Cost: $137.04 │
|
||
│ Avg Cost per Call: $0.11 │
|
||
│ │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ Cost Breakdown by Service │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ Filtering: $45.67 (33%) ████████░░░ │
|
||
│ Assignment: $38.21 (28%) ███████░░░░ │
|
||
│ Evaluation Summary: $28.90 (21%) █████░░░░░░ │
|
||
│ Tagging: $15.12 (11%) ███░░░░░░░░ │
|
||
│ Award Eligibility: $9.14 (7%) ██░░░░░░░░░ │
|
||
│ │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ Token Usage Trend │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ 500K ┤ ╭─ │
|
||
│ 400K ┤ ╭────────╯ │
|
||
│ 300K ┤ ╭────────────╯ │
|
||
│ 200K ┤ ╭─────────────╯ │
|
||
│ 100K ┤ ╭────────╯ │
|
||
│ 0 ┼───┴──────────────────────────────────────────────────── │
|
||
│ Feb 1 Feb 8 Feb 15 Feb 22 Feb 29 │
|
||
│ │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ Recent API Calls │
|
||
│ ────────────────────────────────────────────────────────────────── │
|
||
│ │
|
||
│ 2024-02-15 14:30 | Filtering | Round 2 | 20 projects │
|
||
│ gpt-4 | 3,456 tokens | $0.10 | SUCCESS │
|
||
│ │
|
||
│ 2024-02-15 14:15 | Assignment | Round 3 | 15 projects │
|
||
│ gpt-4 | 4,123 tokens | $0.12 | SUCCESS │
|
||
│ │
|
||
│ 2024-02-15 13:45 | Summary | Proj-123 | 5 evaluations │
|
||
│ gpt-4-turbo | 1,890 tokens | $0.02 | ✓ │
|
||
│ │
|
||
│ [ View All Logs ] [ Export CSV ] [ Set Budget Alert ] │
|
||
│ │
|
||
└────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## API Changes
|
||
|
||
### New tRPC Procedures
|
||
|
||
#### AI Router (`src/server/routers/ai.ts`)
|
||
|
||
```typescript
|
||
export const aiRouter = createTRPCRouter({
|
||
|
||
// Test OpenAI connection
|
||
testConnection: adminProcedure
|
||
.mutation(async ({ ctx }) => {
|
||
const openai = await getOpenAI()
|
||
if (!openai) {
|
||
throw new TRPCError({ code: 'PRECONDITION_FAILED', message: 'OpenAI not configured' })
|
||
}
|
||
|
||
try {
|
||
await openai.models.list()
|
||
return { connected: true, message: 'OpenAI connection successful' }
|
||
} catch (error) {
|
||
return { connected: false, message: error.message }
|
||
}
|
||
}),
|
||
|
||
// Get AI usage stats
|
||
getUsageStats: adminProcedure
|
||
.input(z.object({
|
||
startDate: z.date(),
|
||
endDate: z.date()
|
||
}))
|
||
.query(async ({ ctx, input }) => {
|
||
const logs = await ctx.prisma.aIUsageLog.findMany({
|
||
where: {
|
||
createdAt: {
|
||
gte: input.startDate,
|
||
lte: input.endDate
|
||
}
|
||
}
|
||
})
|
||
|
||
const totalTokens = logs.reduce((sum, log) => sum + log.totalTokens, 0)
|
||
const totalCost = logs.reduce((sum, log) => sum + (log.estimatedCost || 0), 0)
|
||
|
||
const byService = logs.reduce((acc, log) => {
|
||
const service = log.action
|
||
if (!acc[service]) {
|
||
acc[service] = { count: 0, tokens: 0, cost: 0 }
|
||
}
|
||
acc[service].count++
|
||
acc[service].tokens += log.totalTokens
|
||
acc[service].cost += log.estimatedCost || 0
|
||
return acc
|
||
}, {} as Record<string, { count: number, tokens: number, cost: number }>)
|
||
|
||
return {
|
||
totalCalls: logs.length,
|
||
totalTokens,
|
||
totalCost,
|
||
avgCostPerCall: totalCost / logs.length,
|
||
byService
|
||
}
|
||
}),
|
||
|
||
// Tag a single project
|
||
tagProject: adminProcedure
|
||
.input(z.object({
|
||
projectId: z.string()
|
||
}))
|
||
.mutation(async ({ ctx, input }) => {
|
||
const result = await tagProject(input.projectId, ctx.session.user.id)
|
||
return result
|
||
}),
|
||
|
||
// Get tag suggestions (preview only, don't apply)
|
||
getTagSuggestions: adminProcedure
|
||
.input(z.object({
|
||
projectId: z.string()
|
||
}))
|
||
.query(async ({ ctx, input }) => {
|
||
const suggestions = await getTagSuggestions(input.projectId, ctx.session.user.id)
|
||
return suggestions
|
||
}),
|
||
|
||
// Generate evaluation summary
|
||
generateEvaluationSummary: adminProcedure
|
||
.input(z.object({
|
||
projectId: z.string(),
|
||
roundId: z.string()
|
||
}))
|
||
.mutation(async ({ ctx, input }) => {
|
||
const summary = await generateSummary({
|
||
projectId: input.projectId,
|
||
stageId: input.roundId,
|
||
userId: ctx.session.user.id,
|
||
prisma: ctx.prisma
|
||
})
|
||
return summary
|
||
}),
|
||
|
||
// Generate AI assignments
|
||
generateAssignments: adminProcedure
|
||
.input(z.object({
|
||
roundId: z.string(),
|
||
constraints: z.object({
|
||
requiredReviewsPerProject: z.number(),
|
||
minAssignmentsPerJuror: z.number().optional(),
|
||
maxAssignmentsPerJuror: z.number().optional()
|
||
})
|
||
}))
|
||
.mutation(async ({ ctx, input }) => {
|
||
const round = await ctx.prisma.round.findUnique({
|
||
where: { id: input.roundId },
|
||
include: {
|
||
juryGroup: {
|
||
include: {
|
||
members: { include: { user: true } }
|
||
}
|
||
}
|
||
}
|
||
})
|
||
|
||
if (!round?.juryGroup) {
|
||
throw new TRPCError({ code: 'NOT_FOUND', message: 'Round or jury group not found' })
|
||
}
|
||
|
||
const projects = await ctx.prisma.project.findMany({
|
||
where: {
|
||
projectRoundStates: {
|
||
some: {
|
||
roundId: input.roundId,
|
||
state: 'PASSED'
|
||
}
|
||
}
|
||
},
|
||
include: { _count: { select: { assignments: true } } }
|
||
})
|
||
|
||
const existingAssignments = await ctx.prisma.assignment.findMany({
|
||
where: { roundId: input.roundId },
|
||
select: { userId: true, projectId: true }
|
||
})
|
||
|
||
const result = await generateAIAssignments(
|
||
round.juryGroup.members.map(m => m.user),
|
||
projects,
|
||
{
|
||
...input.constraints,
|
||
existingAssignments
|
||
},
|
||
ctx.session.user.id,
|
||
input.roundId
|
||
)
|
||
|
||
return result
|
||
}),
|
||
|
||
// Run filtering on a round
|
||
runFiltering: adminProcedure
|
||
.input(z.object({
|
||
roundId: z.string()
|
||
}))
|
||
.mutation(async ({ ctx, input }) => {
|
||
const round = await ctx.prisma.round.findUnique({
|
||
where: { id: input.roundId }
|
||
})
|
||
|
||
if (round?.roundType !== 'FILTERING') {
|
||
throw new TRPCError({ code: 'BAD_REQUEST', message: 'Not a filtering round' })
|
||
}
|
||
|
||
const rules = await ctx.prisma.filteringRule.findMany({
|
||
where: { roundId: input.roundId, isActive: true },
|
||
orderBy: { priority: 'asc' }
|
||
})
|
||
|
||
const projects = await ctx.prisma.project.findMany({
|
||
where: {
|
||
projectRoundStates: {
|
||
some: { roundId: input.roundId }
|
||
}
|
||
},
|
||
include: {
|
||
files: true,
|
||
_count: { select: { teamMembers: true } }
|
||
}
|
||
})
|
||
|
||
const results = await executeFilteringRules(
|
||
rules,
|
||
projects,
|
||
ctx.session.user.id,
|
||
input.roundId
|
||
)
|
||
|
||
// Store results
|
||
for (const result of results) {
|
||
await ctx.prisma.filteringResult.upsert({
|
||
where: {
|
||
projectId_roundId: {
|
||
projectId: result.projectId,
|
||
roundId: input.roundId
|
||
}
|
||
},
|
||
create: {
|
||
projectId: result.projectId,
|
||
roundId: input.roundId,
|
||
outcome: result.outcome,
|
||
ruleResultsJson: result.ruleResults,
|
||
aiScreeningJson: result.aiScreeningJson
|
||
},
|
||
update: {
|
||
outcome: result.outcome,
|
||
ruleResultsJson: result.ruleResults,
|
||
aiScreeningJson: result.aiScreeningJson
|
||
}
|
||
})
|
||
}
|
||
|
||
return {
|
||
total: results.length,
|
||
passed: results.filter(r => r.outcome === 'PASSED').length,
|
||
rejected: results.filter(r => r.outcome === 'FILTERED_OUT').length,
|
||
flagged: results.filter(r => r.outcome === 'FLAGGED').length
|
||
}
|
||
}),
|
||
|
||
// Check award eligibility
|
||
checkAwardEligibility: adminProcedure
|
||
.input(z.object({
|
||
awardId: z.string()
|
||
}))
|
||
.mutation(async ({ ctx, input }) => {
|
||
const award = await ctx.prisma.specialAward.findUnique({
|
||
where: { id: input.awardId }
|
||
})
|
||
|
||
if (!award?.useAiEligibility || !award.criteriaText) {
|
||
throw new TRPCError({ code: 'BAD_REQUEST', message: 'AI eligibility not enabled for this award' })
|
||
}
|
||
|
||
const projects = await ctx.prisma.project.findMany({
|
||
where: { competitionId: award.competitionId },
|
||
include: {
|
||
files: true,
|
||
_count: { select: { teamMembers: true } }
|
||
}
|
||
})
|
||
|
||
const results = await aiInterpretCriteria(
|
||
award.criteriaText,
|
||
projects,
|
||
ctx.session.user.id,
|
||
award.id
|
||
)
|
||
|
||
// Store results
|
||
for (const result of results) {
|
||
await ctx.prisma.awardEligibility.upsert({
|
||
where: {
|
||
specialAwardId_projectId: {
|
||
specialAwardId: award.id,
|
||
projectId: result.projectId
|
||
}
|
||
},
|
||
create: {
|
||
specialAwardId: award.id,
|
||
projectId: result.projectId,
|
||
isEligible: result.eligible,
|
||
confidence: result.confidence,
|
||
reasoning: result.reasoning,
|
||
method: result.method
|
||
},
|
||
update: {
|
||
isEligible: result.eligible,
|
||
confidence: result.confidence,
|
||
reasoning: result.reasoning,
|
||
method: result.method
|
||
}
|
||
})
|
||
}
|
||
|
||
return {
|
||
total: results.length,
|
||
eligible: results.filter(r => r.eligible).length,
|
||
ineligible: results.filter(r => !r.eligible).length
|
||
}
|
||
})
|
||
})
|
||
```
|
||
|
||
---
|
||
|
||
## Service Functions
|
||
|
||
### Complete Function Signatures
|
||
|
||
```typescript
|
||
// ai-filtering.ts
|
||
export function evaluateFieldRule(
|
||
config: FieldRuleConfig,
|
||
project: ProjectForFiltering
|
||
): { passed: boolean; action: 'PASS' | 'REJECT' | 'FLAG' }
|
||
|
||
export function evaluateDocumentRule(
|
||
config: DocumentCheckConfig,
|
||
project: ProjectForFiltering
|
||
): { passed: boolean; action: 'PASS' | 'REJECT' | 'FLAG' }
|
||
|
||
export async function executeAIScreening(
|
||
config: AIScreeningConfig,
|
||
projects: ProjectForFiltering[],
|
||
userId?: string,
|
||
entityId?: string,
|
||
onProgress?: ProgressCallback
|
||
): Promise<Map<string, AIScreeningResult>>
|
||
|
||
export async function executeFilteringRules(
|
||
rules: FilteringRuleInput[],
|
||
projects: ProjectForFiltering[],
|
||
userId?: string,
|
||
stageId?: string,
|
||
onProgress?: ProgressCallback
|
||
): Promise<ProjectFilteringResult[]>
|
||
|
||
// ai-assignment.ts
|
||
export async function generateAIAssignments(
|
||
jurors: JurorForAssignment[],
|
||
projects: ProjectForAssignment[],
|
||
constraints: AssignmentConstraints,
|
||
userId?: string,
|
||
entityId?: string,
|
||
onProgress?: AssignmentProgressCallback
|
||
): Promise<AIAssignmentResult>
|
||
|
||
export function generateFallbackAssignments(
|
||
jurors: JurorForAssignment[],
|
||
projects: ProjectForAssignment[],
|
||
constraints: AssignmentConstraints
|
||
): AIAssignmentResult
|
||
|
||
// ai-evaluation-summary.ts
|
||
export function anonymizeEvaluations(
|
||
evaluations: EvaluationForSummary[]
|
||
): AnonymizedEvaluation[]
|
||
|
||
export function buildSummaryPrompt(
|
||
anonymizedEvaluations: AnonymizedEvaluation[],
|
||
projectTitle: string,
|
||
criteriaLabels: string[]
|
||
): string
|
||
|
||
export function computeScoringPatterns(
|
||
evaluations: EvaluationForSummary[],
|
||
criteriaLabels: CriterionDef[]
|
||
): ScoringPatterns
|
||
|
||
export async function generateSummary({
|
||
projectId,
|
||
stageId,
|
||
userId,
|
||
prisma
|
||
}: {
|
||
projectId: string
|
||
stageId: string
|
||
userId: string
|
||
prisma: PrismaClient
|
||
}): Promise<EvaluationSummaryResult>
|
||
|
||
// ai-tagging.ts
|
||
export async function getTaggingSettings(): Promise<{
|
||
enabled: boolean
|
||
maxTags: number
|
||
}>
|
||
|
||
export async function getAvailableTags(): Promise<AvailableTag[]>
|
||
|
||
export async function tagProject(
|
||
projectId: string,
|
||
userId?: string
|
||
): Promise<TaggingResult>
|
||
|
||
export async function getTagSuggestions(
|
||
projectId: string,
|
||
userId?: string
|
||
): Promise<TagSuggestion[]>
|
||
|
||
export async function addProjectTag(
|
||
projectId: string,
|
||
tagId: string
|
||
): Promise<void>
|
||
|
||
export async function removeProjectTag(
|
||
projectId: string,
|
||
tagId: string
|
||
): Promise<void>
|
||
|
||
// ai-award-eligibility.ts
|
||
export function applyAutoTagRules(
|
||
rules: AutoTagRule[],
|
||
projects: ProjectForEligibility[]
|
||
): Map<string, boolean>
|
||
|
||
export async function aiInterpretCriteria(
|
||
criteriaText: string,
|
||
projects: ProjectForEligibility[],
|
||
userId?: string,
|
||
awardId?: string
|
||
): Promise<EligibilityResult[]>
|
||
|
||
// anonymization.ts
|
||
export function sanitizeText(text: string): string
|
||
|
||
export function truncateAndSanitize(
|
||
text: string | null | undefined,
|
||
maxLength: number
|
||
): string
|
||
|
||
export function anonymizeForAI(
|
||
jurors: JurorInput[],
|
||
projects: ProjectInput[]
|
||
): AnonymizationResult
|
||
|
||
export function anonymizeProjectForAI(
|
||
project: ProjectWithRelations,
|
||
index: number,
|
||
context: DescriptionContext
|
||
): AnonymizedProjectForAI
|
||
|
||
export function anonymizeProjectsForAI(
|
||
projects: ProjectWithRelations[],
|
||
context: DescriptionContext
|
||
): {
|
||
anonymized: AnonymizedProjectForAI[]
|
||
mappings: ProjectAIMapping[]
|
||
}
|
||
|
||
export function deanonymizeResults<T extends { jurorId: string; projectId: string }>(
|
||
results: T[],
|
||
jurorMappings: JurorMapping[],
|
||
projectMappings: ProjectMapping[]
|
||
): (T & { realJurorId: string; realProjectId: string })[]
|
||
|
||
export function validateNoPersonalData(
|
||
data: Record<string, unknown>
|
||
): PIIValidationResult
|
||
|
||
export function enforceGDPRCompliance(data: unknown[]): void
|
||
|
||
export function validateAnonymization(data: AnonymizationResult): boolean
|
||
|
||
export function validateAnonymizedProjects(
|
||
projects: AnonymizedProjectForAI[]
|
||
): boolean
|
||
|
||
export function toProjectWithRelations(project: unknown): ProjectWithRelations
|
||
```
|
||
|
||
---
|
||
|
||
## Edge Cases
|
||
|
||
| Scenario | Behavior | Fallback |
|
||
|----------|----------|----------|
|
||
| **OpenAI API key missing** | AI services disabled | Use algorithm-based fallbacks |
|
||
| **OpenAI rate limit hit** | Queue requests, retry with exponential backoff | Fail gracefully after 3 retries |
|
||
| **AI returns invalid JSON** | Parse error logged, flagged for manual review | Mark items as "needs review" |
|
||
| **AI model doesn't exist** | Throw clear error with model name | Suggest checking Settings > AI Config |
|
||
| **Anonymization validation fails** | Throw error, log GDPR violation | Reject AI call, require manual review |
|
||
| **All projects rejected by AI** | Admin gets warning notification | Admin reviews AI reasoning |
|
||
| **Zero tag suggestions** | Return empty array | No error, just no tags applied |
|
||
| **Confidence too low (<0.5)** | Tag not applied automatically | Admin can manually apply |
|
||
| **Duplicate project detected (>0.85 similarity)** | Flagged for admin review | Admin marks as duplicate or false positive |
|
||
| **Mentor workspace inactive (>14 days)** | Intervention alert sent to admin | Admin reaches out to mentor |
|
||
| **Winner ranking unclear (close gap <0.3)** | AI flags as "close call", suggests review | Jury deliberates again |
|
||
| **Batch size too large (>50)** | Clamped to 50 projects per batch | Automatically split into multiple batches |
|
||
| **Description too long (>500 chars)** | Truncated with "..." | AI works with truncated text |
|
||
| **PII in feedback text** | Sanitized before sending to AI | Emails/phones replaced with `[email removed]` |
|
||
| **Juror has no expertise tags** | Scored as 0.5 (neutral) in assignment | Fallback algorithm includes them for load balancing |
|
||
| **Project has no description** | AI flags as low quality | Admin decides whether to reject |
|
||
| **All jurors at capacity** | Assignment fails with clear error | Admin adjusts caps or adds more jurors |
|
||
| **Award criteria too vague** | AI gives low-confidence results | Admin refines criteria and re-runs |
|
||
| **Evaluation summary with <3 evaluations** | AI summary still generated, marked as "limited data" | Admin aware that consensus may be weak |
|
||
| **Streaming response timeout** | Fall back to non-streaming request | Complete response returned after wait |
|
||
|
||
---
|
||
|
||
## Integration Map
|
||
|
||
### AI Services × Round Types
|
||
|
||
```mermaid
|
||
graph TD
|
||
R1[Round 1: INTAKE] --> T[ai-tagging.ts]
|
||
T --> TAG[Auto-tag on submit]
|
||
|
||
R2[Round 2: FILTERING] --> F[ai-filtering.ts]
|
||
F --> SCREEN[AI screening + rules]
|
||
F --> DUP[ai-duplicate-detection]
|
||
DUP --> EMBED[Embedding similarity]
|
||
|
||
R3[Round 3: EVALUATION] --> A[ai-assignment.ts]
|
||
A --> ASSIGN[Generate assignments]
|
||
A --> E[ai-evaluation-summary.ts]
|
||
E --> SUM[Summarize evaluations]
|
||
|
||
R4[Round 4: SUBMISSION] --> NONE1[No AI service]
|
||
|
||
R5[Round 5: EVALUATION] --> A
|
||
A --> E
|
||
|
||
R6[Round 6: MENTORING] --> M[ai-mentoring-insights.ts]
|
||
M --> INSIGHT[Workspace analysis]
|
||
|
||
R7[Round 7: LIVE_FINAL] --> NONE2[No AI service]
|
||
|
||
R8[Round 8: CONFIRMATION] --> C[ai-confirmation-helper.ts]
|
||
C --> EXPLAIN[Ranking explanation]
|
||
|
||
AWARDS[Special Awards] --> AE[ai-award-eligibility.ts]
|
||
AE --> ELIG[Eligibility check]
|
||
```
|
||
|
||
### Data Flow Diagram
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant Admin
|
||
participant API
|
||
participant Service
|
||
participant Anonymization
|
||
participant OpenAI
|
||
participant DB
|
||
|
||
Admin->>API: Trigger AI service
|
||
API->>Service: Call service function
|
||
Service->>DB: Fetch data
|
||
DB-->>Service: Raw data
|
||
Service->>Anonymization: Anonymize data
|
||
Anonymization->>Anonymization: Strip PII
|
||
Anonymization->>Anonymization: Replace IDs
|
||
Anonymization->>Anonymization: Validate GDPR
|
||
Anonymization-->>Service: Anonymized + mappings
|
||
Service->>OpenAI: API call
|
||
OpenAI-->>Service: AI response
|
||
Service->>Anonymization: De-anonymize results
|
||
Anonymization-->>Service: Real IDs
|
||
Service->>DB: Store results
|
||
DB-->>Service: Confirmation
|
||
Service->>DB: Log AI usage
|
||
Service-->>API: Results
|
||
API-->>Admin: Success + data
|
||
```
|
||
|
||
---
|
||
|
||
This completes the extremely detailed AI Services documentation covering all current services, new services for the redesign, anonymization pipeline, prompt engineering, OpenAI integration, privacy/security, admin controls, API changes, service functions, edge cases, and integration map.
|
||
|
||
Total lines: 2,900+
|