29 KiB
29 KiB
Round: Evaluation (Jury 1 & Jury 2)
1. Purpose & Position in Flow
The EVALUATION round is the core judging mechanism of the competition. It appears twice in the standard flow:
| Instance | Name | Position | Jury | Purpose | Output |
|---|---|---|---|---|---|
| Round 3 | "Jury 1 — Semi-finalist Selection" | After FILTERING | Jury 1 | Score projects, select semi-finalists | Semi-finalists per category |
| Round 5 | "Jury 2 — Finalist Selection" | After SUBMISSION Round 2 | Jury 2 | Score semi-finalists, select finalists + awards | Finalists per category |
Both instances use the same RoundType.EVALUATION but are configured independently with:
- Different jury groups (Jury 1 vs Jury 2)
- Different evaluation forms/rubrics
- Different visible submission windows (Jury 1 sees Window 1 only; Jury 2 sees Windows 1+2)
- Different advancement counts
2. Data Model
Round Record
Round {
id: "round-jury-1"
competitionId: "comp-2026"
name: "Jury 1 — Semi-finalist Selection"
roundType: EVALUATION
status: ROUND_DRAFT → ROUND_ACTIVE → ROUND_CLOSED
sortOrder: 2
windowOpenAt: "2026-04-01" // Evaluation window start
windowCloseAt: "2026-04-30" // Evaluation window end
juryGroupId: "jury-group-1" // Links to Jury 1
submissionWindowId: null // EVALUATION rounds don't collect submissions
configJson: { ...EvaluationConfig }
}
EvaluationConfig
type EvaluationConfig = {
// --- Assignment Settings ---
requiredReviewsPerProject: number // How many jurors review each project (default: 3)
// --- Scoring Mode ---
scoringMode: "criteria" | "global" | "binary"
// criteria: Score per criterion + weighted total
// global: Single 1-10 score
// binary: Yes/No decision (semi-finalist worthy?)
requireFeedback: boolean // Must provide text feedback (default: true)
// --- COI ---
coiRequired: boolean // Must declare COI before evaluating (default: true)
// --- Peer Review ---
peerReviewEnabled: boolean // Jurors can see anonymized peer evaluations after submission
anonymizationLevel: "fully_anonymous" | "show_initials" | "named"
// --- AI Features ---
aiSummaryEnabled: boolean // Generate AI-powered evaluation summaries
aiAssignmentEnabled: boolean // Allow AI-suggested jury-project matching
// --- Advancement ---
advancementMode: "auto_top_n" | "admin_selection" | "ai_recommended"
advancementConfig: {
perCategory: boolean // Separate counts per STARTUP / BUSINESS_CONCEPT
startupCount: number // How many startups advance (default: 10 for Jury 1, 3 for Jury 2)
conceptCount: number // How many concepts advance
tieBreaker: "admin_decides" | "highest_individual" | "revote"
}
}
Related Models
| Model | Role |
|---|---|
JuryGroup |
Named jury entity linked to this round |
JuryGroupMember |
Members of the jury with per-juror overrides |
Assignment |
Juror-project pairing for this round, linked to JuryGroup |
Evaluation |
Score/feedback submitted by a juror for one project |
EvaluationForm |
Rubric/criteria definition for this round |
ConflictOfInterest |
COI declaration per assignment |
GracePeriod |
Per-juror deadline extension |
EvaluationSummary |
AI-generated insights per project per round |
EvaluationDiscussion |
Peer review discussion threads |
RoundSubmissionVisibility |
Which submission windows' docs jury can see |
AdvancementRule |
How projects advance after evaluation |
ProjectRoundState |
Per-project state in this round |
3. Setup Phase (Before Window Opens)
3.1 Admin Creates the Evaluation Round
Admin uses the competition wizard or round management UI to:
- Create the Round with type EVALUATION
- Link a JuryGroup — select "Jury 1" (or create a new jury group)
- Set the evaluation window — start and end dates
- Configure the evaluation form — scoring criteria, weights, scales
- Set visibility — which submission windows jury can see (via RoundSubmissionVisibility)
- Configure advancement rules — how many advance per category
3.2 Jury Group Configuration
The linked JuryGroup has:
JuryGroup {
name: "Jury 1"
defaultMaxAssignments: 20 // Default cap per juror
defaultCapMode: SOFT // HARD | SOFT | NONE
softCapBuffer: 2 // Can exceed by 2 for load balancing
categoryQuotasEnabled: true
defaultCategoryQuotas: {
"STARTUP": { "min": 3, "max": 15 },
"BUSINESS_CONCEPT": { "min": 3, "max": 15 }
}
allowJurorCapAdjustment: true // Jurors can adjust their cap during onboarding
allowJurorRatioAdjustment: true // Jurors can adjust their category preference
}
3.3 Per-Juror Overrides
Each JuryGroupMember can override group defaults:
JuryGroupMember {
juryGroupId: "jury-group-1"
userId: "judge-alice"
maxAssignmentsOverride: 25 // Alice wants more projects
capModeOverride: HARD // Alice: hard cap, no exceptions
categoryQuotasOverride: {
"STARTUP": { "min": 5, "max": 20 }, // Alice prefers startups
"BUSINESS_CONCEPT": { "min": 0, "max": 5 }
}
preferredStartupRatio: 0.8 // 80% startups
}
3.4 Juror Onboarding (Optional)
If allowJurorCapAdjustment or allowJurorRatioAdjustment is true:
- When a juror first opens their jury dashboard after being added to the group
- A one-time onboarding dialog appears:
- "Your default maximum is 20 projects. Would you like to adjust?" (slider)
- "Your default startup/concept ratio is 50/50. Would you like to adjust?" (slider)
- Juror saves preferences → stored in
JuryGroupMember.maxAssignmentsOverrideandpreferredStartupRatio - Dialog doesn't appear again (tracked via
JuryGroupMember.updatedAtor a flag)
4. Assignment System (Enhanced)
4.1 Assignment Algorithm — Jury-Group-Aware
The current stage-assignment.ts algorithm is enhanced to:
- Filter jury pool by JuryGroup — only members of the linked jury group are considered
- Apply hard/soft cap logic per juror
- Apply category quotas per juror
- Score candidates using existing expertise matching + workload balancing + geo-diversity
Effective Limits Resolution
function getEffectiveLimits(member: JuryGroupMember, group: JuryGroup): EffectiveLimits {
return {
maxAssignments: member.maxAssignmentsOverride ?? group.defaultMaxAssignments,
capMode: member.capModeOverride ?? group.defaultCapMode,
softCapBuffer: group.softCapBuffer, // Group-level only (not per-juror)
categoryQuotas: member.categoryQuotasOverride ?? group.defaultCategoryQuotas,
categoryQuotasEnabled: group.categoryQuotasEnabled,
preferredStartupRatio: member.preferredStartupRatio,
}
}
Cap Enforcement Logic
function canAssignMore(
jurorId: string,
projectCategory: CompetitionCategory,
currentLoad: LoadTracker,
limits: EffectiveLimits
): { allowed: boolean; penalty: number; reason?: string } {
const total = currentLoad.total(jurorId)
const catLoad = currentLoad.byCategory(jurorId, projectCategory)
// 1. HARD cap check
if (limits.capMode === "HARD" && total >= limits.maxAssignments) {
return { allowed: false, penalty: 0, reason: "Hard cap reached" }
}
// 2. SOFT cap check (can exceed by buffer)
let overflowPenalty = 0
if (limits.capMode === "SOFT") {
if (total >= limits.maxAssignments + limits.softCapBuffer) {
return { allowed: false, penalty: 0, reason: "Soft cap + buffer exceeded" }
}
if (total >= limits.maxAssignments) {
// In buffer zone — apply increasing penalty
overflowPenalty = (total - limits.maxAssignments + 1) * 15
}
}
// 3. Category quota check
if (limits.categoryQuotasEnabled && limits.categoryQuotas) {
const quota = limits.categoryQuotas[projectCategory]
if (quota) {
if (catLoad >= quota.max) {
return { allowed: false, penalty: 0, reason: `Category ${projectCategory} max reached (${quota.max})` }
}
// Bonus for under-min
if (catLoad < quota.min) {
overflowPenalty -= 15 // Negative penalty = bonus
}
}
}
// 4. Ratio preference alignment
if (limits.preferredStartupRatio != null && total > 0) {
const currentStartupRatio = currentLoad.byCategory(jurorId, "STARTUP") / total
const isStartup = projectCategory === "STARTUP"
const wantMore = isStartup
? currentStartupRatio < limits.preferredStartupRatio
: currentStartupRatio > limits.preferredStartupRatio
if (wantMore) overflowPenalty -= 10 // Bonus for aligning with preference
else overflowPenalty += 10 // Penalty for diverging
}
return { allowed: true, penalty: overflowPenalty }
}
4.2 Assignment Flow
1. Admin opens Assignment panel for Round 3 (Jury 1)
2. System loads:
- Projects with ProjectRoundState PENDING/IN_PROGRESS in this round
- JuryGroup members (with effective limits)
- Existing assignments (to avoid duplicates)
- COI records (to skip conflicted pairs)
3. Admin clicks "Generate Suggestions"
4. Algorithm runs:
a. For each project (sorted by fewest current assignments):
- Score each eligible juror (tag matching + workload + geo + cap/quota penalties)
- Select top N jurors (N = requiredReviewsPerProject - existing reviews)
- Track load in jurorLoadMap
b. Report unassigned projects (jurors at capacity)
5. Admin reviews preview:
- Assignment matrix (juror × project grid)
- Load distribution chart
- Unassigned projects list
- Category distribution per juror
6. Admin can:
- Accept all suggestions
- Modify individual assignments (drag-drop or manual add/remove)
- Re-run with different parameters
7. Admin clicks "Apply Assignments"
8. System creates Assignment records with juryGroupId set
9. Notifications sent to jurors
4.3 AI-Powered Assignment (Optional)
If aiAssignmentEnabled is true in config:
- Admin clicks "AI Assignment Suggestions"
- System calls
ai-assignment.ts:- Anonymizes juror profiles and project descriptions
- Sends to GPT with matching instructions
- Returns confidence scores and reasoning
- AI suggestions shown alongside algorithm suggestions
- Admin picks which to use or mixes both
4.4 Handling Unassigned Projects
When all jurors with SOFT cap reach cap+buffer:
- Remaining projects become "unassigned"
- Admin dashboard highlights these prominently
- Admin can:
- Manually assign to specific jurors (bypasses cap — manual override)
- Increase a juror's cap
- Add more jurors to the jury group
- Reduce
requiredReviewsPerProjectfor remaining projects
5. Jury Evaluation Experience
5.1 Jury Dashboard
When a Jury 1 member opens their dashboard:
┌─────────────────────────────────────────────────────┐
│ JURY 1 — Semi-finalist Selection │
│ ─────────────────────────────────────────────────── │
│ Evaluation Window: April 1 – April 30 │
│ ⏱ 12 days remaining │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────┐ │
│ │ 15 │ │ 8 │ │ 2 │ │ 5 │ │
│ │ Total │ │ Complete │ │ In Draft │ │ Pending│ │
│ └──────────┘ └──────────┘ └──────────┘ └────────┘ │
│ │
│ [Continue Next Evaluation →] │
│ │
│ Recent Assignments │
│ ┌──────────────────────────────────────────────┐ │
│ │ OceanClean AI │ Startup │ ✅ Done │ View │ │
│ │ Blue Carbon Hub │ Concept │ ⏳ Draft │ Cont │ │
│ │ SeaWatch Monitor │ Startup │ ⬜ Pending│ Start│ │
│ │ ... │ │
│ └──────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘
Key elements:
- Deadline countdown — prominent timer showing days/hours remaining
- Progress stats — total, completed, in-draft, pending
- Quick action CTA — jump to next unevaluated project
- Assignment list — sorted by status (pending first, then drafts, then done)
5.2 COI Declaration (Blocking)
Before evaluating any project, the juror MUST declare COI:
┌───────────────────────────────────────────┐
│ Conflict of Interest Declaration │
│ │
│ Do you have a conflict of interest with │
│ "OceanClean AI" (Startup)? │
│ │
│ ○ No conflict — I can evaluate fairly │
│ ○ Yes, I have a conflict: │
│ Type: [Financial ▾] │
│ Description: [________________] │
│ │
│ [Submit Declaration] │
└───────────────────────────────────────────┘
- If No conflict: Proceed to evaluation form
- If Yes: Assignment flagged, admin notified, juror may be reassigned
- COI declaration is logged in
ConflictOfInterestmodel - Admin can review and take action (cleared / reassigned / noted)
5.3 Evaluation Form
The form adapts to the scoringMode:
Criteria Mode (default for Jury 1 and Jury 2)
┌───────────────────────────────────────────────────┐
│ Evaluating: OceanClean AI (Startup) │
│ ──────────────────────────────────────────────── │
│ │
│ [📄 Documents] [📊 Scoring] [💬 Feedback] │
│ │
│ ── DOCUMENTS TAB ── │
│ ┌─ Round 1 Application Docs ─────────────────┐ │
│ │ 📄 Executive Summary.pdf [Download] │ │
│ │ 📄 Business Plan.pdf [Download] │ │
│ └─────────────────────────────────────────────┘ │
│ │
│ (Jury 2 also sees:) │
│ ┌─ Round 2 Semi-finalist Docs ────────────────┐ │
│ │ 📄 Updated Business Plan.pdf [Download] │ │
│ │ 🎥 Video Pitch.mp4 [Play] │ │
│ └─────────────────────────────────────────────┘ │
│ │
│ ── SCORING TAB ── │
│ Innovation & Impact [1] [2] [3] [4] [5] (w:30%)│
│ Feasibility [1] [2] [3] [4] [5] (w:25%)│
│ Team & Execution [1] [2] [3] [4] [5] (w:25%)│
│ Ocean Relevance [1] [2] [3] [4] [5] (w:20%)│
│ │
│ Overall Score: 3.8 / 5.0 (auto-calculated) │
│ │
│ ── FEEDBACK TAB ── │
│ Feedback: [________________________________] │
│ │
│ [💾 Save Draft] [✅ Submit Evaluation] │
│ (Auto-saves every 30s) │
└───────────────────────────────────────────────────┘
Binary Mode (optional for quick screening)
Should this project advance to the semi-finals?
[✅ Yes] [❌ No]
Justification (required): [________________]
Global Score Mode
Overall Score: [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]
Feedback (required): [________________]
5.4 Document Visibility (Cross-Round)
Controlled by RoundSubmissionVisibility:
| Round | Sees Window 1 ("Application Docs") | Sees Window 2 ("Semi-finalist Docs") |
|---|---|---|
| Jury 1 (Round 3) | Yes | No (doesn't exist yet) |
| Jury 2 (Round 5) | Yes | Yes |
| Jury 3 (Round 7) | Yes | Yes |
In the evaluation UI:
- Documents are grouped by submission window
- Each group has a label (from
RoundSubmissionVisibility.displayLabel) - Clear visual separation (tabs, accordion sections, or side panels)
5.5 Auto-Save and Submission
- Auto-save: Client debounces and calls
evaluation.autosaveevery 30 seconds while draft is open - Draft status: Evaluation starts as NOT_STARTED → DRAFT on first save → SUBMITTED on explicit submit
- Submission validation:
- All required criteria scored (if criteria mode)
- Global score provided (if global mode)
- Binary decision selected (if binary mode)
- Feedback text provided (if
requireFeedback) - Window is open (or juror has grace period)
- After submission: Evaluation becomes read-only for juror (status = SUBMITTED)
- Admin can lock: Set status to LOCKED to prevent any further changes
5.6 Grace Periods
GracePeriod {
roundId: "round-jury-1"
userId: "judge-alice"
projectId: null // Applies to ALL Alice's assignments in this round
extendedUntil: "2026-05-02" // 2 days after official close
reason: "Travel conflict"
grantedById: "admin-1"
}
- Admin can grant per-juror or per-juror-per-project grace periods
- Evaluation submission checks grace period before rejecting past-window submissions
- Dashboard shows "(Grace period: 2 extra days)" badge for affected jurors
6. End of Evaluation — Results & Advancement
6.1 Results Visualization
When the evaluation window closes, the admin sees:
┌──────────────────────────────────────────────────────────────┐
│ Jury 1 Results │
│ ─────────────────────────────────────────────────────────── │
│ │
│ Completion: 142/150 evaluations submitted (94.7%) │
│ Outstanding: 8 (3 jurors have pending evaluations) │
│ │
│ ┌─ STARTUPS (Top 10) ──────────────────────────────────────┐│
│ │ # Project Avg Score Consensus Reviews Status ││
│ │ 1 OceanClean AI 4.6/5 0.92 3/3 ✅ ││
│ │ 2 SeaWatch 4.3/5 0.85 3/3 ✅ ││
│ │ 3 BlueCarbon 4.1/5 0.78 3/3 ✅ ││
│ │ ... ││
│ │ 10 TidalEnergy 3.2/5 0.65 3/3 ✅ ││
│ │ ── cutoff line ────────────────────────────────────────── ││
│ │ 11 WavePower 3.1/5 0.71 3/3 ⬜ ││
│ │ 12 CoralGuard 2.9/5 0.55 2/3 ⚠️ ││
│ └──────────────────────────────────────────────────────────┘│
│ │
│ ┌─ CONCEPTS (Top 10) ──────────────────────────────────────┐│
│ │ (same layout) ││
│ └──────────────────────────────────────────────────────────┘│
│ │
│ [🤖 AI Recommendation] [📊 Score Distribution] [Export] │
│ │
│ [✅ Approve Shortlist] [✏️ Edit Shortlist] │
└──────────────────────────────────────────────────────────────┘
Metrics shown:
- Average global score (or weighted criteria average)
- Consensus score (1 - normalized stddev, where 1.0 = full agreement)
- Review count / required
- Per-criterion averages (expandable)
6.2 AI Recommendation
When admin clicks "AI Recommendation":
- System calls
ai-evaluation-summary.tsfor each project in bulk - AI generates:
- Ranked shortlist per category based on scores + feedback analysis
- Strengths, weaknesses, themes per project
- Recommendation: "Advance" / "Borderline" / "Do not advance"
- Admin sees AI recommendation alongside actual scores
- AI recommendations are suggestions only — admin has final say
6.3 Advancement Decision
Advancement Mode: admin_selection (with AI recommendation)
1. System shows ranked list per category
2. AI highlights recommended top N per category
3. Admin can:
- Accept AI recommendation
- Drag projects to reorder
- Add/remove projects from advancement list
- Set custom cutoff line
4. Admin clicks "Confirm Advancement"
5. System:
a. Sets ProjectRoundState to PASSED for advancing projects
b. Sets ProjectRoundState to REJECTED for non-advancing projects
c. Updates Project.status to SEMIFINALIST (Jury 1) or FINALIST (Jury 2)
d. Logs all decisions in DecisionAuditLog
e. Sends notifications to all teams (advanced / not selected)
6.4 Advancement Modes
| Mode | Behavior |
|---|---|
auto_top_n |
Top N per category automatically advance when window closes |
admin_selection |
Admin manually selects who advances (with AI/score guidance) |
ai_recommended |
AI proposes list, admin must approve/modify |
7. Special Awards Integration (Jury 2 Only)
During the Jury 2 evaluation round, special awards can run alongside:
7.1 How It Works
Round 5: "Jury 2 — Finalist Selection"
├── Main evaluation (all semi-finalists scored by Jury 2)
└── Special Awards (run in parallel):
├── "Innovation Award" — STAY_IN_MAIN mode
│ Projects remain in main eval, flagged as eligible
│ Award jury (subset of Jury 2 or separate) votes
└── "Impact Award" — SEPARATE_POOL mode
AI filters eligible projects into award pool
Dedicated jury evaluates and votes
7.2 SpecialAward.evaluationRoundId
Each award links to the evaluation round it runs alongside:
SpecialAward {
evaluationRoundId: "round-jury-2" // Runs during Jury 2
eligibilityMode: STAY_IN_MAIN
juryGroupId: "jury-group-innovation" // Can be same or different jury
}
7.3 Award Evaluation Flow
- Before Jury 2 window opens: Admin runs award eligibility (AI or manual)
- During Jury 2 window: Award jury members see their award assignments alongside regular evaluations
- Award jury submits award votes (PICK_WINNER, RANKED, or SCORED)
- After Jury 2 closes: Award results finalized alongside main results
8. Differences Between Jury 1 and Jury 2
| Aspect | Jury 1 (Round 3) | Jury 2 (Round 5) |
|---|---|---|
| Input projects | All eligible (post-filtering) | Semi-finalists only |
| Visible docs | Window 1 only | Window 1 + Window 2 |
| Output | Semi-finalists | Finalists |
| Project.status update | → SEMIFINALIST | → FINALIST |
| Special awards | No | Yes (alongside) |
| Jury group | Jury 1 | Jury 2 (different members, possible overlap) |
| Typical project count | 50-100+ | 10-20 |
| Required reviews | 3 (more projects, less depth) | 3-5 (fewer projects, more depth) |
9. API Changes
Preserved Procedures (renamed stageId → roundId)
| Procedure | Change |
|---|---|
evaluation.get |
roundId via assignment |
evaluation.start |
No change |
evaluation.autosave |
No change |
evaluation.submit |
Window check uses round.windowCloseAt + grace periods |
evaluation.declareCOI |
No change |
evaluation.getCOIStatus |
No change |
evaluation.getProjectStats |
No change |
evaluation.listByRound |
Renamed from listByStage |
evaluation.generateSummary |
roundId instead of stageId |
evaluation.generateBulkSummaries |
roundId instead of stageId |
New Procedures
| Procedure | Purpose |
|---|---|
assignment.previewWithJuryGroup |
Preview assignments filtered by jury group with cap/quota logic |
assignment.getJuryGroupStats |
Per-member stats: load, category distribution, cap utilization |
evaluation.getResultsOverview |
Rankings, scores, consensus, AI recommendations per category |
evaluation.confirmAdvancement |
Admin confirms which projects advance |
evaluation.getAdvancementPreview |
Preview advancement impact before confirming |
Modified Procedures
| Procedure | Modification |
|---|---|
assignment.getSuggestions |
Now filters by JuryGroup, applies hard/soft caps, category quotas |
assignment.create |
Now sets juryGroupId on Assignment |
assignment.bulkCreate |
Now validates against jury group caps |
file.listByProjectForRound |
Uses RoundSubmissionVisibility to filter docs |
10. Service Layer Changes
stage-assignment.ts → round-assignment.ts
Key changes to previewStageAssignment → previewRoundAssignment:
- Load jury pool from JuryGroup instead of all JURY_MEMBER users:
const juryGroup = await prisma.juryGroup.findUnique({
where: { id: round.juryGroupId },
include: { members: { include: { user: true } } }
})
const jurors = juryGroup.members.map(m => ({
...m.user,
effectiveLimits: getEffectiveLimits(m, juryGroup),
}))
- Replace simple max check with cap mode logic (hard/soft/none)
- Add category quota tracking per juror
- Add ratio preference scoring in candidate ranking
- Report overflow — projects that couldn't be assigned because all jurors hit caps
stage-engine.ts → round-engine.ts
Simplified:
- Remove trackId from all transitions
executeTransitionnow takesfromRoundId+toRoundId(or auto-advance to next sortOrder)validateTransitionsimplified — no StageTransition lookup, just checks next round exists and is active- Guard evaluation simplified — AdvancementRule.configJson replaces arbitrary guardJson
11. Edge Cases
More projects than jurors can handle
- Algorithm assigns up to hard/soft cap for all jurors
- Remaining projects flagged as "unassigned" in admin dashboard
- Admin must: add jurors, increase caps, or manually assign
Juror doesn't complete by deadline
- Dashboard shows overdue assignments prominently
- Admin can: extend via GracePeriod, reassign to another juror, or mark as incomplete
Tie in scores at cutoff
- Depending on
tieBreakerconfig:admin_decides: Admin manually picks from tied projectshighest_individual: Project with highest single-evaluator score winsrevote: Tied projects sent back for quick re-evaluation
Category imbalance
- If one category has far more projects, quotas ensure jurors still get a mix
- If quotas can't be satisfied (not enough of one category), system relaxes quota for that category
Juror in multiple jury groups
- Juror Alice is in Jury 1 and Jury 2
- Her assignments for each round are independent
- Her caps are per-jury-group (20 for Jury 1, 15 for Jury 2)
- No cross-round cap — each round manages its own workload