Testing & QA
Overview
This document defines the test strategy for the redesigned competition system. It covers the test pyramid, specific test matrices for the Monaco competition flow and deliberation system, regression coverage, audit verification, and performance scenarios.
Test Pyramid
╱╲
╱ E2E ╲ ~10 tests — Full Monaco flow simulation
╱────────╲
╱Integration╲ ~50 tests — Service + database interaction
╱──────────────╲
╱ Unit Tests ╲ ~200+ tests — Pure logic, no I/O
╱────────────────────╲
| Level |
What |
Tools |
Speed |
| Unit |
Pure functions: policy resolution, cap calculation, Borda count, tie-breaking logic, config validation |
Vitest |
< 1s each |
| Integration |
Service + Prisma: assignment algorithm with real DB, round transitions, deliberation vote aggregation |
Vitest + test DB |
< 5s each |
| E2E |
Full flow: create competition → run all 8 rounds → lock results |
Vitest + test DB + tRPC callers |
< 30s each |
Monaco Flow Test Matrix
End-to-end tests that simulate the complete Monaco 2026 competition:
R1: Intake
| Test Case |
Validates |
| Submit project before deadline |
Submission accepted, project created with category |
| Submit project after deadline (HARD policy) |
Submission rejected |
| Submit project after deadline (FLAG policy) |
Submission accepted, marked as Late |
| Submit with missing required doc |
Validation error returned |
| Admin uploads doc for applicant |
File associated, provenance recorded |
| Submission window not yet open |
Submission rejected |
R2: AI Filtering
| Test Case |
Validates |
| AI marks project eligible |
ProjectRoundState = ELIGIBLE |
| AI marks project ineligible |
ProjectRoundState = INELIGIBLE, reason stored |
| AI marks project for manual review |
ProjectRoundState = MANUAL_REVIEW |
| Admin overrides ineligible → eligible |
Override recorded in audit log |
| Admin overrides eligible → ineligible |
Override recorded in audit log |
| Filtering with no AI (deterministic rules only) |
Rules-based filtering works without AI |
R3: Jury 1 Evaluation
| Test Case |
Validates |
| Assignment respects hard cap |
Judge gets ≤ maxProjects |
| Assignment distributes overflow to soft cap judges |
Overflow distributed evenly, up to cap + buffer |
| Unassigned projects enter queue |
Remaining projects have reason codes |
| COI declared → project skipped |
COI judge doesn't get COI project |
| Judge submits score within window |
Score recorded |
| Judge submits score outside window |
Rejected |
| AI shortlist generated at round end |
Ranked recommendations exist per category |
| Admin overrides shortlist selection |
Override recorded, selected projects advance |
R4: Semifinal Submission
| Test Case |
Validates |
| Applicant uploads R2 docs |
Files accepted for correct slots |
| Applicant tries to edit R1 docs |
Rejected (read-only) |
| Admin replaces R1 doc |
Replacement accepted with provenance |
| Applicant downloads R1 doc |
Download succeeds |
R5: Jury 2 Evaluation + Special Awards
| Test Case |
Validates |
| Judge sees R1 + R2 docs clearly separated |
Both sets of docs returned with round labels |
| Award Mode A: projects filtered into award pool |
Eligible projects appear in award pool |
| Award Mode A: admin confirms pull-out |
Pull-out confirmed, project moved |
| Award Mode B: projects flagged but stay in main |
Project in both main and award evaluation |
| Award single-judge decision |
Single judge can select winner |
| Top N finalists selected (N configurable) |
Exactly N projects advance per category |
R6: Mentoring
| Test Case |
Validates |
| Mentor sends message to team |
Message delivered and visible |
| Team uploads file to workspace |
File visible to mentor + team + admin |
| Mentor comments on file |
Comment created with thread support |
| File promoted to official submission |
SubmissionPromotionEvent created with provenance |
| Non-mentored finalist has no workspace |
No workspace features visible |
R7: Live Finals
| Test Case |
Validates |
| Admin advances project cursor |
Current project updates for all Jury 3 members |
| Jury 3 member submits live score |
Score recorded within voting window |
| Audience vote submitted |
Vote counted |
| Audience vote outside window |
Rejected |
| Jury 3 sees prior jury data (when enabled) |
Prior jury scores/feedback visible |
| Jury 3 doesn't see prior jury data (when disabled) |
Prior jury data hidden |
R8: Deliberation
| Test Case |
Validates |
| SINGLE_WINNER_VOTE: juror picks winner |
Vote recorded |
| SINGLE_WINNER_VOTE: tally produces ranking |
Most votes = rank 1, others by vote count |
| FULL_RANKING: juror submits ordinal ranks |
All ranks recorded |
| FULL_RANKING: Borda count aggregation |
Correct Borda scores computed |
| Tie detected → runoff initiated |
Session status = RUNOFF, new vote round |
| Tie detected → admin breaks tie |
Admin decision recorded |
| Admin overrides entire result |
Override result stored, flag set |
| Admin locks result |
ResultLock created with snapshot |
| Super-admin unlocks result |
ResultUnlockEvent created with reason |
| Non-super-admin tries to unlock |
Rejected |
Deliberation-Specific Test Matrix
Detailed tests for the deliberation subsystem:
SINGLE_WINNER_VOTE Mode
| Scenario |
Expected Outcome |
| 3 jurors, unanimous pick |
Winner = picked project, rank 1 |
| 3 jurors, split vote (2-1) |
Winner = project with 2 votes |
| 3 jurors, three-way tie (1-1-1) |
Tie-break method triggered |
| Juror submits vote, then changes |
Latest vote overwrites (within window) |
| Voting window closes |
No more votes accepted |
FULL_RANKING Mode
| Scenario |
Expected Outcome |
| 3 jurors rank 5 projects |
Borda count: 1st=5pts, 2nd=4pts, ... 5th=1pt |
| Clear winner (highest Borda) |
Rank 1 = highest Borda score |
| Two projects tied on Borda |
Tie-break method triggered |
| Juror ranks only top 3 (partial) |
Unranked projects get 0 points from this juror |
Tie-Breaking
| Method |
Test Case |
Expected |
| RUNOFF |
Two projects tied → runoff vote |
New voting round with only tied projects |
| RUNOFF |
Runoff still tied → second runoff |
Session supports multiple runoff rounds |
| ADMIN_DECIDES |
Tie detected → admin picks winner |
Admin decision recorded, tie resolved |
| SCORE_FALLBACK |
Tie → fall back to Jury 3 scores |
Higher Jury 3 score wins |
Participant Management
| Scenario |
Expected |
| Juror marked ABSENT_EXCUSED |
Doesn't count toward quorum |
| Juror REPLACED |
Replacement can vote, original cannot |
| Quorum not met |
Session cannot proceed to VOTING |
Invite/Onboarding Test Matrix
| Test Case |
Validates |
| Admin invites judge to Jury 1 |
JuryGroupMember created |
| Judge accepts invite and onboards |
Member status active, self-service values set |
| Judge adjusts cap during onboarding |
selfServiceCap updated |
| Judge adjusts ratio during onboarding |
selfServiceBias updated |
| Self-service disabled → fields hidden |
No self-service options shown |
| Admin overrides self-service values |
Override takes precedence |
| Invite with pre-assignment intent |
AssignmentIntent record created |
| Invite with pre-assignment + COI conflict |
Intent stays PENDING, reason INTENT_BLOCKED |
| Judge removed from group with pending intent |
Intent status → CANCELLED |
| Round completes with unmatched intent |
Intent status → EXPIRED |
Config Schema Validation Test Matrix
All 7 round-type Zod schemas must validate both correct and malformed input:
| Schema |
Valid Input Test |
Invalid Input Test |
IntakeConfigSchema |
Valid with deadlinePolicy: "FLAG", maxFileSizeMB: 50 |
Rejects negative maxFileSizeMB, unknown deadlinePolicy |
FilteringConfigSchema |
Valid with aiEnabled: true, autoAdvanceEligible: false |
Rejects autoAdvanceEligible without aiEnabled set |
EvaluationConfigSchema |
Valid with rubric, requireFeedback: true, feedbackMinLength: 50 |
Rejects feedbackMinLength < 0, missing rubric |
SubmissionConfigSchema |
Valid with doc slot requirements, deadlinePolicy: "HARD" |
Rejects empty requiredDocSlots, invalid MIME types |
MentoringConfigSchema |
Valid with eligibility: "all_advancing", filePromotionEnabled: true |
Rejects unknown eligibility value |
LiveFinalConfigSchema |
Valid with audienceVoteWeight: 0.3, presentationDurationMinutes: 15 |
Rejects audienceVoteWeight > 1, negative duration |
DeliberationConfigSchema |
Valid with mode: "FULL_RANKING", topN: 3, tieBreakMethod: "RUNOFF" |
Rejects topN < 1, unknown mode |
Edge cases (all schemas):
| Test Case |
Expected |
| Empty object |
Defaults applied, valid |
| Extra unknown fields |
Stripped (Zod .strip()) |
null config |
Rejected — config is required |
| Partial config (some fields) |
Missing fields get defaults |
Assignment Intent Lifecycle Tests
| Test Case |
Expected |
| Create intent at invite time |
AssignmentIntent created with status PENDING, source INVITE |
| Algorithm runs, intent matchable |
Intent transitions to HONORED, Assignment record created |
| Algorithm runs, intent blocked by COI |
Intent stays PENDING, unassigned queue entry with INTENT_BLOCKED |
| Algorithm runs, intent blocked by cap |
Intent stays PENDING, unassigned queue entry with INTENT_BLOCKED |
| Admin reassigns project to different judge |
Intent transitions to OVERRIDDEN, audit logged |
| Judge removed from JuryGroup |
Intent transitions to CANCELLED |
| Round closes with pending intent |
Intent transitions to EXPIRED (batch update) |
| Admin explicitly cancels intent |
Intent transitions to CANCELLED |
Intent already HONORED → no further transitions |
Terminal state enforced |
Intent already EXPIRED → no further transitions |
Terminal state enforced |
Regression Coverage
Existing test files that need updates for the redesign:
| Test File |
Required Updates |
tests/unit/stage-engine.test.ts |
Rename to round-engine, update all type references |
tests/unit/stage-filtering.test.ts |
Update to use RoundType.FILTERING, Competition model |
tests/unit/stage-assignment.test.ts |
Add JuryGroup-based assignment tests, cap mode tests |
tests/integration/pipeline-crud.test.ts |
Rename to competition-crud, test Competition/Round CRUD |
tests/integration/evaluation-flow.test.ts |
Update to use Round model, multi-round doc visibility |
New test files to create:
| Test File |
Coverage |
tests/unit/policy-resolution.test.ts |
5-layer policy precedence |
tests/unit/borda-count.test.ts |
Borda count aggregation |
tests/unit/deliberation-tally.test.ts |
Vote tallying for both modes |
tests/unit/config-validation.test.ts |
All 7 Zod schema validations |
tests/integration/deliberation-flow.test.ts |
Full deliberation lifecycle |
tests/integration/mentor-workspace.test.ts |
Messaging, files, comments, promotion |
tests/integration/submission-windows.test.ts |
Multi-round doc lifecycle |
tests/e2e/monaco-full-flow.test.ts |
Complete 8-round simulation |
Audit Completeness Checks
Every critical operation must emit an audit record to DecisionAuditLog:
| Operation |
Must Audit |
| Admin overrides eligibility |
Yes — who, when, project, old→new status |
| Admin overrides assignment |
Yes — who, when, judge, project, reason |
| Admin overrides shortlist selection |
Yes — who, when, selected projects |
| Admin overrides deliberation result |
Yes — who, when, reason, original→override |
| Result lock |
Yes — who, when, snapshot |
| Result unlock |
Yes — who, when, reason (super-admin only) |
| File promotion |
Yes — who, when, source file, target slot |
| Award pull-out confirmation |
Yes — who, when, projects pulled |
| Judge cap/ratio override |
Yes — who, when, old→new values |
Test: For each operation above, verify an audit record exists with all required fields.
Performance & Capacity Scenarios
| Scenario |
Target |
Test Method |
| Bulk intake: 500 projects submitted in 1 hour |
All submissions processed, no timeouts |
Load test with concurrent submissions |
| Assignment: 500 projects across 30 judges |
Assignment completes < 30s |
Integration test with timing |
| Live voting: 50 audience members voting simultaneously |
All votes recorded, no conflicts |
Concurrent request test |
| Deliberation: 15 jurors submitting rankings |
Aggregation completes < 5s |
Integration test with timing |
| Multi-round doc query: judge views 3 rounds of docs |
Response < 2s |
Query performance test |
Test Data Factories
All tests use factory functions (see tests/helpers.ts):
// New factories needed
createTestCompetition(overrides?)
createTestRound(competitionId, type, overrides?)
createTestJuryGroup(competitionId, overrides?)
createTestJuryGroupMember(juryGroupId, userId, overrides?)
createTestSubmissionWindow(competitionId, roundId, overrides?)
createTestDeliberationSession(competitionId, roundId, overrides?)
createTestDeliberationVote(sessionId, juryMemberId, overrides?)
createTestSpecialAward(competitionId, overrides?)
createTestMentorFile(projectId, mentorId, overrides?)
See 09-implementation-roadmap.md Phase 0 for factory creation timeline.