158 lines
6.2 KiB
Markdown
158 lines
6.2 KiB
Markdown
|
# 07. Testing, Observability, And Release Gates
|
||
|
|
|
||
|
|
## 1) QA Strategy
|
||
|
|
The redesign is policy-heavy. Testing must validate:
|
||
|
|
1. Functional correctness
|
||
|
|
2. Policy compliance
|
||
|
|
3. Audit completeness
|
||
|
|
4. Role-based access integrity
|
||
|
|
5. Operational resilience under load and deadline pressure
|
||
|
|
|
||
|
|
## 2) Test Pyramid
|
||
|
|
|
||
|
|
### 2.1 Unit Tests
|
||
|
|
Target modules:
|
||
|
|
- context resolver
|
||
|
|
- policy resolver
|
||
|
|
- assignment policy evaluator (hard/soft/buffer/category-bias weighting)
|
||
|
|
- round bundle lock evaluator
|
||
|
|
- final confirmation rule evaluator
|
||
|
|
|
||
|
|
### 2.2 Integration Tests
|
||
|
|
Target flows:
|
||
|
|
- stage transitions with purpose validation
|
||
|
|
- filtering outcomes + override
|
||
|
|
- assignment run + unassigned queue + manual assignment
|
||
|
|
- document upload/edit lock transitions
|
||
|
|
- mentor promotion pipeline
|
||
|
|
- live voting -> confirmation -> result lock
|
||
|
|
|
||
|
|
### 2.3 End-to-End Tests
|
||
|
|
Target user journeys:
|
||
|
|
- applicant R1 -> eligibility -> Jury1 -> R2 -> Jury2 -> finals prep -> finals -> result
|
||
|
|
- admin stage manager live finals operations
|
||
|
|
- award mode A and mode B paths
|
||
|
|
- admin invite -> onboarding -> jury assignment materialization
|
||
|
|
- team invite -> accept invite -> team collaboration
|
||
|
|
|
||
|
|
## 3) Monaco Flow Test Matrix
|
||
|
|
|
||
|
|
| Flow Step | Required Test Cases |
|
||
|
|
|---|---|
|
||
|
|
| Submission R1 | open/close behavior, late policy modes, required docs enforcement |
|
||
|
|
| Eligibility filter | deterministic pass/fail, AI banding, manual queue, admin override trail |
|
||
|
|
| Jury1 | cap modes, category-bias preference behavior (suggestive only), deadline reminder triggers, score submission |
|
||
|
|
| Submission R2 | R1 read-only for applicants, R2 editable, admin full override |
|
||
|
|
| Jury2 | visibility of R1+R2 docs, finalist recommendation and override |
|
||
|
|
| Awards | separate pool, dual track, admin-confirmed pull-out behavior, jury overlap, single-judge decision mode |
|
||
|
|
| Mentoring | private file sharing, comments, promotion to official slot with provenance |
|
||
|
|
| Jury3 live | stage manager controls, voting windows, notes, audience vote mode |
|
||
|
|
| Final confirmation | unanimous jury agreement on active quorum (or configured rule), juror replacement/absence handling, single-judge award exception flow, admin finalize, admin override path, result lock |
|
||
|
|
|
||
|
|
## 4) Invite/Member/Onboarding Test Matrix (Critical)
|
||
|
|
|
||
|
|
| Workflow | Must Validate |
|
||
|
|
|---|---|
|
||
|
|
| Admin member invite (`/admin/members/invite`) | creates user + jury membership + selected pre-assignment mode (intent and/or direct policy-validated assignment); respects role authorization |
|
||
|
|
| Bulk invite CSV | invalid rows, duplicate handling, unauthorized role attempts, staged pre-assignment mode selection and materialization |
|
||
|
|
| Invite acceptance (`/accept-invite`) | token validity, role-context preview, routing to correct onboarding path |
|
||
|
|
| Jury onboarding | cap/bias override onboarding when enabled, including explicit disclosure that category bias is a suggestion not a rule |
|
||
|
|
| Team invite (`/applicant/team`) | lead-only invite/remove, account setup vs existing account behavior |
|
||
|
|
| Cross-context conflict | same user invited as jury and team member still resolves correct permissions by context |
|
||
|
|
|
||
|
|
## 5) Regression Coverage Extension Against Existing Test Suite
|
||
|
|
Current suite already has useful coverage (`tests/unit/*`, `tests/integration/*`). Extend with:
|
||
|
|
|
||
|
|
1. `tests/unit/assignment-policy-hard-soft.test.ts`
|
||
|
|
2. `tests/unit/final-confirmation-rules.test.ts`
|
||
|
|
3. `tests/integration/invite-jury-membership-intent.test.ts`
|
||
|
|
4. `tests/integration/submission-bundle-locking.test.ts`
|
||
|
|
5. `tests/integration/mentor-promotion-provenance.test.ts`
|
||
|
|
6. `tests/integration/award-mode-routing.test.ts`
|
||
|
|
7. `tests/integration/live-to-final-lock.test.ts`
|
||
|
|
|
||
|
|
## 6) Audit Completeness Checks
|
||
|
|
Add automated assertions that every critical operation emits both domain event and audit record:
|
||
|
|
- eligibility override
|
||
|
|
- assignment exception
|
||
|
|
- stage advancement override
|
||
|
|
- award winner override
|
||
|
|
- promotion event
|
||
|
|
- final confirmation finalize/lock
|
||
|
|
|
||
|
|
Failure to emit required audit evidence should fail tests.
|
||
|
|
|
||
|
|
## 7) Observability Blueprint
|
||
|
|
|
||
|
|
## 7.1 Metrics
|
||
|
|
|
||
|
|
### Competition Lifecycle Metrics
|
||
|
|
- `projects_submitted_total` by category and stage
|
||
|
|
- `eligibility_pass_rate`
|
||
|
|
- `manual_review_rate`
|
||
|
|
- `assignment_coverage_percent`
|
||
|
|
- `unassigned_projects_count`
|
||
|
|
- `over_cap_exception_count`
|
||
|
|
- `submission_late_rate`
|
||
|
|
- `mentor_promotion_count`
|
||
|
|
- `live_vote_completion_rate`
|
||
|
|
- `final_confirmation_duration_seconds`
|
||
|
|
|
||
|
|
### Reliability Metrics
|
||
|
|
- API latency by router
|
||
|
|
- background job success/failure rates (filtering/assignment/award jobs)
|
||
|
|
- reminder send success/failure
|
||
|
|
- invitation send success/failure
|
||
|
|
|
||
|
|
## 7.2 Logs And Tracing
|
||
|
|
- Correlation id per operation across router -> service -> DB writes
|
||
|
|
- Structured logs include `programId`, `stageId`, `stagePurposeKey`, `juryId`, `projectId`, `userId`
|
||
|
|
|
||
|
|
## 7.3 Alerts
|
||
|
|
- assignment run leaves residual without queue record
|
||
|
|
- final confirmation attempted without satisfied quorum requirement
|
||
|
|
- result lock write failure
|
||
|
|
- result unlock attempted by non-super-admin
|
||
|
|
- invite acceptance failures spike
|
||
|
|
- live voting session state mismatch (open with no active project)
|
||
|
|
|
||
|
|
## 8) Release Gates (Must Pass)
|
||
|
|
|
||
|
|
## Gate A: Schema + Backfill Readiness
|
||
|
|
- all migrations idempotent in staging
|
||
|
|
- no unresolved backfill anomalies
|
||
|
|
|
||
|
|
## Gate B: Policy Engine Correctness
|
||
|
|
- hard/soft cap test pass
|
||
|
|
- ratio policy test pass
|
||
|
|
- access resolver test pass
|
||
|
|
|
||
|
|
## Gate C: End-To-End Flow Simulation
|
||
|
|
- full Monaco dry run with seeded data and mixed categories
|
||
|
|
- award mode A/B dry runs
|
||
|
|
|
||
|
|
## Gate D: Invite/Onboarding Stability
|
||
|
|
- admin invite + team invite + acceptance flows pass under load
|
||
|
|
|
||
|
|
## Gate E: Finals And Confirmation Integrity
|
||
|
|
- live voting and final confirmation produce correct lock artifacts
|
||
|
|
- super-admin unlock/relock path preserves complete audit integrity
|
||
|
|
|
||
|
|
## Gate F: Operational Readiness
|
||
|
|
- runbooks approved
|
||
|
|
- dashboards and alerts active
|
||
|
|
- support playbooks distributed
|
||
|
|
|
||
|
|
## 9) Performance And Capacity Scenarios
|
||
|
|
- Large intake window close with bulk late submissions
|
||
|
|
- Filtering and assignment jobs across high project volume
|
||
|
|
- Live finals peak voting concurrency (jury + audience)
|
||
|
|
- Bulk invitation campaigns with email retries
|
||
|
|
|
||
|
|
## 10) Sign-Off Criteria
|
||
|
|
Release is approved only when:
|
||
|
|
1. All phase gates pass.
|
||
|
|
2. No Sev-1/Sev-2 defects remain open in critical competition flows.
|
||
|
|
3. Audit completeness checks pass for all critical operations.
|
||
|
|
4. Operators can execute runbook drills successfully.
|