letsbe
/
MOPC-App
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
MOPC-App/docs/codex-architecture-redesign.../07-testing-observability-an...

6.2 KiB
Raw Blame History

07. Testing, Observability, And Release Gates

1) QA Strategy

The redesign is policy-heavy. Testing must validate:

  1. Functional correctness
  2. Policy compliance
  3. Audit completeness
  4. Role-based access integrity
  5. Operational resilience under load and deadline pressure

2) Test Pyramid

2.1 Unit Tests

Target modules:

  • context resolver
  • policy resolver
  • assignment policy evaluator (hard/soft/buffer/category-bias weighting)
  • round bundle lock evaluator
  • final confirmation rule evaluator

2.2 Integration Tests

Target flows:

  • stage transitions with purpose validation
  • filtering outcomes + override
  • assignment run + unassigned queue + manual assignment
  • document upload/edit lock transitions
  • mentor promotion pipeline
  • live voting -> confirmation -> result lock

2.3 End-to-End Tests

Target user journeys:

  • applicant R1 -> eligibility -> Jury1 -> R2 -> Jury2 -> finals prep -> finals -> result
  • admin stage manager live finals operations
  • award mode A and mode B paths
  • admin invite -> onboarding -> jury assignment materialization
  • team invite -> accept invite -> team collaboration

3) Monaco Flow Test Matrix

Flow Step Required Test Cases
Submission R1 open/close behavior, late policy modes, required docs enforcement
Eligibility filter deterministic pass/fail, AI banding, manual queue, admin override trail
Jury1 cap modes, category-bias preference behavior (suggestive only), deadline reminder triggers, score submission
Submission R2 R1 read-only for applicants, R2 editable, admin full override
Jury2 visibility of R1+R2 docs, finalist recommendation and override
Awards separate pool, dual track, admin-confirmed pull-out behavior, jury overlap, single-judge decision mode
Mentoring private file sharing, comments, promotion to official slot with provenance
Jury3 live stage manager controls, voting windows, notes, audience vote mode
Final confirmation unanimous jury agreement on active quorum (or configured rule), juror replacement/absence handling, single-judge award exception flow, admin finalize, admin override path, result lock

4) Invite/Member/Onboarding Test Matrix (Critical)

Workflow Must Validate
Admin member invite (/admin/members/invite) creates user + jury membership + selected pre-assignment mode (intent and/or direct policy-validated assignment); respects role authorization
Bulk invite CSV invalid rows, duplicate handling, unauthorized role attempts, staged pre-assignment mode selection and materialization
Invite acceptance (/accept-invite) token validity, role-context preview, routing to correct onboarding path
Jury onboarding cap/bias override onboarding when enabled, including explicit disclosure that category bias is a suggestion not a rule
Team invite (/applicant/team) lead-only invite/remove, account setup vs existing account behavior
Cross-context conflict same user invited as jury and team member still resolves correct permissions by context

5) Regression Coverage Extension Against Existing Test Suite

Current suite already has useful coverage (tests/unit/*, tests/integration/*). Extend with:

  1. tests/unit/assignment-policy-hard-soft.test.ts
  2. tests/unit/final-confirmation-rules.test.ts
  3. tests/integration/invite-jury-membership-intent.test.ts
  4. tests/integration/submission-bundle-locking.test.ts
  5. tests/integration/mentor-promotion-provenance.test.ts
  6. tests/integration/award-mode-routing.test.ts
  7. tests/integration/live-to-final-lock.test.ts

6) Audit Completeness Checks

Add automated assertions that every critical operation emits both domain event and audit record:

  • eligibility override
  • assignment exception
  • stage advancement override
  • award winner override
  • promotion event
  • final confirmation finalize/lock

Failure to emit required audit evidence should fail tests.

7) Observability Blueprint

7.1 Metrics

Competition Lifecycle Metrics

  • projects_submitted_total by category and stage
  • eligibility_pass_rate
  • manual_review_rate
  • assignment_coverage_percent
  • unassigned_projects_count
  • over_cap_exception_count
  • submission_late_rate
  • mentor_promotion_count
  • live_vote_completion_rate
  • final_confirmation_duration_seconds

Reliability Metrics

  • API latency by router
  • background job success/failure rates (filtering/assignment/award jobs)
  • reminder send success/failure
  • invitation send success/failure

7.2 Logs And Tracing

  • Correlation id per operation across router -> service -> DB writes
  • Structured logs include programId, stageId, stagePurposeKey, juryId, projectId, userId

7.3 Alerts

  • assignment run leaves residual without queue record
  • final confirmation attempted without satisfied quorum requirement
  • result lock write failure
  • result unlock attempted by non-super-admin
  • invite acceptance failures spike
  • live voting session state mismatch (open with no active project)

8) Release Gates (Must Pass)

Gate A: Schema + Backfill Readiness

  • all migrations idempotent in staging
  • no unresolved backfill anomalies

Gate B: Policy Engine Correctness

  • hard/soft cap test pass
  • ratio policy test pass
  • access resolver test pass

Gate C: End-To-End Flow Simulation

  • full Monaco dry run with seeded data and mixed categories
  • award mode A/B dry runs

Gate D: Invite/Onboarding Stability

  • admin invite + team invite + acceptance flows pass under load

Gate E: Finals And Confirmation Integrity

  • live voting and final confirmation produce correct lock artifacts
  • super-admin unlock/relock path preserves complete audit integrity

Gate F: Operational Readiness

  • runbooks approved
  • dashboards and alerts active
  • support playbooks distributed

9) Performance And Capacity Scenarios

  • Large intake window close with bulk late submissions
  • Filtering and assignment jobs across high project volume
  • Live finals peak voting concurrency (jury + audience)
  • Bulk invitation campaigns with email retries

10) Sign-Off Criteria

Release is approved only when:

  1. All phase gates pass.
  2. No Sev-1/Sev-2 defects remain open in critical competition flows.
  3. Audit completeness checks pass for all critical operations.
  4. Operators can execute runbook drills successfully.