LetsBeBiz-Redesign/docs/architecture-proposal/gpt/03-deployment-strategy.md

03. Deployment Strategy

1. Goals

• Ship to founding members in ~12 weeks without compromising security invariants.
• Maintain one-VPS-per-customer isolation.
• Keep OpenClaw upstream-pinned and independently upgradeable.
• Make tenant rollout reversible with fast rollback paths.

2. Environment Topology

2.1 Control Plane Environments

Environment	Purpose	Data
dev	Rapid feature iteration	Synthetic/local data
staging	Release-candidate validation, e2e, load, security checks	Sanitized fixtures
prod-eu	EU customers (default EU routing)	Real customer data
prod-us	NA customers (default NA routing)	Real customer data

Control plane services (Hub + worker + notifications) are region-deployed with independent DBs and clear region affinity.

2.2 Tenant Environments

• sandbox tenants: internal QA and interactive demo pool.
• canary tenants: first real-production update recipients.
• general tenants: full customer fleet.

3. Deployment Units

3.1 Control Plane Units

• hub-web-api container (Next.js standalone runtime)
• hub-worker container (automation + billing jobs)
• notifications container (push/email delivery)
• postgres (managed or self-hosted HA)

3.2 Tenant Units (Per Customer VPS)

• openclaw container (upstream image/tag pinned)
• safety-wrapper plugin package mounted into OpenClaw extension dir
• egress-proxy service (localhost-only)
• tool containers and nginx from provisioner
• local SQLite data stores for secrets/approvals/metering

4. Provisioning Deployment Plan

4.1 Provisioner Mode

Continue with existing one-shot SSH provisioner flow, retooled to:

• deploy OpenClaw + Safety components
• remove legacy orchestrator/sysadmin deployment
• strip deprecated stacks and n8n references
• write secrets into encrypted vault only (no plaintext long-lived config)

4.2 Immutable Artifact Inputs

Provisioning uses pinned artifacts only:

• OpenClaw release tag (stable channel pin)
• Safety Wrapper image/package digest
• Tool stack compose templates with hash
• policy bundle version + checksum

5. Secrets And Credential Deployment

• Registration token is one-time and short-lived.
• Tenant API key returned at registration; only hash stored in Hub DB.
• Provisioner writes bootstrap secrets to tmpfs file, consumed once, then shredded.
• Existing plaintext job config path (jobs/<id>/config.json) replaced by encrypted payload + ephemeral decrypt-on-run.

6. Release Strategy

6.1 Control Plane

• Trunk-based merges behind feature flags.
• Deploy via Gitea Actions with staged promotions (dev -> staging -> prod).
• DB migrations run in expand/contract pattern.

6.2 Tenant Plane

Tenant updates split into independent channels:

• policy-only: classification/autonomy/tool policy updates (no binary change)
• wrapper patch: Safety Wrapper version bump
• openclaw bump: upstream release bump (separate tracked campaign)

Rollout:

1. Internal sandbox tenants
2. 5% canary customer tenants
3. 25%
4. 100%

Auto-stop criteria:

• redaction test failure
• approval-routing failure >1%
• tenant heartbeat drop >3%

7. Rollback Strategy

7.1 Control Plane Rollback

• Keep last two container digests deployable.
• Migration rollback policy: only for reversible migrations; otherwise hotfix-forward.

7.2 Tenant Rollback

• Policy rollback via previous signed policy bundle.
• Wrapper rollback to previous plugin package.
• OpenClaw rollback to previous pinned stable tag after compatibility check.

8. Observability And SLOs

8.1 Required Telemetry

• tenant heartbeat latency and freshness
• approval queue latency (request -> decision)
• redaction pipeline counters (matches by layer)
• token usage ingest lag
• provisioning success/failure per step

8.2 Launch SLO Targets

• Hub API availability: 99.9%
• Tenant heartbeat freshness: 99% under 2 minutes
• Approval propagation: p95 < 5 seconds (Hub to mobile push)
• Provisioning success first-attempt: >= 90%

9. Dual-Provider Strategy (Netcup + Hetzner)

• Primary capacity pool on Netcup (EU/US).
• Overflow path on Hetzner with same provisioner scripts and hardened baseline.
• Provider adapter abstraction lives in Hub server-provisioning module; provisioner remains Debian-focused and provider-agnostic.

10. Cutover Plan From Current State

1. Freeze legacy orchestrator/sysadmin deployment paths.
2. Land prerequisite cleanup release (n8n/deprecated removal + credential leak fix).
3. Enable new tenant register/heartbeat APIs in Hub.
4. Provision first new-architecture internal tenant.
5. Execute parallel-run window (old and new provisioning flows side-by-side for internal only).
6. Flip default provisioning to new flow for production orders.