LetsBeBiz-Redesign/docs/architecture-proposal/claude/08-CICD-STRATEGY.md

# LetsBe Biz — CI/CD Strategy

**Date:** February 27, 2026
**Team:** Claude Opus 4.6 Architecture Team
**Document:** 08 of 09
**Status:** Proposal — Competing with independent team

---

## Table of Contents

1. [CI/CD Overview](#1-cicd-overview)
2. [Gitea Actions Pipelines](#2-gitea-actions-pipelines)
3. [Branch Strategy](#3-branch-strategy)
4. [Build & Publish](#4-build--publish)
5. [Deployment Workflows](#5-deployment-workflows)
6. [Rollback Procedures](#6-rollback-procedures)
7. [Secret Management in CI](#7-secret-management-in-ci)
8. [Quality Gates in CI](#8-quality-gates-in-ci)
9. [Monitoring & Alerting](#9-monitoring--alerting)

---

## 1. CI/CD Overview

### Platform: Gitea Actions

Gitea Actions is the CI/CD platform (Architecture Brief §9.1). It uses GitHub Actions-compatible YAML workflow syntax, making migration straightforward if needed later.

### Pipeline Architecture

```
Developer pushes code
        │
        ▼
┌──────────────────┐
│  Gitea Actions    │
│  Trigger: push    │
│                   │
│  1. Lint          │
│  2. Type Check    │
│  3. Unit Tests    │
│  4. Build         │
│  5. Security Scan │
└────────┬─────────┘
         │
    ┌────┴────┐
    │ Branch? │
    └────┬────┘
         │
    ┌────┼────────────┐
    │    │             │
feature  develop      main
    │    │             │
    │    ▼             ▼
    │  Build Docker  Build Docker
    │  Push :dev     Push :latest
    │    │             │
    │    ▼             ▼
    │  Deploy to     Deploy to
    │  staging       production
    │                  │
    │                  ▼
    │              Canary rollout
    │              (tenant servers)
    │
    └─► PR required to merge
```

### Environments

| Environment | Branch | Trigger | Purpose |
|-------------|--------|---------|---------|
| **Local** | Any | Manual | Developer testing |
| **CI** | Any push | Automatic | Lint, test, type check |
| **Staging** | `develop` | Automatic on merge | Integration testing, dogfooding |
| **Production** | `main` | Manual approval | Live customers |

---

## 2. Gitea Actions Pipelines

### 2.1 Monorepo CI Pipeline (All Packages)

```yaml
# .gitea/workflows/ci.yml
name: CI

on:
  push:
    branches: [main, develop, 'feature/**']
  pull_request:
    branches: [main, develop]

env:
  NODE_VERSION: '22'

jobs:
  lint-and-typecheck:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}

      - name: Install dependencies
        run: npm ci

      - name: Lint
        run: npx turbo run lint

      - name: Type check
        run: npx turbo run typecheck

  unit-tests:
    runs-on: ubuntu-latest
    needs: lint-and-typecheck
    strategy:
      matrix:
        package:
          - safety-wrapper
          - secrets-proxy
          - hub
          - shared-types
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}

      - name: Install dependencies
        run: npm ci

      - name: Run tests for ${{ matrix.package }}
        run: npx turbo run test --filter=${{ matrix.package }}

  security-scan:
    runs-on: ubuntu-latest
    needs: lint-and-typecheck
    steps:
      - uses: actions/checkout@v4

      - name: Check for secrets in code
        run: |
          npx @trufflesecurity/trufflehog git file://. --only-verified --fail

      - name: Dependency audit
        run: npm audit --audit-level=high
```

### 2.2 Safety Wrapper Pipeline

```yaml
# .gitea/workflows/safety-wrapper.yml
name: Safety Wrapper

on:
  push:
    paths:
      - 'packages/safety-wrapper/**'
      - 'packages/shared-types/**'
    branches: [main, develop]

jobs:
  p0-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '22'

      - run: npm ci

      - name: P0 Secrets Redaction Tests
        run: npx turbo run test:p0 --filter=secrets-proxy

      - name: P0 Command Classification Tests
        run: npx turbo run test:p0 --filter=safety-wrapper

      - name: P1 Autonomy Tests
        run: npx turbo run test:p1 --filter=safety-wrapper

  build-image:
    runs-on: ubuntu-latest
    needs: p0-tests
    if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/develop'
    steps:
      - uses: actions/checkout@v4

      - name: Set tag
        id: tag
        run: |
          if [ "${{ github.ref }}" = "refs/heads/main" ]; then
            echo "tag=latest" >> $GITHUB_OUTPUT
          else
            echo "tag=dev" >> $GITHUB_OUTPUT
          fi

      - name: Build Safety Wrapper image
        run: |
          docker build \
            -f packages/safety-wrapper/Dockerfile \
            -t code.letsbe.solutions/letsbe/safety-wrapper:${{ steps.tag.outputs.tag }} \
            -t code.letsbe.solutions/letsbe/safety-wrapper:${{ github.sha }} \
            .

      - name: Push to registry
        run: |
          echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login code.letsbe.solutions -u ${{ secrets.REGISTRY_USER }} --password-stdin
          docker push code.letsbe.solutions/letsbe/safety-wrapper:${{ steps.tag.outputs.tag }}
          docker push code.letsbe.solutions/letsbe/safety-wrapper:${{ github.sha }}
```

### 2.3 Secrets Proxy Pipeline

```yaml
# .gitea/workflows/secrets-proxy.yml
name: Secrets Proxy

on:
  push:
    paths:
      - 'packages/secrets-proxy/**'
      - 'packages/shared-types/**'
    branches: [main, develop]

jobs:
  p0-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '22' }
      - run: npm ci

      - name: P0 Redaction Tests (must pass 100%)
        run: npx turbo run test:p0 --filter=secrets-proxy

      - name: Performance Benchmark
        run: npx turbo run test:benchmark --filter=secrets-proxy

  build-image:
    runs-on: ubuntu-latest
    needs: p0-tests
    if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/develop'
    steps:
      - uses: actions/checkout@v4

      - name: Build Secrets Proxy image
        run: |
          docker build \
            -f packages/secrets-proxy/Dockerfile \
            -t code.letsbe.solutions/letsbe/secrets-proxy:${{ github.ref == 'refs/heads/main' && 'latest' || 'dev' }} \
            .

      - name: Push to registry
        run: |
          echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login code.letsbe.solutions -u ${{ secrets.REGISTRY_USER }} --password-stdin
          docker push code.letsbe.solutions/letsbe/secrets-proxy --all-tags
```

### 2.4 Hub Pipeline

```yaml
# .gitea/workflows/hub.yml
name: Hub

on:
  push:
    paths:
      - 'packages/hub/**'
      - 'packages/shared-prisma/**'
    branches: [main, develop]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16-alpine
        env:
          POSTGRES_DB: hub_test
          POSTGRES_USER: hub
          POSTGRES_PASSWORD: testpass
        ports: ['5432:5432']
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '22' }
      - run: npm ci

      - name: Run Prisma migrations
        run: npx turbo run db:push --filter=hub
        env:
          DATABASE_URL: postgresql://hub:testpass@localhost:5432/hub_test

      - name: Run tests
        run: npx turbo run test --filter=hub
        env:
          DATABASE_URL: postgresql://hub:testpass@localhost:5432/hub_test

  build-image:
    runs-on: ubuntu-latest
    needs: test
    if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/develop'
    steps:
      - uses: actions/checkout@v4
      - name: Build Hub image
        run: |
          docker build \
            -f packages/hub/Dockerfile \
            -t code.letsbe.solutions/letsbe/hub:${{ github.ref == 'refs/heads/main' && 'latest' || 'dev' }} \
            .
      - name: Push to registry
        run: |
          echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login code.letsbe.solutions -u ${{ secrets.REGISTRY_USER }} --password-stdin
          docker push code.letsbe.solutions/letsbe/hub --all-tags
```

### 2.5 Integration Test Pipeline

```yaml
# .gitea/workflows/integration.yml
name: Integration Tests

on:
  push:
    branches: [develop]
  workflow_dispatch:

jobs:
  integration:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '22' }
      - run: npm ci

      - name: Start integration stack
        run: docker compose -f test/docker-compose.integration.yml up -d --wait
        timeout-minutes: 5

      - name: Wait for services
        run: |
          for i in $(seq 1 30); do
            curl -sf http://localhost:8200/health && break || sleep 2
          done

      - name: Run integration tests
        run: npx turbo run test:integration

      - name: Collect logs on failure
        if: failure()
        run: docker compose -f test/docker-compose.integration.yml logs > integration-logs.txt

      - name: Upload logs
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: integration-logs
          path: integration-logs.txt

      - name: Teardown
        if: always()
        run: docker compose -f test/docker-compose.integration.yml down -v
```

---

## 3. Branch Strategy

### Git Flow (Simplified)

```
main ─────────────────────────────────────────────────►
  │                                        ▲
  │                                        │ (merge via PR, requires approval)
  │                                        │
develop ──┬───────────┬───────────┬────────┤
           │           │           │
  feature/sw-skeleton  │  feature/hub-billing
           │           │
           │  feature/secrets-proxy
           │
  hotfix/critical-fix ──────────────────────► main (direct merge for critical fixes)
```

### Branch Rules

| Branch | Protection | Merge Requirements |
|--------|-----------|-------------------|
| `main` | Protected; no direct pushes | PR from `develop`; 1 approval; all CI checks pass; security scan pass |
| `develop` | Protected; no direct pushes | PR from feature branch; all CI checks pass |
| `feature/*` | Unprotected | Free to push; PR to develop when ready |
| `hotfix/*` | Unprotected | Can merge to both `main` and `develop`; 1 approval required |

### Naming Conventions

```
feature/sw-command-classification    # Safety Wrapper feature
feature/hub-tenant-api               # Hub feature
feature/mobile-chat-view             # Mobile app feature
feature/prov-step10-rewrite          # Provisioner feature
fix/secrets-proxy-jwt-detection      # Bug fix
hotfix/redaction-bypass-cve          # Critical security fix
```

### Release Tagging

```
v0.1.0   # First internal milestone (M1)
v0.2.0   # M2
v0.3.0   # M3
v1.0.0   # Founding member launch (M4)
v1.0.1   # First patch
v1.1.0   # First feature update post-launch
```

---

## 4. Build & Publish

### Docker Image Strategy

| Image | Registry Path | Build Context | Size Target |
|-------|--------------|---------------|-------------|
| `letsbe/safety-wrapper` | `code.letsbe.solutions/letsbe/safety-wrapper` | `packages/safety-wrapper/` | <150MB |
| `letsbe/secrets-proxy` | `code.letsbe.solutions/letsbe/secrets-proxy` | `packages/secrets-proxy/` | <100MB |
| `letsbe/hub` | `code.letsbe.solutions/letsbe/hub` | `packages/hub/` | <500MB |
| `letsbe/ansible-runner` | `code.letsbe.solutions/letsbe/ansible-runner` | `packages/provisioner/` | Existing |

### Multi-Stage Dockerfile Pattern

```dockerfile
# packages/safety-wrapper/Dockerfile
# Stage 1: Dependencies
FROM node:22-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./
COPY packages/safety-wrapper/package.json ./packages/safety-wrapper/
COPY packages/shared-types/package.json ./packages/shared-types/
RUN npm ci --workspace=packages/safety-wrapper --workspace=packages/shared-types

# Stage 2: Build
FROM node:22-alpine AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY packages/safety-wrapper/ ./packages/safety-wrapper/
COPY packages/shared-types/ ./packages/shared-types/
COPY turbo.json package.json ./
RUN npx turbo run build --filter=safety-wrapper

# Stage 3: Production
FROM node:22-alpine AS runner
WORKDIR /app
RUN addgroup -g 1001 -S letsbe && adduser -S letsbe -u 1001
COPY --from=builder /app/packages/safety-wrapper/dist ./dist
COPY --from=builder /app/packages/safety-wrapper/package.json ./
COPY --from=deps /app/node_modules ./node_modules
USER letsbe
EXPOSE 8200
CMD ["node", "dist/index.js"]
```

### Image Tagging

| Tag | When | Purpose |
|-----|------|---------|
| `:dev` | On merge to `develop` | Staging deployment |
| `:latest` | On merge to `main` | Production deployment |
| `:<git-sha>` | On every build | Immutable reference for debugging |
| `:v1.0.0` | On release tag | Version-pinned deployment |

---

## 5. Deployment Workflows

### 5.1 Central Platform (Hub) Deployment

```yaml
# .gitea/workflows/deploy-hub.yml
name: Deploy Hub

on:
  push:
    branches: [main]
    paths: ['packages/hub/**', 'packages/shared-prisma/**']

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Deploy to production
        run: |
          ssh -o StrictHostKeyChecking=no deploy@hub.letsbe.biz << 'EOF'
            cd /opt/letsbe/hub
            docker compose pull hub
            docker compose up -d hub
            # Wait for health check
            for i in $(seq 1 30); do
              curl -sf http://localhost:3847/api/health && break || sleep 2
            done
            # Run migrations
            docker compose exec hub npx prisma migrate deploy
          EOF
```

### 5.2 Tenant Server Update Pipeline

Tenant servers are updated via the Hub push mechanism (see 03-DEPLOYMENT-STRATEGY §7).

```yaml
# .gitea/workflows/tenant-update.yml
name: Tenant Server Update

on:
  workflow_dispatch:
    inputs:
      component:
        description: 'Component to update'
        required: true
        type: choice
        options: [safety-wrapper, secrets-proxy, openclaw]
      strategy:
        description: 'Rollout strategy'
        required: true
        type: choice
        options: [staging-only, canary-5pct, canary-25pct, full-rollout]

jobs:
  prepare:
    runs-on: ubuntu-latest
    steps:
      - name: Verify image exists
        run: |
          docker manifest inspect code.letsbe.solutions/letsbe/${{ inputs.component }}:latest

  rollout:
    runs-on: ubuntu-latest
    needs: prepare
    steps:
      - name: Trigger Hub rollout API
        run: |
          curl -X POST https://hub.letsbe.biz/api/v1/admin/rollout \
            -H "Authorization: Bearer ${{ secrets.HUB_ADMIN_TOKEN }}" \
            -H "Content-Type: application/json" \
            -d '{
              "component": "${{ inputs.component }}",
              "tag": "latest",
              "strategy": "${{ inputs.strategy }}"
            }'
```

### 5.3 Staging Deployment (Automatic)

```yaml
# .gitea/workflows/deploy-staging.yml
name: Deploy Staging

on:
  push:
    branches: [develop]

jobs:
  deploy-staging:
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - name: Deploy Hub to staging
        run: |
          ssh deploy@staging.letsbe.biz << 'EOF'
            cd /opt/letsbe/hub
            docker compose pull
            docker compose up -d
            docker compose exec hub npx prisma migrate deploy
          EOF

      - name: Deploy tenant stack to staging VPS
        run: |
          ssh deploy@staging-tenant.letsbe.biz << 'EOF'
            cd /opt/letsbe
            docker compose -f docker-compose.letsbe.yml pull
            docker compose -f docker-compose.letsbe.yml up -d
          EOF

      - name: Run smoke tests
        run: |
          curl -sf https://staging.letsbe.biz/api/health
          curl -sf https://staging-tenant.letsbe.biz:8200/health
          curl -sf https://staging-tenant.letsbe.biz:8100/health
```

---

## 6. Rollback Procedures

### 6.1 Hub Rollback

```bash
# Rollback Hub to previous version
ssh deploy@hub.letsbe.biz << 'EOF'
  cd /opt/letsbe/hub

  # Find previous image
  PREVIOUS=$(docker compose images hub --format '{{.Tag}}' | head -1)

  # Pull and deploy previous
  docker compose pull hub  # Uses previous :latest from registry
  docker compose up -d hub

  # Verify health
  for i in $(seq 1 30); do
    curl -sf http://localhost:3847/api/health && break || sleep 2
  done

  # Note: Prisma migrations are forward-only.
  # If a migration needs reverting, use prisma migrate resolve.
EOF
```

### 6.2 Tenant Component Rollback

```bash
# Rollback Safety Wrapper on a specific tenant
ssh deploy@tenant-server << 'EOF'
  cd /opt/letsbe

  # Roll back to pinned SHA
  docker compose -f docker-compose.letsbe.yml \
    -e SAFETY_WRAPPER_TAG=<previous-sha> \
    up -d safety-wrapper

  # Verify health
  curl -sf http://127.0.0.1:8200/health
EOF
```

### 6.3 Rollback Decision Matrix

| Symptom | Action | Automatic? |
|---------|--------|-----------|
| Health check fails after deploy | Rollback to previous image | Yes (Docker restart policy pulls previous on repeated failure) |
| P0 tests fail in CI | Block merge; no deployment | Yes (CI gate) |
| Secrets redaction miss detected | EMERGENCY: rollback all tenants immediately | Manual (requires admin trigger) |
| Hub API errors >5% | Rollback Hub to previous version | Manual (monitoring alert) |
| Billing discrepancy | Investigate first; rollback billing code if confirmed | Manual |

### 6.4 Emergency Rollback Checklist

For critical security issues (e.g., redaction bypass):

1. **STOP** all tenant updates immediately (disable Hub rollout API)
2. **ROLLBACK** all affected components to last known-good version
3. **VERIFY** rollback successful (health checks, P0 tests)
4. **INVESTIGATE** root cause
5. **FIX** and add test case for the specific failure
6. **AUDIT** all tenants for potential exposure during the window
7. **NOTIFY** affected customers if secrets were potentially exposed
8. **POST-MORTEM** within 24 hours

---

## 7. Secret Management in CI

### Gitea Secrets Configuration

| Secret | Scope | Purpose |
|--------|-------|---------|
| `REGISTRY_USER` | Organization | Docker registry login |
| `REGISTRY_PASSWORD` | Organization | Docker registry password |
| `HUB_ADMIN_TOKEN` | Repository | Hub API authentication for deployments |
| `STAGING_SSH_KEY` | Repository | SSH key for staging deployment |
| `PRODUCTION_SSH_KEY` | Repository | SSH key for production deployment |
| `STRIPE_TEST_KEY` | Repository | Stripe test mode for integration tests |

### Rules

1. **Never** put secrets in workflow YAML files
2. **Never** echo secrets in CI logs (use `::add-mask::`)
3. **Never** pass secrets as command-line arguments (use environment variables)
4. SSH keys: use deploy keys with minimal permissions (read-only for CI, write for deploy)
5. Rotate all CI secrets quarterly

---

## 8. Quality Gates in CI

### Gate Configuration

```yaml
# In each pipeline, quality gates are enforced as job dependencies:

jobs:
  # Gate 1: Code quality
  lint:
    # Must pass before tests run
    ...

  typecheck:
    # Must pass before tests run
    ...

  # Gate 2: Correctness
  unit-tests:
    needs: [lint, typecheck]
    # Must pass before build
    ...

  # Gate 3: Security
  security-scan:
    needs: [lint]
    # Must pass before deploy
    ...

  # Gate 4: Build
  build:
    needs: [unit-tests, security-scan]
    # Must succeed before deploy
    ...

  # Gate 5: Deploy (only on protected branches)
  deploy:
    needs: [build]
    if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/develop'
    ...
```

### PR Merge Requirements

| Requirement | Enforcement |
|-------------|------------|
| All CI checks pass | Gitea branch protection rule |
| At least 1 approval | Gitea branch protection rule |
| No unresolved review comments | Convention (not enforced by Gitea) |
| P0 tests pass if security code changed | CI pipeline condition |
| No secrets detected in diff | trufflehog scan |

---

## 9. Monitoring & Alerting

### CI Pipeline Monitoring

| Metric | Alert Threshold | Action |
|--------|----------------|--------|
| Build duration | >15 min | Investigate; optimize caching |
| Test suite duration | >10 min | Investigate; parallelize tests |
| Failed builds on `develop` | >3 consecutive | Freeze merges; investigate |
| Failed deploys | Any | Automatic rollback; notify team |
| Security scan findings | Any critical | Block merge; assign to Security Lead |

### Deployment Monitoring

| Metric | Alert Threshold | Action |
|--------|----------------|--------|
| Hub health after deploy | Unhealthy for >60s | Automatic rollback |
| Tenant health after update | Unhealthy for >120s | Rollback specific tenant; pause rollout |
| Error rate post-deploy | >5% increase | Alert team; investigate |
| Latency post-deploy | >2× baseline | Alert team; investigate |

### Notification Channels

| Event | Channel |
|-------|---------|
| CI failure on `main` | Team chat (immediate) |
| Security scan finding | Team chat + email to Security Lead |
| Deployment success | Team chat (informational) |
| Deployment failure | Team chat + email to on-call |
| Emergency rollback | Team chat + phone call to on-call |

---

*End of Document — 08 CI/CD Strategy*