Include full contents of all nested repositories

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-02-27 16:25:02 +01:00
parent 14ff8fd54c
commit 2401ed446f
7271 changed files with 1310112 additions and 6 deletions

View File

@@ -0,0 +1,162 @@
---
summary: "Debugging tools: watch mode, raw model streams, and tracing reasoning leakage"
read_when:
- You need to inspect raw model output for reasoning leakage
- You want to run the Gateway in watch mode while iterating
- You need a repeatable debugging workflow
title: "Debugging"
---
# Debugging
This page covers debugging helpers for streaming output, especially when a
provider mixes reasoning into normal text.
## Runtime debug overrides
Use `/debug` in chat to set **runtime-only** config overrides (memory, not disk).
`/debug` is disabled by default; enable with `commands.debug: true`.
This is handy when you need to toggle obscure settings without editing `openclaw.json`.
Examples:
```
/debug show
/debug set messages.responsePrefix="[openclaw]"
/debug unset messages.responsePrefix
/debug reset
```
`/debug reset` clears all overrides and returns to the on-disk config.
## Gateway watch mode
For fast iteration, run the gateway under the file watcher:
```bash
pnpm gateway:watch
```
This maps to:
```bash
node --watch-path src --watch-path tsconfig.json --watch-path package.json --watch-preserve-output scripts/run-node.mjs gateway --force
```
Add any gateway CLI flags after `gateway:watch` and they will be passed through
on each restart.
## Dev profile + dev gateway (--dev)
Use the dev profile to isolate state and spin up a safe, disposable setup for
debugging. There are **two** `--dev` flags:
- **Global `--dev` (profile):** isolates state under `~/.openclaw-dev` and
defaults the gateway port to `19001` (derived ports shift with it).
- **`gateway --dev`: tells the Gateway to auto-create a default config +
workspace** when missing (and skip BOOTSTRAP.md).
Recommended flow (dev profile + dev bootstrap):
```bash
pnpm gateway:dev
OPENCLAW_PROFILE=dev openclaw tui
```
If you dont have a global install yet, run the CLI via `pnpm openclaw ...`.
What this does:
1. **Profile isolation** (global `--dev`)
- `OPENCLAW_PROFILE=dev`
- `OPENCLAW_STATE_DIR=~/.openclaw-dev`
- `OPENCLAW_CONFIG_PATH=~/.openclaw-dev/openclaw.json`
- `OPENCLAW_GATEWAY_PORT=19001` (browser/canvas shift accordingly)
2. **Dev bootstrap** (`gateway --dev`)
- Writes a minimal config if missing (`gateway.mode=local`, bind loopback).
- Sets `agent.workspace` to the dev workspace.
- Sets `agent.skipBootstrap=true` (no BOOTSTRAP.md).
- Seeds the workspace files if missing:
`AGENTS.md`, `SOUL.md`, `TOOLS.md`, `IDENTITY.md`, `USER.md`, `HEARTBEAT.md`.
- Default identity: **C3PO** (protocol droid).
- Skips channel providers in dev mode (`OPENCLAW_SKIP_CHANNELS=1`).
Reset flow (fresh start):
```bash
pnpm gateway:dev:reset
```
Note: `--dev` is a **global** profile flag and gets eaten by some runners.
If you need to spell it out, use the env var form:
```bash
OPENCLAW_PROFILE=dev openclaw gateway --dev --reset
```
`--reset` wipes config, credentials, sessions, and the dev workspace (using
`trash`, not `rm`), then recreates the default dev setup.
Tip: if a nondev gateway is already running (launchd/systemd), stop it first:
```bash
openclaw gateway stop
```
## Raw stream logging (OpenClaw)
OpenClaw can log the **raw assistant stream** before any filtering/formatting.
This is the best way to see whether reasoning is arriving as plain text deltas
(or as separate thinking blocks).
Enable it via CLI:
```bash
pnpm gateway:watch --raw-stream
```
Optional path override:
```bash
pnpm gateway:watch --raw-stream --raw-stream-path ~/.openclaw/logs/raw-stream.jsonl
```
Equivalent env vars:
```bash
OPENCLAW_RAW_STREAM=1
OPENCLAW_RAW_STREAM_PATH=~/.openclaw/logs/raw-stream.jsonl
```
Default file:
`~/.openclaw/logs/raw-stream.jsonl`
## Raw chunk logging (pi-mono)
To capture **raw OpenAI-compat chunks** before they are parsed into blocks,
pi-mono exposes a separate logger:
```bash
PI_RAW_STREAM=1
```
Optional path:
```bash
PI_RAW_STREAM_PATH=~/.pi-mono/logs/raw-openai-completions.jsonl
```
Default file:
`~/.pi-mono/logs/raw-openai-completions.jsonl`
> Note: this is only emitted by processes using pi-monos
> `openai-completions` provider.
## Safety notes
- Raw stream logs can include full prompts, tool output, and user data.
- Keep logs local and delete them after debugging.
- If you share logs, scrub secrets and PII first.

View File

@@ -0,0 +1,122 @@
---
summary: "Where OpenClaw loads environment variables and the precedence order"
read_when:
- You need to know which env vars are loaded, and in what order
- You are debugging missing API keys in the Gateway
- You are documenting provider auth or deployment environments
title: "Environment Variables"
---
# Environment variables
OpenClaw pulls environment variables from multiple sources. The rule is **never override existing values**.
## Precedence (highest → lowest)
1. **Process environment** (what the Gateway process already has from the parent shell/daemon).
2. **`.env` in the current working directory** (dotenv default; does not override).
3. **Global `.env`** at `~/.openclaw/.env` (aka `$OPENCLAW_STATE_DIR/.env`; does not override).
4. **Config `env` block** in `~/.openclaw/openclaw.json` (applied only if missing).
5. **Optional login-shell import** (`env.shellEnv.enabled` or `OPENCLAW_LOAD_SHELL_ENV=1`), applied only for missing expected keys.
If the config file is missing entirely, step 4 is skipped; shell import still runs if enabled.
## Config `env` block
Two equivalent ways to set inline env vars (both are non-overriding):
```json5
{
env: {
OPENROUTER_API_KEY: "sk-or-...",
vars: {
GROQ_API_KEY: "gsk-...",
},
},
}
```
## Shell env import
`env.shellEnv` runs your login shell and imports only **missing** expected keys:
```json5
{
env: {
shellEnv: {
enabled: true,
timeoutMs: 15000,
},
},
}
```
Env var equivalents:
- `OPENCLAW_LOAD_SHELL_ENV=1`
- `OPENCLAW_SHELL_ENV_TIMEOUT_MS=15000`
## Env var substitution in config
You can reference env vars directly in config string values using `${VAR_NAME}` syntax:
```json5
{
models: {
providers: {
"vercel-gateway": {
apiKey: "${VERCEL_GATEWAY_API_KEY}",
},
},
},
}
```
See [Configuration: Env var substitution](/gateway/configuration#env-var-substitution-in-config) for full details.
## Secret refs vs `${ENV}` strings
OpenClaw supports two env-driven patterns:
- `${VAR}` string substitution in config values.
- SecretRef objects (`{ source: "env", provider: "default", id: "VAR" }`) for fields that support secrets references.
Both resolve from process env at activation time. SecretRef details are documented in [Secrets Management](/gateway/secrets).
## Path-related env vars
| Variable | Purpose |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `OPENCLAW_HOME` | Override the home directory used for all internal path resolution (`~/.openclaw/`, agent dirs, sessions, credentials). Useful when running OpenClaw as a dedicated service user. |
| `OPENCLAW_STATE_DIR` | Override the state directory (default `~/.openclaw`). |
| `OPENCLAW_CONFIG_PATH` | Override the config file path (default `~/.openclaw/openclaw.json`). |
## Logging
| Variable | Purpose |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `OPENCLAW_LOG_LEVEL` | Override log level for both file and console (e.g. `debug`, `trace`). Takes precedence over `logging.level` and `logging.consoleLevel` in config. Invalid values are ignored with a warning. |
### `OPENCLAW_HOME`
When set, `OPENCLAW_HOME` replaces the system home directory (`$HOME` / `os.homedir()`) for all internal path resolution. This enables full filesystem isolation for headless service accounts.
**Precedence:** `OPENCLAW_HOME` > `$HOME` > `USERPROFILE` > `os.homedir()`
**Example** (macOS LaunchDaemon):
```xml
<key>EnvironmentVariables</key>
<dict>
<key>OPENCLAW_HOME</key>
<string>/Users/kira</string>
</dict>
```
`OPENCLAW_HOME` can also be set to a tilde path (e.g. `~/svc`), which gets expanded using `$HOME` before use.
## Related
- [Gateway configuration](/gateway/configuration)
- [FAQ: env vars and .env loading](/help/faq#env-vars-and-env-loading)
- [Models overview](/concepts/models)

2898
openclaw/docs/help/faq.md Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,21 @@
---
summary: "Help hub: common fixes, install sanity, and where to look when something breaks"
read_when:
- Youre new and want the “what do I click/run” guide
- Something broke and you want the fastest path to a fix
title: "Help"
---
# Help
If you want a quick “get unstuck” flow, start here:
- **Troubleshooting:** [Start here](/help/troubleshooting)
- **Install sanity (Node/npm/PATH):** [Install](/install#nodejs--npm-path-sanity)
- **Gateway issues:** [Gateway troubleshooting](/gateway/troubleshooting)
- **Logs:** [Logging](/logging) and [Gateway logging](/gateway/logging)
- **Repairs:** [Doctor](/gateway/doctor)
If youre looking for conceptual questions (not “something broke”):
- [FAQ (concepts)](/help/faq)

View File

@@ -0,0 +1,28 @@
---
summary: "Repository scripts: purpose, scope, and safety notes"
read_when:
- Running scripts from the repo
- Adding or changing scripts under ./scripts
title: "Scripts"
---
# Scripts
The `scripts/` directory contains helper scripts for local workflows and ops tasks.
Use these when a task is clearly tied to a script; otherwise prefer the CLI.
## Conventions
- Scripts are **optional** unless referenced in docs or release checklists.
- Prefer CLI surfaces when they exist (example: auth monitoring uses `openclaw models status --check`).
- Assume scripts are hostspecific; read them before running on a new machine.
## Auth monitoring scripts
Auth monitoring scripts are documented here:
[/automation/auth-monitoring](/automation/auth-monitoring)
## When adding scripts
- Keep scripts focused and documented.
- Add a short entry in the relevant doc (or create one if missing).

View File

@@ -0,0 +1,407 @@
---
summary: "Testing kit: unit/e2e/live suites, Docker runners, and what each test covers"
read_when:
- Running tests locally or in CI
- Adding regressions for model/provider bugs
- Debugging gateway + agent behavior
title: "Testing"
---
# Testing
OpenClaw has three Vitest suites (unit/integration, e2e, live) and a small set of Docker runners.
This doc is a “how we test” guide:
- What each suite covers (and what it deliberately does _not_ cover)
- Which commands to run for common workflows (local, pre-push, debugging)
- How live tests discover credentials and select models/providers
- How to add regressions for real-world model/provider issues
## Quick start
Most days:
- Full gate (expected before push): `pnpm build && pnpm check && pnpm test`
When you touch tests or want extra confidence:
- Coverage gate: `pnpm test:coverage`
- E2E suite: `pnpm test:e2e`
When debugging real providers/models (requires real creds):
- Live suite (models + gateway tool/image probes): `pnpm test:live`
Tip: when you only need one failing case, prefer narrowing live tests via the allowlist env vars described below.
## Test suites (what runs where)
Think of the suites as “increasing realism” (and increasing flakiness/cost):
### Unit / integration (default)
- Command: `pnpm test`
- Config: `scripts/test-parallel.mjs` (runs `vitest.unit.config.ts`, `vitest.extensions.config.ts`, `vitest.gateway.config.ts`)
- Files: `src/**/*.test.ts`, `extensions/**/*.test.ts`
- Scope:
- Pure unit tests
- In-process integration tests (gateway auth, routing, tooling, parsing, config)
- Deterministic regressions for known bugs
- Expectations:
- Runs in CI
- No real keys required
- Should be fast and stable
- Pool note:
- OpenClaw uses Vitest `vmForks` on Node 22/23 for faster unit shards.
- On Node 24+, OpenClaw automatically falls back to regular `forks` to avoid Node VM linking errors (`ERR_VM_MODULE_LINK_FAILURE` / `module is already linked`).
- Override manually with `OPENCLAW_TEST_VM_FORKS=0` (force `forks`) or `OPENCLAW_TEST_VM_FORKS=1` (force `vmForks`).
### E2E (gateway smoke)
- Command: `pnpm test:e2e`
- Config: `vitest.e2e.config.ts`
- Files: `src/**/*.e2e.test.ts`
- Runtime defaults:
- Uses Vitest `vmForks` for faster file startup.
- Uses adaptive workers (CI: 2-4, local: 4-8).
- Runs in silent mode by default to reduce console I/O overhead.
- Useful overrides:
- `OPENCLAW_E2E_WORKERS=<n>` to force worker count (capped at 16).
- `OPENCLAW_E2E_VERBOSE=1` to re-enable verbose console output.
- Scope:
- Multi-instance gateway end-to-end behavior
- WebSocket/HTTP surfaces, node pairing, and heavier networking
- Expectations:
- Runs in CI (when enabled in the pipeline)
- No real keys required
- More moving parts than unit tests (can be slower)
### Live (real providers + real models)
- Command: `pnpm test:live`
- Config: `vitest.live.config.ts`
- Files: `src/**/*.live.test.ts`
- Default: **enabled** by `pnpm test:live` (sets `OPENCLAW_LIVE_TEST=1`)
- Scope:
- “Does this provider/model actually work _today_ with real creds?”
- Catch provider format changes, tool-calling quirks, auth issues, and rate limit behavior
- Expectations:
- Not CI-stable by design (real networks, real provider policies, quotas, outages)
- Costs money / uses rate limits
- Prefer running narrowed subsets instead of “everything”
- Live runs will source `~/.profile` to pick up missing API keys
- API key rotation (provider-specific): set `*_API_KEYS` with comma/semicolon format or `*_API_KEY_1`, `*_API_KEY_2` (for example `OPENAI_API_KEYS`, `ANTHROPIC_API_KEYS`, `GEMINI_API_KEYS`) or per-live override via `OPENCLAW_LIVE_*_KEY`; tests retry on rate limit responses.
## Which suite should I run?
Use this decision table:
- Editing logic/tests: run `pnpm test` (and `pnpm test:coverage` if you changed a lot)
- Touching gateway networking / WS protocol / pairing: add `pnpm test:e2e`
- Debugging “my bot is down” / provider-specific failures / tool calling: run a narrowed `pnpm test:live`
## Live: Android node capability sweep
- Test: `src/gateway/android-node.capabilities.live.test.ts`
- Script: `pnpm android:test:integration`
- Goal: invoke **every command currently advertised** by a connected Android node and assert command contract behavior.
- Scope:
- Preconditioned/manual setup (the suite does not install/run/pair the app).
- Command-by-command gateway `node.invoke` validation for the selected Android node.
- Required pre-setup:
- Android app already connected + paired to the gateway.
- App kept in foreground.
- Permissions/capture consent granted for capabilities you expect to pass.
- Optional target overrides:
- `OPENCLAW_ANDROID_NODE_ID` or `OPENCLAW_ANDROID_NODE_NAME`.
- `OPENCLAW_ANDROID_GATEWAY_URL` / `OPENCLAW_ANDROID_GATEWAY_TOKEN` / `OPENCLAW_ANDROID_GATEWAY_PASSWORD`.
- Full Android setup details: [Android App](/platforms/android)
## Live: model smoke (profile keys)
Live tests are split into two layers so we can isolate failures:
- “Direct model” tells us the provider/model can answer at all with the given key.
- “Gateway smoke” tells us the full gateway+agent pipeline works for that model (sessions, history, tools, sandbox policy, etc.).
### Layer 1: Direct model completion (no gateway)
- Test: `src/agents/models.profiles.live.test.ts`
- Goal:
- Enumerate discovered models
- Use `getApiKeyForModel` to select models you have creds for
- Run a small completion per model (and targeted regressions where needed)
- How to enable:
- `pnpm test:live` (or `OPENCLAW_LIVE_TEST=1` if invoking Vitest directly)
- Set `OPENCLAW_LIVE_MODELS=modern` (or `all`, alias for modern) to actually run this suite; otherwise it skips to keep `pnpm test:live` focused on gateway smoke
- How to select models:
- `OPENCLAW_LIVE_MODELS=modern` to run the modern allowlist (Opus/Sonnet/Haiku 4.5, GPT-5.x + Codex, Gemini 3, GLM 4.7, MiniMax M2.1, Grok 4)
- `OPENCLAW_LIVE_MODELS=all` is an alias for the modern allowlist
- or `OPENCLAW_LIVE_MODELS="openai/gpt-5.2,anthropic/claude-opus-4-6,..."` (comma allowlist)
- How to select providers:
- `OPENCLAW_LIVE_PROVIDERS="google,google-antigravity,google-gemini-cli"` (comma allowlist)
- Where keys come from:
- By default: profile store and env fallbacks
- Set `OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1` to enforce **profile store** only
- Why this exists:
- Separates “provider API is broken / key is invalid” from “gateway agent pipeline is broken”
- Contains small, isolated regressions (example: OpenAI Responses/Codex Responses reasoning replay + tool-call flows)
### Layer 2: Gateway + dev agent smoke (what “@openclaw” actually does)
- Test: `src/gateway/gateway-models.profiles.live.test.ts`
- Goal:
- Spin up an in-process gateway
- Create/patch a `agent:dev:*` session (model override per run)
- Iterate models-with-keys and assert:
- “meaningful” response (no tools)
- a real tool invocation works (read probe)
- optional extra tool probes (exec+read probe)
- OpenAI regression paths (tool-call-only → follow-up) keep working
- Probe details (so you can explain failures quickly):
- `read` probe: the test writes a nonce file in the workspace and asks the agent to `read` it and echo the nonce back.
- `exec+read` probe: the test asks the agent to `exec`-write a nonce into a temp file, then `read` it back.
- image probe: the test attaches a generated PNG (cat + randomized code) and expects the model to return `cat <CODE>`.
- Implementation reference: `src/gateway/gateway-models.profiles.live.test.ts` and `src/gateway/live-image-probe.ts`.
- How to enable:
- `pnpm test:live` (or `OPENCLAW_LIVE_TEST=1` if invoking Vitest directly)
- How to select models:
- Default: modern allowlist (Opus/Sonnet/Haiku 4.5, GPT-5.x + Codex, Gemini 3, GLM 4.7, MiniMax M2.1, Grok 4)
- `OPENCLAW_LIVE_GATEWAY_MODELS=all` is an alias for the modern allowlist
- Or set `OPENCLAW_LIVE_GATEWAY_MODELS="provider/model"` (or comma list) to narrow
- How to select providers (avoid “OpenRouter everything”):
- `OPENCLAW_LIVE_GATEWAY_PROVIDERS="google,google-antigravity,google-gemini-cli,openai,anthropic,zai,minimax"` (comma allowlist)
- Tool + image probes are always on in this live test:
- `read` probe + `exec+read` probe (tool stress)
- image probe runs when the model advertises image input support
- Flow (high level):
- Test generates a tiny PNG with “CAT” + random code (`src/gateway/live-image-probe.ts`)
- Sends it via `agent` `attachments: [{ mimeType: "image/png", content: "<base64>" }]`
- Gateway parses attachments into `images[]` (`src/gateway/server-methods/agent.ts` + `src/gateway/chat-attachments.ts`)
- Embedded agent forwards a multimodal user message to the model
- Assertion: reply contains `cat` + the code (OCR tolerance: minor mistakes allowed)
Tip: to see what you can test on your machine (and the exact `provider/model` ids), run:
```bash
openclaw models list
openclaw models list --json
```
## Live: Anthropic setup-token smoke
- Test: `src/agents/anthropic.setup-token.live.test.ts`
- Goal: verify Claude Code CLI setup-token (or a pasted setup-token profile) can complete an Anthropic prompt.
- Enable:
- `pnpm test:live` (or `OPENCLAW_LIVE_TEST=1` if invoking Vitest directly)
- `OPENCLAW_LIVE_SETUP_TOKEN=1`
- Token sources (pick one):
- Profile: `OPENCLAW_LIVE_SETUP_TOKEN_PROFILE=anthropic:setup-token-test`
- Raw token: `OPENCLAW_LIVE_SETUP_TOKEN_VALUE=sk-ant-oat01-...`
- Model override (optional):
- `OPENCLAW_LIVE_SETUP_TOKEN_MODEL=anthropic/claude-opus-4-6`
Setup example:
```bash
openclaw models auth paste-token --provider anthropic --profile-id anthropic:setup-token-test
OPENCLAW_LIVE_SETUP_TOKEN=1 OPENCLAW_LIVE_SETUP_TOKEN_PROFILE=anthropic:setup-token-test pnpm test:live src/agents/anthropic.setup-token.live.test.ts
```
## Live: CLI backend smoke (Claude Code CLI or other local CLIs)
- Test: `src/gateway/gateway-cli-backend.live.test.ts`
- Goal: validate the Gateway + agent pipeline using a local CLI backend, without touching your default config.
- Enable:
- `pnpm test:live` (or `OPENCLAW_LIVE_TEST=1` if invoking Vitest directly)
- `OPENCLAW_LIVE_CLI_BACKEND=1`
- Defaults:
- Model: `claude-cli/claude-sonnet-4-6`
- Command: `claude`
- Args: `["-p","--output-format","json","--dangerously-skip-permissions"]`
- Overrides (optional):
- `OPENCLAW_LIVE_CLI_BACKEND_MODEL="claude-cli/claude-opus-4-6"`
- `OPENCLAW_LIVE_CLI_BACKEND_MODEL="codex-cli/gpt-5.3-codex"`
- `OPENCLAW_LIVE_CLI_BACKEND_COMMAND="/full/path/to/claude"`
- `OPENCLAW_LIVE_CLI_BACKEND_ARGS='["-p","--output-format","json","--permission-mode","bypassPermissions"]'`
- `OPENCLAW_LIVE_CLI_BACKEND_CLEAR_ENV='["ANTHROPIC_API_KEY","ANTHROPIC_API_KEY_OLD"]'`
- `OPENCLAW_LIVE_CLI_BACKEND_IMAGE_PROBE=1` to send a real image attachment (paths are injected into the prompt).
- `OPENCLAW_LIVE_CLI_BACKEND_IMAGE_ARG="--image"` to pass image file paths as CLI args instead of prompt injection.
- `OPENCLAW_LIVE_CLI_BACKEND_IMAGE_MODE="repeat"` (or `"list"`) to control how image args are passed when `IMAGE_ARG` is set.
- `OPENCLAW_LIVE_CLI_BACKEND_RESUME_PROBE=1` to send a second turn and validate resume flow.
- `OPENCLAW_LIVE_CLI_BACKEND_DISABLE_MCP_CONFIG=0` to keep Claude Code CLI MCP config enabled (default disables MCP config with a temporary empty file).
Example:
```bash
OPENCLAW_LIVE_CLI_BACKEND=1 \
OPENCLAW_LIVE_CLI_BACKEND_MODEL="claude-cli/claude-sonnet-4-6" \
pnpm test:live src/gateway/gateway-cli-backend.live.test.ts
```
### Recommended live recipes
Narrow, explicit allowlists are fastest and least flaky:
- Single model, direct (no gateway):
- `OPENCLAW_LIVE_MODELS="openai/gpt-5.2" pnpm test:live src/agents/models.profiles.live.test.ts`
- Single model, gateway smoke:
- `OPENCLAW_LIVE_GATEWAY_MODELS="openai/gpt-5.2" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts`
- Tool calling across several providers:
- `OPENCLAW_LIVE_GATEWAY_MODELS="openai/gpt-5.2,anthropic/claude-opus-4-6,google/gemini-3-flash-preview,zai/glm-4.7,minimax/minimax-m2.1" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts`
- Google focus (Gemini API key + Antigravity):
- Gemini (API key): `OPENCLAW_LIVE_GATEWAY_MODELS="google/gemini-3-flash-preview" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts`
- Antigravity (OAuth): `OPENCLAW_LIVE_GATEWAY_MODELS="google-antigravity/claude-opus-4-6-thinking,google-antigravity/gemini-3-pro-high" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts`
Notes:
- `google/...` uses the Gemini API (API key).
- `google-antigravity/...` uses the Antigravity OAuth bridge (Cloud Code Assist-style agent endpoint).
- `google-gemini-cli/...` uses the local Gemini CLI on your machine (separate auth + tooling quirks).
- Gemini API vs Gemini CLI:
- API: OpenClaw calls Googles hosted Gemini API over HTTP (API key / profile auth); this is what most users mean by “Gemini”.
- CLI: OpenClaw shells out to a local `gemini` binary; it has its own auth and can behave differently (streaming/tool support/version skew).
## Live: model matrix (what we cover)
There is no fixed “CI model list” (live is opt-in), but these are the **recommended** models to cover regularly on a dev machine with keys.
### Modern smoke set (tool calling + image)
This is the “common models” run we expect to keep working:
- OpenAI (non-Codex): `openai/gpt-5.2` (optional: `openai/gpt-5.1`)
- OpenAI Codex: `openai-codex/gpt-5.3-codex` (optional: `openai-codex/gpt-5.3-codex-codex`)
- Anthropic: `anthropic/claude-opus-4-6` (or `anthropic/claude-sonnet-4-5`)
- Google (Gemini API): `google/gemini-3-pro-preview` and `google/gemini-3-flash-preview` (avoid older Gemini 2.x models)
- Google (Antigravity): `google-antigravity/claude-opus-4-6-thinking` and `google-antigravity/gemini-3-flash`
- Z.AI (GLM): `zai/glm-4.7`
- MiniMax: `minimax/minimax-m2.1`
Run gateway smoke with tools + image:
`OPENCLAW_LIVE_GATEWAY_MODELS="openai/gpt-5.2,openai-codex/gpt-5.3-codex,anthropic/claude-opus-4-6,google/gemini-3-pro-preview,google/gemini-3-flash-preview,google-antigravity/claude-opus-4-6-thinking,google-antigravity/gemini-3-flash,zai/glm-4.7,minimax/minimax-m2.1" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts`
### Baseline: tool calling (Read + optional Exec)
Pick at least one per provider family:
- OpenAI: `openai/gpt-5.2` (or `openai/gpt-5-mini`)
- Anthropic: `anthropic/claude-opus-4-6` (or `anthropic/claude-sonnet-4-5`)
- Google: `google/gemini-3-flash-preview` (or `google/gemini-3-pro-preview`)
- Z.AI (GLM): `zai/glm-4.7`
- MiniMax: `minimax/minimax-m2.1`
Optional additional coverage (nice to have):
- xAI: `xai/grok-4` (or latest available)
- Mistral: `mistral/`… (pick one “tools” capable model you have enabled)
- Cerebras: `cerebras/`… (if you have access)
- LM Studio: `lmstudio/`… (local; tool calling depends on API mode)
### Vision: image send (attachment → multimodal message)
Include at least one image-capable model in `OPENCLAW_LIVE_GATEWAY_MODELS` (Claude/Gemini/OpenAI vision-capable variants, etc.) to exercise the image probe.
### Aggregators / alternate gateways
If you have keys enabled, we also support testing via:
- OpenRouter: `openrouter/...` (hundreds of models; use `openclaw models scan` to find tool+image capable candidates)
- OpenCode Zen: `opencode/...` (auth via `OPENCODE_API_KEY` / `OPENCODE_ZEN_API_KEY`)
More providers you can include in the live matrix (if you have creds/config):
- Built-in: `openai`, `openai-codex`, `anthropic`, `google`, `google-vertex`, `google-antigravity`, `google-gemini-cli`, `zai`, `openrouter`, `opencode`, `xai`, `groq`, `cerebras`, `mistral`, `github-copilot`
- Via `models.providers` (custom endpoints): `minimax` (cloud/API), plus any OpenAI/Anthropic-compatible proxy (LM Studio, vLLM, LiteLLM, etc.)
Tip: dont try to hardcode “all models” in docs. The authoritative list is whatever `discoverModels(...)` returns on your machine + whatever keys are available.
## Credentials (never commit)
Live tests discover credentials the same way the CLI does. Practical implications:
- If the CLI works, live tests should find the same keys.
- If a live test says “no creds”, debug the same way youd debug `openclaw models list` / model selection.
- Profile store: `~/.openclaw/credentials/` (preferred; what “profile keys” means in the tests)
- Config: `~/.openclaw/openclaw.json` (or `OPENCLAW_CONFIG_PATH`)
If you want to rely on env keys (e.g. exported in your `~/.profile`), run local tests after `source ~/.profile`, or use the Docker runners below (they can mount `~/.profile` into the container).
## Deepgram live (audio transcription)
- Test: `src/media-understanding/providers/deepgram/audio.live.test.ts`
- Enable: `DEEPGRAM_API_KEY=... DEEPGRAM_LIVE_TEST=1 pnpm test:live src/media-understanding/providers/deepgram/audio.live.test.ts`
## BytePlus coding plan live
- Test: `src/agents/byteplus.live.test.ts`
- Enable: `BYTEPLUS_API_KEY=... BYTEPLUS_LIVE_TEST=1 pnpm test:live src/agents/byteplus.live.test.ts`
- Optional model override: `BYTEPLUS_CODING_MODEL=ark-code-latest`
## Docker runners (optional “works in Linux” checks)
These run `pnpm test:live` inside the repo Docker image, mounting your local config dir and workspace (and sourcing `~/.profile` if mounted):
- Direct models: `pnpm test:docker:live-models` (script: `scripts/test-live-models-docker.sh`)
- Gateway + dev agent: `pnpm test:docker:live-gateway` (script: `scripts/test-live-gateway-models-docker.sh`)
- Onboarding wizard (TTY, full scaffolding): `pnpm test:docker:onboard` (script: `scripts/e2e/onboard-docker.sh`)
- Gateway networking (two containers, WS auth + health): `pnpm test:docker:gateway-network` (script: `scripts/e2e/gateway-network-docker.sh`)
- Plugins (custom extension load + registry smoke): `pnpm test:docker:plugins` (script: `scripts/e2e/plugins-docker.sh`)
Manual ACP plain-language thread smoke (not CI):
- `bun scripts/dev/discord-acp-plain-language-smoke.ts --channel <discord-channel-id> ...`
- Keep this script for regression/debug workflows. It may be needed again for ACP thread routing validation, so do not delete it.
Useful env vars:
- `OPENCLAW_CONFIG_DIR=...` (default: `~/.openclaw`) mounted to `/home/node/.openclaw`
- `OPENCLAW_WORKSPACE_DIR=...` (default: `~/.openclaw/workspace`) mounted to `/home/node/.openclaw/workspace`
- `OPENCLAW_PROFILE_FILE=...` (default: `~/.profile`) mounted to `/home/node/.profile` and sourced before running tests
- `OPENCLAW_LIVE_GATEWAY_MODELS=...` / `OPENCLAW_LIVE_MODELS=...` to narrow the run
- `OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1` to ensure creds come from the profile store (not env)
## Docs sanity
Run docs checks after doc edits: `pnpm docs:list`.
## Offline regression (CI-safe)
These are “real pipeline” regressions without real providers:
- Gateway tool calling (mock OpenAI, real gateway + agent loop): `src/gateway/gateway.test.ts` (case: "runs a mock OpenAI tool call end-to-end via gateway agent loop")
- Gateway wizard (WS `wizard.start`/`wizard.next`, writes config + auth enforced): `src/gateway/gateway.test.ts` (case: "runs wizard over ws and writes auth token config")
## Agent reliability evals (skills)
We already have a few CI-safe tests that behave like “agent reliability evals”:
- Mock tool-calling through the real gateway + agent loop (`src/gateway/gateway.test.ts`).
- End-to-end wizard flows that validate session wiring and config effects (`src/gateway/gateway.test.ts`).
Whats still missing for skills (see [Skills](/tools/skills)):
- **Decisioning:** when skills are listed in the prompt, does the agent pick the right skill (or avoid irrelevant ones)?
- **Compliance:** does the agent read `SKILL.md` before use and follow required steps/args?
- **Workflow contracts:** multi-turn scenarios that assert tool order, session history carryover, and sandbox boundaries.
Future evals should stay deterministic first:
- A scenario runner using mock providers to assert tool calls + order, skill file reads, and session wiring.
- A small suite of skill-focused scenarios (use vs avoid, gating, prompt injection).
- Optional live evals (opt-in, env-gated) only after the CI-safe suite is in place.
## Adding regressions (guidance)
When you fix a provider/model issue discovered in live:
- Add a CI-safe regression if possible (mock/stub provider, or capture the exact request-shape transformation)
- If its inherently live-only (rate limits, auth policies), keep the live test narrow and opt-in via env vars
- Prefer targeting the smallest layer that catches the bug:
- provider request conversion/replay bug → direct models test
- gateway session/history/tool pipeline bug → gateway live smoke or CI-safe gateway mock test

View File

@@ -0,0 +1,265 @@
---
summary: "Symptom first troubleshooting hub for OpenClaw"
read_when:
- OpenClaw is not working and you need the fastest path to a fix
- You want a triage flow before diving into deep runbooks
title: "Troubleshooting"
---
# Troubleshooting
If you only have 2 minutes, use this page as a triage front door.
## First 60 seconds
Run this exact ladder in order:
```bash
openclaw status
openclaw status --all
openclaw gateway probe
openclaw gateway status
openclaw doctor
openclaw channels status --probe
openclaw logs --follow
```
Good output in one line:
- `openclaw status` → shows configured channels and no obvious auth errors.
- `openclaw status --all` → full report is present and shareable.
- `openclaw gateway probe` → expected gateway target is reachable.
- `openclaw gateway status``Runtime: running` and `RPC probe: ok`.
- `openclaw doctor` → no blocking config/service errors.
- `openclaw channels status --probe` → channels report `connected` or `ready`.
- `openclaw logs --follow` → steady activity, no repeating fatal errors.
## Decision tree
```mermaid
flowchart TD
A[OpenClaw is not working] --> B{What breaks first}
B --> C[No replies]
B --> D[Dashboard or Control UI will not connect]
B --> E[Gateway will not start or service not running]
B --> F[Channel connects but messages do not flow]
B --> G[Cron or heartbeat did not fire or did not deliver]
B --> H[Node is paired but camera canvas screen exec fails]
B --> I[Browser tool fails]
C --> C1[/No replies section/]
D --> D1[/Control UI section/]
E --> E1[/Gateway section/]
F --> F1[/Channel flow section/]
G --> G1[/Automation section/]
H --> H1[/Node tools section/]
I --> I1[/Browser section/]
```
<AccordionGroup>
<Accordion title="No replies">
```bash
openclaw status
openclaw gateway status
openclaw channels status --probe
openclaw pairing list --channel <channel> [--account <id>]
openclaw logs --follow
```
Good output looks like:
- `Runtime: running`
- `RPC probe: ok`
- Your channel shows connected/ready in `channels status --probe`
- Sender appears approved (or DM policy is open/allowlist)
Common log signatures:
- `drop guild message (mention required` → mention gating blocked the message in Discord.
- `pairing request` → sender is unapproved and waiting for DM pairing approval.
- `blocked` / `allowlist` in channel logs → sender, room, or group is filtered.
Deep pages:
- [/gateway/troubleshooting#no-replies](/gateway/troubleshooting#no-replies)
- [/channels/troubleshooting](/channels/troubleshooting)
- [/channels/pairing](/channels/pairing)
</Accordion>
<Accordion title="Dashboard or Control UI will not connect">
```bash
openclaw status
openclaw gateway status
openclaw logs --follow
openclaw doctor
openclaw channels status --probe
```
Good output looks like:
- `Dashboard: http://...` is shown in `openclaw gateway status`
- `RPC probe: ok`
- No auth loop in logs
Common log signatures:
- `device identity required` → HTTP/non-secure context cannot complete device auth.
- `unauthorized` / reconnect loop → wrong token/password or auth mode mismatch.
- `gateway connect failed:` → UI is targeting the wrong URL/port or unreachable gateway.
Deep pages:
- [/gateway/troubleshooting#dashboard-control-ui-connectivity](/gateway/troubleshooting#dashboard-control-ui-connectivity)
- [/web/control-ui](/web/control-ui)
- [/gateway/authentication](/gateway/authentication)
</Accordion>
<Accordion title="Gateway will not start or service installed but not running">
```bash
openclaw status
openclaw gateway status
openclaw logs --follow
openclaw doctor
openclaw channels status --probe
```
Good output looks like:
- `Service: ... (loaded)`
- `Runtime: running`
- `RPC probe: ok`
Common log signatures:
- `Gateway start blocked: set gateway.mode=local` → gateway mode is unset/remote.
- `refusing to bind gateway ... without auth` → non-loopback bind without token/password.
- `another gateway instance is already listening` or `EADDRINUSE` → port already taken.
Deep pages:
- [/gateway/troubleshooting#gateway-service-not-running](/gateway/troubleshooting#gateway-service-not-running)
- [/gateway/background-process](/gateway/background-process)
- [/gateway/configuration](/gateway/configuration)
</Accordion>
<Accordion title="Channel connects but messages do not flow">
```bash
openclaw status
openclaw gateway status
openclaw logs --follow
openclaw doctor
openclaw channels status --probe
```
Good output looks like:
- Channel transport is connected.
- Pairing/allowlist checks pass.
- Mentions are detected where required.
Common log signatures:
- `mention required` → group mention gating blocked processing.
- `pairing` / `pending` → DM sender is not approved yet.
- `not_in_channel`, `missing_scope`, `Forbidden`, `401/403` → channel permission token issue.
Deep pages:
- [/gateway/troubleshooting#channel-connected-messages-not-flowing](/gateway/troubleshooting#channel-connected-messages-not-flowing)
- [/channels/troubleshooting](/channels/troubleshooting)
</Accordion>
<Accordion title="Cron or heartbeat did not fire or did not deliver">
```bash
openclaw status
openclaw gateway status
openclaw cron status
openclaw cron list
openclaw cron runs --id <jobId> --limit 20
openclaw logs --follow
```
Good output looks like:
- `cron.status` shows enabled with a next wake.
- `cron runs` shows recent `ok` entries.
- Heartbeat is enabled and not outside active hours.
Common log signatures:
- `cron: scheduler disabled; jobs will not run automatically` → cron is disabled.
- `heartbeat skipped` with `reason=quiet-hours` → outside configured active hours.
- `requests-in-flight` → main lane busy; heartbeat wake was deferred.
- `unknown accountId` → heartbeat delivery target account does not exist.
Deep pages:
- [/gateway/troubleshooting#cron-and-heartbeat-delivery](/gateway/troubleshooting#cron-and-heartbeat-delivery)
- [/automation/troubleshooting](/automation/troubleshooting)
- [/gateway/heartbeat](/gateway/heartbeat)
</Accordion>
<Accordion title="Node is paired but tool fails camera canvas screen exec">
```bash
openclaw status
openclaw gateway status
openclaw nodes status
openclaw nodes describe --node <idOrNameOrIp>
openclaw logs --follow
```
Good output looks like:
- Node is listed as connected and paired for role `node`.
- Capability exists for the command you are invoking.
- Permission state is granted for the tool.
Common log signatures:
- `NODE_BACKGROUND_UNAVAILABLE` → bring node app to foreground.
- `*_PERMISSION_REQUIRED` → OS permission was denied/missing.
- `SYSTEM_RUN_DENIED: approval required` → exec approval is pending.
- `SYSTEM_RUN_DENIED: allowlist miss` → command not on exec allowlist.
Deep pages:
- [/gateway/troubleshooting#node-paired-tool-fails](/gateway/troubleshooting#node-paired-tool-fails)
- [/nodes/troubleshooting](/nodes/troubleshooting)
- [/tools/exec-approvals](/tools/exec-approvals)
</Accordion>
<Accordion title="Browser tool fails">
```bash
openclaw status
openclaw gateway status
openclaw browser status
openclaw logs --follow
openclaw doctor
```
Good output looks like:
- Browser status shows `running: true` and a chosen browser/profile.
- `openclaw` profile starts or `chrome` relay has an attached tab.
Common log signatures:
- `Failed to start Chrome CDP on port` → local browser launch failed.
- `browser.executablePath not found` → configured binary path is wrong.
- `Chrome extension relay is running, but no tab is connected` → extension not attached.
- `Browser attachOnly is enabled ... not reachable` → attach-only profile has no live CDP target.
Deep pages:
- [/gateway/troubleshooting#browser-tool-fails](/gateway/troubleshooting#browser-tool-fails)
- [/tools/browser-linux-troubleshooting](/tools/browser-linux-troubleshooting)
- [/tools/chrome-extension](/tools/chrome-extension)
</Accordion>
</AccordionGroup>