docs/TESTING_MATRIX.md

# Agent Canvas — Testing Matrix

**Priority key:** P0 = must pass before any release · P1 = must pass before GA · P2 = best-effort

---

## Install × OS × Agent

Each cell = smoke test: install → onboard → start conversation → agent replies.

| | **macOS** | **Linux** | **Windows** |
|---|:---:|:---:|:---:|
| **npm — OpenHands** | ☐ | ☐ | ☐ |
| **npm — Claude Code** | ☐ | ☐ | ☐ |
| **npm — Codex** | ☐ | ☐ | ☐ |
| **npm — Gemini CLI** | ☐ | ☐ | ☐ |
| **npm — Custom ACP** | ☐ | ☐ | ☐ |
| **Docker — OpenHands** | ☐ | ☐ | ☐ |
| **Docker — Claude Code** | ☐ | ☐ | ☐ |
| **Docker — Codex** | ☐ | ☐ | ☐ |
| **Docker — Gemini CLI** | ☐ | ☐ | ☐ |
| **Docker — Custom ACP** | ☐ | ☐ | ☐ |

---

## Automations × Install × Agent

Requires full stack (automation backend running).

| | **npm** | **Docker** |
|---|:---:|:---:|
| **OpenHands** | ✅ P0 | ✅ P0 |
| **Claude Code** | ✅ P1 | ✅ P1 |
| **Codex** | ✅ P1 | ✅ P1 |
| **Gemini CLI** | ✅ P2 | ✅ P2 |

Each cell = create automation → dispatch run → run reaches COMPLETED → conversation link works.

---

## Auth Modes

| | **npm** | **Docker** |
|---|:---:|:---:|
| **Local (auto-generated key)** | ✅ P0 | ✅ P0 |
| **Public (`--public` + user key)** | ✅ P1 | ✅ P1 |

---

## Feature Checklist

### npm

| Feature | OpenHands | Claude Code | Codex | Gemini CLI |
|---|:---:|:---:|:---:|:---:|
| Onboarding | ☐ | ☐ | ☐ | ☐ |
| Conversation — start, resume, history | ☐ | ☐ | ☐ | ☐ |
| Terminal tool | ☐ | ☐ | ☐ | ☐ |
| File editor tool | ☐ | ☐ | ☐ | ☐ |
| Browser tool | ☐ | ☐ | ☐ | ☐ |
| LLM profiles — create / switch | ☐ | — | — | — |
| Secrets — add / delete / forwarded | ☐ | ☐ | ☐ | ☐ |
| Automations — create, dispatch, COMPLETED | ☐ | ☐ | ☐ | ☐ |
| Files tab + Changes/diff tab | ☐ | ☐ | ☐ | ☐ |
| MCP server install | ☐ | ☐ | ☐ | ☐ |
| Image upload in chat | ☐ | ☐ | ☐ | ☐ |
| Key rotation | ☐ | ☐ | ☐ | ☐ |

### Docker

| Feature | OpenHands | Claude Code | Codex | Gemini CLI |
|---|:---:|:---:|:---:|:---:|
| Onboarding | ☐ | ☐ | ☐ | ☐ |
| Conversation — start, resume, history | ☐ | ☐ | ☐ | ☐ |
| Terminal tool | ☐ | ☐ | ☐ | ☐ |
| File editor tool | ☐ | ☐ | ☐ | ☐ |
| Browser tool | ☐ | ☐ | ☐ | ☐ |
| LLM profiles — create / switch | ☐ | — | — | — |
| Secrets — add / delete / forwarded | ☐ | ☐ | ☐ | ☐ |
| Automations — create, dispatch, COMPLETED | ☐ | ☐ | ☐ | ☐ |
| Files tab + Changes/diff tab | ☐ | ☐ | ☐ | ☐ |
| MCP server install | ☐ | ☐ | ☐ | ☐ |
| Image upload in chat | ☐ | ☐ | ☐ | ☐ |
| Key rotation | ☐ | ☐ | ☐ | ☐ |

---

## Automated Coverage

| Suite | Install | OS | Agents | Automations |
|---|---|---|---|---|
| `vitest` (unit) | — | Linux | — | partial |
| `test:e2e:mock-llm` | npm | Linux | OpenHands, ACP (mock) | ✅ full |
| `test:e2e:mock-llm:docker` | Docker | Linux | OpenHands, ACP (mock) | ✅ full |
| `test:e2e:live` | npm | Linux | OpenHands | ❌ |

**Not yet covered by CI:** real ACP credentials (Claude Code / Codex / Gemini), macOS, public auth mode, subscription login paths, Windows.