loki-mode

Testing

This document describes the test strategy, test types, and how to run and configure tests for Loki Mode.

Loki Mode is a polyglot codebase. Tests span three languages and runtimes:

Because the product’s central promise is verified completion (“Loki does not lie about done”), the test suite is treated as a trust layer, not a formality. Trust-surface code (completion councils, quality gates, evidence receipts, the Bun runner, memory integrity) is held to a higher bar: mutation testing, parity checks, and adversarial detectors back the standard unit and integration tests.

Test Strategy

  1. Test the trust surface hardest. Completion detection, override councils, the verified-completion evidence gate, and the RARV-C closure loop are the highest-risk paths. They get unit tests plus mutation testing plus cross-route parity checks.
  2. Cross-route parity is a first-class concern. Behavior must match between the legacy Bash route and the Bun (loki-ts) route. Parity is checked by tests/test-bash-bun-parity.sh and the per-command parity suites under tests/ (for example, doctor-output text drift); there is no continuous parity workflow.
  3. Prefer real verification over mocked assertions. Dedicated detectors (tests/detect-mock-problems.sh, tests/detect-semantic-test-problems.sh, tests/detect-test-mutations.sh, tests/detect-invariant-violations.sh) flag tests that pass without exercising real logic.
  4. Local CI is the canonical pre-push gate. scripts/local-ci.sh mirrors the GitHub Actions workflows so failures are discovered on the developer machine, not after push. GitHub Actions is the post-push verifier.
  5. Run tests in matrices. Python runs across 3.10 through 3.13; the Node suite runs across the supported Node versions. Version-specific breakage (for example, a Python 3.13-only inode race) must be caught by the matrix.

Test Types

Unit tests

Integration tests

End-to-end tests

Specialized test layers

How to Run Tests

Quick start

# Full default suite (bash syntax checks + node --test suites + managed-memory)
npm test

# Pre-push gate: mirrors every CI workflow. Run this before every git push.
bash scripts/local-ci.sh

If local-ci.sh reports “DO NOT PUSH”, do not push. Fix the failures and re-run.

By language and layer

# Bash shell test suite
bash tests/run-all-tests.sh

# Python tests (all)
python3 -m pytest

# Python tests (single module)
python3 -m pytest tests/test_classifier.py -v

# TypeScript / Bun runner tests
cd loki-ts && bun test

# Bun runner type check
cd loki-ts && bun run typecheck

# Integration suite
npm run test:integration
# or
bash tests/integration/run_integration_suite.sh

# Control Plane tests
cd packages/control-plane && bun test ./test/

Mutation testing (local)

cd loki-ts && npx stryker run

Syntax-only validation (fast smoke)

bash -n autonomy/run.sh
bash -n autonomy/loki
bash -n autonomy/completion-council.sh
python3 -c "import ast, os; [ast.parse(open(f'dashboard/{f}').read()) for f in os.listdir('dashboard') if f.endswith('.py')]"

Cleanup after test runs (mandatory)

Spawned processes and temp files must be cleaned up after any run that starts a server or test runner:

lsof -ti:57374 | xargs kill -9 2>/dev/null || true
pkill -f "loki-run-" 2>/dev/null || true
rm -rf /tmp/loki-* /tmp/test-* /tmp/package /tmp/*.tgz 2>/dev/null || true

# Verify
ps -ef | grep -E "(loki|test)" | grep -v grep || echo "Clean"

Note: never run rm -rf /tmp/loki-* while a live loki run is in progress; the orchestrator stages its run script at /tmp/loki-run-*.sh. Check pgrep -f loki-run- first.

Test Configuration

Area Tool Config / entry point
Default test command mixed (bash, node) package.json test script
Bash suite bash tests/run-all-tests.sh
Python pytest invoked via python3 -m pytest; deps: pytest, pytest-asyncio
Bun runner bun test loki-ts/package.json, loki-ts/bunfig.toml
Node/Deno node --test *_test.ts, *.test.js
Mutation Stryker loki-ts/stryker.config.json
Control Plane UI E2E Playwright packages/control-plane/test/e2e/
Pre-push gate bash scripts/local-ci.sh

Python test dependencies commonly required: fastapi, httpx, pydantic, sqlalchemy[asyncio], aiosqlite, uvicorn. Install with pip before running the Python suite.

Coverage Goals

CI Integration

GitHub Actions workflows (under .github/workflows/) run on push and pull request:

Workflow File What it runs
Tests test.yml Node suite (npm test), Python suite (pytest) across 3.10-3.13, shell suite (tests/run-all-tests.sh), Helm lint
Integrity audit integrity-audit.yml Repository and artifact integrity checks
Post-release smoke post-release-smoke.yml Smoke tests against published artifacts after a release
Security audit security-audit.yml Dependency and security scanning
SBOM sbom.yml Software bill of materials generation

Local CI mirrors GitHub Actions

scripts/local-ci.sh is the canonical pre-push gate. It mirrors every GitHub Actions workflow: bun typecheck/test, the full tests/run-all-tests.sh shell suite (matching CI exactly, not a cherry-picked subset), npm pack contents, SBOM, license audit, npm audit, shellcheck, YAML parse, the no-emoji check, the no-git add -A check, and a cleanup probe.

Run it before every push. The Mac (developer machine) is the discovery channel; GitHub Actions is the post-push verifier. Distinguish real failures from local-environment false alarms (for example a missing pytest-asyncio in a Homebrew Python) before acting on a red result.

Release-time validation

Before a release is committed, the npm tarball is verified to contain the built dashboard and web-app artifacts, and a fresh global install is exercised end to end (loki --version, loki web, /api/status). After a release ships, post-release distribution validation runs across npm, Docker, and Homebrew on both the Bun and legacy-Bash routes.

This site is open source. Improve this page.