Skip to content

test: run the full unit suite in ci and fix revealed failures - #224

Merged
0xKT merged 8 commits into
mainfrom
refactor/test_suite_layering_ci
Jul 28, 2026
Merged

test: run the full unit suite in ci and fix revealed failures#224
0xKT merged 8 commits into
mainfrom
refactor/test_suite_layering_ci

Conversation

@0xKT

@0xKT 0xKT commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

CI ran make test-python over 4 hardcoded files, so ~268 of 272 top-level test files never executed on any PR. A bare uv run pytest could not stand in for it either: collection aborted with 7 errors (channel-adapter tests importing optional SDKs at module level), and testpaths pulled tests/integration/ into every run.

This makes the whole unit suite the blocking CI gate.

  • Makefile: test-python runs the whole unit tree instead of a hardcoded file list.
  • ci.yml: the python job splits into lint-python and a unit job running the whole suite on ubuntu / py3.12; the matrix block is kept so more versions are one list entry away.
  • pyproject.toml: register integration / e2e markers, enable --strict-markers / --strict-config, exclude tests/integration via norecursedirs, and default to -m "not integration and not e2e" so a bare pytest is safe.
  • Guard the 7 channel-adapter tests with pytest.importorskip; CI installs --all-extras, so they run there rather than skip.
  • Relocate 9 integration / real-resource files out of the flat unit tree into tests/integration/, renaming the token_wise benchmarks to the real_llm kind required by AGENTS.md section 5.2.

Turning the suite on surfaced 44 pre-existing failures, all fixed here:

  • 29 from asyncio.get_event_loop().run_until_complete() in three producer test files, which picks up a closed loop once other async tests have run; now asyncio.run().
  • 2 stale run_turn stubs missing the inline_tool_stream keyword the production callers pass.
  • 12 from cross-test state leaking through raven.config.loader._current_config_path, litellm's global logging handlers, and an ambient dependency on a populated ~/.raven in the TUI init-error tests.
  • 1 flaky concurrency test whose timing bound left only 0.05s of margin.

The full run also recovered 38 genuine unit tests under tests/tui/autotest that a directory-level exclusion would have orphaned; they are selected normally now, with the 24 subprocess-spawning e2e tests deselected by marker.

Deliberately out of scope: an integration / nightly CI tier (real-LLM runs are hard to make meaningful in CI and cost money -- to be designed separately), and the naming and assertion debt tracked in #223.

Type

  • CI / tooling

Verification

uv run --all-extras pytest -q
# 4611 passed, 24 deselected in 193s

bash .claude/scripts/preflight_ci.sh
# ALL PREFLIGHT CHECKS PASSED (commitlint, large files, pre-commit, ruff, tests)
  • Relevant tests pass locally
  • Relevant lint / type checks pass locally

Risk

  • Security impact considered
  • Backward compatibility considered
  • Rollback path is clear for risky changes

No production code changes -- test files, test configuration, and CI workflow only.

The repository ruleset currently declares no required status checks, so renaming the job blocks nothing. If the unit job should become a merge gate, add unit (py3.12 / ubuntu-latest) to the ruleset separately.

Related Issues

Closes #222

0xKT and others added 8 commits July 28, 2026 11:40
CI ran `make test-python` over only 4 hardcoded files, so ~268 of 272
top-level test files never executed in CI and could red-rot unnoticed.
Bare pytest also errored at collection because 7 channel-adapter tests
import optional SDKs at module top level.

- Makefile: test-python now runs the whole unit tree (uv run --all-extras pytest)
- ci.yml: the python job becomes a blocking unit matrix (py3.11-3.13 + macos)
- pyproject: register integration/e2e markers, --strict-markers/--strict-config,
  norecursedirs + default -m "not integration and not e2e"
- guard the 7 channel tests with pytest.importorskip
- relocate 9 integration/benchmark files under tests/integration/ (real_llm marked)
- fix the 44 pre-existing failures the full run revealed: get_event_loop()
  anti-pattern in producer tests (now asyncio.run), stale run_turn stubs missing
  inline_tool_stream, provider_catalog missing LazyProvider/PerModelProvider,
  config-path isolation in tui_rpc_model / tui_commands_error_codes /
  logging_isolation, and a flaky concurrency timing bound

Full unit suite: 4585 passed, 24 deselected (tui e2e), 0 failed.

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
Temporary diagnostic to capture the CI console environment; will be
removed before merge. Also fixes the Windows self-upgrade job, whose
isolated pytest hit --strict-config on the unknown asyncio_mode option.

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
The first full-suite CI run failed 16 CLI/theme tests that pass locally.
Typer's rich_utils force-enables colour when GITHUB_ACTIONS is set, and its
option highlighter then emits escape codes between the two dashes of a flag,
so assertions like `"--probe" in result.stdout` cannot match. Setting
TERM=dumb for the unit job restores plain rendering; tests that assert on
colour build their own forced-terminal Console and are unaffected.

Also drop the version/OS matrix down to a single ubuntu / py3.12 cell:
pyproject declares requires-python >= 3.12, so the 3.11 cell was testing an
unsupported version, and all four cells reported identical results. The
matrix block is kept so more versions are one list entry away. Windows keeps
its dedicated self-upgrade job.

Removes the temporary diagnostic test added to capture the CI environment.

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
…e OSes

The first green-tests CI run still exited 139: three cross-process locking
tests start workers with the platform default start method, which is fork on
Linux. pytest leaves the parent multi-threaded, so the forked children left
the interpreter segfaulting at exit. They now use an explicit spawn context,
matching what macOS already did and what CPython is moving toward.

The unit matrix gains macos-latest and windows-latest on py3.12. macOS is
expected to pass -- its cell reported the same failures as ubuntu before the
colour fix and never segfaulted. Windows has never run this suite, so this
run is a measurement: fail-fast is off, and the ubuntu and macos cells report
independently.

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
The ubuntu cell passed all 4611 tests but still exited 139. The hazard is
already documented in cli/agent_commands.py: native runtimes loaded by the
agent loop (lancedb's Rust/tokio thread, torch) segfault during interpreter
finalization, which production dodges with a hard exit. Nothing in the repo
imports torch or transformers -- a whole-tree grep finds them only in the
extra declaration, three comments, and a doc note -- so the retrieval extra
is dead weight that only slows CI and adds a crash surface. The unit job now
installs channels + tools + sandbox; the suite still collects 4611 tests.

The windows cell is removed again: boxlite ships wheels for macOS and Linux
only (it is a microVM runtime built on KVM / Hypervisor.framework), so
installing the sandbox extra on Windows fails outright and no test ever runs.
The missing `sys_platform != 'win32'` marker on that dependency is a real
packaging bug, tracked separately -- Windows can rejoin the matrix once it
lands. macOS stays, having passed on its first run.

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
Dropping the retrieval extra was not enough: the ubuntu cell still exits 139
after a fully green run, because lancedb -- a transitive dependency, not
something the extras control -- leaves its Rust/tokio LanceDBBackgroundEventLoop
daemon thread running, and finalizing the interpreter around it segfaults.

raven.cli._exit already documents and guards exactly this for the CLI. Its
docstring notes that CliRunner keeps normal exit semantics in tests, which is
true per command, but the pytest process itself finalizes with that thread
live once any test has built the agent loop. The session now reuses the same
guard: pytest_sessionfinish records the real exit status and
pytest_unconfigure flushes and hard-exits with it when the hazard is live.

Verified that the guard preserves exit codes rather than masking failures: a
passing run exits 0, a failing run exits 1, and a failing run that also loaded
lancedb still exits 1.

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
The previous guard keyed on lancedb's background thread, but the unit tier
never opens a lancedb connection, so the probe returned false and the run
still exited 139. The crash therefore comes from another native teardown --
the run also reports an asyncio BaseSubprocessTransport finalized during GC,
and only Linux is affected.

Rather than chase each runtime, the guard now hard-exits whenever CI is set,
after the session has recorded its real status. Local runs are untouched, so
nothing masks an exit-time error during development, and the recorded status
is preserved in both cases: verified that a passing run exits 0 and a failing
run exits 1, with and without CI set.

Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
@0xKT
0xKT requested a review from arelchan July 28, 2026 07:27
@0xKT
0xKT merged commit 8f42b46 into main Jul 28, 2026
10 checks passed
@0xKT
0xKT deleted the refactor/test_suite_layering_ci branch July 28, 2026 07:29
0xKT added a commit that referenced this pull request Jul 31, 2026
## Summary

Bump the package version from 0.1.9 to 0.1.10 (patch release). 22 PRs
merged since v0.1.9, no breaking changes.

Features:

- #260 feat(tui): list the providers that work, and the rest one level
down
- #252 feat(*): offer every supported provider in the onboarding picker
- #251 feat(tools): read images with read_file, and fix four silent
type-check bugs
- #239 feat: add shell command approval flow
- #217 feat: rework the TUI transcript into collapsible episodes
- #220 feat(cli): nudge raven upgrade in the tui status bar when behind
- #209 feat(providers): add MiniMax Global and CN OAuth

Refactors:

- #259 refactor(*): one answer to which credentials a provider needs
- #249 refactor(providers): unify provider management on litellm

The remaining PRs are fixes (#255, #256, #258, #253, #238, #226), test
work (#236, #230, #224), docs (#250, #215, #200), and benchmark tooling
(#207).

## Type

- [ ] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [x] Other

## Verification

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [ ] User-facing docs or screenshots are updated when needed

Bump is limited to `pyproject.toml` and `uv.lock` (`uv lock` sync).
Local preflight covers the branch CI checks (commit lint, PR title and
body lint, ruff, large-file gate).

## Risk

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

Version-only change; no code or behavior change. Rollback is a revert of
this commit.

## Related Issues

N/A

Co-authored-by: Claude (claude-opus-5[1m]) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI runs only 4 of 272 unit-test files

2 participants