test: run the full unit suite in ci and fix revealed failures - #224
Merged
Conversation
CI ran `make test-python` over only 4 hardcoded files, so ~268 of 272 top-level test files never executed in CI and could red-rot unnoticed. Bare pytest also errored at collection because 7 channel-adapter tests import optional SDKs at module top level. - Makefile: test-python now runs the whole unit tree (uv run --all-extras pytest) - ci.yml: the python job becomes a blocking unit matrix (py3.11-3.13 + macos) - pyproject: register integration/e2e markers, --strict-markers/--strict-config, norecursedirs + default -m "not integration and not e2e" - guard the 7 channel tests with pytest.importorskip - relocate 9 integration/benchmark files under tests/integration/ (real_llm marked) - fix the 44 pre-existing failures the full run revealed: get_event_loop() anti-pattern in producer tests (now asyncio.run), stale run_turn stubs missing inline_tool_stream, provider_catalog missing LazyProvider/PerModelProvider, config-path isolation in tui_rpc_model / tui_commands_error_codes / logging_isolation, and a flaky concurrency timing bound Full unit suite: 4585 passed, 24 deselected (tui e2e), 0 failed. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
Temporary diagnostic to capture the CI console environment; will be removed before merge. Also fixes the Windows self-upgrade job, whose isolated pytest hit --strict-config on the unknown asyncio_mode option. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
The first full-suite CI run failed 16 CLI/theme tests that pass locally. Typer's rich_utils force-enables colour when GITHUB_ACTIONS is set, and its option highlighter then emits escape codes between the two dashes of a flag, so assertions like `"--probe" in result.stdout` cannot match. Setting TERM=dumb for the unit job restores plain rendering; tests that assert on colour build their own forced-terminal Console and are unaffected. Also drop the version/OS matrix down to a single ubuntu / py3.12 cell: pyproject declares requires-python >= 3.12, so the 3.11 cell was testing an unsupported version, and all four cells reported identical results. The matrix block is kept so more versions are one list entry away. Windows keeps its dedicated self-upgrade job. Removes the temporary diagnostic test added to capture the CI environment. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
…e OSes The first green-tests CI run still exited 139: three cross-process locking tests start workers with the platform default start method, which is fork on Linux. pytest leaves the parent multi-threaded, so the forked children left the interpreter segfaulting at exit. They now use an explicit spawn context, matching what macOS already did and what CPython is moving toward. The unit matrix gains macos-latest and windows-latest on py3.12. macOS is expected to pass -- its cell reported the same failures as ubuntu before the colour fix and never segfaulted. Windows has never run this suite, so this run is a measurement: fail-fast is off, and the ubuntu and macos cells report independently. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
The ubuntu cell passed all 4611 tests but still exited 139. The hazard is already documented in cli/agent_commands.py: native runtimes loaded by the agent loop (lancedb's Rust/tokio thread, torch) segfault during interpreter finalization, which production dodges with a hard exit. Nothing in the repo imports torch or transformers -- a whole-tree grep finds them only in the extra declaration, three comments, and a doc note -- so the retrieval extra is dead weight that only slows CI and adds a crash surface. The unit job now installs channels + tools + sandbox; the suite still collects 4611 tests. The windows cell is removed again: boxlite ships wheels for macOS and Linux only (it is a microVM runtime built on KVM / Hypervisor.framework), so installing the sandbox extra on Windows fails outright and no test ever runs. The missing `sys_platform != 'win32'` marker on that dependency is a real packaging bug, tracked separately -- Windows can rejoin the matrix once it lands. macOS stays, having passed on its first run. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
Dropping the retrieval extra was not enough: the ubuntu cell still exits 139 after a fully green run, because lancedb -- a transitive dependency, not something the extras control -- leaves its Rust/tokio LanceDBBackgroundEventLoop daemon thread running, and finalizing the interpreter around it segfaults. raven.cli._exit already documents and guards exactly this for the CLI. Its docstring notes that CliRunner keeps normal exit semantics in tests, which is true per command, but the pytest process itself finalizes with that thread live once any test has built the agent loop. The session now reuses the same guard: pytest_sessionfinish records the real exit status and pytest_unconfigure flushes and hard-exits with it when the hazard is live. Verified that the guard preserves exit codes rather than masking failures: a passing run exits 0, a failing run exits 1, and a failing run that also loaded lancedb still exits 1. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
The previous guard keyed on lancedb's background thread, but the unit tier never opens a lancedb connection, so the probe returned false and the run still exited 139. The crash therefore comes from another native teardown -- the run also reports an asyncio BaseSubprocessTransport finalized during GC, and only Linux is affected. Rather than chase each runtime, the guard now hard-exits whenever CI is set, after the session has recorded its real status. Local runs are untouched, so nothing masks an exit-time error during development, and the recorded status is preserved in both cases: verified that a passing run exits 0 and a failing run exits 1, with and without CI set. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
arelchan
approved these changes
Jul 28, 2026
This was referenced Jul 28, 2026
0xKT
added a commit
that referenced
this pull request
Jul 31, 2026
## Summary Bump the package version from 0.1.9 to 0.1.10 (patch release). 22 PRs merged since v0.1.9, no breaking changes. Features: - #260 feat(tui): list the providers that work, and the rest one level down - #252 feat(*): offer every supported provider in the onboarding picker - #251 feat(tools): read images with read_file, and fix four silent type-check bugs - #239 feat: add shell command approval flow - #217 feat: rework the TUI transcript into collapsible episodes - #220 feat(cli): nudge raven upgrade in the tui status bar when behind - #209 feat(providers): add MiniMax Global and CN OAuth Refactors: - #259 refactor(*): one answer to which credentials a provider needs - #249 refactor(providers): unify provider management on litellm The remaining PRs are fixes (#255, #256, #258, #253, #238, #226), test work (#236, #230, #224), docs (#250, #215, #200), and benchmark tooling (#207). ## Type - [ ] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [x] Other ## Verification - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed Bump is limited to `pyproject.toml` and `uv.lock` (`uv lock` sync). Local preflight covers the branch CI checks (commit lint, PR title and body lint, ruff, large-file gate). ## Risk - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes Version-only change; no code or behavior change. Rollback is a revert of this commit. ## Related Issues N/A Co-authored-by: Claude (claude-opus-5[1m]) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CI ran
make test-pythonover 4 hardcoded files, so ~268 of 272 top-level test files never executed on any PR. A bareuv run pytestcould not stand in for it either: collection aborted with 7 errors (channel-adapter tests importing optional SDKs at module level), andtestpathspulledtests/integration/into every run.This makes the whole unit suite the blocking CI gate.
Makefile:test-pythonruns the whole unit tree instead of a hardcoded file list.ci.yml: thepythonjob splits intolint-pythonand aunitjob running the whole suite on ubuntu / py3.12; the matrix block is kept so more versions are one list entry away.pyproject.toml: registerintegration/e2emarkers, enable--strict-markers/--strict-config, excludetests/integrationvianorecursedirs, and default to-m "not integration and not e2e"so a barepytestis safe.pytest.importorskip; CI installs--all-extras, so they run there rather than skip.tests/integration/, renaming the token_wise benchmarks to thereal_llmkind required by AGENTS.md section 5.2.Turning the suite on surfaced 44 pre-existing failures, all fixed here:
asyncio.get_event_loop().run_until_complete()in three producer test files, which picks up a closed loop once other async tests have run; nowasyncio.run().run_turnstubs missing theinline_tool_streamkeyword the production callers pass.raven.config.loader._current_config_path, litellm's global logging handlers, and an ambient dependency on a populated~/.ravenin the TUI init-error tests.The full run also recovered 38 genuine unit tests under
tests/tui/autotestthat a directory-level exclusion would have orphaned; they are selected normally now, with the 24 subprocess-spawning e2e tests deselected by marker.Deliberately out of scope: an integration / nightly CI tier (real-LLM runs are hard to make meaningful in CI and cost money -- to be designed separately), and the naming and assertion debt tracked in #223.
Type
Verification
Risk
No production code changes -- test files, test configuration, and CI workflow only.
The repository ruleset currently declares no required status checks, so renaming the job blocks nothing. If the
unitjob should become a merge gate, addunit (py3.12 / ubuntu-latest)to the ruleset separately.Related Issues
Closes #222