Repository navigation
test: Cover bundled app templates with a smoke test - #2504
Open
karangattu wants to merge 14 commits into
Open
karangattu wants to merge 14 commits into
karangattu wants to merge 14 commits into
Conversation
Each template entrypoint now builds its App with dummy keys. The run needs no browser and no model server. The new guard also scans Chat calls for messages= without history=. It covers templates and the bundled skill references. The smoke test found three broken markdown-stream templates. Two missed the input import. One missed the Ollama model. This commit fixes all three.
Real provider constructors validate against servers and packages. That validation changes across chatlas releases. Newer chatlas queries the model server and requires provider packages at build time. The test must not depend on that behavior. Stub all provider classes. Keep the Shiny side real. ui.Chat still raises on messages= without history=. The smoke test still catches that startup failure.
The oldest-deps job pins shiny's own deps to their floors. Two template deps break there for reasons outside this repo. python-dotenv is absent, so app_utils.load_dotenv warns. The xarray floor no longer imports under NumPy 2, which breaks import plotly.express. Stub dotenv when it is missing. Skip a template when its traceback shows it failed on its own import line. A failure on any other template line is still a template bug and still fails.
Walk the exception cause chain when deciding if a template failed on its own import line. Express wraps load errors in RuntimeError, which hid the template frame. Bump the shinychat floor to 0.6.0. The bundled chat templates pass greeting= to ui.Chat, which older releases reject. The lock file already pins 0.6.0.
A missing module skips only for an allowlisted optional dependency. A typo'd helper name or a deleted sibling file fails instead. A bad from-import fails even on an import line: the source resolved, so the name is wrong. A helper that raises while loading (a missing data file, for example) fails too. Only an error raised inside third-party import code skips, with unit tests pinning each rule.
Drop the exception classifier. Any template-load failure now fails the test, including wrapped Express errors. Skips hid real bugs. Exempt this file in the oldest-deps job instead. Its version floors cannot import the templates' plotting stack together. The exemption lives in the workflow, not in the test. Purge imported helper modules by file location. A future helper name cannot leak state into the next template unnoticed.
The bundled chat templates pass greeting= to the ui.Chat constructor. shinychat 0.4.0 only has greeting= on chat.ui(). The constructor form arrived in 0.5.0, so set that floor.
The smoke test is exempt there, so add a tiny compat test that keeps running: build an express Chat with greeting=, bookmark it, register a submit handler, render it. It fails if the shinychat floor ever drops an API the templates rely on. No key is needed under a stub session. Also evict shadowed helper modules before each load, so a stale cached copy cannot win over the template's own file.
Ensure calls using messages= explicitly specify history=False (as ast.Constant(value=False)), catching history=True, history=None, and omitted history. Add unit tests covering these cases.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
shiny createcopiesshiny/templates/verbatim. Nothing in CI ran these files. Five chat templates shipped aChat(messages=...)call that raised on startup (#2488).Change
This PR extends
tests/playwright/examples/test_shiny_create.pywith a parametrized test that launches every bundled template entrypoint (app.py,app-core.py,app-express.py, including the nestedmarkdown-streamchat templates).template_entrypoints()discovers them from the samefind_templates()helper thatshiny createuses, and skips the app templates already covered bytest_template_examples.The apps run as real Shiny apps in a child process, so it needs a browser. No model server and no real API key.
A
sitecustomize.pystub module keeps the provider classes inert. Python importssitecustomizefromPYTHONPATHbeforepython -m shiny, soDummyChatClientreplaces thechatlasprovider constructors (andlangchain_openai.ChatOpenAIwhen that package is absent) inside the app's own process. Any template that actually reaches a provider fails loudly. The Shiny side stays real, soui.Chatstill raises onmessages=withouthistory=.validate_example()now takes an optionalenvthat it forwards torun_shiny_app, which is how the stub path and the dummy provider keys reach the child process.Result
The new test found three broken markdown-stream templates before this PR. Two missed the
inputimport. One missed the Ollama model. This PR fixes all three.Closes #2488.