test: add Gherkin evaluator tests using test-harness specs#1947
Conversation
✅ Deploy Preview for polite-licorice-3db33c ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
dosubot
Bot
added
the
size:XS
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces Gherkin-based integration tests for the evaluator component. It adds a new test file core/pkg/evaluator/gherkin_test.go utilizing the godog framework to execute feature-based tests against the evaluator's logic, including support for various data types, targeting keys, and metadata assertions. Additionally, the Makefile has been updated with a test-evaluator-gherkin target, and necessary dependencies have been added to core/go.mod. I have no feedback to provide.
aepfli
force-pushed
the
feat/evaluator-gherkin-tests
branch
from
1962bd7 to
d43c66e
Compare
|
|
@aepfli Nice. I don't think this is really a feat for end-users. I think let's make it a chore/test? Also - should we run it in the CI? |
aepfli
changed the title
|
I agree with the pr title, and changed it
Maybe my knowledge is off, but we do not have the e2e tag, so this should execute with normal test execution, or am i wrong here? as it does not need docker, etc. it should be fast, and there is no reason to not run it during normal test execution. |
I don't think it's running. The submodule doesn't seem to be checked out in the CI and the coverage is exactly the same. |
|
I will put this onto my todo, as a lot of things have been moving currently, and i try to fix this by the end of next week. |
aepfli
force-pushed
the
feat/evaluator-gherkin-tests
branch
from
db0b3d4 to
7046251
Compare
Add in-process Gherkin tests for the JSON evaluator using godog and the shared test-harness evaluator feature files. Tests validate flag evaluation, error handling, targeting, metadata, fractional bucketing, semver, string operators, zero-values, evaluator refs, and no-default-variant behavior directly against the evaluator without testcontainers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Simon Schrottner <simon.schrottner@gmail.com>
The TestEvaluatorGherkin test depends on the test-harness submodule and silently skips when it is absent. The test job did not check out submodules, so the new gherkin tests never ran in CI. Signed-off-by: Simon Schrottner <simon.schrottner@gmail.com>
aepfli
force-pushed
the
feat/evaluator-gherkin-tests
branch
from
7046251 to
70c27e1
Compare
dosubot
Bot
added
size:S
|
2 participants
Summary
godogand the sharedtest-harness/evaluator/feature specs@fractional-v1excluded (flagd implements v2)test-evaluator-gherkinMakefile targetRelated: open-feature/python-sdk-contrib#377 (equivalent Python testkit)
Part of: open-feature/flagd-testbed#366 (centralized evaluator testkit migration)
See also: open-feature/flagd-testbed#367 (error handling discussion)
Test plan
make test-evaluator-gherkinpasses all 83 scenariosgo test -short ./core/pkg/evaluator/...)🤖 Generated with Claude Code