test(ffe): wait for ready evaluation metric response

[leoromanovsky]

Motivation

FFE metric tests can observe Remote Config ACK before the tracer has installed the UFC payload locally. The sidecar ACK confirms delivery to the sidecar, not that a request thread will immediately evaluate against the new config.

That made Test_FFE_Eval_Metric_Basic too sensitive to startup timing: it could issue one evaluation immediately after RC setup, then assert metrics from a response that was still effectively not-ready/default behavior.

Changes

• Add a small helper that retries the basic string evaluation until the provider returns the expected ready variant.
• Assert the evaluation response is the expected non-default value before checking metrics.
• Select the metric series with the expected successful-result tags instead of taking the first metric for the flag.

Decisions

This is intentionally separate from #7033. #7033 can stay focused on enabling the PHP metric test in the manifest, while this PR makes the shared metric test resilient to the RC ACK/install timing boundary.

Related PRs

• PHP metric implementation: Add FFE evaluation metrics dd-trace-php#3911
• PHP metric system-test enablement: Enable PHP FFE evaluation metric system tests #7033

Validation

Static validation on this branch:

python -m ruff check tests/ffe/test_flag_eval_metrics.py
python -m ruff format --check tests/ffe/test_flag_eval_metrics.py
python -m mypy tests/ffe/test_flag_eval_metrics.py
DOCKER_CONFIG=/tmp/codex-docker-config-no-creds PATH=/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH ./format.sh

Result: all passed.

Behavior validation using the same test change while validating DataDog/dd-trace-php#3911 locally:

TEST_LIBRARY=php ./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_flag_eval_metrics.py

Result: 17 passed in 81.26s.

[github-actions]

CODEOWNERS have been resolved as:

tests/ffe/test_flag_eval_metrics.py                                     @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core

[datadog-prod-us1-3]

 

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 38 Pipeline jobs failed

Testing the test | System Tests (dotnet, prod) / End-to-end #1 / poc 1     

See error

1 failed test. KeyError: 'variant' in test_flag_eval_metrics.py:144

🧪 1 Test failed

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Basic.test_ffe_eval_metric_basic[poc] from system_tests_suite   (Fix with Cursor)

KeyError: 'variant'

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Basic object at 0x7fc4af93b950>

    def test_ffe_eval_metric_basic(self):
        """Test that flag evaluation produces a metric with correct tags."""
        assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}"
        result = json.loads(self.r.text)
>       assert result["variant"] == "on", f"Expected evaluated variant 'on', got response: {result}"
E       KeyError: 'variant'
...

Testing the test | System Tests (golang, dev) / End-to-end #1 / chi 1     

See error

1 test failed due to missing 'variant' key in response: Expected evaluated variant 'on', got response: {'status_code': 200}.

🧪 1 Test failed

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Basic.test_ffe_eval_metric_basic[chi] from system_tests_suite   (Fix with Cursor)

KeyError: 'variant'

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Basic object at 0x7f81b40fa1b0>

    def test_ffe_eval_metric_basic(self):
        """Test that flag evaluation produces a metric with correct tags."""
        assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}"
        result = json.loads(self.r.text)
>       assert result["variant"] == "on", f"Expected evaluated variant 'on', got response: {result}"
E       KeyError: 'variant'
...

Testing the test | System Tests (golang, dev) / End-to-end #1 / echo 1     

See error

1 failed test. KeyError: 'variant' in tests/ffe/test_flag_eval_metrics.py:144. Expected evaluated variant 'on', got response: {result}

🧪 1 Test failed

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Basic.test_ffe_eval_metric_basic[echo] from system_tests_suite   (Fix with Cursor)

KeyError: 'variant'

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Metric_Basic object at 0x7f09240a1160>

    def test_ffe_eval_metric_basic(self):
        """Test that flag evaluation produces a metric with correct tags."""
        assert self.r.status_code == 200, f"Flag evaluation failed: {self.r.text}"
        result = json.loads(self.r.text)
>       assert result["variant"] == "on", f"Expected evaluated variant 'on', got response: {result}"
E       KeyError: 'variant'
...

View all 38 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e188bb5 | Docs | Datadog PR Page | Give us feedback!}