SEP-1730 says Tier-1 requires "all conformance tests pass" / "100% compliance" but doesn't define whether SHOULD-level checks (which emit WARNING) count. Our own tooling disagrees with itself:
| Layer |
Behavior |
Location |
| SEP-1730 |
undefined |
seps/1730 |
tier-check |
WARNING ignored in pass_rate (only SUCCESS/FAILURE counted) |
src/tier-check/checks/test-conformance-results.ts:55-57 |
| CI runner |
warnings > 0 → overallFailure |
src/runner/client.ts:202 |
Today every Tier-1 SDK (Go, TypeScript, Python) shows 0 warnings. So the de-facto policy is "WARNINGs must be fixed," but tier-check would still award Tier-1 if one slipped through.
What a WARNING actually means here: since conformance runs against a maximally-configured everything-client, a WARNING signals "the SDK cannot be configured to satisfy this SHOULD" (missing API surface), not "a typical app skips it." That's a stronger statement than a SHOULD usually carries.
Decide and document one of:
- (a) Tier-1 requires zero WARNINGs → align
tier-check with client.ts:202; scenario authors keep using WARNING for SHOULD and it's effectively required
- (b) Tier-1 tolerates WARNINGs → drop
warnings > 0 from client.ts:202 overallFailure; WARNINGs become genuinely advisory
- (c) Distinguish "SDK can't" (FAILURE) from "SDK can but everything-client didn't opt in" (WARNING) — probably impractical
Since it's roughly "0 cost" to do it today, and having SHOULD checks be required for Tier 1 makes the ecosystem stronger, I'm leaning towards (a).
Whichever we pick should be recorded in AGENTS.md and potentially proposed as a clarification to SEP-1730.
Came up reviewing #200 (SEP-2164), where the same SHOULD-vs-FAILURE question determines whether a wrong error code blocks Tier-1. Related: #243.
SEP-1730 says Tier-1 requires "all conformance tests pass" / "100% compliance" but doesn't define whether SHOULD-level checks (which emit
WARNING) count. Our own tooling disagrees with itself:tier-checkpass_rate(only SUCCESS/FAILURE counted)src/tier-check/checks/test-conformance-results.ts:55-57warnings > 0→overallFailuresrc/runner/client.ts:202Today every Tier-1 SDK (Go, TypeScript, Python) shows 0 warnings. So the de-facto policy is "WARNINGs must be fixed," but
tier-checkwould still award Tier-1 if one slipped through.What a WARNING actually means here: since conformance runs against a maximally-configured everything-client, a WARNING signals "the SDK cannot be configured to satisfy this SHOULD" (missing API surface), not "a typical app skips it." That's a stronger statement than a SHOULD usually carries.
Decide and document one of:
tier-checkwithclient.ts:202; scenario authors keep using WARNING for SHOULD and it's effectively requiredwarnings > 0fromclient.ts:202overallFailure; WARNINGs become genuinely advisorySince it's roughly "0 cost" to do it today, and having SHOULD checks be required for Tier 1 makes the ecosystem stronger, I'm leaning towards (a).
Whichever we pick should be recorded in AGENTS.md and potentially proposed as a clarification to SEP-1730.
Came up reviewing #200 (SEP-2164), where the same SHOULD-vs-FAILURE question determines whether a wrong error code blocks Tier-1. Related: #243.