Add CI job for checking stdlib semver compatibility

[Kobzol]

View all comments

This CI job installs the latest published version of cargo-semver-checks and then runs the x test std-semver-check bootstrap command added in #159671.

If csc finds semver breakage, the job will fail. If csc cannot parse the rustdoc JSON data (e.g. because the format was bumped), the job will print an error, but it will not fail.

If a PR wants to explicitly go around the csc check (either there is an actual semver break or csc produces a false positive), if the src/bootstrap/stdlib-semver-check-stamp is changed in the same PR, then the check will be skipped

CI runs:

• cargo-semver-checks not being up-to-date result: job green
• Renaming an unstable stdlib function result: job green
• Renaming a stable stdlib function result: job red
• Renaming a stable stdlib function while bumping stdlib semver stamp result: job green

CC @rust-lang/libs @obi1kenobi

r? @jieyouxu (for bootstrap and CI stuff)

try-job: x86_64-gnu-stdlib-semver-check

[rustbot]

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

Warning

If you are changing how CI LLVM is built or linked, make sure to bump
src/bootstrap/download-ci-llvm-stamp.

cc @jieyouxu

This PR modifies bootstrap.example.toml.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

[Kobzol]

@bors try jobs=x86_64-gnu-stdlib-semver-check

[rust-bors]

💔 Test for 79f5ace failed: CI. Failed job:

• try - x86_64-gnu-stdlib-semver-check (web logs, enhanced plaintext logs)

[Kobzol]

@bors try jobs=x86_64-gnu-stdlib-semver-check

[rust-bors]

☀️ Try build successful (CI)
Build commit: cf27501 (cf2750129ae79b5df875523f88c46d6fe8a84b4c)
Base parent: 922325b (922325bb13bfea5b41454318563f2a65e83c2336)

[Urgau]

I expect quite a few semver breaking changes over the development of nightly features, how are contributors supposed to bypass the CI for changes that are okay? What about stable feature where the breakage is accepted?

[Kobzol]

That's a very good question, that I don't yet know the answer to :) In terms of implementation, we could have some file in the standard library subtree (something like the LLVM bootstrap stamp files) that could be manually bumped in PRs that expect a breaking change. And for those PRs the test would just be green.

[cuviper]

A file-based bypass sounds good to me.

[rust-bors]

💔 Test for 0bb0391 failed: CI. Failed job:

• try - x86_64-gnu-stdlib-semver-check (web logs, enhanced plaintext logs)

[Kobzol]

@bors try

[rust-bors]

💔 Test for 7920893 failed: CI. Failed job:

• try - x86_64-gnu-stdlib-semver-check (web logs, enhanced plaintext logs)

[Kobzol]

@bors try

[Kobzol]

@bors try

[rust-bors]

☀️ Try build successful (CI)
Build commit: 2d06403 (2d06403073949d757f25faf287ed7514be455eb7)
Base parent: 7c329d6 (7c329d6c76e11ca40c5673818ab0439c1be8962c)

[Kobzol]

Ok, should be ready now, including an opt-out mechanism. I updated the PR description and added CI links to the various cases that can happen.

@rustbot ready

[jieyouxu]

Some behavioral / dealing with job failure questions, but otherwise CI/bootstrap changes look good to me.

@rustbot author

View changes since this review

[jieyouxu]

Question: hm, is it possible that the JSON data remains borked but this swallows the problem? I assume that if std JSON data becomes borked there's additional mechanisms to catch that, or at least is possible for library maintainers to notice?

[obi1kenobi]

Unless the breakage is something super specific, like "only std is broken and nothing else", there are plenty of other things that could catch this including both rustdoc tests and cargo-semver-checks CI which runs vs nightly. std/core etc. aren't that special AFAIK in rustdoc JSON so the likelihood of such narrow breakage is very low.

In the reasonably near future cargo-semver-checks should start connecting rustdoc JSON for std/core/alloc together with the user's crate, at which point its CI will become even more attuned to finding possible breakage in rustdoc output.

So I think we're reasonably well set up for success here, and feel that more complexity here is probably not worth the possible payoff.

[jieyouxu]

Question: could we maybe provide a doc_url for how to handle failures from this job? Or alternatively, if it's possible to allow co-posting a more general "job failure help message".

[jieyouxu]

NB: Hm, this is an auto job. I wonder how often we would hit csc failures in practice for libs changes, I suppose we'll have to find out. But seems fine to start with this to try it out.

[obi1kenobi]

Based on my historical analysis, we expect about one instance of breakage per year. We may get a few more false-positives but I hope not many.

With that in mind, I think having even a pretty manual process like "post in this Zulip topic if you aren't sure what to do" should be okay, and I'd be happy to watch that topic and help people who might be stuck or confused.