diff --git a/src/doc/rustc-dev-guide/README.md b/src/doc/rustc-dev-guide/README.md index 2a686b5471c68..7ce7d4ddf3e7d 100644 --- a/src/doc/rustc-dev-guide/README.md +++ b/src/doc/rustc-dev-guide/README.md @@ -70,46 +70,21 @@ $ ENABLE_LINKCHECK=1 mdbook serve We use `mdbook-toc` to auto-generate TOCs for long sections. You can invoke the preprocessor by including the `` marker at the place where you want the TOC. -## How to fix toolstate failures - -> [!NOTE] -> Currently, we do not track the rustc-dev-guide toolstate due to -> [spurious failures](https://github.com/rust-lang/rust/pull/71731), -> but we leave these instructions for when we do it again in the future. - -1. You will get a ping from the toolstate commit. e.g. https://github.com/rust-lang-nursery/rust-toolstate/commit/8ffa0e4c30ac9ba8546b7046e5c4ccc2b96ebdd4 - -2. The commit contains a link to the PR that caused the breakage. e.g. https://github.com/rust-lang/rust/pull/64321 - -3. If you go to that PR's thread, there is a post from bors with a link to the CI status: https://github.com/rust-lang/rust/pull/64321#issuecomment-529763807 - -4. Follow the check-actions link to get to the Actions page for that build - -5. There will be approximately 1 billion different jobs for the build. They are for different configurations and platforms. The rustc-dev-guide build only runs on the Linux x86_64-gnu-tools job. So click on that job in the list, which is about 60% down in the list. - -6. Click the Run build step in the job to get the console log for the step. - -7. Click on the log and Ctrl-f to get a search box in the log - -8. Search for rustc-dev-guide. This gets you to the place where the links are checked. It is usually ~11K lines into the log. - -9. Look at the links in the log near that point in the log - -10. Fix those links in the rustc-dev-guide (by making a PR in the rustc-dev-guide repo) - -11. Make a PR on the rust-lang/rust repo to update the rustc-dev-guide git submodule in src/docs/rustc-dev-guide. -To make a PR, the following steps are useful. - -```bash -# Assuming you already cloned the rust-lang/rust repo and you're in the correct directory -git submodule update --remote src/doc/rustc-dev-guide -git add -u -git commit -m "Update rustc-dev-guide" -# Note that you can use -i, which is short for --incremental, in the following command -./x test --incremental src/doc/rustc-dev-guide # This is optional and should succeed anyway -# Open a PR in rust-lang/rust -``` - -12. Wait for PR to merge - -Voilà! +## Synchronizing josh subtree with rustc + +This repository is linked to `rust-lang/rust` as a [josh](https://josh-project.github.io/josh/intro.html) subtree. You can use the following commands to synchronize the subtree in both directions. + +### Pull changes from `rust-lang/rust` into this repository +1) Checkout a new branch that will be used to create a PR into `rust-lang/rustc-dev-guide` +2) Run the pull command + ``` + $ cargo run --manifest-path josh-sync/Cargo.toml rustc-pull + ``` +3) Push the branch to your fork and create a PR into `rustc-dev-guide` + +### Push changes from this repository into `rust-lang/rust` +1) Run the push command to create a branch named `` in a `rustc` fork under the `` account + ``` + $ cargo run --manifest-path josh-sync/Cargo.toml rustc-push + ``` +2) Create a PR from `` into `rust-lang/rust` diff --git a/src/doc/rustc-dev-guide/josh-sync/Cargo.lock b/src/doc/rustc-dev-guide/josh-sync/Cargo.lock new file mode 100644 index 0000000000000..844518628c437 --- /dev/null +++ b/src/doc/rustc-dev-guide/josh-sync/Cargo.lock @@ -0,0 +1,430 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "anstream" +version = "0.6.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8acc5369981196006228e28809f761875c0327210a891e941f4c683b3a99529b" +dependencies = [ + "anstyle", + "anstyle-parse", + "anstyle-query", + "anstyle-wincon", + "colorchoice", + "is_terminal_polyfill", + "utf8parse", +] + +[[package]] +name = "anstyle" +version = "1.0.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "55cc3b69f167a1ef2e161439aa98aed94e6028e5f9a59be9a6ffb47aef1651f9" + +[[package]] +name = "anstyle-parse" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b2d16507662817a6a20a9ea92df6652ee4f94f914589377d69f3b21bc5798a9" +dependencies = [ + "utf8parse", +] + +[[package]] +name = "anstyle-query" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "79947af37f4177cfead1110013d678905c37501914fba0efea834c3fe9a8d60c" +dependencies = [ + "windows-sys 0.59.0", +] + +[[package]] +name = "anstyle-wincon" +version = "3.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2109dbce0e72be3ec00bed26e6a7479ca384ad226efdd66db8fa2e3a38c83125" +dependencies = [ + "anstyle", + "windows-sys 0.59.0", +] + +[[package]] +name = "anyhow" +version = "1.0.95" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "34ac096ce696dc2fcabef30516bb13c0a68a11d30131d3df6f04711467681b04" + +[[package]] +name = "bitflags" +version = "2.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b048fb63fd8b5923fc5aa7b340d8e156aec7ec02f0c78fa8a6ddc2613f6f71de" + +[[package]] +name = "cfg-if" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd" + +[[package]] +name = "clap" +version = "4.5.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3135e7ec2ef7b10c6ed8950f0f792ed96ee093fa088608f1c76e569722700c84" +dependencies = [ + "clap_builder", + "clap_derive", +] + +[[package]] +name = "clap_builder" +version = "4.5.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "30582fc632330df2bd26877bde0c1f4470d57c582bbc070376afcd04d8cb4838" +dependencies = [ + "anstream", + "anstyle", + "clap_lex", + "strsim", +] + +[[package]] +name = "clap_derive" +version = "4.5.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ac6a0c7b1a9e9a5186361f67dfa1b88213572f427fb9ab038efb2bd8c582dab" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "clap_lex" +version = "0.7.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f46ad14479a25103f283c0f10005961cf086d8dc42205bb44c46ac563475dca6" + +[[package]] +name = "colorchoice" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b63caa9aa9397e2d9480a9b13673856c78d8ac123288526c37d7839f2a86990" + +[[package]] +name = "directories" +version = "5.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a49173b84e034382284f27f1af4dcbbd231ffa358c0fe316541a7337f376a35" +dependencies = [ + "dirs-sys", +] + +[[package]] +name = "dirs-sys" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "520f05a5cbd335fae5a99ff7a6ab8627577660ee5cfd6a94a6a929b52ff0321c" +dependencies = [ + "libc", + "option-ext", + "redox_users", + "windows-sys 0.48.0", +] + +[[package]] +name = "getrandom" +version = "0.2.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4567c8db10ae91089c99af84c68c38da3ec2f087c3f82960bcdbf3656b6f4d7" +dependencies = [ + "cfg-if", + "libc", + "wasi", +] + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "is_terminal_polyfill" +version = "1.70.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7943c866cc5cd64cbc25b2e01621d07fa8eb2a1a23160ee81ce38704e97b8ecf" + +[[package]] +name = "josh-sync" +version = "0.1.0" +dependencies = [ + "anyhow", + "clap", + "directories", + "xshell", +] + +[[package]] +name = "libc" +version = "0.2.169" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b5aba8db14291edd000dfcc4d620c7ebfb122c613afb886ca8803fa4e128a20a" + +[[package]] +name = "libredox" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c0ff37bd590ca25063e35af745c343cb7a0271906fb7b37e4813e8f79f00268d" +dependencies = [ + "bitflags", + "libc", +] + +[[package]] +name = "option-ext" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d" + +[[package]] +name = "proc-macro2" +version = "1.0.92" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "37d3544b3f2748c54e147655edb5025752e2303145b5aefb3c3ea2c78b973bb0" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.38" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e4dccaaaf89514f546c693ddc140f729f958c247918a13380cccc6078391acc" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "redox_users" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba009ff324d1fc1b900bd1fdb31564febe58a8ccc8a6fdbb93b543d33b13ca43" +dependencies = [ + "getrandom", + "libredox", + "thiserror", +] + +[[package]] +name = "strsim" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" + +[[package]] +name = "syn" +version = "2.0.93" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c786062daee0d6db1132800e623df74274a0a87322d8e183338e01b3d98d058" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "unicode-ident" +version = "1.0.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "adb9e6ca4f869e1180728b7950e35922a7fc6397f7b641499e8f3ef06e50dc83" + +[[package]] +name = "utf8parse" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" + +[[package]] +name = "wasi" +version = "0.11.0+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423" + +[[package]] +name = "windows-sys" +version = "0.48.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "677d2418bec65e3338edb076e806bc1ec15693c5d0104683f2efe857f61056a9" +dependencies = [ + "windows-targets 0.48.5", +] + +[[package]] +name = "windows-sys" +version = "0.59.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b" +dependencies = [ + "windows-targets 0.52.6", +] + +[[package]] +name = "windows-targets" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a2fa6e2155d7247be68c096456083145c183cbbbc2764150dda45a87197940c" +dependencies = [ + "windows_aarch64_gnullvm 0.48.5", + "windows_aarch64_msvc 0.48.5", + "windows_i686_gnu 0.48.5", + "windows_i686_msvc 0.48.5", + "windows_x86_64_gnu 0.48.5", + "windows_x86_64_gnullvm 0.48.5", + "windows_x86_64_msvc 0.48.5", +] + +[[package]] +name = "windows-targets" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" +dependencies = [ + "windows_aarch64_gnullvm 0.52.6", + "windows_aarch64_msvc 0.52.6", + "windows_i686_gnu 0.52.6", + "windows_i686_gnullvm", + "windows_i686_msvc 0.52.6", + "windows_x86_64_gnu 0.52.6", + "windows_x86_64_gnullvm 0.52.6", + "windows_x86_64_msvc 0.52.6", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b38e32f0abccf9987a4e3079dfb67dcd799fb61361e53e2882c3cbaf0d905d8" + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc35310971f3b2dbbf3f0690a219f40e2d9afcf64f9ab7cc1be722937c26b4bc" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" + +[[package]] +name = "windows_i686_gnu" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a75915e7def60c94dcef72200b9a8e58e5091744960da64ec734a6c6e9b3743e" + +[[package]] +name = "windows_i686_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" + +[[package]] +name = "windows_i686_msvc" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f55c233f70c4b27f66c523580f78f1004e8b5a8b659e05a4eb49d4166cca406" + +[[package]] +name = "windows_i686_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53d40abd2583d23e4718fddf1ebec84dbff8381c07cae67ff7768bbf19c6718e" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b7b52767868a23d5bab768e390dc5f5c55825b6d30b86c844ff2dc7414044cc" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.48.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed94fce61571a4006852b7389a063ab983c02eb1bb37b47f8272ce92d06d9538" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" + +[[package]] +name = "xshell" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e7290c623014758632efe00737145b6867b66292c42167f2ec381eb566a373d" +dependencies = [ + "xshell-macros", +] + +[[package]] +name = "xshell-macros" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32ac00cd3f8ec9c1d33fb3e7958a82df6989c42d747bd326c822b1d625283547" diff --git a/src/doc/rustc-dev-guide/josh-sync/Cargo.toml b/src/doc/rustc-dev-guide/josh-sync/Cargo.toml new file mode 100644 index 0000000000000..81d0d1ebd22d8 --- /dev/null +++ b/src/doc/rustc-dev-guide/josh-sync/Cargo.toml @@ -0,0 +1,10 @@ +[package] +name = "josh-sync" +version = "0.1.0" +edition = "2021" + +[dependencies] +anyhow = "1.0.95" +clap = { version = "4.5.21", features = ["derive"] } +directories = "5" +xshell = "0.2.6" diff --git a/src/doc/rustc-dev-guide/josh-sync/README.md b/src/doc/rustc-dev-guide/josh-sync/README.md new file mode 100644 index 0000000000000..a3dd876e8b87b --- /dev/null +++ b/src/doc/rustc-dev-guide/josh-sync/README.md @@ -0,0 +1,4 @@ +# Git josh sync +This utility serves for syncing the josh git subtree to and from the rust-lang/rust repository. + +See CLI help for usage. diff --git a/src/doc/rustc-dev-guide/josh-sync/src/main.rs b/src/doc/rustc-dev-guide/josh-sync/src/main.rs new file mode 100644 index 0000000000000..84613ad86890e --- /dev/null +++ b/src/doc/rustc-dev-guide/josh-sync/src/main.rs @@ -0,0 +1,32 @@ +use clap::Parser; +use crate::sync::GitSync; + +mod sync; + +#[derive(clap::Parser)] +enum Args { + /// Pull changes from the main `rustc` repository. + /// This creates new commits that should be then merged into `rustc-dev-guide`. + RustcPull, + /// Push changes from `rustc-dev-guide` to the given `branch` of a `rustc` fork under the given + /// GitHub `username`. + /// The pushed branch should then be merged into the `rustc` repository. + RustcPush { + branch: String, + github_username: String + } +} + +fn main() -> anyhow::Result<()> { + let args = Args::parse(); + let sync = GitSync::from_current_dir()?; + match args { + Args::RustcPull => { + sync.rustc_pull(None)?; + } + Args::RustcPush { github_username, branch } => { + sync.rustc_push(github_username, branch)?; + } + } + Ok(()) +} diff --git a/src/doc/rustc-dev-guide/josh-sync/src/sync.rs b/src/doc/rustc-dev-guide/josh-sync/src/sync.rs new file mode 100644 index 0000000000000..26963d2e247fc --- /dev/null +++ b/src/doc/rustc-dev-guide/josh-sync/src/sync.rs @@ -0,0 +1,234 @@ +use std::ops::Not; +use std::path::PathBuf; +use std::{env, net, process}; +use std::io::Write; +use std::time::Duration; +use anyhow::{anyhow, bail, Context}; +use xshell::{cmd, Shell}; + +/// Used for rustc syncs. +const JOSH_FILTER: &str = ":/src/doc/rustc-dev-guide"; +const JOSH_PORT: u16 = 42042; +const UPSTREAM_REPO: &str = "kobzol/rust"; + +pub struct GitSync { + dir: PathBuf, +} + +/// This code was adapted from the miri repository +/// (https://github.com/rust-lang/miri/blob/6a68a79f38064c3bc30617cca4bdbfb2c336b140/miri-script/src/commands.rs#L236). +impl GitSync { + pub fn from_current_dir() -> anyhow::Result { + Ok(Self { + dir: std::env::current_dir()? + }) + } + + pub fn rustc_pull(&self, commit: Option) -> anyhow::Result<()> { + let sh = Shell::new()?; + sh.change_dir(&self.dir); + let commit = commit.map(Ok).unwrap_or_else(|| { + let rust_repo_head = + cmd!(sh, "git ls-remote https://github.com/{UPSTREAM_REPO}/ HEAD").read()?; + rust_repo_head + .split_whitespace() + .next() + .map(|front| front.trim().to_owned()) + .ok_or_else(|| anyhow!("Could not obtain Rust repo HEAD from remote.")) + })?; + // Make sure the repo is clean. + if cmd!(sh, "git status --untracked-files=no --porcelain").read()?.is_empty().not() { + bail!("working directory must be clean before performing rustc pull"); + } + // Make sure josh is running. + let josh = Self::start_josh()?; + let josh_url = + format!("http://localhost:{JOSH_PORT}/{UPSTREAM_REPO}.git@{commit}{JOSH_FILTER}.git"); + + // Update rust-version file. As a separate commit, since making it part of + // the merge has confused the heck out of josh in the past. + // We pass `--no-verify` to avoid running git hooks. + // We do this before the merge so that if there are merge conflicts, we have + // the right rust-version file while resolving them. + sh.write_file("rust-version", format!("{commit}\n"))?; + const PREPARING_COMMIT_MESSAGE: &str = "Preparing for merge from rustc"; + cmd!(sh, "git commit rust-version --no-verify -m {PREPARING_COMMIT_MESSAGE}") + .run() + .context("FAILED to commit rust-version file, something went wrong")?; + + // Fetch given rustc commit. + cmd!(sh, "git fetch {josh_url}") + .run() + .inspect_err(|_| { + // Try to un-do the previous `git commit`, to leave the repo in the state we found it. + cmd!(sh, "git reset --hard HEAD^") + .run() + .expect("FAILED to clean up again after failed `git fetch`, sorry for that"); + }) + .context("FAILED to fetch new commits, something went wrong (committing the rust-version file has been undone)")?; + + // This should not add any new root commits. So count those before and after merging. + let num_roots = || -> anyhow::Result { + Ok(cmd!(sh, "git rev-list HEAD --max-parents=0 --count") + .read() + .context("failed to determine the number of root commits")? + .parse::()?) + }; + let num_roots_before = num_roots()?; + + // Merge the fetched commit. + const MERGE_COMMIT_MESSAGE: &str = "Merge from rustc"; + cmd!(sh, "git merge FETCH_HEAD --no-verify --no-ff -m {MERGE_COMMIT_MESSAGE}") + .run() + .context("FAILED to merge new commits, something went wrong")?; + + // Check that the number of roots did not increase. + if num_roots()? != num_roots_before { + bail!("Josh created a new root commit. This is probably not the history you want."); + } + + drop(josh); + Ok(()) + } + + pub fn rustc_push(&self, github_user: String, branch: String) -> anyhow::Result<()> { + let sh = Shell::new()?; + sh.change_dir(&self.dir); + let base = sh.read_file("rust-version")?.trim().to_owned(); + // Make sure the repo is clean. + if cmd!(sh, "git status --untracked-files=no --porcelain").read()?.is_empty().not() { + bail!("working directory must be clean before running `rustc-push`"); + } + // Make sure josh is running. + let josh = Self::start_josh()?; + let josh_url = + format!("http://localhost:{JOSH_PORT}/{github_user}/rust.git{JOSH_FILTER}.git"); + + // Find a repo we can do our preparation in. + if let Ok(rustc_git) = env::var("RUSTC_GIT") { + // If rustc_git is `Some`, we'll use an existing fork for the branch updates. + sh.change_dir(rustc_git); + } else { + // Otherwise, do this in the local repo. + println!( + "This will pull a copy of the rust-lang/rust history into this checkout, growing it by about 1GB." + ); + print!( + "To avoid that, abort now and set the `RUSTC_GIT` environment variable to an existing rustc checkout. Proceed? [y/N] " + ); + std::io::stdout().flush()?; + let mut answer = String::new(); + std::io::stdin().read_line(&mut answer)?; + if answer.trim().to_lowercase() != "y" { + std::process::exit(1); + } + }; + // Prepare the branch. Pushing works much better if we use as base exactly + // the commit that we pulled from last time, so we use the `rust-version` + // file to find out which commit that would be. + println!("Preparing {github_user}/rust (base: {base})..."); + if cmd!(sh, "git fetch https://github.com/{github_user}/rust {branch}") + .ignore_stderr() + .read() + .is_ok() + { + println!( + "The branch '{branch}' seems to already exist in 'https://github.com/{github_user}/rust'. Please delete it and try again." + ); + std::process::exit(1); + } + cmd!(sh, "git fetch https://github.com/{UPSTREAM_REPO} {base}").run()?; + cmd!(sh, "git push https://github.com/{github_user}/rust {base}:refs/heads/{branch}") + .ignore_stdout() + .ignore_stderr() // silence the "create GitHub PR" message + .run()?; + println!(); + + // Do the actual push. + sh.change_dir(&self.dir); + println!("Pushing changes..."); + cmd!(sh, "git push {josh_url} HEAD:{branch}").run()?; + println!(); + + // Do a round-trip check to make sure the push worked as expected. + cmd!(sh, "git fetch {josh_url} {branch}").ignore_stderr().read()?; + let head = cmd!(sh, "git rev-parse HEAD").read()?; + let fetch_head = cmd!(sh, "git rev-parse FETCH_HEAD").read()?; + if head != fetch_head { + bail!( + "Josh created a non-roundtrip push! Do NOT merge this into rustc!\n\ + Expected {head}, got {fetch_head}." + ); + } + println!( + "Confirmed that the push round-trips back to rustc-dev-guide properly. Please create a rustc PR:" + ); + println!( + // Open PR with `subtree update` title to silence the `no-merges` triagebot check + " https://github.com/{UPSTREAM_REPO}/compare/{github_user}:{branch}?quick_pull=1&title=Rustc+dev+guide+subtree+update&body=r?+@ghost" + ); + + drop(josh); + Ok(()) + } + + fn start_josh() -> anyhow::Result { + // Determine cache directory. + let local_dir = { + let user_dirs = + directories::ProjectDirs::from("org", "rust-lang", "rustc-dev-guide-josh").unwrap(); + user_dirs.cache_dir().to_owned() + }; + + // Start josh, silencing its output. + let mut cmd = process::Command::new("josh-proxy"); + cmd.arg("--local").arg(local_dir); + cmd.arg("--remote").arg("https://github.com"); + cmd.arg("--port").arg(JOSH_PORT.to_string()); + cmd.arg("--no-background"); + cmd.stdout(process::Stdio::null()); + cmd.stderr(process::Stdio::null()); + let josh = cmd.spawn().context("failed to start josh-proxy, make sure it is installed")?; + + // Create a wrapper that stops it on drop. + struct Josh(process::Child); + impl Drop for Josh { + fn drop(&mut self) { + #[cfg(unix)] + { + // Try to gracefully shut it down. + process::Command::new("kill") + .args(["-s", "INT", &self.0.id().to_string()]) + .output() + .expect("failed to SIGINT josh-proxy"); + // Sadly there is no "wait with timeout"... so we just give it some time to finish. + std::thread::sleep(Duration::from_millis(100)); + // Now hopefully it is gone. + if self.0.try_wait().expect("failed to wait for josh-proxy").is_some() { + return; + } + } + // If that didn't work (or we're not on Unix), kill it hard. + eprintln!( + "I have to kill josh-proxy the hard way, let's hope this does not break anything." + ); + self.0.kill().expect("failed to SIGKILL josh-proxy"); + } + } + + // Wait until the port is open. We try every 10ms until 1s passed. + for _ in 0..100 { + // This will generally fail immediately when the port is still closed. + let josh_ready = net::TcpStream::connect_timeout( + &net::SocketAddr::from(([127, 0, 0, 1], JOSH_PORT)), + Duration::from_millis(1), + ); + if josh_ready.is_ok() { + return Ok(Josh(josh)); + } + // Not ready yet. + std::thread::sleep(Duration::from_millis(10)); + } + bail!("Even after waiting for 1s, josh-proxy is still not available.") + } +} diff --git a/src/doc/rustc-dev-guide/rust-version b/src/doc/rustc-dev-guide/rust-version new file mode 100644 index 0000000000000..56b1d45b66e19 --- /dev/null +++ b/src/doc/rustc-dev-guide/rust-version @@ -0,0 +1 @@ +20c99373f28ee0e093fd4f5f4979d3bfcd8548df diff --git a/src/doc/rustc-dev-guide/src/external-repos.md b/src/doc/rustc-dev-guide/src/external-repos.md index 533f7eb5e7344..a7ab3d773acb5 100644 --- a/src/doc/rustc-dev-guide/src/external-repos.md +++ b/src/doc/rustc-dev-guide/src/external-repos.md @@ -3,24 +3,24 @@ The `rust-lang/rust` git repository depends on several other repos in the `rust-lang` organization. There are three main ways we use dependencies: 1. As a Cargo dependency through crates.io (e.g. `rustc-rayon`) -2. As a git subtree (e.g. `clippy`) +2. As a git (e.g. `clippy`) or a [josh] (e.g. `miri`) subtree 3. As a git submodule (e.g. `cargo`) -As a general rule, use crates.io for libraries that could be useful for others in the ecosystem; use -subtrees for tools that depend on compiler internals and need to be updated if there are breaking -changes; and use submodules for tools that are independent of the compiler. +As a general rule: +- Use crates.io for libraries that could be useful for others in the ecosystem +- Use subtrees for tools that depend on compiler internals and need to be updated if there are breaking +changes +- Use submodules for tools that are independent of the compiler -## External Dependencies (subtree) +## External Dependencies (subtrees) -As a developer to this repository, you don't have to treat the following external projects -differently from other crates that are directly in this repo: +The following external projects are managed using some form of a `subtree`: -* [Clippy](https://github.com/rust-lang/rust-clippy) -* [Miri] +* [clippy](https://github.com/rust-lang/rust-clippy) +* [miri](https://github.com/rust-lang/miri) * [rustfmt](https://github.com/rust-lang/rustfmt) * [rust-analyzer](https://github.com/rust-lang/rust-analyzer) - -[Miri]: https://github.com/rust-lang/miri +* [rustc_codegen_cranelift](https://github.com/rust-lang/rustc_codegen_cranelift) In contrast to `submodule` dependencies (see below for those), the `subtree` dependencies are just regular files and directories which can @@ -29,6 +29,20 @@ to these tools should be filed against the tools directly in their respective upstream repositories. The exception is that when rustc changes are required to implement a new tool feature or test, that should happen in one collective rustc PR. +`subtree` dependencies are currently managed by two distinct approaches: + +* Using `git subtree` + * `clippy` ([sync guide](https://doc.rust-lang.org/nightly/clippy/development/infrastructure/sync.html#performing-the-sync-from-rust-langrust-to-clippy)) + * `rustfmt` + * `rustc_codegen_cranelift` ([sync script](https://github.com/rust-lang/rustc_codegen_cranelift/blob/113af154d459e41b3dc2c5d7d878e3d3a8f33c69/scripts/rustup.sh#L7)) +* Using the [josh] tool + * `miri` ([sync guide](https://github.com/rust-lang/miri/blob/master/CONTRIBUTING.md#advanced-topic-syncing-with-the-rustc-repo)) + * `rust-analyzer` ([sync script](https://github.com/rust-lang/rust-analyzer/blob/2e13684be123eca7181aa48e043e185d8044a84a/xtask/src/release.rs#L147)) + +The [josh] tool is an alternative to git subtrees, which manages git history in a different way and scales better to larger repositories. Specific tooling is required to work with josh, you can check out the `miri` or `rust-analyzer` scripts linked above for inspiration. If you want to migrate a subtree from `git subtree` to josh, you can check out [this guide](https://hackmd.io/7pOuxnkdQDaL1Y1FQr65xg). + +Below you can find a guide on how to perform push and pull synchronization with the main rustc repo using `git subtree`, although these instructions might differ repo from repo. + ### Synchronizing a subtree Periodically the changes made to subtree based dependencies need to be synchronized between this @@ -84,7 +98,6 @@ Now you're done, the `src/tools/clippy` directory behaves as if Clippy were part of the rustc monorepo, so no one but you (or others that synchronize subtrees) actually needs to use `git subtree`. - ## External Dependencies (submodules) Building Rust will also use external git repositories tracked using [git @@ -111,3 +124,4 @@ the week leading up to the beta cut. [The Rust Reference]: https://github.com/rust-lang/reference/ [toolstate website]: https://rust-lang-nursery.github.io/rust-toolstate/ [Toolstate chapter]: https://forge.rust-lang.org/infra/toolstate.html +[josh]: https://josh-project.github.io/josh/intro.html diff --git a/src/doc/rustc-dev-guide/src/method-lookup.md b/src/doc/rustc-dev-guide/src/method-lookup.md index 8b49e8d004b17..c8d529a32b532 100644 --- a/src/doc/rustc-dev-guide/src/method-lookup.md +++ b/src/doc/rustc-dev-guide/src/method-lookup.md @@ -67,6 +67,7 @@ imported to use an inherent method, they are associated with the type itself (note that inherent impls can only be defined in the same crate as the type itself). + **Extension candidates** are derived from imported traits. If I have the trait `ToString` imported, and I call `to_string()` as a method, diff --git a/src/doc/rustc-dev-guide/src/solve/normalization.md b/src/doc/rustc-dev-guide/src/solve/normalization.md index 8858258b54269..99dc20c46b5d4 100644 --- a/src/doc/rustc-dev-guide/src/solve/normalization.md +++ b/src/doc/rustc-dev-guide/src/solve/normalization.md @@ -1,5 +1,7 @@ # Normalization in the new solver +> FIXME: Normalization has been changed significantly since this chapter was written. + With the new solver we've made some fairly significant changes to normalization when compared to the existing implementation. @@ -52,12 +54,14 @@ before assigning the resulting rigid type to an inference variable. This is used This has to be used whenever we match on the value of some type, both inside and outside of the trait solver. + [structural_norm]: https://github.com/rust-lang/rust/blob/2627e9f3012a97d3136b3e11bf6bd0853c38a534/compiler/rustc_trait_selection/src/solve/alias_relate.rs#L140-L175 [structural-relate]: https://github.com/rust-lang/rust/blob/a0569fa8f91b5271e92d2f73fd252de7d3d05b9c/compiler/rustc_trait_selection/src/solve/alias_relate.rs#L88-L107 @@ -72,11 +76,13 @@ possible. However, this only works for aliases referencing bound variables if th not ambiguous as we're unable to replace the alias with a corresponding inference variable without leaking universes. + [generalize-no-alias]: https://github.com/rust-lang/rust/blob/a0569fa8f91b5271e92d2f73fd252de7d3d05b9c/compiler/rustc_infer/src/infer/relate/generalize.rs#L353-L358 diff --git a/src/doc/rustc-dev-guide/src/solve/opaque-types.md b/src/doc/rustc-dev-guide/src/solve/opaque-types.md index 87531705c2da4..672aab7708018 100644 --- a/src/doc/rustc-dev-guide/src/solve/opaque-types.md +++ b/src/doc/rustc-dev-guide/src/solve/opaque-types.md @@ -60,6 +60,7 @@ Finally, we check whether the item bounds of the opaque hold for the expected ty [eq-prev]: https://github.com/rust-lang/rust/blob/384d26fc7e3bdd7687cc17b2662b091f6017ec2a/compiler/rustc_trait_selection/src/solve/normalizes_to/opaque_types.rs#L51-L59 [insert-storage]: https://github.com/rust-lang/rust/blob/384d26fc7e3bdd7687cc17b2662b091f6017ec2a/compiler/rustc_trait_selection/src/solve/normalizes_to/opaque_types.rs#L68 [item-bounds-ck]: https://github.com/rust-lang/rust/blob/384d26fc7e3bdd7687cc17b2662b091f6017ec2a/compiler/rustc_trait_selection/src/solve/normalizes_to/opaque_types.rs#L69-L74 + [^1]: FIXME: this should ideally only result in a unique candidate given that we require the args to be placeholders and regions are always inference vars [^2]: FIXME: why do we check whether the expected type is rigid for this. @@ -101,6 +102,7 @@ The handling of member constraints does not change in the new solver. See the FIXME: We need to continue to support calling methods on still unconstrained opaque types in their defining scope. It's unclear how to best do this. + ```rust use std::future::Future; use futures::FutureExt; diff --git a/src/doc/rustc-dev-guide/src/tests/compiletest.md b/src/doc/rustc-dev-guide/src/tests/compiletest.md index 17106be46744c..459c082906eba 100644 --- a/src/doc/rustc-dev-guide/src/tests/compiletest.md +++ b/src/doc/rustc-dev-guide/src/tests/compiletest.md @@ -441,6 +441,46 @@ $ COMPILETEST_FORCE_STAGE0=1 x test --stage 0 tests/run-make/ Of course, some tests will not successfully *run* in this way. +#### Using rust-analyzer with `rmake.rs` + +Like other test programs, the `rmake.rs` scripts used by run-make tests do not +have rust-analyzer integration by default. + +To work around this when working on a particular test, temporarily create a +`Cargo.toml` file in the test's directory +(e.g. `tests/run-make/sysroot-crates-are-unstable/Cargo.toml`) +with these contents: + +
+Be careful not to add this `Cargo.toml` or its `Cargo.lock` to your actual PR! +
+ +```toml +# Convince cargo that this isn't part of an enclosing workspace. +[workspace] + +[package] +name = "rmake" +version = "0.1.0" +edition = "2021" + +[dependencies] +run_make_support = { path = "../../../src/tools/run-make-support" } + +[[bin]] +name = "rmake" +path = "rmake.rs" +``` + +Then add a corresponding entry to `"rust-analyzer.linkedProjects"` +(e.g. in `.vscode/settings.json`): + +```json +"rust-analyzer.linkedProjects": [ + "tests/run-make/sysroot-crates-are-unstable/Cargo.toml" +], +``` + #### Using Makefiles (legacy)
diff --git a/src/doc/rustc-dev-guide/src/tests/directives.md b/src/doc/rustc-dev-guide/src/tests/directives.md index b0527da7bf564..69f4c864186cf 100644 --- a/src/doc/rustc-dev-guide/src/tests/directives.md +++ b/src/doc/rustc-dev-guide/src/tests/directives.md @@ -2,7 +2,9 @@ -> **FIXME(jieyouxu)** completely revise this chapter. + Directives are special comments that tell compiletest how to build and interpret a test. They must appear before the Rust source in the test. They may also @@ -248,10 +250,11 @@ Consider writing the test as a proper incremental test instead. |-------------|--------------------------------------------------------------|------------------------------------------|---------------------------| | `doc-flags` | Flags passed to `rustdoc` when building the test or aux file | `rustdoc`, `js-doc-test`, `rustdoc-json` | Any valid `rustdoc` flags | -> **FIXME(rustdoc)**: what does `check-test-line-numbers-match` do? -> -> Asked in -> . + ### Pretty printing diff --git a/src/doc/rustc-dev-guide/src/traits/chalk.md b/src/doc/rustc-dev-guide/src/traits/chalk.md index 78deb367506c5..844f42b9879f8 100644 --- a/src/doc/rustc-dev-guide/src/traits/chalk.md +++ b/src/doc/rustc-dev-guide/src/traits/chalk.md @@ -10,7 +10,7 @@ stream and say hello! [Types team]: https://github.com/rust-lang/types-team [`#t-types`]: https://rust-lang.zulipchat.com/#narrow/stream/144729-t-types -The new-style trait solver is based on the work done in [chalk][chalk]. Chalk +The new-style trait solver is based on the work done in [chalk]. Chalk recasts Rust's trait system explicitly in terms of logic programming. It does this by "lowering" Rust code into a kind of logic program we can then execute queries against. @@ -30,7 +30,7 @@ You can read more about chalk itself in the ## Ongoing work The design of the new-style trait solving happens in two places: -**chalk**. The [chalk][chalk] repository is where we experiment with new ideas +**chalk**. The [chalk] repository is where we experiment with new ideas and designs for the trait system. **rustc**. Once we are happy with the logical rules, we proceed to