Store LiveLoans more densely packed by panstromek · Pull Request #161850 · rust-lang/rust

panstromek · 2026-08-27T05:33:11Z

LiveLoans originally used SparseBitMatrix<PointIndex, BorrowIndex> in a way that didn't use "the sparseness" very effectively. The borrow dimension is usually very low (<10 bits 90% of the time), while the point dimension is typically order of magnitude higher.

This means that we used to allocate a bunch of 32-byte DenseBitSets that only point to a few bits in heap-allocated u64. We only saved some memory by not allocating the whole PointIndex range, but we still often need to allocatate most of it.

This PR changes the representation to use a single flat representation with GrowableBitSet storing all bits in a single Vec. There's still a room for improvement but this should be a positive step forward.

r? lqd

panstromek · 2026-08-27T05:36:42Z

@bors try @rust-timer queue

Store LiveLoans more densely packed

rust-bors · 2026-08-27T06:42:12Z

☀️ Try build successful (CI)
Build commit: 2b3c53b (2b3c53be5d80c9b8d0ffa557c1ee46df8bb6fee3)
Base parent: d9dd070 (d9dd0703ba332dbdd19ae6c9bf7ab98ab7c933c7)

panstromek · 2026-08-27T07:03:02Z

Actually, this might be better stored with dimensions swapped after all. I didn't count it precisely yet, but it looks like there's usually more zeros at a few columns on the right, while last row ~~almost~~ always has some ones.e.g.

live_loans (pointIndex x borrowIndex)
000000000001111111111111000000
000000000000011111111111000000
000000000000000111111111000000

There's also the tradeoff with iteration, though - points are usually assigned in succession (i.e. the iteration is for each borrow { for each point { } }, so this layout might be better for that.

Also , storing this as something packed in the spirit of IndexVec<BorrowIndex, IntervalSet> might be even better. But since this is an improvement on its own, I'd rather iterate on these in a followup.

rust-timer · 2026-08-27T07:25:13Z

Finished benchmarking commit (2b3c53b): comparison URL.

Overall result: ✅ improvements - no action needed

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.4%	[-0.6%, -0.2%]	9
Improvements ✅ (secondary)	-0.3%	[-0.3%, -0.2%]	3
All ❌✅ (primary)	-0.4%	[-0.6%, -0.2%]	9

Max RSS (memory usage)

Results (primary 3.1%, secondary 7.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	4.8%	[2.6%, 10.3%]	4
Regressions ❌ (secondary)	8.9%	[3.0%, 13.3%]	6
Improvements ✅ (primary)	-3.8%	[-3.8%, -3.8%]	1
Improvements ✅ (secondary)	-2.4%	[-2.4%, -2.4%]	1
All ❌✅ (primary)	3.1%	[-3.8%, 10.3%]	5

Cycles

Results (primary -2.4%, secondary -4.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.4%	[2.1%, 2.5%]	4
Improvements ✅ (primary)	-2.4%	[-2.4%, -2.4%]	1
Improvements ✅ (secondary)	-12.7%	[-18.7%, -3.5%]	3
All ❌✅ (primary)	-2.4%	[-2.4%, -2.4%]	1

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 475.228s -> 473.78s (-0.30%)
Artifact size: 402.70 MiB -> 402.62 MiB (-0.02%)

panstromek · 2026-08-27T07:52:09Z

+pub(crate) struct LiveLoans {
+    num_points: usize,
+    // This matrix always has more rows (PointIndex) than columns (BorrowIndex), so we store
+    // it row major to save memory when later columns are unused


I don't think this is row-major actually, I never remember which is which, I'll fix up the comment after review.

View changes since the review

Yes, this is column-major. I'm not sure being technical here helps: I'd just say we store as a flat matrix grouped by columns.

We don't need to mention row-major or column-major, yes. We do need to better describe how the data is stored though, a growable bitset of usizes is not clear enough.

panstromek · 2026-08-27T07:52:53Z


-pub(crate) type LiveLoans = SparseBitMatrix<PointIndex, BorrowIndex>;
+#[derive(Clone)]
+pub(crate) struct LiveLoans {


Not sure if it's worth making this a broadly available datastructure or if it's enough to keep it here for now?

View changes since the review

I think leaving this here for now is fine.

It's only used in a small part of borrowck, and we will have to tweak it in the future anyways: the current behavior of filling this depends on how the graph is traversed. When traversing multiple loans at a time, the exact iteration and filling behavior will be different, and the best live loans representation will likely need to be reinvestigated. This matches your comment

There's also the tradeoff with iteration, though

panstromek · 2026-08-27T07:54:34Z

        &lowered_constraints,
    );

+    let num_points = location_map.num_points();


I'm not sure where is the right place to get this information from location_map. The way location_map is threaded through the code and the fact that it's in Rc makes it unclear at what point in time it is ready to use.

View changes since the review

This is fine. I was realizing too that currently things are all jumbled up and it might make sense to take a cleanup pass at some point.

I end up touching this in #161938 anyways, so not a big deal.

jackh726

This general shape is good and in line with what I'm already working on.

I think the key question here is if we want a DenseBitSet, GrowableBitSet, or MixedBitSet. We know the max size up front, and I doubt that later-indexed borrows go unused, so I doubt GrowableBitSet makes sense. MixedBitSet may be good if there are memory savings can outweigh the additional computational overhead.

(edit: local measurements have DenseBitSet better than the other two pretty convincingly)

I'm fine with whatever we decide here, if we have numbers that justify a benefit over the status quo - we can always optimize more later. (I think DenseBitSet is a safe choice for now.)

r=me with review addressed

View changes since this review

jackh726 · 2026-08-31T02:12:01Z

+    num_points: usize,
+    // This matrix always has more rows (PointIndex) than columns (BorrowIndex), so we store
+    // it row major to save memory when later columns are unused
+    flat_matrix: GrowableBitSet<usize>,


This could/should be a DenseBitSetof num_points * borrow_set.len()? I'm not sure that making this growable will give us meaningful memory benefits.

I think just converting this into a dense, flat set gives significant performance wins.

(That is also what I'm getting to, independently.)

Possibly, using a MixedBitSet/ChunkedBitSet could have a better chance at reducing memory overhead when points for a given borrow are stored next to each other, since I generally would expect loans to occupy some subset of the fn.

In practice, I'm not sure this actually will end up being very helpful. Since I think we want to optimize for speed here, not memory. And these two will have a much larger computational overhead.

jackh726 · 2026-08-31T03:43:27Z

+pub(crate) struct LiveLoans {
+    num_points: usize,
+    // This matrix always has more rows (PointIndex) than columns (BorrowIndex), so we store
+    // it row major to save memory when later columns are unused


Yes, this is column-major. I'm not sure being technical here helps: I'd just say we store as a flat matrix grouped by columns.

jackh726 · 2026-08-31T03:44:16Z


-pub(crate) type LiveLoans = SparseBitMatrix<PointIndex, BorrowIndex>;
+#[derive(Clone)]
+pub(crate) struct LiveLoans {


I think leaving this here for now is fine.

jackh726 · 2026-08-31T03:47:55Z

        &lowered_constraints,
    );

+    let num_points = location_map.num_points();


This is fine. I was realizing too that currently things are all jumbled up and it might make sense to take a cleanup pass at some point.

I end up touching this in #161938 anyways, so not a big deal.

jackh726 · 2026-08-31T03:56:08Z

r? me

lqd · 2026-08-31T08:00:16Z

r? me

lqd · 2026-08-31T08:47:40Z

Nice! Thanks for taking a look a this.

It's unclear to me why a growable bitset would would be better in general. The density and loan reachability values should be dependent on the CFG and outlives graph shapes, and ofc the actual region liveness -- which feels unpredicatable to me, though I guess you could make assumptions that most code is straight-line code, and thus there could be a bias towards loans introduced late in the function being less likely to be live at the beginning of the function). I also don't believe our existing benchmarks are representative of the cases where this structure would matter in practice. I've seen big outliers where we could test this, and the other kinds of the inner bitset you're changing to a growable one. Do you see cpu/memory improvements to http-0.2.0, rust-gate-0.1.1, panda-re-macros-0.26.0, tld-2.37.0 with this PR (or using a dense bitset as jack suggested)?

Having a points-per-loan bitset (instead of the current loans-per-point) is what the non-lazy loan traversal used to have as well. I'm pretty sure we'll need to do something cleverer once we propagate multiple loans at a time, to be able to exploit sparseness better for both the CFG points and loans themselves.

panstromek · 2026-08-31T10:03:22Z

It's unclear to me why a growable bitset would would be better in general.

Like I said in #161850 (comment), it's probably not better if we store it like this, because every borrow has at least one 1 point set, so this allocates almost always close to what DenseBitSet would allocate, just piecewise, so it makes sense to just change it to DenseBitSet if we keep the row/columns the same.

I made the assumption from printing out dimensions, but later when I actually printed out the whole matrices, it became clear that we'd save more space if we flip rows and columns, which would match the assumption of the original code (btw. I assigned you (lqd) because you intruduced the SparseBitMatrix here, so I assumed you made the original assumption - namely that later point indexes are almost always empty).

Do you see cpu/memory improvements to http-0.2.0, rust-gate-0.1.1, panda-re-macros-0.26.0, tld-2.37.0 with this PR (or using a dense bitset as jack suggested)?

I don't have these set up locally, but I can try later. Jack has http IIRC, so I assume he tested on that. I test on serde and serde-derive since they regressed the most with polonius.

I attached the print outs for serde if you want to get a quick idea for how these look like. I think it's pretty clear that there's more room for improvement (probably something with intervals). serde-derive has a few pretty massive ones (and the whole print file has 99MB, so I didn't include that one).

eprintln-Id-serde-1.0.219-Check-Full-Threads1.txt

rust-bors · 2026-08-31T18:10:48Z

✌️ @panstromek, you can now approve this pull request!

If @jackh726 told you to "r=me" after making some further change, then please make that change and post @bors r=jackh726.

View changes since this delegation.

lqd · 2026-08-31T19:04:19Z

Do you want me to test this before merge?

No worries, I will do so myself tomorrow with the try build artifacts.

(though, @lqd re-assigned himself as a reviewer, so probably should wait for him to sign off on this; but maybe he does before your perf run lands)

I had already started looking at this before you removed me, but will wait for the perf run results and the quick tests I mentioned above yes.

rust-timer · 2026-08-31T19:16:53Z

Finished benchmarking commit (a63776d): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.1%	[0.1%, 0.1%]	2
Improvements ✅ (primary)	-0.4%	[-1.0%, -0.2%]	18
Improvements ✅ (secondary)	-0.3%	[-0.4%, -0.2%]	4
All ❌✅ (primary)	-0.4%	[-1.0%, -0.2%]	18

Max RSS (memory usage)

Results (primary -3.0%, secondary 4.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	4.0%	[3.8%, 4.0%]	3
Improvements ✅ (primary)	-3.0%	[-3.2%, -2.7%]	3
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-3.0%	[-3.2%, -2.7%]	3

Cycles

Results (primary -2.2%, secondary -4.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.2%	[-2.4%, -2.1%]	2
Improvements ✅ (secondary)	-4.2%	[-6.6%, -2.0%]	9
All ❌✅ (primary)	-2.2%	[-2.4%, -2.1%]	2

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 475.395s -> 474.673s (-0.15%)
Artifact size: 401.38 MiB -> 400.56 MiB (-0.20%)

lqd · 2026-09-01T08:44:37Z

Looking better than the previous run, and even better in the benchmarks I mentioned:

http-0.2.0: no change
rusty-gate-0.1.1: 16% win on a clean build, 25% win on the leaf crate
panda-re-macros-0.26.0: 5% win on a clean build, 11% win on the leaf crate
tld-2.37.0: 2% win on a clean build, 3% win on the leaf crate

We can slightly improve the comments in the future, and this is already great as-is 🚀

@bors r=lqd,jackh726

rust-bors · 2026-09-01T08:44:39Z

📌 Commit 01a08cb has been approved by lqd,jackh726

It is now in the queue for this repository.

rustbot · 2026-09-01T17:04:48Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

panstromek · 2026-09-01T17:06:38Z

@rustbot ready

Let's see whether the delegation still applies.

@bors r=lqd,jackh726

rust-bors · 2026-09-01T17:06:41Z

📌 Commit e515460 has been approved by lqd,jackh726

It is now in the queue for this repository.

JonathanBrouwer · 2026-09-02T10:45:45Z

@bors rollup=iffy
Creating a rollup of perf-sensitive PRs

…uwer Rollup of 3 perf-sensitive pull requests Successful merges: - #161850 (Store LiveLoans more densely packed) - #162031 (Reduce next-solver memory usage by interning CanonicalQueryInput) - #162047 (Optimize empty token streams)

Rollup merge of #161850 - panstromek:SparseBitMatrix, r=lqd,jackh726 Store LiveLoans more densely packed `LiveLoans` originally used `SparseBitMatrix<PointIndex, BorrowIndex>` in a way that didn't use "the sparseness" very effectively. The borrow dimension is usually very low (<10 bits 90% of the time), while the point dimension is typically order of magnitude higher. This means that we used to allocate a bunch of 32-byte `DenseBitSet`s that only point to a few bits in heap-allocated `u64`. We only saved some memory by not allocating the whole `PointIndex` range, but we still often need to allocatate most of it. This PR changes the representation to use a single flat representation with `GrowableBitSet` storing all bits in a single Vec. There's still a room for improvement but this should be a positive step forward. r? lqd

rust-timer · 2026-09-02T19:43:15Z

Note

This PR was benchmarked as part of triage of its containing rollup: triage URL.

Finished benchmarking commit (a9c8aa3): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.4%	[-1.2%, -0.1%]	22
Improvements ✅ (secondary)	-0.2%	[-0.4%, -0.0%]	7
All ❌✅ (primary)	-0.4%	[-1.2%, -0.1%]	22

Max RSS (memory usage)

Results (primary -1.7%, secondary 0.5%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.6%	[2.6%, 2.6%]	1
Regressions ❌ (secondary)	4.0%	[3.3%, 4.7%]	3
Improvements ✅ (primary)	-3.1%	[-4.1%, -2.5%]	3
Improvements ✅ (secondary)	-3.0%	[-3.1%, -2.8%]	3
All ❌✅ (primary)	-1.7%	[-4.1%, 2.6%]	4

Cycles

Results (primary 2.3%, secondary 5.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.3%	[2.3%, 2.3%]	1
Regressions ❌ (secondary)	5.0%	[2.9%, 7.6%]	7
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.3%	[2.3%, 2.3%]	1

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: missing data
Artifact size: 400.83 MiB -> 401.38 MiB (0.14%)

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Aug 27, 2026

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 27, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Aug 27, 2026

Auto merge of #161850 - panstromek:SparseBitMatrix, r=<try>

2b3c53b

Store LiveLoans more densely packed

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 27, 2026

panstromek marked this pull request as ready for review August 27, 2026 07:50

rustbot assigned lqd Aug 27, 2026

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 27, 2026

panstromek commented Aug 27, 2026

View reviewed changes

This was referenced Aug 29, 2026

Use Box<[Word]> for word storage in DenseBitSet #161957

Open

Store locals in borrowck in BitSets #161930

Draft

jackh726 approved these changes Aug 31, 2026

View reviewed changes

rustbot assigned jackh726 and unassigned lqd Aug 31, 2026

rustbot assigned lqd and unassigned jackh726 Aug 31, 2026

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Aug 31, 2026

This comment has been minimized.

Sign in to view

Store LiveLoans more densely packed

e515460

panstromek force-pushed the SparseBitMatrix branch from 01a08cb to e515460 Compare September 1, 2026 17:04

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 1, 2026

rust-bors Bot added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Sep 1, 2026

JonathanBrouwer mentioned this pull request Sep 2, 2026

Rollup of 3 perf-sensitive pull requests #162183

Merged

rust-bors Bot merged commit 558ae12 into rust-lang:main Sep 2, 2026
13 checks passed

rustbot removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 2, 2026

rustbot added this to the 1.100.0 milestone Sep 2, 2026

rustbot removed the perf-regression Performance regression. label Sep 2, 2026

Uh oh!

Uh oh!

Conversation

panstromek commented Aug 27, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

panstromek commented Aug 27, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-bors Bot commented Aug 27, 2026

Uh oh!

This comment has been minimized.

panstromek commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rust-timer commented Aug 27, 2026

Overall result: ✅ improvements - no action needed

Uh oh!

panstromek Aug 27, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lqd Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

panstromek Aug 27, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lqd Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

panstromek Aug 27, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackh726 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackh726 commented Aug 31, 2026

Uh oh!

lqd commented Aug 31, 2026

Uh oh!

lqd commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

panstromek commented Aug 31, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented Aug 31, 2026

Uh oh!

lqd commented Aug 31, 2026

Uh oh!

rust-timer commented Aug 31, 2026

Overall result: ❌✅ regressions and improvements - please read:

Uh oh!

panstromek commented Aug 27, 2026 •

edited by rustbot

Loading

panstromek commented Aug 27, 2026 •

edited

Loading

panstromek Aug 27, 2026 •

edited by rustbot

Loading

lqd Aug 31, 2026 •

edited

Loading

panstromek Aug 27, 2026 •

edited by rustbot

Loading

lqd Aug 31, 2026 •

edited

Loading

panstromek Aug 27, 2026 •

edited by rustbot

Loading

jackh726 left a comment •

edited

Loading

lqd commented Aug 31, 2026 •

edited

Loading