Improve source indexer deduplication to prefer real implementation over *.notsupported.cs files by Copilot · Pull Request #192 · dotnet/source-indexer

Copilot · 2025-07-22T16:27:32Z

This PR addresses the issue where https://source.dot.net/ shows platform-agnostic source code containing mostly empty classes that throw PlatformNotSupportedException instead of the actual platform-specific implementation files.

Problem

The current deduplication logic in BinLogToSln/Program.cs was too simplistic, using only file path-based deduplication. When multiple builds existed for the same assembly (e.g., generic + Linux-specific), it would select whichever was processed first, often resulting in .notsupported.cs stub files being chosen over real implementation files.

Solution

Implemented improved deduplication logic that:

Groups by Assembly Name: All compiler invocations are now grouped by AssemblyName instead of simple file path deduplication
Scores Each Build: Uses a 5-tier scoring system to select the "best" build based on:
- UseForSourceIndex property: int.MaxValue score for builds explicitly marked for indexing
- Platform support status: -10,000 penalty for IsPlatformNotSupportedAssembly builds
- Framework version: Higher scores for newer frameworks (major1000 + minor100)
- Platform specificity: +500 bonus for platform-specific frameworks, with additional bonuses for Linux (+100) and Unix (+50)
- Source file count: +1 per source file as a tiebreaker
Selects Best Build: For each assembly, only the highest-scored build is included in the solution

Documentation and Testing

Documentation: Added comprehensive documentation in docs/source-selection-algorithm.md explaining the scoring priorities, examples with concrete scores, and configuration options
Unit Tests: Added BinLogToSln.Tests project with 12 test methods covering all scoring scenarios, edge cases, and priority ordering to protect against regressions

Example

Before this change, if there were two builds for System.Net.NameResolution:

Generic build: System.Net.NameResolution.notsupported.cs (score: -1,955)
Linux build: actual implementation files (score: 8,727)

The tool would pick whichever was processed first. Now it consistently selects the Linux build with real implementation.

The changes maintain backward compatibility while significantly improving source quality on https://source.dot.net/.

Fixes #188.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

…er *.notsupported.cs Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

…utput Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

ericstj

@copilot please try plumbing project properties from the binlog as suggested in the feedback.

Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Copilot · 2025-07-22T17:26:27Z

@copilot please try plumbing project properties from the binlog as suggested in the feedback.

Implemented project properties plumbing as requested. The BinLogReader now captures project properties from the binlog and stores them in a ProjectProperties dictionary on CompilerInvocation. The scoring logic uses these properties instead of file/command-line conventions. Commit c59f2a1

ericstj

@copilot give this a try.

…rder scoring priorities Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Copilot · 2025-07-23T22:49:17Z

@copilot give this a try.

Implemented all the requested changes! The BinLogReader now optimizes project access, the scoring system follows the proper priority hierarchy, and platform-specific precedence has been removed. All builds are passing. Commit 56384ee

…BinLogReader Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Copilot · 2025-07-24T02:51:53Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

https://api.github.com/repos/KirillOsenkov/MSBuildStructuredLog/commits/00926a0debc2facd65611f68f91adafe0d56ab14
- Triggering command: curl -s REDACTED (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

This reverts commit 41af551.

joperezr

LGTM barring those two last comments I left. Thanks @ericstj

ericstj · 2025-08-15T00:01:02Z

@copilot can you address @joperezr's feedback?

Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Copilot · 2025-08-15T00:12:37Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

dnceng.pkgs.visualstudio.com
- Triggering command: dotnet build src/SourceBrowser/src/BinLogToSln/BinLogToSln.csproj (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

…t score values Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Copilot · 2025-08-15T16:21:14Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

dnceng.pkgs.visualstudio.com
- Triggering command: dotnet test src/SourceBrowser/src/BinLogToSln.Tests/BinLogToSln.Tests.csproj --logger console;verbosity=normal (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Copilot · 2025-08-15T17:16:59Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

dnceng.pkgs.visualstudio.com
- Triggering command: dotnet build src/SourceBrowser/src/BinLogParser/BinLogParser.csproj (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

joperezr

Great, this looks ready to go! Thanks @ericstj!

ericstj · 2025-08-15T18:12:59Z

Almost, we need to enable tests to run. We should add test step to https://github.com/dotnet/source-indexer/blob/main/.github/workflows/dotnet.yml

The fork has diverged ~21 commits since the last upstream sync (PR dotnet#184, 2025-05-12). Blindly re-running update-source-browser.ps1 risks silently dropping local features (dotnet#183 signing key, dotnet#192 dedup, dotnet#193 source-generated files, dotnet#255 net10 retarget, dotnet#257 Linux binlog fix, plus Dependabot bumps). - 02: prepend warning block to the 'Updating the vendored SourceBrowser' section listing the divergent PRs and recommending cherry-picks over a full re-sync. - 00: cross-reference the warning from the overview bullet. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Initial plan

aa05925

Copilot AI assigned Copilot and ericstj Jul 22, 2025

Copilot started work on behalf of ericstj July 22, 2025 16:27 View session

Copilot AI and others added 3 commits July 22, 2025 16:39

Initial analysis and build environment setup

41af551

Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Implement improved deduplication logic to prefer real source files ov…

d1d62ab

…er *.notsupported.cs Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Refine deduplication logic with improved error handling and cleaner o…

61d0f7b

…utput Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Missing actual source code from https://source.dot.net/~~ Improve source indexer deduplication to prefer real implementation over *.notsupported.cs files Jul 22, 2025

Copilot finished work on behalf of ericstj July 22, 2025 16:47

Copilot AI requested a review from ericstj July 22, 2025 16:47

ericstj reviewed Jul 22, 2025

View reviewed changes

Copilot started work on behalf of ericstj July 22, 2025 17:18 View session

Implement project properties plumbing and improved deduplication logic

c59f2a1

Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Copilot AI requested a review from ericstj July 22, 2025 17:30

Copilot finished work on behalf of ericstj July 22, 2025 17:30

ericstj reviewed Jul 23, 2025

View reviewed changes

Comment thread src/SourceBrowser/src/BinLogParser/BinLogReader.cs Outdated

Comment thread src/SourceBrowser/src/BinLogParser/BinLogReader.cs Outdated

Comment thread src/SourceBrowser/src/BinLogToSln/Program.cs Outdated

Comment thread src/SourceBrowser/src/BinLogToSln/Program.cs Outdated

Copilot started work on behalf of ericstj July 23, 2025 22:38 View session

Address PR review feedback: Optimize BinLogReader performance and reo…

56384ee

…rder scoring priorities Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Copilot finished work on behalf of ericstj July 23, 2025 22:49

Copilot AI requested a review from ericstj July 23, 2025 22:49

ericstj reviewed Jul 24, 2025

View reviewed changes

Copilot started work on behalf of ericstj July 24, 2025 02:31 View session

Address PR review feedback: Revert project file changes and optimize …

fe6459f

…BinLogReader Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Copilot finished work on behalf of ericstj July 24, 2025 02:52

Copilot AI requested a review from ericstj July 24, 2025 02:52

ericstj added 2 commits August 12, 2025 09:24

Revert "Initial analysis and build environment setup"

6682b64

This reverts commit 41af551.

Fix project ranking heuristic

c2e5693

ericstj added 3 commits August 13, 2025 13:27

Merge branch 'main' into copilot/fix-188

623df56

Make sure Linux is preferred over Windows

eff5ae6

Removed workaround now that fix is in

fbc277e

joperezr reviewed Aug 14, 2025

View reviewed changes

Comment thread src/SourceBrowser/src/BinLogToSln/Program.cs

joperezr reviewed Aug 14, 2025

View reviewed changes

Comment thread src/SourceBrowser/src/BinLogToSln/Program.cs Outdated

joperezr approved these changes Aug 14, 2025

View reviewed changes

ericstj assigned Copilot Aug 15, 2025

Copilot started work on behalf of ericstj August 15, 2025 00:01 View session

Add documentation and unit tests for source selection algorithm

e91af1c

Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Copilot finished work on behalf of ericstj August 15, 2025 00:12

ericstj reviewed Aug 15, 2025

View reviewed changes

Comment thread src/SourceBrowser/src/BinLogToSln.Tests/InvocationScoringTests.cs Outdated

Copilot started work on behalf of ericstj August 15, 2025 16:13 View session

Change test methodology to use comparative assertions instead of exac…

f1b6f7d

…t score values Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Copilot AI requested a review from ericstj August 15, 2025 16:21

Copilot finished work on behalf of ericstj August 15, 2025 16:21

ericstj reviewed Aug 15, 2025

View reviewed changes

Comment thread src/SourceBrowser/src/BinLogToSln.Tests/InvocationScoringTests.cs Outdated

Copilot started work on behalf of ericstj August 15, 2025 17:07 View session

Add Clone method to CompilerInvocation and refactor tests to use it

788e60c

Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com>

Copilot AI requested a review from ericstj August 15, 2025 17:17

Copilot finished work on behalf of ericstj August 15, 2025 17:17

joperezr approved these changes Aug 15, 2025

View reviewed changes

Build and test both SLNs in PR validation

aba391f

ericstj approved these changes Aug 15, 2025

View reviewed changes

ericstj merged commit 07dd0cc into main Aug 15, 2025
2 checks passed

Conversation

Copilot AI commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Documentation and Testing

Example

Uh oh!

ericstj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Jul 22, 2025

Uh oh!

ericstj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Jul 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Jul 24, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Uh oh!

Uh oh!

joperezr left a comment

Choose a reason for hiding this comment

Uh oh!

ericstj commented Aug 15, 2025

Uh oh!

Copilot AI commented Aug 15, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Uh oh!

Copilot AI commented Aug 15, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Uh oh!

Copilot AI commented Aug 15, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

joperezr left a comment

Choose a reason for hiding this comment

Uh oh!

ericstj commented Aug 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Jul 22, 2025 •

edited

Loading