Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/design/tools/illink/task-cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ internal sealed class ILLinkCache
- Purge requires an explicit cache root and UTC cutoff. Only published SHA-256 entry directories in `v1` are considered; delete entries whose marker is strictly older than the cutoff and retain equality. Leave staging directories and other layouts alone. Report per-entry errors and continue, then return nonzero if maintenance failed.
- Assume old caches are cleared before using markers. Do not infer ages or migrate entries with missing markers; missing, malformed, or unreadable markers are ordinary reported errors.
- Purge must not run during any build using the cache. Do not add purge locks or change reader locking; coordination with active readers/writers remains a separate follow-up, as does protection against manual deletion.
- CI may record build start after restoring a job-private cache, run linking, then purge against that timestamp before uploading the next snapshot. Preserve tool diagnostics but do not fail the build for maintenance errors. Only purge once all users have exited; incremental skips and unreached work in failed builds do not refresh usage. See the [CI recipe](../../../tools/illink/illink-tasks.md#ci-purge-ordering). This does not enable an ILLink cache in CI automatically.
- CI jobs using `setup-sccache.yml` also enable ILLink caching under its existing Unix x64/arm64 platform gate. Restore a job-private Azure cache, record build start, run linking and all post-build steps, then purge against that timestamp before uploading the next snapshot. Build the maintenance tool from the checkout if it was not part of the main build. Skip maintenance when no versioned cache directory exists. Preserve tool diagnostics but do not fail the build for maintenance errors, and skip canceled jobs or jobs where ILLink caching was not enabled. Only purge once all users have exited; incremental skips and unreached work in failed builds do not refresh usage. See the [CI recipe and wiring](../../../tools/illink/illink-tasks.md#ci-purge-ordering).
- Attempt best-effort cleanup of the current attempt's unpublished staging directory. Log cleanup I/O/access failures without failing a successful link; do not scan or delete other attempts' staging directories. Correctness must tolerate leftovers after crashes.

## Open follow-ups
Expand Down
13 changes: 11 additions & 2 deletions docs/tools/illink/illink-tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,8 +135,17 @@ work that was never reached will count as unused. Likewise, incremental builds t
linking do not refresh cache usage. Skip purge if the cache may still be in use, including
during cancellation. Do not share a live directory with another job while purging.

This is usage guidance, not automatic CI enablement. ILLink cache restore/upload wiring
is separate from runtime's Roslyn/csc cache maintenance.
Runtime's jobs that use `setup-sccache.yml` also enable ILLink caching under its existing
Unix x64/arm64 platform gate. `setup-illink-cache.yml` restores a branch-scoped Azure
Pipelines snapshot into `$(Pipeline.Workspace)/.illink-cache`, enables the experiment, and
records build start. `global-build-job.yml` runs `purge-illink-cache.yml` after all post-build
steps and before the `Cache@2` post-job save. The purge step builds the tool from the checkout
because library-only builds need not include the tool project. It reports missing
prerequisites and maintenance failures as warnings, and runs only when ILLink caching
was enabled and the job was not canceled. When no versioned cache directory exists,
as on a cold native-only build, it skips the maintenance tool build and purge.

This configuration is separate from runtime's Roslyn/csc and native compiler caches.

## ILLink Task Properties

Expand Down
4 changes: 4 additions & 0 deletions eng/pipelines/common/global-build-job.yml
Original file line number Diff line number Diff line change
Expand Up @@ -273,3 +273,7 @@ jobs:
- powershell: ./eng/collect_vsinfo.ps1 -ArchiveRunName postbuild_log
displayName: Collect vslogs on exit
condition: always()

# Some post-build steps compile additional tests. Purge only after those
# steps have completed, before Cache@2 saves its post-job snapshot.
- template: /eng/pipelines/coreclr/templates/purge-illink-cache.yml
44 changes: 44 additions & 0 deletions eng/pipelines/coreclr/templates/purge-illink-cache.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
steps:
- script: |
set -euo pipefail

if [[ -z "${ILLINK_EXPERIMENTAL_CACHE_PATH:-}" ||
-z "${ILLINK_CACHE_BUILD_START:-}" ]]; then
echo "##vso[task.logissue type=warning]ILLink cache configuration or build-start timestamp is unavailable; skipping purge."
exit 0
fi

if [[ ! -e "$ILLINK_EXPERIMENTAL_CACHE_PATH/v1" ]]; then
echo "No ILLink cache entries were created; skipping purge."
exit 0
fi

dotnet="$(Build.SourcesDirectory)/.dotnet/dotnet"
if [[ ! -x "$dotnet" ]]; then
echo "##vso[task.logissue type=warning]The build SDK is unavailable; skipping ILLink cache purge."
exit 0
fi

# Library-only builds need not build the tools.illink subset. Build the
# maintenance tool from this checkout into a known host-tool directory.
toolDirectory="$(Build.SourcesDirectory)/artifacts/bin/ILLink.CacheTool/ci-purge"
if ! "$dotnet" build \
"$(Build.SourcesDirectory)/src/tools/illink/src/ILLink.CacheTool/ILLink.CacheTool.csproj" \
--configuration Release --output "$toolDirectory" \
-m:1 -nodeReuse:false -p:BuildInParallel=false -p:UseSharedCompilation=false; then
echo "##vso[task.logissue type=warning]Could not build the ILLink cache maintenance tool; skipping purge."
exit 0
fi

if ! "$dotnet" "$toolDirectory/dotnet-illink-cache.dll" purge \
--cache-directory "$ILLINK_EXPERIMENTAL_CACHE_PATH" \
--before "$ILLINK_CACHE_BUILD_START"; then
echo "##vso[task.logissue type=warning]ILLink cache purge failed; see the tool diagnostics above."
fi
displayName: Purge unused ILLink cache entries
condition: and(succeededOrFailed(), eq(variables['ILLINK_EXPERIMENTAL_CACHE'], 'true'))
continueOnError: true
env:
DOTNET_PROCESSOR_COUNT: '2'
DOTNET_CLI_USE_MSBUILD_SERVER: '0'
MSBUILDDISABLENODEREUSE: '1'
27 changes: 27 additions & 0 deletions eng/pipelines/coreclr/templates/setup-illink-cache.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
parameters:
archType: 'x64'
osGroup: 'linux'
nameSuffix: ''
osSubgroup: ''

steps:
# Use the same branch-scoped restore policy as the compiler caches. Each job
# works on its own restored copy, so post-build purge cannot affect another job.
- task: Cache@2
displayName: ILLink task cache
continueOnError: true
inputs:
key: illink | v1 | ${{ parameters.osGroup }}${{ parameters.osSubgroup }} | ${{ parameters.archType }} | ${{ parameters.nameSuffix }} | "$(Build.BuildId)"
path: $(Pipeline.Workspace)/.illink-cache
restoreKeys: |
illink | v1 | ${{ parameters.osGroup }}${{ parameters.osSubgroup }} | ${{ parameters.archType }} | ${{ parameters.nameSuffix }}

- script: |
set -euo pipefail
mkdir -p "$(Pipeline.Workspace)/.illink-cache"
buildStart="$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
echo "##vso[task.setvariable variable=ILLINK_EXPERIMENTAL_CACHE_PATH]$(Pipeline.Workspace)/.illink-cache"
echo "##vso[task.setvariable variable=ILLINK_CACHE_BUILD_START]$buildStart"
echo "##vso[task.setvariable variable=ILLINK_EXPERIMENTAL_CACHE]true"
displayName: Configure experimental ILLink task cache
continueOnError: true
7 changes: 7 additions & 0 deletions eng/pipelines/coreclr/templates/setup-sccache.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,13 @@ steps:
archType: ${{ parameters.archType }}
nameSuffix: ${{ parameters.nameSuffix }}

- template: /eng/pipelines/coreclr/templates/setup-illink-cache.yml
parameters:
osGroup: ${{ parameters.osGroup }}
osSubgroup: ${{ parameters.osSubgroup }}
archType: ${{ parameters.archType }}
nameSuffix: ${{ parameters.nameSuffix }}

# Set up the Azure Pipeline Cache for sccache's local cache directory.
# Use a rolling key so each build can update the cache; restoreKeys
# falls back to the most recent saved entry.
Expand Down
Loading