Azure Pipelines / runtime (Formatting windows x64) fails

[noahfalk]

Rolling build 1902291 failed the Formatting windows x64 leg:

https://dev.azure.com/dnceng/public/_build/results?buildId=1902291&view=logs&j=db5b9618-eefc-58e8-4554-fbce452d4dac&t=e007745b-d6a3-5b01-37e0-7d8f8b247608&l=9

2022-07-25T20:17:12.4716969Z ##[section]Starting: Publish format.patch
2022-07-25T20:17:12.4975409Z ==============================================================================
2022-07-25T20:17:12.4976456Z Task         : Publish build artifacts
2022-07-25T20:17:12.4977508Z Description  : Publish build artifacts to Azure Pipelines or a Windows file share
2022-07-25T20:17:12.4978412Z Version      : 1.200.0
2022-07-25T20:17:12.4979109Z Author       : Microsoft Corporation
2022-07-25T20:17:12.4979866Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/publish-build-artifacts
2022-07-25T20:17:12.4980782Z ==============================================================================
2022-07-25T20:17:12.7729414Z ##[error]Publishing build artifacts failed with an error: Not found PathtoPublish: D:\a\_work\1\s\src\coreclr\format.patch
2022-07-25T20:17:12.7813669Z ##[section]Finishing: Publish format.patch

This error occurred in the past in #37500 and #61331

@BruceForstall - Not sure who the right owner is here, but since you fixed this last time I am hoping you know how to triage this further at least?

Build	Kind	Start Time
1902291	Rolling	2022-25-07

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Rolling build 1902291 failed the Formatting windows x64 leg:

https://dev.azure.com/dnceng/public/_build/results?buildId=1902291&view=logs&j=db5b9618-eefc-58e8-4554-fbce452d4dac&t=e007745b-d6a3-5b01-37e0-7d8f8b247608&l=9

2022-07-25T20:17:12.4716969Z ##[section]Starting: Publish format.patch
2022-07-25T20:17:12.4975409Z ==============================================================================
2022-07-25T20:17:12.4976456Z Task         : Publish build artifacts
2022-07-25T20:17:12.4977508Z Description  : Publish build artifacts to Azure Pipelines or a Windows file share
2022-07-25T20:17:12.4978412Z Version      : 1.200.0
2022-07-25T20:17:12.4979109Z Author       : Microsoft Corporation
2022-07-25T20:17:12.4979866Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/publish-build-artifacts
2022-07-25T20:17:12.4980782Z ==============================================================================
2022-07-25T20:17:12.7729414Z ##[error]Publishing build artifacts failed with an error: Not found PathtoPublish: D:\a\_work\1\s\src\coreclr\format.patch
2022-07-25T20:17:12.7813669Z ##[section]Finishing: Publish format.patch

This error occurred in the past in #37500 and #61331

@BruceForstall - Not sure who the right owner is here, but since you fixed this last time I am hoping you know how to triage this further at least?

Build	Kind	Start Time
1902291	Rolling	2022-25-07

Author:	noahfalk
Assignees:	-
Labels:	area-Infrastructure-coreclr, blocking-clean-ci, untriaged
Milestone:	-

[ericstj]

That's just the cascading error, here's what failed:

Downloading https://raw.githubusercontent.com/dotnet/jitutils/main/bootstrap.cmd to C:\Users\CLOUDT~1\AppData\Local\Temp\tmp3z_qpi45\bootstrap.cmd
  File "D:\a\_work\1\s\src\tests\Common\scripts\format.py", line 266, in <module>
    return_code = main(sys.argv[1:])
  File "D:\a\_work\1\s\src\tests\Common\scripts\format.py", line 146, in main
    urlretrieve(bootstrapUrl, bootstrapPath)
  File "C:\python3\lib\urllib\request.py", line 239, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "C:\python3\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "C:\python3\lib\urllib\request.py", line 523, in open
    response = meth(req, response)
  File "C:\python3\lib\urllib\request.py", line 632, in http_response
    response = self.parent.error(
  File "C:\python3\lib\urllib\request.py", line 561, in error
    return self._call_chain(*args)
  File "C:\python3\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\python3\lib\urllib\request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Internal Server Error
##[error]Cmd.exe exited with code '1'.

Seems to be a problem downloading from GitHub. This script might need retries? Or perhaps use the built-in AzDo tasks for cloning repos rather than manually downloading raw files.

Aside - when the build of a particular hash downloads main of some other repo it creates non-determinism in the build. We shouldn't do this sort of thing. Better to either flow a dependency package (EG: source package) or use a submodule.

[jakobbotsch]

Aside - when the build of a particular hash downloads main of some other repo it creates non-determinism in the build. We shouldn't do this sort of thing. Better to either flow a dependency package (EG: source package) or use a submodule.

This is just for the formatting job -- not part of the build. We don't run this job on e.g. the servicing branches since it is cosmetic only.

Agree we should use a retrying download here since it is running as part of CI. We already have the helper function here:

runtime/src/coreclr/scripts/jitutil.py

Lines 582 to 596 in 60e5f40

	def download_with_progress_urlretrieve(uri, target_location, fail_if_not_found=True, display_progress=True):
	""" Do an URI download using urllib.request.urlretrieve with a progress hook.
	Retries the download up to 5 times unless the URL returns 404.
	Outputs messages using the `logging` package.
	Args:
	uri (string) : URI to download
	target_location (string) : local path to put the downloaded object
	fail_if_not_found (bool) : if True, fail if a download fails due to file not found (HTTP error 404).
	Otherwise, ignore the failure.
	display_progress (bool) : if True, display download progress (for URL downloads). Otherwise, do not.
	Returns True if successful, False on failure
	"""

format.py should just be moved so that this function can be referenced and used for the download.

[BruceForstall]

Aside - when the build of a particular hash downloads main of some other repo it creates non-determinism in the build. We shouldn't do this sort of thing. Better to either flow a dependency package (EG: source package) or use a submodule.

This comes up every once-in-a-while, e.g., #61331 (comment), #33697, but it's never been deemed worth changing/fixing.