Transition CI builds over to GitHub Actions by NickEntin · Pull Request #256 · square/Valet

NickEntin · 2021-02-11T21:53:21Z

Travis CI no longer has free macOS builds, so it's time to migrate away. This moves our CI builds over to GitHub Actions.

codecov-io · 2021-03-28T04:27:26Z

Codecov Report

Merging #256 (b6fef0e) into master (6305081) will increase coverage by 0.45%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #256      +/-   ##
==========================================
+ Coverage   86.25%   86.71%   +0.45%     
==========================================
  Files          16       16              
  Lines        1055      986      -69     
==========================================
- Hits          910      855      -55     
+ Misses        145      131      -14

Impacted Files	Coverage Δ
Sources/Valet/Internal/Service.swift	`95.77% <0.00%> (-1.41%)`	⬇️
Sources/Valet/SecureEnclaveAccessControl.swift	`76.27% <0.00%> (-0.78%)`	⬇️
Sources/Valet/Internal/Keychain.swift	`89.00% <0.00%> (-0.50%)`	⬇️
Sources/Valet/SecureEnclaveValet.swift	`76.19% <0.00%> (+0.28%)`	⬆️
Sources/Valet/SinglePromptSecureEnclaveValet.swift	`75.75% <0.00%> (+0.49%)`	⬆️
Sources/Valet/Valet.swift	`94.55% <0.00%> (+4.48%)`	⬆️

NickEntin · 2021-03-28T06:50:23Z

@dfed It looks like the macOS tests are failing due to missing entitlements. Any idea why this would have worked on Travis CI but not GitHub Actions? It should be running exactly the same commands, so presumably there's something different between the two environments?

dfed · 2021-03-31T02:45:38Z

@NickEntin welp. Let's dig in.

Looking at one example test_setStringForKey_neutralizesMacOSAccessControlListVuln:

It's failing in CI
I'm not seeing a guard testEnvironmentIsSigned() in there, so we didn't expect it to fail in CI
I do see that it fails when I run it locally on my M1 Mac using 12.4 without going through the code-signing dance

I vaguely remember macOS 10.15 introducing changes that meant I had to code-sign the environment in order for macOS tests to run. Our xcode11.3 machine on Travis was running macOS 10.14.6, and our new build is on macOS 10.15.

So, with all the above info, I think we hit the macOS Catalina time-bomb. I hate to say it, but we should probably just delete the macOS tests from CI all-together. Speaking of which... I need to land #260 so folk know how on earth to test macOS.

codecov-commenter · 2021-06-09T05:49:22Z

Codecov Report

Merging #256 (b9988d2) into master (a1b2e46) will increase coverage by 0.20%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #256      +/-   ##
==========================================
+ Coverage   86.60%   86.81%   +0.20%     
==========================================
  Files          16       16              
  Lines        1053      986      -67     
==========================================
- Hits          912      856      -56     
+ Misses        141      130      -11

Impacted Files	Coverage Δ
Sources/Valet/SecureEnclaveAccessControl.swift	`76.27% <0.00%> (-6.78%)`	⬇️
Sources/Valet/Internal/Keychain.swift	`89.00% <0.00%> (-0.50%)`	⬇️
Sources/Valet/SecureEnclaveValet.swift	`76.19% <0.00%> (+0.28%)`	⬆️
Sources/Valet/SinglePromptSecureEnclaveValet.swift	`75.75% <0.00%> (+0.49%)`	⬆️
Sources/Valet/Valet.swift	`94.55% <0.00%> (+4.48%)`	⬆️

NickEntin · 2021-06-09T05:51:21Z

Oh boo, turns out GitHub Actions doesn't actually support macOS 10.14, even though the virtual environments repo shows a specification for it. 😞 It silently uses macOS 10.15 instead.

There goes my workaround for testing macOS and running legacy iOS tests.

NickEntin · 2021-06-10T23:35:58Z

I'm not thrilled with what this does to our test matrix, but I think it's good enough for now and unblocks our CI.

Xcode Build and Test

Platform	Before	After
iOS	10, 11, 12, 13	12, 13, 14
tvOS	10, 11, 12, 13	12, 13, 14
watchOS	3, 4, 5, 6	5, 6, 7
macOS	10.15

Swift Package Manager Build

Platform	Before	After
iOS	13	13, 14
tvOS	13	13, 14
watchOS	6	6, 7
macOS	10.15	10.15

NickEntin · 2021-06-10T23:39:22Z

Test Case '-[Valet_tvOS_Tests.SinglePromptSecureEnclaveIntegrationTests test_allKeys_doesNotReflectValetImplementationDetails]' started.
2021-06-10 23:32:12.171511+0000 Valet tvOS Test Host App[2434:43850] [Client,SPI,LAContext] externalizedContext on LAContext[2434:1] cid:2 returned 0
/Users/runner/work/Valet/Valet/Tests/ValetIntegrationTests/SinglePromptSecureEnclaveIntegrationTests.swift:119: error: -[Valet_tvOS_Tests.SinglePromptSecureEnclaveIntegrationTests test_allKeys_doesNotReflectValetImplementationDetails] : XCTAssertEqual failed: threw error "KeychainError.couldNotAccessKeychain"
Test Case '-[Valet_tvOS_Tests.SinglePromptSecureEnclaveIntegrationTests test_allKeys_doesNotReflectValetImplementationDetails]' failed (0.291 seconds).
Test Case '-[Valet_tvOS_Tests.SinglePromptSecureEnclaveIntegrationTests test_allKeys]' started.
2021-06-10 23:32:12.461466+0000 Valet tvOS Test Host App[2434:43850] [Client,SPI,LAContext] externalizedContext on LAContext[2434:1] cid:3 returned 0
/Users/runner/work/Valet/Valet/Tests/ValetIntegrationTests/SinglePromptSecureEnclaveIntegrationTests.swift:97: error: -[Valet_tvOS_Tests.SinglePromptSecureEnclaveIntegrationTests test_allKeys] : XCTAssertEqual failed: threw error "KeychainError.couldNotAccessKeychain"
2021-06-10 23:32:12.503067+0000 Valet tvOS Test Host App[2434:43850] [Client,SPI,LAContext] externalizedContext on LAContext[2434:1] cid:4 returned 0
/Users/runner/work/Valet/Valet/Tests/ValetIntegrationTests/SinglePromptSecureEnclaveIntegrationTests.swift:100: error: -[Valet_tvOS_Tests.SinglePromptSecureEnclaveIntegrationTests test_allKeys] : XCTAssertEqual failed: threw error "KeychainError.couldNotAccessKeychain"
2021-06-10 23:32:12.507018+0000 Valet tvOS Test Host App[2434:43850] [Client,SPI,LAContext] externalizedContext on LAContext[2434:1] cid:5 returned 0
/Users/runner/work/Valet/Valet/Tests/ValetIntegrationTests/SinglePromptSecureEnclaveIntegrationTests.swift:103: error: -[Valet_tvOS_Tests.SinglePromptSecureEnclaveIntegrationTests test_allKeys] : XCTAssertEqual failed: threw error "KeychainError.couldNotAccessKeychain"
2021-06-10 23:32:12.511302+0000 Valet tvOS Test Host App[2434:43850] [Client,SPI,LAContext] externalizedContext on LAContext[2434:1] cid:6 returned 0
/Users/runner/work/Valet/Valet/Tests/ValetIntegrationTests/SinglePromptSecureEnclaveIntegrationTests.swift:106: error: -[Valet_tvOS_Tests.SinglePromptSecureEnclaveIntegrationTests test_allKeys] : XCTAssertEqual failed: threw error "KeychainError.couldNotAccessKeychain"
Test Case '-[Valet_tvOS_Tests.SinglePromptSecureEnclaveIntegrationTests test_allKeys]' failed (0.065 seconds).

🤔 Not sure why these can't access the keychain on tvOS 14.

cc @allisonmoyer this is the same test class you were looking at in #268

…e we can't link to the runtime

NickEntin · 2021-06-11T00:28:16Z

Hmm, we're getting back a -26276 error code in the tvOS tests, which doesn't match any of the OSStatus codes. Not sure how to debug this one.

The closest one is errSecDecode, so maybe it has something to do with that?

errSecDecode                             = -26275,    /* Unable to decode the provided data. */

NickEntin · 2021-06-11T00:33:48Z

The tvOS test failures are reproducing locally, so I think it's a separate bug in our test (filed #270 to track this). Removing tvOS 14 from our test matrix for now.

dfed

LGTM! Thanks for taking this on.

NickEntin closed this Feb 11, 2021

NickEntin reopened this Mar 28, 2021

NickEntin added 13 commits June 8, 2021 22:22

Transition CI builds over to GitHub Actions

ce4f53b

Update to Bundler 2

a2def94

Use iOS 13.7 and 12.4

0f6aaf5

Use 4th gen iPad Pro for 13.7

b70eeb8

Fix watchOS and tvOS version to pair with iOS 13.7

a6c5020

platforms, not platform

95422ff

Use Xcode 11.7 everywhere

f5a2d84

Apparently it's watchOS 6.2.1, not 6.2

2cd34a7

Set up tvOS 12.4 runtime

20e1f6a

Set up watchOS 5.3 simulator

65ad218

Use bash script to prepare simulator runtimes

4d4145a

Move iOS 10 and 11 back to Travis CI

4a36134

Add build for legacy versions (iOS 11, tvOS 11, watchOS 4)

b0e35c6

NickEntin force-pushed the entin/gh-actions branch from f70c98d to b0e35c6 Compare June 9, 2021 05:29

NickEntin added 2 commits June 8, 2021 22:31

Move macOS 10.15 tests to legacy build

06eb46f

Remove iOS 11 tests from Travis CI

d7f44d0

NickEntin mentioned this pull request Jun 9, 2021

Enable customization of biometrics prompt with application-specific fallback #267

Closed

Debugging

b3480a2

NickEntin added 5 commits June 10, 2021 16:07

Remove legacy build, since macOS 10.14 isn't real

6b191da

Remove Travis config

d1680e1

Use Xcode 12 and add build for iOS 14, tvOS 14, and watchOS 7

2551803

Fix runtime paths for Xcode 11

c8bf1a5

Add SPM tests for iOS 14 family

3626419

NickEntin added 3 commits June 10, 2021 16:40

Use Xcode 12 for SPM build

4be030b

Split macOS SPM build into a separate job that runs on Xcode 11, sinc…

94a5da4

…e we can't link to the runtime

🤦🤦🏻 copy pasta

b9988d2

Don't run tests on tvOS 14

e936778

NickEntin marked this pull request as ready for review June 11, 2021 03:34

NickEntin requested review from allisonmoyer and dfed June 11, 2021 03:35

allisonmoyer approved these changes Jun 11, 2021

View reviewed changes

dfed approved these changes Jun 11, 2021

View reviewed changes

dfed mentioned this pull request Jun 11, 2021

Unit tests fail on tvOS 14 #270

Closed

NickEntin merged commit bce7b39 into master Jun 12, 2021

NickEntin deleted the entin/gh-actions branch June 12, 2021 19:51

dfed mentioned this pull request Mar 7, 2022

Start using Github Actions dfed/CacheAdvance#53

Merged

dfed mentioned this pull request Oct 12, 2022

Get codecov working again dfed/CacheAdvance#67

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transition CI builds over to GitHub Actions#256

Transition CI builds over to GitHub Actions#256
NickEntin merged 25 commits into
masterfrom
entin/gh-actions

NickEntin commented Feb 11, 2021

Uh oh!

codecov-io commented Mar 28, 2021

Uh oh!

NickEntin commented Mar 28, 2021

Uh oh!

dfed commented Mar 31, 2021

Uh oh!

codecov-commenter commented Jun 9, 2021 •

edited

Loading

Uh oh!

NickEntin commented Jun 9, 2021

Uh oh!

NickEntin commented Jun 10, 2021

Uh oh!

NickEntin commented Jun 10, 2021

Uh oh!

NickEntin commented Jun 11, 2021 •

edited

Loading

Uh oh!

NickEntin commented Jun 11, 2021

Uh oh!

dfed left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

NickEntin commented Feb 11, 2021

Uh oh!

codecov-io commented Mar 28, 2021

Codecov Report

Uh oh!

NickEntin commented Mar 28, 2021

Uh oh!

dfed commented Mar 31, 2021

Uh oh!

codecov-commenter commented Jun 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

NickEntin commented Jun 9, 2021

Uh oh!

NickEntin commented Jun 10, 2021

Uh oh!

NickEntin commented Jun 10, 2021

Uh oh!

NickEntin commented Jun 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NickEntin commented Jun 11, 2021

Uh oh!

dfed left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Jun 9, 2021 •

edited

Loading

NickEntin commented Jun 11, 2021 •

edited

Loading