Buggify GRV proxy to return errors and fix bugs found

[jzhou77]

GRV proxy may return proxy_memory_limit_exceeded and batch_transaction_throttled errors, which are not properly handled at many places.

100k 20220811-220634-jzhou-2e8c8744412b8112 passed.

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

• The PR has a description, explaining both the problem and the solution.
• The description mentions which forms of testing were done and the testing seems reasonable.
• Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

• This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
• There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: c4c6aeb
• Duration 0:47:52
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Logs (available for 30 days)

[fdb-windows-ci]

Doxense CI Report for Windows 10

• Commit ID: 737b964
• Result: ✔️ SUCCEEDED
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS BigSur 11.5.2

• Commit ID: 737b964
• Duration 0:43:31
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: c4c6aeb
• Duration 1:54:50
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)

[fdb-windows-ci]

Doxense CI Report for Windows 10

• Commit ID: 41e941c
• Result: ✔️ SUCCEEDED
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 737b964
• Duration 1:45:14
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 737b964
• Duration 2:15:24
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: c01c2f2
• Duration 1:47:10
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: c01c2f2
• Duration 2:16:37
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 41e941c
• Duration 2:20:01
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 41e941c
• Duration 2:20:41
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Logs (available for 30 days)

[fdb-windows-ci]

Doxense CI Report for Windows 10

• Commit ID: 5a0a366
• Result: ✔️ SUCCEEDED
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 5a0a366
• Duration 0:59:12
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 5a0a366
• Duration 1:40:35
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)

[fdb-windows-ci]

Doxense CI Report for Windows 10

• Commit ID: 4017a7a
• Result: ✔️ SUCCEEDED
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS BigSur 11.5.2

• Commit ID: 4017a7a
• Duration 0:45:12
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 4017a7a
• Duration 0:52:34
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: f6f5f9e
• Duration 0:10:42
• Result: ❌ FAILED
• Error: reference not found for primary source and source version f6f5f9eb2afb1072d0796294beea48d765999f7f
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: f6f5f9e
• Duration 0:11:00
• Result: ❌ FAILED
• Error: reference not found for primary source and source version f6f5f9eb2afb1072d0796294beea48d765999f7f
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS BigSur 11.5.2

• Commit ID: f6f5f9e
• Duration 0:11:05
• Result: ❌ FAILED
• Error: reference not found for primary source and source version f6f5f9eb2afb1072d0796294beea48d765999f7f
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 4017a7a
• Duration 1:42:09
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)

[fdb-windows-ci]

Doxense CI Report for Windows 10

• Commit ID: ade54d4
• Result: ✔️ SUCCEEDED
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: ade54d4
• Duration 1:18:02
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: ade54d4
• Duration 1:39:31
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 40afd38
• Duration 1:45:17
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)

[sfc-gh-mpilman]

Thanks for fixing this

[xis19]

I think this request can be put outside the loop.

[jzhou77]

This is intentional. Same request object in an actor can cause old response being replayed, i.e., not delivering new responses.

[fdb-windows-ci]

Doxense CI Report for Windows 10

• Commit ID: 8fb6d59
• Result: ❌ FAILED
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 8fb6d59
• Duration 1:16:52
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)
• Build Artifact (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 8fb6d59
• Duration 1:23:41
• Result: ❌ FAILED
• Error: Error in UPLOAD_ARTIFACTS phase: [fdb-kubernetes-tests: [report files not found in build]]
• Build Logs (available for 30 days)
• Build Artifact (available for 30 days)

[fdb-windows-ci]

Doxense CI Report for Windows 10

• Commit ID: 92cae85
• Result: ❌ FAILED
• Build Logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS BigSur 11.5.2

• Commit ID: 92cae85
• Duration 0:44:17
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)
• Build Artifact (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 92cae85
• Duration 1:06:57
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Logs (available for 30 days)
• Build Artifact (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 92cae85
• Duration 1:28:22
• Result: ❌ FAILED
• Error: Error in UPLOAD_ARTIFACTS phase: [fdb-kubernetes-tests: [report files not found in build]]
• Build Logs (available for 30 days)
• Build Artifact (available for 30 days)