Follow-up from #478 review round 4 (accepted as a known window at merge). Infrastructure retries for an eval that finished without a result are counted in infra-retries.json. If the controller crashes after the retry sbatch succeeds but before the submitted marker is rewritten with the new job id, that retry is never counted, so the two-retry cap can be exceeded by one eval per such crash. Fix: journal the retry before submitting and roll the count back if the submit fails.
Follow-up from #478 review round 4 (accepted as a known window at merge). Infrastructure retries for an eval that finished without a result are counted in
infra-retries.json. If the controller crashes after the retrysbatchsucceeds but before thesubmittedmarker is rewritten with the new job id, that retry is never counted, so the two-retry cap can be exceeded by one eval per such crash. Fix: journal the retry before submitting and roll the count back if the submit fails.