More resiliency so that bugs in the worker do not result in "never ending" jobs.

[carter-yagemann]

My worker occasionally fails to decode API responses that are expected to be JSON:

mlsploit-execution-service_1  | Traceback (most recent call last):
mlsploit-execution-service_1  |   File "/usr/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
mlsploit-execution-service_1  |     R = retval = fun(*args, **kwargs)
mlsploit-execution-service_1  |   File "/usr/lib/python3.7/site-packages/celery/app/trace.py", line 648, in __protected_call__
mlsploit-execution-service_1  |     return self.run(*args, **kwargs)
mlsploit-execution-service_1  |   File "/app/mlsploit.py", line 189, in perform_job
mlsploit-execution-service_1  |     job.output_files = output_file_urls
mlsploit-execution-service_1  |   File "/app/api.py", line 125, in __setattr__
mlsploit-execution-service_1  |     RestClient.patch(self.url, {key: value})
mlsploit-execution-service_1  |   File "/app/api.py", line 84, in patch
mlsploit-execution-service_1  |     data = json.loads(r.text)
mlsploit-execution-service_1  |   File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
mlsploit-execution-service_1  |     return _default_decoder.decode(s)
mlsploit-execution-service_1  |   File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
mlsploit-execution-service_1  |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
mlsploit-execution-service_1  |   File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
mlsploit-execution-service_1  |     raise JSONDecodeError("Expecting value", s, err.value) from None
mlsploit-execution-service_1  | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

It's unclear which part of the backend is to blame for this. Regardless, uncaught exceptions will cause jobs to never reach a finished state. Thus, we should make sure there's a generic exception handler for all job handling code that at least puts the job into a FAILURE state.