[Bug] tests/python/contrib/test_rpc_server_device.py::test_check_auto_schedule_tuning is flaky

[driazati]

This fails in CI with some frequency:

• https://github.com/apache/tvm/runs/4518525605?check_suite_focus=true
• https://github.com/apache/tvm/runs/4530093366?check_suite_focus=true
• https://github.com/apache/tvm/runs/4515385368?check_suite_focus=true
• https://github.com/apache/tvm/runs/4515385368?check_suite_focus=true
• https://github.com/apache/tvm/runs/4530093366?check_suite_focus=true