fix(amber): avoids false negative on _check_heartbeat when connection is established but close raises exception

[nathant27]

What changes were proposed in this PR?

Old version of Heartbeat._check_heartbeat(self) wraps the socket connection and socket close in a single try except, and handles with the same try except. The issue is that if close raises after a successful connection, there is a false negative since it will handle it as if socket connection had failed. In Heartbeat.run(self), this could cause the server to shutdown on close raising exception since _check_heartbeat will return False in this case.
New version separates the connection handling and the close in two adjacent try except blocks, and returns true as long as connection is successfully established as well, getting rid of the false negative.

As per discussed in the issue, will keep the double check for _check_heartbeat in Heartbeat.run.

Old Version:

    def _check_heartbeat(self) -> bool:
        """
        Attempt to connect to JVM on the specific port. If succeeds, it means the
        socket is still available and the JVM is still alive. Otherwise, the JVM
        might have been gone.

        :return: bool, indicating if the socket is available.
        """
        try:
            temp_socket = socket.create_connection(
                (self._parsed_server_host, self._parsed_server_port), timeout=1
            )
            temp_socket.close()
            return True
        except Exception as e:
            logger.warning(f"Server is down with exception: {e}")
            return False

New Version:

    def _check_heartbeat(self) -> bool:
        """
        Attempt to connect to the JVM port. If connection failure, then JVM is dead, or if connection success 
        then JVM is alive even if close() raises. Logs on connection failure and on close error.

        :return: bool, True if connect succeeded, False if connect failed.
        """
        try:
            temp_socket = socket.create_connection(
                (self._parsed_server_host, self._parsed_server_port), timeout=1
            )
        except Exception as e:
            logger.warning(f"Server is down with exception: {e}")
            return False

        try:
            temp_socket.close()
        except Exception as e:
            logger.warning(f"Failed to close socket: {e}")
        
        return True

Any related issues, documentation, discussions?

Closes #4726

How was this PR tested?

Changed old test to reflect new behavior(True on connection succeeds but socket close rasises)
OLD TEST(REMOVED):

    def test_returns_false_when_socket_close_raises(self):
        # Pins the false-negative path: connect succeeds but the subsequent
        # close() throws (e.g. broken pipe on a half-open socket). The bare
        # `except Exception` in _check_heartbeat() catches it and reports
        # "server down", and a regression that narrows that handler would be
        # caught here.
        hb = make_heartbeat()
        fake_sock = MagicMock()
        fake_sock.close.side_effect = OSError("close failed")
        with patch(
            "core.runnables.heartbeat.socket.create_connection",
            return_value=fake_sock,
        ):
            assert hb._check_heartbeat() is False

NEW TEST:

    def test_returns_true_when_connection_succeeds_but_socket_close_raises(self):
        hb = make_heartbeat()
        fake_sock = MagicMock()
        fake_sock.close.side_effect = OSError("close failed")
        with patch(
            "core.runnables.heartbeat.socket.create_connection",
            return_value=fake_sock,
        ):
            assert hb._check_heartbeat() is True

Passes on existing old tests

Was this PR authored or co-authored using generative AI tooling?

No

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 42.78%. Comparing base (09828ea) to head (18aa5b1).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #5027      +/-   ##
============================================
- Coverage     42.84%   42.78%   -0.06%     
+ Complexity     2204     2175      -29     
============================================
  Files          1045     1045              
  Lines         40144    39826     -318     
  Branches       4240     4199      -41     
============================================
- Hits          17198    17038     -160     
+ Misses        21879    21732     -147     
+ Partials       1067     1056      -11     

Flag	Coverage Δ		*Carryforward flag
access-control-service	39.53% <ø> (ø)		Carriedforward from dcb457e
agent-service	33.72% <ø> (ø)		Carriedforward from dcb457e
amber	43.20% <ø> (-0.52%)	⬇️	Carriedforward from dcb457e
computing-unit-managing-service	0.00% <ø> (ø)		Carriedforward from dcb457e
config-service	0.00% <ø> (ø)		Carriedforward from dcb457e
file-service	32.18% <ø> (ø)		Carriedforward from dcb457e
frontend	34.20% <ø> (+0.27%)	⬆️	Carriedforward from dcb457e
python	89.00% <100.00%> (+<0.01%)	⬆️
workflow-compiling-service	47.72% <ø> (ø)		Carriedforward from dcb457e

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[nathant27]

/request-review @Yicong-Huang