Why the callback server always listens to 25334 by default

[ricdong]

Why the callback server always listens to 25334 by default, which will cause the pyspark applications to not run on the same server at the same time.
I tried to modify the listening port before starting the callback server like below:

scala_utils.py
self.gateway.start_callback_server(callback_server_parameters = CallbackServerParameters(port = 0))

this can work on the different port for each pyspark application, but the Checks Failure with the constraint_message:

Can't execute the assertion: Error while obtaining a new communication channel

Does anyone have any suggestions on this issue ?

[yk-st]

plz. same here

[cghyzel]

Could you modify the listening port to a valid port #? Say 25444

[dash-hugo-vanthournhout]

We have the same issue here, only when run locally, it is working on aws EMR.

How can we modify the listening port while using pydeequ?

pydeequ -> 1.0.1
spark -> 3.1.3
java -> 11.0.16

[Ashokgoa]

@ricdong did you implement any workaround?

[github-actions]

This is a known limitation. The issue is twofold:

1. Port 25334 conflict: Py4j's callback server defaults to port 25334, so multiple PySpark apps on the same machine collide.
2. "Error while obtaining a new communication channel": When you use port=0 (random port), the JVM side doesn't know which port the Python callback server is listening on, so it can't call back into Python to execute your lambda assertions.

The root cause is in PythonCallback.__init__ in scala_utils.py — it starts the callback server but doesn't propagate the dynamically assigned port back to the JVM's GatewayServer.

Workaround 1: Run your checks sequentially (not parallel PySpark apps on the same host), which avoids the port conflict entirely.

Workaround 2: Avoid Python lambda assertions altogether. Instead of:

check.hasSize(lambda x: x >= 3)

Use the Deequ Java-side assertion directly by not relying on the callback server. Unfortunately, PyDeequ currently requires the callback server for any Check method that takes a Python lambda (like hasSize, hasCompleteness, etc.).

This is a real bug/limitation in PyDeequ's architecture. A proper fix would require either:

• Using port=0 and then updating the JVM-side gateway client with the actual bound port, or
• Allowing users to configure the callback port via an environment variable or parameter.

I'd welcome a PR that addresses this. The fix would be in the PythonCallback class to handle dynamic port assignment and communicate it back to the Java gateway.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.