fix: Split apache-libcloud versions for Python compatibility#619
Merged
Conversation
- Server (Python 3.9): apache-libcloud==3.2.0 (stable Cloudflare DNS) - Client (Python 3.12): apache-libcloud==3.8.0 (Python 3.12 support) Client Dockerfile installs requirements-webapp.txt after requirements.txt, so libcloud 3.8.0 overrides 3.2.0 for the client container only. Fixes Cloudflare DNS issues: - libcloud 3.6.0: RecordAlreadyExistsError constructor bug - libcloud 3.8.0: KeyError 'zone_name' in Cloudflare driver - libcloud 3.2.0: Proven stable with Cloudflare, incompatible with Py3.12 Tested: - Server: Python 3.9 + libcloud 3.2.0 (all 8 DNS tests pass) - Client: Python 3.12 + libcloud 3.8.0 (imports work correctly)
The client Dockerfile was installing the package with [opwen_email_server] extra, which pulled in server dependencies (requirements.txt with libcloud 3.2.0). This conflicted with the client's libcloud 3.8.0. Client only needs the base package dependencies (requirements-webapp.txt).
…comment placement
RabbitMQ 4.0+ deprecated transient non-exclusive queues. The integration test was creating a queue without durable=True, causing CI failures with: InternalError: Queue.declare: (541) INTERNAL_ERROR - Feature `transient_nonexcl_queues` is deprecated.
RabbitMQ 4.0+ deprecated transient non-exclusive queues. This fix: 1. Explicitly defines all task queues as durable using Kombu Queue objects 2. Disables remote control (worker_enable_remote_control=False) to prevent creation of non-durable pidbox queues that trigger the deprecation error 3. Updates test queue to be durable Without these changes, Celery workers fail to start with: amqp.exceptions.InternalError: Queue.declare: (541) INTERNAL_ERROR Feature `transient_nonexcl_queues` is deprecated. Tested successfully with RabbitMQ 4.3.1 - all integration tests pass.
…tion.txt - Created requirements-integration.txt with server dependencies needed by AzureIoc - Updated Dockerfile to install integration deps in both builder and runtime stages - Production client (client.env) uses default Ioc without server dependencies - Integration tests (webapp.env) use AzureIoc with full server dependencies - Integration tests now pass successfully with BUILD_TARGET=runtime
- Define all 7 task queues explicitly as durable Queue objects - Disable worker remote control to prevent transient pidbox queues - Set worker_prefetch_multiplier=1 to avoid deprecated global_qos - Fixes 'Feature transient_nonexcl_queues is deprecated' errors - Integration tests pass with RabbitMQ 4.3.1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
PR #618 upgraded apache-libcloud to 3.8.0 for Python 3.12 compatibility, but introduced a production bug:
Root cause: apache-libcloud Cloudflare driver is unstable across recent versions:
Solution
1. Split apache-libcloud Versions
Use Docker container isolation to run different versions per service:
Implementation:
requirements.txt:apache-libcloud==3.2.0(server default)requirements-webapp.txt:apache-libcloud==3.8.0(client override)[opwen_email_server]extra to avoid version conflict2. RabbitMQ 4.0+ Compatibility
Fixed deprecation warnings causing worker instability:
Queueobjectsworker_prefetch_multiplier=1to avoid deprecated global_qosFeature 'transient_nonexcl_queues' is deprecated3. Clean Integration Test Dependencies
Created
requirements-integration.txtfor server dependencies needed only by AzureIoc:client.env→ defaultIoc→ minimal dependencieswebapp.env→AzureIoc→ full server dependenciesTesting
✅ Local Integration Tests (BUILD_TARGET=runtime):
✅ Version Isolation Verified:
Changes
Modified Files
requirements.txt: Pin apache-libcloud 3.2.0 for server (Python 3.9)requirements-webapp.txt: Override to apache-libcloud 3.8.0 for client (Python 3.12)docker/client/Dockerfile: Install base package without[opwen_email_server]extra, add integration deps to runtime stageopwen_email_server/integration/celery.py: Add explicit durable queue definitions, disable remote controlopwen_email_server/services/dns.py: Refactor with type-safe error handlingtests/opwen_email_server/integration/test_celery.py: Update test queue to be durableNew Files
requirements-integration.txt: Server dependencies for AzureIoc integration testing onlyCI/CD
Deployment Plan
Breaking Changes
None - backward compatible with existing deployments.
Related