Skip to content

[improve][broker] Add topic-aware BookKeeper context for auxiliary ledger placement - #26702

Closed
void-ptr974 wants to merge 1 commit into
apache:masterfrom
void-ptr974:fix/aux-ledger-placement-core
Closed

void-ptr974 wants to merge 1 commit into
apache:masterfrom
void-ptr974:fix/aux-ledger-placement-core

Conversation

@void-ptr974

Copy link
Copy Markdown
Contributor

Motivation

Persistent topics resolve namespace bookieAffinityGroup and strict bookie-affinity settings before selecting a BookKeeper client. Components that create auxiliary ledgers directly only have access to the default client, so they cannot safely reuse the same placement-specific client and persist the matching policy metadata.

Using only the default client can make a configured namespace affinity ineffective. Adding only ledger metadata is also insufficient because a default Rackaware client does not execute IsolatedBookieEnsemblePlacementPolicy.

This is the shared prerequisite for applying topic placement to schema, delayed-delivery bucket, and compaction ledgers in follow-up stacked PRs.

Modifications

  • Extract the existing BrokerService strict/affinity/system-topic decision matrix into BookKeeperPlacementPolicyConfigResolver and continue using it for managed ledgers.
  • Add an asynchronous policy-aware client lookup to BookkeeperManagedLedgerStorageClass.
  • Make ManagedLedgerClientFactory serve that lookup from its existing EnsemblePlacementPolicyConfig client cache.
  • Add BookKeeperClientContext, which keeps the borrowed client and its encoded placement metadata together.
  • Add PulsarService#getBookKeeperClientContext(TopicName) to resolve local namespace policies without loading the full topic policy stack.
  • Fail explicitly when a storage class cannot honor a custom placement policy instead of silently returning the default client.

Behavior and impact

  • Existing managed-ledger placement behavior is preserved, including strict handling for ordinary and system topics.
  • No new BookKeeper client is created per topic. One client is cached per distinct placement-policy configuration and is owned by ManagedLedgerClientFactory.
  • A missing namespace placement policy still uses the existing default client and does not add placement metadata.
  • Policy lookup, client creation, and metadata encoding failures are propagated; they do not fall back to a client that would ignore configured affinity.
  • This PR introduces the shared capability only. The stacked follow-up PRs connect individual auxiliary-ledger creation paths.

Dynamic policy changes are intentionally outside this stack.

Tests

  • ./gradlew :pulsar-broker:test --tests org.apache.pulsar.broker.ManagedLedgerClientFactoryTest --tests org.apache.pulsar.broker.storage.BookKeeperPlacementPolicyConfigResolverTest --tests org.apache.pulsar.broker.storage.BookKeeperClientContextTest
  • ./gradlew :pulsar-broker:checkstyleMain :pulsar-broker:checkstyleTest
  • ./gradlew :pulsar-broker:clean :pulsar-broker:compileJava --no-build-cache --no-configuration-cache

Assisted-by: Codex

Resolve topic placement configuration in one shared path and pair it with the cached BookKeeper client and matching ledger metadata.

Assisted-by: Codex
@void-ptr974

Copy link
Copy Markdown
Contributor Author

Closing in favor of #26705. The shared resolver and BookKeeper client-context commit remains the first of the three commits in that unified PR, whose description now documents the complete problem, implementation, and impact.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant