Skip to content

Polaris Tag Management: tag assignment write path - #5469

Draft
flyingImer wants to merge 3 commits into
apache:mainfrom
flyingImer:tag/pr3-assignment
Draft

Polaris Tag Management: tag assignment write path#5469
flyingImer wants to merge 3 commits into
apache:mainfrom
flyingImer:tag/pr3-assignment

Conversation

@flyingImer

@flyingImer flyingImer commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Tip

Until #5391 merges, the files view includes the spec commit from #5366 and the definition commit from #5391. For this slice's own diff, review the head commit: flyingImer@fff2ce1

GH proposal tracking: #5442

Why

#5391 makes Tag definitions usable, but nothing can be attached to them yet, and an authorized detach-all=true drop answers 501. This PR lets a tag be assigned to a catalog, a namespace, a table-like object, or a top-level Iceberg table column, one value per definition and target, validated against the allowed values, and unassigned again. Dropping a definition with detach-all=true now removes the definition and every assignment together, or nothing at all.

Reads and reverse lookup are the next slice. Management grants for the new privileges are a later slice. NoSQL stays unsupported.

Scope

  • PUT and POST on /tags/{tag-name}/mappings assign and unassign; assigning again replaces the stored value.
  • Both operations check two privileges: TAG_ATTACH or TAG_DETACH on the definition, plus a target-side {CATALOG,NAMESPACE,TABLE}_{ATTACH,DETACH}_TAG. CATALOG_MANAGE_CONTENT covers all of them. Tags classify resources only; they do not grant or deny access.
  • JDBC persistence on schema v7 (new tag_assignment_record table; existing databases need a one-time manual SQL, documented in the CHANGELOG and the metastore docs); writes on an older schema fail with an error naming the v7 requirement. In-memory is supported. NoSQL rejects assignment operations and tag drops explicitly. A backend that cannot guarantee the detach-all result answers 501 to detach-all=true and changes nothing.
  • The selected value is checked against the definition inside the write transaction, and the write conflicts with a concurrent update of the definition's allowed values: an update that completes first rejects a removed value, an assignment that completes first is grandfathered. A first-write race on one identity re-runs once; a second conflict surfaces as a server error. Unassigning an assignment that is already gone answers 404.
  • Allowed values and assigned values are limited to 2000 UTF-8 bytes: the value is part of a database index key, an implementation limit under the spec's section 1.3.
  • A malformed target, a view, a non-Iceberg column, a null or empty path segment, a path segment containing U+001F, or more than one selected value answers 400.
  • Tags remain disabled by default behind ENABLE_TAG_STORE. Target deletion cleans up assignments best-effort on the shared entity drop path; leftover rows are hidden by the read slice.

Extension points

Three mixin interfaces carry the assignment contract, threaded the same way PolicyMappingPersistence was: BasePersistence gains TagAssignmentPersistence, TransactionalPersistence gains TransactionalTagAssignmentPersistence, and PolarisMetaStoreManager gains PolarisTagAssignmentManager. Every persistence method, including the detach-all capability, is a default that reports "not supported", so an out-of-tree backend keeps compiling and stays fail-closed until it opts in. The manager's assignTagToEntity and unassignTagFromEntity are abstract, the same shape as PolarisPolicyMappingManager's attach and detach. A note describing this delta is posted on the design thread with this PR.

Validation

The integration test covers assign, unassign and detach-all end to end on JDBC and in-memory, replacement and grandfathering, malformed targets answering 400 and the value bound. The authorization test covers the privilege matrix and the detach-all two-privilege rule. The schema gate test pins the write rejection below v7. The JDBC unit tests inject a failure between the two deletes of the atomic drop and assert the rollback, and drive the two-writer conflict cases for the definition check and the first-write race. The shared metastore contract runs against PostgreSQL at every schema version.

This PR will remain Draft until #5391 merges and the rebased SHA passes CI.

Context

AI-assisted contribution

AI assistance was used for implementation and consistency review. I reviewed the final behavior and diff and take full responsibility for the contribution.

Checklist

  • 🛡️ Don't disclose security issues! (contact security@apache.org)
  • 🔗 Clearly explained why the changes are needed, with design and dev-list links
  • 🧪 Added/updated tests with good coverage, or manually tested
  • 💡 Added comments for complex logic
  • 🧾 Updated CHANGELOG.md
  • 📚 Updated documentation in site/content/in-dev/unreleased

Add the OpenAPI specification for tag definitions and tag assignments to the
Polaris Catalog API: tag definition CRUD, assign/unassign against a structured
target, a reverse lookup from a definition to its directly assigned targets, and
a per-target read with direct and effective views.

Targets are catalog, namespace, table-like, and one top-level column. v1 supports
Iceberg tables; Iceberg views are excluded because Iceberg view metadata does not
maintain stable column field ids across replaces. Generic tables can be tagged as
whole objects, but not their columns, since v1 defines no stable column identifier
for them.

The mappings route shape follows the existing Policy APIs so both features expose
the same relationship model. This change registers the generated models only; the
API resource is registered together with its service implementation.

Related to apache#5442
Implement the definition side of the Tag Management API: create, list,
load, update and drop a tag definition, with catalog-scoped authorization.

- TagEntity: a TAG entity type parented by the catalog, storing comment,
  allowed-values and target-types in the entity properties map,
  following PolicyEntity's storage pattern; no persistence schema change.
  A live tag blocks dropping its catalog the same way live namespaces do.
- Privileges TAG_CREATE/TAG_READ/TAG_DROP/TAG_WRITE/TAG_LIST/
  TAG_FULL_METADATA and operations CREATE/LOAD/DROP/UPDATE/LIST_TAG,
  registered with the same RBAC semantics shape the policy operations use;
  CATALOG_MANAGE_CONTENT covers the tag privileges through the existing
  super-privilege mapping.
- TagCatalogHandler/TagCatalog: authorize-then-delegate following
  PolicyCatalogHandler, resolving the tag as a catalog-child leaf the same
  way catalog roles resolve; updateTag enforces current-tag-version and
  treats target-types as create-only; a lost compare-and-swap surfaces as
  a retryable 409 conflict, matching the policy store; dropTag authorizes
  first and answers 501 for detach-all until tag assignments land; the
  assignment PR implements the two-privilege detach-all path.
- Tag endpoints are gated by ENABLE_TAG_STORE, disabled by default, the
  same way the policy store is gated: the adapter enforces the flag and
  the catalog config contributor advertises tag endpoints only when it is
  enabled. Enable it on deployments backed by the JDBC or in-memory
  metastores; the NoSQL metastore does not support tags yet.
- TagCatalogAdapter and the TagApi codegen registration land together, so
  the generated resource's service injection resolves; endpoints are
  advertised via the catalog config contributor; before/after events are
  emitted like other catalog APIs.
- updateTag uses the generated UpdateTagRequest model. An omitted mutable
  field stays unchanged, and for the two nullable lists, allowed-values
  and target-types, an explicit JSON null is treated as omitted; a present
  allowed-values list replaces the whole list, an empty-string comment
  clears the comment, and a non-null target-types value, including an
  empty list, is rejected because target-types is create-only. A no-op
  update writes nothing and returns the tag with its current version token.
- Named top-level string enums now generate with their declared wire
  values: TargetType previously serialized Java constant names, so
  table-like could never round-trip. Only enums whose names differ from
  their values change shape on the wire; all others are byte-identical.
- Integration tests mirror the policy service tests across the in-memory
  and JDBC backends, including an order round-trip for allowed-values and
  negative tests for the documented definition-side errors reachable
  before assignments exist.

Related to apache#5442
Adds the tag assignment write path end to end: a TagAssignmentRecord
persistence interface with default-throw capability probes (mirroring the
policy mapping persistence split), JDBC and in-memory implementations, a
new JDBC schema v7 carrying the tag_assignment_record table with a
fail-closed version gate for assignment writes, assignTag and unassignTag
with two-securable authorization and column-to-field-id resolution, and
an atomic combined delete backing dropTag with detach-all=true: no Tag
API read observes the definition without its assignments or a partially
removed state.

Related to apache#5442
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant