Polaris Tag Management: tag definition CRUD end-to-end - #5391
Draft
flyingImer wants to merge 2 commits into
Draft
flyingImer wants to merge 2 commits into
flyingImer wants to merge 2 commits into
Conversation
3 of 6 tasks
jbonofre
self-requested a review
August 27, 2026 12:00
This was referenced Sep 3, 2026
flyingImer
force-pushed
the
tag/pr2-definition
branch
from
September 10, 2026 04:57
69906a2 to
db254b0
Compare
flyingImer
force-pushed
the
tag/pr2-definition
branch
7 times, most recently
from
September 23, 2026 05:03
e304408 to
0ae37bc
Compare
6 tasks done
Add the OpenAPI specification for tag definitions and tag assignments to the
Polaris Catalog API: tag definition CRUD, assign and unassign against one
target, a reverse lookup from a definition to its directly assigned targets, and
a per-target read with direct and effective views.
Target kinds are CATALOG, NAMESPACE, TABLE, VIEW and COLUMN. TABLE covers
Iceberg tables and the generic tables an implementation supports, VIEW covers
whole Iceberg views, and COLUMN is limited to top-level Iceberg table columns
because generic tables and views define no stable column identifier in v1. That
limit applies to columns alone and does not restrict view targets.
A target is named by query parameters, not in a request body, and the same set
addresses all three operations that take one: assign is a PUT with only values
in the body, unassign is a DELETE with no body, and the per-target read uses the
same parameters. target-type is required and is never inferred, so a request
that omits it is rejected rather than treated as addressing the catalog.
Assignments live under a definition at .../tags/{tag-name}/assignments, and the
reverse lookup is a GET on that same collection. The catalog is addressed by the
same prefix path parameter the other catalog-scoped Polaris APIs take, defined in
this document rather than referenced so the specification stands on its own.
A definition's values are the choices a new assignment may select; an
assignment's values carries the value that was assigned. Removing a value from a
definition leaves existing assignments readable. Each definition value is
limited to 2000 bytes, measured on the decoded value in UTF-8. target-types
keeps its name because it lists permitted object kinds rather than one target,
and it is create-only: a request that carries it on update, including as an
explicit null, is rejected. Create, load and update return the definition
directly, with its version token on it; update and rename carry that token back
as current-tag-version in the request body, and the token identifies the
definition as well as its revision, so a token from a same-name replacement is
refused.
Errors are self-described and Polaris-owned. The spec defines
PolarisErrorResponse, every response lists the error types it can carry, and no
error type this spec lists carries an Exception suffix. Java class names are
unchanged. Dropping a definition with detach-all=true removes the definition and
every one of its assignments from every read together, or changes nothing;
physical cleanup may happen later and is never observable.
Collections paginate by default and a full result is asked for explicitly. A
request with no pagination parameters returns the first page, an empty pageToken
also asks for the first page, and a token continues from where the previous page
ended. pagination=false asks for the complete result in one response and has to
arrive alone, so combining it with pageToken or pageSize is rejected with 400.
Tag lists paginate by default while the Iceberg REST catalog returns everything
when pageToken is absent, and the same parameter names do not make the two
defaults interchangeable; the difference is deliberate. Both modes are bounded by
finite response-size and work limits that exist by default, and a full result
that would exceed one is rejected with 400 rather than silently truncated or
switched to paged mode, with a message pointing the client back to paged mode.
pageSize carries no schema minimum, so a zero or negative value reaches the Tag
handler and is answered with the BadRequest type this specification names for
invalid pagination input rather than a generic schema-validation type; pagination
is judged the same way, against the value the query carried, so an empty or
unrecognized value is rejected instead of being read as a full-result request.
The per-target read paginates on the same terms. Whether Polaris later adopts one
shared pagination contract across its APIs is still open on the dev list.
The reverse lookup requires read access to the named definition and then to each
target it would report, with columns checked through their table, and it
excludes targets that are dropped, soft-dropped or missing. Each operation
states the authorization it requires in plain words and names no privilege, so
the contract does not fix how a server authorizes.
Renaming a definition is its own operation, and createTag, updateTag and renameTag
accept an optional Idempotency-Key: where a deployment enables idempotency, a
retry inside the advertised window is recognized as the write that already
succeeded rather than performed again, and the server acknowledges that
completion instead of replaying a stored response.
This change registers the generated models only; the API resource is registered
together with its service implementation.
Related to apache#5442
Implement the definition side of the Tag Management API: create, list, load, update and drop a tag definition, with catalog-scoped authorization. - TagEntity: a TAG entity type parented by the catalog, storing description, values and target-types in the entity properties map, following PolicyEntity's storage pattern; no persistence schema change. A live tag blocks dropping its catalog the same way live namespaces do. Each value is limited to 2000 bytes, measured on the decoded value in UTF-8 so JSON escaping does not count, on update as well as create, since both go through the same validation. - Privileges TAG_CREATE/TAG_READ/TAG_DROP/TAG_WRITE/TAG_LIST/ TAG_FULL_METADATA/TAG_DETACH and operations CREATE/LOAD/DROP/UPDATE/ LIST_TAG, DROP_TAG_DETACH_ALL and RENAME_TAG, registered with the same RBAC semantics shape the policy operations use; CATALOG_MANAGE_CONTENT covers the tag privileges through the existing super-privilege mapping. Renaming is registered two-sided, the way RENAME_TABLE and RENAME_VIEW are: TAG_DROP on the definition whose name disappears plus TAG_CREATE on the catalog that gains one. Neither side alone carries it. - TagCatalogHandler/TagCatalog: authorize-then-delegate following PolicyCatalogHandler, resolving the tag as a catalog-child leaf the same way catalog roles resolve. createTag, loadTag and updateTag return the definition itself, with its version token on it, rather than a wrapper around it. updateTag enforces current-tag-version. The token names the definition as well as its revision, so a token taken before the definition was deleted and recreated under its name, or before another definition was renamed into that name, is refused as a version mismatch exactly like a stale one; the check and the write are one compare-and-swap, not a check followed by a write, and a lost compare-and-swap surfaces as a retryable 409 conflict, matching the policy store. detach-all removes the definition and every assignment of it together, and takes TAG_DETACH on the definition as well as TAG_DROP even now, while no assignment can exist: what the caller asks for decides which permission it takes, not what happens to be there to remove. - Availability is two questions, not one. ENABLE_TAG_STORE, disabled by default, says whether a realm offers tags, the same way the policy store is gated. PolarisMetaStoreManager.supportsEntityType says whether the configured metastore can store them at all: the flag cannot see which metastore is running, so enabling it on one without tag storage would otherwise reach persistence and fail there. The NoSQL metastore has no object mapping for TAG and reports it, so the config endpoint stops advertising tags and the adapter refuses the routes before resolving anything. - updateTag replaces the whole editable definition rather than patching fields. description and values are both required. A null description and an empty one both answer 200 and read back as null, because they are two spellings of the same thing and the stored form is null; a string sets it; omitting the field answers 400 ValidationError and changes nothing. Sending back what a read returned is therefore a no-op even for a definition that never had a description. A request that asks for the state the definition already has still has its token checked, returns the current definition and advances no version. name is no longer a field of the request, because renaming is its own operation, and target-types stays create-only; a request carrying either is rejected with 400. Unknown properties are ignored service-wide, so those rejections are deserializers for these request types rather than a change to how any other schema treats unknown input. - Omitting target-types on create selects every target kind this version defines and stores that set explicitly, so a kind added later cannot widen a definition that already exists. An explicit null is a different request and stays invalid: it names no kinds, which no definition can have. - A definition carries a read-only id, the entity id as a string, in every response that names it. It survives renames and updates, and a definition that is deleted and recreated under the same name receives a different one, so a client can recognize the same definition under a new name and tell a same-name replacement apart. Requests still address definitions by name; a list result pairs each name with the id from the same lookup record, so no response can pair a name with a foreign id. - createTag, updateTag and renameTag honour Idempotency-Key through the shared entity mechanism, so a retry after a lost response is recognized instead of applied twice: create and update answer with the definition as it stands, and rename answers 204. Each key is recorded by the same entity write as the change it stands for, because two metastore calls are two transactions here and a key written on its own would promise a change that may never have landed. Recognition needs the definition to still exist, the key to still be live and the caller to still be authorized. A no-op update records nothing, so a later retry of it can legitimately answer 409 once the definition has moved on. A rename retry cannot be found by either of its own names, since the source is free by then and the destination may have moved again or been reused, so it is located by the entity id inside the version token it still carries; authorization then runs against the name that definition holds now, through the ordinary name-keyed path, and the id is re-checked before the answer. Delete and the assignment operations have no such guarantee, and the shared filter accepting the header does not create one. - The error types the Tag API returns omit the Exception suffix, and an exception class name does not select the wire type: a name collision answers AlreadyExists, a request whose shape the schema rejects answers ValidationError, and a missing surrounding catalog answers NoSuchCatalog through a tag exception that is distinguishable from every other 404. The contract draws a line between its own validation errors and a generic schema failure, so create reads its body through a deserializer of its own: a missing values or target-types list, and a tag name outside the declared pattern on create or on rename, are Tag validation errors and answer BadRequest, while a missing name or version token stays a schema failure and answers ValidationError. Left to the generated model those first cases are answered by the shared required-property and constraint paths, which run before any handler and report the generic literal. Binding each field there also keeps a wrong JSON type from escaping the body reader with no error envelope at all. A field the schema declares a string is read as one rather than converted, so a number or a boolean sent as a description on create or update, or as a version token, is a malformed request. A declared array of strings is read the same way member by member: a number, a boolean, an object or a nested array inside values is refused on create and on update rather than converted to its own text. Converted it would become an allowed value the client never sent, and nothing after the body reader could notice, because the JSON type is gone once a list of strings exists. The exception mappers are shared with every other API and tag also throws exceptions it does not own, so no set of exception classes separates tag errors; a response filter bound to the tag resource rewrites the type instead. Java class names are unchanged and no other API's payload changes. - listTags returns the first page unless the request asks for the whole collection. An omitted or empty pageToken asks for the first page and a pageSize bounds it; pagination=false is the only way to ask for the complete result in one response. That flag has to arrive alone, because a pageToken or a pageSize inside a request that is not paged contradicts its own mode, so either one is refused with 400 even when its value is empty. Presence is read from the request's query map rather than from the bound parameters, since the binding hands a present-but-empty query value to a String as null and would hide exactly that contradiction. The flag's own value comes from the same place and is accepted only as the literal true or false: the generated parameter is a boolean whose conversion never fails, so an empty or misspelled value would otherwise arrive as a silent request for everything. The query map also answers a pagination, pageToken or pageSize sent more than once, which has no single value to act on, with 400 naming the parameter. A full result larger than the configured limit is refused with 400 pointing back to paged mode, decided before a response is committed rather than truncated into something a client would read as complete. A token this server did not issue, or issued for a listing of a different catalog or a different realm, is refused with 400 as well, because a token belongs to the listing that produced it. A catalog id is only unique inside the realm that allocated it, so the realm is part of what a token is checked against. Upstream already answers an unreadable token with 400, so what the tag path adds there is the error type this API documents for one, rather than the generic schema-validation type a 400 raised anywhere else would carry. A pageSize is refused unless it is a positive integer, because it is an input whether or not it is used. An empty one is refused on presence, since an unusable value arrives as no value at all. A value that is not a number fails conversion before any tag code runs, which the platform answers 404, so the tag response filter corrects that one case to the 400 these routes promise; a not-found the tag code raised itself is left alone, and the proper fix is a parameter converter for every API in a separate change. Three configuration entries hold the paged-mode default size, the ceiling that applies when no deployment maximum is set, and the unpaginated limit, all read through one accessor, so a catalog-level override reaches every one of them and adopting a shared setting later is a one-site change. A page stays bounded by default: a listing that is bounded only when an operator remembers to bound it is not bounded. - TagCatalogAdapter and the TagApi codegen registration land together, so the generated resource's service injection resolves; the tag error envelope maps to the Iceberg error response the mappers already build, rather than generating a second model nothing constructs; endpoints are advertised via the catalog config contributor; before/after events are emitted like other catalog APIs. - Integration tests mirror the policy service tests across the in-memory and JDBC backends, and assert the wire status and error type for the definition-side failures reachable before assignments exist. Related to apache#5442
flyingImer
force-pushed
the
tag/pr2-definition
branch
from
September 24, 2026 22:55
0ae37bc to
3fc6a15
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tip
Until #5366 merges, the files view includes the spec commit from #5366. For this slice's own diff, review the head commit: 3fc6a15
GH proposal tracking: #5442
Why
#5366 defines the public Tag API. This PR makes Tag definitions usable: catalogs can create, list, load, update, rename, and delete them.
This is definition CRUD only. Assignment storage and reads remain in the next slices, so this PR does not lock in their persistence model.
Scope
Tags are disabled by default and supported on the JDBC and in-memory metastores. NoSQL remains unsupported.
Create, load and update answer the definition itself, with its
versionon it, and rename answers 204 with no body. A definition'sversionis an opaque string. Clients send the value they last read back incurrent-tag-versionand do not parse, order, or increment it. An update must match the current token, including one that would change nothing, and a stale token is refused with 409TagVersionMismatch; a token that arrives as anything other than a string is a 400ValidationErrorinstead of being converted to one, and so is adescriptionon create or update that arrives as a number, a boolean, an object or an array. The same holds for a member of thevaluesarray on either operation: a number, a boolean, an object or a nested array is refused rather than read as its own text, which would otherwise record an allowed value the client never sent. An explicit null or an empty string as the updatedescriptionclears it, and a cleared description reads back as null. A request that changes nothing leaves the definition alone and returns the token it was given. Any change issues a new token and invalidates earlier ones, including a change that sets a field back to a previous value; a token issued for a definition that was since deleted cannot update a new definition that took over its name, and neither can a token from a definition whose name another definition was renamed into: both are refused with 409TagVersionMismatchexactly as a stale token is. The token check and the change are atomic. Eachvaluesmember is limited to 2000 bytes, measured on the decoded value in UTF-8, so JSON escaping in the request does not count toward it. The wire contract fixes none of this in storage: the backend keeps whatever representation it already has.Until assignments land, an authorized
detach-all=truerequest removes the definition the same way an ordinary drop does.Management grants for Tag privileges remain a separate follow-up. Using Tags for authorization remains outside v1. A query parameter that cannot be read as an integer, such as
pageSize=large, answers 404 from the REST layer today; the tag routes rewrite that one case to 400, and a follow-up change will fix it for every API so the rewrite can go away. ApageTokenthis server did not issue, or issued for a listing of a different catalog or a different realm, is refused with 400 as well, because a catalog id is only unique inside the realm that allocated it.Validation
Coverage includes CRUD, authorization, version and conflict behavior, feature gating, supported backends, and cleanup.
This PR will remain Draft until #5366 merges and the rebased SHA passes CI.
Context
AI-assisted contribution
AI assistance was used for implementation and consistency review. I reviewed the final behavior and diff and take full responsibility for the contribution.
Checklist
CHANGELOG.mdsite/content/in-dev/unreleased