REST Spec: Add resolve endpoint for catalog objects

[stevenzwu]

Design doc: https://docs.google.com/document/d/1VW5hgaaajRWtp5KbOU3s83YyoyPi5WOSvHtoJ_yXzJs/edit?tab=t.0#heading=h.e6w7vgpr8t2f

Adds POST /v1/{prefix}/resolve — a single endpoint that takes one or more typed arrays of catalog items (currently relations; extensible to e.g. functions) and returns their current state with per-item outcomes (loaded, not-modified, not-found) plus a typed unprocessed list for partial-progress capping.

[stevenzwu]

Future materialized-view would look like

{
  "object-type": "materialized-view",
  "view": { },
  "storage-table": { }
}

[danielcweeks]

No, I don't think we would have a new type. It would just have a storage-table associated with it, which is what makes it a materialized view. I'm not sure we need a separate type.

[danielcweeks]

I have a minor objection to the use of relations here since we want this to be a general endpoint for resolution. Objects like table/view are considered relations, but something like a function would not (unless you're for a strictly relational algebra definition, but that's not consistent with sql usage).

We may also include other objects in the future, so a more general term like resolve, identifiers, resources, or entities might be better.

[stevenzwu]

this is related to the other identifier conflict domain discussion where we want to allow the same identifier for a relational object (like table) and a function. With that assumption, the endpoint will need to have object category in the path to distinguish them. Otherwise, we would require identifier uniqueness across all object types, which is not the consensus from the identifier conflict discussion.

[danielcweeks]

Seems like a good time to introduce Identifier that is just the same as a table identifier. Seems odd we would continue to use an object specific identifier type to reference multiple.

Since it has the same structure, you could possibly just have then extend identifier (depending on how that affects the open api structure and generated code).

[stevenzwu]

I am actually about to share a proposal with the community. https://docs.google.com/document/d/1NTQhgNbP2dkIMuXUMA5JdwliVQKCp1TU_ux5J_AaPiw/edit?tab=t.0#heading=h.xzrfzeom8dqa

[danielcweeks]

This feels awkward because we're using HTTP status codes for non-request/internal results. I don't think that makes a lot of sense and prefer we indicate result behavior in a different way.

[stevenzwu]

What about defining a enum schema in the REST spec? it will be similar to the http status name though.

    BatchLoadItemResultStatus:
      type: string
      description: |
        The outcome of loading a single item in a batch load response.
      enum:
        - success
        - not-modified
        - not-found

Open to other suggestions.

[stevenzwu]

BTW, previously @gaborkaszab and @jbonofre suggested using http status code in the design doc comment.

[stevenzwu]

I did a little more research. Two patterns are common.

1. use http status code

• Microsoft Graph API — JSON batching where each item has its own integer HTTP status, headers, and body. Docs
• Facebook/Meta Graph API — Each item has a code field (HTTP integer) plus optional headers and body. Docs
• Elasticsearch Bulk API — Each item has an integer status field (e.g. 200, 201, 404, 409) plus a string result field (e.g. "created", "updated", "not_found"). Docs

2. split into separate lists. AWS services commonly use this pattern.

• AWS DynamoDB BatchGetItem — Found items in Responses, absent items silently omitted, incomplete items in UnprocessedKeys. No status field. Docs
• AWS SQS SendMessageBatch / DeleteMessageBatch — Results split into Successful and Failed lists. Failed entries have Code (string error code like "InvalidParameterValue"), not HTTP status integers. Docs
• AWS S3 DeleteObjects — In verbose mode, successful deletes listed in Deleted, failures in Errors with string Code (e.g. "AccessDenied"). Docs

[danielcweeks]

I would personally prefer a more structured response than just embedding HTTP codes. I ran into this issue with GraphQL, which returns 200 (403) or something which is a bit confusing.

[stevenzwu]

Agreed. Switched the per-item shape to a discriminated union using the same object-type-style pattern already in LoadRelationResult:

BatchLoadRelationResultItem:
  oneOf:
    - $ref: '#/components/schemas/BatchLoadRelationLoaded'
    - $ref: '#/components/schemas/BatchLoadRelationNotModified'
    - $ref: '#/components/schemas/BatchLoadRelationNotFound'
  discriminator:
    propertyName: result-type
    mapping:
      loaded:       '#/components/schemas/BatchLoadRelationLoaded'
      not-modified: '#/components/schemas/BatchLoadRelationNotModified'
      not-found:    '#/components/schemas/BatchLoadRelationNotFound'

Each variant declares exactly its own required fields:

• loaded → result (required) + etag (optional, tables only)
• not-modified → etag (required)
• not-found → identifier (required)

Why domain-native names over HTTP status integers:

• Wire stays honest. HTTP codes describe transport; per-item outcomes describe domain state. Keeping them separate means caches, retries, tracing, and monitoring don't have to peek inside the body to know what actually happened.
• Presence rules become schema-enforced. oneOf lets each variant require its own fields instead of relying on prose in the description.
• Stronger generated clients. result-type generates into sealed interfaces / discriminated unions / sum types, so the compiler catches missed cases. An integer status generates into a plain int that callers switch on by hand.
• Extensible without fake codes. Adding outcomes like skipped or stale-etag-mismatch later is a new variant — no need to overload 429 or invent a meaning for an HTTP code that doesn't fit.
• Internally consistent with this PR. LoadRelationResult already uses object-type as a string discriminator; reusing the same style for per-item outcomes keeps the reader's mental model uniform.

[stevenzwu]

We may add values such as materialized-view or function in the future.

[danielcweeks]

Materialized view is just a type of view, so I'm not sure we need to distinguish.

[stevenzwu]

The response object is different for MV, which contains both view metadata and storage table metadata. If we want to load a MV in one round trip, a specific MV type is needed so that client knows how to parse the response.

In the Java library, we may need to define a MaterializedView type, which could be mostly just a container class for a View and a Table fields.

[ldsantos0911]

At either this layer or the URL layer, would it make sense to honor the referenced-by parameter available to loadTable and loadView?

[danielcweeks]

I think it would have to be at the pre-request level since you need to distinguish different loads (and they may be treated differently for authorization)

[danielcweeks]

This is a little pedantic, but I'm not entirely sold on the resource path here. I looked across many different implementations and found lots of approaches. However, I don't think we need to put the /batch-load at the end. We can just leave it as POST /v1/{prefix}/relations. I could see that you might say "what if we need to create", but we already have a transactions endpoint that is for that operation.

[stevenzwu]

I added batch-load to the path because POST is used for the batch load so that the list of identifiers can be encoded in the request payload.

If we remove it, it is a bit weird to have POST /v1/{prefix}/relations for batch get purpose.

[danielcweeks]

Do we want to describe how to handle if too many items are requested? I understand the server can process a subset and return the unprocessed list, but what if someone lists an entire catalog and then asks for 100K resources? What error code can/should the server return if they consider the request unreasonable (400 might fit that)

[stevenzwu]

Good catch, thanks. Updated the spec.

unprocessed-identifiers was only meant as a cooperative soft cap — the server accepted the request, then stopped partway due to its own cost budget. It doesn't cover the "please reject this upfront" case you described.

Added a typed hard cap to CatalogConfig:

relations-batch-load-max-items:
  type: integer
  minimum: 1

Advertising the limit lets well-behaved clients chunk proactively, and servers that receive an oversized request still have a spec-supported way to reject with 400.

The batchLoadRelations description and unprocessed-identifiers field now spell out the two mechanisms as distinct:

• relations-batch-load-max-items — governs whether the request is accepted at all (400 if exceeded).
• unprocessed-identifiers — reports partial progress within an accepted request.

Went with 400 rather than 413 since the limit is on identifier count (a request property), not body size — happy to switch if you disagree.