`catalog.table_exists()` returns 'False' when table exists in Polaris catalog

[JasperHG90]

Apache Iceberg version

0.8.0 (latest release)

Please describe the bug 🐞

Hi 👋

I see quite a few issues/PRs centered around the table_exists() method when using REST catalogs, e.g.

• Fix table_exists behavior in REST catalog #1096
• Adjust the "table_exists" behavior in the REST Catalog #1018
• pyiceberg always return false for catalog.table_exists when used with Polaris catalog #1006
• Rest Catalog: table_exist does not succeed on 204 response code #735

When using Polaris OSS, I also experience issues with this method. Even though a table exists the catalog.table_exists() method will return 'False'.

I have a devcontainer setup that you can find here to make the example below reproducible.

Python code:

#%%

import httpx
import polars as pl
from pyiceberg.catalog import load_catalog

# Assumes this catalog exists and credentials stored in
#  .pyiceberg.yaml
# Possibly created using https://github.com/JasperHG90/dagster-pyiceberg-example-polaris/blob/main/scripts/set_up_catalog.py
catalog = load_catalog(
    name="dagster_example_catalog",
)

# %% Create a namespace

catalog.create_namespace("example")

# %% Create data and add to table

df = pl.from_dict({
    "a": [1, 2, 3],
    "b": [4, 5, 6]
}).to_arrow()

catalog.create_table(
    identifier="example.table",
    schema=df.schema
)

# %% This will return 'False'

catalog.table_exists("example.table")

# %% But querying the 'table exists' Polaris endpoint  returns 200

catalog_client = httpx.Client(
    base_url="http://polaris:8181/api/catalog/v1",
    headers={
        "Authorization": "Bearer principal:root;realm:default-realm",
        "Content-Type": "application/json",
    }
)

# See endpoint here: https://polaris.apache.org/in-dev/unreleased/rest-catalog-open-api/#tag/Catalog-API/operation/tableExists
resp = catalog_client.head(
    url="dagster_example_catalog/namespaces/example/tables/table"
)

resp.status_code # 200

[kevinjqliu]

Thank you for bringing this up. Its been on my todo list to investigate this as part of #1018 (comment)

can you help me go through the same exercise and step through the table_exists call? In #1018, you can see the response code is 404, which should return False according to the implementation and the spec

I want to see if this is a polaris issue or pyiceberg issue

[JasperHG90]

I've annotated this method with what I see in the debugger

  @retry(**_RETRY_ARGS)
  def table_exists(self, identifier: Union[str, Identifier]) -> bool:
      """Check if a table exists.

      Args:
          identifier (str | Identifier): Table identifier.

      Returns:
          bool: True if the table exists, False otherwise.
      """
      identifier_tuple = self._identifier_to_tuple_without_catalog(identifier) # ('example', 'test')
      # Url: 'http://polaris:8181/api/catalog/v1/dagster_example_catalog/namespaces/example/tables/table'
      response = self._session.head(
          self.url(Endpoints.load_table, prefixed=True, **self._split_identifier_for_path(identifier_tuple))
      )

     # response.status_code is 200

     # This code block is skipped
      if response.status_code == 404:
          return False
      elif response.status_code == 204:
          return True

     # This code block is skipped
      try:
          response.raise_for_status()
      except HTTPError as exc:
          self._handle_non_200_response(exc, {})

      # ... And 'False' is returned because 200 != 204
      return False

Polaris is returning a <200> status code, so that's not correct according to your spec.

This method is somewhat confusing because the code is checking explicitly for two status codes, and raises in case of a non-2xx status code, but will return 'False' in case of 404/200 and 'True' in case of 204. Although it’s probably fair not to expect another 2xx status code, it is confusing for the end-user to not at least raise a warning.

[kevinjqliu]

Thanks @JasperHG90 for the breakdown. The server spec above describes what the server should send and what the client should expect.
According to the spec, Polaris should be sending a 204 for success, not 200.

Do you mind opening an issue with Polaris?

[JasperHG90]

FYI PR that fixes status code for this and other endpoints: apache/polaris#472

[Fokko]

To make PyIceberg a bit friendlier to the catalogs, we could also allow for HTTP 200

[kevinjqliu]

apache/polaris#472 is merged @JasperHG90 do you mind testing it again with the latest code?

[JasperHG90]

Tested and works. Closing this issue. Thanks @kevinjqliu !

[kevinjqliu]

cc @djouallah from #1018 (comment)