table_exists error raisng 400 error code

[dongsupkim-onepredict]

Apache Iceberg version

0.8.0 (latest release)

Please describe the bug 🐞

pyiceberg 0.8
It raises a 400 status exception even though the table exists. I’m currently using the REST catalog with tabulario/iceberg-rest:1.6.0

error log is as follows:

2024-11-24T10:33:10.809 ERROR [org.apache.iceberg.rest.RESTCatalogServlet] - Error processing REST request org.apache.iceberg.exceptions.RESTException: Unhandled error: ErrorResponse(code=400, type=NamespaceNotEmptyException, message=Namespace test_ns is not empty. 10 tables exist.) org.apache.iceberg.exceptions.NamespaceNotEmptyException: Namespace test_ns is not empty. 10 tables exist. at org.apache.iceberg.jdbc.JdbcCatalog.dropNamespace(JdbcCatalog.java:520) at org.apache.iceberg.rest.CatalogHandlers.dropNamespace(CatalogHandlers.java:166) at org.apache.iceberg.rest.RESTCatalogAdapter.handleRequest(RESTCatalogAdapter.java:332) at org.apache.iceberg.rest.RESTServerCatalogAdapter.handleRequest(RESTServerCatalogAdapter.java:42) at org.apache.iceberg.rest.RESTCatalogAdapter.execute(RESTCatalogAdapter.java:552) at org.apache.iceberg.rest.RESTCatalogServlet.execute(RESTCatalogServlet.java:100) at org.apache.iceberg.rest.RESTCatalogServlet.doDelete(RESTCatalogServlet.java:84) at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:722) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:840) at org.apache.iceberg.rest.RESTCatalogAdapter.execute(RESTCatalogAdapter.java:569) at org.apache.iceberg.rest.RESTCatalogServlet.execute(RESTCatalogServlet.java:100) at org.apache.iceberg.rest.RESTCatalogServlet.doDelete(RESTCatalogServlet.java:84) at javax.servlet.http.HttpServlet.service(HttpServlet.java:713) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:722) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:840)

[kevinjqliu]

hi @dongsupkim-onepredict thanks for raising this issue. can you also include the python code that cause this error?

[dongsupkim-onepredict]

@kevinjqliu sure. :)

from pyiceberg.catalog import load_catalog
_catalog.table_exists(identifier=(namespace, table_name))
rest_catalog_config = {
            "uri": uri,
            "s3.endpoint": s3_endpoint,
            "s3.access-key-id": s3_access_key_id,
            "s3.secret-access-key": s3_secret_access_key,
            "py-io-impl": "pyiceberg.io.pyarrow.PyArrowFileIO",
        }
_catalog = load_catalog("rest", **rest_catalog_config)
_catalog.table_exists(identifier=(namespace, table_name))

[kevinjqliu]

type=NamespaceNotEmptyException, message=Namespace test_ns is not empty. 10 tables exist.)

it seems like the issue is from the rest catalog itself. the drop_namespace call errored

iceberg-python/pyiceberg/catalog/sql.py

Line 569 in df804c4

raise NamespaceNotEmptyError(f"Namespace {namespace_str} is not empty. {len(tables)} tables exist.")

[dongsupkim-onepredict]

Is there any plan to update the RestCatalog Docker image regarding this issue?

[kevinjqliu]

yep, in #1389

[sungwy]

Hi @dongsupkim-onepredict ! Please try using this docker image from the dockerhub instead: apache/iceberg-rest-fixture

I'm closing this issue as there is now a table_exists test in the integration test suite against the new image: #1389

[dongsupkim-onepredict]

@sungwy Thanks for help, But unfortunately it lacks postgres driver for catalog internal database that I'm using instead of default sqliteDB. Don't you have any plans to update this issue from tabulario/iceberg-rest image?

[Fokko]

Hey @dongsupkim-onepredict unfortunally the tabulario/iceberg-rest is deprecated in favor of apache/iceberg-rest-fixture. You should be able to use Postgres as a backend, but remember that both images are not intended for production usage.