Enhance iceberg-go to Support Nessie API for All Catalog Operations

Feature Request / Improvement

Description

The current Catalog interface in iceberg-go is designed for the Iceberg REST catalog, but it does not fully align with Project Nessie’s API requirements. Nessie uses different API paths, request payloads, and reference (branch/tag) handling, which needs adjustments across all catalog operations. And it has different API versions too.

For example, Namespace Creation
CreateNamespace function in catalog/rest.go uses POST method while nessie v1 api uses PUT method
API PATH is also different. I have seen similar issues with multiple functions

I am willing to make changes to support Nessie apis, would appreciate some guidance on how to go about it?

Should we modify current catalog/rest

Use ref and other required params from context.Context for Nessie API.
Dynamically determine API paths based on whether Nessie or Iceberg REST Catalog is being used.

Create a new catalog/nessie to support and maintain nessie specific functionality separately

I have created a PR to support Nessie via catalog/rest (namespace related functionalities)

@Fokko

@Fokko @kevinjqliu can you two chime in here? My personal opinion is that we were following the REST spec, so if Nessie isn't following the REST spec, we should do a separate catalog. But, is this actually just a case where Nessie was being more strict than the spec necessarily required? Were we not actually following the spec correctly? Or is Nessie just wrong?

@zeroshade

hey @zeroshade thats a great question. Based on my understanding, Nessie has 2 sets of APIs, its own catalog API and an Iceberg REST compliant API.
Heres the nessie docs on iceberg rest api https://projectnessie.org/guides/try-nessie/

Based on this comment, Nessie should have support for Iceberg REST API. In fact, there are many users of pyiceberg interacting with Nessie.

There are subtle difference in behaviors. One big one is apache/iceberg-python#1105 where Nessie only shows 1 snapshot. This is part of the Nessie implementation so not much we can do.

@shubham-tomar

@shubham-tomar Given that users are using pyiceberg with nessie without needing separate changes to behavior, and iceberg-go is compliant with the REST api (as I based the implementation on pyiceberg), do we actually need changes to the behaviors to support nessie?

Since iceberg-go is a client library for the Apache Iceberg project, I would say adding the custom catalog implementation for Nessie native API's is out of scope. Perhaps the implementation can be adopted by the Nessie project itself

I'd say the same. Just wanted to get other opinions to make sure i'm not alone on that one. The catalog package makes it easy to register new implementations so that any other project (or separate repo) could potentially embed the REST catalog and then override the necessary changes for Nessie's specific features rather than us having to maintain it here.

I'll comment on the PR accordingly.

@zeroshade

Hi @zeroshade, @kevinjqliu,

I want to use Nessie as a catalog for Iceberg in my data pipeline, which is entirely written in Go. Given the discussion around my PR, I want to clarify the best approach moving forward.

Would it make sense to fork iceberg-go and create a dedicated catalog/nessie package that:

Implements Nessie-specific API paths and structures while keeping compatibility with Iceberg.
Use the core functionalities of iceberg-go, ensuring minimal duplication and consistency with the Iceberg ecosystem.

This way, I wouldn’t modify catalog/rest directly but instead build a separate Nessie catalog implementation within my fork. Does this sound like the right approach, or do you have alternative recommendations?

Essentially i wanted to use Nessie for branching logic, and iceberg-go for data CRUD operations

Thanks for your guidance!

Yes you're essentially building a "nessie-go-client" based off of the Nessie REST API, which is different from Iceberg's REST API

Nessie already has 2 other clients, python and java

@shubham-tomar

@shubham-tomar you don't need to fork iceberg-go. You simply just need to implement the catalog interface. Once you register the catalog with the registry, you can use it with the rest of the iceberg-go library as-is.

That's why it's separated using the interfaces. You won't need to fork the whole library (and you shouldn't!)

Got it, thank you for your guidance.

@shubham-tomar

@shubham-tomar If you end up needing help, please reach out because I would really prefer to avoid needing to fork this whole lib 😄

@zeroshade

Hi @zeroshade, i am facing issue while creating table

This is how i am loading catalog

config.URI := "http://localhost:19120/iceberg"
config.WarehouseLocation := "s3://warehouse"

iceCatalog, err := catalog.Load(ctx, catalogName, iceberg.Properties{
		"uri":                  config.URI,
		"warehouse":            config.WarehouseLocation,
		"s3.endpoint":          "http://minio:9000", // MinIO endpoint
		"s3.access-key-id":     "admin",             // Default MinIO access key
		"s3.secret-access-key": "password",          // Default MinIO secret key
		"s3.path-style-access": "true",              // Required for MinIO
		"s3.region":            "us-east-1",         // Default region
	})

Successfully Loaded catalog: &{0x14000aef710 0x14000b02270 nessieCatalog map[prefix:main%7Cs3%3A%2F%2Fwarehouse warehouse:s3://warehouse]}
Do endpoint: http://localhost:19120/iceberg/v1/main%7Cs3%3A%2F%2Fwarehouse/namespaces

i am able to create and List Namespace,

[my_namespace_1]

Create Table function

func CreateTable(ctx context.Context, iceCatalog *rest.Catalog, ref string, tableIdentifier string, schema *iceberg.Schema, opts ...catalog.CreateTableOpt) error {
	if schema == nil {
		return fmt.Errorf("schema is required to create a table")
	}

	ctx = iceberg.WithRef(ctx, ref)

	identifier := catalog.ToIdentifier(tableIdentifier)

	defaultOpts := []catalog.CreateTableOpt{
		catalog.WithLocation(fmt.Sprintf("s3://warehouse/%s", tableIdentifier)),
		catalog.WithProperties(iceberg.Properties{
			"format-version":       "2",
			"write.format.default": "parquet",
		}),
	}

	table, err := iceCatalog.CreateTable(ctx, identifier, schema, opts...)
	if err != nil {
		return fmt.Errorf("failed to create table: %w", err)
	}
	return nil
}

I have add logs in catalog/rest for debugging and getting this

DEBUG: Making POST request to: http://localhost:19120/iceberg/v1/main%7Cs3%3A%2F%2Fwarehouse/namespaces/my_namespace_1/tables
DEBUG: Request payload: {"name":"iceberg_table_1","schema":{"type":"struct","fields":[{"type":"int","id":1,"name":"id","required":true,"doc":"unique identifier"},{"type":"string","id":2,"name":"name","required":true,"doc":"user name"},{"type":"timestamp","id":3,"name":"created_at","required":true,"doc":"creation timestamp"}],"schema-id":0,"identifier-field-ids":[]},"location":"s3://warehouse/my_namespace_1.iceberg_table_1","partition-spec":{"spec-id":0,"fields":[]},"write-order":{"order-id":0,"fields":[]},"properties":{"format-version":"2","write.format.default":"parquet"}}
DEBUG: Request headers: map[Content-Type:[application/json]]
DEBUG: Response status: 500 Internal Server Error
DEBUG: Response headers: map[Content-Length:[118] Content-Type:[application/json;charset=UTF-8]]
DEBUG: Response body: {
  "error" : {
    "message" : "HTTP 400 Bad Request",
    "type" : "WebApplicationException",
    "code" : 500
  }
}
DEBUG: Error from POST request: WebApplicationException: HTTP 400 Bad Request
2025/02/15 11:29:16 Failed to create table: failed to create table: WebApplicationException: HTTP 400 Bad Request

Looks like some s3 config is missing but i am not able to figure out what exactly is missing, let me know

My nessie and minio configs

nessie:
    image: ghcr.io/projectnessie/nessie:0.102.5-java
    container_name: nessie
    depends_on:
      - minio
environment:
      - QUARKUS_PROFILE=iceberg
      - NESSIE_ICEBERG_REST_API_ENABLED=true
      - NESSIE_CATALOG_ICEBERG_DEFAULT_WAREHOUSE=s3://warehouse/
      - NESSIE_CATALOG_DEFAULT_WAREHOUSE=warehouse
      - NESSIE_CATALOG_WAREHOUSES_WAREHOUSE_LOCATION=s3://warehouse/
      - AWS_REGION=us-east-1
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - NESSIE_OBJECT_STORAGE_ACCESS_KEY=admin
      - NESSIE_OBJECT_STORAGE_SECRET_KEY=password
      - AWS_ENDPOINT=http://minio:9000
      - NESSIE_OBJECT_STORAGE_AUTH_MODE=APPLICATION_GLOBAL
       # Additional Nessie S3 Configuration Based on Docs
      - NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_ENDPOINT=http://minio:9000
      - NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_AUTH_TYPE=APPLICATION_GLOBAL
      - NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_ACCESS_KEY=admin
      - NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_SECRET_ACCESS_KEY=password
      - NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_PATH_STYLE_ACCESS=true
    networks:
      iceberg:
    ports:
      - 19120:19120
  # Minio Storage Server
  minio:
    image: minio/minio:latest
    container_name: minio
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=password
      - MINIO_DOMAIN=storage
      - MINIO_REGION_NAME=us-east-1
      - MINIO_REGION=us-east-1
    networks:
      iceberg:
    ports:
      - 9001:9001
      - 9000:9000
    command: ["server", "/data", "--console-address", ":9001"]

This URI seems wrong? http://localhost:19120/iceberg/v1/main%7Cs3%3A%2F%2Fwarehouse/namespaces/my_namespace_1/tables

Looking at Nessie's API description, i'm not seeing /v1/main anywhere in the REST api, it also seems wrong that it would include s3://warehouse/..... in the URI. Is this using the existing REST catalog in iceberg-go or your custom catalog for nessie interactions?

I am using the existing catalog/rest (not my custom nessie code)
It is the Prefix which is being added, but the same URI is used to create and list namespaces (which is working)

> go run main.go
Ignoring extension file:unknown.yaml err:failure reading YAML failed to parse unknown as FuncDefArgType, Skipping it 
DEBUG: Inside init
DEBUG: uri ->  http://localhost:19120/iceberg
DEBUG: r.baseURI, r.props ->  http://localhost:19120/iceberg/v1/main%7Cs3%3A%2F%2Fwarehouse map[prefix:main%7Cs3%3A%2F%2Fwarehouse warehouse:s3://warehouse]
DEBUG: ops.prefix ->  main%7Cs3%3A%2F%2Fwarehouse

Successfully Loaded catalog:  &{0x14000145d40 0x14000262810 nessieCatalog map[prefix:main%7Cs3%3A%2F%2Fwarehouse warehouse:s3://warehouse]}
DEBUG: Inside ListNamespaces
- [my_namespace_1]
DEBUG: Inside CreateTable
DEBUG: ns, tbl,  r.baseURI ->  my_namespace_1 iceberg_table_1 http://localhost:19120/iceberg/v1/main%7Cs3%3A%2F%2Fwarehouse
2025/02/16 18:46:50 Failed to create table: failed to create table: WebApplicationException: HTTP 400 Bad Request

In Nessie logs, i can see this error
problem: Cannot build IcebergCreateTableRequest, some of required attributes are not set [stageCreate]

nessie  | 2025-02-17 10:27:54,004 INFO  [io.qua.htt.access-log] (executor-thread-1) 192.168.65.1 - - [17/Feb/2025:10:27:54 +0000] "GET /iceberg/v1/config?warehouse=s3%3A%2F%2Fwarehouse HTTP/1.1" 200 1591
nessie  | 2025-02-17 10:27:54,017 INFO  [io.qua.htt.access-log] (executor-thread-1) 192.168.65.1 - - [17/Feb/2025:10:27:54 +0000] "GET /iceberg/v1/main%7Cs3%3A%2F%2Fwarehouse/namespaces HTTP/1.1" 200 73
nessie  | 2025-02-17 10:27:54,021 WARN  [org.pro.cat.ser.res.IcebergErrorMapper] (executor-thread-1) Unhandled exception returned as HTTP/500: jakarta.ws.rs.WebApplicationException: HTTP 400 Bad Request: jakarta.ws.rs.WebApplicationException: HTTP 400 Bad Request


nessie  | Caused by: com.fasterxml.jackson.databind.exc.ValueInstantiationException: Cannot construct instance of `org.projectnessie.catalog.formats.iceberg.rest.ImmutableIcebergCreateTableRequest`, problem: Cannot build IcebergCreateTableRequest, some of required attributes are not set [stageCreate]
nessie  |  at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 556]
nessie  | 	at com.fasterxml.jackson.databind.exc.ValueInstantiationException.from(ValueInstantiationException.java:47)

If i set this StageCreate: true it is able to create table but since it is stage create i am not able to see this in catalog
if i try to register it it is failing

failed to register table: failed to register table: NoSuchTableException: Table does not exist: my_namespace_1.iceberg_table_1

in nessie logs, it gives 404 maybe because iceberg_table_1 does not exist

4:23:46 +0000] "POST /iceberg/v1/main/namespaces/my_namespace_1/tables HTTP/1.1" 200 1631
nessie  | 2025-02-17 14:23:46,172 INFO  [io.qua.htt.access-log] (executor-thread-7) 192.168.65.1 - - [17/Feb/2025:14:23:46 +0000] "POST /iceberg/v1/main/namespaces/my_namespace_1/tables/iceberg_table_1 HTTP/1.1" 404 147

how to go about it

I'll see if i can find some time this week to spin up a nessie catalog myself and see if I can figure out what's going wrong

@zeroshade

Thank you @zeroshade , i have shared my docker-compose file above in the comments, let me know if you would need any other info for reproducing this.
Meanwhile i will also look if i can find something.

@zeroshade

Hi @zeroshade, i think i found the issue, it is rather very small fix
PR:link

Also, i faced issue with RenameTable function

type payload struct {
Source identifier json:"source"
Destination identifier json:"destination"
}

above payload works but we have from, to, is this changed recently? should i add this in the same PR?

>  curl -X POST "http://localhost:19120/iceberg/v1/main/tables/rename" \
     -H "Content-Type: application/json" \
     -d '{
           "source": { "namespace": ["my_namespace_1"], "name": "my_table_v2" },
           "destination": { "namespace": ["my_namespace_1"], "name": "my_table_v3" }
         }'

This works, but below curl fails
                                                                                 
>  curl -X POST "http://localhost:19120/iceberg/v1/main/tables/rename" \
     -H "Content-Type: application/json" \
     -d '{
           "from": { "namespace": ["my_namespace_1"], "name": "my_table_v2" },
           "to": { "namespace": ["my_namespace_1"], "name": "my_table_v3" }
         }'
{
  "error" : {
    "message" : "HTTP 400 Bad Request",
    "type" : "WebApplicationException",
    "code" : 500
  }
}%

You're right, looks like a mistake on our end here. It should definitely be source and destination go ahead and add it to the same PR! I should look into trying to create an integration test framework so that we can ensure that the methods are correct

This URI seems wrong? http://localhost:19120/iceberg/v1/main%7Cs3%3A%2F%2Fwarehouse/namespaces/my_namespace_1/tables

Looking at Nessie's API description, i'm not seeing /v1/main anywhere in the REST api, it also seems wrong that it would include s3://warehouse/..... in the URI. Is this using the existing REST catalog in iceberg-go or your custom catalog for nessie interactions?

This is the issue that many people are running into. Is that endpoint wrong? This is what I get when I try to load_catalog, and I get a 404. Others apparently do not.

This is the issue that many people are running into. Is that endpoint wrong? This is what I get when I try to load_catalog, and I get a 404. Others apparently do not.

In my case there was issue with my local nessie config, that weired thing main%7Cs3%3A%2F%2Fwarehouse
is prefix which was being added to the uri because i was sending warehouse path and all which is not required

below function implementation works for me


// config.URI := "http://localhost:19120"
// workingBranch := "main"

func LoadCatalog(ctx context.Context, workingBranch string, catalogName string, config types.CatalogConfig) (*rest.Catalog, error) {

	uri, _ := url.JoinPath(config.URI, "iceberg", workingBranch)

	iceCatalog, err := catalog.Load(ctx, catalogName, iceberg.Properties{
		"uri":                  uri,
		"s3.region":            "us-east-1",
	})
	if err != nil {
		return nil, fmt.Errorf("failed to load catalog: %w", err)
	}

	restCatalog, ok := iceCatalog.(*rest.Catalog)
	if !ok {
		return nil, fmt.Errorf("expected *rest.Catalog, got %T", iceCatalog)
	}

	return restCatalog, nil
}

You're right, looks like a mistake on our end here. It should definitely be source and destination go ahead and add it to the same PR! I should look into trying to create an integration test framework so that we can ensure that the methods are correct

Done

ci(catalog/rest): initial framework for rest catalog integration tests (

Navigation Menu

Repository navigation

Enhance iceberg-go to Support Nessie API for All Catalog Operations #291

Feature Request / Improvement

Description

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Enhance iceberg-go to Support Nessie API for All Catalog Operations #291

Description

Feature Request / Improvement

Description

Activity

shubham-tomar commented on Feb 9, 2025

zeroshade commented on Feb 10, 2025

kevinjqliu commented on Feb 10, 2025

zeroshade commented on Feb 10, 2025

kevinjqliu commented on Feb 10, 2025

zeroshade commented on Feb 10, 2025

shubham-tomar commented on Feb 12, 2025

kevinjqliu commented on Feb 12, 2025

zeroshade commented on Feb 12, 2025

shubham-tomar commented on Feb 12, 2025

zeroshade commented on Feb 12, 2025

shubham-tomar commented on Feb 15, 2025

zeroshade commented on Feb 15, 2025

shubham-tomar commented on Feb 16, 2025

shubham-tomar commented on Feb 17, 2025

zeroshade commented on Feb 17, 2025

shubham-tomar commented on Feb 18, 2025

shubham-tomar commented on Feb 19, 2025

shubham-tomar commented on Feb 19, 2025

zeroshade commented on Feb 19, 2025

adamcodes716 commented on Feb 19, 2025

shubham-tomar commented on Feb 20, 2025

shubham-tomar commented on Feb 20, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions