Repository navigation
[Nessie REST Catalog] Iceberg table not listing all snapshots #1105
Description
Question
Hello, I have a question regarding iceberg table snapshots. I used pyiceberg to create a namespace and a table, then insert a dataframe like so:
from pyiceberg.catalog import load_rest
from pyiceberg.schema import Schema
from pyiceberg.types import StringType, NestedField, DoubleType
from pyiceberg.partitioning import PartitionSpec, PartitionField
import numpy as np
import pandas as pd
rows = 10**1
ncols = 10
countries = ["US", "CA", "UK"]
attr2 = ["OPEN", "CLOSE", "LOW", "HIGH"]
dates = pd.date_range("2020-01-01", "2020-12-31")
data_orig = pd.DataFrame(
[
{
"countries": countries[i % len(countries)],
"status": attr2[i % len(attr2)],
"return_index": np.random.rand(),
}
for i in range(rows)
]
)
schema = Schema(
NestedField(field_id=1, name="countries", field_type=StringType(), required=False),
NestedField(field_id=2, name="status", field_type=StringType(), required=False),
NestedField(
field_id=3, name="return_index", field_type=DoubleType(), required=False
),
)
partition_spec = PartitionSpec(
fields=[
PartitionField(
source_id=1, field_id=1000, name="countries", transform="identity"
),
PartitionField(source_id=2, field_id=1001, name="status", transform="identity"),
]
)
catalog = load_rest(
"rest",
conf={
"uri": "http://localhost:19120/iceberg",
},
)
catalog.create_namespace_if_not_exists("rpmd")
tables = catalog.list_tables(namespace="rpmd")
table = catalog.create_table_if_not_exists(
identifier="rpmd.performance", schema=schema, partition_spec=partition_spec
)
import pyarrow as pa
import pyarrow.parquet as pq
tbl = pa.Table.from_pandas(data_orig)
table.append(tbl)
When i run this script a second time, it is my understanding that the append method would create a new commit, and in turn, a new snapshot of the rpmd.performance iceberg table, and i should be able to see a list of snapshots in the latest metadata.json file. However, i only see the latest snapshot in the array, and running table.inspect.snapshots() only shows 1 snapshot_id, even though i see all the relevant .avro files.
May i know if there is some configuration i need to do to ensure that i can see all snapshots?
Activity
[-]Iceberg table not keeping track of snapshots[/-][+]Iceberg table not listing all snapshots[/+]PetrasTYR commentedon Aug 28, 2024
For some additional background information:
i am using Nessie's REST APIs for iceberg tables.
package versions:
in the latest
metadata.jsonfile, this is how the snapshots section look like:the sequence number is being tracked correctly but only the latest snapshot is being saved. my understanding is that all the past snapshots should be saved.
sungwy commentedon Aug 28, 2024
Hi @PetrasTYR , thank you for reporting this issue, and continuing the investigation on your side. The follow-up comment you posted is indeed very helpful. It looks like the latest metadata file only has a single snapshot, which makes me wonder if this is an issue with the specific implementation of the REST Catalog.
What you expect is correct. We should expect to see two snapshots, each snapshot with different snapshot-id but with the same snapshot summary since you are repeating the same operation twice.
PetrasTYR commentedon Aug 28, 2024
Hi @sungwy , were you able to replicate the issue, by any chance?
PetrasTYR commentedon Aug 28, 2024
Hello, I was asking some folks at project nessie's zulip chat and it seems like this is expected. Nessie provides catalog level versioning providing a consistent and reproducible state across the whole catalog; and to retain Nessie’s consistency and cross-branch/tag isolation guarantees, they deliberately choose to only return the state of a table or view as a single snapshot in Iceberg i.e. Nessie intentionally only returns the latest snapshot. Anything related to time-travel queries are to be done via Nessie instead of directly through Iceberg!
sungwy commentedon Aug 28, 2024
That's good to know! I didn't know much about Nessie, but I think this is helpful information for anyone else looking to debug this issue in the future. Thank you again for raising this issue and following up with the findings @PetrasTYR !
PetrasTYR commentedon Aug 28, 2024
No worries! Closing this issue for now~
[-]Iceberg table not listing all snapshots[/-][+][Nessie REST Catalog] Iceberg table not listing all snapshots[/+]kevinjqliu commentedon Aug 30, 2024
Updated title for future reference
kevinjqliu commentedon Jan 13, 2025
https://projectnessie.org/guides/iceberg-rest/#time-travel-with-iceberg-rest