[QUESTION] Register multiple S3 parquet files as one table?

[danthegoodman1]

Let's say I have a known list of files in s3 like s3://bucket/file1.parquet, s3://bucket/file2.parquet, etc.

How can I register them all under the same table so I could do something like select count(*) from parquet_files?

[mesejo]

Hey! If all the files are under the same bucket, you could do the following:

import os
import datafusion
from datafusion.object_store import AmazonS3

region = "us-east-1"
bucket_name = "yellow-trips"

s3 = AmazonS3(
    bucket_name=bucket_name,
    region=region,
    access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
    secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
)

ctx = datafusion.SessionContext()
path = f"s3://{bucket_name}/"
ctx.register_object_store(path, s3)

ctx.register_parquet("trips", path)

df = ctx.sql("select count(passenger_count) from trips")
df.show()

DataFusion

>>> datafusion.__version__
'28.0.0'

[danthegoodman1]

@mesejo I guess I can do that with the S3 proxy I've made to trick clients, thanks