Skip to content

Webapp export: chunk dataset.parquet file #164

Closed as not planned
Closed as not planned

Description

@do-me

The export option in Embedding Atlas is an absolute killer feature.
I assume, you want to enable people to deploy their apps as easily as somehow possible. For me that usually means GitHub pages.

However, GitHub pages has a hard limit of 100Mb for single file uploads. You can use git lfs but then it becomes quite complicated. I think a better option is to chunk the resulting parquet files by default.

I implemented this feature here and it's working great, see this example app I exported. Would you be up for a PR?

Activity

  1. domoritz commented on Mar 2, 2026

    @domoritz
    Member

    Or something like #163 to deploy somewhere else. But that's not as easy as just deploying on GitHub of course. Arguable, git isn't made for large files (unless lfs which as you said is a bit hard to use).

  2. do-me commented on Mar 3, 2026

    @do-me
    Author

    Yes that's a great proposal!

  3. domoritz commented on Mar 4, 2026

    @domoritz
    Member

    Do you think it addresses the issue here (easy deployment of apps with datasets larger than 100mb)? Maybe nudging people towards putting their data outside git isn't a bad direction.

  4. do-me commented on Mar 4, 2026

    @do-me
    Author

    Yes it does. Generally, neither GitHub without LFS, nor GitHub Pages is really designed to host really large files anyway. Chunking a large file in small pieces is an intermediary solution but not really sustainable if we'd like to go towards the GB range. At some point the hard limits of around 5Gb (if I remember correctly) will become the problem anyway.

    Usually, I'd choose places like Huggingface or source cooperative for such large file hosting.

    So in a nutshell, going ahead with this PR would be great!

  5. domoritz commented on Mar 4, 2026

    @domoritz
    Member

    Great. We are reviewing it. I'll close this issue for now then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions