You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The export option in Embedding Atlas is an absolute killer feature.
I assume, you want to enable people to deploy their apps as easily as somehow possible. For me that usually means GitHub pages.
However, GitHub pages has a hard limit of 100Mb for single file uploads. You can use git lfs but then it becomes quite complicated. I think a better option is to chunk the resulting parquet files by default.
I implemented this feature here and it's working great, see this example app I exported. Would you be up for a PR?
Or something like #163 to deploy somewhere else. But that's not as easy as just deploying on GitHub of course. Arguable, git isn't made for large files (unless lfs which as you said is a bit hard to use).
Do you think it addresses the issue here (easy deployment of apps with datasets larger than 100mb)? Maybe nudging people towards putting their data outside git isn't a bad direction.
Yes it does. Generally, neither GitHub without LFS, nor GitHub Pages is really designed to host really large files anyway. Chunking a large file in small pieces is an intermediary solution but not really sustainable if we'd like to go towards the GB range. At some point the hard limits of around 5Gb (if I remember correctly) will become the problem anyway.
Usually, I'd choose places like Huggingface or source cooperative for such large file hosting.
So in a nutshell, going ahead with this PR would be great!
Activity
domoritz commentedon Mar 2, 2026
Or something like #163 to deploy somewhere else. But that's not as easy as just deploying on GitHub of course. Arguable, git isn't made for large files (unless lfs which as you said is a bit hard to use).
do-me commentedon Mar 3, 2026
Yes that's a great proposal!
domoritz commentedon Mar 4, 2026
Do you think it addresses the issue here (easy deployment of apps with datasets larger than 100mb)? Maybe nudging people towards putting their data outside git isn't a bad direction.
do-me commentedon Mar 4, 2026
Yes it does. Generally, neither GitHub without LFS, nor GitHub Pages is really designed to host really large files anyway. Chunking a large file in small pieces is an intermediary solution but not really sustainable if we'd like to go towards the GB range. At some point the hard limits of around 5Gb (if I remember correctly) will become the problem anyway.
Usually, I'd choose places like Huggingface or source cooperative for such large file hosting.
So in a nutshell, going ahead with this PR would be great!
domoritz commentedon Mar 4, 2026
Great. We are reviewing it. I'll close this issue for now then.