Submitted by rajatarya t3_znfgap in MachineLearning
rajatarya OP t1_j0iemk4 wrote
Reply to comment by BossOfTheGame in [P] XetHub: We scaled Git to support 1 TB repos by rajatarya
There isn’t a hard limit at 1TB currently. The main thing is the experience / performance may degrade. The size of the merkle tree is roughly 1% of total repo size so at 1TB even downloading that can take some time. You can definitely use XetHub past 1TB repo today - but your mileage may vary (in terms of perf/experience).
To avoid downloading the entire repo you can use Xet Mount today to get a file system readonly view of the repo. Or use the —no-smudge flag on clone to simply get pointer files. Then call git xet checkout for the files you want to hydrate.
I would love to talk more about the 2TB DVC repos you are using today - and believe they would be well served by XetHub. Something I would be eager to explore. DM me your email if interested and I will follow up.
Thanks for the question!
Viewing a single comment thread. View all comments