Viewing a single comment thread. View all comments

rajatarya OP t1_j0gyotd wrote

Reply to comment by Keepclamand- in [P] XetHub: We scaled Git to support 1 TB repos by rajatarya

Great questions. Definitely check us out - within 15m of getting started you'll experience answers to your questions :)

  1. Do you support all data types?
    Yes, all file types are supported. The level of deduplication we can achieve varies by file type (some file types are already compressed) but all file types are supported. We have some great example repos with images, text, and other data types.

  2. Can you track versioning of data?
    Yes, since you are just using Git - each commit captures the version of the data (since the data is just files in the repo). This way you have full collaboration features of Git while having full reproducibility. With the added benefit of having confidence that the code will work with the data at each commit.

  3. Do you have APIs?
    Not today. Can you tell me what sort of APIs would be interesting to you? We built Xet Mount specifically for use cases when you don't want to download the entire repo - instead you mount it and get a filesystem view over the repo and stream in the files you want to explore/examine/analyze.

Do check out XetHub - I would love to hear your feedback!

7

rajatarya OP t1_j0gzh2r wrote

Oh I forgot to mention - yes! mapping model to training data is a key part of reproducibility. 100% agree!

Using XetHub you can _finally_ commit the data, features, models, and metadata all in one place (along with the code). Have full confidence everything is aligned & working.

4

Liorithiel t1_j0h19at wrote

> finally

I was doing so with git annex for a long time, so this is a bit of a stretch that it wasn't possible in the past. Kind of a Schmidhuber moment…

Still, nice work with the merkle tree!

2

rajatarya OP t1_j0h7npz wrote

True :) I haven't used `git annex` myself so for me it felt like _finally_ when I could put all parts of the project in one place with XetHub.

How do you like using git annex? Are you working with others on your projects - does git annex help support team collaboration?

Again, appreciate the comment!

3

Liorithiel t1_j0hehga wrote

> How do you like using git annex? Are you working with others on your projects - does git annex help support team collaboration?

Right now I've got one large 5 TB repository with general media and archives, and some smaller project-specific repos. Slow with many small files (like, over 1 million), but very easy to set up. Haven't tried collaboration, I've mostly worked with projects where my collaborators were rather less technical. My main use case was working with the same dataset on different computers, and for that it was more than enough.

2