Thank you very much for all these very clear answe...
# helpp
Paul
04/12/2023, 3:15 PMThank you very much for all these very clear answers.
I understand that it is not possible to connect two lakeFS repositories to the same s3 bucket (even after deleting one of them) but is it possible to connect two s3 buckets to the same lakeFS repo ?
a
Ariel Shaqed (Scolnicov)
04/12/2023, 3:26 PMNot for regular storage. I have some ideas of things you might do -- can you explain the goal of doing this? Maybe I can suggest another way of getting there.
i
Iddo Avneri
04/12/2023, 3:50 PMMaybe worth specifically calling out (following your previous comment), Your code can work against several repositories, similarly to the way that it would work against several buckets. Similar to Ariel, would love to understand what you have in mind to suggest an approach.
p
Paul
04/13/2023, 11:51 AMOk I will try to explain. Let's take the example that my buckets are on minIO, I can import my data from differents minIO buckets as long as these buckets have not already been linked to a lakeFS repository. Is that right ?
Now I want to use other data that is in another AWS bucket. Can I do thos or is it better to transfer my data on my original minIO bucket ?
a
Ariel Shaqed (Scolnicov)
04/13/2023, 11:58 AMThanks for explaining! I was completely off base before, so please allow me to explain what I think is the request. And please complain if I got it wrong again 😕
To my understanding you would like to use the zero-copy import to import data that lives on several different S3 implementations: a few on your proprietary MinIO, and another other on AWS S3. If so, this is not currently supported. You would need to copy data from AWS S3 to MinIO -- although not necessarily to the same bucket.
👍 1
Ariel Shaqed (Scolnicov)
04/13/2023, 12:02 PMNow I want to play🧸 what-if.
You might open an issue, to support multiple blockstores in lakeFS. If you figure them out, I would probably ask you to add some numbers for the desired non-functional characteristics and scale: how many objects, their sizes, how many imports, how often you would re-import, etc.
It may be possible, although even configuration would be very confusing.
Also I believe that -- if implemented -- it could have very strange operational characteristics. For instance, backing storage failures will not only become more common but also open a new family of partial failure modes.
THANKS!
👍 1
2 Views