May I just ask for one confirmation: whenever you ...
# helpa
Alexey Shchedrin
12/05/2023, 2:16 PMMay I just ask for one confirmation: whenever you create a new repository, does it matter how I name my repo and whether that name is in sync with my S3 bucket name?
a
Ariel Shaqed (Scolnicov)
12/05/2023, 2:19 PMNo, it does not. In fact you can have multiple repos in the same S3 bucket, as long as each uses a different prefix on S3.
lakeFS manages metadata for your repository. No matter what you do, the paths that you use on lakeFS will be very different from the paths that lakeFS uses on S3. This is intended.
Ariel Shaqed (Scolnicov)
12/05/2023, 2:24 PMYou can use lakeFS without going any deeper into the internals of why object paths on S3 are so different from object paths on lakeFS. It's an internal implementation details. Just like almost everyone uses minIO without understanding its internal data structures.
But if you're really curious to understand why you might dive deeply into the lakeFS architecture by starting at https://docs.lakefs.io/understand/how/, or with one of our blog posts on "Graveler".
a
Alexey Shchedrin
12/05/2023, 2:25 PMOK, thanks, understood. No, I am not really interested by all these details, I am just trying to investigate my issues 😉 .
Alexey Shchedrin
12/05/2023, 2:27 PMIt is all very strange: the very first test fails and I cannot believe other people did not experience it.
a
Ariel Shaqed (Scolnicov)
12/05/2023, 2:30 PMYeah, probably best for us to understand the issue with DuckDB in the GUI. We suspect a particular browser configuration breaks -- leading to "it works on my machine" syndrome.
If you're ultimately planning also to use Spark, this might be an opportunity to configure a tool outside of the lakeFS GUI to access lakeFS... and read Parquet directly from Spark, maybe!
a
Alexey Shchedrin
12/05/2023, 2:32 PMThanks, I will try that. We are planning to use Spark so yeah, that would be a natural step
a
Ariel Shaqed (Scolnicov)
12/05/2023, 2:33 PMPlease let us know how you get along! Also (obviously) I am sorry you're running into these issues.