Get Help from lakeFS contributors

Ask technical questions and get help from lakeFS contributors.

Question: Heya! in https://docs.lakefs.io/contributing.htmlcontributing> for lakeFS under the Documentation section, there is a broken link - Just the Docs customization guide - https://pmarsceill.github.io/just-the-docs/docs/customization/ , where should I look for guidance on it? if it’s not necessary, I can create a PR to remove it. posted by polakadi8
Originally created at:Slack

Question: Hello, I’ve touched upon this in #lakefs-for-beginners, but has anyone ever implemented lakefs with GBQ?

We’re directly streaming data into GBQ using a Stitch ELT pipeline (see my conversatin with @barak.amar Slack).

I’m new to GBQ and LakeFS, but can’t seem to find where the GBQ data is actually hosted (and therefore where I can add the LakeFS layer). Any suggestions? posted by stephane.burwash
Originally created at:Slack

Question: Hi! I am trying to use the LakeFSFileSystem in Databricks Spark but I get an error when writing to a lakefs:// URI:

Caused by: java.lang.NoSuchMethodException: com.databricks.s3a.S3AFileSystem.getWrappedFs()
The command fails when running in Databricks Runtimes 7.3 LTS and 9.1 LTS. However, the command succeeds in Databricks Runtime 6.4.

Reading a lakefs:// URI works in all three of those Databricks Runtimes.

Is this an issue anyone else has experienced? Does anyone have guidance on how I could get this to work for Databricks Runtime 9.1 LTS (Spark 3.1)? posted by clinton.monk
Originally created at:Slack

Question: Hey,

Taken from https://github.com/treeverse/charts/tree/master/charts/lakefs:
secrets.authEncryptSecretKey: A random (cryptographically safe) generated string that is used for encryption and HMAC signing
Should we reuse the same encryption secret key in all of our deployments? (will we lose access to our data in PostgreSQL / S3 otherwise?) posted by gal.b
Originally created at:Slack

Question: Hi,
I’m I’m planning to setup lakefs with one bucket per repository on s3,

• Is there a way to create s3 bucket though lakefs or the best way forward is to interact with s3 create a bucket and then point lakefs to use as source for a repository?
• Should SSE be enabled on these buckets?
• Should versioning be enabled on these buckets ? posted by mishraprafful
Originally created at:Slack

Question: I’m sure it’s not great practice to do so, but will anything break if we use the backing S3 bucket as a temporary storage location for other applications? Our servers can only write to that bucket, and we to store some artifacts in the cloud. posted by sid.senthilnathan
Originally created at:Slack

Question: Combine ignoring late data with an effective arrival time by layering multiple instances of this strategy.
My question is : What is multiple instances of this strategy?
Set a sequence of deadline intervals. Data goes into the first layer not beyond its deadline, giving a quantized arrival time.
My question is : What is quantized arrival time?What quantized means? posted by r7raul1984
Originally created at:Slack

Question: Hi, I am trying to configure Spark to access data from LakeFS following https://docs.lakefs.io/integrations/spark.htmlthis tutorial> using the S3A gateway. I set up my LakeFS credentials as env vars and start a Spark shell with the following command:

            --conf spark.hadoop.fs.s3a.secret.key=${LAKECTL_CREDENTIALS_SECRET_ACCESS_KEY} \
            --conf spark.hadoop.fs.s3a.endpoint=${LAKECTL_SERVER_ENDPOINT_URL} \
            --conf spark.hadoop.fs.s3a.path.style.access=true```
When I try to read a LakeFS file rom the Spark shell I get the following error (replacing actual repo and branch):
```java.nio.file.AccessDeniedException: s3a://<my-repo>/<my-branch>/<path-to-file>: getFileStatus on s3a://<my-repo>/<my-branch>/<path-to-file>: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 4442587FB7D0A2F9; S3 Extended Request ID: null), S3 Extended Request ID: null:403 Forbidden```
Do I need other key/secret in addition to the LakeFS key/secret? _e.g._ AWS

Thank you! posted by cristian.caloian
Originally created at:https://lakefs.slack.com/archives/C016726JLJW/p1647506877449319?thread_ts=1647506877.449319&cid=C016726JLJW

Question: can lakefs expose s3 path, so i can use juicefsor s3fsmount local path to lakefs s3 path?
like this:
s3fs <s3bucket> <localpath> -o url=http://127.0.0.1:8391
juicefs format --storage s3 --bucket http://127.0.0.1:8391/test --access-key admin --secret-key seaweedfs redis://127.0.0.1:6379/1 data posted by antipowborkasde
Originally created at:Slack

Question: Hi Everyone, Glad to be here… I want to deploy LakeFS on digital ocean using Kubernetes or Docker… Please is there any documentation that can guide me on achieving that. I intend to have it point to a storage on Digital Ocean and also have Delta Lake integrated posted by onaneyetemilola
Originally created at:Slack

Question: Hi, we are migrating our Lakefs installation to a new AWS account. We are leaving the existing data in the backing S3 bucket, we are just moving the servers and database (postgres RDS). I was thinking that once we restore the database that I should be able to log in using my old user credentials, but it says that they are invalid. Am I missing something? Is this approach not going to work? posted by sid.senthilnathan
Originally created at:Slack

Question: i use aws s3 --profile lakefs --endpoint-url https://192.168.0.65:8000 ls s3://s3bk/main to see my bucket, but it errors, how can i fix it? follow are error message:
SSL validation failed for https://192.168.0.65:8391/s3bk?list-type=2&prefix=main&delimiter=%2F&encoding-type=url [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:852) posted by antipowborkasde
Originally created at:Slack

Question: I’m having some trouble with the python client, see the thread posted by yusuf.khan
Originally created at:Slack

Question: This is what my repository looks like, there’s a weird extra empty path I think because when I ingested I had a trailing slash. posted by yusuf.khan
Originally created at:Slack

Question: and then here’s what I’m running, I’m able to list the branches so it is reachable: posted by yusuf.khan
Originally created at:Slack

Question: but then when I try to get the objects:
I tried with a few different combinations of adding slashes to the path and ref posted by yusuf.khan
Originally created at:Slack

Question: this is just directly plugging into the documentation example code posted by yusuf.khan
Originally created at:Slack

Question: this is just directly plugging into the documentation example code posted by yusuf.khan
Originally created at:Slack

Question: Hey @yusuf.khan! First of all, the weird slashes in the beginning of the path are indeed frustrating, and this issue is already fixed in the latest lakeFS version (0.62.0, fixed in https://github.com/treeverse/lakeFS/pull/3108this> PR). posted by yoni.augarten
Originally created at:Slack