proposed change to the lakeFS quickstart - comments, opinions, and heckling welcome 🙂 https://github.com/treeverse/lakeFS/pull/5565

GitHub leaked git SSH server private key 😱 https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/ Spoiler: says

We have no reason to believe that the exposed key was abused and took this action out of an abundance of caution.

Also includes instructions how to tell your SSH client to drop the compromised old public key so you can verify and start trusting the new key. AFAIK your SSH client should not normally be using this key, you are most likely using an EC* key instead.

Planning to release a new lakeFS version today, Anything I should wait for?

Devs, I opened https://github.com/treeverse/lakeFS/issues/5611 to modify GC to write expired addresses only once. I am trying to figure out whether there are any reasons other than historical for doing so. I shall be removing the extra writes tomorrow. So if you have any reason to care or insight about where to write expired addresses, please note them on the issue. If I do not hear from you, we will write only a single Parquet-format output to

.../_lakefs/logs/gc/expired_objects

. THANKS!

A wise person once said,

_"With great scale comes big data_"

And we're getting there. I think it's time for pack files like Git uses, so I made up this issue to design how we can do them. Packs serve as snapshots in big data (or cubes if anyone still remembers OLAP): in this case, I would like us to hold many commits together in a single "file". The issue has a sketch, and I think we need to think about this direction. I believe it can help us with: • GC on large repositories with many commits • lakectl blame and other log-by-path operations • Even FindMergeBase when the base is distant. It won't be a short sprint, but I'd like us to think about this as a possible next step for lakeFS scalability.

hi devs! Does anyone know what

<https://github.com/treeverse/lakeFS/blob/master/docker-compose.yaml>

is for these days? It looks like it was added a while ago. I'd like to [re]move it so that new users aren't confused between it and the quickstart Docker Compose, but wanted to check first before doing so.

Hi, I am going to release a new lakeFS version later on today, if anyone wants me to wait for something please let me know

Hey, trying to push a docs related PR, failing on Esti tests that seem unrelated.. https://github.com/treeverse/lakeFS/pull/5638 Anyone familiar with these failures?

Hi all, I will release a major lakeFS version in an hour, if anyone wants me to wait for any reason please let me know

Realeasing

Testing changes in goreleaser - I'm using release v0.99.1 for testing.

At the moment we use

AKIAIOSFODNN7EXAMPLE

and

wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

as example access&secret keys respectively, throughout the documentation, Everything Bagel, etc. If we replaced these with

AKIA-EXAMPLE-KEY

and

EXAMPLE-SECRET

it would make documentation more accessible by being more readable (the brain doesn't have to grok a bunch of random chars to realise it's just a dummy placeholder). More accessible == easier for users to get to grips with lakeFS and the samples and documentation. Before I do a PR with a global search & replace for formal review, I wanted to check if there were any immediate objections or gotchas to consider ? thanks 🙂

@Barak Amar suggested and I agree, to check-in our generated Go code. Please feel free to comment on the issue. https://github.com/treeverse/lakeFS/issues/5758

Hi people publishing lakeFS API clients, There may have been some long delays on Sonatype in publishing lakeFS Java artefacts. If you tried to publish the Java API client for version 0.99.0 on 2023-04-19, please be advised that there is a closed release on staging. That is, it was never published. Why? • Perhaps a separate operation did publish it; • Perhaps it was never published because of the delay; • Perhaps you decided not to publish it. If you intended to publish an API client for version 0.99.0, please check whether it was published appropriately. And let me know if I can help.

Planning to publish a new version of airflow-provider-lakefs tomorrow. Any objections / requests to delay it?

Not sure if this has any benefit (or implications) for the lakeFS metadata client, but as of Spark 3.4 which was recently released, there's an official Protobuf Data Source

Any plans to support prefect integration?

Hi, I want to release lakeFS tomorrow morning (UTC+3, Asia/Jerusalem). Does anyone want me to delay?

Hi again, I also want to release a new version of the Airflow provider. Does anyone want me to delay that?

I'm looking at the

Dockerfile

and not being totally familiar with them beyond the basics wondered if the duplication of some of these lines is deliberate and if so what purpose it serves? Thanks.

```EXPOSE 8000/tcp

# Setup user

RUN addgroup -S lakefs && adduser -S lakefs -G lakefs

USER lakefs

WORKDIR /home/lakefs

ENTRYPOINT ["/app/lakefs"]

CMD ["run"]```

I'm planning a PR on the file so will submit a second one to tidy things up, if necessary

Hi @Brad, Can you review this PR and suggest changes? https://github.com/limx0/prefect-lakefs/pull/1 I've added prefect tasks for interacting with branches api and modified the credentials block. More coming soon.

While trying to hit the API I am getting auth error. Can someone help?

Hi folks, Can you provide me any logo representing lakeFS? Need one for the prefect-integration. Found this in the repo but a smaller one like just the gopher kinda creature without text will look great. Thanks.

100

Going to release v0.100.0.

Plan a new lakeFS release - let me know if I should wait for something.

I am working on some issue around this client. So I have put some debugger in this code. I used MVN package then and created my jar under target path. Now following this link I am creating a scala session to read file form lakefs under this newly built jar created above. Spark-shell

spark-shell --conf spark.hadoop.fs.s3a.access.key=minioadmin\ 
--conf spark.hadoop.fs.s3a.secret.key=minioadmin\ 
--conf spark.hadoop.fs.s3a.endpoint=<http://127.0.0.1:9090>\
 --conf spark.hadoop.fs.lakefs.impl=io.lakefs.LakeFSFileSystem\
 --conf spark.hadoop.fs.lakefs.access.key=AKIAIOSFODNN7EXAMPLE\
  --conf spark.hadoop.fs.lakefs.secret.key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'\
 --conf spark.hadoop.fs.lakefs.endpoint=<http://localhost:8000/api/v1>\
 —jars /Users/simar/lakeFS/clients/hadoopfs/target/hadoop-lakefs-0.1.0.jar\
 io.lakefs.LakeFSFileSystem

While reading it I am getting the below error

scala> val df = spark.read.parquet("<lakefs://example/main/sample1.json>")

23/05/20 00:24:42 WARN FileStreamSink: Assume no metadata directory. Error while looking for metadata directory in the path: <lakefs://example/main/sample1.json>.
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class io.lakefs.LakeFSFileSystem not found

Does anyone know what could be causing this?

This issue is worth tracking. It might offer a way to add Delta concurrent writer support on lakeFS with no external synchronization.

is the release process documented somewhere? we need to update it to include the new build & publish of the

-duckdb

Docker image and reference in docs quickstart each time. I'd like to help towards this where possible, but not sure where to start 🙂 (at the moment there is no

treeverse/lakefs:0.101.0-duckdb

, only for 0.100)

Planning to release version 101.1 soon. Anything I need to wait for?