Hey all, I made a PR for the OIDC feature we requested. https://github.com/treeverse/lakeFS/pull/4533

Based on

brentahughes/badger-cli

created a fork https://github.com/nopcoder/badger-cli to support the badger file format and couple of more changes for working with our local format k/v metadata.

The next Python and Java API releases will include a license. Does anyone need this immediately? They'll be published anyway as part of the next lakeFS release 8-)

I plan on releasing lakeFS today. Any PR I should wait for?

Just heads up that

blockstore.type

is going to be a required field - the default as local can be applied by using

--local-settings

. Aligned the behaviour with

database.type

. https://github.com/treeverse/lakeFS/pull/4553

Plan to release lakeFS - let me know if there is something I should wait for before going forward.

Heads-up: commit 7fd903f2669af412f130607e2783f86f717af8d3 adds a use of

time.Duration.Abs

, which is Go 1.19. This raises the effective required minimum version of Go to 1.19.

We need to release this week for the fix of 4608 - is there anyone else waiting for a release? Any PR we should be waiting for?

I’m planning to release a new version of lakeFS, is there anything pending I should wait for?

Java experts: What's a modern JVM-ish way to do advice aka AOP (Aspect-Oriented programming)? I'm looking to wrap all API calls performed by our hadoopfs library. It will measure number of calls and duration by context inside the client. So I'd like to wrap e.g. io.lakefs.clients.api.ObjectsApi inside a wrapper object. The wrapper will present the same interface as ObjectsApi (say, inherit from it), and will perform one action before calling the underlying method and another after it. I control the calling code, but I have no control over the API code (it's the autogenerated client). Acceptable compromises: • Change the actual type, say be a Wrapped<ObjectsApi> that is not an ObjectsApi. (I control the caller) • Use reflection. (Every call is an HTTP request+response, so the cost of reflection is acceptable) Thanks!

We have no actions running on the GitHub treeverse/lakeFS repo. This PR should be running some more actions, but nothing's happening, they're waiting. Any clues?

We’re planning on releasing lakeFS server and Spark Metadata Client later today, anything we should be waiting for?

I wonder if anyone wants to share their thoughts in this thread - regarding the feature that you developed that was most significant to you?

Hackaday: Using GitHub Actions To Brew Coffee. https://hackaday.com/2022/12/10/using-github-actions-to-brew-coffee/

Hi! In my company I would like to create/update symlink files directly from Airflow (typically after a Databricks job which updates some data!) So I opened a pull request on airflow provider: https://github.com/treeverse/airflow-provider-lakeFS/pull/41 I am not sure about what to do next? Maybe ask for some review from maintainers?

Google have a special approach to o11y and to RPC, so this might not be directly applicable. OTOH this is an incredible engineering feat, even ignoring the scale at which they present. https://hachyderm.io/@ruidlopes/109587653665864909 Speed is a critical feature we invest a lot of effort in Google Search. A few friends in Search Infra just published an in-depth article in the Communications of the ACM journal on how we do #latency #profiling of our serving stack titled "Distributed Latency Profiling through Critical Path Tracing". Enjoy the reading at https://m-cacm.acm.org/magazines/2023/1/267978-distributed-latency-profiling-through-critical-path-tracing/fulltext https://hachyderm.io/@ruidlopes/109587653665864909

Brilliant explanation why pass by value is faster than pass by reference for small-sized values. Uses cats (and also x86_64) as an ABI. This is also true for Go, except that the fact that references and pointers are mixed together forces a less pure decision. https://mastodon.social/@TartanLlama/109636266260266025

tl;dr: Where is a good place to upload JARs for use in testing? They do not belong to our namespace, so I cannot use Maven Central. Longer explanation: • I want to upload hadoop-aws-2.7.4.jar to somewhere so that I can use it to test Spark 3 with ancient Hadoops. ◦ It is not possible to get this version of the JAR from Maven Central because it's been dead for so long. • I can get a copy of this jar: ◦ Unfortunately it lives inside the full Hadoop 2.7.4 download and that's not what I want to test. ◦ So I extracted this jar from that .tar.gz • It's part of org.apache.hadoop, so Maven Central won't allow me to create a profile for that. • I do want to put it somewhere useful... but where?

Hi Devs! I've just merged https://github.com/treeverse/lakeFS/pull/4933. It tests GC with Hadoop 2 on the Spark host. previously we were testing only Hadoop 2 assemblies in a modern Spark build, which is not the same. This test currently fails - TDD FTW! So it is marked so as not to fail the build. You should see it red but will still be allowed to merge without force. Thanks, and sorry for any confusion.

Planning to release lakeFSFS (hadoopfs) later today. Any objections / stays?

Planning on releasing lakeFS today, anything worth waiting for?

Adding a bunch of config properties to lakeFS. What's the current rule: add accessors, or access the config struct directly? (@Barak Amar was last to touch this IIRC)

This looks weirdly broken right now, questions about "gorm"(?) are showing up on the #stackoverflow channel that I see.

Planning on releasing lakeFS today, anything worth waiting for?

I’ll release lakeFS HadoopFS client & Spark metadata client, let me know if you have something worth waiting for.

Hey, I'm starting lakeFS using docker-compose, with postgres as the backing database. I'm using

replicas: 3

to scale up - and on the first run I'm getting the following error on one of the lakeFS containers, and it fails to start:

setup failed: ERROR: relation \"kv_49\" already exists (SQLSTATE 42P07)

Seems like a race condition. Any recommendations on how to deal with it?

Hi Team, I am working on an issue. In the github Provider are we not creating Airflow for testing the action ? In the below code where will this DAG run? I can’t see airflow env creation in the above provider.yaml?

- name: Run lakeFS DAG

run: |

cd astro

astro dev run connections add conn_lakefs --conn-type=HTTP --conn-host=<http://172.17.0.1:8000> --conn-extra='{"access_key_id":"${{ env.KEY }}","secret_access_key":"${{ env.SECRET }}"}'

astro dev run dags unpause lakeFS_workflow

astro dev run dags trigger lakeFS_workflow

sleep 30

We plan on releasing lakeFS tomorrow to get import for local block adapter out, anything worth waiting for?

Hi Team I am planning to work on this issue Please tag me to this. Github vaibhavk1992 And want to understand more around the issue. Please help me with more details.