Anyone have an idea why automated tests are not running on https://github.com/treeverse/lakeFS/pull/3779?

Hi, I would like to release a new minor version of lakeFS Any objections?

Do we need/want dependabot running on docs-lakefs? Every time it gets busy, it issues PRs for every version of docs. this number alone grows linearly with releases.

I'm having some difficulty bringing up DBT dbt on the lakeFS lakefs Everything Bagel 🥯 . I've updated some things to use up-to-date DBT, avoid docker-compose-prefixes, and bump

ulimit -n

. Now my hive-metastore container fails to start; AFAICT the issue is

hive                      | MetaException(message:Required table missing : "`DBS`" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCreateTables")

(full logs are longer). It looks like Hive needs some startup script (or to set that value). Has anyone run into this?

Hi Everyone, I have a question about the database storage sizing guide in the lakeFS documentation According to the documentation, the storage requirements are about 150MiB per every 100,000 uncommitted writes - which is roughly around 1500 bytes per write. Looking at the code - I see that the lakeFS is writing the following

Entry

struct per write:

ent := &Entry{
		Address:      entry.PhysicalAddress,
		AddressType:  addressTypeToProto(entry.AddressType),
		Metadata:     entry.Metadata,
		LastModified: timestamppb.New(entry.CreationDate),
		ETag:         entry.Checksum,
		Size:         entry.Size,
		ContentType:  ContentTypeOrDefault(entry.ContentType),
	}

Creating a gross calculation taking into account field limits: Address - according to AWS guidelines does not exceed -- 1024 bytes AddressType - int32 -- 4 bytes Metadata - according to AWS limited to 2KB user data -- 2048 bytes LastModified - int64 -- 8bytes Etag - AWS limitation -- 1024 bytes Size - int64 -- 8 bytes ContentType - Lets use the worst case scenario -- 1024 bytes Summing this up we get over 5000 bytes which is far from the given estimation, and this is without taking into consideration other data which is saved such as entry key and checksum Am I missing something here?? (Keep in mind that these are general approximation - not trying to do exact math here but rather get a sense of the size)

Calling all Java experts 🙂 I'm working to support lakeFSFS on Hadoop3. This will be a distinct package on Maven central -- it possesses different semantics (hadoop3 FileSystem has a subtly different spec than hadoop2 FileSystem) and different (provided!) dependencies. I'm trying to figure out the Maven Way to name this package. Does anyone have any suggestions, or ideally examples of how to do this? This is particularly important _*if you are a developer who uses LakeFSFileSystem*_: it will directly impact how you build your code (and a chance to get a strange hard-to-debug error at runtime...). The only option that I can see so far is to incorporate a string (`_hadoop2`/`_hadoop3`, or maybe `-hadoop2`/`-hadoop3`) into the package name. This is similar to the way Scala packages incorporate a Scala language minor version into their name -- and for a similar reason: the Scala runtime library is an implicitly provided dependency to all Scala packages, and that changes incompatibly between Scala minor versions.

Planning to release a fix version with :

Fix DB serialization error during multiple writes to the same key (#3862)

Any objections?

Hi lakeFSers, Wanted to have your input on something: We are currently planning the KV migration procedure. As part of the migration, the lakeFS database configuration parameters change. We added a new section per driver type with its own configuration. The current flow requires the users to copy the postgres configuration into a designated 'postgres' section in the database configuration section, perform the migration and afterwards delete the old configuration parameters. A concern was raised that this might cause user friction, so it was suggested that we take the parameters from the old path and issue a deprecation warning, and remove these configuration paths in a future version. Another approach says that we already require the user to update the configuration (Need to specify the database 'type' in the configuration)

TIL about Maven Central Search. We've been searching for a tool to find Java packages that provide a class. This does pretty much that; it could be useful for debugging shading failures. Example:

❯ mcs class-search -f org.apache.hadoop.fs.FileSystem
Searching for artifacts containing org.apache.hadoop.fs.FileSystem...
Found 743 results (showing 20)

  Coordinates                                                                            Last updated
  ===========                                                                            ============
  org.scray:scray-hdfs-writer:1.1.1                                                      13 Sep 2018 at 10:08 (IDT)
  org.scray:scray-hdfs-writer:1.1.0                                                      12 Sep 2018 at 11:45 (IDT)
  org.apache.hadoop:hadoop-common-instrumented:0.22.0                                    10 Dec 2011 at 04:33 (IST)
  org.apache.hadoop:hadoop-common:0.22.0                                                 10 Dec 2011 at 04:33 (IST)
  org.jvnet.hudson.hadoop:hadoop-core:0.19.1-hudson-3                                    03 Sep 2009 at 00:41 (IDT)
  org.apache.mahout.hadoop:hadoop-core:0.19.1                                            03 Apr 2009 at 09:53 (IDT)
  org.jvnet.hudson.hadoop:hadoop-core:0.19.1-hudson-2                                    16 Mar 2009 at 00:54 (IST)
  org.jvnet.hudson.hadoop:hadoop-core:0.19.1-hudson-1                                    15 Mar 2009 at 22:40 (IST)
  org.jvnet.hudson.hadoop:hadoop-core:0.19.1                                             12 Mar 2009 at 07:15 (IST)
  org.jvnet.hudson.hadoop:hadoop-core:0.19.0                                             11 Mar 2009 at 08:02 (IST)
  com.tencent.bk.base.datahub:hadoop-common:2.7.3-bkbase.1                               26 Oct 2021 at 06:51 (IDT)
  org.apache.hadoop:hadoop-common:0.23.11                                                19 Jun 2014 at 17:17 (IDT)
  org.apache.hadoop:hadoop-common:0.23.10                                                03 Dec 2013 at 07:46 (IST)
  org.apache.hadoop:hadoop-common:0.23.9                                                 01 Jul 2013 at 17:45 (IDT)
  org.apache.hadoop:hadoop-common:0.23.8                                                 28 May 2013 at 18:27 (IDT)
  org.apache.hadoop:hadoop-common:0.23.7                                                 11 Apr 2013 at 21:26 (IDT)
  com.google.code.maven-play-plugin.org.apache.hadoop:hadoop-core:0.20.2-with-200-826    11 Mar 2013 at 22:15 (IST)
  org.apache.hadoop:hadoop-common:0.23.6                                                 29 Jan 2013 at 05:53 (IST)
  org.apache.servicemix.bundles:org.apache.servicemix.bundles.hadoop-core:0.20.203.0_3   21 Jan 2013 at 04:37 (IST)
  org.apache.hadoop:hadoop-common:0.23.5                                                 20 Nov 2012 at 20:29 (IST)

This blogpost makes me suspect that the generated Java API client does not reuse HTTP connections: we probably get a new SSL Socket Factory each time we create an API client, which prevents reuse.

This might be useful..? https://acorn.io/

Hi, I would like to release a new version of lakeFS, any objections?

Thought i’d share a few references that might be highly relevant to lakeFS’ scalability: 1. https://devblogs.microsoft.com/devops/supercharging-the-git-commit-graph-iii-generations/#using-generation-number-in-merge-base-questions 2. Git's https://git-scm.com/docs/commit-graph-format 3. some more https://devblogs.microsoft.com/devops/supercharging-the-git-commit-graph-ii-file-format/ 4. logCommits for a path: https://devblogs.microsoft.com/devops/super-charging-the-git-commit-graph-iv-bloom-filters/ Obviously these are for longer term improvements that include additional indexing or data model tweaks, so putting here as reference

Interesting but tough reading: Conflict resolution for structured merge via version space algebra. Adds some algebraic representations of merging. (AFAICT this is not the operational transform approach behind Google Docs, rather something like a description of Git's 3-way merge (which is also our model.)

This message was deleted.

I'm about to release lakeFS with KV version, any objections?

Hi all, A lot of code changes were merged over the weekend towards release of lakeFS with KV. I highly recommend you rebase your work over latest master and apply the changes. Please let us know if you encounter any issues.

I have a small fix https://github.com/treeverse/lakeFS/pull/4097. It is related to https://github.com/treeverse/lakeFS/issues/4094, but it does not fix that issue. Think of it as a partial resolution. How should I mark it to get around the

No linked issues found. Please add the corresponding issues in the pull request description.

message we get on GitHub? Just open another issue? (@Tal Sofer, can you advise?)

hi all, I started working on the failure in loading a user with % in his name issue. first, I thought that we're missing encoding somewhere, but going deeper into the issue, it seems that it's something broken with react's history library and how it works with percent signs in the URL. Encoding this using

encodeURIComponent

or

encodeURI

doesn't help as % is used as part of the encoded value. Another approach is to encode using base64 which is URL-safe. I believe the right approach would be to disallow some characters for user/group/policy names as we use them in the URL and to avoid other potential bugs. Comparing to AWS IAM's restrictions, they allow only specific characters to be used in the (alphanumeric and '+=,.@-_' characters.) I think that restricting to these characters is a the right approach to take. this means it'll be a breaking change in case someone used % in previous versions, but to be honest, the UX was already broken with entries with % in its name. Thoughts? @Elad Lachmi thanks for the help

Does anyone know of an ASCII representation for timestamps, similar to ISO8601, but that sorts in reverse order? Long form: @Yoni Augarten and I are working on improving usability of run_id in GC. Currently in order to discover which run_id to provide to a run, you pretty much need a KV store to keep the last few run_ids by time. That's an ops problem. As a first step we want to change the format of run_ids to be sort -- so instead of

<UUID>

we will use something like

<TS_ISO_8601>-<RANDOM>

. Now the files

.../retention/gc/commits/run_id.../commits.csv

will list in time-sorted order. So no need for KV, we can simply list these objects in S3. Unfortunately, it is still hard to go backwards in such a list. And that's actually the preferred mode: "find the last run_id from at least 2 weeks ago". So... I'd really like to name them

<REVERSE_8601_ISH>-<TS_ISO_8601>-<RANDOM

, where RANDOM_8601_ISH is an ISO 8601-like printable representation of timestamp that sorts in reverse order, and we include TS_ISO_8601 merely for readability. Now we just need an encoding 🙂 Where I've gotten so far: use a 9s-complement ASCII representation. An even less-readable representation would use e.g.

10000000000 - EPOCH

(right now would be

8337539453

(== 10000000000-1662460547), and 5 seconds later it would be

8337539448

. A more-readable representation might use

<10000-YYYY>-<100-mm>-<100-DD>N<100-HH>:<100-MM>:<100-SS>

, and let you decode in your head with a little effort. Is there any more standard way to do this?

https://www.githubstatus.com/ 😞

I'm reading up on the Spark DataSource V2 API. Does anyone have good reading material? I found this series of blogs, but would be happier with some more official documentation. THANKS!

can anyone tell me what is in the _lakefs directory with a repo? I am guessing it's rocksdb related?

Hi All, I'd like to understand the usage of lakeFS StageObject API (*PUT*/repositories/{repository}/branches/{branch}/objects). Specifically if there are any usecases which provide a physical address which is inside the repository storage namespace. We are in the works of adding a garbage collection process for uncommitted data and considering allowing this API only on physical addresses which are outside the repository namespace

TIL a good way to compute a nonserializable value once in an object that needs to be Serializable to use in Spark:

@transient lazy val notSerializable = computeFromSerializables(param1, param2)

🤷🏼

I want to commit some ranges SSTable to the lakeFS repo for a benchmark. Should be around 1 MiB to 20 MiB in size. Does anyone have one that I could use? (A range SSTable contains object metadata (including pathnames), so you should feel comfortable revealing such information in the file. It does NOT contain object data (the actual contents of the objects)!)

Can delete_object Python API be enhanced to recursively delete all objects under the specified path?

lakectl fs rm

command already has this functionality but Python API doesn’t.

Hi Team I am working on this issue https://github.com/treeverse/lakeFS/issues/2344 As far as I know I have to change the the README under the python package https://github.com/treeverse/lakeFS/tree/master/clients/python . Not sure why the changes doesn't reflect in pypy of lakefs already. Or if it is not how it was supposed to be done please guide me where to add the changes?