Hi, We are evaluating lakefs for an internal project in our company. We have a remote linux server that serves as a common client to host our data. For now we are avoiding using cloud services (but it will most likely be Azure). We are evaluating lakefs as a potential solution before proceeding with a full solution in our project. My question is, is it possible to use lakefs without reference to any large cloud solution providers? Many of the examples use S3, MinIO or Azure referenced. We simply want to test lakefs locally on a linux server with various types of data, (Postgres, XML, json, text files etc). Apologies if this is a basic question but all the Quickstart implicitly assume docker or a cloud solution is used.

Hi @Patrick Glynn! lakeFS also provides a local storage adapter that can be used on your local filesystem You can read more about it here

What @Niro said, of course. But I think also

lakefs --quickstart

will work... except that you'll need to clean up various directories. The idea behind using docker is to avoid forcing you to clean up. Other alternatives: Run the binary with Postgres if you have it on your machine, or even do that but configure database.type to be "local", and database.local.path to be somewhere to write a KV.

You can definitely use

quickstart

for a quick experimentation with lakeFS, but it should not be used for production environments. When you decide to deploy lakeFS with local blockstore for production it is advised to configure lakeFS on your own. Also note that

quickstart

uses the KV local driver for lakeFS's metadata which does not have production guarantees

Hi I tried the local block store solution as outlined https://docs.lakefs.io/howto/deploy/onprem.html https://docs.lakefs.io/howto/deploy/onprem.html#local-blockstore https://docs.lakefs.io/v0.52/quickstart/more_quickstart_options.html#using-the-binary I run into issues where the connection string for postgres is invalid: FATA[0000]/home/runner/work/lakeFS/lakeFS/cmd/lakefs/cmd/root.go:149 github.com/treeverse/lakefs/cmd/lakefs/cmd.initConfig() Load config error="1 error(s) decoding:\n\n* 'Database' has invalid keys: connection_string" fields.file=config.yaml file=config.yaml phase=startup However i checked and I used the original postgres connection string as specific in the docs. I tried with other types of connection strings based on the postgres documentation but nothing works? https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING

@Patrick Glynn you are reading documentation of a very old lakeFS version. Which version are you using?

I am using lakefs version 1.17.0

You should probably look at the latest docs

there is a no real difference, the same connection string is used between the two as far as I can see

Hi @Patrick Glynn, I think I figured this out. This message:

FATA[0000]/home/runner/work/lakeFS/lakeFS/cmd/lakefs/cmd/root.go:149 <http://github.com/treeverse/lakefs/cmd/lakefs/cmd.initConfig()|github.com/treeverse/lakefs/cmd/lakefs/cmd.initConfig()> Load config                                   error="1 error(s) decoding:\n\n* 'Database' has invalid keys: connection_string" fields.file=config.yaml file=config.yaml phase=startup

is not from a Postgres driver, it's from our config parser. can you please check? I think you may be setting

database.connection_string

, but should be setting

database.postgres.connection_string

.

should I modify the config YAML file to match the setting database.postgres.connection_string? This is what I currently have: database: type: "postgres" postgres: connection_string: "postgres://localhost:5432/postgres?sslmode=disable" Or do I need check elsewhere?

I think you may be missing some indentation; I cannot be sure because Slack likes to destroy unquoted text. Could you paste configs inside triple-back-quotes? Anyway, it should look like this:

database:
  type: "postgres"
  postgres:
    connection_string: "<postgres://localhost:5432/postgres?sslmode=disable>"

with

connection_string

indented inside

postgres

.

ahhhh I completely forgot YAML hates identation 😑

Glad we could help.

😁 have a great day, i will have fun exploring lakefs!

Welcome to the lake! Also https://imgflip.com/i/8mdwb1