Please let me know how to change database connecti...
# help
r
Please let me know how to change database connection string from postgres to CockroachDB
Copy code
database:
  connection_string: "<postgres://user:pass@lakefs.rds.amazonaws.com:5432/postgres>"
a
Hi Ravinder! Thanks for the question! Currently lakeFS requires PostgreSQL on the backend. This is used for mutable parts of the repository (only), including branch heads and uncommitted/staging metadata, as well as auth. Could you expand on the use-case for replacing PostgreSQL with CockroachDB? We do have proposals to remove the requirement to use PostgreSQL in favour of a generic key-value store, this is one of them. Specific requirements may inform our choice of proposals and their timing :-)
r
Thank you for the information
Hello Ariel, Thanks for the update. Please let me know if this works databaseConnectionString: "postgres://root@roach.test:26257/lakefs?sslmode=disable"
a
That indeed is a Postgres connection string. So if roach.test on that port is, in fact, PostgreSQL, with an appropriate then that should work. However I think (given the name and the history of this conversation) the name may hint at this being a CockroachDB instance with an SQL compatibility layer? That would help probably not work,; it is certainly not a supported configuration. If you are looking for a quick start, you might try one of the configurations recommended along the https://docs.lakefs.io/quickstart/try.html trail.
Might I ask what you are trying to do? It would allow some quick answers backed by better recommendations.
r
secrets: # replace DATABASE_CONNECTION_STRING with the connection string of the database you created in a previous step. # e.g. postgres://postgres:myPassword@my-lakefs-db.rds.amazonaws.com:5432/lakefs databaseConnectionString: [DATABASE_CONNECTION_STRING] # replace this with a randomly-generated string authEncryptSecretKey: [ENCRYPTION_SECRET_KEY] lakefsConfig: | blockstore: type: s3 s3: region: us-east-1 # optional, fallback in case discover from bucket is not supported
I am trying to replace my-lakefs-db.rds.amazonaws.com with CockroachDB as a data store
a
I understand. Please note that this is not a supported configuration, and I would be surprised if it worked. AFAIK we have never tested it, I have certainly never done so. And I lack the expertise to help you set something like this up. I cannot in good conscience recommend that you start with such a configuration. Database consistency and atomicity semantics are subtle, and we would need to research how that affects lakeFS. If I may: I would instead suggest... 1. Set up a temporary installation using one of the "quick start" options on our website (demo.lakefs.io is probably easiest) and see whether lakeFS is useful to you. 2. If this configuration is important to you, please open an issue for this requirement. Ideally include your use-cases so that I will be able to move it up the backlog queue during triage. On Sunday when everyone is back from the weekend I shall ask around; there is a slight chance we have someone with CockroachDB experience who would be able to say that this should work. Have a great weekend!
A definitive answer from our @Barak Amar: nope, lakeFS is not going to work on top of CockroachDB. We use advisory locks as part of branch locking (e.g. during commit operations), and CockroachDB does not support these. Sorry. You might add a comment on the currently-open issue to generalize KV store usage, or open a new issue. I am sure that supporting general KV stores WILL happen. But there is currently no set date for doing this. The more you (and others) can explain your use-case, the easier it will be for me to get support for doing this.