https://lakefs.io/ logo
#dev
Title
# dev
д

Денис Егоров

12/09/2023, 5:52 PM
Hello everyone ) Are there any plans to add MySQL support?
e

einat.orr

12/09/2023, 5:54 PM
Not in the near future. For versioned MySQL you can check out Dolt or Dolthub for a managed service. Alternatively, you can move your data to an object storage and use lakeFS :-).
д

Денис Егоров

12/09/2023, 5:58 PM
Yes... I can, but I want to use lakeFS c TiDB .
I don't want to start a zoo. ))
And if I make a pull request, can it get into the main branch?
e

einat.orr

12/09/2023, 6:02 PM
I didn't know TiDB. Let me check it out and see what the options may be. I'll probably get to it tomorrow.
Sure! You are welcome to contribute!
д

Денис Егоров

12/09/2023, 6:03 PM
Super. ! ) Thank you.
a

Ariel Shaqed (Scolnicov)

12/09/2023, 6:18 PM
Hi Denis, IIUC you would like to add a MySQL kv implementation? We love PRs! This should be straightforward based on the existing Postgres implementation. But, because this is a core component, I worry about correctness and about ensuring that it continues to work. We will need a full understanding of things like isolation levels used, and also comprehensive testing. Most of the work will have to happen there.
д

Денис Егоров

12/09/2023, 6:26 PM
Hi Ariel, Yes, I understand... But... In this case, the bottleneck is Postgres. I want to remove it. Postgres is very difficult and always has downtime and problems when the master crashes. There is no master-master replication or for a lot of money. Of course, I would like to add and conduct any necessary tests. If someone can coordinate me on this, then I would like to provide normal support for TiDB.
jumping lakefs 1
Adding MySQL, it is possible to use https://galeracluster.com/. I have no negative experiences with this solution, only positive ones.
a

Ariel Shaqed (Scolnicov)

12/09/2023, 6:54 PM
Just to be clear: I want to be that person, and there's a fair chance that I will be. I'm worried about diluting the team effort to support additional kv solutions. I will talk with some people, I want to work out how we support something like this over time. My gut feeling is that we will definitely need a running system test setup ("esti" in the repo). But I hope you understand that we will have to understand what commitments we end up making.
Personally, were I you, I would just start working on it and see where that took me. (Don't tell my boss.) Let's have a VC later this week and see?
д

Денис Егоров

12/09/2023, 7:07 PM
Yes, I understand... Then I'm waiting for the "final decision".
👍 1
n

Niro

12/09/2023, 8:38 PM
As a first measure, the kV package has an acceptance test suite that can be used to check basic implementation logic
👍 1
a

Ariel Shaqed (Scolnicov)

12/10/2023, 2:10 PM
Good news: I talked to the team & we're good to go 🙂 . Let me know when and how you want to proceed! Obviously this is your call. I suggest a VC or just chat.
e

einat.orr

12/10/2023, 2:13 PM
@Денис Егоров Just my personal opinion, but I prefer you go with the Galera cluster than with TiDB.
д

Денис Егоров

12/11/2023, 10:35 AM
Great! ) What are our actions? Am I writing code following the example of Postgres or...? You can communicate, but my English is very bad. The best way is to chat. Yes, we can start with Galera cluster.
Maybe you have some vision of how this should all happen? Maybe a short plan in the form of abstracts.
a

Ariel Shaqed (Scolnicov)

12/11/2023, 10:38 AM
Great, let's get started! I think that, since we are talking about adding an SQL-based KV backend, it will be a lot like the Postgres backend. So I would start with reading the Postgres implementation: it lives here.
д

Денис Егоров

12/11/2023, 10:41 AM
ok... Is this driver for muscle?
a

Ariel Shaqed (Scolnicov)

12/11/2023, 10:41 AM
I would guess most of the operations there will be almost the same? Apart from the code that creates a table the first time lakeFS runs, and of course
Open
. The first you probably don't need at first. Also the whole business with the advisory lock looks like needless complexity. The second is to open a pool of connections to the DB; you probably know more about MySQL than I do.
ok...
Is this driver for muscle?
Not a mySQL expert but that library is what Go says to use in the tutorial, so I'd say yes.
д

Денис Егоров

12/11/2023, 10:44 AM
yes, it is used here.
a

Ariel Shaqed (Scolnicov)

12/11/2023, 11:02 AM
I think what we need to get started it to write mysql/store.go and mysql/store_test.go. The important bit about store_test.go is where it calls kvtest.DriverTest. This runs the existing test of KV functionality. If that passes, we feel much happier about the KV implementation.
I suggest that you look around the code. You will probably want to chat or talk before you start writing (of course this is not required). The full contributor guidelines are here. They might also help you ensure that you have a working development environment, one where you can run
make test
.
д

Денис Егоров

12/11/2023, 11:21 AM
To begin with, everything is there and clear. I’ll take a look and it will be clear later whether it’s clear or not...)) Thank you.
sunglasses lakefs 1
4 Views