Ok so I finally got it working but I'm running int...
# help
i
Ok so I finally got it working but I'm running into an issue. It seems the exporter doesn't support exporting protocol V3,7 of delta..
j
Can you explain what you mean by protocol V3,7?
i
Minimum writer version 3 and minimum reader version 7
You should be able to parse this, including the reader features and writer features
j
Currently, this feature is not supported at the moment. The protocol action can specify only min reader and min writer (without table features).
i
Can this be added then?
I went through all the trouble to add timestampNtz support in delta-rs😆
j
If it’s possible by your end, a feature request with the reasoning behind it could greatly impact the progression of such feature
i
Sure, but reader and writer features are the way forward for delta, so not supporting this means a hard stop at supporting new tables
👍 1
j
It seems like you’ve got knowledge about it, and that you deeply understand the need for this feature. Therefore I think it would be great if you could elaborate on that, via a feature request, so that the team and contributors would be able to grasp it as well, and so that we could prioritize it properly.
i
Yeah I made feature requests now :)
I also haven't tested it yet, but do you guys also properly re export the delta checkpoints?
j
The export process goes back to the latest checkpoint (or first entry) and starts exporting from that point forward. It exports a representation of the table and not a complete copy. It’s purpose is to allow the reading of the delta table from an external client which cannot reach lakeFS.
i
Ok then the exporter is incorrect though
And will create tables with an invalid state
j
why is that?
i
You should include the checkpoint since it contains the state of that table at that point in time. If you skip it, you don't know what the protocol versions are
j
Never said it’s skipped. The exporter grabs it and fetches all of the entries in it. It then uses it as the first log entry in the exported table.
i
Ah ok, that's fine yeah
Still a bit weird, but works
Ideally it's also written as parquet file, since you might have use cases where that parquet checkpoint is 500Mb