Are there any Debezium people here? I just finished an interesting conversation with Gunnar Morling, a contributor to Debezium open source and we thought about creating a sink adapter with lakeFS for cases when we need better management of propagating data from stream processing into a data lake while using webhooks to test and qualify the data. for example, propagating data into an apache iceberg-based data lake as described in this article. wdyt?
In the past hive-catalog had significant issues with keeping db and FS in sync.. Have we considered this within lakefs?
5 months ago
A new flag
lakectl branch create
command, to have a different stage for different users that works on the same branch (currently, users might override each other's staging changes when working on the same branch).
It might be done at the repository/installation level, but it makes more sense (to me) that it should be done at the branch level.
4 months ago
hey folks, does lakefs support fast-forward merges or is this on the roadmap? we want to build a feature-branch workflow on lakefs that mirrors our git PR workflow, where we squash commits and fast-forward merge into master
3 months ago
👋 Hi! Is it possible to limit read/write access of LakeFS to files in S3? Specifically, we plan to use the LakeFS Hadoop FS to read/write many of our files. It seems then that LakeFS doesn't need permissions on those files, as it will be our Spark clusters that need those permissions instead. The downside is that the files could not be downloaded from the LakeFS API, but, for these types of files, I don't think we would be doing that anyway. I can create a restricted IAM policy right now, but I am not sure which files LakeFS needs read/write access to (e.g.
) and which ones are just our data. Is there a way for me to distinguish between LakeFS metadata and our data currently? If not, is there something we can add (e.g. a prefix) that makes such IAM policies easy to create?
3 months ago
👋 Hello everyone! Started a guide draft on how to render lakeFS docs locally to test docs structure before creating a docs pr. if you are contributing to the docs, we will appreciate your feedback! 📚
@Jonathan Rosenberg thanks, i do still get errors when i try two of the commands
(base) paulsingman@Pauls-MacBook-Pro lakeFS % sudo gem install bundler:2.1.4
ERROR: While executing gem ... (Gem::FilePermissionError)
You don't have write permissions for the /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/gems/2.6.0 directory.
and when i try this one:
(base) paulsingman@Pauls-MacBook-Pro docs % bundle install; cd ..; make serve-docs
/Library/Ruby/Site/2.6.0/rubygems.rb:265:in `find_spec_for_exe': can't find gem bundler (>= 0.a) with executable bundle (Gem::GemNotFoundException)
from /Library/Ruby/Site/2.6.0/rubygems.rb:284:in `activate_bin_path'
from /usr/bin/bundle:23:in `<main>'
make: *** Noruleto make target `serve-docs'. Stop.
1 month ago
LakeFS team. integration is going very smoothly. one question: when do you think SAML integration will be supported? (we hate manually managing users) 🙂