Is there a way to increase the default level of parallelism used by the

~lakefs~ lakectl local

CLI? I didn't see a config here for it: https://docs.lakefs.io/reference/configuration.html

@Aaron Taylor I think you are confusing lakefs with lakectl See this

Ah, I mis-typed, I meant

lakectl local

. Is there a way to override the default parallelism for

lakectl

?

yes, -p flag: https://docs.lakefs.io/reference/cli.html#lakectl-local

I have been using that flag! I meant more permanently, e.g. in a config or environment variable so that we can pick a more optimal value for our environments than the default (raise it quite a bit), and have that be set for all team members so that they have faster performance by default.

Hi @Aaron Taylor can you explain how adding a config parameter increases performance?

Sure, my initial goal here was to confirm whether I had missed something in the docs, or whether this was a feature request, happy to open an issue! This is the parallelism flag definition for

lakectl local checkout

-p, --parallelism int   Max concurrent operations to perform (default 25)

I have seen big performance improvements for some of the lakectl commands by bumping that number way up, e.g. to 500. Our use case has lots of smaller files so it might be helping optimize for that scenario.

Sure, what you are saying makes a lot of sense. Documenting it in an issue will help us address it better! And as always we are always appreciative of contributions in the form of issues/PRs 🙏🏽

Issue opened here! https://github.com/treeverse/lakeFS/issues/8159