Hi Team, I am running ACL server separately in pod. It is getting connected to pssql and lakfes is connecting to ACL server. Upon ACL pod restart. Acl shows error. time="2024-10-07T075536Z" level=fatal msg="failed to query auth service" func=github.com/treeverse/lakefs/contrib/auth/acl/cmd/acl/cmd.init.func1 file="/app/contrib/auth/acl/cmd/acl/cmd/run.go:51" error="setup: create base ACL group Admins: save group (groupKey groups/Admins): already exists" run.go file line 51 is responsible. But it should validate the presence of group first. If not then create. Is that correct understanding?

Hey man. I've just run across the same issue. But my Docker's lakeFS and Postgres instances are using the same config as local ACL - keys are the same. Could you please share your configs and how were you able to fix the issue?

@mpn mbn initially, i got this (groupKey groups/Admins): already exists. so, contrib/auth/acl/setup.go (func CreateACLBaseGroups) i added validation step if groups are present then dont create and it went fine and got another error. then added more verbose in to func "IsInitialized" in the same file. time="2024-10-07T132127Z" level=info msg="Checking if ACL server is initialized..." func=github.com/treeverse/lakefs/contrib/auth/acl.SetupACLServer file="/app/contrib/auth/acl/setup.go:105" time="2024-10-07T132127Z" level=info msg="ACL server is not initialized (timestamp not found)." func=github.com/treeverse/lakefs/contrib/auth/acl.IsInitialized file="/app/contrib/auth/acl/setup.go:133" login to pssql database & checked kv table. acluser=> SELECT DISTINCT partition_key, key FROM kv LIMIT 10; partition_key | key decoded values: Partition_key: aclauth Key: installation_metadata/setup_timestamp In file lakeFS/contrib/auth/acl/service.go line 937 - SetupTimestampKeyName (referencing - pkg/auth/metadata.go) in which setup_timestamp is mentioned hence, both function getSetupTimestamp and updateSetupTimestamp are referring the same. I updated that in both function in service.go with value (installation_metadata/setup_timestamp) and it acked the time and on restart it didnt initiate setup again. config.yaml (values referred from pkg/config/config.go) listen_address: ":8001" database: type: "postgres" drop_tables: false postgres: connection_string: "STRING!!!!!!!" max_open_connections: 25 max_idle_connections: 25 connection_max_lifetime: 5m scan_page_size: 100 metrics: false encrypt: secret_key: "KEY!!!!!!!!!"

I encountered the same issue, here is a psql command that can resolve the issue without code changes:

INSERT INTO kv (partition_key, key, value)
SELECT 'aclauth', 'setup_timestamp', value
FROM kv
WHERE key = 'installation_metadata/setup_timestamp';

Seems like this is a bug in the ACL server here: https://github.com/treeverse/lakeFS/blob/d10750248adcceb89aa06fe131a5e071f261090b/contrib/auth/acl/service.go#L937

any chance someone can make a PR ?

ok one sec