I m feeling very stupid not being able to follow the idiot s

Question

I m feeling very stupid not being able to follow the idiot s guide but here I am Using Cloudflare s R2 S3 as the backend I ve created a bucket at `https lt id gt ` Created a lakefs config file ```$ cat config lakefs yaml database type postgres postgres connection string auth encrypt secret key a4d333b733f7b63d69fe9065701a4cf8 blockstore type s3 s3 force path style true endpoint https lt id gt discover bucket region false credentials access key id lt key gt secret access key lt secret gt ``` Running `lakefs config ~ config lakefs yaml run` does what s expected I create an admin account and attempt to create a new repo with the following settings ```Repository ID local dev Storage Namespace s3 lt id gt Default Branch trunk``` That gives me a failed to create repository failed to access storage error ```ERROR 2023 04 18T12 03 06+01 00 lakeFS pkg api auth middleware go 351 pkg api userByAuth authenticate error= 2 errors occurred n t built in authenticator credentials not found n t email authenticator crypto bcrypt hashedSecret too short to be a bcrypted password n n service=api gateway user= ERROR 2023 04 18T12 04 39+01 00 lakeFS pkg block s3 adapter go 309 pkg block s3 Adapter Get failed to get S3 object bucket lt id gt key dummy error= InvalidBucketName The specified bucket name is not valid n tstatus code 400 request id host id host= 127 0 0 1 8000 method=POST operation=GetObject operation id=CreateRepository path= api v1 repositories request id=11d96aa3 7c8e 42f2 8f64 bb35429258a9 service name=rest api user=sg WARNING 2023 04 18T12 04 39+01 00 lakeFS pkg api controller go 1576 pkg api Controller CreateRepository Could not access storage namespace error= InvalidBucketName The specified bucket name is not valid n tstatus code 400 request id host id reason=unknown service=api gateway storage namespace= s3 lt id gt ERROR 2023 04 18T12 05 20+01 00 lakeFS pkg block s3 adapter go 275 pkg block s3 Adapter streamToS3 bad S3 PutObject response error= s3 error lt xml version= 1 0 encoding= UTF 8 gt lt Error gt lt Code gt NotImplemented lt Code gt lt Message gt STREAMING AWS4 HMAC SHA256 PAYLOAD not implemented lt Message gt lt Error gt host= 127 0 0 1 8000 method=POST operation=PutObject operation id=CreateRepository path= api v1 repositories request id=061e03f2 a2e2 471b 9dfe b22f98fb0cb8 service name=rest api status code=501 url= https lt id gt user=sg WARNING 2023 04 18T12 05 20+01 00 lakeFS pkg api controller go 1576 pkg api Controller CreateRepository Could not access storage namespace error= s3 error lt xml version= 1 0 encoding= UTF 8 gt lt Error gt lt Code gt NotImplemented lt Code gt lt Message gt STREAMING AWS4 HMAC SHA256 PAYLOAD not implemented lt Message gt lt Error gt reason=unknown service=api gateway storage namespace= ERROR 2023 04 18T12 05 36+01 00 lakeFS pkg block s3 adapter go 309 pkg block s3 Adapter Get failed to get S3 object bucket lt id gt key dummy error= InvalidBucketName The specified bucket name is not valid n tstatus code 400 request id host id host= 127 0 0 1 8000 method=POST operation=GetObject operation id=CreateRepository path= api v1 repositories request id=c01cf2d9 5be1 4bff be29 467f082de1d4 service name=rest api user=sg WARNING 2023 04 18T12 05 36+01 00 lakeFS pkg api controller go 1576 pkg api Controller CreateRepository Could not access storage namespace error= InvalidBucketName The specified bucket name is not valid n tstatus code 400 request id host id reason=unknown service=api gateway storage namespace= s3 lt id gt ``` If I click the info icon in the create repo modal it takes me to a dead link so I ve no idea what format Storage Namespace is expecting

Or Tzabary · Answer

hey < Sam Gaw> slightly smiling face let me have a look

Or Tzabary · Answer

I haven t tested cloudflare in specific but I think that for the storage namespace you should drop the `https lt id gt ` prefix as it s already configured in the configuration file can you please retry with `` only regarding the broken link sorry to hear that I to address this

Sam Gaw · Answer

That moved things in the right direction New error message uncovers `STREAMING AWS4 HMAC SHA256 PAYLOAD not implemented`

Sam Gaw · Answer

Seems CF doesn t support Streaming SigV4 used for chunked uploading

Sam Gaw · Answer

Only way around it is to disable payload signing on putting objects

Sam Gaw · Answer

So atm CF R2 isn t supported

Or Tzabary · Answer

that s true mind sharing a bit on your use case if that s something you re still interested in you can open an issue in the lakeFS repository to add support for CloudFlare for us to examine and see if there are any other community members looking for the same capability

Sam Gaw · Answer

CF have no egress fees on reading objects so for training data that s regularly accessed it s the most cost effective for us ~2000 files largest ~30GB all updating daily

Sam Gaw · Answer

We re also planning on migrating 180TB from BiqQuery to parquet files stored in R2

Or Tzabary · Answer

interesting thank you for sharing let me check to see if there s a workaround for you to try I wonder if CF support SigV2 or if we could leverage presigned URLs for this to work

Sam Gaw · Answer

Seems they use SigV4 they just don t support Streaming So SigV4 still needs to be used for presigned URLs

Or Tzabary · Answer

I wonder have you used lakeFS with GCP or were you experimenting it with CloudFlare first

Sam Gaw · Answer

We re making the conscious decision to move away from all Google infra GCP isn t an option

Or Tzabary · Answer

I understand I wonder if you were already using lakeFS with GCP and now you re trying to migrate it to CloudFlare or as part of onboarding to CloudFlare you re testing new technologies

Sam Gaw · Answer

Haven t used lakefs before I m brand new

Sam Gaw · Answer

Going to try setting up with Wasabi to see if they have better compatibility CF was just because we already had an existing account

Or Tzabary · Answer

Got it let me know pray

einat.orr · Answer

< Sam Gaw> if you re choosing storage how about It s OSS with great performance and fully compatible with lakeFS slightly smiling face

Sam Gaw · Answer

< einat orr> The problem with minio is hosting costs Rather than juggling multiple systems I d like to get the bigquery data under the same roof And when you go into the 200TB range the costs become crippling Colo would be the only option to make that viable longterm

Sam Gaw · Answer

So Wasabi works well and has comparable costs $6 TB month and no egress charges

Sam Gaw · Answer

Although something else I noticed is that `lakefs` seems to be ignoring the `search path` param given to pgx in the postgres connection string If I try to set the postgres user to a specific schema it barfs `error= setup failed ERROR no schema has been selected to create in SQLSTATE 3F000 ` on startup If I try to enforce it only from the DB side with ALTER USER it ignores the search path set in the session and explicitly tries to create tables in public `error= setup failed ERROR permission denied for schema public SQLSTATE 42501 `

Sam Gaw · Answer

Appreciate I seem to be QAing a lot of edge cases all at once slightly smiling face

Or Tzabary · Answer

no worries slightly smiling face

Or Tzabary · Answer

if I understand correctly following the configuration you shared you d expect lakeFS to use the `lakefs dev` scheme and it seems to be trying to use the `public` scheme regardless of the connection string is that the case

Sam Gaw · Answer

Generally we try to have a monolith DB in Postgres spliting responsiblities by schemas To get it running I had to use `````` But after creating a datalake schema and pinning the `lakefs` user to it via search path the config that produced the errors was `````` Note application name is respected and I can confirm with `SELECT FROM pg stat activity WHERE application name = lakefs `

Or Tzabary · Answer

I see let me check

Or Tzabary · Answer

can you try that without the `application name` filter