Hi I am trying to read data from a LakeFS repo For Object St

Question

Hi I am trying to read data from a LakeFS repo For Object Storage I am making use of the azure blob storage I run the following command ` Reading data` `repo = lt my repo gt ` `branch = main ` `dataPath = s3a lt my repo main bronze inp claimsk lds 5 2020 sample csv ` ` df = spark read dataPath ` `df = spark read format csv load dataPath inferSchema = True ` And I get the following error ```Py4JJavaError An error occurred while calling o85 load java nio file AccessDeniedException smartlens org apache hadoop fs s3a auth NoAuthWithAWSException No AWS Credentials provided by SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider com amazonaws SdkClientException The requested metadata is not found at at org apache hadoop fs s3a S3AUtils translateException S3AUtils java 187 at org apache hadoop fs s3a Invoker once Invoker java 111 at org apache hadoop fs s3a Invoker lambda$retry$3 Invoker java 265 at org apache hadoop fs s3a Invoker retryUntranslated Invoker java 322 at org apache hadoop fs s3a Invoker retry Invoker java 261 at org apache hadoop fs s3a Invoker retry Invoker java 236 at org apache hadoop fs s3a S3AFileSystem verifyBucketExists S3AFileSystem java 375 at org apache hadoop fs s3a S3AFileSystem initialize S3AFileSystem java 311 at org apache hadoop fs FileSystem createFileSystem FileSystem java 3303 at org apache hadoop fs FileSystem access$200 FileSystem java 124 at org apache hadoop fs FileSystem$Cache getInternal FileSystem java 3352 at org apache hadoop fs FileSystem$Cache get FileSystem java 3320 at org apache hadoop fs FileSystem get FileSystem java 479 at org apache hadoop fs Path getFileSystem Path java 361 at org apache spark sql execution streaming FileStreamSink$ hasMetadata FileStreamSink scala 46 at org apache spark sql execution datasources DataSource resolveRelation DataSource scala 377 at org apache spark sql DataFrameReader loadV1Source DataFrameReader scala 325 at org apache spark sql DataFrameReader $anonfun$load$3 DataFrameReader scala 307 at scala Option getOrElse Option scala 189 at org apache spark sql DataFrameReader load DataFrameReader scala 307 at org apache spark sql DataFrameReader load DataFrameReader scala 239 at java base jdk internal reflect NativeMethodAccessorImpl invoke0 Native Method at java base jdk internal reflect NativeMethodAccessorImpl invoke NativeMethodAccessorImpl java 62 at java base jdk internal reflect DelegatingMethodAccessorImpl invoke DelegatingMethodAccessorImpl java 43 at java base java lang reflect Method invoke Method java 566 at py4j reflection MethodInvoker invoke MethodInvoker java 244 at py4j reflection ReflectionEngine invoke ReflectionEngine java 357 at py4j Gateway invoke Gateway java 282 at py4j commands AbstractCommand invokeMethod AbstractCommand java 132 at py4j commands CallCommand execute CallCommand java 79 at py4j GatewayConnection run GatewayConnection java 238 at java base java lang Thread run Thread java 829 Caused by org apache hadoop fs s3a auth NoAuthWithAWSException No AWS Credentials provided by SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider com amazonaws SdkClientException The requested metadata is not found at at org apache hadoop fs s3a AWSCredentialProviderList getCredentials AWSCredentialProviderList java 159 at com amazonaws http AmazonHttpClient$RequestExecutor getCredentialsFromContext AmazonHttpClient java 1257 at com amazonaws http AmazonHttpClient$RequestExecutor runBeforeRequestHandlers AmazonHttpClient java 833 at com amazonaws http AmazonHttpClient$RequestExecutor doExecute AmazonHttpClient java 783 at com amazonaws http AmazonHttpClient$RequestExecutor executeWithTimer AmazonHttpClient java 770 at com amazonaws http AmazonHttpClient$RequestExecutor execute AmazonHttpClient java 744 at com amazonaws http AmazonHttpClient$RequestExecutor access$500 AmazonHttpClient java 704 at com amazonaws http AmazonHttpClient$RequestExecutionBuilderImpl execute AmazonHttpClient java 686 at com amazonaws http AmazonHttpClient execute AmazonHttpClient java 550 at com amazonaws http AmazonHttpClient execute AmazonHttpClient java 530 at com amazonaws services s3 AmazonS3Client invoke AmazonS3Client java 5227 at com amazonaws services s3 AmazonS3Client getBucketRegionViaHeadRequest AmazonS3Client java 6189 at com amazonaws services s3 AmazonS3Client fetchRegionFromCache AmazonS3Client java 6162 at com amazonaws services s3 AmazonS3Client invoke AmazonS3Client java 5211 at com amazonaws services s3 AmazonS3Client invoke AmazonS3Client java 5173 at com amazonaws services s3 AmazonS3Client headBucket AmazonS3Client java 1438 at com amazonaws services s3 AmazonS3Client doesBucketExist AmazonS3Client java 1374 at org apache hadoop fs s3a S3AFileSystem lambda$verifyBucketExists$1 S3AFileSystem java 376 at org apache hadoop fs s3a Invoker once Invoker java 109 30 more Caused by com amazonaws SdkClientException The requested metadata is not found at at com amazonaws internal EC2ResourceFetcher doReadResource EC2ResourceFetcher java 89 at com amazonaws internal EC2ResourceFetcher doReadResource EC2ResourceFetcher java 70 at com amazonaws internal InstanceMetadataServiceResourceFetcher readResource InstanceMetadataServiceResourceFetcher java 75 at com amazonaws internal EC2ResourceFetcher readResource EC2ResourceFetcher java 66 at com amazonaws auth InstanceMetadataServiceCredentialsFetcher getCredentialsEndpoint InstanceMetadataServiceCredentialsFetcher java 58 at com amazonaws auth InstanceMetadataServiceCredentialsFetcher getCredentialsResponse InstanceMetadataServiceCredentialsFetcher java 46 at com amazonaws auth BaseCredentialsFetcher fetchCredentials BaseCredentialsFetcher java 112 at com amazonaws auth BaseCredentialsFetcher getCredentials BaseCredentialsFetcher java 68 at com amazonaws auth InstanceProfileCredentialsProvider getCredentials InstanceProfileCredentialsProvider java 165 at org apache hadoop fs s3a AWSCredentialProviderList getCredentials AWSCredentialProviderList java 137 48 more```

Barak Amar · Answer

From the error it looks like the credentials are missing When you access lakeFS you are passing key secret The application needs these creds to access lakefs too <https docs lakefs io integrations spark html use the s3 gateway> There `spark hadoop fs s3a access key` and `spark hadoop fs s3a secret key` described here

Barak Amar · Answer

You can pass them through the code the example there is in scala