https://lakefs.io/ logo
Join Slack
Powered by
`import {ListObjectsCommand, S3Client} from "@aws-...
# help
u
import {ListObjectsCommand, S3Client} from "@aws-sdk/client-s3";
import { fromIni } from "@aws-sdk/credential-providers";
const client = new S3Client({
credentials: fromIni({profile: 'local'}),
endpoint: "<http://localhost:8000>",
});
var input = {
Bucket:"call-center3"
};
const command = new ListObjectsCommand(input);
const response = await client.send(command);
console.log(response)
u
I see several potential issues: 1. Make sure that you pass lakeFS credentials to the SDK, I can’t tell from the example if 
local
profile in the file is pointing to lakeFS. 2. Unlike S3, lakeFS has branches. When you try to list objects from a repository you need to pass a prefix as well for the branch (must) and a prefix under the branch (optional).
u
@Itai Admi My credentials are fine as I am able to use the ListBucketsCommand just fine. Would my prefix be the same thing as my branch name?
u
Yes, followed by a 
/
u
@Itai Admi I have to apologize, I misstated what I was getting a a response. I am getting a 200 response with no data objects. Just for reference I will post the response from ListBuckets as well. 
import {ListObjectsCommand,ListObjectsV2Command,ListBucketsCommand, S3Client} from "@aws-sdk/client-s3";
import { fromIni } from "@aws-sdk/credential-providers";
const client = new S3Client({
credentials: fromIni({profile: 'local'}),
endpoint: "<http://localhost:8000>",
bucketEndpoint: false,
});
var input = {
Bucket:"call-center3",
Prefix:"main/"
};
const command = new ListBucketsCommand({});
const response = await client.send(command);
console.log(response)
u
That code produces this response 
{
'$metadata': {
httpStatusCode: 200,
requestId: undefined,
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
},
Buckets: [
{ Name: 'call-center', CreationDate: 2022-09-06T14:45:37.661Z },
{ Name: 'call-center2', CreationDate: 2022-09-06T14:51:10.171Z },
{ Name: 'call-center3', CreationDate: 2022-09-06T15:06:38.831Z }
],
Owner: { DisplayName: '', ID: '' }
}
u
Just to make sure I understand - both ListBuckets and ListObjects work as expected?
u
This code 
import {ListObjectsCommand,ListObjectsV2Command,ListBucketsCommand, S3Client} from "@aws-sdk/client-s3";
import { fromIni } from "@aws-sdk/credential-providers";
const client = new S3Client({
credentials: fromIni({profile: 'local'}),
endpoint: "<http://localhost:8000>",
bucketEndpoint: false,
});
var input = {
Bucket:"call-center3",
Prefix:"main/"
};
const command = new ListObjectsV2Command(input);
const response = await client.send(command);
console.log(response)
Produces this response : 
{
'$metadata': {
httpStatusCode: 200,
requestId: undefined,
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
}
}
u
ListObjects returns no data
u
ListBuckets works perfect.
u
Silly question: Do you have objects under the 
main
branch?
u
Second set of code is ListObjects
u
Yes sir. I have a package.json file in there
u
That’s weird. Can you share lakeFS logs? You may be needing to change the level to TRACE for something useful to surface..
u
Can you direct me to the logs?
u
How are you running lakeFS?
u
I have it running in a docker container using the install instructions from the website.
u
I am in the container now.
u
you can get it from the outside too. 
docker logs <container_id>
u
I am not seeing logs that indicate I am making contact either way. How do I turn up the level to TRACE?
u
in my config.yaml? Do I need to restart my container once I make that change?
u
in config yaml https://docs.lakefs.io/reference/configuration.html:
Copy code
logging:
  level: trace
Yes, container needs to restart. But not deleted.
u
Inside of the docker container I have added a config.yaml in /home/lakefs with the following:
Copy code
logging:
  format: json
  level: TRACE
  output: "-"
I then restarted my container and added a file through the UI. I don't see an increased level of logging. Do you have any idea what I have done wrong?
u
That’s unnecessary.. You can add it as env var to the container. 
LAKEFS_LOGGING_LEVEL=TRACE
u
Are you using 
docker-compose
?
u
Yeah. I started the container with docker-compose. Do I have to make of a copy of this container and start that one with the log level set or can I restart my container and set the ENV variable at that time? Sorry for making you give me a mini course in docker.
u
No worries. No need to copy the container, you can use the same one with the env var
u
I have gotten some log data but it doesn't indicate any errors. I ran the 2 commands simultaneously and here are the log from the 2 request.
Copy code
time="2022-09-07T16:56:23Z" level=debug msg="performing S3 action" func=pkg/gateway.EnrichWithOperation.func1.1 file="build/pkg/gateway/middleware.go:116" action=list_repos message_type=action
time="2022-09-07T16:56:23Z" level=debug msg="HTTP call ended" func=net/http.HandlerFunc.ServeHTTP file="usr/local/go/src/net/http/server.go:2047" host=localhost log_audit=API method=GET path=/ request_id=62baec00-fc16-444a-9042-1b8a0027e927 sent_bytes=529 service_name=s3_gateway status_code=200 took=25.8603ms user=admin
Copy code
time="2022-09-07T16:56:23Z" level=debug msg="performing S3 action" func=pkg/gateway.EnrichWithOperation.func1.1 file="build/pkg/gateway/middleware.go:116" action=list_repos message_type=action
time="2022-09-07T16:56:23Z" level=debug msg="HTTP call ended" func=net/http.HandlerFunc.ServeHTTP file="usr/local/go/src/net/http/server.go:2047" host=call-center2.localhost log_audit=API method=GET path="/?prefix=main%2F" request_id=a98659cc-ae3a-44de-a5ef-ce01f248c4f4 sent_bytes=529 service_name=s3_gateway status_code=200 took=2.7925ms user=admin
u
It seems like both requests are ListBuckets requests (or 
list_repos
as lakeFS writes it)
u
The second request should be list objects
u
Notice the prefix there
u
I am using the S3 client from AWS so where do you think the break down is occurring? I am using the ListObjects function but it is running the list_repos action on the LakeFS side.
u
Can you try using path style access for the S3 client, something like
Copy code
AmazonS3 s3client = AmazonS3Client.builder()
            .withCredentials((new AWSStaticCredentialsProvider(credentials)))
            .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("host", "region"))
            .withPathStyleAccessEnabled(true)
            .build();
u
FORCEPATHSTYLE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! I just had to add that option when building my s3 client.
u
Copy code
const client = new S3Client({
    credentials: fromIni({profile: 'local'}),
    endpoint: "<http://localhost:8000>",
    bucketEndpoint: false,
    forcePathStyle:true

  });
u
I only spent 10 hours on that one. 😁
u
I'm glad it's working, let us know if there's anything else we can do to help
u
now the action is list objects
Open in Slack
PreviousNext
Powered by