Robin Moffatt
03/07/2023, 2:46 PMaws
CLI doing the same thing
• The lakeFS server log for the HTTP interaction is the same for the R (doesn't work) and Python (does work)
• The R code works fine listing buckets and objects with Minio.
• R library and S3 HTTP code
Any pointers to what to check next? thanks.
Full details in threadaws s3api list-buckets --endpoint-url <http://127.0.0.1:8000>
json
{
"Buckets": [
{
"Name": "example",
"CreationDate": "2023-03-07T14:08:35.111000+00:00"
}
],
"Owner": {
"DisplayName": "",
"ID": ""
}
}
## List buckets from boto
import boto3
AWS_ACCESS_KEY_ID='AKIAJNSOLJH5YUW4KLBQ'
AWS_SECRET_ACCESS_KEY='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'
endpoint = '<http://localhost:8000>'
session = boto3.Session(
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
)
s3 = session.resource('s3', endpoint_url=endpoint)
print(list(s3.buckets.all()))
Output
[s3.Bucket(name='test-repo')]
Server log
{
"action": "list_repos",
"file": "build/pkg/gateway/middleware.go:124",
"func": "pkg/gateway.EnrichWithOperation.func1.1",
"level": "debug",
"message_type": "action",
"msg": "performing S3 action",
"ref": "",
"repository": "",
"time": "2023-03-07T13:47:17Z",
"user_id": "admin"
}
{
"client": "Boto3/1.26.78 Python/3.11.2 Darwin/22.2.0 Botocore/1.29.78 Resource",
"file": "usr/local/go/src/net/http/server.go:2109",
"func": "net/http.HandlerFunc.ServeHTTP",
"host": "localhost:8000",
"level": "debug",
"log_audit": true,
"method": "GET",
"msg": "HTTP call ended",
"operation_id": "list_buckets",
"path": "/",
"request_id": "81fb3769-dd20-468d-a889-a4cd8d43b7c2",
"sent_bytes": 250,
"service_name": "s3_gateway",
"source_ip": "172.17.0.1:58854",
"status_code": 200,
"time": "2023-03-07T13:47:17Z",
"took": 644584,
"user": "admin"
}
## List buckets from R
# Install packages
install.packages(c("aws.s3"))
# Load necessary libraries
library(aws.s3)
library(data.table)
library(ggplot2)
# Define AWS S3 credentials and endpoint
Sys.setenv("AWS_ACCESS_KEY_ID" = "AKIAIOSFODNN7EXAMPLE",
"AWS_SECRET_ACCESS_KEY" = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY")
baseurl="localhost:8000"
b="drones03"
# List buckets
bucketlist(add_region = FALSE,base_url=baseurl,verbose=FALSE,use_https=FALSE)
# List objects in bucket
get_bucket_df(bucket=b, base_url=baseurl,add_region=FALSE,verbose=FALSE,use_https=FALSE)
Output: Note that nothing is returned for either command
> # List buckets
> bucketlist(add_region = FALSE,base_url=baseurl,verbose=FALSE,use_https=FALSE)
data frame with 0 columns and 0 rows
> # List objects in bucket
> get_bucket_df(bucket=b, base_url=baseurl,add_region=FALSE,verbose=FALSE,use_https=FALSE)
[1] Key LastModified ETag Size
[5] Owner_ID Owner_DisplayName StorageClass Bucket
<0 rows> (or 0-length row.names)
Server log
{
"action": "list_repos",
"file": "build/pkg/gateway/middleware.go:124",
"func": "pkg/gateway.EnrichWithOperation.func1.1",
"level": "debug",
"message_type": "action",
"msg": "performing S3 action",
"ref": "",
"repository": "",
"time": "2023-03-07T13:49:28Z",
"user_id": "admin"
}
{
"client": "libcurl/7.85.0 r-curl/5.0.0 httr/1.4.5",
"file": "usr/local/go/src/net/http/server.go:2109",
"func": "net/http.HandlerFunc.ServeHTTP",
"host": "localhost:8000",
"level": "debug",
"log_audit": true,
"method": "GET",
"msg": "HTTP call ended",
"operation_id": "list_buckets",
"path": "/",
"request_id": "3c5df6e2-0a0e-49bc-9d74-c3fd3dfcccde",
"sent_bytes": 250,
"service_name": "s3_gateway",
"source_ip": "172.17.0.1:52904",
"status_code": 200,
"time": "2023-03-07T13:49:28Z",
"took": 307500,
"user": "admin"
}
## Prove R code works against another S3 implementation
# Define Minio credentials
Sys.setenv("AWS_ACCESS_KEY_ID" = "minioadmin",
"AWS_SECRET_ACCESS_KEY" = "minioadmin")
baseurl="localhost:9000"
b="example"
# List buckets
bucketlist(add_region = FALSE,base_url=baseurl,verbose=FALSE,use_https=FALSE)
# List objects in bucket
get_bucket_df(bucket=b, base_url=baseurl,add_region=FALSE,verbose=FALSE,use_https=FALSE)
Output
> # List buckets
> bucketlist(add_region = FALSE,base_url=baseurl,verbose=FALSE,use_https=FALSE)
Bucket CreationDate
1 example 2023-03-07T14:08:18.427Z
> # List objects in bucket
> get_bucket_df(bucket=b, base_url=baseurl,add_region=FALSE,verbose=FALSE,use_https=FALSE)
Key LastModified ETag Size
1 dummy 2023-03-07T14:08:35.103Z "18ab9e2980f7223750c5ed4833f45dab" 70
Owner_ID
1 02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4
Owner_DisplayName StorageClass Bucket
1 minio STANDARD example
>
Barak Amar
03/10/2023, 6:40 PMctype <- httr::headers(r)[["content-type"]]
if (is.null(ctype) || ctype == "application/xml"){
text/xml
.Robin Moffatt
03/10/2023, 9:56 PM