I used docker <https lakefs io the docker everything bagel s lakeFS #dev

I used docker “<Everything Bagel>” to spin up a la...

Tal Sofer

11/21/2021, 9:43 AM

I used docker “Everything Bagel” to spin up a lakeFS setup connected to hive metastore, and ran

docker compose --profile client run --rm hive-client

to enable the hive-client. Now, I would like to create my first table in the metastore. How do I use the hive-client to do that? @Guy Hardonag you can probably point me to the right direction

Guy Hardonag

11/21/2021, 9:52 AM

Hey @Tal Sofer, You could create a table in hive metastore using one of the following clients: • Hive • Trino • Spark-sql My choice would be Trino Hive Client: run:

Copy code

docker compose --profile client run --rm hive-client

In Hive:

Copy code

CREATE EXTERNAL TABLE `inventory`(
  `inv_item_sk` int,
  `inv_warehouse_sk` int,
  `inv_quantity_on_hand` int)
  PARTITIONED BY (
  `inv_date_sk` int) STORED AS ORC
  LOCATION
  '<s3://REPO_NAME/BRANCH_NAME/PATH_TO_TABLE>';

Trino Run:

Copy code

docker compose --profile client run --rm trino-client

in Trino:

Copy code

CREATE TABLE request_logs (
  request_time timestamp,
  url varchar,
  ip varchar,
  user_agent varchar
)
WITH (
  format = 'PARQUET',
  external_location = '<s3://REPO_NAME/BRANCH_NAME/PATH_TO_TABLE>'
)

Tal Sofer

11/21/2021, 10:19 AM

Thanks Guy!

Tal Sofer

11/21/2021, 10:26 AM

Copy code

trino:default> CREATE TABLE request_logs (
            ->   request_time timestamp,
            ->   url varchar,
            ->   ip varchar,
            ->   user_agent varchar
            -> )
            -> WITH (
            ->   format = 'PARQUET',
            ->   external_location = '<s3://example/main>'
            -> )
            -> ;
Query 20211121_102532_00007_gvtka failed: Got exception: java.io.FileNotFoundException PUT 0-byte object  on main/: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: null; S3 Extended Request ID: null), S3 Extended Request ID: null:404 Not Found

Tal Sofer

11/21/2021, 10:27 AM

What am I missing? (I didn’t configure anything in addition to what i’m getting from the compose file because I assumed it is given, is this a correct assumption?)

Guy Hardonag

11/21/2021, 10:54 AM

Try using the location:

<s3://example/main/request_logs>

Tal Sofer

11/21/2021, 10:56 AM

Worked, thanks. Does this mean that we can’t create table on the root of a branch, is this intentional?

5 Views

Open in Slack

Previous Next