Federation query between hive and Iceberg with kerberos enabled.

[fpierrick]

Hi folks,
I need your help.
I configured a Hive Catalog in gravitino

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name":"hive",
  "comment":"Hive Catalog", 
  "type":"RELATIONAL", 
  "provider":"hive", 
  "properties":{
        "metastore.uris":"thrift://xxx:9083,thrift://yyy:9083",
        "gravitino.bypass.hive.metastore.client.capability.check":"FALSE",
        "kerberos.principal":"hive/gravitinoHost@BDFINT2.REALM",
        "gravitino.bypass.hive.metastore.kerberos.principal":"hive/_HOST@DOMAIN.REALM",
        "kerberos.keytab-uri":"file:///keytabs/hive-gravitino.keytab",
        "gravitino.bypass.hive.strict.managed.tables":"false",
        "gravitino.bypass.hive.metastore.sasl.enabled":"TRUE",
        "hive.metastore.warehouse.dir":"hdfs://MYDOMAIN/apps/hive/warehouse/",
        "gravitino.bypass.hadoop.security.authentication":"kerberos"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs

In the GUI of Gravitino I can see all databases and tables.

I launch a spark-sql
${SPARK_HOME}/bin/spark-sql -v --conf spark.plugins="org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin" --conf spark.sql.gravitino.uri=http://gravitinoHost:8090 --conf spark.sql.gravitino.metalake=metalake

I create a database

CREATE DATABASE IF NOT EXISTS gravitino_db;
I create an External Table

spark-sql ()> use gravitino_db;
use gravitino_db
Time taken: 0.112 seconds
spark-sql (gravitino_db)> CREATE EXTERNAL TABLE IF NOT EXISTS students (
student_ID INT,
FirstName STRING,
LastName STRING,
year STRING,
Major STRING)
COMMENT 'Student Names' ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION 'hdfs://MYDOMAIN/apps/hive/warehouse/gravitino_db.db/students';

But I got this error

25/02/03 11:36:03 WARN ObjectStore: Failed to get database gravitino_db, returning NoSuchObjectException
[SCHEMA_NOT_FOUND] The schema `gravitino_db` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a catalog, verify the current_schema() output, or qualify the name with the correct catalog.
To tolerate the error on drop use DROP SCHEMA IF EXISTS.

spark-sql (gravitino_db)> SELECT current_schema();
SELECT current_schema()
gravitino_db
Time taken: 0.822 seconds, Fetched 1 row(s)

But the directory is well created and I can see the table schema in gravitino

hdfs dfs -ls /apps/hive/warehouse/gravitino_db.db/
Found 1 items
drwxrwxr-x+  - hive hive          0 2025-02-03 11:36 /apps/hive/warehouse/gravitino_db.db/students

$ curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://gravitinoHost:8090/api/metalakes/metalake/catalogs/hive/schemas/gravitino_db/tables/students

{"code":0,"table":{"name":"students","comment":"Student Names","columns":[{"name":"student_id","type":"integer","nullable":true,"autoIncrement":false},{"name":"firstname","type":"string","nullable":true,"autoIncrement":false},{"name":"lastname","type":"string","nullable":true,"autoIncrement":false},{"name":"year","type":"string","nullable":true,"autoIncrement":false},{"name":"major","type":"string","nullable":true,"autoIncrement":false}],"properties":{"owner":"root","input-format":"org.apache.hadoop.mapred.TextInputFormat","external":"true","transient_lastDdlTime":"1738578962","hive.stored-as":"TEXTFILE","output-format":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde.parameter.serialization.format":",","serde.parameter.field.delim":",","table-type":"EXTERNAL_TABLE","location":"hdfs://MYDOMAIN/apps/hive/warehouse/gravitino_db.db/students","serde-lib":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe","serde-name":"students"},"audit":{"creator":"root","createTime":"2025-02-03T10:36:03.200188815Z"},"distribution":{"strategy":"none","number":0,"funcArgs":[]},"sortOrders":[],"partitioning":[],"indexes":[]}}

I cannot also list data from the table :

spark-sql (gravitino_db)> SELECT * FROM students;
SELECT * FROM students
25/02/03 11:55:08 WARN ObjectStore: Failed to get database gravitino_db, returning NoSuchObjectException
[TABLE_OR_VIEW_NOT_FOUND] The table or view `students` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.; line 1 pos 14;
'Project [*]
+- 'UnresolvedRelation [students], [], false

Neither an already existing table

spark-sql (gravitino_db)> use my_existing_db;
use my_existing_db
Time taken: 0.133 seconds
spark-sql (my_existing_db)> select * from my_existing_table;
select * from my_existing_table
25/02/03 12:06:04 WARN ObjectStore: Failed to get database my_existing_db, returning NoSuchObjectException
[TABLE_OR_VIEW_NOT_FOUND] The table or view `my_existing_table` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.; line 1 pos 14;
'Project [*]
+- 'UnresolvedRelation [my_existing_table], [], false

It seems there is a problem with reading the data. I may have forgotten a configuration.
Do you have any idea what could be wrong?
Thanks and Best Regards

[FANNG1]

There are some known limitations for spark-sql client, please try using spark-shell or setting spark.bypass.spark.sql.hive.metastore.jars explicitly in Hive catalog properties, please refer to
https://gravitino.apache.org/docs/0.8.0-incubating/spark-connector/spark-catalog-hive#catalog-properties

[fpierrick]

Hi @FANNG1, thanks for your reply.
I will try it. Do you have any example with spark-shell?

[FANNG1]

spark-shell is similar to spark-sql by leveraging the sql function like sql(select * from xx);

[jerqi]

Hi @FANNG1, thanks for your reply. I will try it. Do you have any example with spark-shell?

You can refer to our playground. https://github.com/apache/gravitino-playground/blob/03be79663520a8cbe82e86ebb79f937622c52124/init/spark/spark-defaults.conf#L30

[fpierrick]

Hello,
my spark-sql is now working with gravitino by following the playground conf + adding other configurations.
I put my configuration here if it can help.

Context : POC HiveMetastore + HDFS + Kerberos on stack HDP 3.1.4
Using spark 3.5 (spark-3.5.4-bin-hadoop3-scala2.13).

Add iceberg-spark-runtime-3.5_2.13-1.7.1.jar + gravitino-spark-connector-runtime-3.5_2.13-0.7.0-incubating.jar to the jars folder of spark.
Add core-site.xml, hdfs-site.xml and hive-site.xml to the conf folder of spark.

file conf/spark-default.conf

spark.plugins org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin
spark.sql.gravitino.uri http://gravitino:8090
spark.sql.gravitino.metalake metalake
spark.sql.gravitino.authType kerberos
 
spark.kerberos.principal gravitino@REALM
spark.kerberos.keytab /etc/security/keytabs/gravitino.headless.keytab
 
spark.sql.gravitino.enableIcebergSupport true
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.catalog_rest org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.catalog_rest.type rest
spark.sql.catalog.catalog_rest.uri http://gravitino:9002/iceberg/
spark.locality.wait.node 0
#spark.sql.warehouse.dir hdfs://DOMAIN/apps/hive/warehouse
 
spark.sql.catalog.hive.spark.sql.hive.metastore.jars path
spark.sql.catalog.hive.spark.sql.hive.metastore.jars.path file:///opt/application/spark-3.5.4-bin-hadoop3-scala2.13/jars/*

Server iceberg installed as standalone (gravitino-iceberg-rest-server-0.8.0-incubating-bin.tar.gz)
Add core-site.xml and hdfs-site.xml to the conf folder
file conf/gravitino-iceberg-rest-server.conf

# THE CONFIGURATION FOR Iceberg REST SERVER
gravitino.iceberg-rest.shutdown.timeout = 3000
 
# THE CONFIGURATION FOR Iceberg REST WEB SERVER
# The host name of the built-in web server
gravitino.iceberg-rest.host = 0.0.0.0
# The http port number of the built-in web server
gravitino.iceberg-rest.httpPort = 9002
# The min thread size of the built-in web server
gravitino.iceberg-rest.minThreads = 24
# The max thread size of the built-in web server
gravitino.iceberg-rest.maxThreads = 200
# The stop timeout of the built-in web server
gravitino.iceberg-rest.stopTimeout = 30000
# The timeout of idle connections
gravitino.iceberg-rest.idleTimeout = 30000
# The executor thread pool work queue size of the built-in web server
gravitino.iceberg-rest.threadPoolWorkQueueSize = 100
# The request header size of the built-in web server
gravitino.iceberg-rest.requestHeaderSize = 131072
# The response header size of the built-in web server
gravitino.iceberg-rest.responseHeaderSize = 131072
 
# HIVE CATALOG BACKEND WITH HDFS
gravitino.iceberg-rest.catalog-backend = hive
gravitino.iceberg-rest.catalog-backend-name = hive
gravitino.iceberg-rest.uri = thrift://thrifturi01:9083,thrift://thrifturi02:9083
gravitino.iceberg-rest.authentication.type = kerberos
gravitino.iceberg-rest.authentication.impersonation-enable = false
gravitino.iceberg-rest.hive.metastore.sasl.enabled = true
gravitino.iceberg-rest.hive.metastore.kerberos.principal = hive/_HOST@REALM
gravitino.iceberg-rest.authentication.kerberos.principal = hive/gravitinohost@REALM
gravitino.iceberg-rest.authentication.kerberos.keytab-uri = file:///etc/security/keytabs/hive-gravitino.service.keytab
gravitino.iceberg-rest.authentication.kerberos.check-interval-sec = 60
gravitino.iceberg-rest.authentication.kerberos.keytab-fetch-timeout-sec = 60
gravitino.iceberg-rest.warehouse = hdfs://DOMAIN/apps/iceberg/warehouse

Catalogs:

# metalake
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{"name":"metalake","comment":"This is a new metalake","properties":{}}' \
http://gravitino:8090/api/metalakes
# hive catalog
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
-d '{"name":"hive","comment":"Catalogue Hive BDF", "type":"RELATIONAL", "provider":"hive", "properties":{"metastore.uris":"thrift://thrifturi01:9083,thrift://thrifturi02:9083", "gravitino.bypass.hive.metastore.client.capability.check":"FALSE", "kerberos.principal":"hive/gravitinohost@REALM", "gravitino.bypass.hive.metastore.kerberos.principal":"hive/_HOST@REALM", "kerberos.keytab-uri":"file:///etc/security/keytabs/hive-gravitino.service.keytab", "gravitino.bypass.hive.metastore.sasl.enabled":"TRUE", "gravitino.bypass.hadoop.security.authentication":"kerberos", "hive.metastore.warehouse.dir":"hdfs://DOMAIN/apps/hive/warehouse/"}}' \
http://gravitino:8090/api/metalakes/metalake/catalogs
# iceberg catalog
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
-d '{"name":"iceberg_rest","comment":"Catalogue Iceberg Rest", "type":"RELATIONAL", "provider":"lakehouse-iceberg", "properties":{"catalog-backend":"rest", "authType":"simple", "uri":"http://icebergrestcataloguri:9002/iceberg/"}}' \
http://gravitino:8090/api/metalakes/metalake/catalogs

Test:

./bin/spark-sql
# HIVE
spark-sql (default)> use hive;
spark-sql ()> use mydatabasehive;
spark-sql (mydatabasehive)> select col1 from mytablehive limit 1;
# Iceberg
spark-sql (mydatabasehive)> use catalog_rest;
spark-sql ()> create database test_sales;
spark-sql ()> use test_sales;
spark-sql (test_sales)> create table customers (customer_id int, customer_name varchar(100), customer_email varchar(100));
spark-sql (test_sales)> insert into customers (customer_id, customer_name, customer_email) values (11,'Rory Brown','rory@123.com');
spark-sql (test_sales)> insert into customers (customer_id, customer_name, customer_email) values (12,'Jerry Washington','jerry@dt.com');
spark-sql (test_sales)> select * from customers;

[FANNG1]

Thanks for sharing your configuration!