How can I create trino / hive tables on top of existing HDFS parquet

[terranceuney]

Context

I have successfully tried the demo to configure (hdfs, hive, trino, superset, spark) using your stackable. Below are details:

• hdfscluster: up & running with kerberos authentication
• hivecluster (metastore): up & running, connected to hdfs with kerberos fine
• trino, superset, also up & running OK

When I define trino-catalog with iceberg connector. I am able to insert data into HDFS (using spark sql) I also able to query via superset & trino using your demo, ALL WORKING FINE

Problems

We have a lots of hdfs data (in parquet format). Before using your stackable, we're able to create superset dashboard by define (hive-metastore, trino, hive-server2) stack that:

• hive-metastore (with postgres): provide metastore for both trino & hive-server2
• hive-server2: we use this to create EXTERNAL TABLE with data on HDFS
• Trino / superset: query hdfs data using tables defained by hive-server2

If using your stackable, seem we cannot migrate our existing hdfs parquet hive table because:

• Seem you're only support hive-metastore, not hive-server2 itself as native query engine
• I tried iceberge trino-catalog connector but seem it does not allow to create table with existing hdfs parquet
• I also tried hive trino-catalog connector but when querying on superset (through trino) I got empty data

Below are my questions:

• Is there anyway to create trino tables using iceberg OR hive connector (both using hive metastore) on EXISTING hdfs parquet?

• Is there any way to integrate hive native with spark / trino so that I can create hive table using native hive query?

• I saw current spark-k8s-operator (24.11.1) only support hive 2.3.9 jars BUT current hive version is 3.1.2. If using spark + hive native, how can we upgrade those jars to 3.1.2?

Looking forward to your help on this, thanks a lot.

If possible, I can share you my trino.yaml, hdfs.yaml, hive.yaml & superset.yaml files