[Spark] Spark UC test abstraction to run test on different env.

[openinx]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

Related issue is: #5624

How was this patch tested?

If we want to do the locally test, which means we set up the local OSS uc server for the testing purpose, then we use this command to run:

./build/sbt -DsparkVersion=3.5.7 "testOnly io.sparkuctest.UCDeltaTableDMLTest"

If we want to run the same tests against the external UC server, then we can use this command:

export UC_URI=$UC_URI
export UC_CATALOG_NAME=main
export UC_SCHEMA_NAME=demo_zh
export UC_BASE_LOCATION=$S3_BASE_LOCATION
export UC_TOKEN=$UC_TOKEN
export UC_EXTENSION_CLASS=io.sparkuctest.extension.RemoteUCExtension
./build/sbt -DsparkVersion=3.5.7 "testOnly io.sparkuctest.UCDeltaTableDMLTest"

In summary, with this abstracted UnityCatalogExtension, we provided two extension implementations:

• LocalUCExtension, which means run all the tests locally.
• RemoteUCExtension, which means run the same tests remotelly.

And the good point is: for the new tests developers, they never need to perceive the underlying setup, all they need is extending the UCDeltaTableIntegrationBaseTest, and write the normal test codes.

Does this PR introduce any user-facing changes?

No

[tdas]

do we need both SQLExecutor interfacve and UCExtension interface? can we not abstract out into one? or if there are 2.. how do they compose together?

and what additional thing is capability is needed that SQLExecutor interface does not provide today?