This repository is about building a generic, extensible Sink Connector for Iceberg from different Input sources, based on Spark (Structured Streaming) 3.5. This project is under active development.
For Testing, the directory testsuite can be utilized
Todo:
- Complete Config and ConfigLoader
- Test entrypoint.py
- Change spark-warehouse in session creation as well as bootstrap servers of kafka in testsuite