Releases: Aiven-Open/metadata-parser
Metadata parser capable of analysing sql in PG
Metadata parser has now the code (in src/pg.py
and src/sql.py
) to parse PG views and create a network graph of column dependencies. The work is based on sqllineage
With the new sql parser, the tool is able to link columns used in views definition to the original sources, giving a column level data lineage capability within PostgreSQL
Metadata parser goes modular
Global Improvements
The Metadata parser explore_services.py
has been divided into several modules, one per technology. Single tech parsers are available under the src
folder having the name of the related tech (e.g. kafka.py
for Apache Kafka®)
This makes contributing to the metadata parser (and understanding what a parser does) easier.
New parsed objects
Metadata parser now also parses:
- Kafka Topics partitions
- Kafka Topics consumer groups
Bug fixes
The code has also been cleaned, and several errors fixed.
The main error fixed was related to Opensearch sink connector not correctly parsing the Opensearch endpoint.