Skip to content

Releases: Aiven-Open/metadata-parser

Metadata parser capable of analysing sql in PG

18 Oct 07:22
09d3bf8
Compare
Choose a tag to compare

Metadata parser has now the code (in src/pg.py and src/sql.py) to parse PG views and create a network graph of column dependencies. The work is based on sqllineage

With the new sql parser, the tool is able to link columns used in views definition to the original sources, giving a column level data lineage capability within PostgreSQL

Metadata parser goes modular

14 Oct 08:08
1ee5d43
Compare
Choose a tag to compare

Global Improvements

The Metadata parser explore_services.py has been divided into several modules, one per technology. Single tech parsers are available under the src folder having the name of the related tech (e.g. kafka.py for Apache Kafka®)

This makes contributing to the metadata parser (and understanding what a parser does) easier.

New parsed objects

Metadata parser now also parses:

  • Kafka Topics partitions
  • Kafka Topics consumer groups

Bug fixes

The code has also been cleaned, and several errors fixed.
The main error fixed was related to Opensearch sink connector not correctly parsing the Opensearch endpoint.