Bump version 7.5.2 (#970)

Author: droserasprout

CHANGELOG.md

@@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog], and this project adheres to [Semantic Versioning].
 
-## [Unreleased]
+## [7.5.2] - 2024-03-20
 
 ### Fixed
 
@@ -1391,7 +1391,8 @@ This release contains no changes except for the version number.
 [semantic versioning]: https://semver.org/spec/v2.0.0.html
 
 <!-- Versions -->
-[Unreleased]: https://github.com/dipdup-io/dipdup/compare/7.5.1...HEAD
+[Unreleased]: https://github.com/dipdup-io/dipdup/compare/7.5.2...HEAD
+[7.5.2]: https://github.com/dipdup-io/dipdup/compare/7.5.1...7.5.2
 [7.5.1]: https://github.com/dipdup-io/dipdup/compare/7.5.0...7.5.1
 [7.5.0]: https://github.com/dipdup-io/dipdup/compare/7.4.0...7.5.0
 [6.5.16]: https://github.com/dipdup-io/dipdup/compare/6.5.15...6.5.16

Makefile

@@ -56,6 +56,7 @@ docs_build:     ## Build docs
 	python scripts/docs.py dump-references
 	python scripts/docs.py dump-demos
 	python scripts/docs.py dump-jsonschema
+	python scripts/docs.py merge-changelog
 	python scripts/docs.py markdownlint
 	python scripts/docs.py build --source docs --destination ${FRONTEND_PATH}/content/docs
 

docs/9.release-notes/1.v7.5.md

@@ -17,29 +17,5 @@ The Hasura adapter now supports the `bulk` request type to apply table customiza
 
 Finally, DipDup 6.5, the stable release branch, has reached end-of-life. 6.5.16 is the last release in this branch. Please, follow the [7.0 Migration Guide](../9.release-notes/6.v7.0.md#migration-guide) to upgrade to the latest version.
 
-## Changes since 7.4
-
-### Added
-
-- config: Added `http.polling_interval` option to set the interval between polling requests (some datasources).
-- hasura: Allow `bulk` request type in custom metadata files.
-
-### Fixed
-
-- abi.etherscan: Raise `AbiNotAvailableError` when contract is not verified.
-- cli: Fixed incorrect indexer status logging.
-- evm.node: Fixed default ratelimit sleep time being too high.
-- evm.node: Fixed memory leak when using realtime subscriptions.
-- evm.node: Fixed processing chain reorgs.
-- evm.node: Respect `http.batch_size` when fetching block headers.
-- evm.subsquid.transactions: Fixed issue with `node_only` flag ignored.
-
-### Performance
-
-- evm.subsquid: Dynamically adjust the batch size when syncing with node.
-- hasura: Apply table customizations in a single request.
-- performance: Collect hit/miss stats for cached models.
-- performance: Decrease main loop and node polling intervals.
-- performance: Drop caches when all indexes have reached realtime.
-
+{{ #include 9.release-notes/_7.5_changelog.md }}
 {{ #include 9.release-notes/_footer.md }}

docs/9.release-notes/2.v7.4.md

@@ -27,17 +27,5 @@ indexes:
     first_level: 4634748
 ```
 
-## Changes since 7.3
-
-### Added
-
-- cli: Added `--template` option to `new` command to skip template selection.
-- evm.subsquid.transactions: Added `evm.subsquid.transactions` index kind to process EVM transactions.
-
-### Fixed
-
-- cli: Fixed crash when running `init` command with a config outside of the project directory.
-- codegen: Don't create intermediate `events.json` file in ABI directory.
-- evm.subsquid: When request to worker fails, ask router for another one instead of retrying the same worker.
-
+{{ #include 9.release-notes/_7.4_changelog.md }}
 {{ #include 9.release-notes/_footer.md }}

docs/9.release-notes/3.v7.3.md

@@ -80,27 +80,5 @@ Read more about Etherlink and DipDup support for it:
 - [Etherlink docs](https://docs.etherlink.com/)
 - [Etherlink: Building The Most Decentralized EVM Layer 2 (On Tezos)](https://news.tezoscommons.org/etherlink-building-the-most-decentralized-evm-layer-2-on-tezos-1c749fb78d34).
 
-## Changes since 7.2
-
-### Added
-
-- env: Added `DIPDUP_NO_VERSION_CHECK` and `DIPDUP_NO_SYMLINK` variables.
-- tezos.tzkt.operations: Added new operation type `sr_execute` for Etherlink smart rollups.
-
-### Fixed
-
-- abi.etherscan: Fixed handling "rate limit reached" errors.
-- cli: Do not consider config as oneshot if `tezos.tzkt.head` index is present.
-- cli: Fixed setting logger levels based on config and env variables.
-- codegen: Allow dots to be used in typenames indicating nested packages.
-- codegen: Always cleanup jsonschemas before generating types.
-- config: Make `ws_url` field optional for `evm.node` datasource.
-- evm.node: Make `withdrawals_root` field optional in `EvmNodeHeadData` model.
-- http: Fixed crash on some datasource URLs.
-- http: Fixed incorrect number of retries performed on failed requests.
-
-### Performance
-
-- evm.subsquid.events: Increase indexing speed when using EVM node.
-
+{{ #include 9.release-notes/_7.3_changelog.md }}
 {{ #include 9.release-notes/_footer.md }}

docs/9.release-notes/4.v7.2.md

@@ -93,37 +93,5 @@ Indexing EVM events with node RPC (`evm.node` datasource) has become faster. Nod
 
 Finally, we have sped up the generation of Pydantic typeclasses with `datamodel-codegen`. Also, resulting classes use `X | Y` union syntax when possible.
 
-## Changes since 7.1
-
-### Added
-
-- api: Added HTTP API to manage a running indexer.
-- cli: Added `DIPDUP_CONFIG` and `DIPDUP_ENV_FILE` environment variables corresponding to `--config` and `--env-file` options.
-- config: Added `http.request_timeout` option to set the total timeout for HTTP requests.
-- evm.subsquid: Added Prometheus metrics required for Subsquid Cloud deployments.
-- project: Added Makefile to the default project template (only for new projects).
-- project: Added optional `package_manager` field to replay config.
-- tezos.tzkt: Added support for Etherlink smart rollups (`sr1…` addresses).
-
-### Fixed
-
-- cli: Don't suppress uncaught exceptions when performance monitoring is disabled.
-- codegen: Use datamodel-code-generator from the project's virtualenv.
-- evm.node: Fixed an issue with realtime subscriptions which led to indexing being stuck in some cases.
-- evm.node: Fixed crash on anonymous event logs during the last mile indexing.
-- evm.node: Raise an exception when no realtime messages have been received in `http.connection_timeout` seconds.
-- evm.subsquid: Last mile indexing is significantly faster now.
-- http: Use `request_timeout` instead of `connection_timeout` for total timeout.
-- install: Don't install datamodel-code-generator as a CLI tool.
-- install: Respect package manager if specified in pyproject.toml.
-- tezos.tzkt: Fixed an issue with approving schema after reindexing triggered by rollback.
-
-### Performance
-
-- evm.subsquid.events: Request logs in batches to speed up the last mile indexing.
-
-### Security
-
-- deps: Updated PyArrow to 14.0.1 to fix [CVE-2023-47248](https://github.com/advisories/GHSA-5wvp-7f3h-6wmm)
-
+{{ #include 9.release-notes/_7.1_changelog.md }}
 {{ #include 9.release-notes/_footer.md }}

docs/9.release-notes/5.v7.1.md

@@ -154,20 +154,5 @@ jobs:
 
 That's all for today. Thanks for reading!
 
-## Changes since 7.0
-
-### Added
-
-- cli: Added `--unsafe`, `--compose`, `--internal` flags to `config env` command.
-- cli: Added missing short equivalents for options in some commands.
-- cli: Relative paths to be initialized now can be passed to the `init` command as arguments.
-- tezos.tzkt.token_balances: Added new index.
-
-### Fixed
-
-- cli: Fixed `DIPDUP_DEBUG` not being applied to the package logger.
-- cli: Fixed crash on early Python 3.11 releases.
-- project: Update default Docker tag for TimescaleDB HA.
-- tezos.tzkt.token_transfers: Fixed filtering transfers by token_id.
-
+{{ #include 9.release-notes/_7.1_changelog.md }}
 {{ #include 9.release-notes/_footer.md }}

docs/9.release-notes/_7.0_changelog.md

@@ -0,0 +1,101 @@
+<!-- markdownlint-disable first-line-h1 -->
+## Changes since 7.-1
+
+### Added
+
+- abi.etherscan: Added `abi.etherscan` datasource to fetch ABIs from Etherscan.
+- api: Added `/performance` endpoint to request indexing stats.
+- cli: Added `report` command group to manage performance and crash reports created by DipDup.
+- config: Added `advanced.api` section to configure monitoring API exposed by DipDup.
+- config: Added `advanced.decimal_precision` field to overwrite precision if it's not guessed correctly based on project models.
+- config: Added `advanced.metrics` field to configure amount of gathered metrics.
+- config: Added `advanced.unsafe_sqlite` field to disable journaling and data integrity checks.
+- config: Added `http.alias` field to overwrite alias of datasource HTTP gateway.
+- database: Added `dipdup_meta` immune table to store arbitrary JSON values.
+- database: Added `dipdup_wipe` and `dipdup_approve` SQL functions to the schema.
+- database: Added experimental support for immune tables in SQLite.
+- env: Added `DIPDUP_DEBUG` environment variable to enable debug logging.
+- evm.node: Added `evm.node` datasource to receive events from Ethereum node and use web3 API.
+- evm.subsquid.events: Added `evm.subsquid.events` index to process event logs from Subsquid Archives.
+- evm.subsquid: Added `evm.subsquid` datasource to fetch historical data from Subsquid Archives.
+- models: Added optional `maxsize` meta field to `CachedModel` to limit the LRU cache size.
+
+### Fixed
+
+- ci: Fixed dipdup package metadata.
+- cli: Fixed `config export --full` command showing original config.
+- cli: Fixed `schema wipe` command crash due to `dipdup_meta` table being always immune.
+- cli: Fixed `schema wipe` command for SQLite databases.
+- cli: Fixed ordering of crash reports in `report` group commands.
+- cli: Generate base template from replay only when --base flag is set.
+- cli: Import package submodules before starting indexing to fail early on import errors.
+- cli: Keep the last 100 reports only.
+- cli: Remove cached jsonschemas when calling init --force.
+- cli: Use correct data path with timescaledb-ha Docker image.
+- codegen: Filter jsonschemas by prefixes supported by code generator.
+- config: Don't create empty SentryConfig if DSN is not set.
+- context: Share internal state between context instances.
+- database: Allow running project scripts and queries on SQLite.
+- database: Don't cleanup model updates on every loop.
+- database: Fixed `OperationalError` raised in some cases after calling `bulk_create`.
+- demos: Don't include database config in root config.
+- demos: Fixed decimal overflow in `demo_uniswap` project.
+- evm.node: Fixed incorrect log request parameters.
+- evm.node: Fixed keepalive loop for websocket connection.
+- evm.node: Fixed parsing empty realtime message payloads.
+- evm.node: Fixed parsing topics and integers in datasource models.
+- evm.node: Set `timestamp` field to the block timestamp.
+- evm.subsquid.events: Fixed incorrect log request parameters.
+- evm.subsquid.events: Fixed issue with determining the last level when syncing with node.
+- evm.subsquid.events: Sync to `last_level` if specified in config.
+- evm.subsquid: Create a separate aiohttp session for each worker.
+- hasura: Increated retry count for initial connection (healthcheck).
+- index: Fixed crash when parsing typed transactions with empty parameter.
+- index: Remove Python limitation on large int<->str conversions.
+- install: Fixed issue with interpreting user answers in some cases.
+- jobs: Don't add jobs before scheduler is started.
+- package: Create jsonschemas directory if not exists.
+- package: Create missing files from project base on init.
+- package: Create package symlink only when needed.
+- package: Don't create empty pyproject.toml during init.
+- package: Fixed discovery of the package when workdir is project root.
+- package: Fixed mypy command in default template.
+- package: Fixed package detection for poetry managed projects.
+- package: Update replay.yaml on init.
+- tezos.tzkt.events: Fixed parsing contract event data.
+- tezos.tzkt.operations: Fixed parsing operations with empty parameters.
+- tezos.tzkt: Fixed issue with processing rollbacks while sync is in progress.
+- tezos.tzkt: Fixed operation matching when contract code hash specified as a string.
+- tezos.tzkt: Fixed regression in `get_transactions` method pagination.
+
+### Changed
+
+- ci: Docker images are now based on Debian 12.
+- cli: Commands to manage local dipdup installation moved to the `self` group.
+- cli: `config env --file` option renamed to `--output`.
+- cli: `init --overwrite-types` flag renamed to `--force` and now also affects ABIs.
+- cli: `report` command renamed to `report ls`.
+- config: `advanced.rollback_depth` value set based on indexes used in the project if not set explicitly.
+- config: `logging` field now can contain either loglevel or name-loglevel mapping.
+- context: Signature of `add_contract` method has changed.
+- database: Querysets are no longer copied between chained method calls (`.filter().order_by().limit()`)
+- database: Store datasource aliases instead of URLs in `dipdup_head` table.
+- database: `EnumField` now uses `TEXT` type instead of `VARCHAR(n)`.
+- models: User models must use field classes from `dipdup.fields` module instead of `tortoise.fields`.
+- tezos.tzkt.head: Replaced `handlers` section with a single `callback` field in config.
+- tezos.tzkt: Signatures of `[get/iter]_similar_contracts` and `[get/iter]_originated_contracts` methods have changed.
+
+### Removed
+
+- ci: Docker images no longer contain git, poetry and custom scripts.
+- ci: `-slim` and `-pytezos` Docker images are no longer published.
+- cli: Removed `dipdup-install` alias to `dipdup.install`.
+- cli: Removed `status` command.
+- config: Removed `# dipdup: ignore` hint used to ignore typeclass during init.
+- config: Removed `advanced.metadata_interface` flag (always enabled).
+- config: Removed `similar_to` filter of `operation` index pattern.
+- sentry: Removed `crash_reporting` flag and built-in DSN.
+
+### Other
+
+- tzkt: Request plain values instead of mappings from TzKT when possible.

docs/9.release-notes/_7.1_changelog.md

@@ -0,0 +1,16 @@
+<!-- markdownlint-disable first-line-h1 -->
+## Changes since 7.0
+
+### Added
+
+- cli: Added `--unsafe`, `--compose`, `--internal` flags to `config env` command.
+- cli: Added missing short equivalents for options in some commands.
+- cli: Relative paths to be initialized now can be passed to the `init` command as arguments.
+- tezos.tzkt.token_balances: Added new index.
+
+### Fixed
+
+- cli: Fixed `DIPDUP_DEBUG` not being applied to the package logger.
+- cli: Fixed crash on early Python 3.11 releases.
+- project: Update default Docker tag for TimescaleDB HA.
+- tezos.tzkt.token_transfers: Fixed filtering transfers by token_id.

docs/9.release-notes/_7.2_changelog.md

@@ -0,0 +1,33 @@
+<!-- markdownlint-disable first-line-h1 -->
+## Changes since 7.1
+
+### Added
+
+- api: Added HTTP API to manage a running indexer.
+- cli: Added `DIPDUP_CONFIG` and `DIPDUP_ENV_FILE` environment variables corresponding to `--config` and `--env-file` options.
+- config: Added `http.request_timeout` option to set the total timeout for HTTP requests.
+- evm.subsquid: Added Prometheus metrics required for Subsquid Cloud deployments.
+- project: Added Makefile to the default project template (only for new projects).
+- project: Added optional `package_manager` field to replay config.
+- tezos.tzkt: Added support for Etherlink smart rollups (`sr1…` addresses).
+
+### Fixed
+
+- cli: Don't suppress uncaught exceptions when performance monitoring is disabled.
+- codegen: Use datamodel-code-generator from the project's virtualenv.
+- evm.node: Fixed an issue with realtime subscriptions which led to indexing being stuck in some cases.
+- evm.node: Fixed crash on anonymous event logs during the last mile indexing.
+- evm.node: Raise an exception when no realtime messages have been received in `http.connection_timeout` seconds.
+- evm.subsquid: Last mile indexing is significantly faster now.
+- http: Use `request_timeout` instead of `connection_timeout` for total timeout.
+- install: Don't install datamodel-code-generator as a CLI tool.
+- install: Respect package manager if specified in pyproject.toml.
+- tezos.tzkt: Fixed an issue with approving schema after reindexing triggered by rollback.
+
+### Performance
+
+- evm.subsquid.events: Request logs in batches to speed up the last mile indexing.
+
+### Security
+
+- deps: Updated PyArrow to 14.0.1 to fix [CVE-2023-47248](https://github.com/advisories/GHSA-5wvp-7f3h-6wmm)