7.3.0 release docs (#920)

Author: droserasprout

.github/workflows/docs.yml

@@ -46,6 +46,9 @@ jobs:
       - name: Install PDM
         run: pip install pdm
 
+      - name: Install markdownlint-cli
+        run: npm install -g markdownlint-cli
+
       - name: Install project
         run: pdm sync
 

CHANGELOG.md

@@ -121,7 +121,7 @@ The format is based on [Keep a Changelog], and this project adheres to [Semantic
 - cli: Import package submodules before starting indexing to fail early on import errors.
 - cli: Fixed ordering of crash reports in `report` group commands.
 - evm.node: Fixed parsing topics and integers in datasource models.
-- evm.subsquid.events: Fixed incorrrect log request parameters.
+- evm.subsquid.events: Fixed incorrect log request parameters.
 - install: Fixed issue with interpreting user answers in some cases.
 - tezos.tzkt: Fixed operation matching when contract code hash specified as a string.
 - tezos.tzkt: Fixed issue with processing rollbacks while sync is in progress.

Makefile

@@ -0,0 +1,83 @@
+.ONESHELL:
+.PHONY: $(MAKECMDGOALS)
+MAKEFLAGS += --no-print-directory
+##
+##  🚧 DipDup developer tools
+##
+PACKAGE=dipdup
+TAG=latest
+COMPOSE=deploy/compose.yaml
+SOURCE=src tests scripts
+DEMO=''
+FRONTEND_PATH=../interface
+
+
+help:           ## Show this help (default)
+	@grep -Fh "##" $(MAKEFILE_LIST) | grep -Fv grep -F | sed -e 's/\\$$//' | sed -e 's/##//'
+
+all:            ## Run an entire CI pipeline
+	make format lint test
+
+format:         ## Format with all tools
+	make black
+
+lint:           ## Lint with all tools
+	make ruff mypy
+
+test:           ## Run tests
+	pytest --cov-report=term-missing --cov=dipdup --cov-report=xml -n auto -s -v tests
+
+##
+
+black:          ## Format with black
+	black ${SOURCE}
+
+ruff:           ## Lint with ruff
+	ruff check --fix ${SOURCE}
+
+mypy:           ## Lint with mypy
+	mypy --no-incremental --exclude ${PACKAGE} ${SOURCE}
+
+##
+
+image:          ## Build Docker image
+	docker buildx build . -t ${PACKAGE}:${TAG}
+
+##
+
+demos:          ## Recreate demo projects from templates
+	python scripts/demos.py render ${DEMO}
+	python scripts/demos.py init ${DEMO}
+	pdm run format
+	pdm run lint
+
+docs_build:     ## Build docs
+	python scripts/docs.py check-links --source docs
+	python scripts/docs.py markdownlint
+	python scripts/docs.py dump-references
+	python scripts/docs.py dump-jsonschema
+	python scripts/docs.py build --source docs --destination ${FRONTEND_PATH}/content/docs
+
+docs_serve:     ## Build docs and start frontend server
+	python scripts/docs.py build --source docs --destination ${FRONTEND_PATH}/content/docs --watch --serve
+
+docs_watch:     ## Build docs and watch for changes
+	python scripts/docs.py build --source docs --destination ${FRONTEND_PATH}/content/docs --watch
+
+fixme:          ## Find FIXME and TODO comments
+	grep -r -e 'FIXME: ' -e 'TODO: ' -e 'type: ignore' -n src/dipdup --color
+
+update:         ## Update dependencies and dump requirements.txt
+	pdm update
+	pdm export --without-hashes -f requirements --prod -o requirements.txt
+	pdm export --without-hashes -f requirements --dev -o requirements.dev.txt
+
+before_release: ## Prepare for a new release after updating version in pyproject.toml
+	make format
+	make lint
+	make update
+	make demos
+	make test
+	make docs_build
+	echo "🎉 Commit changes, merge `aux/X.Y.Z`, run 'git checkout next && git pull && git tag X.Y.Z && git push origin X.Y.Z'" 
+##

README.md

@@ -1,5 +1,5 @@
 [![Twitter](https://badgen.net/badge/icon/dipdup_io?icon=twitter&label=)](https://twitter.com/dipdup_io)
-[![GitHub stars](https://img.shields.io/github/stars/dipdup-io/dipdup?color=2c2c2c)](https://github.com/dipdup-io/dipdup)
+[![GitHub stars](https://img.shields.io/github/stars/dipdup-io/dipdup?color=2c2c2c&style=plain)](https://github.com/dipdup-io/dipdup)
 [![PyPI monthly downloads](https://img.shields.io/pypi/dm/dipdup?color=2c2c2c)](https://pypi.org/project/dipdup/)
 [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/dipdup?color=2c2c2c)](https://www.python.org)
 [![License: MIT](https://img.shields.io/github/license/dipdup-io/dipdup?color=2c2c2c)](https://github.com/dipdup-io/dipdup/blob/next/LICENSE)

docs/1.getting-started/3.config.md

@@ -5,29 +5,31 @@ description: "Developing a DipDup indexer begins with creating a YAML config fil
 
 # Creating config
 
-Developing a DipDup indexer begins with creating a YAML config file of a specific format. DipDup expect it to be located in the project root and named `dipdup.yaml`. However, you can provide any path with a `-c` CLI option.
+Developing a DipDup indexer begins with creating a YAML config file of a specific format. DipDup expects it to be located in the project root and named `dipdup.yaml`. However, you can provide any path with a `-c` CLI option.
 
 ## Structure
 
 Config consists of multiple top-level mappings. In the table below they are grouped by sections for convenience, but in the config file, they can be in any order. The only required section is **Header**.
 
-|                  |                                                 |                                                                       |
-| ---------------- | ----------------------------------------------- | --------------------------------------------------------------------- |
-| **Header**       | [spec_version](../9.config/14.spec_version.md)* | DipDup project specification; currently "2.0"                         |
-|                  | [package](../9.config/11.package.md)*           | Python package name                                                   |
-| **Inventory**    | [database](../9.config/4.database.md)           | Database configuration                                                |
-|                  | [contracts](../9.config/2.contracts.md)         | Contracts that need to be indexed                                     |
-|                  | [datasources](../9.config/5.datasources.md)     | Datasources to gather information from                                |
-| **Indexes**      | [indexes](../9.config/8.indexes.md)             | Index definitions                                                     |
-|                  | [templates](../9.config/15.templates.md)        | Also index definitions, but with placeholders that make them reusable |
-| **Hooks**        | [hooks](../9.config/7.hooks.md)                 | Callbacks to run manually or by schedule                              |
-|                  | [jobs](../9.config/9.jobs.md)                   | Schedules for hooks                                                   |
-| **Integrations** | [hasura](../9.config/6.hasura.md)               | Hasura GraphQL Engine configuration                                   |
-|                  | [sentry](../9.config/13.sentry.md)              | Sentry configuration                                                  |
-|                  | [prometheus](../9.config/12.prometheus.md)      | Prometheus configuration                                              |
-| **Miscellaneous**| [advanced](../9.config/1.advanced.md)           | Tunables that affect framework behavior                               |
-|                  | [custom](../9.config/3.custom.md)               | Mapping of user-defined values; neither typed nor validated           |
-|                  | [logging](../9.config/10.logging.md)            | Configure logging verbosity                                           |
+See [Config reference guide](../7.references/2.config.md) for the full list of available options.
+
+|                   |                 |                                                                       |
+| ----------------- | --------------- | --------------------------------------------------------------------- |
+| **Header**        | `spec_version`* | DipDup project specification; currently "2.0"                         |
+|                   | `package`*      | Python package name                                                   |
+| **Inventory**     | `database`      | Database configuration                                                |
+|                   | `contracts`     | Contracts that need to be indexed                                     |
+|                   | `datasources`   | Datasources to gather information from                                |
+| **Indexes**       | `indexes`       | Index definitions                                                     |
+|                   | `templates`     | Also index definitions, but with placeholders that make them reusable |
+| **Hooks**         | `hooks`         | Callbacks to run manually or by schedule                              |
+|                   | `jobs`          | Schedules for hooks                                                   |
+| **Integrations**  | `hasura`        | Hasura GraphQL Engine configuration                                   |
+|                   | `sentry`        | Sentry configuration                                                  |
+|                   | `prometheus`    | Prometheus configuration                                              |
+| **Miscellaneous** | `advanced`      | Tunables that affect framework behavior                               |
+|                   | `custom`        | Mapping of user-defined values; neither typed nor validated           |
+|                   | `logging`       | Configure logging verbosity                                           |
 
 ## Merging multiple files
 
@@ -80,4 +82,4 @@ contracts:
     typename: quipu_fa12
 ```
 
-If multiple contracts you index have the same interface but different code, see [F.A.Q.](../12.faq.md) learn how to avoid conflicts.
+If multiple contracts you index have the same interface but different code, see [F.A.Q.](../12.faq.md) to learn how to avoid conflicts.

docs/1.getting-started/4.package.md

@@ -58,7 +58,7 @@ hooks:
 
 After running the `init` command, you'll get the following directory tree (shortened for brevity)
 
-```
+```text
 {{ project.package }}
 ├── hooks
 │   ├── backup

docs/1.getting-started/6.datasources.md

@@ -9,16 +9,16 @@ Datasources are DipDup connectors to various APIs. They are defined in config an
 
 Index datasources, ones that can be attached to a specific index, are prefixed with blockchain name, e.g. `tezos.tzkt` or `evm.subsquid`.
 
-| kind                                      | blockchain     | description                     |
-| ----------------------------------------- | -------------- | ------------------------------- |
-| [abi.etherscan](2.abi_etherscan.md)       | EVM-compatible | Provides ABIs for EVM contracts |
-| [coinbase](3.coinbase.md)                 | any            | Coinbase price feed             |
-| [evm.node](4.evm_node.md)                 | EVM-compatible | Ethereum node                   |
-| [evm.subsquid](5.evm_subsquid.md)         | EVM-compatible | Subsquid Archives node                   |
-| [http](6.http.md)                         | any            | Generic HTTP API                        |
-| [ipfs](7.ipfs.md)                         | any            | IPFS gateway                            |
-| [tezos.tzkt](8.tezos_tzkt.md)             | Tezos          | TzKT API                        |
-| [tezos.tzip_metadata](9.tzip_metadata.md) | Tezos          | TZIP-16 metadata                |
+| kind                                                       | blockchain     | description                     |
+| ---------------------------------------------------------- | -------------- | ------------------------------- |
+| [abi.etherscan](../3.datasources/1.abi_etherscan.md)       | EVM-compatible | Provides ABIs for EVM contracts |
+| [coinbase](../3.datasources/2.coinbase.md)                 | any            | Coinbase price feed             |
+| [evm.node](../3.datasources/3.evm_node.md)                 | EVM-compatible | Ethereum node                   |
+| [evm.subsquid](../3.datasources/4.evm_subsquid.md)         | EVM-compatible | Subsquid Archives node          |
+| [http](../3.datasources/5.http.md)                         | any            | Generic HTTP API                |
+| [ipfs](../3.datasources/6.ipfs.md)                         | any            | IPFS gateway                    |
+| [tezos.tzkt](../3.datasources/7.tezos_tzkt.md)             | Tezos          | TzKT API                        |
+| [tezos.tzip_metadata](../3.datasources/8.tzip_metadata.md) | Tezos          | TZIP-16 metadata                |
 
 ## Connection settings
 

docs/1.getting-started/7.indexes.md

@@ -42,7 +42,7 @@ indexes:
       datasource: tzkt_mainnet
 ```
 
-You can also spawn indexes from templates in runtime; see [Variables](../1.getting-started/7.env-variables.md) page.
+You can also spawn indexes from templates in runtime; see [Spawning in runtime](../1.getting-started/7.indexes.md#spawning-in-runtime).
 
 ## Limiting scope
 

docs/12.faq.md

@@ -4,6 +4,8 @@ description: "This page contains answers to the most popular questions about Dip
 nested: Resources
 ---
 
+<!-- markdownlint-disable no-trailing-punctuation -->
+
 # F.A.Q.
 
 This page contains answers to the most popular questions about DipDup guts. If you landed here - congrats, you're writing a pretty advanced indexer!
@@ -128,6 +130,6 @@ advanced:
 
 Don't forget to reindex after this change. When decimal context precision is adjusted you'll get a warning in the logs.
 
-```
+```text
 WARNING  dipdup.database      Decimal context precision has been updated: 28 -> 128
 ```

docs/13.troubleshooting.md

@@ -61,7 +61,6 @@ dipdup -c . -c configs/dipdup.sqlite.yaml config export
 
 ### Check env variables
 
-
 `config env` command can help you to find missing environment variables.
 
 ```shell [Terminal]