Fix and update Etherlink demo (#923)

Author: droserasprout

CHANGELOG.md

@@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog], and this project adheres to [Semantic Versioning].
 
-## [Unreleased]
+## [7.3.0] - 2024-01-23
 
 ### Added
 
@@ -1289,7 +1289,8 @@ This release contains no changes except for the version number.
 [semantic versioning]: https://semver.org/spec/v2.0.0.html
 
 <!-- Versions -->
-[Unreleased]: https://github.com/dipdup-io/dipdup/compare/7.2.2...HEAD
+[Unreleased]: https://github.com/dipdup-io/dipdup/compare/7.3.0...HEAD
+[7.3.0]: https://github.com/dipdup-io/dipdup/compare/7.2.2...7.3.0
 [7.2.2]: https://github.com/dipdup-io/dipdup/compare/7.2.1...7.2.2
 [7.2.1]: https://github.com/dipdup-io/dipdup/compare/7.2.0...7.2.1
 [7.2.0]: https://github.com/dipdup-io/dipdup/compare/7.1.1...7.2.0

Makefile

@@ -1,12 +1,10 @@
-.ONESHELL:
 .PHONY: $(MAKECMDGOALS)
 MAKEFLAGS += --no-print-directory
 ##
 ##  🚧 DipDup developer tools
 ##
 PACKAGE=dipdup
 TAG=latest
-COMPOSE=deploy/compose.yaml
 SOURCE=src tests scripts
 DEMO=''
 FRONTEND_PATH=../interface

docs/1.getting-started/6.datasources.md

@@ -22,7 +22,7 @@ Index datasources, ones that can be attached to a specific index, are prefixed w
 
 ## Connection settings
 
-All datasources now share the same code under the hood to communicate with underlying APIs via HTTP. Their configs have an optional section `http` to configure connection settings. You can use it to set timeouts, retry policies, and other parameters.
+All datasources share the same code under the hood to communicate with underlying APIs via HTTP. Their configs have an optional section `http` to configure connection settings. You can use it to set timeouts, retry policies, and other parameters.
 
 Each datasource kind has its defaults. Usually, there's no reason to alter these settings unless you use self-hosted instances. In the example below, default values are shown:
 
@@ -38,6 +38,7 @@ datasources:
       ratelimit_sleep: 0.0
       connection_limit: 100
       connection_timeout: 60
+      request_timeout: 60
       batch_size: 10_000
       replay_path: None
       alias: None

docs/1.getting-started/7.indexes.md

@@ -9,8 +9,8 @@ Index is a primary DipDup entity connecting the inventory and data handling rule
 
 Multiple indexes are available for different workloads. Every index is linked to a specific datasource and provides a set of handlers for different kinds of data. Use this table to choose the right index for the task:
 
-| kind                                                                      | blockchain     | datasource | indexed data                |
-| ------------------------------------------------------------------------- | -------------- | ---------- | --------------------------- |
+| kind                                                                                   | blockchain     | datasource | indexed data                |
+| -------------------------------------------------------------------------------------- | -------------- | ---------- | --------------------------- |
 | [evm.subsquid.events](../2.indexes/1.evm_subsquid_events.md)                           | EVM-compatible | Subsquid   | event logs                  |
 | [tezos.tzkt.big_maps](../2.indexes/2.tezos_tzkt_big_maps.md)                           | Tezos          | TzKT       | big map diffs               |
 | [tezos.tzkt.events](../2.indexes/3.tezos_tzkt_events.md)                               | Tezos          | TzKT       | events                      |

docs/2.indexes/5.tezos_tzkt_operations.md

@@ -4,7 +4,7 @@ description: "Operation index allows you to query only operations related to you
 network: "tezos"
 ---
 
-<!-- markdownlint-disable first-line-h1 -->
+<!-- markdownlint-disable no-inline-html-->
 <!-- TODO: add images to the side <div class="float-img">
   <img src="../../assets/operation-bcd.png" alt="Operation group in BCD explorer">
   <img src="../../assets/operation-config.png" alt="`operation` index config">
@@ -16,10 +16,10 @@ Operation index allows you to query only operations related to your dapp and mat
 
 ## Handlers
 
-Each operation handler contains two required fields:
+Each operation handler config item contains two required fields:
 
-* `callback` — a name of async function with a particular signature; DipDup will search for it in `{{ project.package }}.handlers.<callback>` module.
-* `pattern` — a non-empty list of items that need to be matched.
+- `callback` — a name of async function with a particular signature; DipDup will search for it in `{{ project.package }}.handlers.<callback>` module.
+- `pattern` — a non-empty list of items that need to be matched.
 
 ```yaml [dipdup.yaml]
 indexes:
@@ -37,26 +37,47 @@ indexes:
 
 You can think of the operation pattern as a regular expression on a sequence of operations (both external and internal) with a global flag enabled (there can be multiple matches). Multiple operation parameters can be used for matching (source, destination, etc.).
 
-You will get slightly different callback argument types depending on whether pattern item is typed or not. If so, DipDup will generate the dataclass for a particular entrypoint/storage, otherwise you will have to handle untyped parameters/storage updates stored in `TzktOperationData` model.
+When the operation group matches the pattern, DipDup creates arguments for the callback function and invokes it.
 
-## Matching originations
+## Typed and untyped arguments
 
-| name                            | description                                                                                                                            | supported | typed |
-| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |:---------:|:-----:|
-| `originated_contract.address`   | Origination of exact contract.                                                                                                         |     ✅     |   ✅   |
-| `originated_contract.code_hash` | Originations of all contracts having the same code.                                                                                    |     ✅     |   ✅   |
-| `source.address`                | Special cases only. This filter is very slow and doesn't support strict typing. Usually, `originated_contract.code_hash` suits better. |     ⚠     |   ❌   |
-| `source.code_hash`              | Currently not supported.                                                                                                               |     ❌     |   ❌   |
+You will get slightly different callback argument types depending on whether the pattern item is typed or not. If so, DipDup will generate the dataclass for a particular entrypoint/storage, otherwise, you will have to handle untyped parameters/storage updates stored in `TzktOperationData` model.
 
-## Matching transactions
+## Applying filters
+
+In pattern items, you can specify filters to narrow down the search. For example, you can match only operations sent to a particular address. Each operation type has its own set of filters.
+
+### Originations
+
+| name                            | description                                                  | supported | typed |
+| ------------------------------- | ------------------------------------------------------------ |:---------:|:-----:|
+| `originated_contract.address`   | Origination of exact contract.                               |     🟢    |   🟢  |
+| `originated_contract.code_hash` | Originations of all contracts having the same code.          |     🟢    |   🟢  |
+| `source.address`                | Originations sent by address. Special cases only, see below. |     🟡    |   🔴  |
+| `source.code_hash`              | Not supported.                                               |     🔴    |   🔴  |
+
+Filtering originations by `source.address` is very slow and strict typing is not supported. For most cases `originated_contract.code_hash` suits better.
+
+### Transactions
 
 | name                     | description                                | supported | typed |
 | ------------------------ | ------------------------------------------ |:---------:|:-----:|
-| `source.address`         | Sent by exact address.                     |     ✅     |  N/A  |
-| `source.code_hash`       | Sent by any contract having this code hash |     ✅     |  N/A  |
-| `destination.address`    | Invoked contract address                   |     ✅     |   ✅   |
-| `destination.code_hash`  | Invoked contract code hash                 |     ✅     |   ✅   |
-| `destination.entrypoint` | Entrypoint called                          |     ✅     |   ✅   |
+| `source.address`         | Sent by exact address                      |     🟢    |   🔴  |
+| `source.code_hash`       | Sent by any contract having this code hash |     🟢    |   🔴  |
+| `destination.address`    | Invoked contract address                   |     🟢    |  🟢¹  |
+| `destination.code_hash`  | Invoked contract code hash                 |     🟢    |  🟢¹  |
+| `destination.entrypoint` | Entrypoint called                          |     🟢    |   🟢  |
+
+<sup>¹ when entrypoint is specified too</sup>
+
+### Smart rollup calls
+
+| name                    | description              | supported | typed |
+| ----------------------- | ------------------------ |:---------:|:-----:|
+| `source.address`        | Sent by exact address.   |     🟢    |   🔴  |
+| `source.code_hash`      | Not supported            |     🔴    |   🔴  |
+| `destination.address`   | Invoked contract address |     🟢    |   🔴  |
+| `destination.code_hash` | Not supported            |     🔴    |   🔴  |
 
 ## Optional items
 
@@ -65,15 +86,15 @@ Pattern items have `optional` field to continue matching even if this item is no
 ```yaml [dipdup.yaml]
 pattern:
   # Implicit transaction
-  - destination: some_contract
+  - type: transaction
+    destination: some_contract
     entrypoint: mint
-
-  # Internal transactions below
-  - destination: another_contract
+  # Internal transactions
+  - type: transaction
+    destination: another_contract
     entrypoint: transfer
-
-  - source: some_contract
-    type: transaction
+  - type: transaction
+    source: some_contract
 ```
 
 ## Specifying contracts to index

docs/9.release-notes/1.v7.3.md

@@ -5,8 +5,83 @@ description: DipDup 7.3 release notes
 
 # Release Notes: 7.3
 
-We're glad to announce DipDup 7.2! This release brings better support for Etherlink smart rollups and a bunch of minor improvements.
+We're glad to announce DipDup 7.3! This release brings better support for Etherlink smart rollups and a bunch of minor improvements.
 
 ## Indexing `sr_execute` operations on Tezos
 
+We continue to improve support for Etherlink smart rollups introduced in 7.2. In case you've missed it, Etherlink is an EVM-compatible smart rollup on Tezos, which prioritizes fair ordering of transactions, low latency and security. They are WASM applications, providing versatility in what language you write them in.
+
+Now you can index `sr_execute` operations with DipDup. This is a special operation created when rollup outbox messages are executed. You can filter them by `sr1` source/destination address. `demo_etherlink` demo project was updated to reflect these changes. It's an indexer for a bridge contract having two handlers, for depositing tokens to the rollup address (L1 -> L2) and for withdrawing them (L2 -> L1).
+
+```yaml [dipdup.yaml]
+spec_version: 2.0
+package: demo_etherlink
+
+datasources:
+  tzkt:
+    kind: tezos.tzkt
+    url: ${TZKT_URL:-https://api.nairobinet.tzkt.io}
+
+contracts:
+  ticketer:
+    kind: tezos
+    address: KT1PmYUomF3HDxsGWYQUCbLi2X8WvT7ZHv8o
+    typename: ticketer
+  ticket_helper:
+    kind: tezos
+    address: KT1TZg9EwGHKbfWvsHGsqBjm3J5NhJBtHPKX
+    typename: ticket_helper
+  rollup:
+    kind: tezos
+    address: sr1QgYF6ARMSLcWyAX4wFDrWFaZTyy4twbqe
+    typename: rollup
+
+indexes:
+  rollup_operations:
+    kind: tezos.tzkt.operations
+    datasource: tzkt
+    contracts:
+      - ticketer
+      - ticket_helper
+      - rollup
+    types:
+      - transaction
+      - sr_execute
+    handlers:
+      - callback: on_deposit
+        pattern:
+          - type: transaction
+            destination: ticketer
+            entrypoint: deposit
+          - type: transaction
+            destination: ticket_helper
+            entrypoint: default
+          - type: transaction
+            destination: rollup
+            entrypoint: default
+            alias: rollup_default
+      - callback: on_withdraw
+        pattern:
+          - type: sr_execute
+            destination: rollup
+          - type: transaction
+            destination: ticketer
+            entrypoint: withdraw
+          - type: transaction
+            source: ticketer
+            entrypoint: transfer
+```
+
+To use this demo as a template for your own Etherlink project, run `dipdup new` and choose `demo_etherlink` template.
+
+Read more about Etherlink and DipDup support for it:
+
+- [`tezos.tzkt.operations` index](../2.indexes/5.tezos_tzkt_operations.md)
+- [Etherlink docs](https://docs.etherlink.com/)
+- [Etherlink: Building The Most Decentralized EVM Layer 2 (On Tezos)](https://news.tezoscommons.org/etherlink-building-the-most-decentralized-evm-layer-2-on-tezos-1c749fb78d34).
+
 ## Fixes and improvements
+
+- abi.etherscan: Fixed handling "rate limit reached" errors.
+- cli: Fixed setting logger levels based on config and env variables.
+- http: Fixed incorrect number of retries performed on failed requests.

docs/9.release-notes/2.v7.2.md

@@ -11,7 +11,7 @@ We're glad to announce DipDup 7.2! This release brings Etherlink smart rollup co
 
 Etherlink is an EVM-compatible smart rollup, which prioritizes fair ordering of transactions, low latency and security.
 
-Smart rollups are Tezos’ permissionless L2 scaling solution. They are WASM applications, providing versatility in what language you write them in. You can learn more in [Etherlink docs](https://docs.etherlink.com/welcome/what-is-etherlink)
+Smart rollups are Tezos’ permissionless L2 scaling solution. They are WASM applications, providing versatility in what language you write them in. You can learn more in [Etherlink docs](https://docs.etherlink.com/)
 
 Since version 7.2 you can index Etherlink smart rollups like regular Tezos contracts. Choose `demo_etherlink` template when creating a new project:
 

pdm.lock

@@ -77,7 +77,7 @@ pytest-aiohttp==1.0.5
 pytest-asyncio==0.23.3
 pytest-cov==4.1.0
 pytest-xdist==3.5.0
-python-dotenv==1.0.0
+python-dotenv==1.0.1
 pytz==2023.3.post1
 pyunormalize==15.1.0
 pywin32==306; platform_system == "Windows" or sys_platform == "win32"

requirements.dev.txt

@@ -60,7 +60,7 @@ pydantic==1.10.14
 pyhumps==3.8.0
 pypika-tortoise==0.1.6
 pysignalr==0.2.0
-python-dotenv==1.0.0
+python-dotenv==1.0.1
 pytz==2023.3.post1
 pyunormalize==15.1.0
 pywin32==306; platform_system == "Windows"