Testing different synchronizers

[tclemos]

The current zkEVM-Node architecture allows different types of Synchronizers depending on the node type which is running, basically, we can have a Trusted Sequencer Node, a Permissionless Sequencer Node, and a Read-only Node. We want to make sure all the types of Synchronizers are tested properly since the synchronization processes are different for each type.

Background

Before we start discussing how to test the different Synchronizers, and even though we understand the forced batch is a very important feature that affects directly the Synchronizers, we decided to not take this topic(forced batch) into consideration now to make it easier to create the test strategy focusing in the Synchronizers based on the Trusted Sequencer Node being the only one in the networks affecting the state.

Firstly let's recap the known node types we have, so we can understand more about them and properly think about a solution for the synchronization tests.

When a zkEVM-Node is set up as a Trusted Sequencer Node, this is the node responsible for managing the Trusted State, also this node is the one that decides when the Trusted State should be Virtualized in order to allow an Aggregator to Consolidate the state further. The Synchronization process of this type of node requires reading the already Virtualized State from L1 and checking it against the Trusted State in order to handle reorgs and forced batches properly.

When a zkEVM-Node is set up as a Permissionless Sequencer Node or Read-only Node, they will synchronize information from both L1 and the Trusted Sequencer Node - Broadcast Service, which is the component responsible for providing information about the Trusted State. So, when the node is behind the Virtualized State it synchronizes the information from the L1, when the Virtualized State is fully synchronized, it starts to synchronize information from the Trusted Sequencer Node via the `Broadcast Service.

Well, now that the node types and how they synchronize were clarified, I want to explain what are the things we want to test related to these types of Synchronizers, so we must make sure that:

• Data synchronized from L1 has priority in order to the data stored as the Trusted State.
• Any change to the Trusted State in the Trusted Sequencer Node should reflect in all nodes
• Continuity of the blockchain must be validated, we must make sure the blockchain will not break after a certain amount of blocks or after a specific accumulative scenario.

Now we need to talk about the types of tests we think we can use to solve the problem

Smoke tests generally are handled manually and have a focus to test the core functionalities of a system, just to make sure the very happy paths are still working and the system has the minimum capacity to work with the new version being released. A basic use case is a tester person running the system in a controlled environment and using it as a regular system user, trying to perform the most important operations in the system.

Unit Tests are meant to test a single layer and instead of integrating with a real dependency component, it use mocks to fake the integration layer. A basic use case is a component depending on an external API, where a fake HTTP response ir returned.

Integrated Tests generally tend to validate an integration point even though the whole process can be a bigger chain of events. A basic use case is a component depending on a database, with integration tests you can also test the SQL commands running it against a real SQL database.

E2E Tests These tests tend to validate a workflow from the beginning until the end, making sure all the real components required by the workflow are called, but generally the call is made in a fully controlled environment and a proposed scenario is prepared before running the expected call to get the expected response. A basic use case for it is testing a system that needs to call an external API and also save data to a database and after that send an e-mail with confirmation to the user.

Continuity Tests try to guarantee the system will continue to run over time, no matter what happens to the data inserted into it, it tries to explore failures due to the accumulation of data, resource usage, and limitations and checks against date and time. A basic use case is testing the sequencer of blocks in a blockchain that has the relation between blocks and each block on the chain depending on the previous one, testing the block reorgs, different sizes of blocks, and the behavior of the blockchain as it grows in size of blocks.

Proposal

We can have different tests for all the different types of Synchronizers, but IMO opinion the Continuity Tests in this case are the ones we want to have because the synchronization process is something that we need to check growing the blockchain with different use cases along the journey to make sure the state is synchronized properly on all types of Synchronizers

The idea is to create a robot that will keep executing actions periodically on the L2 blockchain, performing user operations like sending multiple transactions, getting information from the blockchain, and also checking it against the information in the blockchain from the point of view of each different node.

Theoretically, we should have all the types of the Synchronizers running in isolated environments to mimic a realistic scenario about different blockchain consumers in different places and make sure they are all available to be consumed by the robot in order to check the data.

In other words, we should run some nodes on AWS to be the different types of nodes we want to run, they must not know about each other unless is it strictly necessary, theoretically, the protocol is the one that will handle the passive communication between them via the Synchronizers.

[arnaubennassar]

About the proposal:

• We can split responsibility of making the network to receive more txs and checking that the synchronization is working. As an starting point, we could assume that users will send txs so we can completely omit the part of automating tx sending
• Could you elaborate further how each component will be tested using this strategy?
• Beyond the continuity test, is there any other test that we should think of?

[arnaubennassar]

I think that beyond continuity testing we should think about specific pain points such as:

• L1 reorgs
• L2 reorgs
• encoding/decoding of batch data

[tclemos]

For L1 and L2 reorgs I recommend testing via Integration Tests running a local L1 network and DB.

We can force the reorg to happen in the L1 network, so this is going to be transparent to the Node not forcing the responses we get from Geth client in this case, so we can make sure we are able to handle the responses properly when a reorg happens.

For encoding/decoding the Unit test seems to be the simplest choice, they can get outdate if the Executor has changed, but we can also test it via Integration Tests by running a local Executor.

[cool-develope]

Mostly, we are using unit test, integrated test and e2e tests right now. There would be no problem in unit tests. The only problem is a e2e test.
I believe the type of Synchronizers should be managed by config. We can setup different scenarios using script and workflow.