Skip to content

feat: add support for scenario testing #660

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Jul 29, 2022
Merged

feat: add support for scenario testing #660

merged 14 commits into from
Jul 29, 2022

Conversation

tmigone
Copy link
Member

@tmigone tmigone commented Jul 28, 2022

Motivation

This PR adds support for e2e "scenario" testing. Scenarios are special tests that:

  • Test the execution of common protocol actions.
  • Can be run against any network at any time and the tests should pass.
  • Can perform read and write interactions with the blockchain. Requires an account with sufficient balance!
  • Example: a test validating that a user can add signal to a subgraph.

Scenarios are defined by an optional script and a test file:

  • Optional ts script
    • The objective of this script is to perform actions on the protocol to advance it's state to the desired one.
    • Should follow hardhat script convention.
    • Should be named e2e/scenarios/{scenario-name}.ts.
    • They run before the test file.
  • Test file
    • Should be named e2e/scenarios/{scenario-name}.test.ts.
    • Standard chai/mocha/hardhat/ethers test file.

Usage

The following command will run a scenario against the provided network:

# Run a single scenario
npx hardhat e2e:scenario <scenario-name> --network <network> --graph-config config/config.<network>.yml

Scenarios are also run as part of the e2e test suite, in this case scenarios are run one by one, meaning the script for a scenario is run followed by it's tests and only then the next scenario is run. This is to preserve the state between scenario runs as we don't know how the actions in one scenario can affect the next one.

# Run all e2e tests, including all scenarios
npx hardhat e2e --network <network> --graph-config config/config.<network>.yml

Finally, the scenario scripts can also be ran independently from the tests as they are regular hardhat scripts:

# Run only the script
npx hardhat run e2e/scenario/<scenario-name>.ts --network <network> --graph-config config/config.<network>.yml

Note that in all cases the protocol needs to be deployed and setup on the target network beforehand. Alternatively, you can run yarn test:e2e which will deploy a local network and run all e2e tests.

Changes

  • Add e2e:scenario hardhat task to run a specific scenario
  • Modify e2e hardhat task to run all scenarios as part of the test suite
  • Add WIP "scenario1" (need better name), which corresponds to step 1.2 on L2 Testnet Validation plan.

@tmigone tmigone marked this pull request as draft July 28, 2022 14:36
@tmigone tmigone force-pushed the tmigone/scenarios-demo branch from 65c8aa2 to 1e4e888 Compare July 29, 2022 09:09
Signed-off-by: Tomás Migone <[email protected]>
@codecov
Copy link

codecov bot commented Jul 29, 2022

Codecov Report

Merging #660 (91b467b) into dev (358a8ec) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev     #660   +/-   ##
=======================================
  Coverage   90.35%   90.35%           
=======================================
  Files          35       35           
  Lines        1762     1762           
  Branches      296      296           
=======================================
  Hits         1592     1592           
  Misses        170      170           
Flag Coverage Δ
unittests 90.35% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 358a8ec...91b467b. Read the comment docs.

@tmigone tmigone marked this pull request as ready for review July 29, 2022 09:43
@tmigone tmigone changed the title Tmigone/scenarios demo feat: add support for scenario testing Jul 29, 2022
@tmigone tmigone requested a review from abarmat July 29, 2022 09:53
@tmigone tmigone merged commit 7df6669 into dev Jul 29, 2022
@tmigone tmigone deleted the tmigone/scenarios-demo branch July 29, 2022 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants