Skip to content

feat(specs,tests): exception_test marker #1436

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Apr 12, 2025
Merged

Conversation

marioevz
Copy link
Member

@marioevz marioevz commented Apr 10, 2025

🗒️ Description

exception_test marker

Introduce a new marker to mark tests that produce an invalid transaction or block.

@pytest.mark.exception_test

Test filling now fails if this marker is not applied and the test contains either an invalid transaction or an invalid block.

This PR also marks all existing negative tests.

Make request a PrivateAttr in BaseTest

The pytest fixture request is now a private attribute of BaseTest and a new class method from_test is introduced as a generic to automatically generate a test type from an instance of another test type, which BaseTest attributes automatically.

Also request has no longer to be passed to generate method, so it has been removed from all unit tests.

🔗 Related Issues

None

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
  • Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
  • Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

@marioevz marioevz added scope:tests Scope: Changes EL client test cases in `./tests` type:feat type: Feature scope:fw Scope: Framework (evm|tools|forks|pytest) labels Apr 10, 2025
@danceratopz
Copy link
Member

negative has quite a broad connotation. How about something slightly more specific, like:

  1. @pytest.mark.invalid # also quite broad
  2. @pytest.mark.reject_execution
  3. @pytest.mark.exception_test
  4. @pytest.mark.must_raise

@marioevz marioevz changed the title feat(specs,tests): Negative test marker feat(specs,tests): exception_test marker Apr 10, 2025
@marioevz
Copy link
Member Author

negative has quite a broad connotation. How about something slightly more specific, like:

1. `@pytest.mark.invalid`  # also quite broad
2. `@pytest.mark.reject_execution`
3. `@pytest.mark.exception_test`
4. `@pytest.mark.must_raise`

Selected @pytest.mark.exception_test in the end, thanks for the suggestion :)

Copy link
Contributor

@spencer-tb spencer-tb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice addition!! :D

Just a couple follow up comments/nits!

@marioevz marioevz force-pushed the negative-test-marker branch from f86a4da to fe4c01f Compare April 11, 2025 17:50
Copy link
Contributor

@spencer-tb spencer-tb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@spencer-tb spencer-tb merged commit 2fe8ca5 into main Apr 12, 2025
22 checks passed
@spencer-tb spencer-tb deleted the negative-test-marker branch April 12, 2025 05:24
pacrob pushed a commit to pacrob/execution-spec-tests that referenced this pull request May 5, 2025
* feat(specs): Introduce/enforce `negative` marker

* feat(tests): Add marker to all invalid transaction/block tests

* changelog

* refactor(specs): Rename marker to `pytest.mark.exception_test`

* refactor(tests): Rename marker to `pytest.mark.exception_test`

* refactor(specs): Make `request` a `PrivateAttr`

* nit

* fix(specs): Review suggestions

Co-authored-by: spencer <[email protected]>

---------

Co-authored-by: spencer <[email protected]>
felix314159 pushed a commit to felix314159/execution-spec-tests that referenced this pull request May 16, 2025
* feat(specs): Introduce/enforce `negative` marker

* feat(tests): Add marker to all invalid transaction/block tests

* changelog

* refactor(specs): Rename marker to `pytest.mark.exception_test`

* refactor(tests): Rename marker to `pytest.mark.exception_test`

* refactor(specs): Make `request` a `PrivateAttr`

* nit

* fix(specs): Review suggestions

Co-authored-by: spencer <[email protected]>

---------

Co-authored-by: spencer <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scope:fw Scope: Framework (evm|tools|forks|pytest) scope:tests Scope: Changes EL client test cases in `./tests` type:feat type: Feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants