Unit & Integration Testing with Pytest

[andyrids]

Context

I've been developing MicroPython packages for an irrigation project, which led me going down the rabbit hole on how best to create and manage these types of packages. I was also looking for a way to better automate and streamline unit tests and run integration tests on devices using pytest.

The repo and the code referenced here are at:

1. GitHub (mirrored)
2. GitLab

There is extra and more in depth documentation in the repo. Hopefully people find this useful or can give extra insight based on their experience.

Package Structure

I use Astral uv package manager, along with uv workspaces and a Python namespace package structure, which can replicate the extension package concept outlined in @ micropython-lib. This also allows the package to be installed locally in the uv .venv, enabling easier testing.

micropython-networkutils
├── networkutils           <-- Core `networkutils` package
│   ├── manifest.py
│   ├── networkutils       <-- Device installation dir i.e. `lib/networkutils/`
│   │   └── core.py        <-- Core package module
│   └── package.json       <-- Package URLs & dependencies (for `mip install`)
├── networkutils-mqtt      <-- Extension package for `networkutils`
│   ├── manifest.py
│   ├── networkutils       <-- Device installation dir i.e. `lib/networkutils/`
│   │   └── mqtt.py        <-- Extension package module
│   ├── package.json       <-- Extension package URLs & dependencies (includes core `networkutils`)
│   └── pyproject.toml     <-- Extension package `pyproject.toml` enables uv workspace & namespace 

Testing

I ran into the same issues as others, in that the unix port does not have certain modules such as device and network and cannot be relied on for testing all packages. As others have found, mocking or patching these modules and the interfaces they expose is an acceptable way to test the logic.

...
├── tests                                      <-- Main test directory
│   ├── integration                            <-- Integration tests
│   │   ├── __init__.py
│   │   └── test_connection_and_package.py     <-- WIP for integration tests
│   ├── unit                                   <-- Unit tests
│   │   ├── __init__.py
│   │   ├── test_network_config.py             <-- Network configuration tests
│   │   ├── test_network_interface_complete.py <-- Complete interface tests
│   │   └── test_network_interface_complex.py  <-- Complex interface unit tests
│   ├── __init__.py
│   └── conftest.py                            <-- Fixtures & mocks for MicroPython modules

Unit Tests

There's definitely room for improvement with my conftest fixtures, which patch and mock the machine and network modules and the network.WLAN interface. I'm still working out how best to go about it and I'm keen to hear the solutions of others.

These unit tests are in the /tests/unit/ repo directory.

Integration Tests

Having experimented with mpremote, I found a way to connect to a micro-controller in a script context, rather than through the CLI. Through a programmatic connection to the REPL, it is possible to send commands that utilise and test the package on the device, returning any output as a string for assertions within pytest functions.

These strings can be JSON, which might be useful if you had for example, an object that tracks its state, which could be printed in JSON and loaded into the test function for assertions as a dict.

A serial_connection fixture with 'module' scope ensures all integration tests use the same SerialTransport instance, which is used to send commands over the raw REPL. If the fixture raises an exception or all tests have skipped/failed/completed, cleanup and teardown of the connection is carried out by the fixture.

These integration tests are in /tests/integration/test_connection_and_package.py

I've only gotten as far as testing my network config logic, but I will be adding more integration tests that get a WLAN connection and connect to WiFi in STA or create an AP as per the interface my package exposes. I currently only have RPi Pico W devices to test on.

flowchart TD
    A([Run Pytest])
    B([serial_connection fixture])
    C{SerialTransport}
    D([Assert open connection])
    E{Connected}
    F([Yield SerialTransport])
    G([test_enter_raw_repl])
    H([test_package_installation])
    I([test_package_import])
    J([test_network_config])
    K{Results}
    L([Log output])
    M([Test Summary])
    N([serial_connection fixture<br/>cleanup & teardown])

    A --> B
    B --> C
    C -- No --> N
    C -- Yes --> D
    D --> E
    E -- True --> F
    E -- False --> N
    F --> G
    G --> H
    H --> I
    I --> J
    J --> K
    K -- Fail --> L
    K -- Pass --> M
    L --> M
    M --> N

Loading

[andrewleech]

I too have an interest in automated testing with micropython, thanks for sharing! I find myself using the test runners in particular the multi-test system from micropython itself a fair bit; I've extended the docs for it in #17301 (pending review) maybe something could help.

Separately a while back I worked on a minimal port of pytest for micropython itself in https://github.com/andrewleech/micropython-pytest/ - not sure if that would help or not?

At work we've got a more comprehensive mock-machine package to extend what can be tested with the unix port; we've previously been granted permission internally to release this open source I just haven't had time to clean it up enough to push anywhere, I'll try to take another run at that soon!

[andyrids]

Hi @andrewleech thanks for sharing those, they're definitely useful. Will be great to see how that mock-machine package has been setup from a professional working context, when you get a chance to release it properly.

I going to spend some time going through the multi-test properly and your extended docs for them as well. I'll also be giving the pytest port a try too, so thanks for your time invested in those projects.

On a sidenote, I do allot of development on Linux distros run on WSL2, so I've actually used your wsl-usb-gui quite a lot. Previously, I was running a PS script that monitored for events and auto attached my devices to a running WSL distro, so your tool was much appreciated.