Round 2 of Test Organization improvements

[reactive-firewall]

Background

Starting with #199 and the related changes cumulating in the v2.0.5 pre-release ( PR #330 ) the initial groundwork for reorganizing the testing workflow has been laid.

Update: As of v2.0.8 stable builds are possible in CI and can be pushed and pulled from the github CI infrastructure. This allows for CI pipeline style testing.

Next Steps

Further improvements are needed; Namely:

• Regressions in the coupling between changes (from pushes or PR/MRs) and the results of the CI/CD testing via GHAs, need to be corrected so that testing results are provided readily to maintainers through the GitHub platform similar to as it was the case before splitting the Tests.yml workflow.

  • CI-Test need to once again apply to each pull-requests check suites, while still being dependent on a successful CI-BUILD workflow run.
  • CI-Test need to once again apply to relevant pull-requests check suites, while still being dependent on a successful CI-MATs workflow run.
  • The result of the CI-BUILD should include an artifact with the built python package in a form that can be pip install-ed while testing.
  • CI-MATs should use the same environment as that used by CI-BUILD which it depends
  • CI-DOCs should NOT use an environment unless needed
  • CI-Tests should use the same environment as that used by CI-BUILD (and CI-MATs) which it depends
  • CD-DOCs should use the same environment as that used by CI-DOCs which it depends
  • CI-DOCs should use the new Documentation environment

• Improved CI/CD result summaries, (eg. what passed, or failed?)
  • Comments on GitHub Commits (namely for the commit by SHA of the CI run)
  • Feedback on GitHub PRs (when relevant)
    

General idea for CI/CD

New Dependency Graph

graph TD;
    PUSH-EVENT-->CI-Build.yml;
    CI-Build.yml-->Build-Artifact.whl;
    CI-Build.yml-->Build-Info-Artifact.txt;
    CI-Build.yml-->Build-Summary-Artifact.txt;
    Build-Artifact.whl-->CI-MATs.yml;
    Build-Info-Artifact.txt-->CI-MATs.yml;
    CI-MATs.yml-->MATs-Info-Artifact.txt;
    CI-MATs.yml-->MATs-Summary-Artifact.txt;
    Build-Artifact.whl-->CI-Tests.yml;
    MATs-Info-Artifact.txt-->CI-Tests.yml;
    CI-Tests.yml-->COVERAGE-Artifact.zip;
    CI-Tests.yml-->Test-Results-Artifact.zip;
    CI-Tests.yml-->Test-Summary-Artifact.txt;
    Build-Artifact.whl-->CD-DOCs.yml;
    MATs-Info-Artifact.txt-->CD-DOCs.yml;
    CD-DOCs.yml-->Documentation-Artifact.zip;

Loading

Improved CI/CD process

Note

This diagram is only a summary, a lot more steps are performed, but these are only the more relevant sequences of interactions. See implementation for exact behavior.

sequenceDiagram
    Actor User
    participant GH as GitHub
    User->>+GH: Pushes Changes
    Note left of GH: eg. triggers actions
    create participant BUILD as .github/workflows/CI-Build.yml
    GH->>BUILD: TRIGGER
    GH-->>User: Pushed
    deactivate GH
    critical Build Action
        BUILD->>BUILD: Builds "dist/multicast-Artifact.whl"
    option Build Success
        BUILD->>GH: Uploads "dist/multicast-Artifact.whl"
        BUILD-->>BUILD: Creates "Build-Summary-Artifact.txt"
    option Build Failure
        BUILD->>BUILD: Creates "Build-Summary-Artifact.txt"
    end
    destroy BUILD
    BUILD-->>GH: Uploads "Build-Summary-Artifact.txt"
    Note right of BUILD: BUILD is Done.
    create participant MATs as .github/workflows/CI-MATs.yml
    GH->>MATs: TRIGGER
    alt IF Build Success
        Note Left of MATs: FIND "dist/multicast-Artifact.whl"
        MATs<<->>GH: Downloads "dist/multicast-Artifact.whl"
        MATs-->>MATs: Invoke MATs
        MATs->>MATs: Creates "MATs-Summary-Artifact.txt"
        MATs-->>GH: Uploads "MATs-Summary-Artifact.txt"
    else IF Build Failure
        GH-xMATs: CANCEL
    end
    Note right of MATs: MATs are Done.
    destroy MATs
    GH--xMATs: 
    create participant TESTs as .github/workflows/CI-Tests.yml
    GH->>TESTs: TRIGGER
    opt If PR
        create participant DOCs as .github/workflows/CI-DOCs.yml
        GH->>DOCs: TRIGGER
        alt IF Build AND MATs Success
            Note Left of DOCs: Found "dist/multicast-Artifact.whl"
            DOCs<<->>GH: Downloads "dist/multicast-Artifact.whl"
            Note right of DOCs: Found "MATs-Summary-Artifact.txt"
            DOCs->>DOCs: Invoke Documentation Linting/Testing
            alt Documentation Linting Success
                DOCs-->DOCs: Creates "DOCs-Summary-Artifact.txt"
                DOCs->>DOCs: Creates "Documentation-Artifact.zip"
                DOCs-->>DOCs: Uploads "Documentation-Artifact.zip"
            else Documentation Linting failure
                DOCs->>DOCs: Creates "DOCs-Summary-Artifact.txt"
            end
            destroy DOCs
            DOCs-->>GH: Uploads "DOCs-Summary-Artifact.txt"
        end
    end
    alt IF Build AND MATs Success
        Note Left of TESTs: Found "dist/multicast-Artifact.whl"
        TESTs<<->>GH: Downloads "dist/multicast-Artifact.whl"
        Note Left of TESTs: Found "MATs-Summary-Artifact.txt"
        TESTs->>TESTs: Invoke Coverage Testing
        TESTs->>TESTs: Invoke Style Testing
        alt Style Test Success
            alt ALL Coverage tests are a success
                TESTs-->>TESTs: Invoke Integration Coverage Testing
                alt ALL Integration tests are ALSO a success
                    TESTs->>TESTs: Creates "COVERAGE-Artifact.zip"
                    TESTs-->>TESTs: Uploads "COVERAGE-Artifact.zip"
                else Integration tests have a failure
                    TESTs->>TESTs: Creates "TESTs-Summary-Artifact.txt"
                end
            else Coverage tests have a failure
                TESTs->>TESTs: Creates "TESTs-Summary-Artifact.txt"
            end
        else Style tests have a failure
            TESTs->>TESTs: Creates "TESTs-Summary-Artifact.txt"
        end
        TESTs->>TESTs: Creates "Test-Results-Artifact.zip"
        TESTs-->>GH: Uploads "MATs-Summary-Artifact.txt"
        TESTs-->>GH: Uploads "Test-Results-Artifact.zip"
    else IF Build Failure
        GH-xTESTs: CANCEL
    else IF MATs Failure
        Note Left of TESTs: Found "dist/multicast-Artifact.whl"
        TESTs<<->>GH: Downloads "dist/multicast-Artifact.whl"
        GH-xTESTs: CANCEL
    end
    Note right of TESTs: TESTs are Done.

Loading