HTML Report: Successful Tests Can Be Incorrectly Coloured Red

[Tenzian]

In the HTML report, when using a Cest with a dataprovider, if one of the tests fails then all subsequent tests using that dataprovider are coloured in red even if they pass.

Example

Cest

<?php
namespace { # global namespace

    class FailsCest
    {

       /**
        * @dataprovider pageProvider
        */
        public function withProvider(AcceptanceTester $I, \Codeception\Example $example)
        {
            $I->amOnPage($example['url']);
            $I->seeInTitle($example['title']);
        }

        public function failedWithoutProvider(AcceptanceTester $I)
        {
            $I->amOnPage("http://www.bbc.co.uk");
            $I->see('rumplestiltskin', 'h1');
        }

        public function passedWithoutProvider(AcceptanceTester $I)
        {
            $I->amOnPage("http://www.bbc.co.uk");
            $I->see('BBC Homepage', 'h1');
        }

        public function failedAssert(AcceptanceTester $I)
        {
            $I->assertEquals(true, false);
        }

        public function passedAssert(AcceptanceTester $I)
        {
            $I->assertEquals(true, true);
        }

        /**
         * @return array
         */
        protected function pageProvider()
        {
            return [
                [
                    'url'   => "http://www.bbc.co.uk",
                    'title' => "BBC - Home"
                ],
                [
                    'url'   => "http://codeception.com",
                    'title' => "Zebra"
                ],
                [
                    'url'   => "http://www.php.net",
                    'title' => "PHP: Hypertext Preprocessor"
                ],
                [
                    'url'   => "http://www.google.com",
                    'title' => "Google"
                ],

            ];
        }
    }
}

CLI

$ codecept run acceptance --html
Codeception PHP Testing Framework v2.4.1
Powered by PHPUnit 7.0.2 by Sebastian Bergmann and contributors.

Acceptance Tests (8) ---------------------------------------------------------------------------------------------------------------
✔ FailsCest: With provider | "http://www.bbc.co.uk","BBC - Home" (0.56s)
✖ FailsCest: With provider | "http://codeception.com","Zebra" (1.25s)
✔ FailsCest: With provider | "http://www.php.net","PHP: Hypertext Preprocessor" (0.15s)
✔ FailsCest: With provider | "http://www.google.com","Google" (0.14s)
✖ FailsCest: Failed without provider (0.45s)
✔ FailsCest: Passed without provider (0.55s)
✖ FailsCest: Failed assert (0.00s)
✔ FailsCest: Passed assert (0.00s)
------------------------------------------------------------------------------------------------------------------------------------


Time: 3.24 seconds, Memory: 12.00MB

There were 3 failures:

---------
1) FailsCest: With provider | "http://codeception.com","Zebra"
 Test  tests/acceptance/FailsCest.php:withProvider
 Step  See in title "Zebra"
 Fail  page title contains Zebra
Failed asserting that '\n
        \n
        Codeception\n
        \n
    ' contains "Zebra".

Scenario Steps:

 2. $I->seeInTitle("Zebra") at tests/acceptance/FailsCest.php:13
 1. $I->amOnPage("http://codeception.com") at tests/acceptance/FailsCest.php:12


---------
2) FailsCest: Failed without provider
 Test  tests/acceptance/FailsCest.php:failedWithoutProvider
 Step  See "rumplestiltskin","h1"
 Fail  Failed asserting that any element by 'h1' on page / 
+ <h1 class="hp-main-heading" id="hp-bbc-homepage-content">BBC Homepage</h1>
contains text 'rumplestiltskin'

Scenario Steps:

 2. $I->see("rumplestiltskin","h1") at tests/acceptance/FailsCest.php:19
 1. $I->amOnPage("http://www.bbc.co.uk") at tests/acceptance/FailsCest.php:18


---------
3) FailsCest: Failed assert
 Test  tests/acceptance/FailsCest.php:failedAssert
 Step  Assert equals true,false
 Fail  Failed asserting that false matches expected true.

Scenario Steps:

 1. $I->assertEquals(true,false) at tests/acceptance/FailsCest.php:30


FAILURES!
Tests: 8, Assertions: 8, Failures: 3.
- HTML report generated in file:///Users/Tenz/Repositories/demo/tests/_output/report.html

HTML

Expected Results

"With provider" results for http://www.php.net and http://www.google.com should be green.

[Tenzian]

I think I know a fix for this, bear with me while I fork a copy and see if it works.

[axboory]

This issue is still happening to me. I have to run 10 tests but for like 30 times. And in the report.html, the first 2 times I can see some Passed green tests but the next 28 times are all red even if they Passed.

[Tenzian]

@axboory
I'm not seeing this, and I have one cest that uses a dataprovider to run the same test 16,435 times (don't ask 😁 ).

Can you provide an example test that replicates the problem you are seeing?

[axboory]

So this is an example. "Close pop up and choose card" and "Change password" tests are dependent on "Register test" test to pass, that is why in the first screenshot those two tests are skipped. The second screenshot is from the same run but on another --env and in my console output all three tests have passed, but in the html report the "Register test" is considered as failed and the text from the last step is bolded.

So after a few different tests are failing for the two or first three environments, the next 30 something environments are all red even if tests passed.
This issue makes it a lot harder to get accurate results when running hundreds of tests.

Edit: Now I realized that this might not be the same exact issue because for me it is happening between suites with different environments not in the same suite.

[Tenzian]

This issue was caused by the signatures for all iterations of a test using @dataprovider being the same.

That meant once one had failed all the other examples following it also got marked as failed because the HTML report was identifying them as the same test.

Yours looks like it might be something else.

Unfortunately it's impossible to tell exactly what's actually happening in your case just from a description and a couple of screenshots.

I'm happy to have a go at troubleshooting it if you can provide an example test that has the same behaviour as yours (like I did at the top of this issue) so the problem can be reproduced.

[axboory]

Hello. I used the example test you used at the top except that I changed amOnPage to amOnUrl because of some environment settings.

<?php
namespace { # global namespace

    class FailsCest
    {

       /**
        * @dataprovider pageProvider
        */
        public function withProvider(AcceptanceTester $I, \Codeception\Example $example)
        {
            $I->amOnUrl($example['url']);
            $I->seeInTitle($example['title']);
        }

        public function failedWithoutProvider(AcceptanceTester $I)
        {
            $I->amOnUrl("http://www.bbc.co.uk");
            $I->see('rumplestiltskin', 'h1');
        }

        public function passedWithoutProvider(AcceptanceTester $I)
        {
            $I->amOnUrl("http://www.bbc.co.uk");
            $I->see('BBC Homepage', 'h1');
        }

        public function failedAssert(AcceptanceTester $I)
        {
            $I->assertEquals(true, false);
        }

        public function passedAssert(AcceptanceTester $I)
        {
            $I->assertEquals(true, true);
        }

        /**
         * @return array
         */
        protected function pageProvider()
        {
            return [
                [
                    'url'   => "http://www.bbc.co.uk",
                    'title' => "BBC - Home"
                ],
                [
                    'url'   => "http://codeception.com",
                    'title' => "Zebra"
                ],
                [
                    'url'   => "http://www.php.net",
                    'title' => "PHP: Hypertext Preprocessor"
                ],
                [
                    'url'   => "http://www.google.com",
                    'title' => "Google"
                ],

            ];
        }
    }
}

I managed to reproduce the issue by running with 4 different environments for a couple of times.
As you can see in the screenshot below the test for google.com on the testenv4 is displayed as failed with the last step bolded, but after the test failed for the previous two environments.

And below is the result from the console output for testenv4. And as you can see the test for google.com actually passed.

testfolder (testenv4) Tests (8) -------------------------------------------

FailsCest: With provider | "http://www.bbc.co.uk","BBC - Home"
Signature: FailsCest:withProvider
Test: tests\casino_mobile\OfTestCest.php:withProvider
Scenario --
 I am on url "http://www.bbc.co.uk"
 I see in title "BBC - Home"
 PASSED

FailsCest: With provider | "http://codeception.com","Zebra"
Signature: FailsCest:withProvider
Test: tests\casino_mobile\OfTestCest.php:withProvider
Scenario --
 I am on url "http://codeception.com"
 I see in title "Zebra"
 FAIL

FailsCest: With provider | "http://www.php.net","PHP: Hypertext Preprocessor"
Signature: FailsCest:withProvider
Test: tests\casino_mobile\OfTestCest.php:withProvider
Scenario --
 I am on url "http://www.php.net"
 I see in title "PHP: Hypertext Preprocessor"
 FAIL

FailsCest: With provider | "http://www.google.com","Google"
Signature: FailsCest:withProvider
Test: tests\casino_mobile\OfTestCest.php:withProvider
Scenario --
 I am on url "http://www.google.com"
 I see in title "Google"
 PASSED

FailsCest: Failed without provider
Signature: FailsCest:failedWithoutProvider
Test: tests\casino_mobile\OfTestCest.php:failedWithoutProvider
Scenario --
 I am on url "http://www.bbc.co.uk"
 I see "rumplestiltskin","h1"
 FAIL

FailsCest: Passed without provider
Signature: FailsCest:passedWithoutProvider
Test: tests\casino_mobile\OfTestCest.php:passedWithoutProvider
Scenario --
 I am on url "http://www.bbc.co.uk"
 I see "BBC Homepage","h1"
 FAIL

FailsCest: Failed assert
Signature: FailsCest:failedAssert
Test: tests\casino_mobile\OfTestCest.php:failedAssert
Scenario --
 I assert equals true,false
 FAIL

FailsCest: Passed assert
Signature: FailsCest:passedAssert
Test: tests\casino_mobile\OfTestCest.php:passedAssert
Scenario --
 I assert equals true,true
 PASSED

The environments I'm running the tests in are mobile (testenv1=android mobile, testenv2=iphone, testenv3=ipad, testenv4=android tablet), but the issue doesn't seem to happen on a specific device.
There are so many fails because I use browser stack for mobile devices and the pages are loading slow, but it helped me to reproduce the issue easier.

I hope this helps. If you need any more info, please let me know.

Thanks for you help!

[Tenzian]

Aha... OK. I know what the problem here is.

You run your environments sequentially into the same report using something like:

codecept run --env=testenv1 --env=testenv2 --env=testenv3 --env=testenv4 --html

The problem is the test signature isn't unique across environments.

The workaround is to run them separately:

codecept run --env=testenv1 --html=testenv1.html
codecept run --env=testenv2 --html=testenv2.html
codecept run --env=testenv3 --html=testenv3.html
codecept run --env=testenv4 --html=testenv4.html

I've always run mine this way - as I want a separate report file per env - so hadn't spotted this 🤦‍♂️

I know what to do to fix it - I'll submit a PR when I have some time.