Is it possible to debug unit tests?

[brunoscopelliti]

Hi there!

Sorry if this question was asked before, but I didn't find anything about it.

I would like to add some unit tests for the select-dropdown component, for which today my PR was merged; so I just started to explore how you tackle the unit tests flow.

I've read the doc about tests, played a bit, and now I've a couple of questions.

Since I'm new to React component unit testing, the possibility to debug a unit test would be useful in my situation.

Question 1: Does calypso permit this out of the box? How?

I guess the answer might revolve around the possibility to run mocha directly (that is mentioned in the doc); in this regard I would also like to report that running the following command:

$ NODE_ENV=test NODE_PATH=client:test node_modules/.bin/mocha test/run-mocha.js 

does not behave how I would have expected: it executes 0 tests.

I digged in the code and I noticed that test files never get added.

// in load-suite.js
requireTestFiles( setup.getConfig() );

// setup.addFile is never executed

Looking at the history, I found that in the past that line was something like

requireTestFiles( require( 'tests.json' ) );

and in fact when I hardcoded the following configuration

requireTestFiles( {
    "components/accordion": {
        "test": [ "index" ]
    }
} );

the tests for the accordion component were run.

Question 2: Am I perhaps doing something wrong, or having wrong expectations?

[gwwar]

Howdy @brunoscopelliti the single test runner supports automatic test discovery. This was updated back in #4596 The runner uses this glob pattern @(client|server|test)/**/test/*.@(js|jsx) to find test files. Basically this means if you add a new test file in a test folder they should be picked up by the runner. You can run a specific test file with the following command, where the file path is relative to the client dir.

 npm run test-client client/state/posts/test/actions.js

See also: https://github.com/Automattic/wp-calypso/blob/master/test/README.md for more test documentation

[gwwar]

Here's also an example of a component test: https://github.com/Automattic/wp-calypso/blob/master/client/components/bulk-select/test/index.js though do let us know if you have any other questions

[brunoscopelliti]

Hi @gwwar .

Thank you for the answers.
Probably I put down my questions badly; I should improve my English.
My question was not about how to execute unit test for a single/specific component. I would like to know if there's a way to run the tests (or a subset of them) in the browser, or inside chrome devtools, so that I could more easily inspect and debug them.

Currently, looking at the code, I don't understand how this could work:

NODE_ENV=test NODE_PATH=client:test node_modules/.bin/mocha test/run-mocha.js 

and in fact it does not work for me.
I copied/pasted that command from here. Is the wiki outdated?

In my fork I've created a branch try/debug-test-in-browser where I managed to make the tests run in Chrome... it's not a bulletproof solution (there're issues with sinon/mochery, probably others I don't have discovered yet) but currently it works quite well for my needs.

To make it works I created a third, test-specific, webpack configuration. What do you think? Do you think that it could be worth of further investigations?

[gziolo]

I would like to know if there's a way to run the tests (or a subset of them) in the browser, or inside chrome devtools, so that I could more easily inspect and debug them.

In the current setup we run all tests in the node environment and fake browser features only when it's required. We have test helper useFakeDom which takes care of everything that is necessary to make it happen.

We also have e2e Automated Tests which allows us to test features using real browsers. In addition we have components playground inside Calypso Live Docs. This is where we usually add visual components and are able to see how they work inside the browser in isolation.

NODE_ENV=test NODE_PATH=client:test node_modules/.bin/mocha test/run-mocha.js 

I copied/pasted that command from here. Is the wiki outdated?

It is broken for a long time now. Related issue is #6076. That would be great if we could fix it, then we could have IDE integration and all benefits that it gives.

To make it works I created a third, test-specific, webpack configuration. What do you think? Do you think that it could be worth of further investigations?

That's interesting. How it performs? Does it work with all modern browsers?

There are couple of reasons why we didn' want use real browsers for unit testing:

• using browser makes it harder to configure and run test suite
• we execute our test suite also on CirlceCI which is our Continuous Integration platform, so it we want it to perform as fast as possible
• we are server-side rendering some pages so we need to make sure they also work in Node env

[gwwar]

if there's a way to run the tests (or a subset of them) in the browser, or inside chrome devtools, so that I could more easily inspect and debug them.

It looks like node is in the process of adding this natively: https://nodejs.org/api/debugger.html#debugger_v8_inspector_integration_for_node_js
And the related PR: nodejs/node#6792

I haven't been keeping up with how maintained the packages are, but in the past I've used https://github.com/node-inspector/node-inspector to debug node processes in Chrome. This currently looks a bit stale, but there might be other package alternatives.

[gwwar]

Theoretically this should just work with mocha if we pass through: mocha --debug-brk --inspect though it will take a little bit of work to pass through the extra flags in our current test setup.

[brunoscopelliti]

@gziolo Thank you for the clarification.

I understand that you have in place different systems which help you in preventing bug, and regressions. Having the possibility to run tests in the browser is not meant to be a replacement for any of these. In fact I mentioned browsers only because they usually have included nice dev tools. Having the possibility to debug the tests using chrome devtools detached from the browser (as @gwwar suggested) would be great as well.

Keep in mind however that this is just meant to help people (like me 😄) understand the test ecosystem (by debugging it), and make writing new tests easier.

How it performs? Does it work with all modern browsers?

Currently it's pretty fast (a matter of seconds), but it can run the tests for a single source file; It shouldn't be too hard to extend to make it run all the tests for a single component.

I've tried it only on Chrome (iOS Sierra)... but since, I've not touched the bundling process too much, I don't see why it shouldn't work also on other browsers.
However, I will be back to you, after I'll have verified.

However you all have answered my initial questions (thank you)... so feel free to close this issue.

[gwwar]

Thanks @brunoscopelliti. I think we'll probably investigate fixing the webstorm integration and wiring up the experimental v8 inspector, but it may take me a bit of time before I free up.

In general I would say that a browser test harness isn't the greatest match for a web application, but it can make a lot of sense for testing visual client libraries. For example, if you were developing a canvas or svg library.

[brunoscopelliti]

but since, I've not touched the bundling process too much, I don't see why it shouldn't work also on other browsers.

Just tested.
It does not work neither on Safari, neither on Firefox; it breaks somewhere inside jsdom... I guess that's what you mean when you said

using browser makes it harder to configure and run test suite

[brunoscopelliti]

It is broken for a long time now. Related issue is #6076. That would be great if we could fix it, then we could have IDE integration and all benefits that it gives.

I managed to debug the tests using my IDE of choice (currently it is visual studio code)...
I launched the same program specified in the package.json.

Since this could be helpful to someone else, this is the configuration I used:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Launch",
            "type": "node2",
            "request": "launch",
            "program": "${workspaceRoot}/test/runner.js",
            "stopOnEntry": false,
            "args": [
                "--grep",
                "components accordion"
            ],
            "cwd": "${workspaceRoot}",
            "preLaunchTask": null,
            "runtimeExecutable": null,
            "runtimeArgs": [
                "--nolazy"
            ],
            "env": {
                "NODE_ENV": "test",
                "NODE_PATH": "test:client",
                "TEST_ROOT": "client"
            },
            "console": "internalConsole",
            "sourceMaps": false,
            "outFiles": []
        }
    ]
}

More on visual studio code launch configuration