Add attribute `file` to JUnit XML report to accommodate CircleCI algorithm for splitting tests

[yterradas]

Hi,

CircleCI allows to split tests by timings using a JUnit XML report. Unfortunately it requires testcase to have an file attribute in order to properly work as detailed in this CircleCI forum question. I have confirmed it by manipulating the JUnit XML report output of gotestsum. I understand that file is not part of the JUnit XSD but it would really help if it could be added to the output if the user wanted it. What are your thoughts?

[dnephin]

Thanks for opening this issue! The JUnit XML report should definitely work with CircleCI. This is a blog post from CircleCI that covers usage.

With Go, tests can only be split reliably by package because go test allows you to easily run tests by package, but running individual tests within the page is much harder (requires very special handling of -run).

So CircleCI should use the filename on testsuite, instead of testcase to split the files. There's an example of that here. There are many projects using the JUnit xml from gotestsum to support test splitting on CircleCI, so I'm pretty confident it works.

You may be missing the --timings-type=classname flag to make it work?

[yterradas]

I read the post (dated 2018) while investigating the issue. I have no issue generating the report. However as stated in the forum post i linked (date 2020) the issue is that CircleCI ignores classname if file is not present. By the time I opened this issue I had already tested using classname. I confirmed it doesnt work, as the forum post states. Only after adding file attribute did CircleCI test splitting algorithm worked.

• logs from worker 0

++ circleci tests split --split-by=timings --timings-type=classname --show-counts
Read 53 lines of classname(s)
No timing found for "test/folder/XXX_test.go"
No timing found for "test/folder/XXX_test.go"
...
Bucket 1: assigning 6 classname(s), total weight 32024621988

• logs from worker 6

++ circleci tests split --split-by=timings --timings-type=classname --show-counts
Read 53 lines of classname(s)
No timing found for "test/folder/XXX_test.go"
No timing found for "test/folder/XXX_test.go"
...
Bucket 6: assigning 5 classname(s), total weight 26687184990

after adding attribute file

• logs from worker 0

++ circleci tests split --split-by=timings --show-counts
Read 53 lines of autodetect(s)
Autodetected filename timings.
Bucket 0: assigning 4 autodetect(s), total weight 490860000

Unfortunately i dont see CircleCI fixing the issue, if they even think it is an issue, because the forum question is over 1year old and the problem persists. Hence i am hoping i can make the case to you to add the extra attribute. I can volunteer my time if it helps.

[dnephin]

Thank you for the additional details! I do want to make sure this works properly with CircleCI test splits.

What are you passing as stdin into circleci tests split ? From the output it seems like you might be passing a list of files, but for this to work with Go it needs to be a list of package names.

In the example I linked, it's passing in go list ./...:

go list ./... | circleci tests split --split-by=timings --timings-type=classname

[yterradas]

you are right the input is a list of files. these are integration tests and they all in the same package. nonetheless i will test your suggestion.

[dnephin]

Ah, that is an interesting use case! I guess if you run individual files with go test , then none of the files can reference variables or functions in the other files, is that right?

I think it's generally safer to split by package. If the only change required is to add another property to the xml, I'd be happy to accept that change.

The challenge might be that the json we get from go test -json does not include a filename. So I'm not sure how we'll get the name to add to the file.

It might be possible to post-process the xml, use go list -f ... to look up all the filenames for the test functions, and add them to the xml that way. If that's the only way to do it, I think it might be better as a separate tool. gotestsum has some additional tools at gotestsum tool, we could add the post-processing logic there so that it's still able to reference the internal/junitxml package.

[yterradas]

that sounds good. i will rework the code a little to better test classname just in case the initial tests werent valid. i will comeback with the results.

[yterradas]

@dnephin
after some refactoring i was able to use classname. i can confirm CircleCI does use classname correctly. thank you for your time, much appreciated.

[elfiyang16]

@yterradas hi sorry for tagging you here. Do you mind sharing how the problem got resolved? As I'm currently having the same problem... Thanks!

[yterradas]

@elfiyang16 sorry i missed your comment. the issue was solved by switching from running test files to test packages.