twister: Add support for Cpputest

[vChavezB]

Similar to gTest, CppuTest is a CPP framework for unit tests. This commit adds support to detect Cpputest console output when verbose mode is enabled (-v).

I am open for any feedback. My main use case was some unit tests that were written with Cpputest that I needed to port to Zephyr and integrate with twister.

[MaureenHelm]

@vChavezB please take a look at the failed checks

[vChavezB]

Thanks for the feedback, I must have missed it.

[hakehuang]

would be possible to a test application for this?

[PerMac]

Why the need for "-v"? I don't see it in the code. Also, it shouldn't be the case. Level of output verbosity mustn't affect a test result.

[vChavezB]

The verbose argument must be applied to your CPPUtest application. Correct me if I am wrong but if you do not add the verbose option, you do not get information about which test cases have passed. Instead you will only get a dot for each test that has passed (e.g. ....) and at the end something such as OK (2 tests, 2 ran, 94 checks, 0 ignored, 0 filtered out, 2 ms).

Without verbose

..
OK (2 tests, 2 ran, 94 checks, 0 ignored, 0 filtered out, 2 ms)

With verbose

TEST(MyGroup, test1)
TEST(MyGroup, test2)
 OK (2 tests, 2 ran, 94 checks, 0 ignored, 0 filtered out, 2 ms)

[vChavezB]

would be possible to a test application for this?

I made a sample here

https://github.com/vChavezB/zephyr_cpputest_sample

git clone https://github.com/vChavezB/zephyr_cpputest_sample
wes init -l zephyr_cpputest_sample
west update -o=--depth=1 -n
./zephyr/scripts/twister -vvv -T zephyr_cpputest_sample

[hakehuang]

would be possible to a test application for this?

I made a sample here

https://github.com/vChavezB/zephyr_cpputest_sample

git clone https://github.com/vChavezB/zephyr_cpputest_sample
wes init -l zephyr_cpputest_sample
west update -o=--depth=1 -n
./zephyr/scripts/twister -vvv -T zephyr_cpputest_sample

I would support you to move this test into tests/cpputest

[vChavezB]

These are the test cases that are identified by twister:

INFO    - 1/1 native_sim/native/64      tests/cpputest/base/base.my_test_group             PASSED (native 0.005s)
INFO    -                                    base.my_test_group                                                          NONE
INFO    -                                    base.my_test_group.my_test_group.test_1                                     PASSED

The one marked as NONE is actually the test suite group I defined in the yaml.

The only test case that I defined in the sample code is (test_1 , see here)

TEST(my_test_group, test_1) {

}

However, it seems that twister uses the name base.my_test_group.

tests:
  base.my_test_group:
    platform_allow:
      - native_sim/native/64
    harness: "cpputest"

This is implemented here:

zephyr/scripts/pylib/twister/twisterlib/testsuite.py

Line 462 in c50777a

if not parsed_subcases:

[vChavezB]

I have opened an issue (#81176) with the error that you mentioned. This is related to how test cases are parsed for other frameworks that are not ztest.

[vChavezB]

I have made a temporal solution to allow that my CPPUTest harness class deletes the fake testcase at start, but I would prefer that issue #81176 is solved.

[github-actions]

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.