twister: Test suite name parsed as a test case if not using ZTest

[vChavezB]

Describe the bug

For harnesses that are not ZTest, the suite name declared in the yaml file is parsed as a test case.

Example:

tests:
  base.my_test_suite:
    harness: "my_harness"

In this example, base.my_test_suite, is parsed as a test case. This is wrong and will make twister complain when running the unit tests. Example:

ERROR   - A None status detected in instance native_sim/native/64/tests/cpputest/base/base.my_test_suite, test case base.my_test_suite.
INFO    - 1/1 native_sim/native/64      tests/cpputest/base/base.my_test_suite            PASSED (native 0.005s)
INFO    -                                    base.my_test_suite                                                          NONE
INFO    -                                    base.my_test_suite.my_test_suite.test_1                       PASSED

In this log, we can see that test_1 was discovered while running the test. The first test case (i.e., base.my_test_suite is the suite name I defined in the yaml.

To Reproduce

I am developing my own harness for cpputest, so this should allow to test it with an example:

git clone https://github.com/vChavezB/zephyr -b twister_cpputest --depth=1
cd twister_cpputest
python ./scripts/twister /tests/cpputest/base

Expected behavior

Test suite names from the yaml should not be considered as test cases. The relevant code is here

zephyr/scripts/pylib/twister/twisterlib/testsuite.py

Lines 462 to 463 in c50777a

	if not parsed_subcases:
	self.add_testcase(self.id, freeform=True)

Impact

Harnesses that are not ztest will always create a fake testcase derived from the test-suite names. This erroneously adds a new testcase that does not exist.

[vChavezB]

I was checking other harnesses, and I see for example that Pytest clears the test cases before running:

zephyr/scripts/pylib/twister/twisterlib/harness.py

Lines 544 to 546 in c50777a

	def _update_test_status(self):
	if self.status == TwisterStatus.NONE:
	self.instance.testcases = []

In this case, we dont get the error because pytests clears any test case before running the tests.

However, for harnesses that inherit from class Harness, there is no way to do this.

I could do something hacky to cleary the testcases:

class MyHarness(Harness):
    def __init__(self):
        self.started = False

    def handle(self, line):
        if not self.started:
            self.instance.testcases = []
            self.started = True

But I think it would be better to not add a fake test case if the harnesss does not support parsing the source code or that the base class Harness does this.

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.

[vChavezB]

This is still a pending issue.

[nashif]

@golowanow Can you please take a look at this and #74372.

[golowanow]

Harnesses that are not ztest will always create a fake testcase derived from the test-suite names.
This erroneously adds a new testcase that does not exist.

currently it is Twister behavior for non-Ztest Test Suites where it creates a 'standalone' default Test Case:

zephyr/doc/develop/test/twister.rst

Lines 283 to 294 in 176676d

	#. Depending on the Test Suite implementation, its Test Case identifiers consist
	of **at least three sections** delimited with a dot (``.``):
	* **Ztest tests**:
	a Test Scenario identifier from the corresponding ``testcase.yaml`` file,
	a Ztest suite name, and a Ztest test name:
	``<Test Scenario identifier>.<Ztest suite name>.<Ztest test name>``
	* **Standalone tests and samples**:
	a Test Scenario identifier from the corresponding ``testcase.yaml`` (or
	``sample.yaml``) file where the last section signifies the standalone
	Test Case name, for example: ``debug.coredump.logging_backend``.

I am developing my own harness for cpputest, .. I see for example that Pytest clears the test cases before running:

In the new harness you should add test cases the similar way, as other harnesses, e.g. recently added #83297.

Please rebase #74372 and let's resolve your problem there if it still remains.

[vChavezB]

Thanks for the clarification. I will clear the list of self.instance.testcases