Skip Greentea tests for Mbed OS code coverage on Fast Models #7805
Conversation
FastModels targets are simulator running on the x86 hosts. As the nature of non-RealTime x86 OS and FastModels, timing accuracy is not guaranteed So skipping the time drifting tests on FastModel targets
The thread stack size was restricted due to some boards have really limited RAM sizes, and out of heap memory on multiple threads tests. The side effect was on the debug profile build, the tests will get stack overflow. We need the build the test with debug profile in order to do the code coverage analysis. So increased the thread stack size on FastModel targets.
|
Here attached the greentea test log results for reference: |
Merged. This is ready for review! |
cmonr
changed the title
TESTS/mbed_drivers/ticker/main.cpp
Outdated
| @@ -104,6 +104,10 @@ void increment_multi_counter(void) | |||
| */ | |||
| void test_case_1x_ticker() | |||
| { | |||
| #if defined(__ARM_FM) | |||
There was a problem hiding this comment.
If it's required to skip tests, the pattern that we use is to add the #if def in the Cases cases[] array.
This makes determining if tests are being skipped much easier to follow.
Please make this change to this and the other tests.
There was a problem hiding this comment.
Actually, that was my first attempt. with more then one case it just fine.
However, some of the tests only have one test case, e.g. here
I can't skip the only test case in the Case case[ ] . any suggestions on this case ? should I block the whole test file as NOT_SUPPORTTED
I took the currently method purely becasue it give a message while skipping the tests. I admit it not as easy as the other on to follow in the code level.
@cmonr
There was a problem hiding this comment.
Ah, that's a good point.
As an example for how things are done right now (doesn't mean that they should be like this, just means that this is how we do it right now), take a look at this test: https://github.com/ARMmbed/mbed-os/blob/master/TESTS/mbed_hal/common_tickers/main.cpp
There's an ifdef on the top of the file for targets that don't have the supported feature, and another ifdef for if a different feature is available. Please follow this pattern for this PR, and I would appreciate an issue being opened for this alternate method.
|
@cmonr please have another look |
| Case("Test timers: 1x ticker", test_case_1x_ticker), | ||
| Case("Test timers: 2x ticker", test_case_2x_ticker) | ||
| #endif |
There was a problem hiding this comment.
if test_case_2x_ticker is skipped from compiled time do we need runtime check on line 149 ?
What limitations are there for ARM_FM for this 2 test cases ? I would like to avoid having this ifdef there
There was a problem hiding this comment.
Good catch. my bad forgot to remove line 149, which suggested by @cmonr.
The reason was stated in the PR description:
FastModels targets are simulator running on the x86 hosts.
As the nature of non-RealTime x86 OS and FastModels, timing accuracy is not guaranteed
mbed time-drifting-tests will failing all the time on FastModel targets
my first version to skip the test was in line 149, now it been changed to like line 338
|
@cmonr, Thanks for pointing out, my case is a bit different though. putting |
|
@cmonr, your suggestion been addressed. please review again |
|
/morph build |
Build : SUCCESSBuild number : 2896 Triggering tests/morph test |
Exporter Build : SUCCESSBuild number : 2521 |
Test : FAILUREBuild number : 2647 |
|
We've been having F401 issues in CI. |
Test : FAILUREBuild number : 2655 |
|
em..... this time GCC passed, IAR failed on the same test, same F401. |
|
@jamesbeyond The 401's have now been power cycled... /morph test |
Test : SUCCESSBuild number : 2656 |
Description
This PR updated the Greentea tests which failed on code coverage testing on FastModels.
Skip time drifting test on FastModels targets
FastModels targets are simulator running on the x86 hosts.
As the nature of non-RealTime x86 OS and FastModels, timing accuracy is not guaranteed
So skipping the time drifting tests on FastModel targets
Increase the thread stack size on FastModel targets
The thread stack size was restricted due to some boards have really limited RAM sizes,
and out of heap memory on multiple threads tests.
The side effect was on the debug profile build, the tests will get stack overflow.
We need the build the test with debug profile in order to do the code coverage analysis.
So increased the thread stack size on FastModel targets.
This PR have a dependency on #7706
Pull request type