Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unit test - split "AND"-ed conditionals into separate asserts #596

Closed
jphickey opened this issue Apr 9, 2020 · 3 comments · Fixed by #1624 or #1632
Closed

Unit test - split "AND"-ed conditionals into separate asserts #596

jphickey opened this issue Apr 9, 2020 · 3 comments · Fixed by #1624 or #1632
Assignees
Milestone

Comments

@jphickey
Copy link
Contributor

jphickey commented Apr 9, 2020

Is your feature request related to a problem? Please describe.
Debugging unit tests can be very difficult, frustrating, and time consuming. One major part of the problem which makes them very debugging-unfriendly is something like the following:

    UT_Report(__FILE__, __LINE__,
              UT_EventIsInHistory(CFE_ES_PCR_ERR2_EID) &&
              CFE_ES_Global.AppTable[Id].ControlReq.AppTimer == 0 &&
              CFE_ES_Global.AppTable[Id].ControlReq.AppControlRequest == CFE_ES_RunStatus_SYS_DELETE,
              "CFE_ES_ScanAppTable",
              "Waiting; process control request");

The problem with this type of construct is that there are 3 separate tests being combined into one single assert. When it fails, it is not possible to see which of the three conditions are evaluating false. Many of them call functions within the test case, too, which further obfuscate what the actual return value was. The only way to test this is run it in a debugger, break at the start of the test, then set a breakpoint inside e.g. UT_EventIsInHistory to see what it returned.

Describe the solution you'd like

  1. At a minimum - split the && conditions into separate asserts. This would at least let the developer know which one is actually the fault.
  2. Nice to have - employ the macros similar to what @CDKnightNASA added in Fix #397, Remove old unit test example, add README.md, further macro cleanup osal#405, which show the values tested in the log, not simply just a pass/fail.

Describe alternatives you've considered
Continue struggling to figure out what actually went wrong every time a UT failure comes up.

Requester Info
Joseph Hickey, Vantage Systems, Inc.

@skliper skliper added this to the 6.8.0 milestone Apr 9, 2020
@skliper
Copy link
Contributor

skliper commented Apr 9, 2020

Added milestone, targeted for this round for unit test consistency.

@CDKnightNASA
Copy link
Contributor

Now that the UT macros for OSAL have been accepted I can take a look at using that model for CFE as well (I have a crack at sb_UT.c but I like the OSAL macros even more so I will revisit.)

@CDKnightNASA CDKnightNASA self-assigned this Apr 9, 2020
@skliper skliper modified the milestones: 6.8.0, 7.0.0 Jun 5, 2020
@jphickey jphickey self-assigned this Jun 16, 2021
@jphickey
Copy link
Contributor Author

In order to cross reference/validate that the test cases are covering the documented functions, need to do a scrub of all "Report" outputs and make them use the proper macros that include full context info.

jphickey added a commit to jphickey/cFE that referenced this issue Jun 16, 2021
Scrub through all UT reporting calls in es_UT.c and replace
with preferred macro where possible.

- All calls to check status/return code are replaced with macro
  that logs all arguments and return value
- All calls that involved multiple conditions AND'ed together
  are replaced with individual asserts on each condition.
jphickey added a commit to jphickey/cFE that referenced this issue Jun 16, 2021
Scrub through all UT reporting calls in es_UT.c and replace
with preferred macro where possible.

- All calls to check status/return code are replaced with macro
  that logs all arguments and return value
- All calls that involved multiple conditions AND'ed together
  are replaced with individual asserts on each condition.
jphickey added a commit to jphickey/cFE that referenced this issue Jun 16, 2021
Scrub through all UT reporting calls in es_UT.c and replace
with preferred macro where possible.

- All calls to check status/return code are replaced with macro
  that logs all arguments and return value
- All calls that involved multiple conditions AND'ed together
  are replaced with individual asserts on each condition.
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update ES coverage test to use preferred macros.

Adds dedicated assert macros for checking fixed-length
string buffers, and for checking memory offsets.

Also adds an improved implemention of the syslog/printf check
which filters out newlines (keeps log more parseable).
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update EVS coverage test to use preferred macros
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update SB coverage test to use preferred macros.

Adds a dedicated assert macro for checking SB MsgId values.
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update MSG coverage test to use preferred macros
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update SBR coverage test to use preferred macros
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update TBL coverage test to use preferred macros
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update TIME coverage test to use preferred macros
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update FS coverage test to use preferred macros.
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Clean up the assert functions and macros which are no longer
used after updating all coverage tests to use preferred macros.
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update ES coverage test to use preferred macros.

Adds dedicated assert macros for checking fixed-length
string buffers, and for checking memory offsets.

Also adds an improved implemention of the syslog/printf check
which filters out newlines (keeps log more parseable).
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update EVS coverage test to use preferred macros
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update SB coverage test to use preferred macros.

Adds a dedicated assert macro for checking SB MsgId values.
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update MSG coverage test to use preferred macros
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update SBR coverage test to use preferred macros
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update TBL coverage test to use preferred macros
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update TIME coverage test to use preferred macros
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Update FS coverage test to use preferred macros.
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Clean up the assert functions and macros which are no longer
used after updating all coverage tests to use preferred macros.
jphickey added a commit to jphickey/cFE that referenced this issue Jun 22, 2021
Clean up the assert functions and macros which are no longer
used after updating all coverage tests to use preferred macros.
jphickey added a commit that referenced this issue Jun 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants