Rare failures of the unit tests

[bruc]

As part of a new R environment, I've been repeatedly running the unit tests that come with XLConnect using the command, R CMD check --no-stop-on-test-error --no-examples --ignore-vignettes --no-manual --install=skip on version 0.2-15. A week ago, the test failed, but after rerunning the test, it succeeded.

I thought that this could be a problem that occurs occasionally depending on the test data values, so I ran the test 1151 times. Out of those 1151 tests, it failed 5 times. Here's an example failure message:

* checking tests ... ERROR
  Running ‘run_tests.R’
Running the tests in ‘tests/run_tests.R’ failed.
Last 13 lines of output:
  Number of test functions: 32 
  Number of errors: 0 
  Number of failures: 1 
  
   
  1 Test Suite : 
  XLConnect Test Suite - 32 test functions, 0 errors, 1 failure
  FAILURE in test.writeWorksheetToFile: Error in checkEquals(normalizeDataframe(df), res, check.attributes = FALSE,  : 
    Component "Column.I": Mean absolute difference: 0.001000166
  
  Error in runUnitTests() : 
  
  Unit Testing failed (#test failures: 1, #R errors: 0)
  
  Execution halted
* DONE
Status: 1 ERROR

I can provide you the full set of files from the failed tests if you wish. Please let me know if you'd like me to include them on this issue or provide them in some other way.

[spoltier]

@bruc thanks for submitting the issue and sorry we didn't get to it.

I've ran the test 1500 times and didn't see any failures. Since we upgraded to a new major version of POI and did several other small changes, it's quite likely that the issue no longer occurs.

[spoltier]

Observed failure again: https://github.com/miraisolutions/xlconnect/runs/2937007634?check_suite_focus=true
Possibly this occurs on specific OSs (or processors) - in this case the macOs build failed.

The failure occurred during R CMD check, and was similar to what was reported above:

Running the tests in ‘tests/run_tests.R’ failed.
Last 13 lines of output:
  1 Test Suite : 
  XLConnect Test Suite - 32 test functions, 0 errors, 1 failure
  FAILURE in test.dataframeConversion: Error in checkEquals(normalizeDataframe(df), res, check.attributes = FALSE,  : 
    Component "Column.I": Mean absolute difference: 0.0009999275

[spoltier]

Latest one on February 8th
https://github.com/miraisolutions/xlconnect/runs/5113416799?check_suite_focus=true

1 Test Suite : 
XLConnect Test Suite - 32 test functions, 0 errors, 1 failure
FAILURE in test.writeNamedRegionToFile: Error in checkEquals(normalizeDataframe(df), res, check.attributes = FALSE,  : 
  Component "Column.J": Mean absolute difference: 0.001000166

Error in runUnitTests() : 

Unit Testing failed (#test failures: 1, #R errors: 0)

In addition: Warning messages:
1: In RNGkind(kind = testSuite$rngKind, normal.kind = testSuite$rngNormalKind) :
  RNGkind: Marsaglia-Multicarry has poor statistical properties
2: In RNGkind(kind = testSuite$rngKind, normal.kind = testSuite$rngNormalKind) :
  RNGkind: severe deviations from normality for Kinderman-Ramage + Marsaglia-Multicarry
Execution halted

Occurred on ubuntu-18.04 R devel - java 8