cli: Missing output from failed Example_ tests in teamcity

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/ab8a7b014197c38e6e3198d232f48393acea59b5

Parameters:

COCKROACH_PROPOSER_EVALUATED_KV=true
TAGS=
GOFLAGS=

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=108436&tab=buildLog

ERROR: exit status 255

Makefile:147: recipe for target 'stress' failed
make: *** [stress] Error 1

[a-robinson]

@knz - any chance that the lack of output from the failed test here is at all related to the other lack-of-output problem you were fixing recently in #12662?

We saw a similar failure for a different Example_ test yesterday in #12666

[knz]

No these seem to be unrelated issues.

The log in both cases seems truncated. It looks to me as if go test is messing up the logs of the failing test. This exit status 255 is ... weird -- that's definitely not a signal or a normal "exit on failure".

[knz]

Yeah sorry no 1) there is zero evidence left over by the test to investigate and 2) Go's handling of example-based tests is too much magic, I just don't understand how it works.

Under no circumstance a cockroach process could terminate legitimately with exit code 255, so I'll just blame Go's go test on this one.

My bet is that there was insufficient disk space and some temp file could not be created by go test (I'll bet even a bit higher that it couldn't copy the comment in the source file to a reference file to compare or something like that).

[knz]

Either I'll just not do anything on this issue,

or someone gives me permission to replace the Example_ tests by expect tests.

[jordanlewis]

Seems like this is probably a TeamCity issue. Let's close this until it reoccurs.