roachtest: disk-stalled/log=false,data=true failed #52533

cockroach-teamcity · 2020-08-08T07:09:29Z

(roachtest).disk-stalled/log=false,data=true failed on release-20.1@413b1c209b0a9e87b0a32cea8d3b1a26bd9283a4:

		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | set -exuo pipefail;
		  |   |   thrift_dir="/opt/thrift"
		  |   |
		  |   |   if [ ! -f "/usr/bin/thrift" ]; then
		  |   | 	sudo apt-get update;
		  |   | 	sudo apt-get install -qy automake bison flex g++ git libboost-all-dev libevent-dev libssl-dev libtool make pkg-config python-setuptools libglib2.0-dev
		  |   |
		  |   |     sudo mkdir -p "${thrift_dir}"
		  |   |     sudo chmod 777 "${thrift_dir}"
		  |   |     cd "${thrift_dir}"
		  |   |     curl "https://downloads.apache.org/thrift/0.10.0/thrift-0.10.0.tar.gz" | sudo tar xvz --strip-components 1
		  |   |     sudo ./configure --prefix=/usr
		  |   |     sudo make -j$(nproc)
		  |   |     sudo make install
		  |   |     (cd "${thrift_dir}/lib/py" && sudo python setup.py install)
		  |   |   fi
		  |   |
		  |   |   charybde_dir="/opt/charybdefs"
		  |   |   nemesis_path="${charybde_dir}/charybdefs-nemesis"
		  |   |
		  |   |   if [ ! -f "${nemesis_path}" ]; then
		  |   |     sudo apt-get install -qy build-essential cmake libfuse-dev fuse
		  |   |     sudo rm -rf "${charybde_dir}" "${nemesis_path}" /usr/local/bin/charybdefs{,-nemesis}
		  |   |     sudo mkdir -p "${charybde_dir}"
		  |   |     sudo chmod 777 "${charybde_dir}"
		  |   |     git clone --depth 1 "https://github.com/scylladb/charybdefs.git" "${charybde_dir}"
		  |   |
		  |   |     cd "${charybde_dir}"
		  |   |     thrift -r --gen cpp server.thrift
		  |   |     cmake CMakeLists.txt
		  |   |     make -j$(nproc)
		  |   |
		  |   |     sudo modprobe fuse
		  |   |     sudo ln -s "${charybde_dir}/charybdefs" /usr/local/bin/charybdefs
		  |   |     cat > "${nemesis_path}" <<EOF
		  |   | #!/bin/bash
		  |   | cd /opt/charybdefs/cookbook
		  |   | ./recipes "\$@"
		  |   | EOF
		  |   |     chmod +x "${nemesis_path}"
		  |   | 	sudo ln -s "${nemesis_path}" /usr/local/bin/charybdefs-nemesis
		  |   | fi
		  |   |
		  |   | ```
		  | Wraps: (3) exit status 2
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (2) exit status 20
		Error types: (1) *main.withCommandDetails (2) *exec.ExitError

More

Artifacts: /disk-stalled/log=false,data=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

darinpp · 2020-08-11T03:12:46Z

Seems like some issue with Ruby's bundler and Apache's Thirft gem. Looks like the thirdft gem doesn't like bundler's version

make[3]: Entering directory '/opt/thrift/lib/rb'
/usr/local/bin/bundle install
Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.
NOTE: Gem::Specification#has_rdoc= is deprecated with no replacement. It will be removed on or after 2018-12-01.
Gem::Specification#has_rdoc= called from /opt/thrift/lib/rb/thrift.gemspec:15.
Fetching gem metadata from http://rubygems.org/...
Fetching gem metadata from http://rubygems.org/.
Resolving dependencies...
Bundler could not find compatible versions for gem "bundler":
  In Gemfile:
    bundler (~> 1)

  Current Bundler version:
    bundler (2.1.4)
This Gemfile requires a different version of Bundler.
Perhaps you need to update Bundler by running `gem install bundler`?

Could not find gem 'bundler (~> 1)' in any of the relevant sources:
  the local ruby installation
Makefile:618: recipe for target 'all-local' failed
make[3]: Leaving directory '/opt/thrift/lib/rb'
make[3]: *** [all-local] Error 6

cockroach-teamcity · 2020-08-13T07:22:55Z

(roachtest).disk-stalled/log=false,data=true failed on release-20.1@8352fd645f4517ab0650ad637e22b20d9e524d53:

		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | set -exuo pipefail;
		  |   |   thrift_dir="/opt/thrift"
		  |   |
		  |   |   if [ ! -f "/usr/bin/thrift" ]; then
		  |   | 	sudo apt-get update;
		  |   | 	sudo apt-get install -qy automake bison flex g++ git libboost-all-dev libevent-dev libssl-dev libtool make pkg-config python-setuptools libglib2.0-dev
		  |   |
		  |   |     sudo mkdir -p "${thrift_dir}"
		  |   |     sudo chmod 777 "${thrift_dir}"
		  |   |     cd "${thrift_dir}"
		  |   |     curl "https://downloads.apache.org/thrift/0.10.0/thrift-0.10.0.tar.gz" | sudo tar xvz --strip-components 1
		  |   |     sudo ./configure --prefix=/usr
		  |   |     sudo make -j$(nproc)
		  |   |     sudo make install
		  |   |     (cd "${thrift_dir}/lib/py" && sudo python setup.py install)
		  |   |   fi
		  |   |
		  |   |   charybde_dir="/opt/charybdefs"
		  |   |   nemesis_path="${charybde_dir}/charybdefs-nemesis"
		  |   |
		  |   |   if [ ! -f "${nemesis_path}" ]; then
		  |   |     sudo apt-get install -qy build-essential cmake libfuse-dev fuse
		  |   |     sudo rm -rf "${charybde_dir}" "${nemesis_path}" /usr/local/bin/charybdefs{,-nemesis}
		  |   |     sudo mkdir -p "${charybde_dir}"
		  |   |     sudo chmod 777 "${charybde_dir}"
		  |   |     git clone --depth 1 "https://github.com/scylladb/charybdefs.git" "${charybde_dir}"
		  |   |
		  |   |     cd "${charybde_dir}"
		  |   |     thrift -r --gen cpp server.thrift
		  |   |     cmake CMakeLists.txt
		  |   |     make -j$(nproc)
		  |   |
		  |   |     sudo modprobe fuse
		  |   |     sudo ln -s "${charybde_dir}/charybdefs" /usr/local/bin/charybdefs
		  |   |     cat > "${nemesis_path}" <<EOF
		  |   | #!/bin/bash
		  |   | cd /opt/charybdefs/cookbook
		  |   | ./recipes "\$@"
		  |   | EOF
		  |   |     chmod +x "${nemesis_path}"
		  |   | 	sudo ln -s "${nemesis_path}" /usr/local/bin/charybdefs-nemesis
		  |   | fi
		  |   |
		  |   | ```
		  | Wraps: (3) exit status 2
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (2) exit status 20
		Error types: (1) *main.withCommandDetails (2) *exec.ExitError

More

Artifacts: /disk-stalled/log=false,data=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-20T07:11:15Z

(roachtest).disk-stalled/log=false,data=true failed on release-20.1@18c2c6bb8079a123e5fa37d2e99211d6feefb29b:

		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | set -exuo pipefail;
		  |   |   thrift_dir="/opt/thrift"
		  |   |
		  |   |   if [ ! -f "/usr/bin/thrift" ]; then
		  |   | 	sudo apt-get update;
		  |   | 	sudo apt-get install -qy automake bison flex g++ git libboost-all-dev libevent-dev libssl-dev libtool make pkg-config python-setuptools libglib2.0-dev
		  |   |
		  |   |     sudo mkdir -p "${thrift_dir}"
		  |   |     sudo chmod 777 "${thrift_dir}"
		  |   |     cd "${thrift_dir}"
		  |   |     curl "https://downloads.apache.org/thrift/0.10.0/thrift-0.10.0.tar.gz" | sudo tar xvz --strip-components 1
		  |   |     sudo ./configure --prefix=/usr
		  |   |     sudo make -j$(nproc)
		  |   |     sudo make install
		  |   |     (cd "${thrift_dir}/lib/py" && sudo python setup.py install)
		  |   |   fi
		  |   |
		  |   |   charybde_dir="/opt/charybdefs"
		  |   |   nemesis_path="${charybde_dir}/charybdefs-nemesis"
		  |   |
		  |   |   if [ ! -f "${nemesis_path}" ]; then
		  |   |     sudo apt-get install -qy build-essential cmake libfuse-dev fuse
		  |   |     sudo rm -rf "${charybde_dir}" "${nemesis_path}" /usr/local/bin/charybdefs{,-nemesis}
		  |   |     sudo mkdir -p "${charybde_dir}"
		  |   |     sudo chmod 777 "${charybde_dir}"
		  |   |     git clone --depth 1 "https://github.com/scylladb/charybdefs.git" "${charybde_dir}"
		  |   |
		  |   |     cd "${charybde_dir}"
		  |   |     thrift -r --gen cpp server.thrift
		  |   |     cmake CMakeLists.txt
		  |   |     make -j$(nproc)
		  |   |
		  |   |     sudo modprobe fuse
		  |   |     sudo ln -s "${charybde_dir}/charybdefs" /usr/local/bin/charybdefs
		  |   |     cat > "${nemesis_path}" <<EOF
		  |   | #!/bin/bash
		  |   | cd /opt/charybdefs/cookbook
		  |   | ./recipes "\$@"
		  |   | EOF
		  |   |     chmod +x "${nemesis_path}"
		  |   | 	sudo ln -s "${nemesis_path}" /usr/local/bin/charybdefs-nemesis
		  |   | fi
		  |   |
		  |   | ```
		  | Wraps: (3) exit status 2
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (2) exit status 20
		Error types: (1) *main.withCommandDetails (2) *exec.ExitError

More

Artifacts: /disk-stalled/log=false,data=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-25T07:14:09Z

(roachtest).disk-stalled/log=false,data=true failed on release-20.1@5ccfb78ceeefd88a89ee4fad08ec415770c9a15a:

		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | set -exuo pipefail;
		  |   |   thrift_dir="/opt/thrift"
		  |   |
		  |   |   if [ ! -f "/usr/bin/thrift" ]; then
		  |   | 	sudo apt-get update;
		  |   | 	sudo apt-get install -qy automake bison flex g++ git libboost-all-dev libevent-dev libssl-dev libtool make pkg-config python-setuptools libglib2.0-dev
		  |   |
		  |   |     sudo mkdir -p "${thrift_dir}"
		  |   |     sudo chmod 777 "${thrift_dir}"
		  |   |     cd "${thrift_dir}"
		  |   |     curl "https://downloads.apache.org/thrift/0.10.0/thrift-0.10.0.tar.gz" | sudo tar xvz --strip-components 1
		  |   |     sudo ./configure --prefix=/usr
		  |   |     sudo make -j$(nproc)
		  |   |     sudo make install
		  |   |     (cd "${thrift_dir}/lib/py" && sudo python setup.py install)
		  |   |   fi
		  |   |
		  |   |   charybde_dir="/opt/charybdefs"
		  |   |   nemesis_path="${charybde_dir}/charybdefs-nemesis"
		  |   |
		  |   |   if [ ! -f "${nemesis_path}" ]; then
		  |   |     sudo apt-get install -qy build-essential cmake libfuse-dev fuse
		  |   |     sudo rm -rf "${charybde_dir}" "${nemesis_path}" /usr/local/bin/charybdefs{,-nemesis}
		  |   |     sudo mkdir -p "${charybde_dir}"
		  |   |     sudo chmod 777 "${charybde_dir}"
		  |   |     git clone --depth 1 "https://github.com/scylladb/charybdefs.git" "${charybde_dir}"
		  |   |
		  |   |     cd "${charybde_dir}"
		  |   |     thrift -r --gen cpp server.thrift
		  |   |     cmake CMakeLists.txt
		  |   |     make -j$(nproc)
		  |   |
		  |   |     sudo modprobe fuse
		  |   |     sudo ln -s "${charybde_dir}/charybdefs" /usr/local/bin/charybdefs
		  |   |     cat > "${nemesis_path}" <<EOF
		  |   | #!/bin/bash
		  |   | cd /opt/charybdefs/cookbook
		  |   | ./recipes "\$@"
		  |   | EOF
		  |   |     chmod +x "${nemesis_path}"
		  |   | 	sudo ln -s "${nemesis_path}" /usr/local/bin/charybdefs-nemesis
		  |   | fi
		  |   |
		  |   | ```
		  | Wraps: (3) exit status 2
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		Wraps: (2) exit status 20
		Error types: (1) *main.withCommandDetails (2) *exec.ExitError

More

Artifacts: /disk-stalled/log=false,data=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

petermattis · 2020-08-25T12:43:49Z

@itsbilal Seems like we have to update the charybedfs install process.

@thtruo

…53702 #53703 #53704 #53706 53574: sql: fix race in execStmtInOpenState r=solongordon a=solongordon Fixes #53573 Release justification: low-risk update to new functionality Release note: None 53583: sql: implement `CREATE SCHEMA ... AUTHORIZATION` r=rohany a=rohany Fixes #53559. This commit adds the `CREATE SCHEMA ... AUTHORIZATION` command. When authorization is provided, the target user is given ownership of the schema. If the schema name is not provided, then the schema is named the same name as the target role. Release justification: low risk updates to new functionality Release note (sql change): Support the `CREATE SCHEMA ... AUTHORIZATION` command. 53670: vendor: Bump pebble to e6a9f9a3c936adb8ee8642e4afb2c1ff8dee4562 r=itsbilal a=itsbilal Changes pulled in: - e6a9f9a3c936adb8ee8642e4afb2c1ff8dee4562 compaction: Don't set grandparent limit "behind" keys in fragmenter - 04c6a8caf99b691124110f869ef288832cb89356 cmd/pebble: fix rocksdb build errors - 4c3a7a171486d287c872cd9bfb38fd811fb0199b internal/manifest: add clone method to iterator - 778e980e984175bcb1adb1d09c88a5e04b7762e1 internal/manifest: update FileMetadata refcounts within B-Tree - 841f44b62f4bba532d646486928a75c24b66d380 internal/manifest: prohibit overwritten keys in B-Tree - 341164583f87cb754452e7c085d22e0119f25c24 *: Add diskHealthChecking{FS,File}, use it to wrap disk writes Only changes that affect Cockroach are the first and last list items above. Release justification: Bugfix change in Pebble, plus a low-risk change for better visibility into slow disks. Release note: None. 53679: roachtest: deflake some tests r=irfansharif a=knz Fixes #53462 Fixes #53520 Release justification: non-production code changes Release note: None 53685: security: add telemetry for OCSP server checks r=irfansharif a=knz Fixes #53473 cc @thtruo This commit adds two telemetry counters: - `server.ocsp.conn-verifications` counts the number of connections for which the OCSP feature is enabled - `server.ocsp.cert-verifications` counts the number of times a certificate actually underwent OCSP verification. Release justification: low risk, high benefit changes to existing functionality Release note: None 53688: geo/geomfn: implement ST_SymmetricDifference r=otan a=CyrusJavan Added ST_SymmetricDifference builtin. ST_SymmetricDifference is simply an alias for ST_SymDifference. Release justification: low-risk update to new functionality Release note (sql change): Implemented the geometry based builtin `ST_SymmetricDifference`. Resolves: #49052 53689: roachtests: re-enable election-after-restart and kv/gracefuldraining/nodes=3 r=irfansharif a=knz Fixes #35047 Fixes #33501 **Note that I dislike the idea to run any roachtest under stressrace in the first place.** Release justification: non-production code changes Release note: None 53693: roachtest: re-enable acceptance/bank/zerosum-restart r=irfansharif a=knz Fixes #33683 This test had been flaking earlier in 2019 due to unclean graceful shutdowns. Since we've heavily reworked that logic since, let's give the test a second chance. Release justification: non-production code changes Release note: None 53695: roachtest: re-enable acceptance/bank/zerosum-splits (revert #34080) r=irfansharif a=knz First commit from #53693. PR #34080 which intended to skip `bank/zerosum-restarts`, mistakenly skipped `bank/zerosum-splits` instead. (As `zerosum-restarts` had been skipped already prior) There was no reason to skip `zerosum-splits`, so this commit re-enables it. Release justification: non-production code changes Release note: None 53702: geos: fix compile error with -Wsign-conversion r=sumeerbhola a=otan Not sure why this wasn't caught in #53647. Release justification: bug fixes and low-risk updates to new functionality Release note: None 53703: cloud: update orchestrator configs to point to v20.1.5 r=jlinder a=arulajmani Release justification: changes as part of v20.1.5 release. Release note: None 53704: sql: update `format_type` to handle user defined types r=rohany a=rohany Fixes #53684. Update the `format_type` builtin to handle user defined types. Release justification: bug fix Release note: None 53706: roachprod: Build Charybdefs against thrift 0.13 r=itsbilal a=itsbilal As we use Ubuntu 18.04 on Roachprod clusters, which has a newer version of bundler and ruby, we have to use a newer version of Thrift. Using newer Thrift also necessitates some changes in Charybdefs' build files, so I've forked it, used the fork here, and submitted the changes upstream: scylladb/charybdefs#21 Fixes #52533. Release justification: Roachtest/roachprod only fix. Release note: None. Co-authored-by: Solon Gordon <solon@cockroachlabs.com> Co-authored-by: Rohan Yadav <rohany@alumni.cmu.edu> Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com> Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net> Co-authored-by: Cyrus Javan <javan.cyrus@gmail.com> Co-authored-by: Oliver Tan <otan@cockroachlabs.com> Co-authored-by: arulajmani <arulajmani@gmail.com>

cockroach-teamcity added branch-release-20.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Aug 8, 2020

cockroach-teamcity added this to the 20.2 milestone Aug 8, 2020

darinpp mentioned this issue Aug 11, 2020

roachtest: disk-stalled/log=true,data=true failed #52397

Closed

cockroach-teamcity mentioned this issue Aug 23, 2020

roachtest: disk-stalled/log=true,data=false failed #53281

Closed

petermattis assigned itsbilal Aug 24, 2020

This was referenced Aug 24, 2020

roachtest: disk-stalled/log=false,data=false failed #53015

Closed

roachtest: disk-stalled/log=false,data=false failed #52669

Closed

itsbilal mentioned this issue Aug 31, 2020

roachprod: Build Charybdefs against thrift 0.13 #53706

Merged

craig bot closed this as completed in a8bd0df Sep 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: disk-stalled/log=false,data=true failed #52533

roachtest: disk-stalled/log=false,data=true failed #52533

cockroach-teamcity commented Aug 8, 2020

darinpp commented Aug 11, 2020

cockroach-teamcity commented Aug 13, 2020

cockroach-teamcity commented Aug 20, 2020

cockroach-teamcity commented Aug 25, 2020

petermattis commented Aug 25, 2020

roachtest: disk-stalled/log=false,data=true failed #52533

roachtest: disk-stalled/log=false,data=true failed #52533

Comments

cockroach-teamcity commented Aug 8, 2020

darinpp commented Aug 11, 2020

cockroach-teamcity commented Aug 13, 2020

cockroach-teamcity commented Aug 20, 2020

cockroach-teamcity commented Aug 25, 2020

petermattis commented Aug 25, 2020