Database Aborting when attempting Benchmark with OLTPBench

[saatviks]

I'm attempting to benchmark ComDB2 with OLTPBench. The table creation and data loading seems to be working for most tables. However when loading data into the tables, ComDB2 suddenly crashes with the following log:

LONG REQUEST 2056 MS logged in /Users/saatvik/comdb2/var/log/cdb2/tpcc.longreqs [SOCKSQL num=1 tot=[2 2] rtt=[2 2] rtrs=[0 0]]
LONG REQUEST 2334 MS logged in /Users/saatvik/comdb2/var/log/cdb2/tpcc.longreqs [SOCKSQL num=1 tot=[2 2] rtt=[2 2] rtrs=[0 0]]
LONG REQUEST 2070 MS logged in /Users/saatvik/comdb2/var/log/cdb2/tpcc.longreqs [SOCKSQL num=1 tot=[2 2] rtt=[2 2] rtrs=[0 0]]
LONG REQUEST 2173 MS logged in /Users/saatvik/comdb2/var/log/cdb2/tpcc.longreqs [SOCKSQL num=1 tot=[2 2] rtt=[2 2] rtrs=[0 0]]
LONG REQUEST 2274 MS logged in /Users/saatvik/comdb2/var/log/cdb2/tpcc.longreqs [SOCKSQL num=1 tot=[2 2] rtt=[2 2] rtrs=[0 0]]
LONG REQUEST 2184 MS logged in /Users/saatvik/comdb2/var/log/cdb2/tpcc.longreqs [SOCKSQL num=1 tot=[2 2] rtt=[2 2] rtrs=[0 0]]
LONG REQUEST 2264 MS logged in /Users/saatvik/comdb2/var/log/cdb2/tpcc.longreqs [SOCKSQL num=1 tot=[2 2] rtt=[2 2] rtrs=[0 0]]
LONG REQUEST 2331 MS logged in /Users/saatvik/comdb2/var/log/cdb2/tpcc.longreqs [SOCKSQL num=1 tot=[2 2] rtt=[2 2] rtrs=[0 0]]
LONG REQUEST 2125 MS logged in /Users/saatvik/comdb2/var/log/cdb2/tpcc.longreqs [SOCKSQL num=1 tot=[2 2] rtt=[2 2] rtrs=[0 0]]
LONG REQUEST 2455 MS logged in /Users/saatvik/comdb2/var/log/cdb2/tpcc.longreqs [SOCKSQL num=1 tot=[2 2] rtt=[2 2] rtrs=[0 0]]
2 LONG REQUESTS 2188 MS - 2203 MS logged in /Users/saatvik/comdb2/var/log/cdb2/tpcc.longreqs [last SOCKSQL num=1 tot=[2 2] rtt=[2 2] rtrs=[0 0]]
LONG REQUEST 2000 MS logged in /Users/saatvik/comdb2/var/log/cdb2/tpcc.longreqs [SOCKSQL num=1 tot=[2 2] rtt=[2 2] rtrs=[0 0]]
[FATAL] **aborting due thd_req mismatch thd id 0 (my thd 4893118464)
Abort trap: 6

I'll add the Java Exception stack trace as well in a bit.
Here is the Java Stack Trace:

java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
	at java.util.ArrayList.rangeCheck(ArrayList.java:653)
	at java.util.ArrayList.get(ArrayList.java:429)
	at com.bloomberg.comdb2.jdbc.Comdb2Handle.reopen(Comdb2Handle.java:1788)
	at com.bloomberg.comdb2.jdbc.Comdb2Handle.open(Comdb2Handle.java:1618)
	at com.bloomberg.comdb2.jdbc.Comdb2Handle.runStatementInt(Comdb2Handle.java:911)
	at com.bloomberg.comdb2.jdbc.Comdb2Handle.runStatement(Comdb2Handle.java:1342)
	at com.bloomberg.comdb2.jdbc.Comdb2Handle.runStatement(Comdb2Handle.java:751)
	at com.bloomberg.comdb2.jdbc.Comdb2Statement.executeQuery(Comdb2Statement.java:118)
	at com.bloomberg.comdb2.jdbc.Comdb2Statement.executeQuery(Comdb2Statement.java:70)
	at com.bloomberg.comdb2.jdbc.Comdb2Statement.executeUpdate(Comdb2Statement.java:157)
	at com.bloomberg.comdb2.jdbc.Comdb2PreparedStatement.executeUpdate(Comdb2PreparedStatement.java:132)
	at com.bloomberg.comdb2.jdbc.Comdb2PreparedStatement.executeBatch(Comdb2PreparedStatement.java:437)
	at com.oltpbenchmark.benchmarks.tpcc.TPCCLoader.loadOrders(TPCCLoader.java:668)
	at com.oltpbenchmark.benchmarks.tpcc.TPCCLoader$2.load(TPCCLoader.java:121)
	at com.oltpbenchmark.api.Loader$LoaderThread.run(Loader.java:65)
	at com.oltpbenchmark.util.ThreadUtil$LatchRunnable.run(ThreadUtil.java:343)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

[akshatsikarwar]

The macOS build is experimental and not as mature at this point. I suggest trying on a Linux machine for the rest of your project. Meanwhile, steps to reproduce this abort will be useful.

[saatviks]

Reproducing this error would need a lot of configuration files be setup and many changes in OLTPBench source. I'm not sure how to provide a straightforward way to reproduce - Any suggestions?

[akshatsikarwar]

Possible to share your changes on github? Feel free to dm me or reach out on hangouts so we can debug quickly.

[adizaimi]

Did the abort produce a core? It would be quite useful to get the stack trace of all the running threads in that core, if it is possible.

[saatviks]

This issue has been resolved - Im sorry I should close it - though I am still having Serializability issues with ComDB2 - I'll add an issue about that.

[akshatsikarwar]

We discussed offline. I don't have a way to reproduce, but @saatviks sent me the backtraces. I still need to fix this. Seems to affect macOS only.

[akshatsikarwar]

bt.txt

[akshatsikarwar]

bt all.txt

[riverszhang89]

The IndexOutOfBoundsException exception may also be worth looking into.