Is there any sample app that shows those most important features comdb2 provides?

[AnDongLi]

I want to learn to know how comdb2 implements some great features such as No Concurrency, Perfect Availablity,etc. I went through the source codes, so far the closest sample I could find is the tests.

Where should I begin? Is there any design document that provide some implementation details?
If I want to debug in an IDE which file is the entry point I should set break point on?

Thanks.

[mponomar]

Implementation notes should probably be a separate document in the docs. If you just wanted a really quick summary:

1. Query starts on the replicant (handle_newsql_requests -> dispatch_sql_query -> execute_sql_query)
2. Query builds up a schedule of operations to be done as writes are requested (still on replicant) and forwards them to master (eg: sqlite3BtreeInsert -> osql_insrec)
3. Master stuffs them in a temp table until it sees a commit (osql_sess_rcvop -> osql_sess_rcvop)
4. Master applies transaction (somewhat convoluted codepath for historical reasons) with important bits being in handle_buf_main -> toblock_main -> osql_bplog_commit ... eventually calling osql_process_packet on each basic request (add/insert/delete record) forwarded from the replicant)

If you're looking for a program that demonstrates some of these features, there's nothing included in the repository, unfortunately.

[AnDongLi]

Thanks for the information, I'll look into those functions later.
What interests me in comdb2 are those 4 abstractions in Alex Scotti's slides here:
https://www.slideshare.net/AlexScotti1/allyourbase-55212398
I'd like to create a simple sample that could show how those 4 abstractions really works.
Four Abstractions:

1. Relational Model (Comdb2 use relation model? Does it use foreign key to link tables?)
2. Perfect Availablity (Client app only check return code, don't need to retry? Does a test exist?)
3. Serializable Isolation Level (Does a test exist?)
4. One big Single Computer (Does a test exist?)

How to debug in local single node?

[mponomar]

1. We support foreign keys.
2. Enable snapshot + ha_sql, run query, kill node
3. Not sure what you mean by "no concurrency"
4. You can refer to a cluster in your connect string, and the API/DB abstract details where the database runs. Is that what you mean?

[AnDongLi]

Updated "No Concurrency" to Serializable Isolation level.
4.Yes, that's what I meant. Is it implemented by modified SQLite? -- I'm reading the paper from Bloomberg
but not sure.

When you have a cluster of comdb2, how to tell which one is master?

[mponomar]

You shouldn't ever need to know who is master.
exec procedure sys.cmd.send('bdb cluster') will tell you the cluster information if you want to know, but you should never attempt to contact individual machines

[AnDongLi]

Thanks for the info. I'll check it out.

I found tests under "linearizable" is very helpful.

I tried to build the project in CLION(an Intellij C/C++ IDE) but I found I have to manully import lots of header files and there are some erros when some structure points to its field, it looks like it's related to header files is not used or marco is undefined. I wonder what IDE BloomBerg comdb2 developers used.

[mponomar]

vim, emacs, vscode, in decreasing order of popularity. The build/install instructions are your best bet for building the software.

[AnDongLi]

Yes, in command line my build and install is finished successfully.
I just want to have a way to debug.

[AnDongLi]

OK, I use Eclipse CDT to import the Makefile project and everything works!
Now I got a new issue:
When I run "docker-compose up" in contrib/docker dir, I got this error for all five nodes:
node4 | pmux: error while loading shared libraries: libprotobuf-c.so.0: cannot open shared object file: No such file or directory

I added a line in testdb.lrl of every node to make it a cluster:
cluster nodes node1 node2 node3 node4 node5

I'm in Ubuntu 14.04, I had to manully install lots of dependent packages to make the comdb2 build pass.
On my local machine, there is no problem to create a db, open a db and run cdb2sql;
When I run 'ldd comdb2' on my local machine, the output is:
linux-vdso.so.1 => (0x00007fffdafb5000)
libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f99b8c12000)
libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f99b8836000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f99b861d000)
liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f99b8414000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f99b820f000)
libprotobuf-c.so.0 => /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.0 (0x00007f99b7fff000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f99b7de1000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f99b7bd9000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f99b78d3000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f99b76cf000)
libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8 (0x00007f99b74b4000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f99b70eb000)
/lib64/ld-linux-x86-64.so.2 (0x00007f99b8e71000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f99b6ec9000)

I saw that in the Dockerfile.build it run apt-get to install libprotobuf-c1, also the image based on
xenial, not the same as my host ubuntu version(trusty).

Any idea what goes wrong?

[nirbhayc]

@AnDongLi That seems strange. Can you try rebuilding the images after switching to libprotobuf-c-dev in Dockerfile.standalone?

[AnDongLi]

same result.
In dev container, I search for libprotobuf-c* and there is no libprotobuf-c.so.0.

One thing I realize is that my host ubuntu is trusty while both Dockerfile.build and Dockerfile.standalone
are from ubuntu:xenial and I guess they install different versions of packages. Is it the reason?

Does these docker file config still work when the development host's OS is different from ubuntu:xenial?

[riverszhang89]

Can you please run make docker-cluster from the top level directory?

[nirbhayc]

In dev container, I search for libprotobuf-c* and there is no libprotobuf-c.so.0.

What was your search command? dpkg -S ? Since the standalone docker file did attempt to install libprotobuf-c, the image must have the shared object file, also evident from the ldd command output.
(though its pmux reporting the problem, not comdb2)

One thing I realize is that my host ubuntu is trusty while both Dockerfile.build and Dockerfile.standalone
are from ubuntu:xenial and I guess they install different versions of packages. Is it the reason?

I don't think version mismatch could cause this.

[AnDongLi]

@riverszhang89
Yes, I did that before, it gave the same error.

@nirbhayc
I used find to search it; Now I used dpkg in my host and in dev container, here are the outputs:
===Host output===
~$ dpkg -S libprotobuf-c*
libprotobuf-c0-dev: /usr/share/doc/libprotobuf-c0-dev
libprotobuf-c0-dev: /usr/lib/x86_64-linux-gnu/libprotobuf-c.a
libprotobuf-c0: /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.0.0.0
libprotobuf-c0: /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.0
libprotobuf-c0: /usr/share/doc/libprotobuf-c0/copyright
libprotobuf-c0: /usr/share/doc/libprotobuf-c0
libprotobuf-c0-dev: /usr/share/doc/libprotobuf-c0-dev/changelog.Debian.gz
libprotobuf-c0-dev: /usr/share/doc/libprotobuf-c0-dev/copyright
libprotobuf-c0: /usr/share/doc/libprotobuf-c0/changelog.Debian.gz
libprotobuf-c0-dev: /usr/lib/x86_64-linux-gnu/libprotobuf-c.so

===Container output===
root@dev:/# dpkg -S libprotobuf-c*
libprotobuf-c-dev: /usr/share/doc/libprotobuf-c-dev/copyright
libprotobuf-c-dev: /usr/lib/x86_64-linux-gnu/libprotobuf-c.a
libprotobuf-c1: /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.1.0.0
libprotobuf-c1: /usr/share/doc/libprotobuf-c1/copyright
libprotobuf-c-dev: /usr/lib/x86_64-linux-gnu/pkgconfig/libprotobuf-c.pc
libprotobuf-c-dev: /usr/share/doc/libprotobuf-c-dev
libprotobuf-c1: /usr/share/doc/libprotobuf-c1/changelog.Debian.gz
libprotobuf-c1: /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.1
libprotobuf-c-dev: /usr/lib/x86_64-linux-gnu/libprotobuf-c.so
libprotobuf-c1: /usr/share/doc/libprotobuf-c1
libprotobuf-c-dev: /usr/share/doc/libprotobuf-c-dev/changelog.Debian.gz

When the comdb2, cdb2sql are built in the host, do they depends on that specific version of share library? Isn't it the reason that libprotobuf-c.so.0 is asked for when I run cdb2sql in container?

[riverszhang89]

These binaries are built inside the container. Did you copy the binaries built on the host to the container?

[nirbhayc]

@AnDongLi : Looks like you built the binaries on the host, trusty (libprotobuf-c.so.0) and then trying to run it in xenial (libprotobuf-c.so.1), hence the error.

[AnDongLi]

@nirbhayc
Yes, looks like that.

@riverszhang89
I see.
Now I just did some experiment, I run 'make clean' and 'make docker-clean', and then 'make docker-cluster', I got error that hasn't been seen before:

docker build -t comdb2-build:7.0.0pre -f contrib/docker/Dockerfile.build .
Sending build context to Docker daemon 74.87 MB
Step 1 : FROM ubuntu:xenial
---> ccc7a11d65b1
Step 2 : RUN apt-get update && apt-get install -y bc bison build-essential flex gawk liblz4-dev libprotobuf-c-dev libreadline-dev libsqlite3-dev libssl-dev libunwind-dev libz-dev make ncurses-dev protobuf-c-compiler tcl uuid-dev libz1 liblz4-tool libprotobuf-c-1 libreadline6 libsqlite3-0 libuuid1 libz1 tzdata && rm -rf /var/lib/apt/lists/*
---> Running in 0e0e84f66325
Err:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Temporary failure resolving 'security.ubuntu.com'
Err:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
Temporary failure resolving 'archive.ubuntu.com'
Reading package lists...
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-updates/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-backports/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/xenial-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package bc
E: Unable to locate package bison
E: Unable to locate package build-essential
E: Unable to locate package flex
E: Unable to locate package gawk
E: Unable to locate package liblz4-dev
E: Unable to locate package libprotobuf-c-dev
E: Unable to locate package libreadline-dev
E: Unable to locate package libsqlite3-dev
E: Unable to locate package libssl-dev
E: Unable to locate package libunwind-dev
E: Unable to locate package libz-dev
E: Unable to locate package make
E: Unable to locate package ncurses-dev
E: Unable to locate package protobuf-c-compiler
E: Unable to locate package tcl
E: Unable to locate package uuid-dev
E: Unable to locate package liblz4-tool
E: Unable to locate package libprotobuf-c-1
E: Unable to locate package libsqlite3-0
E: Unable to locate package tzdata
The command '/bin/sh -c apt-get update && apt-get install -y bc bison build-essential flex gawk liblz4-dev libprotobuf-c-dev libreadline-dev libsqlite3-dev libssl-dev libunwind-dev libz-dev make ncurses-dev protobuf-c-compiler tcl uuid-dev libz1 liblz4-tool libprotobuf-c-1 libreadline6 libsqlite3-0 libuuid1 libz1 tzdata && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100
make: *** [build-build-container] Error 100

If I run 'make' first and then 'make docker-cluster' there is no such error.
I suspect make docker-cluster just copy something from the host build.

[riverszhang89]

Temporary failure resolving 'archive.ubuntu.com'

It looks like a DNS issue.

[riverszhang89]

Thank you for the valuable feedback. However, the docker discussion should be moved to a new thread. I'm going to close the issue. Feel free to open a new one and we can continue there.

[AnDongLi]

Yes, that's a Docker DNS issue, it is fixed by uncommented this line in /etc/default/docker:
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4" and then restart docker service.
Now make docker-cluster succeeded and all the nodes are up!

[riverszhang89]

Awesome! Thank you for sharing the solution with us.