Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using zetcd instead Zookeeper #777

Closed
Slach opened this issue May 9, 2017 · 17 comments
Closed

Using zetcd instead Zookeeper #777

Slach opened this issue May 9, 2017 · 17 comments
Labels
comp-zookeeper Zookeeper st-community-taken External developer is working on that

Comments

@Slach
Copy link
Contributor

Slach commented May 9, 2017

i create repo
https://github.com/Slach/clickhouse-zetcd
for reproducing stand clickhouse + zetcd
latest clickhouse Dockerimage core dumped after run following query

sudo docker-compose run clickhouse-client.local -h clickhouse-ru-1.local -q "CREATE TABLE IF NOT EXISTS zetcd_test.test_replicated (timestamp UInt32, date MATERIALIZED toDate(timestamp), trackerId String,   userId String  ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/hits_replicated', '{replica}', date, cityHash64( userId), (trackerId, date, cityHash64(userId), timestamp), 8192);"

in docker-compose logs clickhouse-ru-1.local i see

2017.05.09 17:54:32.706519 [ 1 ] <Information> : Starting daemon with revision 54198

clickhouse-ru-1.local_1    | 2017.05.09 17:56:46.979663 [ 4 ] <Debug> executeQuery: (from 172.18.0.10:38834) CREATE TABLE IF NOT EXISTS zetcd_test.test_replicated (timestamp UInt32, date MATERIALIZED toDate(timestamp), trackerId String,   userId String  ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/hits_replicated', '{replica}', date, cityHash64( userId), (trackerId, date, cityHash64(userId), timestamp), 8192)
clickhouse-ru-1.local_1    | 2017.05.09 17:56:46.980335 [ 4 ] <Information> BackgroundProcessingPool: Create BackgroundProcessingPool with 16 threads
clickhouse-ru-1.local_1    | 2017.05.09 17:56:46.983051 [ 4 ] <Debug> zetcd_test.test_replicated (Data): Loading data parts
clickhouse-ru-1.local_1    | 2017.05.09 17:56:46.983116 [ 4 ] <Debug> zetcd_test.test_replicated (Data): Loaded data parts (0 items)
clickhouse-ru-1.local_1    | 2017.05.09 17:56:46.984556 [ 4 ] <Debug> zetcd_test.test_replicated (StorageReplicatedMergeTree): Creating table /clickhouse/tables/1/hits_replicated
clickhouse-ru-1.local_1    | /home/robot-metrika-test/jenkins/workspace/clickhouse-packages-build@2/sources/contrib/libtcmalloc/src/tcmalloc.cc:286] Attempt to free invalid pointer 0x7f98402a4b60
clickhouse-ru-1.local_1    | 2017.05.09 17:56:47.003653 [ 6 ] <Error> BaseDaemon: ########################################
clickhouse-ru-1.local_1    | 2017.05.09 17:56:47.003685 [ 6 ] <Error> BaseDaemon: (from thread 5) Received signal Aborted (6).
clickhouse-ru-1.local_1    | 2017.05.09 17:56:47.040628 [ 6 ] <Error> BaseDaemon: 1. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7f9842740428]
clickhouse-ru-1.local_1    | 2017.05.09 17:56:47.040660 [ 6 ] <Error> BaseDaemon: 2. /usr/bin/clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0xa5) [0x33fb245]
clickhouse-ru-1.local_1    | 2017.05.09 17:56:47.040664 [ 6 ] <Error> BaseDaemon: 3. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f98431f06ba]
clickhouse-ru-1.local_1    | 2017.05.09 17:56:47.040667 [ 6 ] <Error> BaseDaemon: 4. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f984281182d]
clickhouse-ru-1.local_1    | Aborted (core dumped)

and zktraffic container
show following logs

17:54:32:723097 ConnectRequest(ver=0, zxid=0, timeout=30000, session=0x0, readonly=False, client=172.18.0.4:39578)
————►17:54:32:723724 ConnectReply(ver=0, timeout=30000, session=0x74085bee5926d104, readonly=False, server=172.18.0.3:2181)
17:54:32:733217 CreateRequest(size=0, xid=1494352473, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:734662 CreateReply(xid=1494352473, zxid=2, error=0, path=/clickhouse, server=172.18.0.3:2181)
17:54:32:734776 CreateRequest(size=0, xid=1494352474, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:735977 CreateReply(xid=1494352474, zxid=3, error=0, path=/clickhouse/task_queue, server=172.18.0.3:2181)
17:54:32:736095 CreateRequest(size=0, xid=1494352475, path=/clickhouse/task_queue/resharding, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:739219 CreateReply(xid=1494352475, zxid=4, error=0, path=/clickhouse/task_queue/resharding, server=172.18.0.3:2181)
17:54:32:742606 CreateRequest(size=0, xid=1494352476, path=/clickhouse/task_queue/resharding/2ed7cf718922, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:746256 CreateReply(xid=1494352476, zxid=5, error=0, path=/clickhouse/task_queue/resharding/2ed7cf718922, server=172.18.0.3:2181)
17:54:32:746556 CreateRequest(size=0, xid=1494352477, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:747985 FailedCreateReply(xid=1494352477, zxid=6, error=-110, path=, server=172.18.0.3:2181)
17:54:32:748271 CreateRequest(size=0, xid=1494352478, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:757310 FailedCreateReply(xid=1494352478, zxid=7, error=-110, path=, server=172.18.0.3:2181)
17:54:32:751792 ConnectRequest(ver=0, zxid=0, timeout=30000, session=0x0, readonly=False, client=172.18.0.5:36334)
————►17:54:32:758219 ConnectReply(ver=0, timeout=30000, session=0x74085bee5926d112, readonly=False, server=172.18.0.3:2181)
17:54:32:758356 CreateRequest(size=0, xid=1494352473, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:759921 FailedCreateReply(xid=1494352473, zxid=9, error=-110, path=, server=172.18.0.3:2181)
17:54:32:757654 CreateRequest(size=0, xid=1494352479, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:760829 CreateReply(xid=1494352479, zxid=10, error=0, path=/clickhouse/task_queue/resharding_distributed, server=172.18.0.3:2181)
17:54:32:759980 CreateRequest(size=0, xid=1494352474, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:769516 FailedCreateReply(xid=1494352474, zxid=11, error=-110, path=, server=172.18.0.3:2181)
17:54:32:761023 CreateRequest(size=0, xid=1494352480, path=/clickhouse/task_queue/resharding_distributed/online, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:770649 CreateReply(xid=1494352480, zxid=12, error=0, path=/clickhouse/task_queue/resharding_distributed/online, server=172.18.0.3:2181)
17:54:32:770889 CreateRequest(size=0, xid=1494352481, path=/clickhouse/task_queue/resharding_distributed/online/2ed7cf718922, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:779403 CreateReply(xid=1494352481, zxid=13, error=0, path=/clickhouse/task_queue/resharding_distributed/online/2ed7cf718922, server=172.18.0.3:2181)
17:54:32:770894 CreateRequest(size=0, xid=1494352475, path=/clickhouse/task_queue/resharding, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:779860 FailedCreateReply(xid=1494352475, zxid=14, error=-110, path=, server=172.18.0.3:2181)
17:54:32:779658 CreateRequest(size=0, xid=1494352482, path=/clickhouse/task_queue/resharding_distributed/lock, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:782157 CreateReply(xid=1494352482, zxid=15, error=0, path=/clickhouse/task_queue/resharding_distributed/lock, server=172.18.0.3:2181)
17:54:32:782274 CreateRequest(size=0, xid=1494352483, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:785940 FailedCreateReply(xid=1494352483, zxid=16, error=-110, path=, server=172.18.0.3:2181)
17:54:32:785207 CreateRequest(size=0, xid=1494352476, path=/clickhouse/task_queue/resharding/3074d827f15e, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:787088 CreateReply(xid=1494352476, zxid=17, error=0, path=/clickhouse/task_queue/resharding/3074d827f15e, server=172.18.0.3:2181)
17:54:32:787177 CreateRequest(size=0, xid=1494352477, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:789305 FailedCreateReply(xid=1494352477, zxid=19, error=-110, path=, server=172.18.0.3:2181)
17:54:32:786246 CreateRequest(size=0, xid=1494352484, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:789376 FailedCreateReply(xid=1494352484, zxid=18, error=-110, path=, server=172.18.0.3:2181)
17:54:32:789405 CreateRequest(size=0, xid=1494352478, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:792920 FailedCreateReply(xid=1494352478, zxid=20, error=-110, path=, server=172.18.0.3:2181)
17:54:32:789567 CreateRequest(size=0, xid=1494352485, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:793309 FailedCreateReply(xid=1494352485, zxid=21, error=-110, path=, server=172.18.0.3:2181)
17:54:32:793481 CreateRequest(size=0, xid=1494352486, path=/clickhouse/task_queue/resharding_distributed/coordination, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:796984 CreateReply(xid=1494352486, zxid=22, error=0, path=/clickhouse/task_queue/resharding_distributed/coordination, server=172.18.0.3:2181)
17:54:32:803827 CreateRequest(size=0, xid=1494352481, path=/clickhouse/task_queue/resharding_distributed/online/3074d827f15e, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:807504 CreateReply(xid=1494352481, zxid=25, error=0, path=/clickhouse/task_queue/resharding_distributed/online/3074d827f15e, server=172.18.0.3:2181)
17:54:32:807574 CreateRequest(size=0, xid=1494352482, path=/clickhouse/task_queue/resharding_distributed/lock, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:816901 FailedCreateReply(xid=1494352482, zxid=26, error=-110, path=, server=172.18.0.3:2181)
17:54:32:818265 CreateRequest(size=0, xid=1494352483, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:819282 FailedCreateReply(xid=1494352483, zxid=27, error=-110, path=, server=172.18.0.3:2181)
17:54:32:819365 CreateRequest(size=0, xid=1494352484, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:820434 FailedCreateReply(xid=1494352484, zxid=28, error=-110, path=, server=172.18.0.3:2181)
17:54:32:972017 ConnectRequest(ver=0, zxid=0, timeout=30000, session=0x0, readonly=False, client=172.18.0.6:44866)
————►17:54:33:048144 ConnectReply(ver=0, timeout=30000, session=0x74085bee5926d142, readonly=False, server=172.18.0.3:2181)
17:54:33:048408 CreateRequest(size=0, xid=1494352473, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:049644 FailedCreateReply(xid=1494352473, zxid=32, error=-110, path=, server=172.18.0.3:2181)
17:54:33:050483 CreateRequest(size=0, xid=1494352474, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:051912 FailedCreateReply(xid=1494352474, zxid=33, error=-110, path=, server=172.18.0.3:2181)
17:54:33:051966 CreateRequest(size=0, xid=1494352475, path=/clickhouse/task_queue/resharding, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:053203 FailedCreateReply(xid=1494352475, zxid=34, error=-110, path=, server=172.18.0.3:2181)
17:54:33:053263 CreateRequest(size=0, xid=1494352476, path=/clickhouse/task_queue/resharding/1ee79866c57e, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:054653 CreateReply(xid=1494352476, zxid=35, error=0, path=/clickhouse/task_queue/resharding/1ee79866c57e, server=172.18.0.3:2181)
17:54:33:054714 CreateRequest(size=0, xid=1494352477, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:057955 FailedCreateReply(xid=1494352477, zxid=36, error=-110, path=, server=172.18.0.3:2181)
17:54:33:058027 CreateRequest(size=0, xid=1494352478, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:061401 FailedCreateReply(xid=1494352478, zxid=37, error=-110, path=, server=172.18.0.3:2181)
17:54:33:061473 CreateRequest(size=0, xid=1494352479, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:063499 FailedCreateReply(xid=1494352479, zxid=38, error=-110, path=, server=172.18.0.3:2181)
17:54:33:063558 CreateRequest(size=0, xid=1494352480, path=/clickhouse/task_queue/resharding_distributed/online, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:067791 FailedCreateReply(xid=1494352480, zxid=39, error=-110, path=, server=172.18.0.3:2181)
17:54:33:067863 CreateRequest(size=0, xid=1494352481, path=/clickhouse/task_queue/resharding_distributed/online/1ee79866c57e, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:069866 CreateReply(xid=1494352481, zxid=40, error=0, path=/clickhouse/task_queue/resharding_distributed/online/1ee79866c57e, server=172.18.0.3:2181)
17:54:33:069934 CreateRequest(size=0, xid=1494352482, path=/clickhouse/task_queue/resharding_distributed/lock, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:127621 FailedCreateReply(xid=1494352482, zxid=41, error=-110, path=, server=172.18.0.3:2181)
17:54:33:127689 CreateRequest(size=0, xid=1494352483, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:128840 FailedCreateReply(xid=1494352483, zxid=42, error=-110, path=, server=172.18.0.3:2181)
17:54:33:128885 CreateRequest(size=0, xid=1494352484, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:129947 FailedCreateReply(xid=1494352484, zxid=43, error=-110, path=, server=172.18.0.3:2181)
17:54:33:129992 CreateRequest(size=0, xid=1494352485, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:131275 FailedCreateReply(xid=1494352485, zxid=44, error=-110, path=, server=172.18.0.3:2181)
17:54:33:131321 CreateRequest(size=0, xid=1494352486, path=/clickhouse/task_queue/resharding_distributed/coordination, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:132699 FailedCreateReply(xid=1494352486, zxid=45, error=-110, path=, server=172.18.0.3:2181)
17:54:33:132858 GetChildren2Request(xid=1494352487, path=/clickhouse/task_queue/resharding/1ee79866c57e, watch=False, size=0, client=172.18.0.6:44866)
————►17:54:33:133600 GetChildren2Reply(xid=1494352487, zxid=45, error=0, count=0, server=172.18.0.3:2181)
17:54:33:133650 GetChildren2Request(xid=1494352488, path=/clickhouse/task_queue/resharding/1ee79866c57e, watch=True, size=0, client=172.18.0.6:44866)
————►17:54:33:134170 GetChildren2Reply(xid=1494352488, zxid=45, error=0, count=0, server=172.18.0.3:2181)
17:54:33:484583 ConnectRequest(ver=0, zxid=0, timeout=30000, session=0x0, readonly=False, client=172.18.0.7:39238)
————►17:54:33:486373 ConnectReply(ver=0, timeout=30000, session=0x74085bee5926d161, readonly=False, server=172.18.0.3:2181)
17:54:33:500355 CreateRequest(size=0, xid=1494352474, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:502105 FailedCreateReply(xid=1494352474, zxid=47, error=-110, path=, server=172.18.0.3:2181)
17:54:33:502732 CreateRequest(size=0, xid=1494352475, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:504349 FailedCreateReply(xid=1494352475, zxid=48, error=-110, path=, server=172.18.0.3:2181)
17:54:33:504511 CreateRequest(size=0, xid=1494352476, path=/clickhouse/task_queue/resharding, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:505941 FailedCreateReply(xid=1494352476, zxid=49, error=-110, path=, server=172.18.0.3:2181)
17:54:33:506075 CreateRequest(size=0, xid=1494352477, path=/clickhouse/task_queue/resharding/655dcde35ccc, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:507706 CreateReply(xid=1494352477, zxid=50, error=0, path=/clickhouse/task_queue/resharding/655dcde35ccc, server=172.18.0.3:2181)
17:54:33:507884 CreateRequest(size=0, xid=1494352478, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:509274 FailedCreateReply(xid=1494352478, zxid=51, error=-110, path=, server=172.18.0.3:2181)
17:54:33:509376 CreateRequest(size=0, xid=1494352479, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:510983 FailedCreateReply(xid=1494352479, zxid=52, error=-110, path=, server=172.18.0.3:2181)
17:54:33:511183 CreateRequest(size=0, xid=1494352480, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:513039 FailedCreateReply(xid=1494352480, zxid=53, error=-110, path=, server=172.18.0.3:2181)
17:54:33:513145 CreateRequest(size=0, xid=1494352481, path=/clickhouse/task_queue/resharding_distributed/online, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:514593 FailedCreateReply(xid=1494352481, zxid=54, error=-110, path=, server=172.18.0.3:2181)
17:54:33:514730 CreateRequest(size=0, xid=1494352482, path=/clickhouse/task_queue/resharding_distributed/online/655dcde35ccc, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:516409 CreateReply(xid=1494352482, zxid=55, error=0, path=/clickhouse/task_queue/resharding_distributed/online/655dcde35ccc, server=172.18.0.3:2181)
17:54:33:516594 CreateRequest(size=0, xid=1494352483, path=/clickhouse/task_queue/resharding_distributed/lock, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:520364 FailedCreateReply(xid=1494352483, zxid=56, error=-110, path=, server=172.18.0.3:2181)
17:54:33:520530 CreateRequest(size=0, xid=1494352484, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:521882 FailedCreateReply(xid=1494352484, zxid=57, error=-110, path=, server=172.18.0.3:2181)
17:54:33:522066 CreateRequest(size=0, xid=1494352485, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:526435 FailedCreateReply(xid=1494352485, zxid=58, error=-110, path=, server=172.18.0.3:2181)
17:54:33:526582 CreateRequest(size=0, xid=1494352486, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:530760 FailedCreateReply(xid=1494352486, zxid=59, error=-110, path=, server=172.18.0.3:2181)
17:54:33:530875 CreateRequest(size=0, xid=1494352487, path=/clickhouse/task_queue/resharding_distributed/coordination, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:532444 FailedCreateReply(xid=1494352487, zxid=60, error=-110, path=, server=172.18.0.3:2181)
17:54:33:532644 GetChildren2Request(xid=1494352488, path=/clickhouse/task_queue/resharding/655dcde35ccc, watch=False, size=0, client=172.18.0.7:39238)
————►17:54:33:534792 GetChildren2Reply(xid=1494352488, zxid=60, error=0, count=0, server=172.18.0.3:2181)
17:54:33:534876 GetChildren2Request(xid=1494352489, path=/clickhouse/task_queue/resharding/655dcde35ccc, watch=True, size=0, client=172.18.0.7:39238)
————►17:54:33:535667 GetChildren2Reply(xid=1494352489, zxid=60, error=0, count=0, server=172.18.0.3:2181)
17:54:33:551575 ConnectRequest(ver=0, zxid=0, timeout=30000, session=0x0, readonly=False, client=172.18.0.8:50670)
————►17:54:33:552834 ConnectReply(ver=0, timeout=30000, session=0x74085bee5926d180, readonly=False, server=172.18.0.3:2181)
17:54:33:554481 CreateRequest(size=0, xid=1494352474, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:555888 FailedCreateReply(xid=1494352474, zxid=62, error=-110, path=, server=172.18.0.3:2181)
17:54:33:555943 CreateRequest(size=0, xid=1494352475, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:557797 FailedCreateReply(xid=1494352475, zxid=63, error=-110, path=, server=172.18.0.3:2181)
17:54:33:557889 CreateRequest(size=0, xid=1494352476, path=/clickhouse/task_queue/resharding, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:559473 FailedCreateReply(xid=1494352476, zxid=64, error=-110, path=, server=172.18.0.3:2181)
17:54:33:559568 CreateRequest(size=0, xid=1494352477, path=/clickhouse/task_queue/resharding/89515b3d7f8b, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:560953 CreateReply(xid=1494352477, zxid=65, error=0, path=/clickhouse/task_queue/resharding/89515b3d7f8b, server=172.18.0.3:2181)
17:54:33:561020 CreateRequest(size=0, xid=1494352478, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:562095 FailedCreateReply(xid=1494352478, zxid=66, error=-110, path=, server=172.18.0.3:2181)
17:54:33:562191 CreateRequest(size=0, xid=1494352479, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:563304 FailedCreateReply(xid=1494352479, zxid=67, error=-110, path=, server=172.18.0.3:2181)
17:54:33:563398 CreateRequest(size=0, xid=1494352480, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:564441 FailedCreateReply(xid=1494352480, zxid=68, error=-110, path=, server=172.18.0.3:2181)
17:54:33:564536 CreateRequest(size=0, xid=1494352481, path=/clickhouse/task_queue/resharding_distributed/online, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:566551 FailedCreateReply(xid=1494352481, zxid=69, error=-110, path=, server=172.18.0.3:2181)
17:54:33:566666 CreateRequest(size=0, xid=1494352482, path=/clickhouse/task_queue/resharding_distributed/online/89515b3d7f8b, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:567772 CreateReply(xid=1494352482, zxid=70, error=0, path=/clickhouse/task_queue/resharding_distributed/online/89515b3d7f8b, server=172.18.0.3:2181)
17:54:33:567840 CreateRequest(size=0, xid=1494352483, path=/clickhouse/task_queue/resharding_distributed/lock, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:569151 FailedCreateReply(xid=1494352483, zxid=71, error=-110, path=, server=172.18.0.3:2181)
17:54:33:571107 CreateRequest(size=0, xid=1494352484, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:572133 FailedCreateReply(xid=1494352484, zxid=72, error=-110, path=, server=172.18.0.3:2181)
17:54:33:572270 CreateRequest(size=0, xid=1494352485, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:573614 FailedCreateReply(xid=1494352485, zxid=73, error=-110, path=, server=172.18.0.3:2181)
17:54:33:573715 CreateRequest(size=0, xid=1494352486, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:576892 FailedCreateReply(xid=1494352486, zxid=74, error=-110, path=, server=172.18.0.3:2181)
17:54:33:576995 CreateRequest(size=0, xid=1494352487, path=/clickhouse/task_queue/resharding_distributed/coordination, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:578439 FailedCreateReply(xid=1494352487, zxid=75, error=-110, path=, server=172.18.0.3:2181)
17:54:33:578866 GetChildren2Request(xid=1494352488, path=/clickhouse/task_queue/resharding/89515b3d7f8b, watch=False, size=0, client=172.18.0.8:50670)
————►17:54:33:580944 GetChildren2Reply(xid=1494352488, zxid=75, error=0, count=0, server=172.18.0.3:2181)
17:54:33:581393 GetChildren2Request(xid=1494352489, path=/clickhouse/task_queue/resharding/89515b3d7f8b, watch=True, size=0, client=172.18.0.8:50670)
————►17:54:33:582713 GetChildren2Reply(xid=1494352489, zxid=75, error=0, count=0, server=172.18.0.3:2181)
17:54:34:512765 ConnectRequest(ver=0, zxid=0, timeout=30000, session=0x0, readonly=False, client=172.18.0.9:56390)
————►17:54:34:521424 ConnectReply(ver=0, timeout=30000, session=0x74085bee5926d19f, readonly=False, server=172.18.0.3:2181)
17:54:34:521899 CreateRequest(size=0, xid=1494352475, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:525570 FailedCreateReply(xid=1494352475, zxid=77, error=-110, path=, server=172.18.0.3:2181)
17:54:34:525649 CreateRequest(size=0, xid=1494352476, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:526771 FailedCreateReply(xid=1494352476, zxid=78, error=-110, path=, server=172.18.0.3:2181)
17:54:34:526829 CreateRequest(size=0, xid=1494352477, path=/clickhouse/task_queue/resharding, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:527872 FailedCreateReply(xid=1494352477, zxid=79, error=-110, path=, server=172.18.0.3:2181)
17:54:34:527929 CreateRequest(size=0, xid=1494352478, path=/clickhouse/task_queue/resharding/6974dc99f908, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:529064 CreateReply(xid=1494352478, zxid=80, error=0, path=/clickhouse/task_queue/resharding/6974dc99f908, server=172.18.0.3:2181)
17:54:34:529125 CreateRequest(size=0, xid=1494352479, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:529969 FailedCreateReply(xid=1494352479, zxid=81, error=-110, path=, server=172.18.0.3:2181)
17:54:34:530030 CreateRequest(size=0, xid=1494352480, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:531215 FailedCreateReply(xid=1494352480, zxid=82, error=-110, path=, server=172.18.0.3:2181)
17:54:34:531273 CreateRequest(size=0, xid=1494352481, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:532346 FailedCreateReply(xid=1494352481, zxid=83, error=-110, path=, server=172.18.0.3:2181)
17:54:34:532477 CreateRequest(size=0, xid=1494352482, path=/clickhouse/task_queue/resharding_distributed/online, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:535046 FailedCreateReply(xid=1494352482, zxid=84, error=-110, path=, server=172.18.0.3:2181)
17:54:34:535123 CreateRequest(size=0, xid=1494352483, path=/clickhouse/task_queue/resharding_distributed/online/6974dc99f908, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:536696 CreateReply(xid=1494352483, zxid=85, error=0, path=/clickhouse/task_queue/resharding_distributed/online/6974dc99f908, server=172.18.0.3:2181)
17:54:34:536798 CreateRequest(size=0, xid=1494352484, path=/clickhouse/task_queue/resharding_distributed/lock, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:541213 FailedCreateReply(xid=1494352484, zxid=86, error=-110, path=, server=172.18.0.3:2181)
17:54:34:541289 CreateRequest(size=0, xid=1494352485, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:543614 FailedCreateReply(xid=1494352485, zxid=87, error=-110, path=, server=172.18.0.3:2181)
17:54:34:543666 CreateRequest(size=0, xid=1494352486, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:544954 FailedCreateReply(xid=1494352486, zxid=88, error=-110, path=, server=172.18.0.3:2181)
17:54:34:545003 CreateRequest(size=0, xid=1494352487, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:546192 FailedCreateReply(xid=1494352487, zxid=89, error=-110, path=, server=172.18.0.3:2181)
17:54:34:546240 CreateRequest(size=0, xid=1494352488, path=/clickhouse/task_queue/resharding_distributed/coordination, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:548786 FailedCreateReply(xid=1494352488, zxid=90, error=-110, path=, server=172.18.0.3:2181)
17:54:34:549071 GetChildren2Request(xid=1494352489, path=/clickhouse/task_queue/resharding/6974dc99f908, watch=False, size=0, client=172.18.0.9:56390)
————►17:54:34:549863 GetChildren2Reply(xid=1494352489, zxid=90, error=0, count=0, server=172.18.0.3:2181)
17:54:34:549917 GetChildren2Request(xid=1494352490, path=/clickhouse/task_queue/resharding/6974dc99f908, watch=True, size=0, client=172.18.0.9:56390)
————►17:54:34:550566 GetChildren2Reply(xid=1494352490, zxid=90, error=0, count=0, server=172.18.0.3:2181)
17:56:46:981019 ExistsRequest(xid=1494352489, path=/clickhouse/tables/1/hits_replicated/replicas/clickhouse-ru-1.local/flags/force_restore_data, watch=False, size=0, client=172.18.0.4:39578)
————►17:56:46:982127 FailedExistsReply(xid=1494352489, zxid=90, error=-101, server=172.18.0.3:2181)
17:56:46:983186 ExistsRequest(xid=1494352490, path=/clickhouse/tables/1/hits_replicated, watch=False, size=0, client=172.18.0.4:39578)
————►17:56:46:983957 FailedExistsReply(xid=1494352490, zxid=90, error=-101, server=172.18.0.3:2181)
17:56:46:984629 CreateRequest(size=0, xid=1494352491, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:56:46:986060 FailedCreateReply(xid=1494352491, zxid=91, error=-110, path=, server=172.18.0.3:2181)
17:56:46:987046 CreateRequest(size=0, xid=1494352492, path=/clickhouse/tables, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:56:46:993787 CreateReply(xid=1494352492, zxid=92, error=0, path=/clickhouse/tables, server=172.18.0.3:2181)
17:56:46:993953 CreateRequest(size=0, xid=1494352493, path=/clickhouse/tables/1, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:56:46:995372 CreateReply(xid=1494352493, zxid=93, error=0, path=/clickhouse/tables/1, server=172.18.0.3:2181)zktraffic_1                | 17:54:32:723097 ConnectRequest(ver=0, zxid=0, timeout=30000, session=0x0, readonly=False, client=172.18.0.4:39578)
————►17:54:32:723724 ConnectReply(ver=0, timeout=30000, session=0x74085bee5926d104, readonly=False, server=172.18.0.3:2181)
17:54:32:733217 CreateRequest(size=0, xid=1494352473, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:734662 CreateReply(xid=1494352473, zxid=2, error=0, path=/clickhouse, server=172.18.0.3:2181)
17:54:32:734776 CreateRequest(size=0, xid=1494352474, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:735977 CreateReply(xid=1494352474, zxid=3, error=0, path=/clickhouse/task_queue, server=172.18.0.3:2181)
17:54:32:736095 CreateRequest(size=0, xid=1494352475, path=/clickhouse/task_queue/resharding, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:739219 CreateReply(xid=1494352475, zxid=4, error=0, path=/clickhouse/task_queue/resharding, server=172.18.0.3:2181)
17:54:32:742606 CreateRequest(size=0, xid=1494352476, path=/clickhouse/task_queue/resharding/2ed7cf718922, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:746256 CreateReply(xid=1494352476, zxid=5, error=0, path=/clickhouse/task_queue/resharding/2ed7cf718922, server=172.18.0.3:2181)
17:54:32:746556 CreateRequest(size=0, xid=1494352477, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:747985 FailedCreateReply(xid=1494352477, zxid=6, error=-110, path=, server=172.18.0.3:2181)
17:54:32:748271 CreateRequest(size=0, xid=1494352478, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:757310 FailedCreateReply(xid=1494352478, zxid=7, error=-110, path=, server=172.18.0.3:2181)
17:54:32:751792 ConnectRequest(ver=0, zxid=0, timeout=30000, session=0x0, readonly=False, client=172.18.0.5:36334)
————►17:54:32:758219 ConnectReply(ver=0, timeout=30000, session=0x74085bee5926d112, readonly=False, server=172.18.0.3:2181)
17:54:32:758356 CreateRequest(size=0, xid=1494352473, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:759921 FailedCreateReply(xid=1494352473, zxid=9, error=-110, path=, server=172.18.0.3:2181)
17:54:32:757654 CreateRequest(size=0, xid=1494352479, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:760829 CreateReply(xid=1494352479, zxid=10, error=0, path=/clickhouse/task_queue/resharding_distributed, server=172.18.0.3:2181)
17:54:32:759980 CreateRequest(size=0, xid=1494352474, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:769516 FailedCreateReply(xid=1494352474, zxid=11, error=-110, path=, server=172.18.0.3:2181)
17:54:32:761023 CreateRequest(size=0, xid=1494352480, path=/clickhouse/task_queue/resharding_distributed/online, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:770649 CreateReply(xid=1494352480, zxid=12, error=0, path=/clickhouse/task_queue/resharding_distributed/online, server=172.18.0.3:2181)
17:54:32:770889 CreateRequest(size=0, xid=1494352481, path=/clickhouse/task_queue/resharding_distributed/online/2ed7cf718922, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:779403 CreateReply(xid=1494352481, zxid=13, error=0, path=/clickhouse/task_queue/resharding_distributed/online/2ed7cf718922, server=172.18.0.3:2181)
17:54:32:770894 CreateRequest(size=0, xid=1494352475, path=/clickhouse/task_queue/resharding, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:779860 FailedCreateReply(xid=1494352475, zxid=14, error=-110, path=, server=172.18.0.3:2181)
17:54:32:779658 CreateRequest(size=0, xid=1494352482, path=/clickhouse/task_queue/resharding_distributed/lock, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:782157 CreateReply(xid=1494352482, zxid=15, error=0, path=/clickhouse/task_queue/resharding_distributed/lock, server=172.18.0.3:2181)
17:54:32:782274 CreateRequest(size=0, xid=1494352483, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:785940 FailedCreateReply(xid=1494352483, zxid=16, error=-110, path=, server=172.18.0.3:2181)
17:54:32:785207 CreateRequest(size=0, xid=1494352476, path=/clickhouse/task_queue/resharding/3074d827f15e, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:787088 CreateReply(xid=1494352476, zxid=17, error=0, path=/clickhouse/task_queue/resharding/3074d827f15e, server=172.18.0.3:2181)
17:54:32:787177 CreateRequest(size=0, xid=1494352477, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:789305 FailedCreateReply(xid=1494352477, zxid=19, error=-110, path=, server=172.18.0.3:2181)
17:54:32:786246 CreateRequest(size=0, xid=1494352484, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:789376 FailedCreateReply(xid=1494352484, zxid=18, error=-110, path=, server=172.18.0.3:2181)
17:54:32:789405 CreateRequest(size=0, xid=1494352478, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:792920 FailedCreateReply(xid=1494352478, zxid=20, error=-110, path=, server=172.18.0.3:2181)
17:54:32:789567 CreateRequest(size=0, xid=1494352485, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:793309 FailedCreateReply(xid=1494352485, zxid=21, error=-110, path=, server=172.18.0.3:2181)
17:54:32:793481 CreateRequest(size=0, xid=1494352486, path=/clickhouse/task_queue/resharding_distributed/coordination, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:54:32:796984 CreateReply(xid=1494352486, zxid=22, error=0, path=/clickhouse/task_queue/resharding_distributed/coordination, server=172.18.0.3:2181)
17:54:32:803827 CreateRequest(size=0, xid=1494352481, path=/clickhouse/task_queue/resharding_distributed/online/3074d827f15e, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:807504 CreateReply(xid=1494352481, zxid=25, error=0, path=/clickhouse/task_queue/resharding_distributed/online/3074d827f15e, server=172.18.0.3:2181)
17:54:32:807574 CreateRequest(size=0, xid=1494352482, path=/clickhouse/task_queue/resharding_distributed/lock, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:816901 FailedCreateReply(xid=1494352482, zxid=26, error=-110, path=, server=172.18.0.3:2181)
17:54:32:818265 CreateRequest(size=0, xid=1494352483, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:819282 FailedCreateReply(xid=1494352483, zxid=27, error=-110, path=, server=172.18.0.3:2181)
17:54:32:819365 CreateRequest(size=0, xid=1494352484, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.5:36334)
————►17:54:32:820434 FailedCreateReply(xid=1494352484, zxid=28, error=-110, path=, server=172.18.0.3:2181)
17:54:32:972017 ConnectRequest(ver=0, zxid=0, timeout=30000, session=0x0, readonly=False, client=172.18.0.6:44866)
————►17:54:33:048144 ConnectReply(ver=0, timeout=30000, session=0x74085bee5926d142, readonly=False, server=172.18.0.3:2181)
17:54:33:048408 CreateRequest(size=0, xid=1494352473, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:049644 FailedCreateReply(xid=1494352473, zxid=32, error=-110, path=, server=172.18.0.3:2181)
17:54:33:050483 CreateRequest(size=0, xid=1494352474, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:051912 FailedCreateReply(xid=1494352474, zxid=33, error=-110, path=, server=172.18.0.3:2181)
17:54:33:051966 CreateRequest(size=0, xid=1494352475, path=/clickhouse/task_queue/resharding, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:053203 FailedCreateReply(xid=1494352475, zxid=34, error=-110, path=, server=172.18.0.3:2181)
17:54:33:053263 CreateRequest(size=0, xid=1494352476, path=/clickhouse/task_queue/resharding/1ee79866c57e, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:054653 CreateReply(xid=1494352476, zxid=35, error=0, path=/clickhouse/task_queue/resharding/1ee79866c57e, server=172.18.0.3:2181)
17:54:33:054714 CreateRequest(size=0, xid=1494352477, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:057955 FailedCreateReply(xid=1494352477, zxid=36, error=-110, path=, server=172.18.0.3:2181)
17:54:33:058027 CreateRequest(size=0, xid=1494352478, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:061401 FailedCreateReply(xid=1494352478, zxid=37, error=-110, path=, server=172.18.0.3:2181)
17:54:33:061473 CreateRequest(size=0, xid=1494352479, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:063499 FailedCreateReply(xid=1494352479, zxid=38, error=-110, path=, server=172.18.0.3:2181)
17:54:33:063558 CreateRequest(size=0, xid=1494352480, path=/clickhouse/task_queue/resharding_distributed/online, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:067791 FailedCreateReply(xid=1494352480, zxid=39, error=-110, path=, server=172.18.0.3:2181)
17:54:33:067863 CreateRequest(size=0, xid=1494352481, path=/clickhouse/task_queue/resharding_distributed/online/1ee79866c57e, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:069866 CreateReply(xid=1494352481, zxid=40, error=0, path=/clickhouse/task_queue/resharding_distributed/online/1ee79866c57e, server=172.18.0.3:2181)
17:54:33:069934 CreateRequest(size=0, xid=1494352482, path=/clickhouse/task_queue/resharding_distributed/lock, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:127621 FailedCreateReply(xid=1494352482, zxid=41, error=-110, path=, server=172.18.0.3:2181)
17:54:33:127689 CreateRequest(size=0, xid=1494352483, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:128840 FailedCreateReply(xid=1494352483, zxid=42, error=-110, path=, server=172.18.0.3:2181)
17:54:33:128885 CreateRequest(size=0, xid=1494352484, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:129947 FailedCreateReply(xid=1494352484, zxid=43, error=-110, path=, server=172.18.0.3:2181)
17:54:33:129992 CreateRequest(size=0, xid=1494352485, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:131275 FailedCreateReply(xid=1494352485, zxid=44, error=-110, path=, server=172.18.0.3:2181)
17:54:33:131321 CreateRequest(size=0, xid=1494352486, path=/clickhouse/task_queue/resharding_distributed/coordination, ephemeral=False, sequence=False, client=172.18.0.6:44866)
————►17:54:33:132699 FailedCreateReply(xid=1494352486, zxid=45, error=-110, path=, server=172.18.0.3:2181)
17:54:33:132858 GetChildren2Request(xid=1494352487, path=/clickhouse/task_queue/resharding/1ee79866c57e, watch=False, size=0, client=172.18.0.6:44866)
————►17:54:33:133600 GetChildren2Reply(xid=1494352487, zxid=45, error=0, count=0, server=172.18.0.3:2181)
17:54:33:133650 GetChildren2Request(xid=1494352488, path=/clickhouse/task_queue/resharding/1ee79866c57e, watch=True, size=0, client=172.18.0.6:44866)
————►17:54:33:134170 GetChildren2Reply(xid=1494352488, zxid=45, error=0, count=0, server=172.18.0.3:2181)
17:54:33:484583 ConnectRequest(ver=0, zxid=0, timeout=30000, session=0x0, readonly=False, client=172.18.0.7:39238)
————►17:54:33:486373 ConnectReply(ver=0, timeout=30000, session=0x74085bee5926d161, readonly=False, server=172.18.0.3:2181)
17:54:33:500355 CreateRequest(size=0, xid=1494352474, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:502105 FailedCreateReply(xid=1494352474, zxid=47, error=-110, path=, server=172.18.0.3:2181)
17:54:33:502732 CreateRequest(size=0, xid=1494352475, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:504349 FailedCreateReply(xid=1494352475, zxid=48, error=-110, path=, server=172.18.0.3:2181)
17:54:33:504511 CreateRequest(size=0, xid=1494352476, path=/clickhouse/task_queue/resharding, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:505941 FailedCreateReply(xid=1494352476, zxid=49, error=-110, path=, server=172.18.0.3:2181)
17:54:33:506075 CreateRequest(size=0, xid=1494352477, path=/clickhouse/task_queue/resharding/655dcde35ccc, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:507706 CreateReply(xid=1494352477, zxid=50, error=0, path=/clickhouse/task_queue/resharding/655dcde35ccc, server=172.18.0.3:2181)
17:54:33:507884 CreateRequest(size=0, xid=1494352478, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:509274 FailedCreateReply(xid=1494352478, zxid=51, error=-110, path=, server=172.18.0.3:2181)
17:54:33:509376 CreateRequest(size=0, xid=1494352479, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:510983 FailedCreateReply(xid=1494352479, zxid=52, error=-110, path=, server=172.18.0.3:2181)
17:54:33:511183 CreateRequest(size=0, xid=1494352480, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:513039 FailedCreateReply(xid=1494352480, zxid=53, error=-110, path=, server=172.18.0.3:2181)
17:54:33:513145 CreateRequest(size=0, xid=1494352481, path=/clickhouse/task_queue/resharding_distributed/online, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:514593 FailedCreateReply(xid=1494352481, zxid=54, error=-110, path=, server=172.18.0.3:2181)
17:54:33:514730 CreateRequest(size=0, xid=1494352482, path=/clickhouse/task_queue/resharding_distributed/online/655dcde35ccc, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:516409 CreateReply(xid=1494352482, zxid=55, error=0, path=/clickhouse/task_queue/resharding_distributed/online/655dcde35ccc, server=172.18.0.3:2181)
17:54:33:516594 CreateRequest(size=0, xid=1494352483, path=/clickhouse/task_queue/resharding_distributed/lock, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:520364 FailedCreateReply(xid=1494352483, zxid=56, error=-110, path=, server=172.18.0.3:2181)
17:54:33:520530 CreateRequest(size=0, xid=1494352484, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:521882 FailedCreateReply(xid=1494352484, zxid=57, error=-110, path=, server=172.18.0.3:2181)
17:54:33:522066 CreateRequest(size=0, xid=1494352485, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:526435 FailedCreateReply(xid=1494352485, zxid=58, error=-110, path=, server=172.18.0.3:2181)
17:54:33:526582 CreateRequest(size=0, xid=1494352486, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:530760 FailedCreateReply(xid=1494352486, zxid=59, error=-110, path=, server=172.18.0.3:2181)
17:54:33:530875 CreateRequest(size=0, xid=1494352487, path=/clickhouse/task_queue/resharding_distributed/coordination, ephemeral=False, sequence=False, client=172.18.0.7:39238)
————►17:54:33:532444 FailedCreateReply(xid=1494352487, zxid=60, error=-110, path=, server=172.18.0.3:2181)
17:54:33:532644 GetChildren2Request(xid=1494352488, path=/clickhouse/task_queue/resharding/655dcde35ccc, watch=False, size=0, client=172.18.0.7:39238)
————►17:54:33:534792 GetChildren2Reply(xid=1494352488, zxid=60, error=0, count=0, server=172.18.0.3:2181)
17:54:33:534876 GetChildren2Request(xid=1494352489, path=/clickhouse/task_queue/resharding/655dcde35ccc, watch=True, size=0, client=172.18.0.7:39238)
————►17:54:33:535667 GetChildren2Reply(xid=1494352489, zxid=60, error=0, count=0, server=172.18.0.3:2181)
17:54:33:551575 ConnectRequest(ver=0, zxid=0, timeout=30000, session=0x0, readonly=False, client=172.18.0.8:50670)
————►17:54:33:552834 ConnectReply(ver=0, timeout=30000, session=0x74085bee5926d180, readonly=False, server=172.18.0.3:2181)
17:54:33:554481 CreateRequest(size=0, xid=1494352474, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:555888 FailedCreateReply(xid=1494352474, zxid=62, error=-110, path=, server=172.18.0.3:2181)
17:54:33:555943 CreateRequest(size=0, xid=1494352475, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:557797 FailedCreateReply(xid=1494352475, zxid=63, error=-110, path=, server=172.18.0.3:2181)
17:54:33:557889 CreateRequest(size=0, xid=1494352476, path=/clickhouse/task_queue/resharding, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:559473 FailedCreateReply(xid=1494352476, zxid=64, error=-110, path=, server=172.18.0.3:2181)
17:54:33:559568 CreateRequest(size=0, xid=1494352477, path=/clickhouse/task_queue/resharding/89515b3d7f8b, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:560953 CreateReply(xid=1494352477, zxid=65, error=0, path=/clickhouse/task_queue/resharding/89515b3d7f8b, server=172.18.0.3:2181)
17:54:33:561020 CreateRequest(size=0, xid=1494352478, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:562095 FailedCreateReply(xid=1494352478, zxid=66, error=-110, path=, server=172.18.0.3:2181)
17:54:33:562191 CreateRequest(size=0, xid=1494352479, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:563304 FailedCreateReply(xid=1494352479, zxid=67, error=-110, path=, server=172.18.0.3:2181)
17:54:33:563398 CreateRequest(size=0, xid=1494352480, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:564441 FailedCreateReply(xid=1494352480, zxid=68, error=-110, path=, server=172.18.0.3:2181)
17:54:33:564536 CreateRequest(size=0, xid=1494352481, path=/clickhouse/task_queue/resharding_distributed/online, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:566551 FailedCreateReply(xid=1494352481, zxid=69, error=-110, path=, server=172.18.0.3:2181)
17:54:33:566666 CreateRequest(size=0, xid=1494352482, path=/clickhouse/task_queue/resharding_distributed/online/89515b3d7f8b, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:567772 CreateReply(xid=1494352482, zxid=70, error=0, path=/clickhouse/task_queue/resharding_distributed/online/89515b3d7f8b, server=172.18.0.3:2181)
17:54:33:567840 CreateRequest(size=0, xid=1494352483, path=/clickhouse/task_queue/resharding_distributed/lock, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:569151 FailedCreateReply(xid=1494352483, zxid=71, error=-110, path=, server=172.18.0.3:2181)
17:54:33:571107 CreateRequest(size=0, xid=1494352484, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:572133 FailedCreateReply(xid=1494352484, zxid=72, error=-110, path=, server=172.18.0.3:2181)
17:54:33:572270 CreateRequest(size=0, xid=1494352485, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:573614 FailedCreateReply(xid=1494352485, zxid=73, error=-110, path=, server=172.18.0.3:2181)
17:54:33:573715 CreateRequest(size=0, xid=1494352486, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:576892 FailedCreateReply(xid=1494352486, zxid=74, error=-110, path=, server=172.18.0.3:2181)
17:54:33:576995 CreateRequest(size=0, xid=1494352487, path=/clickhouse/task_queue/resharding_distributed/coordination, ephemeral=False, sequence=False, client=172.18.0.8:50670)
————►17:54:33:578439 FailedCreateReply(xid=1494352487, zxid=75, error=-110, path=, server=172.18.0.3:2181)
17:54:33:578866 GetChildren2Request(xid=1494352488, path=/clickhouse/task_queue/resharding/89515b3d7f8b, watch=False, size=0, client=172.18.0.8:50670)
————►17:54:33:580944 GetChildren2Reply(xid=1494352488, zxid=75, error=0, count=0, server=172.18.0.3:2181)
17:54:33:581393 GetChildren2Request(xid=1494352489, path=/clickhouse/task_queue/resharding/89515b3d7f8b, watch=True, size=0, client=172.18.0.8:50670)
————►17:54:33:582713 GetChildren2Reply(xid=1494352489, zxid=75, error=0, count=0, server=172.18.0.3:2181)
17:54:34:512765 ConnectRequest(ver=0, zxid=0, timeout=30000, session=0x0, readonly=False, client=172.18.0.9:56390)
————►17:54:34:521424 ConnectReply(ver=0, timeout=30000, session=0x74085bee5926d19f, readonly=False, server=172.18.0.3:2181)
17:54:34:521899 CreateRequest(size=0, xid=1494352475, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:525570 FailedCreateReply(xid=1494352475, zxid=77, error=-110, path=, server=172.18.0.3:2181)
17:54:34:525649 CreateRequest(size=0, xid=1494352476, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:526771 FailedCreateReply(xid=1494352476, zxid=78, error=-110, path=, server=172.18.0.3:2181)
17:54:34:526829 CreateRequest(size=0, xid=1494352477, path=/clickhouse/task_queue/resharding, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:527872 FailedCreateReply(xid=1494352477, zxid=79, error=-110, path=, server=172.18.0.3:2181)
17:54:34:527929 CreateRequest(size=0, xid=1494352478, path=/clickhouse/task_queue/resharding/6974dc99f908, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:529064 CreateReply(xid=1494352478, zxid=80, error=0, path=/clickhouse/task_queue/resharding/6974dc99f908, server=172.18.0.3:2181)
17:54:34:529125 CreateRequest(size=0, xid=1494352479, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:529969 FailedCreateReply(xid=1494352479, zxid=81, error=-110, path=, server=172.18.0.3:2181)
17:54:34:530030 CreateRequest(size=0, xid=1494352480, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:531215 FailedCreateReply(xid=1494352480, zxid=82, error=-110, path=, server=172.18.0.3:2181)
17:54:34:531273 CreateRequest(size=0, xid=1494352481, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:532346 FailedCreateReply(xid=1494352481, zxid=83, error=-110, path=, server=172.18.0.3:2181)
17:54:34:532477 CreateRequest(size=0, xid=1494352482, path=/clickhouse/task_queue/resharding_distributed/online, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:535046 FailedCreateReply(xid=1494352482, zxid=84, error=-110, path=, server=172.18.0.3:2181)
17:54:34:535123 CreateRequest(size=0, xid=1494352483, path=/clickhouse/task_queue/resharding_distributed/online/6974dc99f908, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:536696 CreateReply(xid=1494352483, zxid=85, error=0, path=/clickhouse/task_queue/resharding_distributed/online/6974dc99f908, server=172.18.0.3:2181)
17:54:34:536798 CreateRequest(size=0, xid=1494352484, path=/clickhouse/task_queue/resharding_distributed/lock, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:541213 FailedCreateReply(xid=1494352484, zxid=86, error=-110, path=, server=172.18.0.3:2181)
17:54:34:541289 CreateRequest(size=0, xid=1494352485, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:543614 FailedCreateReply(xid=1494352485, zxid=87, error=-110, path=, server=172.18.0.3:2181)
17:54:34:543666 CreateRequest(size=0, xid=1494352486, path=/clickhouse/task_queue, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:544954 FailedCreateReply(xid=1494352486, zxid=88, error=-110, path=, server=172.18.0.3:2181)
17:54:34:545003 CreateRequest(size=0, xid=1494352487, path=/clickhouse/task_queue/resharding_distributed, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:546192 FailedCreateReply(xid=1494352487, zxid=89, error=-110, path=, server=172.18.0.3:2181)
17:54:34:546240 CreateRequest(size=0, xid=1494352488, path=/clickhouse/task_queue/resharding_distributed/coordination, ephemeral=False, sequence=False, client=172.18.0.9:56390)
————►17:54:34:548786 FailedCreateReply(xid=1494352488, zxid=90, error=-110, path=, server=172.18.0.3:2181)
17:54:34:549071 GetChildren2Request(xid=1494352489, path=/clickhouse/task_queue/resharding/6974dc99f908, watch=False, size=0, client=172.18.0.9:56390)
————►17:54:34:549863 GetChildren2Reply(xid=1494352489, zxid=90, error=0, count=0, server=172.18.0.3:2181)
17:54:34:549917 GetChildren2Request(xid=1494352490, path=/clickhouse/task_queue/resharding/6974dc99f908, watch=True, size=0, client=172.18.0.9:56390)
————►17:54:34:550566 GetChildren2Reply(xid=1494352490, zxid=90, error=0, count=0, server=172.18.0.3:2181)
17:56:46:981019 ExistsRequest(xid=1494352489, path=/clickhouse/tables/1/hits_replicated/replicas/clickhouse-ru-1.local/flags/force_restore_data, watch=False, size=0, client=172.18.0.4:39578)
————►17:56:46:982127 FailedExistsReply(xid=1494352489, zxid=90, error=-101, server=172.18.0.3:2181)
17:56:46:983186 ExistsRequest(xid=1494352490, path=/clickhouse/tables/1/hits_replicated, watch=False, size=0, client=172.18.0.4:39578)
————►17:56:46:983957 FailedExistsReply(xid=1494352490, zxid=90, error=-101, server=172.18.0.3:2181)
17:56:46:984629 CreateRequest(size=0, xid=1494352491, path=/clickhouse, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:56:46:986060 FailedCreateReply(xid=1494352491, zxid=91, error=-110, path=, server=172.18.0.3:2181)
17:56:46:987046 CreateRequest(size=0, xid=1494352492, path=/clickhouse/tables, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:56:46:993787 CreateReply(xid=1494352492, zxid=92, error=0, path=/clickhouse/tables, server=172.18.0.3:2181)
17:56:46:993953 CreateRequest(size=0, xid=1494352493, path=/clickhouse/tables/1, ephemeral=False, sequence=False, client=172.18.0.4:39578)
————►17:56:46:995372 CreateReply(xid=1494352493, zxid=93, error=0, path=/clickhouse/tables/1, server=172.18.0.3:2181)

unfontunally i can't catch coredump files ;( i will improve this issue later maybe
@ztlpn
Copy link
Contributor

ztlpn commented May 11, 2017

Abort is caused by a double free bug in libzookeeper (haven't investigated deeply). But the root cause seems to be that zetcd does not yet support multi-operations: etcd-io/zetcd#38

So I guess the takeaway is that zetcd currently cannot be used with ClickHouse.

@bobrik
Copy link
Contributor

bobrik commented Jun 10, 2017

@ztlpn, looks like zetcd now supports multi-operations.

Slach added a commit to Slach/clickhouse-zetcd that referenced this issue Jun 10, 2017
@Slach
improve zetcd clickhouse testing see ClickHouse/ClickHouse#777
3dd90e8 
Signed-off-by: Slach <bloodjazman@gmail.com>
Slach added a commit to Slach/clickhouse-zetcd that referenced this issue Jun 10, 2017
@Slach
improve zetcd clickhouse testing see ClickHouse/ClickHouse#777
88a7b47 
Signed-off-by: Slach <bloodjazman@gmail.com>
@Slach
Copy link
Contributor Author

Slach commented Jun 10, 2017

yay ;) @bobrik @ztlpn
currently replicated table created OK with zetcd

but clickhouse failed when i try insert data

2017.06.10 11:23:51.659616 [ 3 ] <Debug> zetcd_test.test_replicated (StorageReplicatedMergeTree, RestartingThread): Activating replica.
/home/robot-metrika-test/jenkins/workspace/clickhouse-packages-build/sources/contrib/libtcmalloc/src/tcmalloc.cc:286] Attempt to free invalid pointer 0x7f71d4079b60
2017.06.10 11:23:51.679476 [ 5 ] <Error> BaseDaemon: ########################################
2017.06.10 11:23:51.679500 [ 5 ] <Error> BaseDaemon: (from thread 4) Received signal Aborted (6).
2017.06.10 11:23:51.685327 [ 5 ] <Error> BaseDaemon: 1. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7f71d6515428]
2017.06.10 11:23:51.687880 [ 5 ] <Error> BaseDaemon: 2. /usr/bin/clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0xa5) [0x34b2e45]
2017.06.10 11:23:51.688644 [ 5 ] <Error> BaseDaemon: 3. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f71d6fc56ba]
2017.06.10 11:23:51.688670 [ 5 ] <Error> BaseDaemon: 4. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f71d65e682d]

see https://github.com/Slach/clickhouse-zetcd latest commits for reproduce

Slach added a commit to Slach/clickhouse-zetcd that referenced this issue Jun 10, 2017
@Slach
improve zetcd clickhouse testing see ClickHouse/ClickHouse#777
fcad0f2 
Signed-off-by: Slach <bloodjazman@gmail.com>
@Slach
Copy link
Contributor Author

Slach commented Oct 16, 2017

I checked latest version Docker image

all working fine
when you can use latest zetcd and latest clickhouse together ;) thank you guys for your great job

@Slach Slach closed this as completed Oct 16, 2017
@Slach
Copy link
Contributor Author

Slach commented Oct 16, 2017

=(( oops sorry
when i first run test queries it was OK
but second vagrant provision failed with error

==> clickhouse_zetcd: INSERT INTO zetcd_test.test_replicated (timestamp, trackerId, userId ) VALUES (1, 'test1','test1')
==> clickhouse_zetcd: Code: 32. DB::Exception: Attempt to read after eof: while receiving packet from clickhouse-ru-1.local:9000, 172.18.0.6

clickhouse-ru-1.local_1    | 2017.10.16 06:59:08.503243 [ 10 ] <Debug> executeQuery: (from 172.18.0.10:48808) INSERT INTO zetcd_test.test_replicated (timestamp, trackerId, userId ) VALUES
clickhouse-ru-1.local_1    | 2017.10.16 06:59:08.506013 [ 10 ] <Debug> zetcd_test.test_replicated (Replicated OutputStream): Wrote block with ID '6956073591313983161_11012326548367728807', 1 rows
clickhouse-ru-1.local_1    | 2017.10.16 06:59:08.512849 [ 10 ] <Trace> zetcd_test.test_replicated (Data): Renaming temporary part tmp_insert_19700102_19700102_2_2_0 to 19700102_19700102_2_2_0.
clickhouse-ru-1.local_1    | 2017.10.16 06:59:08.515243 [ 22 ] <Error> BaseDaemon: ########################################
clickhouse-ru-1.local_1    | 2017.10.16 06:59:08.515277 [ 22 ] <Error> BaseDaemon: (from thread 21) Received signal Segmentation fault (11).
clickhouse-ru-1.local_1    | 2017.10.16 06:59:08.515284 [ 22 ] <Error> BaseDaemon: Address: 0xb1
clickhouse-ru-1.local_1    | 2017.10.16 06:59:08.526198 [ 22 ] <Error> BaseDaemon: 0. /lib/x86_64-linux-gnu/libc.so.6(strlen+0x26) [0x7fdf65ce5746]
clickhouse-ru-1.local_1    | 2017.10.16 06:59:08.526670 [ 22 ] <Error> BaseDaemon: 1. /usr/bin/clickhouse-server() [0x3a46cc9]
clickhouse-ru-1.local_1    | 2017.10.16 06:59:08.526675 [ 22 ] <Error> BaseDaemon: 2. /usr/bin/clickhouse-server() [0x3a473d8]
clickhouse-ru-1.local_1    | 2017.10.16 06:59:08.526679 [ 22 ] <Error> BaseDaemon: 3. /usr/bin/clickhouse-server() [0x3a47241]
clickhouse-ru-1.local_1    | 2017.10.16 06:59:08.526683 [ 22 ] <Error> BaseDaemon: 4. /usr/bin/clickhouse-server(zookeeper_process+0x10e5) [0x3a4c135]
clickhouse-ru-1.local_1    | 2017.10.16 06:59:08.526778 [ 22 ] <Error> BaseDaemon: 5. /usr/bin/clickhouse-server(do_io+0xde) [0x3a5d4fe]
clickhouse-ru-1.local_1    | 2017.10.16 06:59:08.527188 [ 22 ] <Error> BaseDaemon: 6. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fdf667406ba]
clickhouse-ru-1.local_1    | Segmentation fault (core dumped)

@Slach Slach reopened this Oct 16, 2017
@Slach Slach mentioned this issue Jan 15, 2018
Closed
@filimonov filimonov mentioned this issue Feb 25, 2018
Closed
@Slach Slach mentioned this issue Apr 5, 2018
Closed
@Slach
Copy link
Contributor Author

Slach commented Apr 19, 2018

Latest Clickhouse release completelly broke zetcd compatibility
previous releases created table successfully

2018.04.19 05:00:53.427019 [ 3 ] <Trace> TCPHandlerFactory: TCP Request. Address: 172.18.0.10:53392
2018.04.19 05:00:53.427255 [ 3 ] <Debug> TCPHandler: Connected ClickHouse client version 1.1.54378, user: default.
2018.04.19 05:00:53.460833 [ 3 ] <Debug> executeQuery: (from 172.18.0.10:53392, query_id: 3bc4e288-78f8-4e0b-ae3b-4b180c83cfd8) CREATE TABLE IF NOT EXISTS zetcd_test.test_replicated (timestamp UInt32,
 date MATERIALIZED toDate(timestamp), trackerId String,   userId String  ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/hits_replicated', '{replica}', date, cityHash64( userId), (trackerId
, date, cityHash64(userId), timestamp), 8192)
2018.04.19 05:00:53.650627 [ 3 ] <Trace> ZooKeeper: initialized, hosts: zetcd:2181
2018.04.19 05:00:53.696119 [ 3 ] <Information> BackgroundProcessingPool: Create BackgroundProcessingPool with 16 threads
2018.04.19 05:00:53.700943 [ 3 ] <Debug> zetcd_test.test_replicated (Data): Loading data parts
2018.04.19 05:00:53.701054 [ 3 ] <Debug> zetcd_test.test_replicated (Data): Loaded data parts (0 items)
2018.04.19 05:00:53.703204 [ 3 ] <Debug> zetcd_test.test_replicated (StorageReplicatedMergeTree): Creating table /clickhouse/tables/1/hits_replicated
2018.04.19 05:00:53.938584 [ 4 ] <Error> void ZooKeeperImpl::ZooKeeper::receiveThread(): Code: 999, e.displayText() = ZooKeeperImpl::Exception: Unexpected op_num received at the end of results for mul
ti transaction (Marshalling error), e.what() = ZooKeeperImpl::Exception, Stack trace:

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x15) [0x84e2995]
1. /usr/bin/clickhouse-server(ZooKeeperImpl::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0xcd) [0x8275c4d]
2. /usr/bin/clickhouse-server(ZooKeeperImpl::ZooKeeper::MultiResponse::readImpl(DB::ReadBuffer&)+0x2eb) [0x827652b]
3. /usr/bin/clickhouse-server(ZooKeeperImpl::ZooKeeper::receiveEvent()+0x374) [0x827c454]
4. /usr/bin/clickhouse-server(ZooKeeperImpl::ZooKeeper::receiveThread()+0x2b1) [0x827cca1]
5. /usr/bin/clickhouse-server() [0x8dcbe7e]
6. /lib/x86_64-linux-gnu/libpthread.so.0(+0x77fb) [0x7fe0e5f167fb]
7. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3e) [0x7fe0e56e9b5e]

@alexey-milovidov
Copy link
Member

alexey-milovidov commented Apr 19, 2018 •
edited
Loading

zetcd sent Unexpected op_num at the end of results for multi transaction
Could you please provide the result of:

strace -t -f -e trace=network -s1000 -x -p $(pidof clickhouse-server)

Few pages before this exception happens?

@alexey-milovidov alexey-milovidov added the st-need-info We need extra data to continue (waiting for response) label Jan 26, 2019
kshvakov added a commit to kshvakov/zetcd that referenced this issue Mar 15, 2019
@kshvakov
fix done header for the multi request/response ClickHouse/ClickHouse#777
8aebc29
@blinkov
Copy link
Contributor

blinkov commented Mar 26, 2019

@kshvakov is it working fine after kshvakov/zetcd@8aebc29 ?

@Slach
Copy link
Contributor Author

Slach commented Mar 26, 2019

wow guys, thanks you for your efforts, i can try actualize and https://github.com/kshvakov/zetcd fork with my https://github.com/Slach/clickhouse-zetcd, i'll report here about results

Slach added a commit to Slach/clickhouse-zetcd that referenced this issue Mar 26, 2019
@Slach
test fixes from etcd-io/zetcd#106, ClickHouse/ClickHouse#777 worked w…
5923bb4 
…ith latest Clickhouse

Signed-off-by: Slach <bloodjazman@gmail.com>
@Slach
Copy link
Contributor Author

Slach commented Mar 26, 2019

@kshvakov yeah bro, it's live ;) i commited latest changes for clickhouse-zetcd

@Felixoid
Copy link
Member

How performant it by comparing with zookeeper?

@filimonov filimonov added the comp-zookeeper Zookeeper label Jun 11, 2019
@Slach Slach changed the title clickhouse core dumped when try work over zetcd Using zetcd instead Zookeeper Jul 12, 2019
@stale stale bot added the stale label Oct 20, 2019
@mnacharov
Copy link
Contributor

What's the status?

@Slach
Copy link
Contributor Author

Slach commented Jan 15, 2020

@mnach
you can try https://github.com/Slach/clickhouse-zetcd
but without any guarantee
also zetcd it's a proxy to etcd
so one request to zetcd fire one and more request to etcd
so performance of zetcd is more slow than Zookeeper

@ClickHouse ClickHouse deleted a comment from stale bot Jan 15, 2020
@alexey-milovidov
Copy link
Member

alexey-milovidov commented Jan 15, 2020 •
edited
Loading

@Slach zetcd doesn't work and that's guaranteed.

@alexey-milovidov alexey-milovidov added the st-community-taken External developer is working on that label Jan 15, 2020
@alexey-milovidov
Copy link
Member

This task is in progress, it is assigned to external contributor @levysh
Currently it depends on #8435

@den-crane den-crane mentioned this issue Feb 18, 2020
Closed
@levysh levysh mentioned this issue Apr 20, 2020
Closed
@filimonov filimonov removed the st-need-info We need extra data to continue (waiting for response) label Aug 11, 2020
@amosbird
Copy link
Collaborator

Do we still need this?

@Slach
Copy link
Contributor Author

Slach commented Jul 12, 2021

no, I think clickhouse-keeper and nuRaft implementation in future clickhouse releases will enough for ReplicatedMergeTree

@Slach Slach closed this as completed Jul 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
comp-zookeeper Zookeeper st-community-taken External developer is working on that
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Allow to use Etcd as a replacement for ZooKeeper levysh/ClickHouse
9 participants
@bobrik @Slach @blinkov @mnacharov @filimonov @ztlpn @Felixoid @amosbird @alexey-milovidov