Self hosting stuck on preparing/syncing

[rhyst]

Hello, I am running a self hosted instance of any sync. I have successfully connected to it using the Android and desktop apps and notes appear to sync between them.

However in both clients the status indicator always shows “preparing” or “syncing” and never “synced”. This is despite the fact that syncing does appear to work.

Is there anyway to debug this? My network topology is the default (1 of each container except sync node of which there are three) and none of the logs appear to show any errors.

Thanks!

[fb929]

hi!
It looks like you don't have a connection with self-hosted, files are syncing only between clients via P2P.
You need to check the availability of your self-hosted cluster from the clients and also ensure that its address is present in the configuration file client.yml (or heart.yml).
For this, you can use the utilities telnet.
Additionally, you can use netcheck, but at the moment it can only check the availability of coordinators.

[rhyst]

Thanks for the reply.

I checked all the tcp ports with telnet and they connected to the containers correctly. I used nc to check that I could connect to the UDP ports on the server though the containers did not produce any output from anything I sent. I ran another nc instance on the server with the containers stopped to at least confirm that the udp ports were open which they were.

I did notice one error in the nodes as a result of starting/stopping them:

2024-05-29T17:48:46.262Z        INFO    common.net.peer serve connection error  {"error": "manager closed: EOF", "errorVerbose": "manager closed: EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:234"}

Is this relevant? They stop after a small amount of time so I think it is just due to the containers booting.

[rhyst]

BTW: I could not get netcheck to run. The downloaded binaries gave me an error:

❯ any-sync-netcheck -v
any-sync-netcheck: /usr/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by any-sync-netcheck)
any-sync-netcheck: /usr/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by any-sync-netcheck)

And installing with go install did not produce a binary as far as I could tell.

[fb929]

BTW: I could not get netcheck to run. The downloaded binaries gave me an error:

❯ any-sync-netcheck -v
any-sync-netcheck: /usr/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by any-sync-netcheck)
any-sync-netcheck: /usr/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by any-sync-netcheck)

And installing with go install did not produce a binary as far as I could tell.

The packages are built on Ubuntu 22.04, so it looks like your system is not compatible. Try installing from the source code.
ubuntu-22.04
go install github.com/anyproto/any-sync-tools/any-sync-netcheck@latest

[rhyst]

Cloning repo and running make build worked :)

[rhyst]

If I used my heart.yml file unmodified then I get the following:

GoOS: linux
Kernel: Linux
Core: 5.15.146.1-microsoft-standard-WSL2
Platform: x86_64
OS: GNU/Linux
Hostname: pc
CPUs: 16
2024-05-30T20:09:17.911+0100    INFO    common.net.secure       secure service init     {"peerId": "<my-peer-id>"}
2024-05-30T20:09:17.911+0100    WARN    netcheck        open TCP conn error     {"addr": "yamux://<my-ip>:4830", "error": "dial tcp: address yamux://<my-ip>:4830: too many colons in address", "dur": "4.847µs"}
2024/05/30 20:09:17 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2024-05-30T20:09:18.419+0100    INFO    netcheck        success {"addr": "<my-ip>:5830", "ip": "quic://<my-ip>:5830", "dur": "507.123562ms"}

If I modify the heart.yml to remove yamux:// then I get:

GoOS: linux
Kernel: Linux
Core: 5.15.146.1-microsoft-standard-WSL2
Platform: x86_64
OS: GNU/Linux
Hostname: pc
CPUs: 16
2024-05-30T20:11:59.939+0100    INFO    common.net.secure       secure service init     {"peerId": "<my-peer-id>"}
2024-05-30T20:12:00.207+0100    INFO    netcheck        success {"addr": "<my-ip>:4830", "ip": "<my-ip>:4830", "dur": "267.738282ms"}
2024/05/30 20:12:00 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2024-05-30T20:12:00.429+0100    INFO    netcheck        success {"addr": "<my-ip>:5830", "ip": "quic://<my-ip>:5830", "dur": "221.861385ms"}

However with the modfied file syncing still does not happen.

[fb929]

Please show your etc/client.yml file.

[rhyst]

networkId: N5AH7JCDmLN7vQCbkAQwWgamKNULKBg4dtXum67yVVNWWbKV
nodes:
  - peerId: 12D3KooWPvYs9oopYTkSFdfdBsJDcRZQYtXyj3dN1sYzyGYfEgVV
    addresses:
      - yamux://<my-ip>:4830
      - quic://<my-ip>:5830
    types:
      - coordinator
  - peerId: 12D3KooWLS9fNC4SqhaREmDKwTg4FFExfwQiWofgJjq3aLb12f47
    addresses:
      - yamux://<my-ip>:4530
      - quic://<my-ip>:5530
    types:
      - consensus
  - peerId: 12D3KooWMSAYxpwz1Kiu2DsBwWp3pbywwBK65zpcB9RdLKEi4yT6
    addresses:
      - yamux://<my-ip>:4430
      - quic://<my-ip>:5430
    types:
      - tree
  - addresses:
      - yamux://<my-ip>:4431
      - quic://<my-ip>:5431
    peerId: 12D3KooWPGLGcbxnUviSr4gxjBeRgjaDcQVjiBzSMxTX8iUwUATZ
    types:
      - tree
  - addresses:
      - yamux://<my-ip>:4432
      - quic://<my-ip>:5432
    peerId: 12D3KooWQSBuQvJmq8co4GxmHkrD9Q1uhPyDu6t3VRBFSKXriGaz
    types:
      - tree
  - peerId: 12D3KooWKUjJSxgZzyU42yd4vyjiE1zkgvfhLD6PMbSr1wvreMyz
    addresses:
      - yamux://<my-ip>:4730
      - quic://<my-ip>:5730
    types:
      - file

[fb929]

and same config from mongo ?

use coordinator
db.nodeConf.find().sort( { _id: -1 } ).limit(1)

[rhyst]

Ah the containers are using the internal docker hostnames:

  {
    _id: ObjectId("6656f8eb762998a025d82078"),
    networkId: 'N5AH7JCDmLN7vQCbkAQwWgamKNULKBg4dtXum67yVVNWWbKV',
    nodes: [
      {
        peerId: '12D3KooWPvYs9oopYTkSFdfdBsJDcRZQYtXyj3dN1sYzyGYfEgVV',
        addresses: [
          'yamux://any-sync-coordinator:4830',
          'quic://any-sync-coordinator:5830'
        ],
        types: [ 'coordinator' ]
      },
      {
        peerId: '12D3KooWLS9fNC4SqhaREmDKwTg4FFExfwQiWofgJjq3aLb12f47',
        addresses: [
          'yamux://any-sync-consensusnode:4530',
          'quic://any-sync-consensusnode:5530'
        ],
        types: [ 'consensus' ]
      },
      {
        peerId: '12D3KooWMSAYxpwz1Kiu2DsBwWp3pbywwBK65zpcB9RdLKEi4yT6',
        addresses: [
          'yamux://any-sync-node-1:4430',
          'quic://any-sync-node-1:5430'
        ],
        types: [ 'tree' ]
      },
      {
        peerId: '12D3KooWPGLGcbxnUviSr4gxjBeRgjaDcQVjiBzSMxTX8iUwUATZ',
        addresses: [
          'yamux://any-sync-node-2:4431',
          'quic://any-sync-node-2:5431'
        ],
        types: [ 'tree' ]
      },
      {
        peerId: '12D3KooWQSBuQvJmq8co4GxmHkrD9Q1uhPyDu6t3VRBFSKXriGaz',
        addresses: [
          'yamux://any-sync-node-3:4432',
          'quic://any-sync-node-3:5432'
        ],
        types: [ 'tree' ]
      },
      {
        peerId: '12D3KooWKUjJSxgZzyU42yd4vyjiE1zkgvfhLD6PMbSr1wvreMyz',
        addresses: [
          'yamux://any-sync-filenode:4730',
          'quic://any-sync-filenode:5730'
        ],
        types: [ 'file' ]
      }
    ],
    creationTime: ISODate("2024-05-29T09:44:11.450Z"),
    enable: true
  }

Do they need to be the same?

[fb929]

yes, the static config "client.yml" and "network config" from mongo should be same.
because "clients" use the static file client.yml ONLY ONCE - during the first start.
after that, clients download "network config" from the coordinator (config from mongo) and use that.

In fact, you can leave only the coordinators in client.yml, but it's easier to keep all the configs identical.

also, if You are using our docker-compose, You only need to configure EXTERNAL_LISTEN_HOSTS

[rhyst]

That was it! I set the hosts to be the externally accessible ones in my network configs, deleted most of the persisted storage, restarted the containers, and started with a new key in the client. Its now showing "Synced" 😄

Thank you for your help!