Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Synapse backup/restore failed #154

Closed
llinfrance opened this issue Oct 11, 2019 · 8 comments
Closed

Synapse backup/restore failed #154

llinfrance opened this issue Oct 11, 2019 · 8 comments
Labels
question

Comments

@llinfrance
Copy link

Hello,

I post on the yunohost forum, but it may be more appropriate to open an issue here 😃
So I have updated my post:

Hardware: Raspberry Pi for backup create / AMD64 for backup restore
YunoHost version: 3.6.5.2 (stable)
I have access to my server : Through SSH
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no

Description of my issue

Hi everyone,

I have an yunohost on a raspberry pi working fine, and I want to upgrade my hardware on a amd64 board.

The main issue is with the synapse backup / restore. The backup works with the command :
sudo yunohost backup create --apps synapse --debug

20551 INFO Now creating a backup archive from the files collected…
20552 DEBUG Creating the backup tar archive…
331786 DEBUG Backup tar archive created
331794 SUCCESS Backup created
331795 DEBUG action [22561.1] executed in 330.154s
331796 DEBUG lock has been released
name: 20191007-200134
results: 
  apps: 
    synapse: Success
  system: 
size: 1330918692

Then, when I restore the backup like this :
sudo yunohost backup restore 20191007-200134 --debug
And it failed at that moment:

284624 DEBUG CREATE INDEX
284627 DEBUG CREATE INDEX
284628 DEBUG + systemctl daemon-reload
284931 DEBUG + systemctl enable matrix-synapse.service
285034 WARNING Created symlink /etc/systemd/system/multi-user.target.wants/matrix-synapse.service → /etc/systemd/system/matrix-synapse.service.
285337 DEBUG + ynh_use_logrotate /var/log/matrix-synapse
285341 DEBUG + local legacy_args=lnuya
285342 DEBUG + args_array=([l]=logfile= [n]=nonappend [u]=specific_user= [y]=non [a]=append)
285343 DEBUG + declare -Ar args_array
285345 DEBUG + local logfile
285346 DEBUG + local nonappend
285347 DEBUG + local specific_user
285348 DEBUG + ynh_handle_getopts_args /var/log/matrix-synapse
285349 DEBUG + set +x
285351 DEBUG + echo '! Helper used in legacy mode !'
285356 DEBUG + set +x
285358 DEBUG + local logfile=/var/log/matrix-synapse
285360 DEBUG + local nonappend=0
285361 DEBUG + local specific_user=
285363 DEBUG + '[' 1 -gt 0 ']'
285365 DEBUG + '[' /var/log/matrix-synapse == --non-append ']'
285367 DEBUG + '[' 1 -gt 1 ']'
285371 DEBUG + '[' 1 -gt 0 ']'
285374 DEBUG ++ echo /
285377 DEBUG + '[' / '!=' - ']'
285379 DEBUG + '[' -f /var/log/matrix-synapse ']'
285383 DEBUG ++ echo /var/log/matrix-synapse
285386 DEBUG + '[' /var/log/matrix-synapse == log ']'
285390 DEBUG + local 'logfile=/var/log/matrix-synapse/*.log'
285392 DEBUG + local 'customtee=tee -a'
285396 DEBUG + '[' 0 -eq 1 ']'
285399 DEBUG + '[' -n '/var/log/matrix-synapse/*.log' ']'
285401 DEBUG + '[' '!' -f /var/log/matrix-synapse ']'
285403 DEBUG ++ echo log
285405 DEBUG + '[' log '!=' log ']'
285407 DEBUG + local su_directive=
285408 DEBUG + [[ -n '' ]]
285410 DEBUG + cat
285412 DEBUG ++ dirname '/var/log/matrix-synapse/*.log'
285414 DEBUG + sudo mkdir -p /var/log/matrix-synapse
285516 DEBUG + cat synapse-logrotate
285519 DEBUG + sudo tee -a /etc/logrotate.d/synapse
285621 DEBUG + chown matrix-synapse:root -R /opt/yunohost/matrix-synapse
285925 DEBUG + chmod 770 /opt/yunohost/matrix-synapse/Coturn_config_rotate.sh
285927 DEBUG + chown matrix-synapse:root -R /var/lib/matrix-synapse
286029 DEBUG + chown matrix-synapse:root -R /var/log/matrix-synapse
286032 DEBUG + chown matrix-synapse:root -R /etc/matrix-synapse
286034 DEBUG + chmod u=rwX,g=rX,o= -R /etc/matrix-synapse
286036 DEBUG + chmod 600 /etc/matrix-synapse/xxxxxxxxxxx.com.signing.key
286038 DEBUG + setfacl -R -m user:turnserver:rX /etc/matrix-synapse
286040 DEBUG + setfacl -R -m user:turnserver:rwX /var/log/matrix-synapse
286042 DEBUG + yunohost service add matrix-synapse -l /var/log/matrix-synapse/homeserver.log
287150 DEBUG The service 'matrix-synapse' has been added
287253 DEBUG + yunohost service add coturn-synapse
288359 DEBUG The service 'coturn-synapse' has been added
288362 DEBUG + systemctl reload nginx.service
288867 DEBUG + systemctl restart coturn-synapse.service
289079 DEBUG + ynh_check_starting 'Synapse now listening on TCP port 8448' /var/log/matrix-synapse/homeserver.log 300 matrix-synapse
289082 WARNING Starting of matrix-synapse
289084 DEBUG + local 'line_to_match=Synapse now listening on TCP port 8448'
289091 WARNING Job for matrix-synapse.service failed because the control process exited with error code.
289094 DEBUG + local service_name=matrix-synapse
289097 WARNING See "systemctl status matrix-synapse.service" and "journalctl -xe" for details.
289099 DEBUG + local app_log=/var/log/matrix-synapse/homeserver.log
289101 DEBUG + local timeout=300
289103 DEBUG + echo 'Starting of matrix-synapse'
289105 DEBUG + systemctl restart matrix-synapse
289106 DEBUG + ynh_exit_properly
289108 DEBUG + local exit_code=1
289110 DEBUG + '[' 1 -eq 0 ']'
289111 DEBUG + trap '' EXIT
289113 DEBUG + set +eu
289114 DEBUG + sleep 0.5
289619 DEBUG + ynh_print_err '--message=!!\n  synapse'\''s script has encountered an error. Its execution was cancelled.\n!!'
289641 WARNING [ERR] !!
289644 DEBUG + local legacy_args=m
289649 WARNING  synapse's script has encountered an error. Its execution was cancelled.
289652 DEBUG + args_array=([m]=message=)
289655 WARNING !!
289658 DEBUG + declare -Ar args_array
289660 WARNING Please find here an extract of the log before the crash:
289663 DEBUG + local message
289665 DEBUG + ynh_handle_getopts_args '--message=!!\n  synapse'\''s script has encountered an error. Its execution was cancelled.\n!!'
289668 DEBUG + set +x
289670 DEBUG + ynh_print_log '\e[91m\e[1m[ERR]\e[0m !!\n synapse'\''s script has encountered an error. Its execution was cancelled.\n!!'
289678 DEBUG + echo -e '\e[91m\e[1m[ERR]\e[0m !!\n synapse'\''s script has encountered an error. Its execution was cancelled.\n!!'
289680 DEBUG + '[' cli == cli ']'
289683 DEBUG + set +x
289787 DEBUG + type -t ynh_clean_setup
289790 WARNING [DEBUG]: DEBUG      moulinette.interface __init__ - initializing base actions map parser for cli
289792 DEBUG + ynh_die
289793 WARNING [DEBUG]: DEBUG      moulinette.interface __init__ - registering new callback action 'yunohost.utils.packages.ynh_packages_version' to ['-v', '--version']
289795 DEBUG + local legacy_args=mc
289797 WARNING [DEBUG]: DEBUG      moulinette.actionsmap process - loading python module yunohost.service took 0.004s
289798 DEBUG + args_array=([m]=message= [c]=ret_code=)
289799 WARNING [DEBUG]: INFO        moulinette.actionsmap process - processing action [5563.1]: yunohost.service.add with args={'status': None, 'description': None, 'need_lock': False, 'log': None, 'runlevel': None, 'log_type': 'file', 'name': 'coturn-synapse'}
289801 DEBUG + declare -Ar args_array
289802 WARNING [DEBUG]: SUCCESS  yunohost.service success - [5563.1] The service 'coturn-synapse' has been added
289803 DEBUG + local message
289804 WARNING [DEBUG]: DEBUG      moulinette.actionsmap process - action [5563.1] executed in 0.081s
289806 DEBUG + local ret_code
289807 WARNING [DEBUG]: DEBUG   - The service 'coturn-synapse' has been added
289808 DEBUG + ynh_handle_getopts_args
289809 WARNING [DEBUG]: DEBUG   - + systemctl reload nginx.service
289811 DEBUG + set +x
289812 WARNING [DEBUG]: DEBUG   - + systemctl restart coturn-synapse.service
289813 DEBUG + echo ''
289814 WARNING [DEBUG]: DEBUG   - + ynh_check_starting 'Synapse now listening on TCP port 8448' /var/log/matrix-synapse/homeserver.log 300 matrix-synapse
289815 DEBUG + exit 1
289816 WARNING [DEBUG]: WARNING - Starting of matrix-synapse
289818 WARNING [DEBUG]: DEBUG   - + local 'line_to_match=Synapse now listening on TCP port 8448'
289819 WARNING [DEBUG]: WARNING - Job for matrix-synapse.service failed because the control process exited with error code.
289820 WARNING [DEBUG]: DEBUG   - + local service_name=matrix-synapse
289821 WARNING [DEBUG]: WARNING - See "systemctl status matrix-synapse.service" and "journalctl -xe" for details.
289823 WARNING [DEBUG]: DEBUG   - + local app_log=/var/log/matrix-synapse/homeserver.log
289824 WARNING [DEBUG]: DEBUG   - + local timeout=300
289825 WARNING [DEBUG]: DEBUG   - + echo 'Starting of matrix-synapse'
289826 WARNING [DEBUG]: DEBUG   - + systemctl restart matrix-synapse
289827 WARNING [DEBUG]: DEBUG   - + ynh_exit_properly
289829 WARNING 
289973 ERROR Unable to restore the app 'synapse'
Traceback (most recent call last):
  File "/usr/lib/moulinette/yunohost/backup.py", line 1325, in _restore_app
    env=env_dict)[0]
  File "/usr/lib/moulinette/yunohost/hook.py", line 397, in hook_exec
    raise YunohostError('hook_exec_failed', path=path)

Finaley, I can share the following log : sudo yunohost log display 20191007-203407-backup_restore_app-synapse

description: Restore 'synapse' from a backup archive
log_path: /var/log/yunohost/categories/operation/20191007-203407-backup_restore_app-synapse.log
logs: 
  - ct of the log before the crash:
  - 2019-10-07 22:36:31,228: DEBUG - + local message
  - 2019-10-07 22:36:31,229: WARNING - [DEBUG]: DEBUG      moulinette.actionsmap process - action [20357.1] executed in 0.075s
  - 2019-10-07 22:36:31,231: DEBUG - + ynh_handle_getopts_args '--message=!!\n  synapse'\''s script has encountered an error. Its execution was cancelled.\n!!'
  - 2019-10-07 22:36:31,233: WARNING - [DEBUG]: DEBUG   - The service 'matrix-synapse' has been added
  - 2019-10-07 22:36:31,234: DEBUG - + set +x
  - 2019-10-07 22:36:31,235: WARNING - [DEBUG]: DEBUG   - + yunohost service add coturn-synapse
  - 2019-10-07 22:36:31,236: DEBUG - + ynh_print_log '\e[91m\e[1m[ERR]\e[0m !!\n synapse'\''s script has encountered an error. Its execution was cancelled.\n!!'
  - 2019-10-07 22:36:31,237: WARNING - [DEBUG]: DEBUG      moulinette.actionsmap __init__ - loading actions map namespace 'yunohost'
  - 2019-10-07 22:36:31,239: DEBUG - + echo -e '\e[91m\e[1m[ERR]\e[0m !!\n synapse'\''s script has encountered an error. Its execution was cancelled.\n!!'
  - 2019-10-07 22:36:31,240: WARNING - [DEBUG]: DEBUG      moulinette.actionsmap __init__ - extra parameter classes loaded: ['comment', 'ask', 'password', 'required', 'pattern']
  - 2019-10-07 22:36:31,241: DEBUG - + '[' cli == cli ']'
  - 2019-10-07 22:36:31,242: WARNING - [DEBUG]: DEBUG      moulinette.interface __init__ - initializing base actions map parser for cli
  - 2019-10-07 22:36:31,243: DEBUG - + set +x
  - 2019-10-07 22:36:31,244: WARNING - [DEBUG]: DEBUG      moulinette.interface __init__ - registering new callback action 'yunohost.utils.packages.ynh_packages_**********' to ['-v', '--**********']
  - 2019-10-07 22:36:31,245: DEBUG - + type -t ynh_clean_setup
  - 2019-10-07 22:36:31,247: WARNING - [DEBUG]: DEBUG      moulinette.actionsmap process - loading python module yunohost.service took 0.004s
  - 2019-10-07 22:36:31,248: DEBUG - + ynh_die
  - 2019-10-07 22:36:31,249: WARNING - [DEBUG]: INFO        moulinette.actionsmap process - processing action [20358.1]: yunohost.service.add with args={'status': None, 'description': None, 'need_lock': False, 'log': None, 'runlevel': None, 'log_type': 'file', 'name': 'coturn-synapse'}
  - 2019-10-07 22:36:31,251: DEBUG - + local legacy_args=mc
  - 2019-10-07 22:36:31,252: WARNING - [DEBUG]: SUCCESS  yunohost.service success - [20358.1] The service 'coturn-synapse' has been added
  - 2019-10-07 22:36:31,253: DEBUG - + args_array=([m]=message= [c]=ret_code=)
  - 2019-10-07 22:36:31,254: WARNING - [DEBUG]: DEBUG      moulinette.actionsmap process - action [20358.1] executed in 0.080s
  - 2019-10-07 22:36:31,256: DEBUG - + declare -Ar args_array
  - 2019-10-07 22:36:31,257: WARNING - [DEBUG]: DEBUG   - The service 'coturn-synapse' has been added
  - 2019-10-07 22:36:31,258: DEBUG - + local message
  - 2019-10-07 22:36:31,259: WARNING - [DEBUG]: DEBUG   - + systemctl reload nginx.service
  - 2019-10-07 22:36:31,260: DEBUG - + local ret_code
  - 2019-10-07 22:36:31,262: WARNING - [DEBUG]: DEBUG   - + systemctl restart coturn-synapse.service
  - 2019-10-07 22:36:31,263: DEBUG - + ynh_handle_getopts_args
  - 2019-10-07 22:36:31,264: WARNING - [DEBUG]: DEBUG   - + ynh_check_starting 'Synapse now listening on TCP port 8448' /var/log/matrix-synapse/homeserver.log 300 matrix-synapse
  - 2019-10-07 22:36:31,265: DEBUG - + set +x
  - 2019-10-07 22:36:31,267: WARNING - [DEBUG]: WARNING - Starting of matrix-synapse
  - 2019-10-07 22:36:31,268: DEBUG - + echo ''
  - 2019-10-07 22:36:31,269: WARNING - [DEBUG]: DEBUG   - + local 'line_to_match=Synapse now listening on TCP port 8448'
  - 2019-10-07 22:36:31,270: DEBUG - + exit 1
  - 2019-10-07 22:36:31,272: WARNING - [DEBUG]: DEBUG   - + local service_name=matrix-synapse
  - 2019-10-07 22:36:31,273: WARNING - [DEBUG]: DEBUG   - + local app_log=/var/log/matrix-synapse/homeserver.log
  - 2019-10-07 22:36:31,274: WARNING - [DEBUG]: DEBUG   - + local timeout=300
  - 2019-10-07 22:36:31,276: WARNING - [DEBUG]: DEBUG   - + echo 'Starting of matrix-synapse'
  - 2019-10-07 22:36:31,277: WARNING - [DEBUG]: DEBUG   - + systemctl restart matrix-synapse
  - 2019-10-07 22:36:31,278: WARNING - [DEBUG]: DEBUG   - + ynh_exit_properly
  - 2019-10-07 22:36:31,279: WARNING - 
  - 2019-10-07 22:36:31,412: ERROR - Unable to restore the app 'synapse'
  - Traceback (most recent call last):
  -   File "/usr/lib/moulinette/yunohost/backup.py", line 1325, in _restore_app
  -     env=env_dict)[0]
  -   File "/usr/lib/moulinette/yunohost/hook.py", line 397, in hook_exec
  -     raise YunohostError('hook_exec_failed', path=path)
  - YunohostError: Script execution failed: /tmp/restorexd7FKL/restore
metadata: 
  ended_at: 2019-10-07 22:36:31
  env: 
    YNH_APP_BACKUP_DIR: /home/yunohost.backup/tmp/20191007-200134/apps/synapse/backup
    YNH_APP_ID: synapse
    YNH_APP_INSTANCE_NAME: synapse
    YNH_APP_INSTANCE_NUMBER: 1
    YNH_BACKUP_CSV: /home/yunohost.backup/tmp/20191007-200134/backup.csv
    YNH_BACKUP_DIR: /home/yunohost.backup/tmp/20191007-200134
    YNH_CWD: /home/yunohost.backup/tmp/20191007-200134/apps/synapse/backup
    YNH_INTERFACE: cli
    YNH_STDINFO: /tmp/tmpbpQ1cA/stdinfo
    YNH_STDRETURN: /tmp/tmpb1ZmJz/stdreturn
  error: Unable to restore the app 'synapse'
  operation: backup_restore_app
  related_to: 
    - app
    - synapse
  started_at: 2019-10-07 22:34:07
  success: False
metadata_path: /var/log/yunohost/categories/operation/20191007-203407-backup_restore_app-synapse.yml
name: 20191007-203407-backup_restore_app-synapse

I have updated yunohost to the latest version 3.6.5.2.

In my look for a solution, I found that they mention your work on the matrix github, which I found it rather cool 😃

Cheers,
Ludovic
@llinfrance
Copy link
Author

Hi,

I try to investigate, but it is hard to progress for me 😞

So far, I understood that the restore failed here :

systemctl reload nginx.service
systemctl restart coturn-$app.service
ynh_check_starting "Synapse now listening on TCP port $synapse_tls_port" "/var/log/matrix-$app/homeserver.log" 300 "matrix-$app"

Unfortunately (but I could imaging why), the script remove everything after a failed, so I don't know where to looked.
I try to execute the script step by step, but now I don't know how to prepare the archive for restoration. I looked for it in the yunohost github (backup.py), but it is not easy to read/understand the steps for how to prepare the archive.

Any clue ?

@llinfrance
Copy link
Author

Hi again,

On the yunohost forum I've got some help from Aleks. He told me to look in /var/log/daemon.log.

I guess the important line was the following:

Oct 14 22:01:56 mydomain systemd[12195]: matrix-synapse.service: Failed at step EXEC spawning /opt/yunohost/matrix-synapse/bin/python: Exec format error

So I did try to run /opt/yunohost/matrix-synapse/bin/python, which gives:

bash: ./python: cannot execute binary file: Exec format error

Looking on the web how to solved this, I run file /opt/yunohost/matrix-synapse/bin/python, which gives:

./python: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=33a2bd0786c4fe4fa40f029319ceb4738cec9b82, stripped

It seems to be the issue: the file is an ARM one. I recall that the backup was made from a raspberry, then the restore on a amd64.

@Josue-T, do you know if it is something fixable ?

@Josue-T
Copy link

Josue-T commented Oct 14, 2019

Well, Yes the cross architecture restoration is NOT supported.

If your are good in command line you should be able to do something. I purpose recreate a new python virtualenvironement for your new architecture. The idea is to use theses command. For the version it's written in the manifest here so actually it's the version 1.3.1 (for the testing).

@llinfrance
Copy link
Author

Thank you for your answer, the issue is clear now.

Unfortunately, I did not understand your comment. I have updated my synapse to the testing branch on the raspberry, backup, then restore on the amd64. I have the same issue.

Did you write that I should install the matrix-synapse in /opt/yunohost using the virtualenv commands you point out, then restore the app with this fresh synapse ?

PS: Just in case, I should have mentioned that I am able to install synapse on the amd64. It worked well, but of course I hadn't any of the data/rooms/contacts.

@Josue-T
Copy link

Josue-T commented Oct 15, 2019

Thank you for your answer, the issue is clear now.

Unfortunately, I did not understand your comment. I have updated my synapse to the testing branch on the raspberry, backup, then restore on the amd64. I have the same issue.

It's normal

Did you write that I should install the matrix-synapse in /opt/yunohost using the virtualenv commands you point out, then restore the app with this fresh synapse ?

Well, my explanation was in case of a full manual restoration (like to launch each command of this script manually). So the idea is to do the restoration manually. And after before to launch synapse you needo to recreate the venv.

@llinfrance
Copy link
Author

Ok, thank you. I have started tonight and made some progress. I will keep you updated when I will arrive to something.

Cheers !

@llinfrance
Copy link
Author

Hi,

I was finally able to restore my synapse server 😃

First, I install a fresh version of synapse, then I run manually the restore script with few modifications.

I start by preparing the archive :

sudo tar -xzvf synapse-archive.tar.gz -C ../tmp/
cd /home/yunohost.backup/tmp/apps/synapse/settings

Then I run the restore script changing the following parts.

In load settings, set the app name.

#=================================================
# LOAD SETTINGS
#=================================================
ynh_script_progression --message="Loading settings..."

#app=$YNH_APP_INSTANCE_NAME
app=synapse

In RESTORE DATA, copy-paste the old stuff in the new place.

#=================================================
# RESTORE DATA and LOG and CONFIG
#=================================================
cp -r ../backup/var/lib/matrix-synapse "/var/lib/matrix-$app" 
cp -r ../backup/var/log/matrix-synapse "/var/log/matrix-$app" 
cp -r ../backup/etc/matrix-synapse "/etc/matrix-$app"

In recreate dedicate user, the ynh_system_user_create was stopping the scripts 😮

#=================================================
# RECREATE THE DEDICATED USER
#=================================================
ynh_script_progression --message="Recreating the dedicated system user..." --weight=3

# Create the dedicated user (if not existing)
#ynh_system_user_create --username=$synapse_user --home_dir=/var/lib/matrix-$app
adduser $synapse_user ssl-cert
adduser turnserver ssl-cert

The only remaining small issue is they is no automatic connection in riot web app...

Thanks again for the help. I will close the issue.

@llinfrance
Copy link
Author

Hi,

I have to mention that after restoring my server as describes in the previous post, I was ale to receive messages from others, but not able to send messages to others 😢
Since I hadn't to much contacts/rooms, I reset my instance and start a fresh one.

Unfortunately, if you change your architecture (from arm to amd64 in my case), there is no known way to restore/backup your synapse instance.

But thanks anyway for the work on synapse !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Josue-T @llinfrance