Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker ttn dashboard v1.0 #56

Merged
merged 26 commits into from
May 20, 2020
Merged

Docker ttn dashboard v1.0 #56

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
c2e13b1
Merge pull request #1 from mcci-catena/master
Mar 27, 2020
207a11c
version updates
Apr 10, 2018
a9e223e
Install old version of influxdb
Apr 10, 2018
4214647
latest changes
Apr 10, 2018
682ddd7
version updates
Apr 10, 2018
72d80fc
default_Version_changes
Oct 15, 2018
2421651
influxdb online backup changes
Oct 25, 2018
8d8817b
small correction
Oct 25, 2018
d2b47f0
small correction
Oct 25, 2018
ab623ac
No need influxdb-backup directory
Nov 7, 2018
90e443e
No need influxdb-backup directory
Nov 7, 2018
79e0bd5
Hostname and domain name modification
Nov 7, 2018
56aee15
Updated files with Terry
Jul 4, 2019
3469909
added mqtts setup
Nov 29, 2019
265a3a8
enabled automatic backup
Dec 18, 2019
2b1251e
enabled automatic backup
Dec 18, 2019
202384f
enabled automatic backup
Dec 18, 2019
1bd06ba
enabled HTTP Endpoint
Dec 18, 2019
ee50578
Removing unwanted entries
Jan 10, 2020
13b9693
some adjustments
Jan 10, 2020
12a8ae9
adjustments
Jan 10, 2020
218c8ea
adjustments
Jan 10, 2020
a98ea79
adjustments
Jan 10, 2020
77e77f9
Update Dockerfile
Jan 10, 2020
bd16b9a
Update Dockerfile
Jan 23, 2020
aa7d9ea
Fixing: Loading env for crontab
Mar 17, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
influxdb online backup changes
  • Loading branch information
cmurugan authored and MuruganChandrasekar committed Mar 30, 2020
commit 2421651d8ea49ed77a8dbcbd699d29d0d39b469f
28 changes: 8 additions & 20 deletions influxdb-backup/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,39 +1,27 @@
#
# Dockerfile for building the extra-instance-for-influxdb-backup
# Dockerfile for building the influxdb instance with backup setup
#

# Build the extra-instance using phusion base image
FROM phusion/baseimage
# passing arguments to install influxdb specific version
ARG version

FROM influxdb:${version}

# Default InfluxDB host
ENV INFLUX_HOST=influxdb

# Change workdir
RUN mkdir -p /opt/influxdb-backup
WORKDIR "/opt/influxdb-backup"

# passing arguments to install influxdb specific version
ARG version

# Install influxdb database
RUN apt-get update && apt-get install -y wget
RUN wget https://dl.influxdata.com/influxdb/releases/influxdb_${version}_amd64.deb
RUN dpkg -i influxdb_${version}_amd64.deb

# To backup influxdb to S3 Bucket, some packages need to be installed as follows:
RUN apt-get update && apt-get install -y python-pip
RUN pip install awscli --upgrade

# Default InfluxDB host
ENV INFLUX_HOST=influxdb

# Amazon S3 bucket's backup working Directory
RUN mkdir -p /var/lib/amazon-bucket

# Backup script
COPY showdb.sh /bin/showdb.sh
COPY backup.sh /bin/backup.sh
RUN chmod +x /bin/showdb.sh
RUN chmod +x /bin/backup.sh

# Backup directory
RUN mkdir -p /var/lib/influxdb-backup

# end of file
146 changes: 80 additions & 66 deletions influxdb-backup/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,40 +2,39 @@

```console

cmurugan@iotserver:/iot/main-server/docker-ttn-dashboard_version_updates/docker-ttn-dashboard$ docker-compose up -d
Creating network "dockerttndashboard_default" with the default driver
Creating dockerttndashboard_postfix_1 ...
Creating dockerttndashboard_influxdb_1 ...
Creating dockerttndashboard_influxdb_1
Creating dockerttndashboard_influxdb_1 ... done
Creating dockerttndashboard_postfix_1 ... done
Creating dockerttndashboard_influxdb-backup_1
Creating dockerttndashboard_node-red_1 ...
Creating dockerttndashboard_grafana_1 ...
Creating dockerttndashboard_grafana_1
Creating dockerttndashboard_node-red_1 ... done
Creating dockerttndashboard_apache_1 ...
Creating dockerttndashboard_apache_1 ... done
cmurugan@iot:~/server/influx_api_version/docker-ttn-dashboard$ docker-compose up -d

dockerttndashboard_postfix_1 is up-to-date
Recreating dockerttndashboard_influxdb_1 ...
Recreating dockerttndashboard_influxdb_1 ... done
Recreating dockerttndashboard_grafana_1 ...
Recreating dockerttndashboard_grafana_1
Recreating dockerttndashboard_node-red_1 ...
Recreating dockerttndashboard_node-red_1 ... done
Recreating dockerttndashboard_apache_1 ...
Recreating dockerttndashboard_apache_1 ... done

```

### status of docker container and databases

```console

cmurugan@iotserver:/iot/main-server/docker-ttn-dashboard_version_updates/docker-ttn-dashboard$ docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------------------------------------------
dockerttndashboard_apache_1 /sbin/my_init Up 0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp
dockerttndashboard_grafana_1 /run.sh Up 3000/tcp
dockerttndashboard_influxdb-backup_1 /sbin/my_init Up
dockerttndashboard_influxdb_1 /entrypoint.sh influxd Up 8086/tcp
dockerttndashboard_node-red_1 npm start -- --userDir /da ... Up 1880/tcp
dockerttndashboard_postfix_1 /sbin/my_init Up 0.0.0.0:25->25/tcp

cmurugan@iotserver:/iot/main-server/docker-ttn-dashboard_version_updates/docker-ttn-dashboard$ docker-compose exec influxdb influx
Connected to http://localhost:8086 version 1.4.0
InfluxDB shell version: 1.4.0
cmurugan@iot:~/server/influx_api_version/docker-ttn-dashboard$ docker-compose ps

Name Command State Ports
-----------------------------------------------------------------------------------------------------------------
dockerttndashboard_apache_1 /sbin/my_init Up 0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp
dockerttndashboard_grafana_1 /run.sh Up 3000/tcp
dockerttndashboard_influxdb_1 /entrypoint.sh influxd Up 8086/tcp
dockerttndashboard_node-red_1 npm start -- --userDir /da ... Up 1880/tcp
dockerttndashboard_postfix_1 /sbin/my_init Up 0.0.0.0:25->25/tcp

cmurugan@iot:~/server/influx_api_version/docker-ttn-dashboard$ docker-compose exec influxdb bash

root@a16175bb4ce0:/opt/influxdb-backup# influx
Connected to http://localhost:8086 version 1.6.4
InfluxDB shell version: 1.6.4
> show databases
name: databases
name
Expand All @@ -62,32 +61,49 @@ time host region value

## BACKUP DATABASE

### Backup can be taken through shell script by using extra instance (influxdb-backup) and synced with Amazon S3 cloud
### Backup can be taken through shell script and synced with Amazon S3 cloud


```console

cmurugan@iotserver:/iot/main-server/docker-ttn-dashboard_version_updates/docker-ttn-dashboard$ docker exec -it dockerttndashboard_influxdb-backup_1 bash
root@a16175bb4ce0:/opt/influxdb-backup# backup.sh

root@56a388b52486:/opt/influxdb-backup# backup.sh
Backup Influx metadata
2018/04/10 09:50:52 backing up metastore to /var/lib/influxdb-backup/meta.03
2018/04/10 09:50:52 backup complete
2018/10/25 11:00:44 backing up metastore to /var/lib/influxdb-backup/meta.00
2018/10/25 11:00:44 No database, retention policy or shard ID given. Full meta store backed up.
2018/10/25 11:00:44 Backing up all databases in portable format
2018/10/25 11:00:44 backing up db=
2018/10/25 11:00:44 backing up db=_internal rp=monitor shard=1 to /var/lib/influxdb-backup/_internal.monitor.00001.00 since 0001-01-01T00:00:00Z
2018/10/25 11:00:44 backing up db=_internal rp=monitor shard=6 to /var/lib/influxdb-backup/_internal.monitor.00006.00 since 0001-01-01T00:00:00Z
2018/10/25 11:00:44 backing up db=testdb rp=autogen shard=7 to /var/lib/influxdb-backup/testdb.autogen.00007.00 since 0001-01-01T00:00:00Z
2018/10/25 11:00:44 backup complete:
2018/10/25 11:00:44 /var/lib/influxdb-backup/20181025T110044Z.meta
2018/10/25 11:00:44 /var/lib/influxdb-backup/20181025T110044Z.s1.tar.gz
2018/10/25 11:00:44 /var/lib/influxdb-backup/20181025T110044Z.s6.tar.gz
2018/10/25 11:00:44 /var/lib/influxdb-backup/20181025T110044Z.s7.tar.gz
2018/10/25 11:00:44 /var/lib/influxdb-backup/20181025T110044Z.manifest
Creating backup for _internal
2018/04/10 09:50:52 backing up db=_internal since 0001-01-01 00:00:00 +0000 UTC
2018/04/10 09:50:52 backing up metastore to /var/lib/influxdb-backup/meta.04
2018/04/10 09:50:52 backing up db=_internal rp=monitor shard=1 to /var/lib/influxdb-backup/_internal.monitor.00001.00 since 0001-01-01 00:00:00 +0000 UTC
2018/04/10 09:50:52 backing up db=_internal rp=monitor shard=2 to /var/lib/influxdb-backup/_internal.monitor.00002.00 since 0001-01-01 00:00:00 +0000 UTC
2018/04/10 09:50:52 backup complete
2018/10/25 11:00:44 backing up metastore to /var/lib/influxdb-backup/meta.00
2018/10/25 11:00:44 backing up db=_internal
2018/10/25 11:00:44 backing up db=_internal rp=monitor shard=1 to /var/lib/influxdb-backup/_internal.monitor.00001.00 since 0001-01-01T00:00:00Z
2018/10/25 11:00:44 backing up db=_internal rp=monitor shard=6 to /var/lib/influxdb-backup/_internal.monitor.00006.00 since 0001-01-01T00:00:00Z
2018/10/25 11:00:44 backup complete:
2018/10/25 11:00:44 /var/lib/influxdb-backup/20181025T110044Z.meta
2018/10/25 11:00:44 /var/lib/influxdb-backup/20181025T110044Z.s1.tar.gz
2018/10/25 11:00:44 /var/lib/influxdb-backup/20181025T110044Z.s6.tar.gz
2018/10/25 11:00:44 /var/lib/influxdb-backup/20181025T110044Z.manifest
Creating backup for testdb
2018/04/10 09:50:52 backing up db=testdb since 0001-01-01 00:00:00 +0000 UTC
2018/04/10 09:50:52 backing up metastore to /var/lib/influxdb-backup/meta.05
2018/04/10 09:50:52 backing up db=testdb rp=autogen shard=3 to /var/lib/influxdb-backup/testdb.autogen.00003.00 since 0001-01-01 00:00:00 +0000 UTC
2018/04/10 09:50:52 backup complete
2018/10/25 11:00:44 backing up metastore to /var/lib/influxdb-backup/meta.00
2018/10/25 11:00:44 backing up db=testdb
2018/10/25 11:00:44 backing up db=testdb rp=autogen shard=7 to /var/lib/influxdb-backup/testdb.autogen.00007.00 since 0001-01-01T00:00:00Z
2018/10/25 11:00:44 backup complete:
2018/10/25 11:00:44 /var/lib/influxdb-backup/20181025T110044Z.meta
2018/10/25 11:00:44 /var/lib/influxdb-backup/20181025T110044Z.s7.tar.gz
2018/10/25 11:00:44 /var/lib/influxdb-backup/20181025T110044Z.manifest
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
upload: ../../var/lib/amazon-bucket/data_directory_backup_2018-04-10.tar.gz to s3://mcci-influxdb-test/data_directory_backup_2018-04-10.tar.gz
upload: ../../var/lib/amazon-bucket/metdata_db_backup_2018-04-10.tar.gz to s3://mcci-influxdb-test/metdata_db_backup_2018-04-10.tar.gz
upload: ../../var/lib/influxdb-S3-bucket/data_directory_backup_2018-10-25.tar.gz to s3://mcci-influxdb-test/data_directory_backup_2018-10-25.tar.gz
upload: ../../var/lib/influxdb-S3-bucket/metdata_db_backup_2018-10-25.tar.gz to s3://mcci-influxdb-test/metdata_db_backup_2018-10-25.tar.gz

```

Expand Down Expand Up @@ -119,9 +135,10 @@ drwxr-xr-x 35 root root 4096 Apr 10 09:50 ..

```console

cmurugan@iotserver:/iot/main-server/docker-ttn-dashboard_version_updates/docker-ttn-dashboard$ docker-compose exec influxdb influx
Connected to http://localhost:8086 version 1.4.0
InfluxDB shell version: 1.4.0
root@a16175bb4ce0:/opt/influxdb-backup# influx

Connected to http://localhost:8086 version 1.6.4
InfluxDB shell version: 1.6.4
> show databases
name: databases
name
Expand All @@ -137,34 +154,31 @@ _internal
> exit

```

`(Stop the influxdb database in order to restore dropped "testdb" database)`
### Restoring metadata and database

```console

cmurugan@iotserver:/iot/main-server/docker-ttn-dashboard_version_updates/docker-ttn-dashboard$ docker-compose stop influxdb
Stopping dockerttndashboard_influxdb_1 ... done
root@a16175bb4ce0:/opt/influxdb-backup# influxd restore -portable -host $INFLUX_HOST:8088 /var/lib/influxdb-backup

cmurugan@iotserver:/iot/main-server/docker-ttn-dashboard_version_updates/docker-ttn-dashboard$ docker exec -it dockerttndashboard_influxdb-backup_1 bash
root@56a388b52486:/opt/influxdb-backup# influxd restore -metadir /var/lib/influxdb/meta /var/lib/influxdb-backup
Using metastore snapshot: /var/lib/influxdb-backup/meta.05
root@56a388b52486:/opt/influxdb-backup# influxd restore -database testdb -datadir /var/lib/influxdb/data /var/lib/influxdb-backup
Restoring from backup /var/lib/influxdb-backup/testdb.*
unpacking /var/lib/influxdb/data/testdb/autogen/3/000000001-000000001.tsm
2018/10/25 11:02:48 Restoring shard 7 live from backup 20181025T110044Z.s7.tar.gz
2018/10/25 11:02:48 Meta info not found for shard 5 on database testdb. Skipping shard file 20181025T100801Z.s5.tar.gz
2018/10/25 11:02:48 Meta info not found for shard 1 on database _internal. Skipping shard file 20181025T095242Z.s1.tar.gz
2018/10/25 11:02:48 Meta info not found for shard 6 on database _internal. Skipping shard file 20181025T095242Z.s6.tar.gz
2018/10/25 11:02:48 Meta info not found for shard 3 on database testdb. Skipping shard file 20181024T143005Z.s3.tar.gz
2018/10/25 11:02:48 Meta info not found for shard 2 on database testdb. Skipping shard file 20181024T142358Z.s2.tar.gz

root@a16175bb4ce0:/opt/influxdb-backup# influxd restore -portable -host $INFLUX_HOST:8088 -database testdb /var/lib/influxdb-backup

```

### Start the influxdb database and check for whether database has been restored
2018/10/25 11:03:04 error updating meta: DB metadata not changed. database may already exist
restore: DB metadata not changed. database may already exist

```console
### Checking the Database has been restored

cmurugan@iotserver:/iot/main-server/docker-ttn-dashboard_version_updates/docker-ttn-dashboard$ docker-compose start influxdb
Starting influxdb ... done
```console

cmurugan@iotserver:/iot/main-server/docker-ttn-dashboard_version_updates/docker-ttn-dashboard$ docker-compose exec influxdb influx
Connected to http://localhost:8086 version 1.4.0
InfluxDB shell version: 1.4.0
root@a16175bb4ce0:/opt/influxdb-backup# influx
Connected to http://localhost:8086 version 1.6.4
InfluxDB shell version: 1.6.4
> show databases
name: databases
name
Expand All @@ -177,7 +191,7 @@ Using database testdb
name: cpu
time host region value
---- ---- ------ -----
1523353042145216096 serverA us_west 0.64
1540391379121807732 serverA us_west 0.64
> exit

```
12 changes: 6 additions & 6 deletions influxdb-backup/backup.sh
100644 → 100755
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,22 @@
DATABASES=$(/bin/showdb.sh)

echo 'Backup Influx metadata'
influxd backup -host $INFLUX_HOST:8088 /var/lib/influxdb-backup
influxd backup -portable -host $INFLUX_HOST:8088 /var/lib/influxdb-backup

# Replace colons with spaces to create list.
for db in ${DATABASES//:/ }; do
echo "Creating backup for $db"
influxd backup -database $db -host $INFLUX_HOST:8088 /var/lib/influxdb-backup
influxd backup -portable -database $db -host $INFLUX_HOST:8088 /var/lib/influxdb-backup
done

if [ $? -eq 0 ]; then

tar czf /var/lib/amazon-bucket/metdata_db_backup_`date +%F`.tar.gz /var/lib/influxdb-backup/
tar czf /var/lib/amazon-bucket/data_directory_backup_`date +%F`.tar.gz /var/lib/influxdb/
aws s3 sync /var/lib/amazon-bucket/ s3://${S3_BUCKET_INFLUXDB}/
tar czf /var/lib/influxdb-S3-bucket/metdata_db_backup_`date +%F`.tar.gz /var/lib/influxdb-backup/
tar czf /var/lib/influxdb-S3-bucket/data_directory_backup_`date +%F`.tar.gz /var/lib/influxdb/
aws s3 sync /var/lib/influxdb-S3-bucket/ s3://${S3_BUCKET_INFLUXDB}/

fi

# Remove the old backup data in local directory to avoid excessive storage use
find /var/lib/amazon-bucket/ -type f -mtime +90 -exec rm {} \;
find /var/lib/influxdb-S3-bucket/ -type f -mtime +90 -exec rm {} \;