Skip to content

Transfer files from MongoDB GridFS to Minio file server, and MongoDB text to SQLite

License

Notifications You must be signed in to change notification settings

wekan/minio-metadata

Repository files navigation

Transfer files from MongoDB GridFS to Minio file server, and MongoDB text to SQLite

Introduction

Related, not used here:

Requirements

Installed commands:

Transferring files

  1. Edit settings.sh to change MongoDB, SQLite and Minio settings.

  2. a) Do everything of 2b) at one step

./start.sh

  1. b) Do in separate steps

./mongoexport.sh (Copy text from MongoDB to SQLite)

./transfer.sh (Copy files from MongoDB to Minio)

Logging transfer to textfile:

./transfer.sh > logfile.txt

Looking at transfer progress at other terminal:

tail -f logfile.txt

mc command has command line option to add metadata

Metadata can be added be selected from SQLite database in transfer.sh script.

Metadata needs to need in JSON format properly.

mc command also has SQL to search.

How to remove extra text when looking logfile, not needed working results:

cat logfile.txt | grep -v success | less
  • That -v option removes that kind of text from logfile, only showing errors or other remaining text.
  • In remaining log text can be seen, is there any file transfer errors, or does some file need to be transferred again.
  • To be tested, does exporting newer attachments require different OID1 etc format.

What will happen while tranferring files

  1. Export MongoDB text to wekan.db (Temporary .csv files at directory csv will be deleted after they have been imported to SQLite)

  2. Export file from MongoDB GridFS to current directory

  3. Upload file to Minio

  4. Delete file from current directory, to not fill more disk space.

  5. Go back to step 2. for each file

NOTE: Running scripts again currently will overwrite wekan.db file new data and transfer files to minio again.

TODO

  • Current bugs:
    • When transferring files, some of files are not transferred. Is this bug in mc command, bash script, or MinIO server?
    • When metadata is added, some metadata is missing for some files.
    • Does MinIO not work on ext4 on Linux? minio/minio#16602 (comment)
    • This MinIO bug is problem, someone recommended to not use MinIO because of it minio/minio#9419
    • Would MinIO server AGPLv3 license change WeKan MIT licence to AGPLv3 ? Some recommend staying away from MinIO: https://news.ycombinator.com/item?id=35328316
    • What alternatives there are for MinIO ?
    • Attachment filename sanitize when storing files serverside, like use only ObjectID or hash wekan/wekan#4416
  • Add more metadata to minio.
    • Create SQL query for FILEID
    • For each FILEID, do queries for metadata, like cardname, username, etc
    • Combine metadata to correct JSON file format that is OK to save to minio

About

Transfer files from MongoDB GridFS to Minio file server, and MongoDB text to SQLite

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published