May 8

Writing directly to S3 from mongodump

In my new position one of the first tasks was to rework the MongoDB installations to enable the application to automatically fail to a new Master in the event of a failure. Taking stock of things was a bit scary as things were out of date and needed some care. I created new nodes with updated OS and updated MongoDB. I fixed a couple mounting options with this new build and then got around to working on the backups. Backups where dumping to a file system they shouldn’t have been and I started thinking wouldn’t it be cool if I could just dump to the end destination within the current process which happens to be S3.

So after a bit of trial and error I found this to work:

mongodump --host <hostname> --db <database_name> --gzip --archive | aws s3 cp - s3://Your_S3_bucket/<database_name>.archive

This all assumes you have awscli installed and have the proper IAM roles/permissions to write to your desired S3 bucket.

While testing the restore I setup a local VM and followed my install guide and when running the restore command I kept getting:

mongorestore failed no reachable servers

Because I followed my setup guide this included setting up replication in the mongod.conf file. Apparently configure replication causes you to not be ableĀ  to restore, which makes sense. Removing that I was able to run the restore command after downloading the .archive from S3:

mongorestore --gzip --archive=<database_name>.archive

Well this job is now done, on to the next thing(s).


Tags: , , ,
Copyright 2018. All rights reserved.

Posted May 8, 2018 by administrator in category "Computer

Leave a Reply

Your email address will not be published. Required fields are marked *

Not a Robot * Time limit is exhausted. Please reload the CAPTCHA.