Writing directly to S3 from mongodump
In my new position one of the first tasks was to rework the MongoDB installations to enable the application to automatically fail to a new Master in the event of a failure. Taking stock of things was a bit scary as things were out of date and needed some care. I created new nodes with updated OS and updated MongoDB. I fixed a couple mounting options with this new build and then got around to working on the backups. Backups where dumping to a file system they shouldn’t have been and I started thinking wouldn’t it be cool if I could just dump to the end destination within the current process which happens to be S3.
So after a bit of trial and error I found this to work:
mongodump --host <hostname> --db <database_name> --gzip --archive | aws s3 cp - s3://Your_S3_bucket/<database_name>.archive
This all assumes you have awscli installed and have the proper IAM roles/permissions to write to your desired S3 bucket.
While testing the restore I setup a local VM and followed my install guide and when running the restore command I kept getting:
mongorestore failed no reachable servers
Because I followed my setup guide this included setting up replication in the mongod.conf file. Apparently configure replication causes you to not be able to restore, which makes sense. Removing that I was able to run the restore command after downloading the .archive from S3:
mongorestore --gzip --archive=<database_name>.archive
Well this job is now done, on to the next thing(s).