MongoDB Ops Manager java.lang.OutOfMemoryError: unable to create new native thread

When starting MongoDB Ops Manager version 1.8.2.312, we have this error:

Migrate MMS data
Running migrations…[  OK  ]
Start MMS server
Instance 0 starting……….[  OK  ]
Start Backup HTTP Server
Instance 0 starting…….[FAILED]
instance: 0  – msg: unable to create new native thread

Solution: Increase the max number of process for user mongodb-mms:

$ sudo vi /etc/security/limits.d/90-nproc.conf

mongodb-mms     soft    nproc     unlimited

MongoDB Ops Manager java.lang.OutOfMemoryError: unable to create new native thread

MongoDB MMS com.xgen.svc.brs.svc.DaemonAssignmentSvc: ERROR Assignment failed: No Daemon found with enough free space

When adding new backup job to Mongo DB MMS, we can have this error
com.xgen.svc.brs.svc.DaemonAssignmentSvc: ERROR Assignment failed: No Daemon found with enough free space

But checking the storage, the usage is just under 20%.

Solution: MMS requires manually configure the backup daemon when you add a new backup job to an existing one. Go to MMS Admin -> Backup -> Jobs -> Set binding, click on it and forces choosing the daemon in select list.

MongoDB MMS com.xgen.svc.brs.svc.DaemonAssignmentSvc: ERROR Assignment failed: No Daemon found with enough free space

MongoDB Ops Manager Insufficient oplog size: The oplog window must be at least 3 hours over the last 24 hours for all members of replica set

When configuring MongoDB Ops Manager (MMS), MMS backup check the oplog of last 24 hours. If there is a oplog windows does not cover minimum 3 hours, Ops Manager will not allow you to continue the backup configuration:

Insufficient oplog size: The oplog window must be at least 3 hours over the last 24 hours for all members of replica set . Please increase the oplog

Solution: If the current oplog size is indeed insufficient for 3 normal hours operations for your application, increase its size following this guide: http://docs.mongodb.org/v3.0/tutorial/change-oplog-size/
If it is the case that you just set up the DB and do an intensive insert, then you don’t need to increase the oplog. Wait for another 24 hours and try the backup configuration process again (https://jira.mongodb.org/browse/MMS-2374)

MongoDB Ops Manager Insufficient oplog size: The oplog window must be at least 3 hours over the last 24 hours for all members of replica set

MongoDB restore DB from MMS backup on shard clusters with different IPs than original shard

When restore shards using MongoDB MMS backup files, if the new shard IP addresses are not the same as the IPs from where you do backup, MongoDB will raise the error “11002 exception: socket exception [CONNECT_ERROR] for since the old IPs is stored in config server DB.

Solution: Open a shell to MongoDB config server(s) and update the shard cluster IP in config DB:

use config;
db.shards.find()
db.shards.update({_id: “shard_id”}, {$set: {“host”: “shard_id/new_ip:port”}} })

MongoDB restore DB from MMS backup on shard clusters with different IPs than original shard

MongoDB MMS failed to decrypt password for job when restoring using SCP

Causes:

MMS Application Server and Backup Daemon are installed on different machines. And these two machines do not share the gen.key file.

Solutions:

On the MMS Application Server, run the following:

grep ENC_KEY /opt/mongodb/mms/conf/mms.conf

On the Backup Daemon Server:

grep ENC_KEY /opt/mongodb/mms-backup-daemon/conf/daemon.conf

If the gen.key file is missing on Backup Daemon Server or gen.key are different on the two machines, copy gen.key from MMS Application Server to Backup Daemon Server and restart Backup Daemon:

sudo service mongodb-mms-backup-daemon restart

MongoDB MMS failed to decrypt password for job when restoring using SCP