Tag: mms

  • MongoDB Ops Manager java.lang.OutOfMemoryError: unable to create new native thread

    When starting MongoDB Ops Manager version 1.8.2.312, we have this error:

    Migrate MMS data
    Running migrations…[  OK  ]
    Start MMS server
    Instance 0 starting……….[  OK  ]
    Start Backup HTTP Server
    Instance 0 starting…….[FAILED]
    instance: 0  – msg: unable to create new native thread

    Solution: Increase the max number of process for user mongodb-mms:

    $ sudo vi /etc/security/limits.d/90-nproc.conf

    mongodb-mms     soft    nproc     unlimited

  • MongoDB MMS com.xgen.svc.brs.svc.DaemonAssignmentSvc: ERROR Assignment failed: No Daemon found with enough free space

    When adding new backup job to Mongo DB MMS, we can have this error
    com.xgen.svc.brs.svc.DaemonAssignmentSvc: ERROR Assignment failed: No Daemon found with enough free space

    But checking the storage, the usage is just under 20%.

    Solution: MMS requires manually configure the backup daemon when you add a new backup job to an existing one. Go to MMS Admin -> Backup -> Jobs -> Set binding, click on it and forces choosing the daemon in select list.

  • MongoDB Ops Manager Insufficient oplog size: The oplog window must be at least 3 hours over the last 24 hours for all members of replica set

    When configuring MongoDB Ops Manager (MMS), MMS backup check the oplog of last 24 hours. If there is a oplog windows does not cover minimum 3 hours, Ops Manager will not allow you to continue the backup configuration:

    Insufficient oplog size: The oplog window must be at least 3 hours over the last 24 hours for all members of replica set . Please increase the oplog

    Solution: If the current oplog size is indeed insufficient for 3 normal hours operations for your application, increase its size following this guide: http://docs.mongodb.org/v3.0/tutorial/change-oplog-size/
    If it is the case that you just set up the DB and do an intensive insert, then you don’t need to increase the oplog. Wait for another 24 hours and try the backup configuration process again (https://jira.mongodb.org/browse/MMS-2374)

  • MongoDB restore DB from MMS backup on shard clusters with different IPs than original shard

    When restore shards using MongoDB MMS backup files, if the new shard IP addresses are not the same as the IPs from where you do backup, MongoDB will raise the error “11002 exception: socket exception [CONNECT_ERROR] for <shard id/ip:port>” since the old IPs is stored in config server DB.

    Solution: Open a shell to MongoDB config server(s) and update the shard cluster IP in config DB:

    use config;
    db.shards.find()
    db.shards.update({_id: “shard_id”}, {$set: {“host”: “shard_id/new_ip:port”}} })

  • MongoDB MMS failed to decrypt password for job when restoring using SCP

    Causes:

    MMS Application Server and Backup Daemon are installed on different machines. And these two machines do not share the gen.key file.

    Solutions:

    On the MMS Application Server, run the following:

    grep ENC_KEY /opt/mongodb/mms/conf/mms.conf

    On the Backup Daemon Server:

    grep ENC_KEY /opt/mongodb/mms-backup-daemon/conf/daemon.conf

    If the gen.key file is missing on Backup Daemon Server or gen.key are different on the two machines, copy gen.key from MMS Application Server to Backup Daemon Server and restart Backup Daemon:

    sudo service mongodb-mms-backup-daemon restart

  • MongoDB MMS Backup failed to enqueue an oplog.

    Causes: Backup DB write error due to:

    1. run out of disk space

    2. no primary/master found

    Solution: Restart backup DB hosts should solve this error.