Tutorial: Migrate MongoDB to Azure Cosmos DB's API for MongoDB offline using MongoDB native tools

APPLIES TO: MongoDB

This MongoDB migration guide is part of a series on MongoDB migration. The critical MongoDB migration steps are pre-migration, migration, and post-migration, as shown in the following diagram.

Overview of data migration using MongoDB native tools

You can use MongoDB native tools to perform an offline (one-time) migration of databases from an on-premises or cloud instance of MongoDB to Azure Cosmos DB's API for MongoDB.

In this tutorial, you learn how to:

In this tutorial, you migrate a dataset in MongoDB hosted in an Azure virtual machine to the Azure Cosmos DB API for MongoDB by using MongoDB native tools. The MongoDB native tools are a set of binaries that facilitate data manipulation on an existing MongoDB instance. Since Azure Cosmos DB exposes an API for MongoDB, the MongoDB native tools can insert data into Azure Cosmos DB. The focus of this article is on migrating data out of a MongoDB instance using mongoexport/mongoimport or mongodump/mongorestore. Because the native tools connect to MongoDB by using connection strings, you can run the tools anywhere, but we recommend that you run these tools within the same network as the MongoDB instance to avoid firewall problems.

The MongoDB native tools can move data only as fast as the host hardware allows. The native tools can be the simplest solution for small datasets where total migration time isn't a concern. MongoDB Spark connector, Azure Data Migration Service (DMS), or Azure Data Factory (ADF) can be better alternatives if you need a scalable migration pipeline.

If you don't have a MongoDB source set up already, see the article Install and configure MongoDB on a Windows VM in Azure.

Prerequisites

To complete this tutorial, you need to:

• Complete the pre-migration steps such as estimating throughput, choosing a partition key, and the indexing policy.
• Create an Azure Cosmos DB for MongoDB account.
• Sign in to your MongoDB instance.
  • Download and install the MongoDB native tools from this link.
    • Ensure that your MongoDB native tools version matches your existing MongoDB instance.
    • If your MongoDB instance has a different version than Azure Cosmos DB for MongoDB, then install both MongoDB native tool versions and use the appropriate tool version for MongoDB and Azure Cosmos DB for MongoDB, respectively.
  • Add a user with readWrite permissions, unless one already exists. Later in this tutorial, provide this username and password to the mongoexport and mongodump tools.

Configure Azure Cosmos DB Server Side Retry

When you migrate from MongoDB to Azure Cosmos DB, you benefit from resource governance capabilities that guarantee the ability to fully use your provisioned RU/s of throughput. Azure Cosmos DB might throttle a given request during migration if that request exceeds the container provisioned RU/s. The request needs to be retried. The round-trip time involved in the network hop between the migration tool and Azure Cosmos DB affects the overall response time of that request. Furthermore, MongoDB native tools might not handle retries. The Server Side Retry feature of Azure Cosmos DB enables the service to intercept throttle error codes and retry with a much lower round-trip time, which dramatically improves request response times. From the perspective of MongoDB native tools, the need to handle retries is minimized, which positively affects your experience during migration.

The Server Side Retry capability is in the Features blade of the Azure Cosmos DB portal.

If it's Disabled, we recommend that you enable it, as shown in the following screenshot.

Choose the proper MongoDB native tool

• mongoexport/mongoimport is the best pair of migration tools for migrating a subset of your MongoDB database.
  • mongoexport exports your existing data to a human-readable JSON or CSV file. mongoexport takes an argument that specifies the subset of your existing data to export.
  • mongoimport opens a JSON or CSV file and inserts the content into the target database instance (Azure Cosmos DB in this case).
  • Note that JSON and CSV aren't compact formats. You might incur excess network charges as mongoimport sends data to Azure Cosmos DB.
• mongodump/mongorestore is the best pair of migration tools for migrating your entire MongoDB database. The compact BSON format makes more efficient use of network resources as the data is inserted into Azure Cosmos DB.
  • mongodump exports your existing data as a BSON file.
  • mongorestore imports your BSON file dump into Azure Cosmos DB.
• If you have a small JSON file that you want to import into Azure Cosmos DB for MongoDB, the mongoimport tool is a quick solution.

Collect the Azure Cosmos DB for MongoDB credentials

Azure Cosmos DB for MongoDB provides compatible access credentials that MongoDB native tools can use. You need these access credentials to migrate data into Azure Cosmos DB for MongoDB. To find these credentials:

1. Open the Azure portal.

2. Go to your Azure Cosmos DB for MongoDB account.

3. In the left navigation, select the Connection String blade. You see something like this:

   

   • HOST - the Azure Cosmos DB endpoint functions as a MongoDB hostname
   • PORT - when MongoDB native tools connect to Azure Cosmos DB, you must specify this port explicitly
   • USERNAME - the prefix of the Azure Cosmos DB endpoint domain name functions as the MongoDB username
   • PASSWORD - the Azure Cosmos DB master key functions as the MongoDB password
   • Also, note the SSL field which is true - the MongoDB native tool must enable SSL when writing data into Azure Cosmos DB

Perform the migration

1. Choose which databases and collections you want to migrate. In this example, you migrate the query collection in the edx database from MongoDB to Azure Cosmos DB.

The rest of this section guides you through using the pair of tools you selected in the previous section.

mongoexport/mongoimport

1. To export the data from the source MongoDB instance, open a terminal on the MongoDB instance machine. If it's a Linux machine, type the following command:

   mongoexport --host HOST:PORT --authenticationDatabase admin -u USERNAME -p PASSWORD --db edx --collection query --out edx.json
   

   On Windows, the executable is mongoexport.exe. Fill in HOST, PORT, USERNAME, and PASSWORD based on the properties of your existing MongoDB database instance.

   You can also choose to export only a subset of the MongoDB dataset. One way to export a subset is to add a filter argument:

   mongoexport --host HOST:PORT --authenticationDatabase admin -u USERNAME -p PASSWORD --db edx --collection query --out edx.json --query '{"field1":"value1"}'
   

   Only documents that match the filter {"field1":"value1"} will be exported.

   When you run the command, you see that it creates an edx.json file:

   

2. You can use the same terminal to import edx.json into Azure Cosmos DB. If you're running mongoimport on a Linux machine, type the following command:

   mongoimport --host HOST:PORT -u USERNAME -p PASSWORD --db edx --collection importedQuery --ssl --type json --writeConcern="{w:0}" --file edx.json
   

   On Windows, mongoimport.exe is the executable. Fill in HOST, PORT, USERNAME, and PASSWORD by using the Azure Cosmos DB credentials you collected earlier.

3. Monitor the terminal output from mongoimport. You should see that it prints lines of text to the terminal containing updates on the migration status:

   

4. Finally, examine Azure Cosmos DB to validate that migration was successful. Open the Azure Cosmos DB portal and go to Data Explorer. You should see that an edx database with an importedQuery collection is created, and if you exported only a subset of data, importedQuery contains only documents matching the desired subset of the data. In the following example, only one document matched the filter {"field1":"value1"}:

   

mongodump/mongorestore

1. To create a BSON data dump of your MongoDB instance, open a terminal on the MongoDB instance machine. If it is a Linux machine, type

   mongodump --host HOST:PORT --authenticationDatabase admin -u USERNAME -p PASSWORD --db edx --collection query --ssl --out edx-dump
   

   HOST, PORT, USERNAME, and PASSWORD should be filled in based on the properties of your existing MongoDB database instance. You should see that an edx-dump directory is produced and that the directory structure of edx-dump reproduces the resource hierarchy (database and collection structure) of your source MongoDB instance. Each collection is represented by a BSON file:

   

2. You can use the same terminal to restore the contents of edx-dump into Azure Cosmos DB. If you're running mongorestore on a Linux machine, type the following command:

   mongorestore --host HOST:PORT --authenticationDatabase admin -u USERNAME -p PASSWORD --db edx --collection importedQuery --writeConcern="{w:0}" --ssl edx-dump/edx/query.bson
   

   On Windows, the executable is mongorestore.exe. Replace HOST, PORT, USERNAME, and PASSWORD with the Azure Cosmos DB credentials you collected earlier.

3. Monitor the terminal output from mongorestore. You should see that it prints lines to the terminal updating on the migration status:

   

4. Finally, examine Azure Cosmos DB to validate that migration was successful. Open the Azure Cosmos DB portal and navigate to Data Explorer. You should see that an edx database with an importedQuery collection is created, and importedQuery should contain the entire dataset from the source collection:

   

Post-migration optimization

After you migrate the data stored in MongoDB database to Azure Cosmos DB’s API for MongoDB, you can connect to Azure Cosmos DB and manage the data. You can also perform other post-migration optimization steps such as optimizing the indexing policy, updating the default consistency level, or configuring global distribution for your Azure Cosmos DB account. For more information, see the Post-migration optimization article.

Additional resources

• Azure Cosmos DB service information
• MongoDB database tools documentation
• Trying to do capacity planning for a migration to Azure Cosmos DB?
  • If you only know the number of vcores and servers in your existing database cluster, read about estimating request units using vCores or vCPUs
  • If you know typical request rates for your current database workload, read about estimating request units using Azure Cosmos DB capacity planner

Next steps

• Review migration guidance for additional scenarios in the Microsoft Database Migration Guide