Migrating from TXE

January 14, 2022 | Glynn Bird | Migration TXE

In this blog post we’ll show how data stored in a Cloudant on Transaction Engine (TXE) instance can be easily migrated to Cloudant Standard. There are few differences betweeen the two offerrings, so we’ll explore ways to avoid any pitfalls along the way.

There isn’t a way of converting an existing account from TXE to Standard, so the first step is to provision a new Cloudant Standard account.

migration

Photo by Julia Craice on Unsplash

Create a new Cloudant Standard account 🔗

In the IBM Cloud Dashboard, locate the Cloudant service and complete the form to create a new Cloudant Standard service:

Note Cloudant Standard offers two authentication mechanisms: IAM only, or IAM and Legacy Credentials. TXE only has IAM.

Create new empty databases 🔗

For each of your databases that needs to be copied over, in the new Cloudant Standard dashboard choose “Create Database” and enter the name of the database to create a new, empty target database.

Note Cloudant Standard has two types of databases: partitioned and non-partitioned. As all of your TXE databases will be non-partitioned, choose the “non-partitioned” option.

We will also need a _replicator database in the Cloudant Standard instance which will handle the replication jobs. Follow the same steps to create a new, empty _replicator database.

Replicating the data 🔗

Cloudant’s replication capabilities can be used to copy the data from the source (Cloudant TXE) to the target (Standard) and it is recommend to use the Standard account to mediate the replication - i.e. we will be sending our replication document to the Standard account and it will “pull” the data from TXE.

replication plan

You’ll need a replication document for each of the TXE databases that are to be copied over. Generate the JSON ahead of time and then add a document for each database to the Cloudant Standard account’s _replicator database:

{
  "_id": "txe_to_standard_orders",
  "source": {
    "url": "https://txe.cloudant.com/orders",
    "auth": {
      "iam": {
        "api_key": "abc123"
      }
    }
  },
  "target": {
    "url": "https://standard.cloudant.com/orders",
    "auth": {
      "iam": {
        "api_key": "def456"
      }
    }
  },
  "continuous": true
}

Note:

Replication jobs use up the read allocation of the source Cloudant service (TXE) and the write allocation of the target Cloudant service (Standard), so it is advisable to increase the capacity of your source and target services during the data copying process.

Once the replication jobs are set up, the replication jobs can be monitored using the _scheduler/docs and _scheduler/jobs endpoints.

Also bear in mind that once data is copied to the new target databases, any secondary indexes will build. You should monitor the _active_tasks endpoint to ensure that index building is complete before switching traffic to the new Cloudant service.

Note if you have a large number of databases, it is advisable to copy databases over in small batches. Copying databases one at a time makes it easier to monitor.

Differences between TXE and Standard 🔗

Before we switch application traffic over to the new Cloudant Standard URL, we need to do some checks to make sure everything is in order. Here are some pitfalls that should be avoided.

JavaScript versions 🔗

Cloudant TXE allows newer (ES6) JavaScript syntax than Cloudant Standard, so it’s worth querying each of your secondary indexes to make sure they’re returning the data you are expecting. If you are seeing errors, then simplify your JavaScript code: var instead of const/let, simple for loops and conditional statements.

Note that if you modify an index definition, then the secondary index will need to rebuild again.

Other differences 🔗