Elasticsearch hybrid

November 30, 2017 | Glynn Bird | ElasticSearch Serverless

On the face of it, IBM Cloudant and Elasticsearch seem to be very similar products. They both:

Cloudant is a resilient database that guarantees that data is stored on disk on multiple nodes when you write your data. Elasticsearch is not a database as such — it prioritises speed over resiliance and is designed to be used as a secondary index rather than as your primary source of truth.

In some cases, it may be desirable to take a hybrid approach:

Such a setup needs a means of moving data from Cloudant to Elasticsearch as it changes. In this article, we’ll see how to automatically keep the data in sync using a “serverless” function.

schematic

Cloudant-to-Elasticsearch bridge

The Cloudant database has a changes feed, an HTTP stream that spools out each add/update/delete that happens to the dataset. By consuming that feed, we can trigger other actions as each change arrives, in this case, writing to an Elasticsearch index. The IBM Cloud Functions platform can be configured to listen to the Cloudant changes feed on your behalf, and execute custom code on each change. As IBM Cloud Functions is a “serverless” platform, you don’t pay for any fixed computing power to run the actions. You simply pay for the number of invocations of your code, which is proportional to the rate of change of your data.

I’ve written a script to set up the serverless infrastructure. Clone the code:

git clone https://github.com/ibm-watson-data-lab/cloudant-es-bridge
cd cloudant-es-bridge

All it needs is our Cloudant and Elasticsearch authentication credentials entered as commandline environment variables. (Read the project’s README to see how to sign up for Cloudant, Elasticsearch, and IBM Cloud Functions.)

export CLOUDANT_HOST="HOST.cloudant.com"
export CLOUDANT_USERNAME="CLOUDANTUSERNAME"
export CLOUDANT_PASSWORD="CLOUDANTPASSWORD"
export CLOUDANT_DB="esbridge"
export ELASTIC_URL="https://ESUSERNAME:ESPASSWORD@HOST.composedb.com:PORT/INDEX/TYPE"

Then run the deployment script (which assumes you have the bx wsk tool installed and configured on your machine):

./deploy.sh

This script performs the following tasks:

The upshot is that every time you create, modify or delete a document in Cloudant, the equivalent action happens in your Elasticsearch cluster!

Try creating documents in your Cloudant dashboard, and then check that they appear in the Compose dashboard’s “Browser” feature.

The data can then be queried with the Elasticsearch API:

curl "$ELASTIC_URL/_search?q=aimee"
{
  "took": 110,
  "timed_out": false,
  "hits": {
    "total": 1,
    "max_score": 0.2824934,
    "hits": [
      {
        "_index": "myesindex",
        "_type": "default",
        "_id": "5be1886e8ff5f340947b907ce2ac9e9d",
        "_score": 0.2824934,
        "_source": {
          "firstname": "Aimee",
          "lastname": "Mann",
          "description": "singer songwriter",
          "timestamp": 241
        }
      }
    ]
  }
}

Do I need separate Cloudant & Elasticsearch?

Whether you use Cloudant on its own, or pick Cloudant for storage and Elasticsearch for search, depends on your application and how it’s used.

If your application relies heavily on search, then you may want to direct the search requests to an Elasticsearch cluster that contains a copy of your searchable data. This would allow you to scale your search capability separately to your storage engine.

If search is an occasional or secondary activity in your app, or if you prefer the convenience of fewer moving parts, then you may wish to use Cloudant’s built-in free-text search. While not as fully-featured as Elasticsearch, it can easily handle free-text searches, range queries, and faceting as it is built on the same Lucene library that powers Elasticsearch.

Open-Source ftw

If you want to build on this utility, the code is on GitHub. It’s free and open-sourced via the Apache-2.0 license. You may want to modify the onchange.js action to alter the document before it’s written to Elasticsearch. You may also need to deploy multiple changes feed listeners for multiple data sources. It’s up to you.