Partitioned Databases and Node.js

May 24, 2019 | Glynn Bird | Node.js Partitioned Databases

The Cloudant database has three supported client libraries: Node.js, Java and Python. In this post, we’ll see examples on how the Node.js library can be used with the new Partition Databases feature.

Version 4.1.0 and above of the Cloudant Node.js library has support for partitioned databases.

Here’s a table of all the functions we’ll be using:

Operation Raw API Call Node.js function call
Create database PUT /db?partitioned=true db.create
Add document POST /db db.insert
Get document GET /db/<id> db.get
Get info GET /db/<partition_key> db.partitionInfo
Get all docs GET /db/<partition_key>/_all_docs db.partitionedList
Create Cloudant Query (CQ) Index PUT /db/_index db.index
Query CQ Index GET /db/<partition_key>/_find db.partitionedFind
Create Cloudant Search (CS) Index PUT /db/<design_doc_name> db.insert
Query CS Index GET /db/<partition_key>/_design/<ddoc>/_search/<search_name> db.partitionedSearch
Create MapReduce (MR) Index PUT /db/<design_doc_name> db.insert
Query MR Index GET /db/<partition_key>/_design/<ddoc>/_view/<view_name> db.partitionedView

All of the following code snippets sit in the main function in the code below:

const Cloudant = require('@cloudant/cloudant');
const cloudant = Cloudant('https://username:password@host');
const db = cloudant.db.use('mypartitioneddb')

const main = async () => {
  // code snippet goes here
}

main()

Remember to replace the Cloudant credentials with your own.

library

Photo by Amelie Ohlrogge on Unsplash

Create database

To create a partitioned database, simply add the { partitioned: true } object as the second parameter to create:

await cloudant.db.create('mypartitioneddb', {partitioned: true})
// { ok: true }

Insert data into a partition

No special functions are required to add data, just remember to supply a two part _id field - <partition_key>:<document_key>:

// document to add - notice the two-part _id
const doc = { _id: 'canidae:dog', name: 'Dog', latin: 'Canis lupus familiaris' }

// insert the document
await db.insert(doc)
// { "ok": true, "id": "canidae:dog", "rev": "1-3a4c4c5d65709bcb3ec675ec895d4051" }

Fetch document by _id from a partition

Again, no special functions required here. Just use get to pull back a single document by its _id:

// fetch a document by its id
await db.get('canidae:dog')
// { _id: 'canidae:dog', _rev: '1-3a4c4c5d65709bcb3ec675ec895d4051', name: 'Dog', latin: 'Canis lupus familiaris' }

Get Partition Info

To fetch the information about a single partition, use the partitionInfo function and pass a partition key:

// get partition information from the 'canidae' partition
await db.partitionInfo('canidae')
// {"db_name":"myhost-bluemix/mypartitioneddb","sizes":{"active":392,"external":332},"partition":"canidae","doc_count":4,"doc_del_count":0}

Get all documents from a partition

To fetch all of the documents from a partition, use the partitionedList function:

// fetch all documents in the 'canidae' partition, returning document bodies too.
await db.partitionedList('canidae', { include_docs: true })
// { "total_rows": 4, "offset": 0, "rows": [ ... ] }

Partitioned Cloudant Query

Create a partitioned index

To create an index that is partitioned, ensure that the partitioned: true field is set when calling the insert function, to instruct Cloudant to create a partitioned query, instead of a global one:

// index definition
const i = {
index: {
  fields: ['name']
},
name: 'nameindex',
ddoc: 'partitionedquery',
type: 'json',
partitioned: true
}
// instruct Cloudant to create the index
await db.index(i)
// { result: 'created', id: '_design/partitionedquery', name: 'nameindex' }

Find within a partition

To perform a Cloudant Query in a single partition, use the partitionedFind (or partitionedFindAsStream) function:

// find document whose name is 'wolf' in the 'canidae' partition
await db.partitionedFind('canidae', { 'selector' : { 'name': 'Wolf' }})
// { "docs": [ ... ], "bookmark": "..." }  

Create a partitioned search index

To create Cloudant Search index that is partitioned, write a Design Document to the database containing the index definition. Use options.partitioned = true to specify that this is a partitioned index:

// the search definition
const func = function(doc) {
  index('name', doc.name)
  index('latin', doc.latin)
}

// the design document containing the search definition function
const ddoc = {
  _id: '_design/searchddoc',
  indexes: {
    searchindex: {
      index: func.toString()
    }
  },
  options: {
    partitioned: true
  }
}
await db.insert(ddoc)
// { ok: true, id: '_design/searchddoc', rev: '1-e7257e575d666ca062b4fe0bdeb6fba1' }

Search within a partition

To perform a Cloudant Search against a pre-existing Cloudant search index, use the partitionedSearch function:

const params = {
  q: 'name:\'Wolf\''
}
await db.partitionedSearch('canidae', 'searchddoc', 'searchindex', params)
// { total_rows: ... , bookmark: ..., rows: [ ...] }

MapReduce Views

Creating a partitioned MapReduce view

To create a MapReduce view, ensure the options.partitioned flag is set to true to indicate to Cloudant that this is a partitioned rather than a global view:

const func = function(doc) {
  emit(doc.family, doc.weight)
}

// Design Document
const ddoc = {
  _id: '_design/viewddoc',
  views: {
    familyweight: {
      map: func.toString(),
      reduce: '_sum'
    }
  },
  options: {
    partitioned: true
  }
}

// create design document
await db.insert(ddoc)
// { ok: true, id: '_design/viewddoc', rev: '1-a062b4fe0bdeb6fbe7257e575d666ca1' }

Querying a partitioned MapReduce view

To direct a query to a pre-existing partitioned MapReduce view, use the partitionedView (or partitionedViewAsStream) function:

const params = {}
await db.partitionedView('canidae', 'viewddoc', 'familyweight', params)
// { rows: [ { key: ... , value: [Object] } ] }

Global indexes

A partitioned database may still have global Cloudant Query, Cloudant Search and MapReduce indexes. Create the indexes as normal but be sure to supply false as the partitioned flag, to indicate you need a global index. Then query your index as normal using db.find, db.search or db.view.