Cloudant Blog

Announcing BigCouch 0.4

Alan Hoffman — Thu, 26 Jan 2012 00:00:00 PST

It is a big day here at Cloudant HQ; we are announcing the release of BigCouch 0.4! This release, which brings BigCouch into API equivalence with Apache CouchDB 1.1.1, has been baking for a while, and we are excited that it is now ready for public consumption. Instructions for installing and using BigCouch 0.4 can be found on the BigCouch page. Users running Debian Squeeze, Ubuntu (LTS or newer) or RedHat / CentOS / Amazon Linux are welcome and encouraged to use our prebuilt distributions based on Erlang/OTP R14B01 and SpiderMonkey 1.8.5.

There are a whole slew of new features and performance improvements in this release. You can read the entire list here, but I wanted to highlight a few of our favorite features. These should convince you to upgrade if you haven’t already.

For my money, the killer addition to BigCouch 0.4 is support for ‘zones,’ which allows you control over where you place the individual copies of your data. Cloudant’s hosted service makes extensive use of this feature to protect against isolated failures in individual data centers. As an example, our west coast cluster “Meritage” is actually spread over two different data centers in two distinct geographic regions. We use zone-support to ensure that at least one copy of all of our customers’ data exists in both locations. That way, if one location falls into the ocean or is subject to nuclear attack, our customers’ data is still safe and available. With this feature now in BigCouch, you can extend clusters across multiple locations.

Another important improvement we added, taken from Apache CouchDB 1.1, is support for the replicator DB to help manage multiple replications. The BigCouch replicator DB works the same as the Apache CouchDB version. A nice writeup of usage can be found here.

We’d like to thank everyone in the CouchDB community that has contributed bug reports, patches, feature requests, etc. for BigCouch and to all the brave souls willing to try out the bleeding-edge releases. This release would not be nearly the upgrade it is without all of you. Special shout out goes to Benoit for his help with FreeBSD and native SSL support.

Finally, for those following along with recent events in the land of CouchDB, we’re dedicated to donating and integrating BigCouch to Apache CouchDB. This sort of integration won’t happen overnight, so we are still planning on making releases to BigCouch while we iron out the divergences in the two projects. We very much appreciate the support and enthusiasm we have recieved from the CouchDB community to date, and we look forward to working with that community to strengthen Apache CouchDB.

Cloudant SF Drinkup

Sam Bisbee — Tue, 17 Jan 2012 00:00:00 PST

Cloudant is happy to be sponsoring Node Summit next week in San Francisco. To keep the party going an extra day we will be hosting a CouchDB drinkup Thursday night to touch base with our west coast crew. This is instead of Benoit and Randall's meetup which they kindly merged with us - community powers unite!

Just look for the folks in good lookin' Cloudant swag.

Thursday Jan 26th
5:30 - 8:30
21st Amendment Brewery
563 2nd St, San Francisco
All Beers on Cloudant's Tab

The Future of Apache CouchDB

Adam Kocoloski — Thu, 05 Jan 2012 00:00:00 PST

The future of CouchDB is Apache CouchDB.

Cloudant over the past 3 years has built and refined BigCouch, a fault-tolerant, horizontally scalable clustering framework purpose-built for CouchDB. Today we announce our intent to contribute this work back to the community. Working with the ASF and the CouchDB community, we hope to integrate the core capabilities of BigCouch into Apache CouchDB. Hopefully this will put to rest the tired (and false) “CouchDB doesn’t scale” meme. BigCouch forms the bedrock of a globally distributed, sophisticated technology stack that we’ve had in production operation for over two years at scale.

Those of you building applications with CouchDB know that its performance has been improving by leaps and bounds. We’ve achieved this through low-level implementations of performance-critical routines and smart refactoring of the Erlang codebase, but without sacrificing the rock-solid durability and stable REST API people have come to expect from CouchDB. The growing community of active committers includes passionate, stellar developers. More and more people are using CouchDB for projects, big and small, every day.

We, along with a host of other companies, strongly support the open source community in building CouchDB and we do not plan on stopping. We have been fortunate in our ability to attract outstanding engineers, investors, and customers. We intend to continue devoting resources to Apache CouchDB and offer our help in any way the community desires. The future of CouchDB is CouchDB.

Damien Katz has been an excellent founder and steward of CouchDB. It’s been an honor to work with him, and we wish him well in his new location and mission.

A Big Finish for 2011

Alan Hoffman — Thu, 15 Dec 2011 00:00:00 PST

2011 has been quite an eventful year for Cloudant. In the spring we outgrew our tiny basement offices in Davis Square and moved Cloudant HQ into Boston proper. We added a number of fantastic engineers to the team and we said goodbye to some of our earliest hires.

Going into the new year, we’ve expanded our platform to 5 geographic zones in the US and 3 more around the globe. Most importantly we more than doubled the number of developers, applications, small businesses, and enterprises using our hosted service – and found quite a few independent BigCouch users out there along the way. We continue to be surprised and delighted by the passion our users show for our products. It is that passion that drives us to continuously improve BigCouch and our hosted service.

With all that has been going on here, the founders decided it was time to expand the team. So today we are happy to announce that Derek Schoettle has joined Cloudant as our new CEO. Yes, Derek has an MBA – don’t hold that against him. His passion is working within early-stage companies bringing great technology to market, most recently at Vertica, and numerous startups before that. Derek has been an advisor to Cloudant for most of 2011 and was a natural choice when the time came. We are very excited to have him on board helping Cloudant grow.

With Derek on board, Adam, Mike and I will have more time to focus on the things we are passionate about . For me, that means working closely with our customers and users, focusing on building a great product that solves their data problems. Mike will continue to focus on customer success and outreach, and will hopefully have more time ‘on the road,’ evangelizing BigCouch to a wider audience. Adam will continue to work deep in the guts of Apache CouchDB, BigCouch, and the Cloudant platform, making all of our crazy thought experiments into real features.

2011 was a big year for Cloudant, but we have even bigger goals for 2012 (assuming the world doesn’t end.) Stay tuned to this channel as we announce some of the big features and plans we have.

Alan (for Adam and Mike)

Co-Founder, Cloudant

Slides and Presentation from 2600hz

Alan Hoffman — Wed, 16 Nov 2011 00:00:00 PST

On December 14th we hosted a webinar with our friends at 2600hz, the open-source VOIP platform. Darren Schreiber, CEO of 2600hz, gave a presentation on how his company uses the BigCouch data platform to better serve its customers. Darren gave some great examples of how moving to a scalable document store has helped his company move faster, save money, and avoid operational headaches.

In case you missed it, we recorded the webinar and you can view it below. If you just want the slides, follow the link below the video.

Download the slides here

Monsanto Chooses Cloudant

Tim Anglade — Wed, 12 Oct 2011 00:00:00 PDT

Big day for Cloudant — and big news in the near future for all BigCouch users. We are happy to announce that biotech heavyweight and perennial Fortune 500 company Monsanto has chosen Cloudant to be the core of their new genome analysis platform.

We’ve been working with them for a few weeks now and we couldn’t be more thrilled with the partnership. We are no strangers to the analysis of the big bytes generated by the smallest of things. Our founders first devised our open-source platform BigCouch while at MIT, working on Large Hadron Collider experiments generating millions of data points per second from the collision of atomic particles. Turns out, Biotechnology & genome analysis is no different in this regard. As GigaOM summarized it: “innovation is rampant, but data growth is outpacing the ability to analyze it, making faster, cheaper and more scalable data systems integral to leveling the playing field”.

This is new ground for us — and for NOSQL as a whole. This Monsanto platform is not destined to toil away in one of their (many) R&D labs, only used by scientists with lab coats & beakers. We’re not powering the CMS of some administrative division or a minor part of their website. Cloudant’s BigCouch will be the core, for both storage and analysis of a new, company-wide platform powering a fundamental aspect of a Fortune 500 business: the analysis & identification of new traits & genomic combinations in agricultural crops. The data & reporting interfaces will be used across Monsanto and should be instrumental in the making of key business decisions.

Now, this also means big news for our users. Monsanto has been very eager to let us open-source many of the core enhancements we make to BigCouch for them. So while this won’t quite make it into the impending BigCouch 0.4 release (which is days away now), or on our cloudant.com clusters for another few months, you can definitely expect to see many Monsanto-driven improvements folded back into our offerings, such as support for more languages on view servers, new tools to model data workflows, as well as a flurry of improvements to our storage & retrieval algorithms.

At Cloudant, we’re all excited to see NOSQL platforms, including BigCouch, being used to power the next generation of business technology. So when announcements like this also translate to a more stable, performant & productive experience, for everybody else that uses a Cloudant product, we find it to be a big day indeed.

You can always contact us if you have any questions. There is also a press release available.

Optimizing your CouchDB Calls by 99%

Tim Anglade — Tue, 27 Sep 2011 00:00:00 PDT

Earlier this month, I gave a presentation at GoGaRuCo about using CouchDB, and how many people do it in inappropriate or wasteful ways.

The centerpiece of my talk was a step-by-step example of best-practices for CouchDB interaction, leading to an improvement of 99% in the median time it takes to do a simple insert to CouchDB (or Cloudant). Admittedly, I start from a worst-case scenario, but the scary part is that it is not a totally unrealistic one; in fact, I’ve seen it replicated many times over.

After factoring my advice in, we have a performance improvement of 96% (if you stick to conversing with CouchDB through JSON, which is recommended for now), or 99% with my experimental patches to support MessagePack in CouchDB & CouchRest.

What’s more, none of the advice I give leads to a relaxation in consistency or durability. It’s plain optimization, no strings attached, just there for the taking. Many of the points I touch on are also valid no matter what language or library you are using (or are easily translatable to other ecosystems), so I would strongly encourage anybody using CouchDB and interested in performance optimization to look at the video.

Head on over over to Confreaks to see the video (the meat of the talk starts at the 5:10 mark).

Using Cloudant from Scala on CloudBees

Michael Neale (CloudBees) — Fri, 15 Jul 2011 00:00:00 PDT

This blogpost originally appeared as “NoSQL: CouchDB with CloudBees and Cloudant” on the CloudBees Blog.

In this tutorial I will build a CouchDB based url shortener service (http://cloudbe.es) using CloudBees and the services/add-on platform with Cloudant. CouchDB is a popular NoSQL database - specifically you could called it a “document database” - i.e. it is all about storing documents which consist of sets of fields in a JSON document against a key. There is no schema - a document can consist of any JSON structure - there is no requirement that the same fields be present in each document unlike rows in a RDBMS. Of course CouchDB has a lot of the other benefits of NoSQL databases - high scalability and ease of use. Cloudant provides an excellent hosted CouchDB service. This takes care of all the management of CouchDB you would normally need to do - compaction, backups, and more.

Building a URL shortener

A url shortener is just that - takes a long messy URL and generates a token/shorter URL you can pass around.

Step 1: Get a cloudbees.com account, signup to a free RUN@cloud service

Step 2: Go to the subscription page — and choose Cloudant

Step 3: Create a new web app skeleton

For this I will use the Play! Framework - a simple framework for building RESTful web apps. I can use the command

play new urlshort --with scala

This will create an empty project using scala. To deploy to CloudBees- a handy way is to add cloudbees 0.2.1 to the dependencies file (don’t worry - source repo will be available to look at).

Step 4: Create an empty database

From grandcentral.cloudbees.com/services you can navigate to your Cloudant/CouchDB console. Any time you need to get back to the CouchDB console you can navigate from grandcentral.

This shows the “Futon” web interface to browse your data, quite a handy tool. Here you can create a new database - I called mine “urly”. Once you have a database, at the top left corner you can see an icon which will take you to the Cloudant management screen - click through to the newly created database, and you will be able to set permissions, and generate an api key:

The important bit is to note the api key details (a user name and password that you will use in your app). The user just generated should also get read/write access to your database (set it with the checkboxes).

Step 5: Set up credentials in your application config

In this case, as I am using the Play! framework - the configuration is in application.conf. The keys I set up were:

    #CloudBees details
    bees.api.key= key for deploying
    bees.api.secret= secret for deploying
    bees.api.domain=michaelvideo
    bees.api.name=urly

    #Cloudant details
    couch.user=api key user
    couch.password=api key password

I put the CloudBees details in there for easy deployment but technically you don’t have to (you can pass them to command line if you like).

Step 6: Talking to CouchDB

One of the really nice things about CouchDB is that you don’t need any drivers or dependencies to talk to it — https is a “native” api for it. So all that is needed is a simple http/REST/WS client.

Urls are how you address content - you can store a document in a nominated key for example by PUT’ing a JSON body to https://michaelvideo.cloudbees.cloudant.com/urly/ (urly is the name of the database). Updates to data require you to specify the current version tag of the data (field in the JSON body) as a form of optimistic locking.

Getting the data back is a simple GET ! (obviously there is a lot more you can read about if you need). In this case the keys are the randomly generated “slugs” that form the short URL.

Step 7: Wire up the application

Descending from the mountain top, a simple codebase which generates random url “slugs” and then does appropriate http redirects was carved in stone (also available on github):

Authentication with CouchDB is using http basic over SSL (using the api keys mentioned earlier).

Step 8: Deploy

(This will use the Play! CloudBees plugin - thanks Ivan !)

I then set up a domain and a record pointing to the host for the app, running at http://cloudbe.es/

Note that there was no need to migrate any data, or setup a schema (just create the database by name - which I could have also done programmatically).

The data is safely stored in CouchDB (below is browsing it with Futon):

MapReduce and Views

A further enhancement is to create views. CouchDB uses a Map Reduce model to provide views and aggregate calculations.

A view is similar to what relational databases offer and only requires a map function to be provided. Reducing is a more complicated topic and I won’t show it, but it could be used, for example, to calculate aggregates (sum or average over some range of documents). The advantage of a Map Reduce model is that the calculation can be distributed and run in parallel with the data - good for very large sets of data.

The Map function above simply takes a document and emits another document which is made up of the slug and the agent string (each browser provides an agent string). The key in this case is the url we stored - this means we can (if we want) lookup slugs based on URL (the reverse of what we were doing before). Note that as the fields in each document are optional - even if there is no agent field the above view will still work (just will have no agent field in the view output).

The functions are written in javascript - and are themselves stored as documents (with a special naming convention). The results of these views are accessed the same as the normal documents you write data to.

Hopefully from this you get a taste of the convenience and power of CouchDB and Cloudant.

You can sign up for both CloudBees (if you don’t have an account) and the Cloudant service on the CloudBees website.

A Technical Look at Cloudant Search

David Hardtke — Thu, 14 Jul 2011 00:00:00 PDT

As we just announced, Cloudant has added fault-tolerant, federated full-text search to our document database offering.

Cloudant Search builds upon our existing database to offer full-text indexing and search. It makes use of the Apache Lucene query syntax and large portions of the Lucene framework, but uses a new data storage model entirely within the CouchDB database framework. The inverted index (the map of terms to documents in which they appear) is calculated using our incremental map-reduce framework. This means the database is always available for document insertion and reading even if the indexing task backs up - other search solutions only allow documents to be added as quickly as they can be indexed. Newly indexed documents are immediately searchable – there is no latency while the inverted indices are rebuilt.

Documents within Cloudant are distributed to multiple nodes using consistent hashing in a ring like structure. The inverted index data associated with each document is saved on the same node as the document itself. Searches are done at the shard level, and results are combined in a federation layer. If a particular node in the cluster is offline, other copies of the shard are used. Cloudant utilizes the same underlying distribution and federation technology from the open-source Bigcouch project.

Important features of Cloudant Search include:

Text analysis can utilize existing Java based Lucene analyzer. Java code can be loaded into the database and stored along with the search indices.
Support for indexing in many languages (Java, Javascript, python, erlang, ruby, etc.) User indexing code is also stored in database.
Ability to run ad-hoc queries against multiple independent map-reduce views of data.
Full support for all JSON types during indexing (Strings, Numbers, JSONObjects). Pure Lucene search solutions only support String tokens.
Support for following query types: Boolean, Range (text, numerical, date, and compound key ranges), Phrase, Sorted, Prefix, Dates
Ability to query a “stale” snapshot of the database in case where indexing has not caught up to document insertion
Horizontal scalability through addition of database nodes
HTTP based search API consistent with CouchDB view API.

We see several distinct use cases where Cloudant Search will be superior to existing products. Since Cloudant runs as both a low cost hosted service and as a dedicated deployment for the largest customers, developers can build products around Cloudant Search and know that they will scale to terabytes of indexed data and high IOPS rates. An ideal use case would be mobile applications that require real-time geographical bounding box search coupled with text queries. Cloudant Search’s native support for numerical index types allows this. Cloudant Search’s reliance on CouchDB views also opens up unique possibilities. Multiple algorithms can be used simultaneously to search, score and sort documents.

To get started with Cloudant search, you first need to specify how your database is indexed by uploading a new CouchDB design document (indexing instructions). Then you can use the search API. There is also a full description of the query syntax.

Announcing Cloudant Search

Tim Anglade — Thu, 14 Jul 2011 00:00:00 PDT

I’ve always strongly felt that using NOSQL wasn’t so much a choice as a necessity. That most successful NOSQL deployments start with the intimate knowledge that your set of requirements — from speed & availability to operational considerations and budget — cannot be met with a relational database, coupled with a deep understanding of the tradeoffs you are making. Among those, perhaps no tradeoff has been felt more deeply by NOSQL users worldwide, than the eponymous loss of a natural, instantaneous way of accessing your data through a structured query language. We all came up with our own remedies; more often than not, that substitute was based on MapReduce: Google’s novel, elegant way of explicitly parallelizing computation over distributed, unstructured data. But as the joke goes, it’s always been a non-starter for the more novice users out there, and where suits & ties are involved.

CouchDB Views (as our brand of MapReduce is called) come with additional concerns, as they are pre-computed and written to disk. While this is fine — and actually, extremely useful — for the use-cases and small scales a lot of Apache CouchDB deployments reside at (single instances working off a limited dataset), this behavior is somewhere North of nagging and South of suicidal for the data sizes & use-cases most Cloudant customers have to deal with. Part of the promise of our industry is — or should be, anyway — to make your life & business easier, no matter how much data you have. And so, while CouchDB Views have been, and will undoubtedly remain, an essential tool to index, filter & transform your data, once you know what to do with it; and while its various weaknesses (explicitly parallelized syntax, lengthy computation, heavy disk usage) are also the source of its most meaningful strengths (distributed processing, high performance on repeated queries, persistent transformations), we at Cloudant saw a clear opportunity to offer a novel, complementary way to interact with your data.

A way that would allow you to interact with your data instantaneously; wouldn’t force you to mess around with MapReduce jobs or complex languages; a way that would not require you to set up a third-party, financially or operationally expensive solution.

We call this way Cloudant Search. And today, we’re proud to announce its immediate availability, as a public beta.

Want to easily find all the documents that contain the word “bieber”? This is the Cloudant Search query you have to write:

bieber

Want to find all the records that have “my world” in their title? Just write:

title:"my world"

How about only finding all the artists who were born in Canada in February & March 1994?

type:artist country:canada dob<date>:[1994-02-01 TO 1994-03-31]

How about people who are fan of either Justin Bieber or Justin Timberlake, that live in JBiebz’ hometown in Ontario — wait, what’s it called, Stratwood? Stratburg? Strat- something. Oh, I’ll just search it:

type:person fan-of:(justin (bieber OR timberlake)) city:strat* state:ontario

And that’s just scratching the surface and you can see more query examples in our knowledge base. Of course, all those queries are also served instantly, and new records you put in your database will be indexed in real time.

But Cloudant Search goes beyond ease of access for developers, analysts & end-users.

For the IT & DevOps teams out there, and for our own sake (since, as maintainers of hosting service cloudant.com, we have the distinction of being the only NOSQL company tasked with operating its own product around the clock), we worked tirelessly to make it capable of tackling terabytes and easy to maintain under load. It is baked in directly into the Cloudant codebase and leverages existing, proven technology, such as the Lucene indexers, CouchDB’s view indexing logic and of course, Cloudant’s own Dynamo-inspired distribution algorithms. Compared to external search & analytics solutions such as Hadoop, SAS, Solr, Sphinx or ElasticSearch, this integration & simplicity effectively translates to more flexibility to address ever-changing use-cases, a drastically reduced operational complexity and, at the end of the day, a much lower total cost of ownership for start-ups and established businesses alike.

Finally, for the power-users out there, we’ve made Cloudant Search extremely extensible. As our documentation details, you can write your own indexing algorithms, leverage Lucene tokenizers or even, customize the Cloudant Search indexer itself, with a bit of Java code.

Now, I’d be engaging in doublespeak if I didn’t make it clear that Search is still bound by some of the tradeoffs that epitomize our industry. Like some of our esteemed colleagues, we strive to make our users & customers aware of the adjustments that, yes, define our weaknesses but more importantly, are also the source of our strengths. Among those, I should mention that Cloudant Search will require an initial, one-time indexing and this single index itself will take up disk space. To limit resource consumption and self-induced podiatric harm, wildcard (“som*thing”) or suffix (“*thing”) searches are not allowed (only prefix searches like “some*”). And yes, some of its more advanced features like the dynamic sorting of results, are still memory-bound and as such, should be used carefully.

There are no silver bullets in this industry. That is something that the Cloudant team has kept in mind through the months we spent developing Cloudant Search. But more than an admission of eventual defeat, we’ve always taken it as a challenge to innovate at the pain-points. To explore the areas beyond the boundaries of our existing trade-offs. To find tasteful solutions to gritty problems. Today, we’re happy to bring you the fruits of the labor, share our response to this problem, and in a way, pass the challenge on to you: we can’t wait to see what you will do with it. We’re confident that Cloudant Search will open new doors, from user-facing analytics dashboards, to efficient Object-Document Mappers, to killer apps we haven’t even dared to think of.

So go ahead and read more about the technology, peruse the docs, activate Search on a database through your cloudant.com dashboard or contact us to see how this technology can be deployed in your datacenter.

.NET + CouchDB = ♥: Announcing the Cloudant Add-On for AppHarbor

Tim Anglade — Tue, 28 Jun 2011 00:00:00 PDT

The NOSQL movement originally found its community roots in some the “edgier” platform & languages of this world; CouchDB in particular, has always felt strongly backed by a certain kinship with the Ruby and Rails communities, amongst others. But as one of the most mature NOSQL projects out there, CouchDB now also enjoys energetic support in the more established communities such as PHP, Java or even .NET.

As it turns out NOSQL isn’t the only movement kindling a relationship with the .NET framework. The “Cloud” has also taken root in that community, and a service called AppHarbor has arisen to meet the demand for reliable, elastic & pay-as-you-grow .NET hosting.

I’d be beating around the bush if I introduced AppHarbor as anything else but “Heroku for .NET”. That description may sound catchy but in and of itself, does not a great service make. No, what really caught our eye about our fellow ex-YCombinator brethren Michael, Rune & Troels, is their flawless execution of the concept and their tactful tweaks to the Platform-as-a-Service model. By giving you access to the Common Language Runtime and the .NET framework, AppHarbor opens more doors for developers than just single-language execution. You instantly gain access to the years of loving care, performance tuning and functional extensions that Microsoft has poured into its platform, as well as support for a flurry of languages.

So today, we’re happy to announce the Cloudant add-on for AppHarbor. It will give you instant provisioning of our hosted CouchDB solutions, for all your hosted .NET apps. Your Cloudant databases will be co-located in the same datacenter as AppHarbor, so you can enjoy the same reliability & performance you’ve come to expect from Cloudant, with the absolute minimal latency possible. And on the business side, you’ll also get the accounting convenience of a single monthly bill for both your app & data hosting expenses. (Don’t forget to put a coversheet on that TPS report before running it by the boss, though!)

So go on ahead and get started with one of the many CouchDB libraries available for the .NET framework, such as Divan, Relax, Hammock or EasyCouchDB. Or peruse this recent introduction to CouchDB for .NET developers. (And please send us an email to show us what you’ve built with AppHarbor + Cloudant!)

What are you still doing here? Go sign up for AppHarbor… You’re in good hands with Michael, Rune & Troels.

Paul Davis is Joining Cloudant!

Tim Anglade — Mon, 27 Jun 2011 00:00:00 PDT

We are thrilled to announce that noted Apache CouchDB community member & committer Paul Davis will be joining Cloudant in July.

Paul had been on our “must-hire” list for a while now, and he will be joining a technical team that now includes a third of the currently active committers to Apache CouchDB! We’ve always tried to be faithful stewards and dedicated supporters of the Open Source community behind CouchDB, and this commitment will only grow stronger with Paul’s inclusion in Team Cloudant.

Paul’s background in large distributed systems & data analytics, combined with his expertise in Erlang and hardware optimization will undoubtedly come in handy as he helps us solidify our quality of service, add features to our platform and get BigCouch ever closer to 1.0.

You can learn more about Paul through his github, twitter or website.

Bottoms up, Paul. We couldn’t be happier to have you working with us.

Mike Miller Talks Cloud Databases at GigaOM Structure

Alan Hoffman — Fri, 24 Jun 2011 00:00:00 PDT

Yesterday, Cloudant Chief Scientist (and VP of powder-blue shirts) sat on a panel at GigaOm’s Structure Conference. Along with representatives from Xeround, ParAccel, and NimbusDB, Mike discussed the ins and outs of running databases in a Cloud environment. As the other three panelists represented more traditional relational and data-warehousing tchnologies, they turned to Mike to represent NOSQL databases as uniquely suited for high performance, elasticity, and operational stability.

We at Cloudant are delighted to have been chosen for this panel. This is the third year in a row we have sent delegates to Structure and we have always found the speakers, panels, and hallway conversations to be very enlightening.

You can watch a video of the panel, or read PCWorld’s writeup.

CouchDB & Java in the Cloud, Now Easier than Ever with CloudBees

Tim Anglade — Tue, 21 Jun 2011 00:00:00 PDT

Today, Cloudant is proud to be a launch partner of the CloudBees Ecosystem. Effective immediately, our line of hosted CouchDB cloud solutions is available directly through CloudBees’ Java platform, with a simple API, unified billing and integrated account management.

CloudBees started out in 2010, with the goal of giving Java developers & organizations a platform, that not only lets them run their applications online easily, but also gives them specific tools to develop, test & integrate them. This twist in the usual PaaS model means they’re the only platform out there that aims to cover the entire lifecycle of traditional Java application development. So today, we’re thrilled to be part of this vision, by helping them launch an add-on program that also features New Relic, SauceLabs, Sonar & JFrog.

Considering the recent increase in CouchDB adoption in the Java community (bolstered by the existence of several drivers & persistence layers, such as our own fork of couchdb4j, ektorp, jcouchdb and jrelax), we’re more than happy we answered CloudBees call, integrated deeply into their platform, and we can’t wait to see what you will build with us!

So go on ahead and check out the details of their announcement or sign up for the CloudBees platform right away!

A Closer Cloudant? Now Accepting Reservations for a European Cluster!

Tim Anglade — Tue, 24 May 2011 00:00:00 PDT

One of the hardest parts of running a global business like Cloudant isn’t necessarily on the technical side of things. While we certainly got better at developing & operating a fleet of clusters that now spans both US coasts & Asia, we’ll admit to still being somewhat under-skilled at tackling some of our marketing concerns, such as estimating latent demand in foreign markets.

In recent months, our support team has been receiving more & more inquiries about provisioning Cloudant accounts in Europe. But instead of trying to guess who else is interested, run complex spreadsheet models or hire a team of consultants, we figured we’d just ask you. (Yes, you: Time magazine’s 2006 Person of the Year.) To limit the noise in this poll, we’ll ask you to put your money where your mouth is: we’ll only accept reservations from paying customers (i.e., users on an “Argon” plan or above).

Having your data located a cluster close to your application layer (or other consumers of your CouchDB docs) has a big primary benefit: latency. With a cluster in Europe, your median read & write latencies to Europe should go down by approximately 200ms (or by ~500ms for SSL access), when compared to our existing East Coast cluster .

So go on ahead to our reservation page. You’ll be asked to sign up for Cloudant if you don’t already have an account; or upgrade to a paid account if you’re currently on our “Oxygen” plan.

We’ll let you know how the process is going with a handy reservation progress bar. We’ll set up our European cluster and load it with the accounts that reserved a spot, as soon as that bar hits 100%!

Go ahead and tell us you’re interested!

Search Indexing in Javascript

David Hardtke — Thu, 19 May 2011 00:00:00 PDT

In order to leverage existing Lucene analyzers, our Search feature (currently in beta) makes use of Java views, rather than the traditional Javascript views you’re used to in CouchDB. Recently, we wanted to do some performance comparisons between the two; would the Java view server be significantly faster (or slower) than the Javascript view server on typical indexing tasks? To test this, we wrote a simple Javascript view (in the form of a CouchApp) that creates a Cloudant Search-compatible inverted index. The code is available on github here.

To use it, clone the git repository and use the couchapp tool to push it to your database:

$ git clone git@github.com:cloudant/jsindexer.git
$ cd jsindexer
$ couchapp push http://<user>.cloudant.com:5984/<db_or_couchapp_you_want_to_search>

With this simple push, you gain the ability to do a search against this inverted index. For example, if you want to search all the docs in that DB whose “comments” field contains “cloudant is awesome” and whose “rating” value is between 80 and 100:

$ curl 'http://<user>.cloudant.com:5984/<db_or_couchapp_you_want_to_search>/_search?q=comments:"cloudant is awesome" AND rating:[80 TO 100]&index=_design/jsindexer/_view/whitespace'

Obviously, that’s just scratching the surface. Our knowledge base has several articles about Search. You can go there to read our introduction to Search, learn more about the Search API or how to write your own Indexer.

As for the results of the performance comparison, our full text indexing speed is limited by the writing of the data to the CouchDB B-Tree, so Javascript and Java perform the same. So there you have it: full-text search from Javascript, a guilt-free geeky pleasure. Go ahead and have at it.

Fukushima from Afar

Mike Miller — Sat, 02 Apr 2011 00:00:00 PDT

In the past month I had the all too rare opportunity to do something that was both terrifically exciting and, I believe, deeply meaningful. On Friday, March 11, I awoke in Seattle to the news of the devastating earthquake and tsunami that struck Japan. As the day developed the sheer magnitude of the true devastation emerged and it was clear that sobering stories would continue to unfold for quite awhile.

I was doubly surprised by the emerging media coverage of the Fukushima reactor complex. The Japanese are absolutely top-tier in both their science and engineering and the very fact that Tokyo had so little damage from a 9.0 earthquake (imagine if that happened here!) suggested that they were equally prepared for the aftermath. To a large extent, they were — just not for a series of events that were so powerful that within days it was clear that the Fukushima situation had changed dramatically.

In this era of real-time data, it occurred to me just how hard it was for the educated public to gain access to the right data. What was the level of radiation being released by Fukushima? Was it enough to be dangerous to the people in Japan? Was it enough to impact our sensitive (and expensive!) physics experiments here in Seattle? Was it enough to be a public health concern in the US?

These questions, plus a little prodding from senior faculty at UW, put the wheels in motion for me. If information was going to continue to be so hard to find, then maybe we could do something about it. We are physicists, after all. Emails were sent, grad students were diverted from thesis writing and qualifying exam preparation, and by Monday we had worked to convert my lab at UW from Dark Matter detector R&D to an air radioactivity monitoring lab, containing some very expensive, one-of-a-kind equipment, along with the requisite quantities of duct tape and zip-ties holding everything together. This was no small feat, and it never would’ve happened without the work of a superb team and the right tools.

We knew that any radiation that was released would be heavily diluted in transit to the west coast, so we had to (i) find a way to sample enormous quantities of air (nearly 150,000 cubic meters per day) and (ii) protect our radiation counter and air samples from the copious amounts of normal background activity. A little known (or appreciated) fact is that natural radiation is part of your daily life, from radon decay products to bananas, and natural radiation would likely drown out any faint signal from Fukushima. By Wednesday we started recording data, and then it suddenly hit me — we were going to be producing a lot of data, and it was already coming in. The data had to be catalogued, processed, condensed, distributed, data models modified, reprocessed, redistributed, and we were learning as we went along. Not only was the data was important to us and our experiments, but the public was hungry for realtime results (that had to be correct!), so we didn’t have the luxury of writing a spec, reviewing it, building it, prototyping, or any of the typical processes. We already were running TV crews through the lab daily, time was short, and so I turned to the tool that I happened to know best: open source BigCouch, and specifically the free hosted version on Cloudant.

We began putting together a real-time BigCouch workflow that ultimately enabled us to complete complex processing (data transformation and aggregation, signal processing, statistical analyses and the like), real-time data monitoring and visualization, collaborative analyses, and ultimately to serve the results directly to the public from the database. At the end of this post I will briefly review my BigCouch experiences, both good and bad.

On Friday, March 18, I had just placed the overnight air filter sample into our monitoring station and was adding some additional signal processing to our workflow. After 20 minutes of counting I happened to look at the computer screen to check the realtime data acquisition histogram, and there it was, clear as day, the first arrival of 131-I in the Seattle air. I spent about 10 minutes double checking, looking for other characteristic fission products in the spectra, and then realized we had something very unique.

Racing down the hall, I gathered a team and the next week is an absolute blur. That Friday night, around 10, we knew that we had unambiguously detected fission fragments from Fukushima, and we had a good handle on the total level of the radioactivity levels. While results were far below any local health concerns (roughly 1000 times below the EPA limits), it was higher than I had personally expected, and we suddenly knew that the situation in Japan was indeed serious. Our thoughts go out to those that live near Fukushima, as well as the heroic workers at the plant as they fight to contain the incident.

As the week progressed, we saw the radiation levels peak, slowly decline, and peak again as we detected events that occurred roughly 7 days after they happened in Japan. Within a day of our initial sighting we had posted the numbers and plots online, and they continued to update in real-time as the situation unfolded. By the middle of the week we posted the first publication draft online, and are now witnessing the radiation levels drop below the limit of detectability in our experiment. If you are interested in the details, the pieces in the Washington Post, Nature News, and NY Times were particularly well written, in my opinion.

Before closing, I want to briefly review the pros and cons of using BigCouch/Cloudant for this critical application.

Pros

Flexibility. The ability to smoothly handle both structured and unstructured data was critical. The document philosophy allowed us to be agile with data types (metadata + binary data side-by-side) and date models (sometimes changing by 20x per day).
Cloud. Not having to install and maintain my own database server for this purpose was a huge win.
Scalability. This application isn’t the world’s biggest data producer, but it’s no slouch either. We were able to easily store and process Terabytes of data without breaking a sweat, and it was nice to know that we wouldn’t be limited by scaling concerns in the future.
Real-time Analytics. We were able to easily insert our domain specific libraries/application into an incremental MapReduce workflow to provide up-to-the-minute results.
Collaborative. The database is a web server, so we were immediately able to collaborate as a team. The fact that the db is also a web server that speaks JSON over HTTP enables simple integration with other standard tools for client-side visualization, etc.
Thin client applications. I was able to write ultra-lightweight client-side javascript applications for data monitoring and presentation. These applications were typically around 100 lines of code and served directly from the cloud to the browser, which means I didn’t have to set up my own web-tier stack, nor learn Rails, Django, etc. This has been on my list of “things to learn” for a year, and this pushed me to finally do it.

Now the things that can definitely be improved

Analytics integration. Like Hadoop, building a BigCouch analytics workflow is a bit raw and has a substantial learning curve. This experience makes me realize that there is tremendous low-hanging fruit for tools that bridge that gap for the average developer and/or analyst.
CouchApp. The client-side JS model is a huge win here, and the couchapp tool was a big help in turning my JS and HTML files immediately into an application in the cloud. However, I had to strip out a significant amount of unused baggage to make sense of it all. Further, there is still some work to be done to allow for rich applications that securely allow not just data reads from the client(which currently works well), but also writes.

In summary, BigCouch was a key enabling technology, and I will definitely use it for future large scale experiments. In this case, it helped maximize the value in a very time-sensitive big data problem.

Our thoughts go to the people nearest Fukushima. We are fortunate that the radioactivity in the US is so low. I feel both proud and humbled to have been able to contribute in some way, by bringing scientific information to the public in a time of great uncertainty.

Rise & Shine

Tim Anglade — Sun, 20 Mar 2011 00:00:00 PDT

You wouldn’t know it from looking out your window here in Boston but spring is supposed to start tomorrow. And so the Cloudant team once again spends a lazy March Sunday, shut indoors, hoping for milder spells.

But it’s already Monday in Japan and it seems like the tech community there has had a busy weekend.

In an effort spearheaded by the local branches of Google, Amazon, Yahoo!, Microsoft and many others, more than 200 developers have been sharing ideas, teaming up and designing apps that will help speed up Japan’s recovery after the disastrous earthquake, tsunami, nuclear scare and 632+ aftershocks they have been experiencing since Friday, March 11.

The endeavor is dubbed Hack For Japan and while the English language details are still sparse (see the idea poll or the ongoing Twitter stream), we immediately knew we wanted to help out in whatever way we could.

The Japanese CouchDB community is 150+ strong, has organized several “relaxons”, is writing an impressive community translation of “CouchDB: the Definitive Guide” and has a very active leader in Yohei Sasaki. Cloudant in particular, has enjoyed some traction in Japan for a while and we set up a cluster dedicated to serving that community last year.

Today, we’re moved by the resilience, kindness and positive outlook of the Japanese tech community. So we’re announcing we’re donating our spare Asia-Pacific cluster capacity to any Hack For Japan project that may need it. It should be enough for several hundred Argon or a few Krypton accounts. The details are here.

That’s just our way of saying thanks to the Japanese community, for always having our back, for keeping the lost, the injured & the stranded in their hearts, and for showing that you can always count on your fellow geeks to unite, build & help rebuild.

After all, it’s already the season of love, life & new hopes, over there.

“Using Incentive Prizes to Crowdsource Brilliance”

Mike Miller — Wed, 09 Mar 2011 00:00:00 PST

I can’t take credit for the title, it’s a direct quote from a fantastic talk that I just saw by Peter Diamandis, the force behind the X Prize Foundation. Peter is a superb speaker. The room was packed and he didn’t disappoint. He is passionate, eloquent and flat-out inspiring. Now, I’ve been incredibly lucky to be surrounded by inspiration at MIT, Cloudant and Y-Combinator, but… Something about this talk really struck home.

From our time in YC and beyond, Paul Graham has always been pounding home a single message: make something people want. By now I’m sure everyone has seen the t-shirts and knows the slogan. Something about this motto has always struck me as just a bit, well, incomplete. Yes, I want to make something people want. Yes, that’s as good of a guiding principle as any on the path to success. However, following that mantra to its logical end can actually lead to sub-optimal outcomes on a whole. I’d bet a decent chunk of entrepreneurs (PG clearly included) actually want to make the world a better place while following their passions. Most of us at Cloudant spent the better parts of our young careers trying (sometimes in vain) to understand what the universe is made of and how it behaves! And somehow I’ve always felt this tension between those two poles: what the market wants and what actually makes the world a better place. And that’s when Peter’s message finally hit home.

Peter recognizes that sometimes free market forces alone don’t converge on the global minimum. Sometimes they need a nudge to escape a local minimum, and Peter has made it his life mission to use incentive prizes to nudge the market as needed.

The X Prize Foundation is awesome. They identify hard problems with high stakes impact on society as a whole (water, medicine, energy, enviornment) and create an innovative/entrepreneurial environment to solve those problems. For profit. Yes the X Prize Foundation is strictly philanthropic, but the contestants are anything but. The foundation only gives prize money to the winning team, but that doesn’t matter – they’ve just created a new market from scratch, and a big one to boot! In the case of the first X Prize, for space flight (actually 100 km elevation, to be precise), they’ve created a billion dollar industry, nearly overnight¹. To an entrepreneur that is simply amazing: the prize and the ecosystem it spawned ameliorate the risk. You don’t have to win the prize to be successful. Great team, great gumption, and clarity of vision: you’ve got a fighting chance to do anything. Great team, great gumption, clarity, revolutionary space, and guaranteed market‽ Now you’re really talking!

Finally, Peter opened and closed with a question: “What are you passionate about?” This is a big one. At Cloudant, we are passionate about innovation and it’s heartening to see the explosion of not just new companies and technology, but also forces such as YCombinator and the X Prize Foundation that are changing both the way innovation happens as well as what it targets. PG and Peter D. — keep it up!

Ok, we can quibble about the market’s existence prior to the 1st prize (hell, Boeing is in the news daily here in Seattle). But.. I walked away convinced that there was at least $1B of new market created, in addition to anything pre-existing.
↩

New Cloudant HQ (with pictures!)

Alan Hoffman — Wed, 23 Feb 2011 00:00:00 PST

The number 402 is following me, you could call it my lucky number. And apparently I’m sharing the luck with my cofounders. I should probably take our investors’ money to Vegas and bet on 402. Back when I first moved to the east coast my first apartment was at 402 Broadway in Cambridge. My office at MIT, where I first met and worked with my cofounders was in Building 26, office 402. When we started Cloudant we set up shop in a small basement office at 402 Highland Ave. in Somerville. Starting to get creepy right? Well, this week we finally moved into our new office space in the south end. We had outgrown the place in Davis Sq. and as much as we are going to miss its horror-movie ambiance (no really, when we moved in that office looked like the set of “Saw”) we couldn’t be more excited about our new digs. Where are they? 580 Harrison Avenue, Suite 402. If I were a man that believed in such things, I would think the universe was trying to tell me something. My colleagues tell me that 402 is an obscure HTTP error code for “Payment Required.” Maybe the universe is calling in some IOUs, or maybe it’s just suggesting that we need to charge more.

Anyway, here are a few pictures of the team working in the new space. The new office has exposed brick wall, large windows without bars on them, and enough room for more than three desks. All of these amenities were lacking in the old office.

It’s a big upgrade from our previous place. If you have the time and the inclination, please stop by to say hi. We’re always happy to chat about CouchDB, startups, complex data problems, NOSQL, and our favorite Twilight Zone episodes. Oh yeah, and if all this sounds enticing (and how could it not?) we’re looking for a crack devops engineer to join the team. If you are serious about keeping the servers well-oiled, building distributed systems, and providing a great cloud service, apply here

BigCouch 0.3

Bob Dionne — Fri, 11 Feb 2011 00:00:00 PST

Time sure flies when you’re having fun. It’s been a little over three months since the first BigCouch release so we thought it might be a good time to package and release a new version. A lot of folks forked this first release and started using it, providing valuable feedback, logging issues and bug reports. We greatly appreciate the effort and hopefully have covered most of the issues. This release sports a major new feature in addition to lots of consolidation and bug fixes.

BigCouch has several simply architected layers that provide the core functionality. These are now broken out into separate sub-projects that are managed as dependencies by rebar, along with ibrowse, mochiweb and the other dependencies. This allows these projects to evolve independently and enables BigCouch developers to switch between various branches as needed. We’ve fixed several bugs and added specs and docs to fabric, which together with rexi provides our distributed CouchDB API. Rexi is actually a very nice RPC like layer that can be easily used standalone if needed.

We’ve done a lot to tighten up propagation of errors through the system, reducing the number of dead_shards errors that occur. For the most part these errors are harmless, often the result of a bad Javascript function in view code. At the suggestion of our operations team we’ve tagged each request with a unique id which is returned in the HTTP headers but is also written in the logs. This helps us track errors better as we’re able to map specific stack traces to API calls. We’ve also created an in-memory buffer in the rexi server that is accessible on the back end and maintains a configurable number of errors and stack traces.

The major feature addition in this release is shard synchronization in the mem3 layer. By proactively triggering replications between database shard copies we reduce the amount of time required to restore the proper replica level for all documents after a node outage. Towards this end we’ve implemented a new lightweight replicator customized for use with shards, and integrated it into mem3. We saw that this replicator could also be used to replicate the nodes and shard_db databases which are used to manage the cluster information, thereby allowing us to not need the heavier weight CouchDB replicator internally. It’s still new and has some kinks to work out but we wanted to get it into developers hands as soon as we could. We encourage your feedback.

Last, but surely not least, we’ve added Robert Newson to the BigCouch team. He’s hit the ground running and has already fixed some of the trickier issues with which we’ve been grappling.

These are just a few of the items covered in this release; all specific changes are enumerated in the release notes. You can always build BigCouch straight from the source, but for your convenience BigCouch 0.3 is now available via precompiled RPM and .deb packages on the downloads page at Github and as a cookbook for installation via the Opscode Chef platform.

Thanks for using BigCouch!

Some Thoughts on the Couchbase Announcement

Alan Hoffman — Tue, 08 Feb 2011 00:00:00 PST

That will teach me to go to sleep before midnight on a weekday.

I woke up this morning to find a shakeup in the world of NOSQL. CouchOne has merged with Membase to form Couchbase, “the most comprehensive data management solution for scalable web and mobile applications.” I’ll add mine to the many voices wishing luck to the new company and the communities it supports. We’re excited to see more and more resources being poured into the NOSQL space. Congratulations to Damien, James, and everyone at both companies.

We’ve been asked if this changes anything at Cloudant. The answer is no. We’ll continue to offer Cloudant’s hosted, managed, scalable data platform. Apache CouchDB remains an important part of our technology offering; it forms the core data store around which we’ve built our wider platform. We have always been advocates, contributors, and leaders in the CouchDB community and we see no reason to change that. We’ll continue our support for both CouchDB and BigCouch. We’ll continue to provide our customers and partners with a robust database, search, and analytics platform to power their mission-critical business applications – as we’ve done for the last two years.

I do want to take issue with one thing said in the press release for Couchbase. They say,

“Couchbase becomes the only document database capable of safely storing your data
whether stored on a single server, or spread across hundreds.”

Some of our customers have billions of documents stored safely on dozens of nodes in datacenters around the world. It’s too soon to say what Couchbase will become, but if you need a safe, scalable, and easy-to-use document storage platform, our technology already provides that today: cloudant.com/solutions.

Alan Hoffman

Co-founder and CEO, Cloudant

SearchApp – Add a Searchbox to your CouchApps and CouchDB Databases in 10 Minutes

David Hardtke — Thu, 03 Feb 2011 00:00:00 PST

Recently we released the developer preview of our full-text indexing and search for Cloudant’s hosted CouchDB. Here we will explain how to use a simple CouchApp to add search to your existing CouchApps and CouchDB databases. The goal is to make your data searchable in 10 minutes are less. Start your timers.

To get this running, you’ll need 4 things:

Git version control system
The couchapp command line tool is used to generate code templates in your application and to push your changes to an instance of CouchDB. This has been tested on couchapp version 0.6.1 and higher.
Free Cloudant account
An existing CouchDB database, CouchApp, or a data importer.

(Note: CouchApps are JavaScript and HTML5 applications served directly from CouchDB. couchapp is a command line tool used to create and manage CouchApps.)

If your CouchDB database is not already hosted at Cloudant, you’ll need to replicate it to Cloudant. I’m assuming your CouchDB database or CouchApp (named mydb) is on your local CouchDB instance (http://localhost:5984/mydb), and that your Cloudant username/password is <user>/<password>:

$ curl http://localhost:5984/_replicate -H 'Content-Type: application/json' -d '{ "source": "mydb", "target": "https://<user>:<pass>@<user>.cloudant.com/mydb", "create_target":true}'

Now, you’ll need to clone the SearchApp. To do this, use the git clone command:

$ git clone git@github.com:cloudant/SearchApp.git
$ cd SearchApp

For convenience, you can make your database the default target for your SearchApp. Modify the file .couchapprc as follows:

{"env":{"default":{"db":"http://<user>:<pass>@<user>.cloudant.com:5984/mydb"}}}

Now for the last step. Push your SearchApp on top of your database:

$ couchapp push

If you want to push to a different location that specified in the .couchapprc file, do:

$ couchapp push http://<user>:<pass>@<user>.cloudant.com:5984/mydb

You should see the following message:

2011-02-01 11:49:36 [INFO] Visit your CouchApp here:
http://<user>.cloudant.com:5984/mydb/_design/searchapp/index.html

Visit that page and you’ll have a working search box. Now check your watch – has it been more that 10 minutes? Your searches might come up blank at first if you have a large database. Your database needs to be indexed before it can be searched. The indexing is a distributed Map-Reduce view. You can check the status of indexing using the Cloudant dashboard at:

https://cloudant.com/#!/dashboard/<user>/mydb/stats

(be sure to insert your username in the url).

For a more details on configuring SearchApp, please look at the README file. To configure the indexing strategy, check out the search indexing page.

MapReduce from the basics to the actually useful (in under 30 minutes)

Mike Miller — Thu, 13 Jan 2011 00:00:00 PST

Recently, Tim Anglade culminated his NOSQL World Tour with the release of the NOSQL Tapes, a collection of live interviews collected globally. Not only do I find that site aesthetically pleasing (courtesy of our own Steadicat), but there is a tremendous amount of solid content, from technical discussions to opinions and conjectures from many of the young leaders in the field. I was fortunate to get a bit of Tim’s time when he stopped in Seattle and we recorded a longish video on MapReduce in my lab at UW. While I love the old school feel of a chalk-talk, I must admit I’m pretty embarrassed at how rambling and fragmented my explanations are, not to mention my shoddy use of available board space. Therefore the Physics Professor in me feels obligated to try again, albeit offline this time.

Brief NOSQL Taxonomy

To set the MapReduce stage, I’m going to offer a very naive taxonomy of NoSQL data stores (apologies to the experts). I have yet to meet a database that isn’t a key/value store, so let’s focus on some higher level distinction. In particular, let’s focus on the difference between big tables (BigTable, HBase, Cassandra, etc.) and document stores (CouchDB, MongoDB). All of these solutions are finding solid acceptance in the wild, so what – if any – difference is there? My one-word answer is flexibility. Huh, isn’t all NoSQL flexible? Well… yes and no. In my eyes, BigTable and its derivatives excel at one thing in particular: structuring your data for the efficient I/O access that is necessary to get blazing read speeds for large data sets. Somewhat orthogonal, document stores, especially CouchDB, prioritize flexibility. In particular, there is no requirement to define columns, column families, or super columns up front. You simply encapsulate your data as documents and push them into the store. How then, you may ask, can you get efficient queries? MongoDB treats the problem with blazingly fast in-memory table scans and document introspection for index builds, whereas CouchDB takes a different approach with something called incremental MapReduce. I’m going to briefly explain the latter and then show you how it works in action.

MapReduce Primer

MapReduce is an old design pattern that was recently made famous by Google. There are many different implementations, and I won’t even try to address them all. It suffices to say that MapReduce is all about giving programmers an efficient way to consume data without needing to know how or where it is actually stored. Further, MapReduce excels at traversing datasets that live on more than one machine. It is simple, a bit restrictive, but extremely powerful. I’m going to show you how it works in CouchDB (well, actually in BigCouch hosted on Cloudant.com).

Broadly speaking, CouchDB (and Cloudant’s BigCouch) use MapReduce as a tool to let you introspect your data and build persistent ‘views’ (indices) for fast query responses. And, oh yeah, it does it incrementally. Let’s examine the pieces in detail:

View builds. The process of scanning over your data to create indices that make queries fast. This step also often performs data normalization.
MapReduce. The framework by which user code gets executed for index builds.
View queries. How you get the information out of your indices
Incremental. When new documents are added or existing documents are modified/removed, you don’t need to rescan your entire dataset. Instead the view engine only rescans the new/modified documents, and deletes the contributions from the deleted/modified documents. It’s a complex implementation, but one that really separates the Couch MapReduce model from, e.g., Hadoop’s or Riak’s implementations.

For full details on Cloudant’s implementation of MapReduce, take a look at Dynamo and CouchDB Clusters and Adam’s O’Reilly Webcast. So what do you actually have to do? Well, you simply need to write one or two functions. The ‘map’ function is required, and the ‘reduce’ is optional. For simple sorting problems (e.g. building a SEO reverse index) map alone suffices. For data aggregation, reduce comes into the picture. CouchDB allows you to write these functions in any language you choose, although JavaScript is the most common. Let’s put some meat on the bones with examples. Below I’m going to show you how to do three of the most common things people do with CouchDB’s MapReduce:

sort documents based on arbitrary information
calculate aggregate statistics
perform a time-series analysis

Our Example Problem

We are going to load and analyze FAA plane crash data from data.gov using Cloudant. If you want to follow along at home, you have two choices.

Use CouchDB replication to copy my db into yours via:

$ curl 'http://<usr>:<pwd>@<usr>.cloudant.com/_replicate' -Hcontent-type:application/json -d '{"source":"http://mlmiller.cloudant.com/planes","target":"http://<usr>:<pwd>@<usr>.cloudant.com/planes","create_target":true}'

Repeat from scratch by getting the code from Github.

(1) is simpler and doesn’t require any special software installed on your machine. All you need is to have a free account on Cloudant. (2) is a bit more effort but should work with any CouchDB server. If you go with (1), the best way to check the progress is to login to your account, click on your user dashboard (top right) and then click on the newly created ‘planes’ database. If you examine the Stats tab and refresh periodically, you will see the db size increasing, as well as some MapReduce ‘views’ running as the data imports:

That bar that registers 31.7% is telling you that MapReduce is being used to index your documents in real time without you having to write any code at all. The ‘2839 of 8952’ on the right represents the fraction of freshly added/updated documents that is processed in this current pass, and if you click on ‘show details’ you can see the results of the MapReduce processes on each of the cluster nodes that contain any chunk of your data. That’s just how easy it is to get your code running, in parallel, in realtime!

Data Import

I’ve chosen to import the FAA crash data using my favorite tools: Python and Benoit’s wonderful Couchdbkit library. If you use the Python Boto module for AWS, you will feel right at home. In upload.py, I use the very handy DictReader class from Python’s csv module to parse the AviationData.txt.gz. DictReader slurps a line of a CSV (or similarly structured text file) with column headings and gives you back a dictionary. It’s a one-liner to serialize that Python dictionary and push it into Cloudant. If you want to try this yourself the README file has instructions on installing the Couchdbkit module and executing the upload.py script, but it just boils down to:

python upload.py AviationData.txt.gz 'http://<username>:<password>@<username>.cloudant.com' planes

My inported data lives at http://mlmiller.cloudant.com/planes. One thing to note: you can see that I don’t take any steps to discover the type of the data on import. Instead, we just throw it all into Cloudant and we can sort it out later using MapReduce (_flexible_). Once the data is fully imported, I can login into cloudant.com and view the ‘planes’ database in the dashboard:

I’ve also gone into the Permissions tab and granted read access to the data:

Each document contains quite a bit of data, too much to fit onto a single screen shot, in fact:

But the important thing to note is that we have data for nearly 70,000 domestic accidents, spanning the range 1988–2010. If you want to look at a single document from your command line, you can via:

$ curl -X GET 'http://<usr>:<pwd>@<usr>.cloudant.com/planes/_all_docs?limit=1&include_docs=true'

and if you want it pretty printed, my favorite is:

$ curl -X GET 'http://<usr>:<pwd>@<usr>.cloudant.com/planes/_all_docs?limit=1&include_docs=true' | python -m json.tool

Since I’ve granted read permissions on mlmiller/planes, you can execute this yourself via:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_all_docs?limit=1&include_docs=true' | python -m json.tool
{
    "offset": 0,
    "rows": [
        {
            "doc": {
                "": "",
                "Accident Number": "LAX08CA100",
                "Air Carrier": "",
                "Aircraft Category": "",
                "Aircraft Damage": "Substantial",
                "Airport Code": "U42",
                "Airport Name": "Salt Lake City Muni 2",
                "Amateur Built": "Yes",
                "Broad Phase of Flight": "",
                "Country": "United States",
                "Engine Type": "Reciprocating",
                "Event Date": "04/12/2008",
                "Event Id": "20080513X00662",
                "FAR Description": "",
                "Injury Severity": "Non-Fatal",
                "Investigation Type": "Accident",
                "Latitude": "40.619445",
                "Location": "Salt Lake City, UT",
                "Longitude": "-111.992777",
                "Make": "Questair, Inc.",
                "Model": "Venture",
                "Number of Engines": "1",
                "Publication Date": "05/28/2008",
                "Purpose of Flight": "Personal",
                "Registration Number": "N36V",
                "Report Status": "Probable Cause",
                "Schedule": "",
                "Total Fatal Injuries": "",
                "Total Minor Injuries": "2",
                "Total Serious Injuries": "",
                "Total Uninjured": "",
                "Weather Condition": "VMC",
                "_id": "0fedbabcd21f15785750457f83000947",
                "_rev": "1-3e9637bb5214c82fc7b29bb4a75e765b"
            },
            "id": "0fedbabcd21f15785750457f83000947",
            "key": "0fedbabcd21f15785750457f83000947",
            "value": {
                "rev": "1-3e9637bb5214c82fc7b29bb4a75e765b"
            }
        }
    ],
    "total_rows": 68388
}

Example 1: Sort

Now that we have the data, let’s get down to business. Key/Value is great, but I want to be able get find all the documents for a given brand of plane, say Cessna. This is really an inverted index, and is in the absolute sweet spot for MapReduce. We can accomplish this with a ‘map-only’ view that looks like:

function(doc) {
    emit(doc.Make, null);
}

The MapReduce code is actually stored in the database as part of a view in a design document (see the CouchDB Book for more details). My favorite method is to use a client-side tool such as couchapp or couchdbkit that allows me to organize map.js and reduce.js files according to a filesystem hierarchy and then automatically upload them to the database as a design document. I actually snuck that two-liner into upload.py, where I have:

loader = FileSystemDocsLoader('_design/')
loader.sync(db, verbose=True)

Those two lines unpack the contents of the _design directory and build a design document. That directory looks like:

This gives me a design document with the name example; that contains two views, one called date and one called make. You can see the entire design document from the command line via:

$ curl -X GET http://mlmiller.cloudant.com/planes/_design/example

or you can just browse the ‘_design’ code in the file _design/example/views/make/map.js. The code looks like this:

function(doc) {
    emit(doc.Make, 1);
}

The emit() function generates a key/value pair that gets sorted according to CouchDB’s deterministic collation order. These key/value pairs are either stored directly in the view index, or passed through the reducer first (see below). Don’t worry about the second argument to the emit method, that’s for the the reduce phase and we don’t need that for a basic sort. The moment that we upload that design document to Cloudant, the index is kept up to date automatically. That’s quite different than the default CouchDB behavior, but it’s a nice optimization to make sure that the query response is snappy.

Query (finally!). Let’s find all of the documents (‘rows’) that have doc.Make=="Cessna". Since we emitted doc.Make in our map code, doc.Make is therefore the ‘key’ for this index, and that means we get snappy responses to queries like:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/make?reduce=false&key="Cessna"&limit=10'
{"total_rows":68387,"offset":13905,"rows":[
{"id":"0fedbabcd21f15785750457f8300324c","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f83003a2f","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f830060ec","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f83006bfb","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f8300a5ae","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f8300e15d","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f830162b7","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f8301a3a4","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f8301a54e","key":"Cessna","value":1},
{"id":"0fedbabcd21f15785750457f83058740","key":"Cessna","value":1}
]}

The full list of query options is given here, but really there are only a handful of options I typically use. For this query, if you remove the limit argument you will get the entire list of crashes with a Cessna involved. If you want to do some fun statistics, you can try:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/make?reduce=false&key="Airbus"' | wc -l 15
$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/make?reduce=false&key="Boeing"' | wc -l 773

Well that’s interesting! But wait, isn’t there a more efficient way to calculate statistics than slurping the entire set of Boeing lines to the client and doing a client-side count? You bet, and that’s where reduce comes in.

Example 2: Data Aggregation

Now we want to go beyond simple sorts and aggregate to find the total number of crashes for different types of planes. Reduce is perfect for this, and that’s why we did emit(doc.Make,1) in map.js. If you look at the corresponding reduce.js file for that view you’ll see nothing more than: _sum.

What the heck is that? Well there are certain aggregation tasks that are simply so common (count, sum, average, min, max, etc.) that we have coded them up in erlang and pushed them into the database. Cloudant has an ever growing number of built-in reduces (_builtins_), and using a builtin is always faster and less error prone than writing it yourself. As an avid user, I can admit that I haven’t used a non-builtin reduce in at least 9 months. So the _sum that we use here is simply going to add up the numbers that we emitted as the 2nd argument to emit() in the map.js function. Now we can answer the Boeing vs. Airbus question without any client side work via:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/make?key="Boeing"'
{"rows":[
{"key":null,"value":771}
]}
$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/make?key="Airbus"'
{"rows":[
{"key":null,"value":13}
]}

Check that out! Hmm, now suppose I want to find the total number of accidents, I can then just do:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/make'
{"rows":[
{"key":null,"value":68387}
]}

Very cool, and if I want to make a graph of how they plot out vs manufacture I can add the group=true option:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/make?group=true'
{"rows":[
{"key":"","value":53},
{"key":"107.5 Flying Corporation","value":1},
{"key":"1200","value":1},
{"key":"1977 COLFER-CHAN","value":1},
{"key":"1ST FTR GP","value":1},
{"key":"2000 McCoy","value":1},
{"key":"2001 MCGIRL","value":1},
{"key":"2003 Nash","value":1},
{"key":"2007 Savage Air LLC","value":1},
{"key":"67 FLYING DUTCHMAN","value":1},
{"key":"85 MANISTA","value":1},
{"key":"A Pair of Jacks","value":1},
{"key":"A. H. GETTINGS","value":1},
{"key":"A. LE FRANCOIS","value":1},
…

(truncated because it’s a looong list!)

Ok, so aggregation seems pretty straightforward. Now let’s go the last step to time-series.

Example 3: Time Series Analysis

Let’s get a bit more numerical and ask, what’s the distribution of crashes and fatalities vs time. That’s what the date view is for. If you inspect any of our crash documents you will see that they have a two other important fields: Event Date and Total Fatal Injuries. Let’s create an index that allows us to answer time-based queries about both the number of accidents and the number of fatalities. While a bit morbid, it’s good to know what the odds are! This example is a bit more complex because we are going to use an array for both arguments of emit(). Take a look at the code in the map.js:

function(doc) {
    var then = new Date(Date.parse(doc['Event Date']));
    var fatalities = 0;
    if (doc['Total Fatal Injuries']!="") {
        fatalities = parseInt(doc['Total Fatal Injuries']);
    }
    emit([then.getFullYear(), then.getMonth()], [1, fatalities]);
}

This is a bit more complex because we do some processing of the data. In line #2 we unpack the Event Date variable into a JavaScript Date object, very handy. I find myself using new Date(Date.parse(some_weird_timestamp)) all the time. Line #6 simply casts our fatalities info into an integer. The magic is in emit, where we use JavaScript arrays [] for both the key and the value. Then, for reduce we use good ol’ _sum. One of Cloudant’s additional builtins is a _sum that works on arrays, very handy. Now to query this index we get rows that are sorted by year, then month. The index looks like:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/date?group_level=2' | more
{"rows":[
{"key":[1982,0],"value":[207,185]},
{"key":[1982,1],"value":[232,96]},
{"key":[1982,2],"value":[280,95]},
{"key":[1982,3],"value":[303,113]},
{"key":[1982,4],"value":[387,133]},
{"key":[1982,5],"value":[349,106]},
{"key":[1982,6],"value":[433,283]},
{"key":[1982,7],"value":[399,96]},
{"key":[1982,8],"value":[332,119]},
{"key":[1982,9],"value":[239,129]},
{"key":[1982,10],"value":[224,134]},
{"key":[1982,11],"value":[208,96]},
{"key":[1983,0],"value":[199,102]},
{"key":[1983,1],"value":[210,72]},
{"key":[1983,2],"value":[264,91]},
…

So, e.g., 1982 month 0 (January) had 207 crashes and 185 fatalities. What was that group_level command? Let’s try changing it:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/date?group_level=1' | more
{"rows":[
{"key":[1982],"value":[3593,1585]},
{"key":[1983],"value":[3556,1273]},
{"key":[1984],"value":[3457,1229]},
{"key":[1985],"value":[3096,1648]},
{"key":[1986],"value":[2880,1180]},
{"key":[1987],"value":[2828,1237]},
…

Well that’s interesting! group_level=1 vs 2 has a big effect: with group_level=2 we get the full granularity/dimensionality of the array we used for the key (first argument to emit). But we can, at query time, reduce even further by integrating over month! Very cool, and the gory details are few: group_level runs between 0 and length(key), so in this case valid values are 0, 1, 2. If we specify group_level=0, we’ll get sum statistics over the entire date range:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/date?group_level=0'
{"rows":[
{"key":null,"value":[68387,38652]}
]}

Awesome. And therein lies the power of using MapReduce for an efficient index build. With Cloudant you don’t need to install any software, you simply use the cloud hosted cluster to run your code. If you think about it, all we really had to do was write 9 lines of javascript for the map method, and no original code for the reduce method. We can use this same exact index to query for a given date range, say 1994:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/date?group_level=2&startkey=\[1994\]&endkey=\[1995\]'
{"rows":[
{"key":[1994,0],"value":[116,60]},
{"key":[1994,1],"value":[142,42]},
{"key":[1994,2],"value":[191,67]},
{"key":[1994,3],"value":[179,72]},
{"key":[1994,4],"value":[224,74]},
{"key":[1994,5],"value":[223,98]},
{"key":[1994,6],"value":[267,131]},
{"key":[1994,7],"value":[260,101]},
{"key":[1994,8],"value":[212,200]},
{"key":[1994,9],"value":[169,129]},
{"key":[1994,10],"value":[131,79]},
{"key":[1994,11],"value":[143,130]}
]}

or if we want to do can turn off the reduce in the query and get all of the crashes from a given year/month combo:

$ curl -X GET 'http://mlmiller.cloudant.com/planes/_design/example/_view/date?key=\[1994,4\]&reduce=false' | more
{"total_rows":68387,"offset":34959,"rows":[
{"id":"88601d815897032516f87f0c510c7c60","key":[1994,4],"value":[1,1]},
{"id":"88601d815897032516f87f0c510c87b7","key":[1994,4],"value":[1,0]},
{"id":"88601d815897032516f87f0c510c951a","key":[1994,4],"value":[1,2]},
{"id":"88601d815897032516f87f0c510c9b0a","key":[1994,4],"value":[1,0]},
{"id":"88601d815897032516f87f0c510c9bbb","key":[1994,4],"value":[1,0]},
{"id":"88601d815897032516f87f0c510c9c15","key":[1994,4],"value":[1,1]},
{"id":"88601d815897032516f87f0c510c9d6d","key":[1994,4],"value":[1,0]},
{"id":"88601d815897032516f87f0c510ca925","key":[1994,4],"value":[1,0]},
…

Wow, handy! And if you want to really blow your mind, add the query-string parameter ?include_docs=true to get the full documents themselves.

Summary

There’s a lot of power in MapReduce, and I hope I’ve given you enough to get your hands dirty running it an an extremely simple way using a hosted service. If you have more questions, there are lots of examples on the CouchDB wiki or on the (Cloudant support center). But the rule of thumb is:

sort: map only
aggregate: use a _sum reduce + group_level
time-series: use an array of [year, month, day, …] + group_level

In the next post, we’ll show you how to link Cloudant MapReduce views with builtin full-text/numerical search for killer data visualizations.

Developer Preview: Cloudant Search for CouchDB

David Hardtke — Tue, 11 Jan 2011 00:00:00 PST

CouchDB is a compelling choice for database-backed web apps and analytics applications. Cloudant’s hosted CouchDB service adds scaling, redundancy, and fault-tolerance to CouchDB. Today we are releasing a new feature for developers to try out: full text indexing and search.

There are several products that implement search for CouchDB as external services (couchdb-lucene and ElasticSearch). Like these, our full text search uses Lucene query syntax and is based on Lucene. For Cloudant search, however, the search application is directly integrated into the Cloudant CouchDB database. This has several distinct advantages over other Lucene based search products:

Real-time Search. Search results reflect state of database at moment it is searched. Little to no latency on document additions, changes, or deletions.
Indexing and Search using all valid JSON types. In Lucene, the search terns are required to be strings. This is especially problematic for date range and numerical range searches. With Cloudant search, the tokens can be any valid JSON type (string, number, object, array, boolean, null).
Fault-tolerance and redundancy. The search indices are stored on the same nodes as their parent documents.
Little to no setup and configuration. For basic search, the procedure is upload documents, enable search, search documents using REST interface.
Scalability. Cloudant Core is designed to be horizontally scalable document store. Cloudant Search is a horizontally scalable search product.

Under the hood, our search consists of two parts. The first step is the creation of a reverse index, or posting list, for your documents. This reverse index maps terms to the documents in which they appear. We have implemented document indexing as a MapReduce view using couchjava, our Java language view server for CouchDB. This allows for usage of custom application-specific Lucene analyzers written in Java. In fact, your existing Lucene analyzers can be imported directly into Cloudant. If you prefer, you can write your document indexer in Javascript as long as you write a MapReduce view using the correct format (in a later blog post, we’ll show how our search application can be used for ad-hoc queries of your CouchDB database).

The second part of Cloudant search is a RESTful API that takes your search query and performs the necessary boolean logic to return the most relevant documents.

What is Cloudant search good for? Here are some applications that are easy (and cheap) to build using Cloudant full text indexing and search:

Searchable Real Time Catalog. If you are starting the next Ebay, build it using Cloudant. We combine full text search and numerical range queries.
Document Discovery by the Hour. If you have a collection of documents that you wish to search through but don’t need to archive, you can upload them to Cloudant, enable search, run your queries, and delete the database.

Currently, the beta version of Cloudant search is available to all hosted customers with paid accounts (including users of the Cloudant Heroku Addon). The version of the search product we are rolling out today is designed for smaller databases (less that a few gigabytes). An enhanced version that uses distributed search for larger databases will be available soon. The feature is free, but the search index counts against your data storage quota and the search queries count against your query quotas.

A technical description of the search API can be found on the Search API page. Instructions for making your databases searchable can be found on the Search Indexing page. Please submit your questions and suggestions to our discussion forum. If your application requires advanced indexing or search capabilities, please contact us for a quote on a custom solution. Please contact us for further information.

Over the next weeks, we’ll be posting tutorials on various ways the search can be used for web applications and ad-hoc analytics. Please stay tuned.

Bob Newson and Tim Anglade Join Cloudant!

Alan Hoffman — Wed, 05 Jan 2011 00:00:00 PST

It’s a huge day here Cloudant HQ; we are very excited to announce the two newest additions to our team: Bob Newson and Tim Anglade.

Tim Anglade certainly needs no introduction, at least in the world of NOSQL. I first met Tim here in Boston back in early 2009 when he gave the keynote at the “NOSQL Live” conference we cosponsored with 10Gen. Since that time, Tim has been a tireless promoter, interlocutor, and sometimes skeptic of all things NOSQL. This past year, he organized the NOSQL Summer reading group that had participation from more than 35 cities in 20 countries around the world. Since then he has been globetrotting on his “NOSQL World Tour,” organizing and promoting events large and small, and working on the NoSQL Tapes, a series of conversations with domain experts, thought-leaders, trailblazers, and Cloudant founders. Tim will be moving from his home country of France to join us here at Cloudant HQ in Boston (in January! buy a thick parka Tim.) Tim will be taking on various challenges at Cloudant including developer advocacy, community development, customer outreach and support, technical writing, and directing our social media efforts. We’re basically piling onto Tim all the stuff that we have never been able to do well. This means that the blog you’re reading is about to become much more exciting and useful. Please join us in welcoming Tim by directing all your questions and complaints to tim@cloudant.com. You should also follow him on Twitter: @timanglade.

Many of you already know Bob Newson from his prolific work with CouchDB. Bob has been a contributor on the project since 2008. Among his countless contributions to that project is the CouchDB-lucene extension, which provides full-text indexing capabilities. Bob is an expert in distributed systems, big data processing, and erlang. His official title at Cloudant will be ‘Senior Database Engineer,’ but unofficially we’re already referring to him as the resident ‘super-hard-problem solver’. He’ll be putting his software expertise to good use building new features for Cloudant and BigCouch, and streamlining our systems to make them faster, stronger, and more reliable. For now, Bob will continue to live and work from Windsor in his native land of England (God Save The Queen!) which is nice, because he’ll be able to provide early morning support while those of us in The States are still sleeping soundly. In his free time, Bob enjoys reading about Paxos and drinks with sparklers in them. Welcome to the team Bob! We couldn’t be more excited to have you on board.

P.S. – The pop-culture nerd in me would be remiss if I didn’t point out that both Bob Dionne and Bob Newson are now working with Cloudant. This means that I now have the chance to make endless Office Space allusions (“I have a meeting with The Bobs,” “Hey Bob! Bob!” etc.) which brings me no end of joy. It also gives me an excuse to put this picture up and have a good laugh.

“What would you say, you do here?”

Scaling CouchDB with BigCouch

Alan Hoffman — Mon, 25 Oct 2010 00:00:00 PDT

On Friday Adam gave an free webcast on BigCouch. I, in my completely biased opinion, think that he killed it. The discussion in the chat room and the questions were fantastic. I hope the community got a lot of this. I’m posting the slides above. The good people at O’Reilly have told us that they will put up a full video of the talk (thanks Kathryn!) When that goes up, I encourage everyone to see it, you’ll definitely get more out of that than just reading the slides. Thanks to all those who attended and made it such a great presentation. Hopefully we’ll get to do another one soon.

P.S. – We’d love your feedback on BigCouch and Cloudant. Help us out by picking one of the choices on our survey page cloudant.com/feedback. You could win a free upgrade to an Argon or Helium plan for a year.

Welcome Bob Dionne to Team Cloudant!

Alan Hoffman — Thu, 07 Oct 2010 00:00:00 PDT

The Cloudant team just got a little bit bigger. We’re super excited to welcome Bob Dionne (@bitdiddle) as our new Senior Software Engineer. Bob has been developing software for more than 25 years, and spent a significant chunk of that time at IBM. Since leaving he has built his own software company and, more recently, founded the consulting firm Dionne Associates. Needless to say, the man has serious chops when it comes to writing code, which is good because we’re going to be throwing a lot of hard problems at him. Bob is a true New Englander (with a wicked awesome accent to boot) and though he’ll be mostly working remotely, we’re looking forward to him joining us at the Boston office a couple times a month.

We (or I should say, I) first met Bob at the NoSQL Live Boston event. I was aware of his stellar work with CouchDB, where he focused some of his efforts on the etap testing framework. I was struck by his quiet thoughtfulness, and his readily-apparent passion for the project and the community. I feel privileged to have him join our team. So far, he’s been digging into BigCouch learning the codebase and helping us build up our testing framework. Over the coming months I’m sure he’ll be building and testing features for BigCouch and Cloudant and working very closely with Adam (our CTO) to make sure our hosted offering is a solid as possible.

Please join me in welcoming Bob to the team!

Java Language MapReduce Views

David Hardtke — Wed, 08 Sep 2010 00:00:00 PDT

One of the core principles at Cloudant is push code to data. In the era of big data and open source software, it’s much easier to move your code around than to move the data. Code is not tied to machine by software licenses, and the code is generally smaller than the data one wishes to analyze. When I first heard the concept push code to data, the retained marketing synapse in my brain immediate fired with write once run anywhere, the Sun corporation marketing slogan for Java.

The Cloudant Hosted CouchDB service uses MapReduce views to analyze data. Code is contained in a design document. The design document is passed to a view server along with each document in the corresponding database. CouchDB view servers have been written for many interpreted languages (see the CouchDB wiki for a list), but until now there has not been a Java Language View Server for CouchDB.

Today we are releasing the Java Language MapReduce View Interface for Cloudant’s Hosted CouchDB service. This interface defines the protocol for writing MapReduce views in Java that can be run on our hosted CouchDB platform. The Java language has several features that make it extremely useful for large scale analytics within CouchDB:

Compile-Time Type Checking. CouchDB views are generally written in weakly typed interpreted languages (javascript, python). Anyone who has worked extensively with CouchDB, however, knows that debugging complicated views can be very challenging. Sometimes it is easier to let the compiler find the syntax errors.
Libraries, Libraries, and More Libraries. Java is the most widely used computer language at the moment. As such, there is a library for almost everything. CouchJava allows for the re-use of existing Java langauge libraries within the CouchDB framework. Libraries can be packaged in jar files and uploaded directly to the view server.

When might you use this? Let’s assume you want to use a neural network to analyze financial options data in real time. You can directly import your favorite neural network library(there are many written for Java)directly into CouchDB along with the weights that you have calculated during training (the weights will be configuration parameters in the design document). Each new piece of data will be automatically analyzed in real-time. You can update the weights by modifying the design document. In fact, one could build the unsupervised learning cycle directly into CouchDB.

The Java view server works differently than a standard CouchDB view server. The design document does not contain code. Instead, the design document specifies which class should be called for the Map and Reduce steps. The code (a jar) is attached to the design document in the form of a binary attachment. This jar contains both user defined classes and external libraries that are needed. This paradigm (libraries as binary attachments) is a non-standard extension of the CouchDB view server API.

The interface and instructions for writing Java Language MapReduce views are now available on Github. Let us know what you think.

Cloudant’s BigCouch is open-source

Brad Anderson — Mon, 30 Aug 2010 00:00:00 PDT

The Cloudant team is pleased to make available its ‘BigCouch’ software project as open source software under the Apache 2.0 license. We have been the beneficiary of countless open source projects while constructing this system, so it is only fitting that we share our efforts in hopes that people will not only find utility, but also assist us in making it better.

It has taken us a while to get to this point. The version we are opening has benefitted from Cloudant operating our systems in production for our customers for almost a year. The recent (and heavy) refactoring effort has encompassed our learning over this time, and we believe it is now time to share this system with the world. We will now be focusing our efforts on documenting and ease of use, as is the case with most newly opened projects.

What does it do? Think of BigCouch as a set of Erlang/OTP applications that allow you to create a cluster of CouchDBs that is distributed across many nodes/servers. Instead of one big honking CouchDB, the result is an elastic data store which is fully CouchDB API-compliant. The top is a picture of the standalone CouchDB setup that is common today. When you download BigCouch and build with make dev (see the README.md file), the result is the bottom picture, a three-node development cluster.

The clustering layer is most closely modeled after Amazon's Dynamo, with consistent hashing, replication, and quorum for read/write operations. CouchDB view indexing occurs in parallel on each partition, and can achieve impressive speedups as compared to standalone serial indexing.

Also contained within BigCouch is a tailor-made RPC server application called Rexi. Rexi better fits the needs of this distributed data store by dropping some unneeded overhead in rex, the RPC server that ships with Erlang/OTP. Rexi is optimized for the case when you need to spawn a bunch of remote processes. Cast messages are sent from the origin to the remote rexi server, and local processes are spawned from there, which is vastly more efficient than spawning remote processes from the origin.

So, let us know what you think. We will be hanging around Freenode IRC in the #cloudant channel. Pull the code, build it, install, and begin playing. Document, enhance, give back, benchmark, blog, tweet, …and enjoy.

Dynamo and CouchDB Clusters

Brad Anderson — Fri, 13 Aug 2010 00:00:00 PDT

For a while, CouchDB was described as a “distributed, fault-tolerant and schema-free document-oriented database accessible via a RESTful HTTP/JSON API.” The story about CouchDB’s ‘distributed’ description has always involved its multi-master replication. In this sense, it is not truly a horizontally scalable database, as noted here. With the availability of Cloudant’s new hosted service, a new option has entered the scene. Our clustering is similar to Voldemort, Cassandra, or Riak](http://github.com/basho/riak), as it implements a version of Amazon’s Dynamo. We’d like to tell you a bit about how it works.

Clustering in a Ring

Cloudant makes use of partitions for sharding data. Picture a ring of partitions that each are responsible for a range of the data. The partitions, sometimes referred to as virtual nodes or vnodes, are placed on separate physical nodes as the primary means of distribution. Partitioning databases is nothing new, and with Dynamo systems, a keyspace is usually employed. We use CRC32, so from 0 to 2^32. We’ve seen MD5, fnv, and SHA-1 used, and previous versions of our software used SHA-1. When creating a database, users may provide Q, a constant that determines how many partitions the database will be spread across. Let’s check out an example:

$ curl -X PUT http://username.cloudant.com/dbname?q=10

For a Q of 10, there will be 10 partitions put across all the nodes in the cluster. Each partition will be responsible for a range of 2^32/Q keys.

Any Node can Handle a Request

Each request to the cluster will most likely go through a round-robin load balancer, because each of the nodes knows how to find the data in O(1) lookup time. To do this, the document id is hashed, and the resulting hash is compared against the database’s partition map, as we discussed above. If you’re referring to the shiny picture on this page, you will see that our example hashes ‘doc_id’ to the E partition. In reality, this could be something like hashing a value to c34b and that falls between c000 and cfff hex values, so we call that range the E partition. The partition map for that database knows which node or nodes have copies of the document, and requests are routed via distributed Erlang messages to those nodes.

Quorum system (N, R, W)

The durability and consistency settings of Dynamo systems are governed by quorum constants. As you can see from the diagram, N=3. This means that three replica copies of ‘doc_id’ document will be stored in the cluster. When the ‘write’ arrives at Node 1 and the consistent hashing algorithm tells us the doc will live on the E partition, the system attempts three asynchronous writes, one to each of the nodes that house the E partition replicas. You can be reasonably certain of the writes succeeding, because the request will wait for W (in this case, W=2) responses. When two successful responses return to Node 1, the response of 201 Created is returned to the client. The same thing occurs on ‘reads.’ When a GET request arrives at a node, N ‘read’ requests go out to the nodes which house that document’s partition. If R=2, then the system waits for two agreed-upon values to be returned, and then replies to the client with a 200 OK and the document’s contents.

You can set the defaults of N, R, and W for your cluster, but you may also override them at request time. N may be overridden at database creation time, just like Q above. R and W may be overridden for each request individually. A higher W value means more durable writes, where lower W value means more write throughput. High R values mean consistent values returned, while low R values mean first read wins, and higher read throughput.

Masterless (no SPOF)

Server failure is handled transparently. The load balancer will not route requests to a downed node. With consistent hashing, the other nodes are capable of serving requests. Also, when a node is down, other partitions will provide the data that is located on the downed node. If N=3, W=2, R=2 as in our example, a node may be down but read and write quorums are still satisfied. When the node returns, operations return to normal. The node is placed back into service by the load balancer, and a ‘hinted-handoff-like’ system is used to replicate the writes that the downed node may have missed. The other nodes holding data for the partitions of the downed node replicate the updates continuously after the node returns.

Horizontally Scalable

When the cluster gets closer to capacity, or could use more nodes to crunch data in a more parallel fashion, extra nodes may be added to the cluster. When this happens, partitions are moved from existing nodes to the new nodes. The system has the flexibility to move partitions on a database level, and can split large partitions if required. This is an evolving portion of the code base, and we are attempting to do partition merging and splitting all while the database remains online, i.e. other partitions serve data while the work is being done.

Above, there was mention of partition merging. This may be required as the cluster shrinks. Or, it is also possible to re-partition an over-partitioned database into fewer shards. The flexibility exists to shrink a cluster by removing nodes, just as it may be grown, providing a truly elastic data store.

Transparent to the application

All of these features – distributed, horizontally scalable, durable, consistent – happen with little or no change required in applications that have been written for CouchDB. A cluster looks just like a stand-alone CouchDB, and API compliance has been our goal from the beginning. Granted, there are a few extra options like overriding quorum constant defaults and there are a few vagaries, like views always performing rereduce due to the views being distributed. But on the whole, the extras in Cloudant are transparent to the application.

Feel free to comment away, email us, or join us on Freenode IRC in the #cloudant room. We’d love to hear from you.

Using Cloudant’s Heroku Add-on

Alan Hoffman — Thu, 05 Aug 2010 00:00:00 PDT

Heroku has recently jumped with both feet into the NoSQL world, providing add-ons for many different NoSQL data stores to complement it’s easy ruby platform. We’re very excited to be able to offer CouchDB 1.0 to Heroku users via the Cloudant add-on. Cloudant is a distributed CouchDB service, with the added benifits of data redundancy, clustering, and automatic maintenance (view builds and compaciton). You can use it just like you would standalone CouchDB but you have the advantage of your data being distributed over multiple machines. The following is a (very short) guide to help you get started with Cloudant and Heroku. It is by no means comprehensive. If you have quesitons, feel free to get in touch. You can read more about the add-on at the Heroku Blog.

Local setup

Since CouchDB and Cloudant have the same REST API you can use them the same way. You just need a http client to interface with the database and a JSON parser.

In the simplest case you can just use the rest-client and json gems:

$ sudo gem install rest-client json

If you don’t already have CouchDB on your local machine:

OSX:

$ brew install couchdb

Ubuntu:

$ sudo apt-get install couchdb

Using from Sinatra

Installing the Cloudant add-on gives you an environmental variable CLOUDANT_URL. This contains your API URL and credentials and is how you access your database(s) at Cloudant:

$ heroku config --long CLOUDANT_URL
https://USER:PASSWORD@APP.heroku.cloudant.com

When you first install the add-on you need to create a database:

Using cURL:

$ curl -X PUT https://USER:PASSWORD@APP.heroku.cloudant.com/DATABASENAME

Using rest-client:

$ RestClient.put("https://USER:PASSWORD@APP.heroku.cloudant.com/DATABASENAME", "")

You could also do this in your code by doing something like:

begin
  RestClient.put("#{ENV['CLOUDANT_URL']}/SOMEDATABASE", "")
rescue
  puts "database already created"
end

The following code will allow you to GET a document from the music database and then print it to the page:

DB = "#{ENV['CLOUDANT_URL']}/music"
get "/doc/:doc" do
  doc = RestClient.get("#{DB}/#{params[:doc]}")
  @result = JSON.parse(doc)
  haml :doc_id
end

The view would look something like:

%h1 A Doc from CouchDB!
- @result.each do |k,v|
  %b=k
  %em=v
  %br

To request this page by pointing your browser to:

http://SOMEAPP.heroku.com/doc/YOURDOCUMENT

Deploying to Heroku

To use Cloudant on Heroku, install the Cloudant add-on:

$ heroku addons:add cloudant:basic

Cloudant’s Own to Speak at SurgeCon

Joe Williams — Tue, 27 Jul 2010 00:00:00 PDT

As Cloudant’s ops guy and utility infielder, I (@williamsjoe) will be speaking at the Surge conference in September. Surge is a conference presented by OmitTI that focuses on infrastructure and web operations. As such I will be discussing how we have built Cloudant’s database platform on top of Amazon’s EC2. The conference will include keynotes from the always excellent John Allspaw (of Flickr and Etsy fame) and Theo Schlossnagle (OmniTI).

Cloudant and SETI: Crowdsourcing the search for E.T.

Alan Hoffman — Wed, 21 Jul 2010 00:00:00 PDT

Going to OSCON? SETI needs your help!

Cloudant is teaming up with the SETI Institute to help engage citizen-scientists in the search for intelligent life elswhere in the universe. With some help from Amazon, we’ve built a ‘croudsourced data analytics platform’ where ordinary citizens can search and analyze data from the Allen telescope array. Are you a citizen-scientist? Do you know about signal processing? Are you interested in helping in the search? Come find out how!

Thursday, July 22nd at OSCON, our own Dr. Dave Hardtke will be demonstrating SETICloud. He’ll be describing the Cloudant platform and how we’re bringing the wisdom of the crowds to SETI’s search.

Time: 12:20–1:20 pm

Place: Room E143/144

SETI is hacking on CouchDB! Come find out how you can help.

And don’t miss Dr. Jill Tarter’s keynote talk on “Open SEITQuest” Thursday morning at OSCON.

Learning to Relax

Alan Hoffman — Mon, 28 Jun 2010 00:00:00 PDT

Over the weekend I traveled to Chicago to give a talk on ‘CouchDB for Beginners’ at the WindyCityDB conference. You can read the slides above. The conference exceeded my expectations and many of the talks, both long and short, were stellar. Big thanks to Ray Hightower and all the people who helped put WindyCityDB together. I very much appreciate the opportunity to give this talk and I hope everyone enjoyed it.

Cloudant Nationwide Tour

Alan Hoffman — Wed, 16 Jun 2010 00:00:00 PDT

It turns out that next week (the week of June 21st) is a big week for tech conferences. A bunch of the Cloudant people will be hitting up various conferences and giving a couple talks. If anyone is interested in meeting up over beers to talk Cloudant, CouchDB, NoSQL, Startups, etc. get in touch (info at cloudant dot com is a good place to start.) Here are the tour dates:

Structure Conference. June 23–24. San Francisco, CA: Both Adam Kocoloski (CTO) and David Hardtke (Search) will be milling about this conference, whose tagline is “Put Cloud Computing To Work.” On the 23rd in the afternoon, Cloudant will be taking part in the Launchpad! event. Adam will be working the crowd, drumming up some love for Cloudant, giving a short pitch about our service. Check it out.
Velocity Conference. June 22–24. Santa Clara, CA: No Cloudant talk at this conference but Joe Williams (Operations and Utility Infeilding) will be in attendence. No one I know gets more pumped up about server uptime, configuration management, and monitoring than Joe. If you’re the same way, say hi.
Momentum Summit. June 23. Cambridge, MA: This one day conference focuses on how to turn a startup into big business. CEO Alan Hoffman (that’s me) will be in attendence. Looking forward to hearing from Boston startup gurus like Steve Kaufer and Scott Griffith. I’m always eager to talk with other local startup types so come find me.
Windy City DB. June 26. Chicago, IL: I’m lucky enough to be heading back to my hometown to attend and speak at this one day event on the south side of Chicago. My talk is entitiled “Learning to Relax: CouchDB for Beginners” and it is about, uh, CouchDB for beginners. It’s my first long-format NoSQL talk so if you’re in attendance, please no heckling. I’m happy to meet up after the conference; if you’re looking for me on Friday afternoon, I’ll be at U.S. Cellular Field watching the White Sox.

Just Opensourced: Gaff and Deckard

Joe Williams — Fri, 04 Jun 2010 00:00:00 PDT

Today we released two open source projects that have been in use internally at Cloudant for some time now, Gaff and Deckard.

All of our infrastructure is in the cloud and as such we need a way for disperate systems to all request resources, this is where Gaff comes in. Gaff is a pubsub daemon for asynchronously talking to cloud APIs using AMQP. Currently it supports a subset of the Dynect (DNS), Slicehost and EC2 APIs and uses @geemus’s awesome fog Ruby library. The basic workflow for Gaff is to send JSON-RPC formated messages to an AMQP exchange with a routing key corresponding to the API you are talking to, you could be sending these messages from a web application or another service. Each message gets routed to an API specific queue and is picked up by Gaff and turned into the appropriate API call, starting, stopping, modifying your servers on EC2 or elsewhere.

We have a lot of CouchDB instances to keep tabs on to do this we wrote Deckard. Deckard is a HTTP check monitoring system based on CouchDB. Yo dawg! What better than to monitor CouchDB with CouchDB (and some Ruby)? Deckard supports basic HTTP content checks, email alerts, SMS alerts (via email) for on-call rotations, basic maintenance scheduling, replication latency alerts (between two Couches) and even has EC2 Elastic IP support for failover between two EC2 instances. Best of all since it’s based on Couch you get an API for free, just PUT a doc in the HTTP checks database and you get a new HTTP check the next time Deckard runs.

Checkout these and my other projects on Github and follow @cloudant and myself on Twitter.

Ode to a Utility Infielder

Alan Hoffman — Wed, 17 Feb 2010 00:00:00 PST

Today marks the one year anniversary of Joe Williams (@williamsjoe) becoming a member of Team Cloudant. Those of us who are married will recall that one year is the paper anniversary so it’s fitting that Joe will be receiving paper (in the form of vested stock options) from Cloudant today. I thought I would take a moment on this occasion to thank Joe for his work over the past year.

We first hired Joe on a contract basis to help us build an internal infrastructure tool we call The Deployer. This is the tool that allows us to quickly provision and configure resources from a number of different cloud providers. We can, with the push of one button, ‘inflate’ an X-node Cloudant database cluster, and have it configured and available in a matter of minutes, all because of Joe’s ongoing efforts.

Since joining full-time, Joe has acted as our ‘utility infielder.’ All the problems in the operations and infrastructure realm fell to Joe. He built and mantains our monitoring system, our logging system, and our build system. That alarm system we have where we get a text message in the middle of the night when something goes wrong, Joe did that (thanks Joe!). Joe, margarita in hand, has pushed hot code updates to live systems from a Mexican karaoke bar.

Needless to say Joe has been and continues to be a staggering presence at Cloudant. We feel privileged to work with him on a daily basis. I only hope I’m able to write a similar post on his 5 year anniversary. Plus, the man has excellent mutton chops.

Thanks Joe! Here’s to the big things (and fewer alarms) coming in the next year.

NoSQL Live From Boston

Alan Hoffman — Wed, 27 Jan 2010 00:00:00 PST

We’re very excited to be joinin up with our friends at 10gen to sponsor the NoSQL Live conference right here in Boston, our backyard. The official press release can be found here.

The event will take place on March 11th at the John Hancock hotel. A number of us from Cloudant will be at the conference in both official (panel discussions, lightning talks) and unofficial (milling about, drinking) capacities. If you are in the New England area or plan on coming to the New England area, you should register and attend the conference. You can register here.

From the 10gen events site:

About:

A one-size-fits-all approach to databases no longer applies. Relational databases have worked well - and will continue to - for highly transactional systems. But today’s web applications require enormous scalability and performance. This has spurred the growth of a new class of databases that trade off some of the features of relational databases to offer high performance, ease of programming, high availability & the ability to scale in cloud environments. They are collectively called NoSQL or non-relational databases.

NoSQL Live from Boston is a full-day interactive conference that brings together the thinking in this space. Picking up where other NoSQL events have left off, NoSQL Live goes beyond understanding the basics of these databases to how they are used in production systems. It features panel discussions on use cases, lightning talks, networking session, and a NoSQL Lab where attendees can get a practical view of programming with NoSQL databases.

Interested in presenting or sponsorship opportunities? Contact meghan@10gen.com.

Welcome Dr. Dave Hardtke

Alan Hoffman — Wed, 06 Jan 2010 00:00:00 PST

The Cloudant team is growing!

We’re excited to welcome Dr. Dave Hardtke who has joined Cloudant as Director of Search. Like many of us here at Cloudant, Dave began his career as an experimental particle physicist. In over a decade of experience Dave illuminated the secrets of the universe at various experiments such as STAR (Brookhaven National Lab), ICECUBE (Antarctica), and NA44 (CERN). Dave’s seminal work lead to the discovery of the Quark Gluon Plasma, the 2005 physics story of the year. More recently Dave has turned his talents in algorithms and analyses to search, where he has built his own startup stinkyteddy.com, a general purpose search engine combining multiple keyword-driven search feeds and intelligent semantic analysis to deliver the most timely search results regardless of source and content types. Prior to that Dave served as Chief Scientist at Surf Canyon, a startup focusing on personalization of real-time search.

Dave is accelerating Cloudant Search. As we add customers it has become clear that they are hungry not just for a flexible way to store thier data, but a powerful way to “discover” their data. Search is a critical component in nearly all data driven applications. We are therefore expanding our efforts to integrate a robust search platform into our CouchDB-based, cloud database service. Dave brings his expertise in internet and enterpise search technology to the project.

Please join me in welcoming Dave to the team.

Cloudant.com Refresh: CouchDB in the Cloud

Stefano J. Attardi — Tue, 08 Dec 2009 00:00:00 PST

Our new front page just went live on Cloudant.com. The main goal of this redesign is to give more information about the service we are currently rolling out. While we are still in private beta, we are starting to accept more people onto our platform and therefore we are ready to tell you more about what we are doing. We are currently aiming at existing CouchDB users, and to them we are offering professional hosting of CouchDB in the cloud.

The new design follows the AIDA principle, according to which a succesful marketing flow involves four main phases: attention (A), interest (I), desire (D), and action (A).

The “CouchDB in the Cloud” headline is designed to attract the attention of CouchDB users. The screenshot behind it is meant to evoke interest by giving a peak to the features available in our dashboard. Underneath it, the three “features” listed aim to create desire. They link our offering to CouchDB, by stating that we provide all the benefits of the database that our potential customers already love, but also highlight what we have to offer on top of that: a scalable and managed infrastructure and the chance to use it for free. Finally, the two buttons, placed in what the Gutenberg Rule calls the “terminal area”, provide options for action.

Let us know what you think about this new design.

Changing the World, One Password at a Time

Stefano J. Attardi — Tue, 01 Dec 2009 00:00:00 PST

Since launching the beta we’ve received a bunch of feedback along the lines of “Uh… I can’t change my password, wtf?” Yes, wtf indeed. We knew we would be building this feature eventually but we didn’t want to hold up the release for it. Well, now we have it on our new Account Options page.

You can access it by selecting ‘account options’ from the user account pulldown on the top right of the page. Thanks for all the feedback and keep it coming.

API Keys: No Really, We’re Listening

Alan Hoffman — Thu, 05 Nov 2009 00:00:00 PST

Since Monday we’ve been sending out invites to the private beta (if you haven’t gotten yours yet, be patient, it’s coming). And we’ve gotten a lot of great feedback on how to make the service better. One suggestion that came from a few users was to allow database access to non-Cloudant users through API keys that can be shared with anyone. Well, your requests did not fall on deaf ears. For any of your databases you can now navigate to the permissions tab and create a database-specific API key and password. You can then set permissions for that key as if it were a Cloudant user. The screenshot below shows what that should look like. Thanks to everyone for the great suggestion. You keep the feature requests coming and we’ll get to them as fast as we can.

Benchmarking CouchDB with Baracus

Joe Williams — Wed, 04 Nov 2009 00:00:00 PST

Baracus is a new project built specifically for benchmarking CouchDB. Using trusty httperf, Baracus makes it easy to create and perform a battery of tests based on simple configuration files. In this post I’ll go into the details of a Baracus run and show you how to run your own benchmarks.

First, let’s install Baracus. It depends on httperf so install it first either from source (check the README for compile options for httperf) or from your distribution. Note that Baracus uses gemcutter for gem hosting.

gem sources -a [http://gemcutter.org](http://gemcutter.org)
gem install baracus

Next, let’s examine a simple Baracus config file.

httperf: "/usr/bin/httperf"
host: "localhost"
port: "5984"
user:
password:
db: "testdb"
report_url: "http://localhost:5984/bench"
name: "test123"
config:
  timeout: 30
  sessions: 100
  rate: 100
  doc_size: 100
  writes: 10
  reads: 10
  batchok: false
info:
  storage: "raid10, 4 disks, ebs"
  anything: "stuff"
  more: "foo"

This configuration file specifies where the httperf executable lives, the host, port, user, password, and database to use for the test. Baracus puts all of the results and wsesslog files back into CouchDB for later consumption, the report_url is where you define this database.

As I mentioned, Baracus uses httperf to perform benchmarks. Specifically it uses httperf’s wsesslog feature. wsesslog is a simple file format that describes what httperf should do each session. Here’s an example of an entry generated with Baracus that does an HTTP GET:

/testdb/a12bf8572f788824dfd5169e2eb496ab method=GET

Baracus creates two wsesslogs for each run, one for writes (POST) and one for reads (GET). Baracus generates these files based on a few items in your configuration. Specifically, the number of sessions, document size and how many reads or writes to do per session. Additionally in the configuration you can specify at what rate httperf should perform the operations within each wsesslog file. Baracus creates the wsesslog for writing with random data and creates the read file with random docs from alldocs. Since Baracus uses alldocs, it performs all the writes first. The above config will create 100 sessions with 10 writes and reads each at a rate of 100 per second. Each document will be roughly 100 bytes.

The info section at the end of the config contains any additional information you may want to have tag along with your results. You can put anything you want in there; I generally use this to describe the system I just ran the tests on.

Once you have a configuration you are happy with it’s very simple to run your tests.

$ baracus baracus.yml

This will print the standard httperf output as well as a tab delimited version. If you want to checkout the wsesslog files they will be written to the directory you ran Baracus in as well as to the results document in the reporting database. Here’s what the results look like in Couch.

Baracus is meant to make repeatable benchmarking easy. If you have any questions, bugs or would like to contribute check out Baracus on Github.

Cloudant at No:SQL (east)

Alan Hoffman — Mon, 31 Aug 2009 00:00:00 PDT

Cloudant’s own Brad Anderson is helping to organize No:SQL (east), a conference centered around the latest nonrelational database systems. The conference will be held at the Georgia Tech Research Institute in Atlanta on October 28–30. The description from the website reads:

NoSQL East, a conference of non-relational data stores, aims to present the experiences of different companies using so-called ‘NoSQL’ solutions in production.

The reasons for using these new systems are varied, but often involve scale that relational solutions cannot achieve. We have a new tool in our toolbelts, and as you will see, some people are using them quite effectively.

Join us in Atlanta as we discuss ‘Big Data’ and the systems that are completely transforming how people look at data.

The organizers have a great venue picked out (courtesy of Georgia Tech) and an exciting group of speakers. A number of nonrelational systems will be covered, and Cloudant co-founder Mike Miller will be giving a talk on CouchDB. Anyone interested in the NoSQL space should really try to make it down to Atlanta to attend.

Cloudant is one of two primary sponsors of the event, the other being GTRI. Many, if not all, of us will be attending the conference. We’re always happy to talk about NoSQL, CouchDB, JQuery, The Bears winning the Superbowl, etc. You should come say hi.

More information about No:SQL (east) can be found at nosqleast.com. Check it out.

Update (10/22/09): Since posting the above, a couple other great companies, including Basho and Rackspace Cloud, have also joined on as sponsors for the event.

Well, how did I get here? – Talking Heads

Alan Hoffman — Wed, 08 Jul 2009 00:00:00 PDT

Dharmesh tells me that I’m supposed to write a blog post explaining how we got to where we are. I’ll try my best, but sometimes I feel like the guy in that Talking Heads song, looking around saying, “This is not my beautiful house. This is not my beautiful wife.” This company formed sometime between 2004 and 2008, but definitely around one of the rather large lunch tables on the first floor of the Stata Center at MIT. Adam, Mike, and I all worked in the same group in the physics department and every day we ate lunch around one of those oddly shaped tables. By January of 2008 came to the conclusion that we couldn’t leave MIT without starting a company. We didn’t really know what we wanted to do, but we did know 1) We were all fairly bright and industrious guys, 2) we enjoyed working together, and 3) we didn’t want to go look for “real” jobs.

Mike’s original idea was for an automatic tamale maker. I’m not kidding. That one got nixed when we found out we wereway behindalready.

Eventually we decided to play to our strengths, which were in large-scale data processing and analysis. Our high-energy physics experiments generated some of the largest data sets in the world. It was our job to sift through the terabytes and petabytes to find rare particles and strange interactions. This is what we did for fun. Turns out that we’re living in a data-centric world (more so every day) and many of the problems we had faced were popping up in the business world. What we wanted was a way to bring physics-style data analysis tools to the masses. All we had to do was build it.

Toward the end of March someone sent me Paul Graham’s essay “A Student’s Guide to Startups.” I had never heard of PG before, but his essay resonated with me, especially the part about getting to stop treading water. I stumbled upon YCombinator literally by clicking random links on Graham’s website. As it turned out, this YCombinator Company was pretty well known and respected, and they were asking for applications for startups that needed funding. Hey! We were a startup that needed funding and we were already in Boston. So we applied, almost on a whim. And to be honest, we were a little bit floored when we got accepted.

After a number of back-and-forths with PG and other YC companies, we decided to focus on building a database designed for the specific needs of the web (hence our tagline “A database for the web,” clever huh?) One that would scale horizontally, could run on commodity hardware in a fault tolerant fashion, with a simple interface, and a powerful built-in analysis engine. We stumbled on Apache CouchDB, which at the time was in its relative infancy, and were very impressed with what they had built, especially the natural JSON-over-HTTP interface. We decided to take Couch and build it into a true ‘cloud-ready’ database.

It’s been more than a year since we started working on Cloudant. Since then a huge community has sprung up around CouchDB. Our own Adam Kocoloski was tapped to be an official committer on the project. We were able to raise significant additional funding late last year, just as the economy was collapsing – a true testament to the vision of our amazing VC. The Cloudant Team has grown from 3 to 6 as we were able to convince a few top-notch hackers to help us build our vision.

We’ll soon be releasing a beta version of Cloudant: CouchDB-in-the-cloud. It seems as though we were on to something when we began tossing ideas around in the Stata Center 18 months ago. The NoSQL movement, which is gaining steam, indicates the need for a new type of database, one built for the specific needs of web applications. At Cloudant we hope to fill that need. If you want to learn more or sign-up to be notified of our impending release, click here.