It is with great pleasure and pride we announce the fifth beta of our quasardb 2.0 release. Things are getting exciting as we are approaching the final release!
The impatient may want to head right away to the download page of our web site and try the software for free! This version isn't crippled in anyway, except that the beta will expire after one month of usage (by then a new beta will probably be available).
The curious might be interested in learning more about where we are coming from, in which case, the curious might want to read on.
Looking back
It's funny to think this project has been started in 2009 as an experiment over the course of a week-end. Back then the company was just really me freelancing for funds in Paris.
"What would happen if I applied the techniques I learned from kernel programming to a database?"
The preliminary performance results were really surprising. I spent another week-end to make sure nothing was wrong but basically I outperformed all the other existing caches and key value stores I could test.
I kept hacking on this project for a while. It grew. I poured in money from consulting into it. Hired world recognized experts to help me with some parts. I had the feeling I was on to something.
Looking back it was a crazy bet because on the paper my database had nothing special to offer, except better performance and a very scalable architecture. Nice.
Still, it was just a distributed key-value store with a very narrow set of features. You could only add, update, get and remove data. Compared to Redis, which was already packed with features, the scalable architecture of quasardb wasn't such a strong differentiator.
The network code was good. I made sure of that by getting the best help possible on that topic. And I have been obsessive about memory management and multi-core friendliness.
Most of all, it had to be "pure". I wanted it to be perfectly symmetric. For example it used the Chord algorithm for distribution and every single thing in the database was distributed. I'd refuse every feature that would break that symmetry.
The project keep growing, near-impossible to understand bugs were fixed one-after another. Then, one day one of my customers exhibited a problem that the "experiment" could solve.
They needed to distribute a large amount of market data to thousands of processors sitting on a gigabit network. They tried a couple of open source and commercial products but reliability and/or performance were not meeting the requirements.
"I've been writing this kind of database on the side, I think you should give it a try"
You would think that the network is the bottleneck in that kind of test, but actually very often the software breaks first. Don't believe me? Submit your favorite cache/datagrid/database to a massively parallel get test of significant size.
Because it had a strong focus on asynchronous I/O, multi-threading and memory management from the outset, quasardb passed the "let's make 2,000 computers download large chunks of data at the same time" test with flying colors.
And it did that with just two nodes.
A year of pre-production later we sold our first subscription.
Keeping it simple
Back then quasardb was more a datagrid than a database. Along the path the team grew, we raised money and had more customers.
We always kept it simple.
At heart, quasardb is a key value store and doesn't pretend to be something else.
Our philosophy is that the new data challenges require more than one tool and trying to do everything is the best way to deliver a poor product.
For example, quasardb isn't and doesn't want to be an analytics tool. But we have features that make it easy and convenient to plug it to your choice of analytics.
"Whatever business intelligence you want to do do, you want to open your incoming data pipeline as wide as possible, and for that, you need quasardb".
I will tell you another thing: relational databases are great. You should use them. Just don't use them everywhere.
Key-value stores are also great because they are simple to use and understand.
Nevertheless if you want your product to be successful, you need features, because customers want their problem solved. Just being able to add/get/remove entries is clearly not enough, even if the product is fast and scalable.
We also realized that key-value stores had a huge weakness: they are too simple.
Multi key transactions
If you are interested in NoSQL you must read every week an announcement about a database that has "transactions" and how about they're the only one to have that and how ACID they are.
quasardb has distributed transactions based on Multiversion Concurrency Control and two phase commit. How boring we're doing the same stuff that relational database do, I guess we must suck. Or maybe the gray beards that designed RDBMS knew what they were doing.
The twist is how we integrated that with the Chord algorithm and managed to pull of a truly distributed multi-key transaction database with a repeatable reads isolation level.
That was hard. Even though quasardb had from version one atomic updates (which is great but not enough to qualify the database as ACID) it was not easy to work around all the horrors that can happen in a distributed database when you apply the principle to a truly symmetric and distributed system.
But thanks to the abundant litterature on the topic and man-years of hard work, we pulled it out.
True multi-key transactions sounds like a nerdy feature, but that enabled us to create incredible value for our customers with our unique tagging system.
Unlimited tagging
"Cool story bro your thing about that distributed key-value store of yours, but my bro who is a specialist in cool stories tell me you're screwed when you don't have the key"
This problem is often solved by content indexing or by transforming the database into a document oriented database (or sometimes a graph oriented database). Or by doing exhaustive search (Map/Reduce for example) on the database. Or by trying to plug-in some SQL engine on top of a key-value store.
That works. But it's extremely inefficient and generally cumbersome.
And that would be trying to be what we are not. quasardb is a key-value store. If you want a document oriented database, there are popular tools like MongoDB. If you need SQL, there is a small startup in the valley that sells something and I heard the boss is a cool guy always open to deals that will help him buy a new boat (and if you don't like that, you can have a look at great open source alternatives like PostgreSQL).
Because quasardb is transactional, we managed to build on a truly tagging system that enables you to tag any entry and do immediate lookup based on tags.
Again, the symmetry, simplicity, scalability, and speed of the database had to be preserved. Tags were perfect because if you don't tag they cost nothing. And when you tag, you have total control over the cost in terms of storage and performance.
In a way, tags are manual secondary indexes. Indexes that are distributed and require little to no maintenance.
This was a game changer. Now using quasardb as a data lake was convenient, affordable and... still very fast!
The tagging approach also makes it very easy to integrate quasardb with other tools. That fits well our vision of "your data pipeline has several stages" and pleases customers that see they can easily integrate with their legacy software.
Where we are going
Enough with this old man blabbing! You want to know about the new version, right? So what does quasardb 2.0 beta 5 brings?
Performance and reliability improvements
We greatly increased the scale of our automated tests and found unsatisfactory behavior in low-memory conditions. Most of these issues (which were related to eviction) have been fixed.
We also had some weaknesses in some devious split-brain scenarii. Most of them have been fixed. There are still some problems with tagging and queues updates during partitions but they will be fixed in beta 6.
API improvements
A great deal of work actually has been done by the API. We improved, for example, the Java API packaging so that you would get everything you need in one convenient JAR (fat jar approach).
A similar work has been done on the Python API as we upgraded the package from easy install to setuptools. We're sure Python fans will appreciate.
The C/C++ headers have also been reorganized to be more convenient.
Queues have been renamed to deques as it reflects more clearly the features of the entry type. This is a breaking change but we think names are important and this change was actually helpful to signal the true nature of the type (you can have a look at the nodes js documentation if you are curious).
The roadmap
Beta 6 will be released later in October with some more performance and reliability improvements and of course with the brand new streaming API.
Streaming will enable you to conveniently load very large objects (hundreds of gigabytes large) into (and from) your cluster without worrying about all the little details.
We hand it to you, our users, to find new exciting things to build with quasardb!