I just fixed an interesting bug with RavenDB. Basically, a user was complaining that if they went ahead and (effectively randomly) deleted files from the RavenDB directory, RavenDB would break. The bug is that it used to work previously and now it doesn’t work anymore. The actual bug is a bit more complex than that, but I think that the description I just gave is a good indication of how I think of this bug.

Nevertheless, this actually demonstrate a problem with RavenDB. RavenDB stores index information in 3 different locations:

• Index statistics are tracked inside the storage (the RavenDB data file)
• Index definition are stored in the /IndexDefinitions directory, as plain JSON files.
• The index data itself is tracked inside the /Indexes directory

Granted, the only way that those locations can go out of sync is if the user is actively trying to corrupt RavenDB by messing with the internal stuff directly, but still…

I decided that I wanted to add explicit support for detecting and recovering from this scenario, so I wrote the following:

namespace Raven.Database.Plugins.Builtins
{
    public class DeleteRemovedIndexes : IStartupTask
    {
        public void Execute(DocumentDatabase database)
        {
            database.TransactionalStorage.Batch(actions =>
            {
                var indexNames = actions.Indexing.GetIndexesStats().Select(x => x.Name).ToList();
                foreach (var indexName in indexNames)
                {
                    if(database.IndexDefinitionStorage.Contains(indexName) )
                        continue;

                    // index is not found on disk, better kill for good
                    // Even though technically we are running into a situation that is considered to be corrupt data
                    // we can safely recover from it by removing the other parts of the index.
                    database.IndexStorage.DeleteIndex(indexName);
                    actions.Indexing.DeleteIndex(indexName);
                }
            });
        }
    }
}

IStartupTask is marked with InheritableExport, and the rest just plugs into RavenDB using MEF. Just by the action of creating this class, I have added new behavior to Raven. Moreover, this is now clearly executed in the proper place and in its own class, making it very easy to understand what is going on.

A lot of the stuff in RavenDB is wired in this fashion, and it makes adding things to it a snap.

Yesterday I gave a visual explanation about map/reduce, and the question came up about how to handle computing navigation instructions using map/reduce. That made it clear that while (I hope) what map/reduce is might be clearer, what it is for is not.

Map/reduce is a technique aimed to solve a very simple problem, you have a lot of data and you want to go through it in parallel, probably on multiple machines. The whole idea with the concept is that you can crunch through massive data sets in realistic time frame. In order for map/reduce to be useful, you need several things:

• The calculation that you need to run is one that can be composed. That is, you can run the calculation on a subset of the data, and merge it with the result of another subset.
• Most aggregation / statistical functions allow this, in one form or another.
• The final result is smaller than the initial data set.
• The calculation has no dependencies external input except the dataset being processed.
• Your dataset size is big enough that splitting it up for independent computations will not hurt overall performance.

Now, given those assumptions, you can create a map/reduce job, and submit it to a cluster of machines that would execute it. I am ignoring data locality and failover to make the explanation simple, although they do make for interesting wrinkles in the implementation.

Map/reduce is not applicable, however, in scenarios where the dataset alone is not sufficient to perform the operation. In the case of the navigation computation example, you can’t really handle this via map/reduce because you lack key data point (the starting and ending points). Trying to computing paths from all points to all other points is probably a losing proposition, unless you have a very small graph. The same applies if you have a 1:1 mapping between input and output. Oh, Map/Reduce will still work, but the resulting output is probably going to be too big to be really useful. It also means that you have a simple parallel problem, not a map/reduce sort of problem.

If you need fresh results, map/reduce isn’t applicable either, it is an inherently a batch operation, not an online one. Trying to invoke map/reduce operation for a user request is going to be very expensive, and not something that you really want to do.

Another set of problems that you can’t really apply map/reduce to are recursive problems. Fibonacci being the most well known among them. You cannot apply map/reduce to Fibonacci for the simple reason that you need the previous values before you can compute the current one. That means that you can’t break it apart to sub computations that can be run independently.

If you data size is small enough to fit on a single machine, it is probably going to be faster to process it as a single reduce(map(data)) operation, than go through the entire map/reduce process (which require synchronization). In the end, map/reduce is just a simple paradigm for gaining concurrency, as such it is subject to the benefits and limitations of all parallel programs. The most basic one of them is Amdahl's law.

Map/reduce is a very hot topic, but you need to realize what it is for. It isn’t some magic formula from Google to make things run faster, it is just Select and GroupBy, run over a distributed network.

Map/Reduce is a term commonly thrown about these days, in essence, it is just a way to take a big task and divide it into discrete tasks that can be done in parallel. A common use case for Map/Reduce is in document database, which is why I found myself thinking deeply about this.

Let us say that we have a set of documents with the following form:

{
  "type": "post",
  "name": "Raven's Map/Reduce functionality",
  "blog_id": 1342,
  "post_id": 29293921,
  "tags": ["raven", "nosql"],
  "post_content": "<p>...</p>",
  "comments": [
    { 
      "source_ip": '124.2.21.2',
      "author": "martin",
      "text": "..."
  }]
}

And we want to answer a question over more than a single document. That sort of operation requires us to use aggregation, and over large amount of data, that is best done using Map/Reduce, to split the work.

Map / Reduce is just a pair of functions, operating over a list of data. In C#, LInq actually gives us a great chance to do things in a way that make it very easy to understand and work with. Let us say that we want to be about to get a count of comments per blog. We can do that using the following Map / Reduce queries:

from post in docs.posts
select new {
  post.blog_id, 
  comments_length = comments.length 
  };

from agg in results
group agg by agg.key into g
select new { 
  agg.blog_id, 
  comments_length = g.Sum(x=>x.comments_length) 
  };

There are a couple of things to note here:

• The first query is the map query, it maps the input document into the final format.
• The second query is the reduce query, it operates over a set of results and produces an answer.
• Note that the reduce query must return its result in the same format that it received it, why will be explained shortly.
• The first value in the result is the key, which is what we are aggregating on (think the group by clause in SQL).

Let us see how this works, we start by applying the map query to the set of documents that we have, producing this output:

The next step is to start reducing the results, in real Map/Reduce algorithms, we partition the original input, and work toward the final result. In this case, imagine that the output of the first step was divided into groups of 3 (so 4 groups overall), and then the reduce query was applied to it, giving us:

You can see why it was called reduce, for every batch, we apply a sum by blog_id to get a new Total Comments value. We started with 11 rows, and we ended up with just 10. That is where it gets interesting, because we are still not done, we can still reduce the data further.

This is what we do in the third step, reducing the data further still. That is why the input & output format of the reduce query must match, we will feed the output of several the reduce queries as the input of a new one. You can also see that now we moved from having 10 rows to have just 7.

And the final step is:

And now we are done, we can't reduce the data any further because all the keys are unique.

There is another interesting property of Map / Reduce, let us say that I just added a comment to a post, that would obviously invalidate the results of the query, right?

Well, yes, but not all of them. Assuming that I added a comment to the post whose id is 10, what would I need to do to recalculate the right result?

• Map Doc #10 again
• Reduce Step 2, Batch #3 again
• Reduce Step 3, Batch #1 again
• Reduce Step 4

What is important is that I did not have to touch quite a bit of the data, making the recalculation effort far cheaper than it would be otherwise.

And that is (more or less) the notion of Map / Reduce.

I just finished drawing the following:

It makes me feel dirty inside, to do so. Mostly because I really don’t like or believe in building applications in this manner anymore. I would really like to be able to do this:

Unfortunately, I am talking about another subject in the context where I am showing the first architectural diagram, and I need to present only a single new concept at a time.