1. Working Through Data-Intensive Text Processing with MapReduce
2. Working Through Data-Intensive Text Processing with MapReduce – Local Aggregation Part II
3. Calculating A Co-Occurrence Matrix with Hadoop
4. MapReduce Algorithms – Order Inversion

This post covers the pattern of secondary sorting, found in chapter 3 of Data-Intensive Text Processing with MapReduce. While Hadoop automatically sorts data emitted by mappers before being sent to reducers, what can you do if you also want to sort by value? You use secondary sorting of course. With a slight manipulation to the format of the key object, secondary sorting gives us the ability to take the value into account during the sort phase. There are two possible approaches here. The first approach involves having the reducer buffer all of the values for a given key and do an in-reducer sort on the values. Since the reducer will be receiving all values for a given key, this approach could possibly cause the reducer to run out of memory. The second approach involves creating a composite key by adding a part of, or the entire value to the natural key to achieve your sorting objectives. The trade off between these two approaches is doing an explicit sort on values in the reducer would most likely be faster(at the risk of running out of memory) but implementing a “value to key” conversion approach, is offloading the sorting the MapReduce framework, which lies at the heart of what Hadoop/MapReduce is designed to do. For the purposes of this post, we will consider the “value to key” approach. We will need to write a custom partitioner to ensure all the data with same key (the natural key not including the composite key with the value) is sent to the same reducer and a custom Comparator so the data is grouped by the natural key once it arrives at the reducer.

Value to Key Conversion

Creating a composite key is straight forward. What we need to do is analyze what part(s) of the value we want to account for during the sort and add the appropriate part(s) to the natural key. Then we need to work on the compareTo method either in key class, or comparator class to make sure the composite key is accounted. We will be re-visiting the weather data set and include the temperature as part of the natural key (the natural key being the year and month concatenated together). The result will be a listing of the coldest day for a given month and year. This example was inspired from the secondary sorting example found in Hadoop, The Definitive Guide book. While there are probably better ways to achieve this objective, but it will be good enough to demonstrate how secondary sorting works.

Mapper Code

Our mapper code already concatenates the year and month together, but we will also include the temperature as part of the key. Since we have included the value in the key itself, the mapper will emit a NullWritable, where in other cases we would emit the temperature.

public class SecondarySortingTemperatureMapper extends Mapper<LongWritable, Text, TemperaturePair, NullWritable> {

    private TemperaturePair temperaturePair = new TemperaturePair();
    private NullWritable nullValue = NullWritable.get();
    private static final int MISSING = 9999;
@Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String line = value.toString();
        String yearMonth = line.substring(15, 21);

        int tempStartPosition = 87;

        if (line.charAt(tempStartPosition) == '+') {
            tempStartPosition += 1;
        }

        int temp = Integer.parseInt(line.substring(tempStartPosition, 92));

        if (temp != MISSING) {
            temperaturePair.setYearMonth(yearMonth);
            temperaturePair.setTemperature(temp);
            context.write(temperaturePair, nullValue);
        }
    }
}

Now we have added the temperature to the key, we set the stage for enabling secondary sorting. What’s left to do is write code taking temperature into account when necessary. Here we have two choices, write a Comparator or adjust the compareTo method on the TemperaturePair class (TemperaturePair implements WritableComparable). In most cases I would recommend writing a separate Comparator, but the TemperaturePair class was written specifically to demonstrate secondary sorting, so we will modify the TemperaturePair class compareTo method.

 @Override
    public int compareTo(TemperaturePair temperaturePair) {
        int compareValue = this.yearMonth.compareTo(temperaturePair.getYearMonth());
        if (compareValue == 0) {
            compareValue = temperature.compareTo(temperaturePair.getTemperature());
        }
        return compareValue;
    }

If we wanted to sort in descending order, we could simply multiply the result of the temperature comparison by a -1.
Now that we have completed the part necessary for sorting, we need to write a custom partitioner.

Partitoner Code

To ensure only the natural key is considered when determining which reducer to send the data to, we need to write a custom partitioner. The code is straight forward and only considers the yearMonth value of the TemperaturePair class when calculating the reducer the data will be sent to.

public class TemperaturePartitioner extends Partitioner<TemperaturePair, NullWritable>{
    @Override
    public int getPartition(TemperaturePair temperaturePair, NullWritable nullWritable, int numPartitions) {
        return temperaturePair.getYearMonth().hashCode() % numPartitions;
    }
}

While the custom partitioner guarantees that all of the data for the year and month arrive at the same reducer, we still need to account for the fact the reducer will group records by key.

Grouping Comparator

Once the data reaches a reducer, all data is grouped by key. Since we have a composite key, we need to make sure records are grouped solely by the natural key. This is accomplished by writing a custom GroupPartitioner. We have a Comparator object only considering the yearMonth field of the TemperaturePair class for the purposes of grouping the records together.

public class YearMonthGroupingComparator extends WritableComparator {
    public YearMonthGroupingComparator() {
        super(TemperaturePair.class, true);
    }
    @Override
    public int compare(WritableComparable tp1, WritableComparable tp2) {
        TemperaturePair temperaturePair = (TemperaturePair) tp1;
        TemperaturePair temperaturePair2 = (TemperaturePair) tp2;
        return temperaturePair.getYearMonth().compareTo(temperaturePair2.getYearMonth());
    }
}

Results

Here are the results of running our secondary sort job:

new-host-2:sbin bbejeck$ hdfs dfs -cat secondary-sort/part-r-00000
190101	-206
190102	-333
190103	-272
190104	-61
190105	-33
190106	44
190107	72
190108	44
190109	17
190110	-33
190111	-217
190112	-300

Conclusion

While sorting data by value may not be a common need, it’s a nice tool to have in your back pocket when needed. Also, we have been able to take a deeper look at the inner workings of Hadoop by working with custom partitioners and group partitioners. Thank you for your time.

Resources

• Data-Intensive Processing with MapReduce by Jimmy Lin and Chris Dyer
• Hadoop: The Definitive Guide by Tom White
• Source Code and Tests from blog
• Hadoop API
• MRUnit for unit testing Apache Hadoop map reduce jobs

Mapper Code

First we need to modify our mapper from the Pairs approach. At the bottom of each loop after we have emitted all the word pairs for a particular word, we will emit the special token WordPair(“word”,”*”) along with the count of times the word on the left was found.

public class PairsRelativeOccurrenceMapper extends Mapper<LongWritable, Text, WordPair, IntWritable> {
    private WordPair wordPair = new WordPair();
    private IntWritable ONE = new IntWritable(1);
    private IntWritable totalCount = new IntWritable();

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        int neighbors = context.getConfiguration().getInt("neighbors", 2);
        String[] tokens = value.toString().split("\\s+");
        if (tokens.length > 1) {
            for (int i = 0; i < tokens.length; i++) {
                    tokens[i] = tokens[i].replaceAll("\\W+","");

                    if(tokens[i].equals("")){
                        continue;
                    }

                    wordPair.setWord(tokens[i]);

                    int start = (i - neighbors < 0) ? 0 : i - neighbors;
                    int end = (i + neighbors >= tokens.length) ? tokens.length - 1 : i + neighbors;
                    for (int j = start; j <= end; j++) {
                        if (j == i) continue;
                        wordPair.setNeighbor(tokens[j].replaceAll("\\W",""));
                        context.write(wordPair, ONE);
                    }
                    wordPair.setNeighbor("*");
                    totalCount.set(end - start);
                    context.write(wordPair, totalCount);
            }
        }
    }
}

Now that we’ve generated a way to track the total numbers of times a particular word has been encountered, we need to make sure those special characters reach the reducer first so a total can be tallied to calculate the relative frequencies. We will have the sorting phase of the MapReduce process handle this for us by modifying the compareTo method on the WordPair object.

Modified Sorting

We modify the compareTo method on the WordPair class so when a “*” caracter is encountered on the right that particular object is pushed to the top.

    @Override
    public int compareTo(WordPair other) {
        int returnVal = this.word.compareTo(other.getWord());
        if(returnVal != 0){
            return returnVal;
        }
        if(this.neighbor.toString().equals("*")){
            return -1;
        }else if(other.getNeighbor().toString().equals("*")){
            return 1;
        }
        return this.neighbor.compareTo(other.getNeighbor());
    }

By modifying the compareTo method we now are guaranteed that any WordPair with the special character will be sorted to the top and arrive at the reducer first. This leads to our second specialization, how can we guarantee that all WordPair objects with a given left word will be sent to the same reducer? The answer is to create a custom partitioner.

Custom Partitioner

Intermediate keys are shuffled to reducers by calculating the hashcode of the key modulo the number of reducers. But our WordPair objects contain two words, so taking the hashcode of the entire object clearly won’t work. We need to wright a custom Partitioner that only takes into consideration the left word when it comes to determining which reducer to send the output to.

public class WordPairPartitioner extends Partitioner<WordPair,IntWritable> {

    @Override
    public int getPartition(WordPair wordPair, IntWritable intWritable, int numPartitions) {
        return wordPair.getWord().hashCode() % numPartitions;
    }
}

Now we are guaranteed that all of the WordPair objects with the same left word are sent to the same reducer. All that is left is to construct a reducer to take advantage of the format of the data being sent.

Reducer

Building the reducer for the inverted order inversion pattern is straight forward. It will involve keeping a counter variable and a “current” word variable. The reducer will check the input key WordPair for the special character “*” on the right. If the word on the left is not equal to the “current” word we will re-set the counter and sum all of the values to obtain a total number of times the given current word was observed. We will now process the next WordPair objects, sum the counts and divide by our counter variable to obtain a relative frequency. This process will continue until another special character is encountered and the process starts over.

public class PairsRelativeOccurrenceReducer extends Reducer<WordPair, IntWritable, WordPair, DoubleWritable> {
    private DoubleWritable totalCount = new DoubleWritable();
    private DoubleWritable relativeCount = new DoubleWritable();
    private Text currentWord = new Text("NOT_SET");
    private Text flag = new Text("*");

    @Override
    protected void reduce(WordPair key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
        if (key.getNeighbor().equals(flag)) {
            if (key.getWord().equals(currentWord)) {
                totalCount.set(totalCount.get() + getTotalCount(values));
            } else {
                currentWord.set(key.getWord());
                totalCount.set(0);
                totalCount.set(getTotalCount(values));
            }
        } else {
            int count = getTotalCount(values);
            relativeCount.set((double) count / totalCount.get());
            context.write(key, relativeCount);
        }
    }
  private int getTotalCount(Iterable<IntWritable> values) {
        int count = 0;
        for (IntWritable value : values) {
            count += value.get();
        }
        return count;
    }
}

By manipulating the sort order and creating a custom partitioner, we have been able to send data to a reducer needed for a calculation, before the data needed for those calculation arrive. Although not shown here, a combiner was used to run the MapReduce job. This approach is also a good candidate for the “in-mapper” combining pattern.

Example & Results

Given that the holidays are upon us, I felt it was timely to run an example of the order inversion pattern against the novel “A Christmas Carol” by Charles Dickens. I know it’s corny, but it serves the purpose.

new-host-2:sbin bbejeck$ hdfs dfs -cat relative/part* | grep Humbug
{word=[Humbug] neighbor=[Scrooge]}	0.2222222222222222
{word=[Humbug] neighbor=[creation]}	0.1111111111111111
{word=[Humbug] neighbor=[own]}	0.1111111111111111
{word=[Humbug] neighbor=[said]}	0.2222222222222222
{word=[Humbug] neighbor=[say]}	0.1111111111111111
{word=[Humbug] neighbor=[to]}	0.1111111111111111
{word=[Humbug] neighbor=[with]}	0.1111111111111111
{word=[Scrooge] neighbor=[Humbug]}	0.0020833333333333333
{word=[creation] neighbor=[Humbug]}	0.1
{word=[own] neighbor=[Humbug]}	0.006097560975609756
{word=[said] neighbor=[Humbug]}	0.0026246719160104987
{word=[say] neighbor=[Humbug]}	0.010526315789473684
{word=[to] neighbor=[Humbug]}	3.97456279809221E-4
{word=[with] neighbor=[Humbug]}	9.372071227741331E-4

Conclusion

While calculating relative word occurrence frequencies probably is not a common task, we have been able to demonstrate useful examples of sorting and using a custom partitioner, which are good tools to have at your disposal when building MapReduce programs. As stated before, even if most of your MapReduce is written at higher level of abstraction like Hive or Pig, it’s still instructive to have an understanding of what is going on under the hood. Thanks for your time.

Resources

• Data-Intensive Processing with MapReduce by Jimmy Lin and Chris Dyer
• Hadoop: The Definitive Guide by Tom White
• Source Code and Tests from blog
• Hadoop API
• MRUnit for unit testing Apache Hadoop map reduce jobs

1. Working Through Data-Intensive Text Processing with MapReduce
2. Working Through Data-Intensive Text Processing with MapReduce – Local Aggregation Part II

A co-occurrence matrix could be described as the tracking of an event, and given a certain window of time or space, what other events seem to occur. For the purposes of this post, our “events” are the individual words found in the text and we will track what other words occur within our “window”, a position relative to the target word. For example, consider the phrase “The quick brown fox jumped over the lazy dog”. With a window value of 2, the co-occurrence for the word “jumped” would be [brown,fox,over,the]. A co-occurrence matrix could be applied to other areas that require investigation into when “this” event occurs, what other events seem to happen at the same time. To build our text co-occurrence matrix, we will be implementing the Pairs and Stripes algorithms found in chapter 3 of Data-Intensive Text Processing with MapReduce. The body of text used to create our co-occurrence matrix is the collective works of William Shakespeare.

Pairs

Implementing the pairs approach is straightforward. For each line passed in when the map function is called, we will split on spaces creating a String Array. The next step would be to construct two loops. The outer loop will iterate over each word in the array and the inner loop will iterate over the “neighbors” of the current word. The number of iterations for the inner loop is dictated by the size of our “window” to capture neighbors of the current word. At the bottom of each iteration in the inner loop, we will emit a WordPair object (consisting of the current word on the left and the neighbor word on the right) as the key, and a count of one as the value. Here is the code for the Pairs implementation:

public class PairsOccurrenceMapper extends Mapper<LongWritable, Text, WordPair, IntWritable> {
    private WordPair wordPair = new WordPair();
    private IntWritable ONE = new IntWritable(1);

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        int neighbors = context.getConfiguration().getInt("neighbors", 2);
        String[] tokens = value.toString().split("\\s+");
        if (tokens.length > 1) {
          for (int i = 0; i < tokens.length; i++) {
              wordPair.setWord(tokens[i]);

             int start = (i - neighbors < 0) ? 0 : i - neighbors;
             int end = (i + neighbors >= tokens.length) ? tokens.length - 1 : i + neighbors;
              for (int j = start; j <= end; j++) {
                  if (j == i) continue;
                   wordPair.setNeighbor(tokens[j]);
                   context.write(wordPair, ONE);
              }
          }
      }
  }
}

The Reducer for the Pairs implementation will simply sum all of the numbers for the given WordPair key:

public class PairsReducer extends Reducer<WordPair,IntWritable,WordPair,IntWritable> {
    private IntWritable totalCount = new IntWritable();
    @Override
    protected void reduce(WordPair key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
        int count = 0;
        for (IntWritable value : values) {
             count += value.get();
        }
        totalCount.set(count);
        context.write(key,totalCount);
    }
}

Stripes

Implementing the stripes approach to co-occurrence is equally straightforward. The approach is the same, but all of the “neighbor” words are collected in a HashMap with the neighbor word as the key and an integer count as the value. When all of the values have been collected for a given word (the bottom of the outer loop), the word and the hashmap are emitted. Here is the code for our Stripes implementation:

public class StripesOccurrenceMapper extends Mapper<LongWritable,Text,Text,MapWritable> {
  private MapWritable occurrenceMap = new MapWritable();
  private Text word = new Text();

  @Override
 protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
   int neighbors = context.getConfiguration().getInt("neighbors", 2);
   String[] tokens = value.toString().split("\\s+");
   if (tokens.length > 1) {
      for (int i = 0; i < tokens.length; i++) {
          word.set(tokens[i]);
          occurrenceMap.clear();

          int start = (i - neighbors < 0) ? 0 : i - neighbors;
          int end = (i + neighbors >= tokens.length) ? tokens.length - 1 : i + neighbors;
           for (int j = start; j <= end; j++) {
                if (j == i) continue;
                Text neighbor = new Text(tokens[j]);
                if(occurrenceMap.containsKey(neighbor)){
                   IntWritable count = (IntWritable)occurrenceMap.get(neighbor);
                   count.set(count.get()+1);
                }else{
                   occurrenceMap.put(neighbor,new IntWritable(1));
                }
           }
          context.write(word,occurrenceMap);
     }
   }
  }
}

The Reducer for the Stripes approach is a little more involved due to the fact we will need to iterate over a collection of maps, then for each map, iterate over all of the values in the map:

public class StripesReducer extends Reducer<Text, MapWritable, Text, MapWritable> {
    private MapWritable incrementingMap = new MapWritable();

    @Override
    protected void reduce(Text key, Iterable<MapWritable> values, Context context) throws IOException, InterruptedException {
        incrementingMap.clear();
        for (MapWritable value : values) {
            addAll(value);
        }
        context.write(key, incrementingMap);
    }

    private void addAll(MapWritable mapWritable) {
        Set<Writable> keys = mapWritable.keySet();
        for (Writable key : keys) {
            IntWritable fromCount = (IntWritable) mapWritable.get(key);
            if (incrementingMap.containsKey(key)) {
                IntWritable count = (IntWritable) incrementingMap.get(key);
                count.set(count.get() + fromCount.get());
            } else {
                incrementingMap.put(key, fromCount);
            }
        }
    }
}

Conclusion

When looking at the two approaches, we can see that the Pairs algorithm will generate more key value pairs compared to the Stripes algorithm. Also, the Pairs algorithm captures each individual co-occurrence event while the Stripes algorithm captures all co-occurrences for a given event. Both the Pairs and Stripes implementations would benefit from using a Combiner. Because both produce commutative and associative results, we can simply re-use each Mapper’s Reducer as the Combiner. As stated before, creating a co-occurrence matrix has applicability to other fields beyond text processing, and represent useful MapReduce algorithms to have in one’s arsenal. Thanks for your time.

Resources

• Data-Intensive Processing with MapReduce by Jimmy Lin and Chris Dyer
• Hadoop: The Definitive Guide by Tom White
• Source Code and Tests from blog
• Hadoop API
• MRUnit for unit testing Apache Hadoop map reduce jobs

Setup

To get started, download MRUnit from here. After you have extracted the tar file, cd into the mrunit-0.9.0-incubating/lib directory. In there you should see the following:

1. mrunit-0.9.0-incubating-hadoop1.jar
2. mrunit-0.9.0-incubating-hadoop2.jar

As I’m sure can guess, the mrunit-0.9.0-incubating-hadoop1.jar is for MapReduce version 1 of Hadoop and mrunit-0.9.0-incubating-hadoop2.jar is for working the new version of Hadoop’s MapReduce. For this post, and all others going forward, we will be using hadoop-2.0 version from Cloudera’s CDH4.1.1 release so we will need the mrunit-0.9.0-incubating-hadoop2.jar file. I added MRUnit, JUnit and Mockito as libraries in Intellij (JUnit and Mockito are found in the same directory as the MRUnit jar files). Now that we have set up our dependencies, let’s start testing.

Testing Mappers

Setting up to test a mapper is very straight forward and is best explained by looking at some code first. We will use the in-mapper combining example from the previous post:

@Test
public void testCombiningMapper() throws Exception {
   new MapDriver<LongWritable,Text,Text,TemperatureAveragingPair>()
           .withMapper(new AverageTemperatureCombiningMapper())
           .withInput(new LongWritable(4),new Text(temps[3]))
           .withOutput(new Text("190101"),new TemperatureAveragingPair(-61,1))
           .runTest();
 }

Notice the fluent api style which adds the ease of creating the test. To write your test you would:

1. Instantiate an instance of the MapDriver class parameterized exactly as the mapper under test.
2. Add an instance of the Mapper you are testing in the withMapper call.
3. In the withInput call pass in your key and input value, in this case a LongWritable with an arbitrary value and a Text object that contains a line from from the NCDC weather dataset contained in a String array called ‘temps’ that was set up earlier in the test (not displayed here as it would take away from the presentation).
4. Specify the expected output in the withOutput call, here we are expecting a Text object with the value of “190101″ and a TemperatureAveragingPair object containing the values -61 (temperature) and a 1 (count).
5. The last call runTest feeds the specified input values into the mapper and compares the actual output against the expected output set in the ‘withOutput’ method.

One thing to note is the MapDriver only allows one input and output per test. You can call withInput and withOutput multiple times if you want, but the MapDriver will overwrite the existing values with the new ones, so you will only ever be testing with one input/output at any time. To specify multiple inputs we would use the MapReduceDriver, covered a couple of sections later, but next up is testing the reducer.

Testing Reducers

Testing the reducer follows the same pattern as the mapper test. Again, let’s start by looking at a code example:

@Test
public void testReducerCold(){
  List<TemperatureAveragingPair> pairList = new ArrayList<TemperatureAveragingPair>();
      pairList.add(new TemperatureAveragingPair(-78,1));
      pairList.add(new TemperatureAveragingPair(-84,1));
      pairList.add(new TemperatureAveragingPair(-28,1));
      pairList.add(new TemperatureAveragingPair(-56,1));

      new ReduceDriver<Text,TemperatureAveragingPair,Text,IntWritable>()
                .withReducer(new AverageTemperatureReducer())
                .withInput(new Text("190101"), pairList)
                .withOutput(new Text("190101"),new IntWritable(-61))
                .runTest();
    }

1. The test starts by creating a list of TemperatureAveragingPair objects to be used as the input to the reducer.
2. A ReducerDriver is instantiated, and like the MapperDriver, is parameterized exactly as the reducer under test.
3. Next we pass in an instance of the reducer we want to test in the withReducer call.
4. In the withInput call we pass in the key of “190101″ and the pairList object created at the start of the test.
5. Next we specify the output that we expect our reducer to emit, the same key of “190101″ and an IntWritable representing the average of the temperatures in the list.
6. Finally runTest is called, which feeds our reducer the inputs specified and compares the output from the reducer against the expect output.

The ReducerDriver has the same limitation as the MapperDriver of not accepting more than one input/output pair. So far we have tested the Mapper and Reducer in isolation, but we would also like to test them together in an integration test. Integration testing can be accomplished by using the MapReduceDriver class. The MapReduceDriver is also the class to use for testing the use of combiners, custom counters or custom partitioners.

Integration Testing

To test your mapper and reducer working together, MRUnit provides the MapReduceDriver class. The MapReduceDriver class as you would expect by now, with 2 main differences. First, you parameterize the input and output types of the mapper and the input and output types of the reducer. Since the mapper output types need to match the reducer input types, you end up with 3 pairs of parameterized types. Secondly you can provide multiple inputs and specify multiple expected outputs. Here is our sample code:

@Test
public void testMapReduce(){

new MapReduceDriver<LongWritable,Text,
                      Text,TemperatureAveragingPair,
                      Text,IntWritable>()
                .withMapper(new AverageTemperatureMapper())
                .withInput(new LongWritable(1),new Text(temps[0]))
                .withInput(new LongWritable(2),new Text(temps[1]))
                .withInput(new LongWritable(3),new Text(temps[2]))
                .withInput(new LongWritable(4),new Text(temps[3]))
                .withInput(new LongWritable(5),new Text(temps[6]))
                .withInput(new LongWritable(6),new Text(temps[7]))
                .withInput(new LongWritable(7),new Text(temps[8]))
                .withInput(new LongWritable(8),new Text(temps[9]))
                .withCombiner(new AverageTemperatureCombiner())
                .withReducer(new AverageTemperatureReducer())
                .withOutput(new Text("190101"),new IntWritable(-22))
                .withOutput(new Text("190102"),new IntWritable(-40))
                .runTest();
    }

As you can see from the example above, the setup is the same as the MapDriver and the ReduceDriver classes. You pass in instances of the mapper, reducer and optionally a combiner to test. The MapReduceDriver allows us to pass in multiple inputs that have different keys. Here the ‘temps’ array is the same one referenced in the mapper sample and contains a few lines from the NCDC weather dataset and the keys in those sample lines are the months of January and February of the year 1901 represented as “190101″ and “190102″ respectively. This test is successful, so we gain a little more confidence around the correctness of our mapper and reducer working together.

Conclusion

Hopefully, we have made the case for how useful MRUnit can be for testing Hadoop programs. I’d like to wrap this post up with some of my own observations. Although MRUnit makes unit testing easy for mapper and reducer code, the mapper and reducer examples presented here are fairly simple. If your map and/or reduce code starts to become more complex, it’s probably a better to decouple the code from the Hadoop framework and test the new classes on their own. Also, as useful as the MapReduceDriver class is for integration testing, it’s very easy to get to a point where you are no longer testing your code, but the Hadoop framework itself, which has already been done. I’ve come up with my own testing strategy that I intend to use going forward:

1. Unit test the map/reduce code.
2. Possibly write one integration test with the MapReduceDriver class.
3. As a sanity check, run a MapReduce job on a single node install (on my laptop) to ensure it runs on the Hadoop framework.
4. Then run my code on a test cluster, on EC2 using Apache Whirr in my case.

Covering how to set up a single node install on my laptop (OSX Lion) and standing up a cluster on EC2 using Whirr would make this post too long, so I’ll cover those topics in the next one. Thanks for your time.

Resources

• Data-Intensive Processing with MapReduce by Jimmy Lin and Chris Dyer
• Hadoop: The Definitive Guide by Tom White
• Source Code from blog
• Hadoop API
• MRUnit for unit testing Apache Hadoop map reduce jobs
• Project Gutenberg a great source of books in plain text format, great for testing Hadoop jobs locally.

One Size Does Not Fit All

Last time we described two approaches for reducing data in a MapReduce job, Hadoop Combiners and the in-mapper combining approach. Combiners are considered an optimization by the Hadoop framework and there are no guarantees on how many times they will be called, if at all. As a result, mappers must emit data in the form expected by the reducers so if combiners aren’t involved, the final result is not changed. To adjust for calculating averages, we need to go back to the mapper and change it’s output.

Mapper Changes

In the word-count example, the non-optimized mapper simply emitted the word and the count of 1. The combiner and in-mapper combining mapper optimized this output by keeping each word as a key in a hash map with the total count as the value. Each time a word was seen the count was incremented by 1. With this setup, if the combiner was not called, the reducer would receive the word as a key and a long list of 1′s to add together, resulting in the same output (of course using the in-mapper combining mapper avoided this issue because it’s guaranteed to combine results as it’s part of the mapper code). To compute an average, we will have our base mapper emit a string key (the year and month of the weather observation concatenated together) and a custom writable object, called TemperatureAveragingPair. The TemperatureAveragingPair object will contain two numbers (IntWritables), the temperature taken and a count of one. We will take the MaximumTemperatureMapper from Hadoop: The Definitive Guide and use it as inspiration for creating an AverageTemperatureMapper:

public class AverageTemperatureMapper extends Mapper<LongWritable, Text, Text, TemperatureAveragingPair> {
 //sample line of weather data
 //0029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF10899199999999999


    private Text outText = new Text();
    private TemperatureAveragingPair pair = new TemperatureAveragingPair();
    private static final int MISSING = 9999;

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String line = value.toString();
        String yearMonth = line.substring(15, 21);

        int tempStartPosition = 87;

        if (line.charAt(tempStartPosition) == '+') {
            tempStartPosition += 1;
        }

        int temp = Integer.parseInt(line.substring(tempStartPosition, 92));

        if (temp != MISSING) {
            outText.set(yearMonth);
            pair.set(temp, 1);
            context.write(outText, pair);
        }
    }
}

By having the mapper output a key and TemperatureAveragingPair object our MapReduce program is guaranteed to have the correct results regardless if the combiner is called.

Combiner

We need to reduce the amount of data sent, so we will sum the temperatures, and sum the counts and store them separately. By doing so we will reduce data sent, but preserve the format needed for calculating correct averages. If/when the combiner is called, it will take all the TemperatureAveragingPair objects passed in and emit a single TemperatureAveragingPair object for the same key, containing the summed temperatures and counts. Here is the code for the combiner:

  public class AverageTemperatureCombiner extends Reducer<Text,TemperatureAveragingPair,Text,TemperatureAveragingPair> {
    private TemperatureAveragingPair pair = new TemperatureAveragingPair();

    @Override
    protected void reduce(Text key, Iterable<TemperatureAveragingPair> values, Context context) throws IOException, InterruptedException {
        int temp = 0;
        int count = 0;
        for (TemperatureAveragingPair value : values) {
             temp += value.getTemp().get();
             count += value.getCount().get();
        }
        pair.set(temp,count);
        context.write(key,pair);
    }
}
 

But we are really interested in being guaranteed we have reduced the amount of data sent to the reducers, so we’ll have a look at how to achieve that next.

In Mapper Combining Averages

Similar to the word-count example, for calculating averages, the in-mapper-combining mapper will use a hash map with the concatenated year+month as a key and a TemperatureAveragingPair as the value. Each time we get the same year+month combination, we’ll take the pair object out of the map, add the temperature and increase the count by by one. Once the cleanup method is called we’ll and emit all pairs with their respective key:

public class AverageTemperatureCombiningMapper extends Mapper<LongWritable, Text, Text, TemperatureAveragingPair> {
 //sample line of weather data
 //0029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF10899199999999999


    private static final int MISSING = 9999;
    private Map<String,TemperatureAveragingPair> pairMap = new HashMap<String,TemperatureAveragingPair>();


    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String line = value.toString();
        String yearMonth = line.substring(15, 21);

        int tempStartPosition = 87;

        if (line.charAt(tempStartPosition) == '+') {
            tempStartPosition += 1;
        }

        int temp = Integer.parseInt(line.substring(tempStartPosition, 92));

        if (temp != MISSING) {
            TemperatureAveragingPair pair = pairMap.get(yearMonth);
            if(pair == null){
                pair = new TemperatureAveragingPair();
                pairMap.put(yearMonth,pair);
            }
            int temps = pair.getTemp().get() + temp;
            int count = pair.getCount().get() + 1;
            pair.set(temps,count);
        }
    }


    @Override
    protected void cleanup(Context context) throws IOException, InterruptedException {
        Set<String> keys = pairMap.keySet();
        Text keyText = new Text();
        for (String key : keys) {
             keyText.set(key);
             context.write(keyText,pairMap.get(key));
        }
    }
}

By following the same pattern of keeping track of data between map calls, we can achieve reliable data reduction by implementing an in-mapper combining strategy. The same caveats apply for keeping state across all calls to the mapper, but looking at the gains that can be made in processing efficiency, using this approach merits some consideration.

Reducer

At this point, writing our reducer is easy, take a list of pairs for each key, sum all the temperatures and counts then divide the sum of the temperatures by the sum of the counts.

public class AverageTemperatureReducer extends Reducer<Text, TemperatureAveragingPair, Text, IntWritable> {
    private IntWritable average = new IntWritable();

    @Override
    protected void reduce(Text key, Iterable<TemperatureAveragingPair> values, Context context) throws IOException, InterruptedException {
        int temp = 0;
        int count = 0;
        for (TemperatureAveragingPair pair : values) {
            temp += pair.getTemp().get();
            count += pair.getCount().get();
        }
        average.set(temp / count);
        context.write(key, average);
    }
}

Results

The results are predictable with the combiner and in-mapper-combining mapper options showing substantially reduced data output.
Non-Optimized Mapper Option:

12/10/10 23:05:28 INFO mapred.JobClient:     Reduce input groups=12
12/10/10 23:05:28 INFO mapred.JobClient:     Combine output records=0
12/10/10 23:05:28 INFO mapred.JobClient:     Map input records=6565
12/10/10 23:05:28 INFO mapred.JobClient:     Reduce shuffle bytes=111594
12/10/10 23:05:28 INFO mapred.JobClient:     Reduce output records=12
12/10/10 23:05:28 INFO mapred.JobClient:     Spilled Records=13128
12/10/10 23:05:28 INFO mapred.JobClient:     Map output bytes=98460
12/10/10 23:05:28 INFO mapred.JobClient:     Total committed heap usage (bytes)=269619200
12/10/10 23:05:28 INFO mapred.JobClient:     Combine input records=0
12/10/10 23:05:28 INFO mapred.JobClient:     Map output records=6564
12/10/10 23:05:28 INFO mapred.JobClient:     SPLIT_RAW_BYTES=108
12/10/10 23:05:28 INFO mapred.JobClient:     Reduce input records=6564

Combiner Option:

12/10/10 23:07:19 INFO mapred.JobClient:     Reduce input groups=12
12/10/10 23:07:19 INFO mapred.JobClient:     Combine output records=12
12/10/10 23:07:19 INFO mapred.JobClient:     Map input records=6565
12/10/10 23:07:19 INFO mapred.JobClient:     Reduce shuffle bytes=210
12/10/10 23:07:19 INFO mapred.JobClient:     Reduce output records=12
12/10/10 23:07:19 INFO mapred.JobClient:     Spilled Records=24
12/10/10 23:07:19 INFO mapred.JobClient:     Map output bytes=98460
12/10/10 23:07:19 INFO mapred.JobClient:     Total committed heap usage (bytes)=269619200
12/10/10 23:07:19 INFO mapred.JobClient:     Combine input records=6564
12/10/10 23:07:19 INFO mapred.JobClient:     Map output records=6564
12/10/10 23:07:19 INFO mapred.JobClient:     SPLIT_RAW_BYTES=108
12/10/10 23:07:19 INFO mapred.JobClient:     Reduce input records=12

In-Mapper-Combining Option:

12/10/10 23:09:09 INFO mapred.JobClient:     Reduce input groups=12
12/10/10 23:09:09 INFO mapred.JobClient:     Combine output records=0
12/10/10 23:09:09 INFO mapred.JobClient:     Map input records=6565
12/10/10 23:09:09 INFO mapred.JobClient:     Reduce shuffle bytes=210
12/10/10 23:09:09 INFO mapred.JobClient:     Reduce output records=12
12/10/10 23:09:09 INFO mapred.JobClient:     Spilled Records=24
12/10/10 23:09:09 INFO mapred.JobClient:     Map output bytes=180
12/10/10 23:09:09 INFO mapred.JobClient:     Total committed heap usage (bytes)=269619200
12/10/10 23:09:09 INFO mapred.JobClient:     Combine input records=0
12/10/10 23:09:09 INFO mapred.JobClient:     Map output records=12
12/10/10 23:09:09 INFO mapred.JobClient:     SPLIT_RAW_BYTES=108
12/10/10 23:09:09 INFO mapred.JobClient:     Reduce input records=12

Calculated Results:
(NOTE: the temperatures in the sample file are in Celsius * 10)

Conclusion

We have covered local aggregation for both the simple case where one could reuse the reducer as a combiner and a more complicated case where some insight on how to structure the data while still gaining the benefits of locally aggregating data for increase processing efficiency.

Resources

• Data-Intensive Processing with MapReduce by Jimmy Lin and Chris Dyer
• Hadoop: The Definitive Guide by Tom White
• Source Code from blog
• Hadoop API
• MRUnit for unit testing Apache Hadoop map reduce jobs
• Project Gutenberg a great source of books in plain text format, great for testing Hadoop jobs locally.

Local Aggregation

At a very high level, when Mappers emit data, the intermediate results are written to disk then sent across the network to Reducers for final processing. The latency of writing to disk then transferring data across the network is an expensive operation in the processing of a MapReduce job. So it stands to reason that whenever possible, reducing the amount of data sent from mappers would increase the speed of the MapReduce job. Local aggregation is a technique used to reduce the amount of data and improve the efficiency of our MapReduce job. Local aggregation can not take the place of reducers, as we need a way to gather results with the same key from different mappers. We are going to consider 3 ways of achieving local aggregation:

1. Using Hadoop Combiner functions.
2. Two approaches of “in-mapper” combining presented in the Text Processing with MapReduce book.

Of course any optimization is going to have tradeoffs and we’ll discuss those as well.
To demonstrate local aggregation, we will run the ubiquitous word count job on a plain text version of A Christmas Carol by Charles Dickens (downloaded from Project Gutenberg) on a pseudo distributed cluster installed on my MacBookPro, using the hadoop-0.20.2-cdh3u3 distribution from Cloudera. I plan in a future post to run the same experiment on an EC2 cluster with more realistic sized data.

Combiners

A combiner function is an object that extends the Reducer class. In fact, for our examples here, we are going to re-use the same reducer used in the word count job. A combiner function is specified when setting up the MapReduce job like so:

 job.setReducerClass(TokenCountReducer.class);

Here is the reducer code:

public class TokenCountReducer extends Reducer<Text,IntWritable,Text,IntWritable>{
    @Override
    protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
        int count = 0;
        for (IntWritable value : values) {
              count+= value.get();
        }
        context.write(key,new IntWritable(count));
    }
}

The job of a combiner is to do just what the name implies, aggregate data with the net result of less data begin shuffled across the network, which gives us gains in efficiency. As stated before, keep in mind that reducers are still required to put together results with the same keys coming from different mappers. Since combiner functions are an optimization, the Hadoop framework offers no guarantees on how many times a combiner will be called, if at all.

In Mapper Combining Option 1

The first alternative to using Combiners (figure 3.2 page 41) is very straight forward and makes a slight modification to our original word count mapper:

public class PerDocumentMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        IntWritable writableCount = new IntWritable();
        Text text = new Text();
        Map<String,Integer> tokenMap = new HashMap<String, Integer>();
        StringTokenizer tokenizer = new StringTokenizer(value.toString());

        while(tokenizer.hasMoreElements()){
            String token = tokenizer.nextToken();
            Integer count = tokenMap.get(token);
            if(count == null) count = new Integer(0);
            count+=1;
            tokenMap.put(token,count);
        }

        Set<String> keys = tokenMap.keySet();
        for (String s : keys) {
             text.set(s);
             writableCount.set(tokenMap.get(s));
             context.write(text,writableCount);
        }
    }
}

As we can see here, instead of emitting a word with the count of 1, for each word encountered, we use a map to keep track of each word already processed. Then when all of the tokens are processed we loop through the map and emit the total count for each word encountered in that line.

In Mapper Combining Option 2

The second option of in mapper combining (figure 3.3 page 41) is very similar to the above example with two distinctions – when the hash map is created and when we emit the results contained in the map. In the above example, a map is created and has its contents dumped over the wire for each invocation of the map method. In this example we are going make the map an instance variable and shift the instantiation of the map to the setUp method in our mapper. Likewise the contents of the map will not be sent out to the reducers until all of the calls to mapper have completed and the cleanUp method is called.

public class AllDocumentMapper extends Mapper<LongWritable,Text,Text,IntWritable> {

    private  Map<String,Integer> tokenMap;

    @Override
    protected void setup(Context context) throws IOException, InterruptedException {
           tokenMap = new HashMap<String, Integer>();
    }

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        StringTokenizer tokenizer = new StringTokenizer(value.toString());
        while(tokenizer.hasMoreElements()){
            String token = tokenizer.nextToken();
            Integer count = tokenMap.get(token);
            if(count == null) count = new Integer(0);
            count+=1;
            tokenMap.put(token,count);
        }
    }


    @Override
    protected void cleanup(Context context) throws IOException, InterruptedException {
        IntWritable writableCount = new IntWritable();
        Text text = new Text();
        Set<String> keys = tokenMap.keySet();
        for (String s : keys) {
            text.set(s);
            writableCount.set(tokenMap.get(s));
            context.write(text,writableCount);
        }
    }
}

As we can see from the above code example, the mapper is keeping track of unique word counts, across all calls to the map method. By keeping track of unique tokens and their counts, there should be a substantial reduction in the number of records sent to the reducers, which in turn should improve the running time of the MapReduce job. This accomplishes the same effect as using the combiner function option provided by the MapReduce framework, but in this case you are guaranteed that the combining code will be called. But there are some caveats with this approach also. Keeping state across map calls could prove problematic and definitely is a violation of the functional spirit of a “map” function. Also, by keeping state across all mappers, depending on the data used in the job, memory could be another issue to contend with. Ultimately, one would have to weigh all of the trade offs to determine the best approach.

Results

Now lets take a look at the some results of the different mappers. Since the job was run in pseudo-distributed mode, actual running times are irrelevant, but we can still infer how using local aggregation could impact the efficiency of MapReduce job running on a real cluster.

Per Token Mapper:

12/09/13 21:25:32 INFO mapred.JobClient:     Reduce shuffle bytes=366010
12/09/13 21:25:32 INFO mapred.JobClient:     Reduce output records=7657
12/09/13 21:25:32 INFO mapred.JobClient:     Spilled Records=63118
12/09/13 21:25:32 INFO mapred.JobClient:     Map output bytes=302886

In Mapper Reducing Option 1:

12/09/13 21:28:15 INFO mapred.JobClient:     Reduce shuffle bytes=354112
12/09/13 21:28:15 INFO mapred.JobClient:     Reduce output records=7657
12/09/13 21:28:15 INFO mapred.JobClient:     Spilled Records=60704
12/09/13 21:28:15 INFO mapred.JobClient:     Map output bytes=293402

In Mapper Reducing Option 2:

12/09/13 21:30:49 INFO mapred.JobClient:     Reduce shuffle bytes=105885
12/09/13 21:30:49 INFO mapred.JobClient:     Reduce output records=7657
12/09/13 21:30:49 INFO mapred.JobClient:     Spilled Records=15314
12/09/13 21:30:49 INFO mapred.JobClient:     Map output bytes=90565

Combiner Option:

12/09/13 21:22:18 INFO mapred.JobClient:     Reduce shuffle bytes=105885
12/09/13 21:22:18 INFO mapred.JobClient:     Reduce output records=7657
12/09/13 21:22:18 INFO mapred.JobClient:     Spilled Records=15314
12/09/13 21:22:18 INFO mapred.JobClient:     Map output bytes=302886
12/09/13 21:22:18 INFO mapred.JobClient:     Combine input records=31559
12/09/13 21:22:18 INFO mapred.JobClient:     Combine output records=7657

As expected the Mapper that did no combining had the worst results, followed closely by the first in-mapper combining option (although these results could have been made better had the data been cleaned up before running the word count). The second in-mapper combining option and the combiner function had virtually identical results. The significant fact is that both produced 2/3 less reduce shuffle bytes as the first two options. Reducing the amount of bytes sent over the network to the reducers by that amount would surely would have a positive impact on the efficiency of a MapReduce job. There is one point to keep in mind here and that is Combiners/In-Mapper combining can not just be used in all MapReduce jobs, in this case the word count lends itself very nicely to such an enhancement, but that might not always be true.

Conclusion

As you can see the benefits of using either in-mapper combining or the Hadoop combiner function require serious consideration when looking to improve the performance of your MapReduce jobs. As for which approach, it is up to you the weigh the trade offs for each approach.

Resources

• Data-Intensive Processing with MapReduce by Jimmy Lin and Chris Dyer
• Hadoop: The Definitive Guide by Tom White
• Source Code from blog
• MRUnit for unit testing Apache Hadoop map reduce jobs
• Project Gutenberg a great source of books in plain text format, great for testing Hadoop jobs locally.

BloomFilter Crash Course

BloomFilters are essentially bit vectors. At a high level BloomFilters work in the following manner:

1. Add the element to the filter.
2. Hash it a few times, than set the bits to 1 where the index matches the results of the hash.

When testing if an element is in the set, you follow the same hashing procedure and check if the bits are set to 1 or 0. This process is how a BloomFilter can guarantee an element does not exist. If the bits aren’t set, it’s simply impossible for the element to be in the set. However, a positive answer means the element is in the set or a hashing collision occurred. A more detaild description of a BloomFilter can be found here and a good tutorial on BloomFilters here. According to wikipedia, Google uses BloomFilters in BigTable to avoid disk lookups for non-existent items. Another interesting usage is using a BloomFilter to optimize a sql querry.

Using the Guava BloomFilter

A Guava BloomFilter is created by calling the static method create on the BloomFilter class,
passing in a Funnel object and an int representing the expected number of insertions. A Funnel, also new in Guava 11, is an object that can send data into a Sink. The following example is the default implementation and has a percentage of false positives of 3%. Guava provides a Funnels class containing two static methods providing implementations of the Funnel interface for inserting a CharSequence or byte Array into a filter.

//Creating the BloomFilter
BloomFilter bloomFilter = BloomFilter.create(Funnels.byteArrayFunnel(), 1000);

//Putting elements into the filter
//A BigInteger representing a key of some sort
bloomFilter.put(bigInteger.toByteArray());

//Testing for element in set
boolean mayBeContained = bloomFilter.mayContain(bitIntegerII.toByteArray());

UPDATE: based on the comment from Louis Wasserman, here’s how to create a BloomFilter for BigIntegers with a custom Funnel implementation:

//Create the custom filter
class BigIntegerFunnel implements Funnel<BigInteger> {
        @Override
        public void funnel(BigInteger from, Sink into) {
            into.putBytes(from.toByteArray());
        }
    }

//Creating the BloomFilter
BloomFilter bloomFilter = BloomFilter.create(new BigIntegerFunnel(), 1000);

//Putting elements into the filter
//A BigInteger representing a key of some sort
bloomFilter.put(bigInteger);

//Testing for element in set
boolean mayBeContained = bloomFilter.mayContain(bitIntegerII);

Considerations

It’s critical to estimate the number of expected insertions correctly. As insertions into the filter approach or exceeds the expected number, the BloomFilter begins to fill up and as a result will generate more false positives to the point of being useless. There is another version of the BloomFilter.create method taking an additional parameter, a double representing the desired level of false hit probability (must be greater than 0 and less than one). The level of false hit probability affects the number of hashes for storing or searching for elements. The lower the desired percentage, the higher number of hashes performed.

Conclusion

A BloomFilter is a useful item for a developer to have in his/her toolbox. The Guava project now makes it very simple to begin using a BloomFilter when the need arises. I hope you enjoyed this post. Helpful comments and suggestions are welcomed.

References

• Unit Test Demo of Guava BloomFilter.
• BloomFilter class
• All You Want to Know about BloomFilters.
• BloomFilter Tutorial.
• BloomFilter on Wikipedia.

Why Use the EventBus

There are two main reasons for using the EventBus with a WatchService.

1. We don’t want poll for events, but would rather receive asynchronous notification.
2. Once events are processed, the WatchKey.reset method needs to be called to enable any new changes to be queued. While the WatchKey object is thread safe, it’s important that the reset method is called only after all threads have finished processing events, leading to somewhat of a coordination hassle. Using a single thread to process the events, invoke the reset method, then publish the changes via the EventBus, eliminates this problem.

Our plan to accomplish this is simple and will involve taking the following steps:

1. Instantiate an instance of the WatchService.
2. Register every directory recursively, starting with a given Path object.
3. Take events off the WatchService queue, then process and publish those events.
4. Start up a separate thread for taking events off the queue and publishing.

The code examples that follow are the more relevant highlights from the DirectoryEventWatcherImpl class that is going to do all of this work.

Registering Directories with the WatchService

While adding or deleting a sub-directory will generate an event, any changes inside a sub-directory of a watched directory will not. We are going to compensate for this by recursively going through all sub-directories (via the Files.walkFileTree method) and register each one with the WatchService object (previously defined in the example here):

private void registerDirectories() throws IOException {
        Files.walkFileTree(startPath, new WatchServiceRegisteringVisitor());
}

private class WatchServiceRegisteringVisitor extends SimpleFileVisitor<Path>{
    @Override
    public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
         dir.register(watchService,ENTRY_CREATE,ENTRY_DELETE,ENTRY_MODIFY);
         return FileVisitResult.CONTINUE;
    }
}

On line 2 the Files.walkFileTree method uses the WatchServiceRegisteringVisitor class defined on line 5 to register every directory with the WatchService. The registered events are creation of files/directories, deletion of files/directories or updates to a file.

Publishing Events

The next step is to create a FutureTask that will do the work of checking the queue and publishing the events.

 private void createWatchTask() {
    watchTask = new FutureTask<>(new Callable<Integer>() {
       private int totalEventCount;
       @Override
       public Integer call() throws Exception {
           while (keepWatching) {
               WatchKey watchKey = watchService.poll(10, TimeUnit.SECONDS);
               if (watchKey != null) {
                  List<WatchEvent<?>> events = watchKey.pollEvents();
                  Path watched = (Path) watchKey.watchable();
                  PathEvents pathEvents = new PathEvents(watchKey.isValid(), watched);
                  for (WatchEvent event : events) {
                        pathEvents.add(new PathEvent((Path) event.context(), event.kind()));
                        totalEventCount++;
                  }
                  watchKey.reset();
                  eventBus.post(pathEvents);
                }
          }
           return totalEventCount;
        }
      });
    }

private void startWatching() {
  new Thread(watchTask).start();
}

On line 7, we are checking the WatchService every 10 seconds for queued events. When a valid WatchKey is returned, the first step is to retrieve the events (line 9) then get the directory where the events occurred (line 10). On line 11 a PathEvents object is created, taking a boolean and the watched directory as constructor arguments. Lines 12-15 are looping over the events retrieved on line 9, using the target Path and event type as arguments to create PathEvent object. The WatchKey.reset method is called on line 16, setting the WatchKey state back to ready, making it eligible to receive new events and be placed back into the queue. Finally on line 17 the EventBus publishes the PathEvents object to all subscribers. It’s important to note here that the PathEvents and PathEvent classes are immutable. The totalEventCount that is returned from the Callable is never exposed in the API, but is used for testing purposes. The startWatching method on line 25 starts the thread to run the watching/publishing task defined above.

Conclusion

By pairing the WatchService with the Guava EventBus we are able to manage the WatchKey and process events in a single thread and notify any number of subscribers asynchronously of the events. It is hoped the reader found this example useful. As always comments and suggestions are welcomed.

Resources

• Source code and unit test for this post
• EventBus API
• WatchService API
• Previous post on the WatchService.
• Previous post on the EventBus

Requirements

Here are the requirements:

1. Starting from a given Path object, it should recursively go through each directory looking for files that match a given pattern.
2. It needs to be a single method call, something like DirUtils.glob(Path path, String pattern).
3. We want the search to be asynchronous, so we can iterate over files as they are found.

The plan to meet our objective is straight forward:

1. Create a DirectoryStream.Filter object from the given pattern.
2. Use the Files.walkFileTree method to recursively search for files that match our pattern.
3. Run the search in a separate thread, and when a matching file is found, place it in a queue.
4. The DirectoryStream iterator will pull the files out of the queue as they become available

Disclosure: This post is merely an attempt to see if making an asynchronous, recursive DirectorySteam will work, I make no guarantees about the code presented here, YMMV.

The Filter

DirectoryStream has a static interface, Filter, that is used to determine if the path object should be accepted. The filter object is constructed by first creating a PathMatcher object then using the PathMatcher in the Filter’s accept method:

public static DirectoryStream.Filter<Path> buildGlobFilter(String pattern) {
        final PathMatcher pathMatcher = getPathMatcher("glob:"+pattern);
        return new DirectoryStream.Filter<Path>() {
            @Override
            public boolean accept(Path entry) throws IOException {
                return pathMatcher.matches(entry);
            }
        };
    }

The Recursive Search

To perform the search we are going the use the Files.walkFileTree method with a FunctionVisitor object. FunctionVisitor is a subclass of SimpleFileVisitor that takes a Guava Function as a constructor argument. The provided function is called as each file is visited and is checked by the DirectoryStream.Filter object for a match. We wrap the creation of the Function object in a method call:

 private Function<Path, FileVisitResult> getFunction(final Filter<Path> filter) {
        return new Function<Path, FileVisitResult>() {
            @Override
            public FileVisitResult apply(Path input) {
                try {
                    if (filter.accept(input.getFileName())) {
                        pathsBlockingQueue.offer(input);
                    }
                } catch (IOException e) {
                    throw new RuntimeException(e.getMessage());
                }
                return (pathTask.isCancelled()) ? FileVisitResult.TERMINATE : FileVisitResult.CONTINUE;
            }
        };
    }

On line 6 the filter is checking for a match on the filename. If a match is found, the path object is placed in a LinkedBlockingQueue, pathsBlockingQueue, on line 7. On line 12, we see the pathTask instance variable, which is a FutureTask handle to the search thread. If pathTask has been cancelled we terminate the search, otherwise continue.

The Search Thread

To run the search, we wrap the Files.walkFileTree method in a Callable object, which is used as a constructor argument to pathTask, the FutureTask object. By using a FutureTask, we have a hook into canceling the search as well as being able to check it’s running status.

 private void findFiles(final Path startPath, final Filter filter) {
        pathTask = new FutureTask<Void>(new Callable<Void>() {
            @Override
            public Void call() throws Exception {
                Files.walkFileTree(startPath, new FunctionVisitor(getFunction(filter)));
                return null;
            }
        });
        start(pathTask);
    }

On line 5 we see the call to the getFunction method from the previous example. On line 9 the method call start is used to spin off the search in it’s own thread:

private void start(FutureTask<Void> futureTask) {
        new Thread(futureTask).start();
    }

We use a Thread object instead of an ExecutorService because we only need a single thread to execute once. Since we are not submitting any subsequent tasks, an ExecutorService really is not necessary.

Implementing A DirectoryStream

To implement a DirectoryStream there are two methods that need to be overridden – iterator and close. DirectoryStream extends the AutoCloseable interface, so when used with the try-with-resources construct, the close method is automatically invoked, releasing any underlying resources. The most interesting part of this code is overriding the iterator method:

public Iterator<Path> iterator() {
        confirmNotClosed();
        findFiles(startPath, filter);
        return new Iterator<Path>() {
            Path path;
            @Override
            public boolean hasNext() {
                try {
                    path = pathsBlockingQueue.poll();
                    while (!pathTask.isDone() && path == null) {
                        path = pathsBlockingQueue.poll(5, TimeUnit.MILLISECONDS);
                    }
                    return (path != null) ? true : false;
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
                return false;
            }

            @Override
            public Path next() {
                return path;
            }

            @Override
            public void remove() {
                throw new UnsupportedOperationException("Removal not supported");
            }
        };
    }

On line 2 we check to see if the iterator was previously closed and throw an UnsupportedOperationException if that is the case. Line 3 kicks off the search thread as we saw from the previous example. The hasNext method is where the real “brains” of the class resides. Since the search is asynchronous the DirectoryStream will attempt to iterate over the files immediately, but there needs to be some coordination with the search thread as it finds matching path objects and places them into the queue. On line 9 we first call poll (a non blocking call) and set the result to the path variable defined on line 5. If the path object is null we drop into a while loop that calls poll again, but this time it’s a blocking call with a timeout of 5 milliseconds. We’ll stay in that loop until a non-null path is returned or the search thread has completed or cancelled. On line 13 we check for null to determine if we have a valid result or if there are no more path objects to return. If hasNext returned true, the next method returns the path object that was retrieved by the previous hasNext call. This process will continue until all results have been returned.

Conclusion

Putting this all together we now have the AsynchronousRecursiveDirectoryStream class. By combining our new class with DirUtils we can iterate over an entire directory tree:

try(DirectoryStream<Path> directoryStream = DirUtils.glob(path,"*")){
   for(Path path : directoryStream){
      ....
    }
}

It is hoped that the reader found this useful. As always, comments and suggestions are welcomed. Thanks for your time.

Resources

• source code and unit test for material covered
• Java 7 java.nio.file api
]]> http://codingjunkie.net/globbing-directories-in-java/feed/ 4