BloomFilter Crash Course

BloomFilters are essentially bit vectors. At a high level BloomFilters work in the following manner:

1. Add the element to the filter.
2. Hash it a few times, than set the bits to 1 where the index matches the results of the hash.

When testing if an element is in the set, you follow the same hashing procedure and check if the bits are set to 1 or 0. This process is how a BloomFilter can guarantee an element does not exist. If the bits aren’t set, it’s simply impossible for the element to be in the set. However, a positive answer means the element is in the set or a hashing collision occurred. A more detaild description of a BloomFilter can be found here and a good tutorial on BloomFilters here. According to wikipedia, Google uses BloomFilters in BigTable to avoid disk lookups for non-existent items. Another interesting usage is using a BloomFilter to optimize a sql querry.

Using the Guava BloomFilter

A Guava BloomFilter is created by calling the static method create on the BloomFilter class,
passing in a Funnel object and an int representing the expected number of insertions. A Funnel, also new in Guava 11, is an object that can send data into a Sink. The following example is the default implementation and has a percentage of false positives of 3%. Guava provides a Funnels class containing two static methods providing implementations of the Funnel interface for inserting a CharSequence or byte Array into a filter.

//Creating the BloomFilter
BloomFilter bloomFilter = BloomFilter.create(Funnels.byteArrayFunnel(), 1000);

//Putting elements into the filter
//A BigInteger representing a key of some sort
bloomFilter.put(bigInteger.toByteArray());

//Testing for element in set
boolean mayBeContained = bloomFilter.mayContain(bitIntegerII.toByteArray());

UPDATE: based on the comment from Louis Wasserman, here’s how to create a BloomFilter for BigIntegers with a custom Funnel implementation:

//Create the custom filter
class BigIntegerFunnel implements Funnel<BigInteger> {
        @Override
        public void funnel(BigInteger from, Sink into) {
            into.putBytes(from.toByteArray());
        }
    }

//Creating the BloomFilter
BloomFilter bloomFilter = BloomFilter.create(new BigIntegerFunnel(), 1000);

//Putting elements into the filter
//A BigInteger representing a key of some sort
bloomFilter.put(bigInteger);

//Testing for element in set
boolean mayBeContained = bloomFilter.mayContain(bitIntegerII);

Considerations

It’s critical to estimate the number of expected insertions correctly. As insertions into the filter approach or exceeds the expected number, the BloomFilter begins to fill up and as a result will generate more false positives to the point of being useless. There is another version of the BloomFilter.create method taking an additional parameter, a double representing the desired level of false hit probability (must be greater than 0 and less than one). The level of false hit probability affects the number of hashes for storing or searching for elements. The lower the desired percentage, the higher number of hashes performed.

Conclusion

A BloomFilter is a useful item for a developer to have in his/her toolbox. The Guava project now makes it very simple to begin using a BloomFilter when the need arises. I hope you enjoyed this post. Helpful comments and suggestions are welcomed.

References

• Unit Test Demo of Guava BloomFilter.
• BloomFilter class
• All You Want to Know about BloomFilters.
• BloomFilter Tutorial.
• BloomFilter on Wikipedia.

Why Use the EventBus

There are two main reasons for using the EventBus with a WatchService.

1. We don’t want poll for events, but would rather receive asynchronous notification.
2. Once events are processed, the WatchKey.reset method needs to be called to enable any new changes to be queued. While the WatchKey object is thread safe, it’s important that the reset method is called only after all threads have finished processing events, leading to somewhat of a coordination hassle. Using a single thread to process the events, invoke the reset method, then publish the changes via the EventBus, eliminates this problem.

Our plan to accomplish this is simple and will involve taking the following steps:

1. Instantiate an instance of the WatchService.
2. Register every directory recursively, starting with a given Path object.
3. Take events off the WatchService queue, then process and publish those events.
4. Start up a separate thread for taking events off the queue and publishing.

The code examples that follow are the more relevant highlights from the DirectoryEventWatcherImpl class that is going to do all of this work.

Registering Directories with the WatchService

While adding or deleting a sub-directory will generate an event, any changes inside a sub-directory of a watched directory will not. We are going to compensate for this by recursively going through all sub-directories (via the Files.walkFileTree method) and register each one with the WatchService object (previously defined in the example here):

private void registerDirectories() throws IOException {
        Files.walkFileTree(startPath, new WatchServiceRegisteringVisitor());
}

private class WatchServiceRegisteringVisitor extends SimpleFileVisitor<Path>{
    @Override
    public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
         dir.register(watchService,ENTRY_CREATE,ENTRY_DELETE,ENTRY_MODIFY);
         return FileVisitResult.CONTINUE;
    }
}

On line 2 the Files.walkFileTree method uses the WatchServiceRegisteringVisitor class defined on line 5 to register every directory with the WatchService. The registered events are creation of files/directories, deletion of files/directories or updates to a file.

Publishing Events

The next step is to create a FutureTask that will do the work of checking the queue and publishing the events.

 private void createWatchTask() {
    watchTask = new FutureTask<>(new Callable<Integer>() {
       private int totalEventCount;
       @Override
       public Integer call() throws Exception {
           while (keepWatching) {
               WatchKey watchKey = watchService.poll(10, TimeUnit.SECONDS);
               if (watchKey != null) {
                  List<WatchEvent<?>> events = watchKey.pollEvents();
                  Path watched = (Path) watchKey.watchable();
                  PathEvents pathEvents = new PathEvents(watchKey.isValid(), watched);
                  for (WatchEvent event : events) {
                        pathEvents.add(new PathEvent((Path) event.context(), event.kind()));
                        totalEventCount++;
                  }
                  watchKey.reset();
                  eventBus.post(pathEvents);
                }
          }
           return totalEventCount;
        }
      });
    }

private void startWatching() {
  new Thread(watchTask).start();
}

On line 7, we are checking the WatchService every 10 seconds for queued events. When a valid WatchKey is returned, the first step is to retrieve the events (line 9) then get the directory where the events occurred (line 10). On line 11 a PathEvents object is created, taking a boolean and the watched directory as constructor arguments. Lines 12-15 are looping over the events retrieved on line 9, using the target Path and event type as arguments to create PathEvent object. The WatchKey.reset method is called on line 16, setting the WatchKey state back to ready, making it eligible to receive new events and be placed back into the queue. Finally on line 17 the EventBus publishes the PathEvents object to all subscribers. It’s important to note here that the PathEvents and PathEvent classes are immutable. The totalEventCount that is returned from the Callable is never exposed in the API, but is used for testing purposes. The startWatching method on line 25 starts the thread to run the watching/publishing task defined above.

Conclusion

By pairing the WatchService with the Guava EventBus we are able to manage the WatchKey and process events in a single thread and notify any number of subscribers asynchronously of the events. It is hoped the reader found this example useful. As always comments and suggestions are welcomed.

Resources

• Source code and unit test for this post
• EventBus API
• WatchService API
• Previous post on the WatchService.
• Previous post on the EventBus

Requirements

Here are the requirements:

1. Starting from a given Path object, it should recursively go through each directory looking for files that match a given pattern.
2. It needs to be a single method call, something like DirUtils.glob(Path path, String pattern).
3. We want the search to be asynchronous, so we can iterate over files as they are found.

The plan to meet our objective is straight forward:

1. Create a DirectoryStream.Filter object from the given pattern.
2. Use the Files.walkFileTree method to recursively search for files that match our pattern.
3. Run the search in a separate thread, and when a matching file is found, place it in a queue.
4. The DirectoryStream iterator will pull the files out of the queue as they become available

Disclosure: This post is merely an attempt to see if making an asynchronous, recursive DirectorySteam will work, I make no guarantees about the code presented here, YMMV.

The Filter

DirectoryStream has a static interface, Filter, that is used to determine if the path object should be accepted. The filter object is constructed by first creating a PathMatcher object then using the PathMatcher in the Filter’s accept method:

public static DirectoryStream.Filter<Path> buildGlobFilter(String pattern) {
        final PathMatcher pathMatcher = getPathMatcher("glob:"+pattern);
        return new DirectoryStream.Filter<Path>() {
            @Override
            public boolean accept(Path entry) throws IOException {
                return pathMatcher.matches(entry);
            }
        };
    }

The Recursive Search

To perform the search we are going the use the Files.walkFileTree method with a FunctionVisitor object. FunctionVisitor is a subclass of SimpleFileVisitor that takes a Guava Function as a constructor argument. The provided function is called as each file is visited and is checked by the DirectoryStream.Filter object for a match. We wrap the creation of the Function object in a method call:

 private Function<Path, FileVisitResult> getFunction(final Filter<Path> filter) {
        return new Function<Path, FileVisitResult>() {
            @Override
            public FileVisitResult apply(Path input) {
                try {
                    if (filter.accept(input.getFileName())) {
                        pathsBlockingQueue.offer(input);
                    }
                } catch (IOException e) {
                    throw new RuntimeException(e.getMessage());
                }
                return (pathTask.isCancelled()) ? FileVisitResult.TERMINATE : FileVisitResult.CONTINUE;
            }
        };
    }

On line 6 the filter is checking for a match on the filename. If a match is found, the path object is placed in a LinkedBlockingQueue, pathsBlockingQueue, on line 7. On line 12, we see the pathTask instance variable, which is a FutureTask handle to the search thread. If pathTask has been cancelled we terminate the search, otherwise continue.

The Search Thread

To run the search, we wrap the Files.walkFileTree method in a Callable object, which is used as a constructor argument to pathTask, the FutureTask object. By using a FutureTask, we have a hook into canceling the search as well as being able to check it’s running status.

 private void findFiles(final Path startPath, final Filter filter) {
        pathTask = new FutureTask<Void>(new Callable<Void>() {
            @Override
            public Void call() throws Exception {
                Files.walkFileTree(startPath, new FunctionVisitor(getFunction(filter)));
                return null;
            }
        });
        start(pathTask);
    }

On line 5 we see the call to the getFunction method from the previous example. On line 9 the method call start is used to spin off the search in it’s own thread:

private void start(FutureTask<Void> futureTask) {
        new Thread(futureTask).start();
    }

We use a Thread object instead of an ExecutorService because we only need a single thread to execute once. Since we are not submitting any subsequent tasks, an ExecutorService really is not necessary.

Implementing A DirectoryStream

To implement a DirectoryStream there are two methods that need to be overridden – iterator and close. DirectoryStream extends the AutoCloseable interface, so when used with the try-with-resources construct, the close method is automatically invoked, releasing any underlying resources. The most interesting part of this code is overriding the iterator method:

public Iterator<Path> iterator() {
        confirmNotClosed();
        findFiles(startPath, filter);
        return new Iterator<Path>() {
            Path path;
            @Override
            public boolean hasNext() {
                try {
                    path = pathsBlockingQueue.poll();
                    while (!pathTask.isDone() && path == null) {
                        path = pathsBlockingQueue.poll(5, TimeUnit.MILLISECONDS);
                    }
                    return (path != null) ? true : false;
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
                return false;
            }

            @Override
            public Path next() {
                return path;
            }

            @Override
            public void remove() {
                throw new UnsupportedOperationException("Removal not supported");
            }
        };
    }

On line 2 we check to see if the iterator was previously closed and throw an UnsupportedOperationException if that is the case. Line 3 kicks off the search thread as we saw from the previous example. The hasNext method is where the real “brains” of the class resides. Since the search is asynchronous the DirectoryStream will attempt to iterate over the files immediately, but there needs to be some coordination with the search thread as it finds matching path objects and places them into the queue. On line 9 we first call poll (a non blocking call) and set the result to the path variable defined on line 5. If the path object is null we drop into a while loop that calls poll again, but this time it’s a blocking call with a timeout of 5 milliseconds. We’ll stay in that loop until a non-null path is returned or the search thread has completed or cancelled. On line 13 we check for null to determine if we have a valid result or if there are no more path objects to return. If hasNext returned true, the next method returns the path object that was retrieved by the previous hasNext call. This process will continue until all results have been returned.

Conclusion

Putting this all together we now have the AsynchronousRecursiveDirectoryStream class. By combining our new class with DirUtils we can iterate over an entire directory tree:

try(DirectoryStream<Path> directoryStream = DirUtils.glob(path,"*")){
   for(Path path : directoryStream){
      ....
    }
}

It is hoped that the reader found this useful. As always, comments and suggestions are welcomed. Thanks for your time.

Resources

• source code and unit test for material covered
• Java 7 java.nio.file api
]]> http://codingjunkie.net/globbing-directories-in-java/feed/ 4 http://codingjunkie.net/java-7-copy-move/ http://codingjunkie.net/java-7-copy-move/#comments Mon, 30 Jan 2012 05:10:34 +0000 Bill B http://codingjunkie.net/?p=1673 This post is a continuation of my series on the Java 7 java.nio.file package, this time covering the copying and moving of files and complete directory trees. If you have ever been frustrated by Java’s lack of copy and move methods, then read on, for relief is at hand. Included in the coverage is the very useful Files.walkFileTree method. Before we dive into the main content however, some background information is needed.

Path Objects

Path objects represent a sequence of directories that may or may not include a file. There are three ways to construct a Path object:

1. FileSystems.getDefault().getPath(String first, String… more)
2. Paths.get(String path, String… more), convenience method that calls FileSystems.getDefault().getPath
3. Calling the toPath method on a java.io.File object

From this point forward in all our examples, we will use the Paths.get method. Here are some examples of creating Path objects:

//Path string would be "/foo"
Paths.get("/foo");
//Path string "/foo/bar"
Paths.get("/foo","bar");

To manipulate Path objects there are the Path.resolve and Path.relativize methods. Here is an example of using Path.resolve:

//This is our base path "/foo"
Path base = Paths.get("/foo");
//filePath is "/foo/bar/file.txt" while base still "/foo"
Path filePath = base.resolve("bar/file.txt");

Using the Path.resolve method will append the given String or Path object to the end of the calling Path, unless the given String or Path represents an absolute path, the the given path is returned, for example:

Path path = Paths.get("/foo");
//resolved Path string is "/usr/local"
Path resolved = path.resolve("/usr/local");

The Path.relativize works in the opposite fashion, returning a new relative path that if resolved against the calling Path would result in the same Path string. Here’s an example:

        // base Path string "/usr"
        Path base = Paths.get("/usr");
        // foo Path string "/usr/foo"
        Path foo = base.resolve("foo");
        // bar Path string "/usr/foo/bar"
        Path bar = foo.resolve("bar");
        // relative Path string "foo/bar"
        Path relative = base.relativize(bar);

Another method on the Path class that is helpful is the Path.getFileName, that returns the name of the farthest element represented by this Path object, with the name being an actual file or just a directory. For example:

//assume filePath constructed elsewhere as "/home/user/info.txt"
//returns Path with path string "info.txt"
filePath.getFileName()

//now assume dirPath constructed elsewhere as "/home/user/Downloads"
//returns Path with path string "Downloads"
dirPath.getFileName()

In the next section we are going to take a look at how we can use Path.resolve and Path.relativize in conjunction with Files class for copying and moving files.

Files Class

The Files class consists of static methods that use Path objects to work with files and directories. While there are over 50 methods in the Files class, at this point we are only going to discuss the copy and move methods.

Copy A File

To copy one file to another you would use the (any guesses on the name?) Files.copy method – copy(Path source, Path target, CopyOption… options) very concise and no anonymous inner classes, are we sure it’s Java?. The options argument are enums that specify how the file should be copied. (There are actually 2 different Enum classes, LinkOption and StandardCopyOption, but both implement the CopyOption interface) Here is the list of available options for Files.copy:

1. LinkOption.NOFOLLOW_LINKS
2. StandardCopyOption.COPY_ATTRIBUTES
3. StandardCopyOption.REPLACE_EXISTING

There is also a StandardCopyOption.ATOMIC_MOVE enum, but if this option is specified, an UsupportedOperationException is thrown. If no options are specified, the default is to throw an error if the target file exists or is a symbolic link. If the path object is a directory then an empty directory is created in the target location. (Wait a minute! didn’t it say in the introduction that we could copy the entire contents of a directory? The answer is still yes and that is coming!) Here’s an example of copying a file to another with Path objects using the Path.resolve and Path.relativize methods:

        Path sourcePath ...
        Path basePath ...
        Path targetPath ...

        Files.copy(sourcePath, targetPath.resolve(basePath.relativize(sourcePath));

Move A File

Moving a file is equally as straight forward – move(Path source, Path target, CopyOption… options);
The available StandardCopyOptions enums available are:

1. StandardCopyOption.REPLACE_EXISTING
2. StandardCopyOption.ATOMIC_MOVE

If Files.move is called with StandardCopyOption.COPY_ATTRIBUTES an UnsupportedOperationException is thrown. Files.move can be called on an empty directory or if it does not require moving a directories contents, re-naming for example, the call will succeed, otherwise it will throw an IOException (we’ll see in the following section how to move non-empty directories). The default is to throw an Exception if the target file already exists. If the source is a symbolic link, then the link itself is moved, not the target of the link. Here’s an example of Files.move, again tying in the Path.relativize and Path.resolve methods:

        Path sourcePath ...
        Path basePath ...
        Path targetPath ...

        Files.move(sourcePath, targetPath.resolve(basePath.relativize(sourcePath));

Copying and Moving Directories

One of the more interesting and useful methods found in the Files class is Files.walkFileTree. The walkFileTree method performs a depth first traversal of a file tree. There are two signatures:

1. walkFileTree(Path start,Set options,int maxDepth,FileVisitor visitor)
2. walkFileTree(Path start,FileVisitor visitor)

The second option for Files.walkFileTree calls the first option with EnumSet.noneOf(FileVisitOption.class) and Integer.MAX_VALUE. As of this writing, there is only one file visit option – FOLLOW_LINKS. The FileVisitor is an interface that has four methods defined:

1. preVisitDirectory(T dir, BasicFileAttributes attrs) called for a directory before all entires are traversed.
2. visitFile(T file, BasicFileAttributes attrs) called for a file in the directory.
3. postVisitDirectory(T dir, IOException exc) only called after all files and sub-directories have been traversed.
4. visitFileFailed(T file, IOException exc) called for files that could not be visited

All of the methods return one of the four possible FileVisitResult enums :

1. FileVistitResult.CONTINUE
2. FileVistitResult.SKIP_SIBLINGS (continue without traversing siblings of the directory or file)
3. FileVistitResult.SKIP_SUBTREE (continue without traversing contents of the directory)
4. FileVistitResult.TERMINATE

To make life easier there is a default implementation of the FileVisitor, SimpleFileVisitor (validates arguments are not null and returns FileVisitResult.CONTINUE), that can be subclassed co you can override just the methods you need to work with. Let’s take a look at a basic example for copying an entire directory structure.

Copying A Directory Tree Example

Let’s take a look at a class that extends SimpleFileVisitor used for copying a directory tree (some details left out for clarity):

 public class CopyDirVisitor extends SimpleFileVisitor<Path> {
    private Path fromPath;
    private Path toPath;
    private StandardCopyOption copyOption = StandardCopyOption.REPLACE_EXISTING;
    ....
    @Override
    public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
        Path targetPath = toPath.resolve(fromPath.relativize(dir));
        if(!Files.exists(targetPath)){
            Files.createDirectory(targetPath);
        }
        return FileVisitResult.CONTINUE;
    }

    @Override
    public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
        Files.copy(file, toPath.resolve(fromPath.relativize(file)), copyOption);
        return FileVisitResult.CONTINUE;
    }
}

On line 9, each directory will be created in the target, ‘toPath’, as each directory from the source, ‘fromPath’,is traversed. Here we can see the power the Path object with respect to working with directories and files. As the code moves deeper into the directory structure, the correct Path objects are constructed simply from calling relativize and resolve on the fromPath and toPath objects, respectively. At no point do we need to be aware of where we are in the directory tree, and as a result no cumbersome StringBuilder manipulations are needed to create the correct paths. On line 17, we see the Files.copy method used to copy the file from the source directory to the target directory. Next is a simple example of deleting an entire directory tree.

Deleting A Directory Tree Example

In this example SimpleFileVisitor has been subclassed for deleting a directory structure:

public class DeleteDirVisitor  extends SimpleFileVisitor<Path> {

    @Override
    public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
        Files.delete(file);
        return FileVisitResult.CONTINUE;
    }

    @Override
    public FileVisitResult postVisitDirectory(Path dir, IOException exc) throws IOException {
        if(exc == null){
            Files.delete(dir);
            return FileVisitResult.CONTINUE;
        }
        throw exc;
    }
}

As you can see, deleting is a very simple operation. Simply delete each file as you find them then delete the directory on exit.

Combining Files.walkFileTree with Google Guava

The previous two examples, although useful, were very ‘vanilla’. Let’s take a look at two more examples that are a little more creative by combining the Google Gauva Function and Predicate interfaces.

public class FunctionVisitor extends SimpleFileVisitor<Path> {
    Function<Path,FileVisitResult> pathFunction;

    public FunctionVisitor(Function<Path, FileVisitResult> pathFunction) {
        this.pathFunction = pathFunction;
    }

    @Override
    public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
        return pathFunction.apply(file);
    }
}

In this very simple example, we subclass SimpleFileVisitor to take a Function object as a constructor parameter and as the directory structure is traversed, apply the function to each file.

public class CopyPredicateVisitor extends SimpleFileVisitor<Path> {
    private Path fromPath;
    private Path toPath;
    private Predicate<Path> copyPredicate;

    public CopyPredicateVisitor(Path fromPath, Path toPath, Predicate<Path> copyPredicate) {
        this.fromPath = fromPath;
        this.toPath = toPath;
        this.copyPredicate = copyPredicate;
    }

    @Override
    public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
        if (copyPredicate.apply(dir)) {
            Path targetPath = toPath.resolve(fromPath.relativize(dir));
            if (!Files.exists(targetPath)) {
                Files.createDirectory(targetPath);
            }
            return FileVisitResult.CONTINUE;
        }
        return FileVisitResult.SKIP_SUBTREE;
    }

    @Override
    public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
        Files.copy(file, toPath.resolve(fromPath.relativize(file)));
        return FileVisitResult.CONTINUE;
    }
}

In this example the CopyPredicateVisitor takes a Predicate object and based on the returned boolean, parts of the directory structure are not copied. I would like to point out that the previous two examples, usefulness aside, do work in the unit tests foe the source code provided with this post.

DirUtils

Building on everything we’ve covered so far, I could not resist the opportunity to create a utility class, DirUtils, as an abstraction for working with directories that provides the following methods:

 //deletes all files but leaves the directory tree in place
 DirUtils.clean(Path sourcePath);
 //completely removes a directory tree
 DirUtils.delete(Path sourcePath);
 //replicates a directory tree
 DirUtils.copy(Path sourcePath, Path targetPath);
 //not a true move but performs a copy then a delete of a directory tree
 DirUtils.move(Path sourcePath, Path targetPath);
 //apply the function to all files visited
 DirUtils.apply(Path sourcePath,Path targetPath, Function<Path> function);

While I wouldn’t go as far to say it’s production ready, it was fun to write.

Conclusion

That wraps up the new copy and move functionality provided by the java.nio.file package. I personally think it’s very useful and will take much of the pain out of working with files in Java. There’s much more to cover, working with symbolic links, stream copy methods, DirectoryStreams etc, so be sure to stick around. Thanks for your time. As always comments and suggestions are welcomed.

Resouces

• Java 7 java.nio.file api
• Source code and tests for the post
]]> http://codingjunkie.net/java-7-copy-move/feed/ 2 http://codingjunkie.net/java7-file-revolution/ http://codingjunkie.net/java7-file-revolution/#comments Fri, 20 Jan 2012 06:32:43 +0000 Bill B http://codingjunkie.net/?p=1700 Java 7 (“Project Coin”) has been out since July of last year. The additions with this release are useful, for example Try with resources – having closable resources handled automatically from try blocks, Strings in switch statements, multicatch for Exceptions and the ‘<>‘ operator for working with generics. The addition that everyone was anticipating the most, closures, has been deferred to version 8. Surprisingly though, there was a small ‘revolution’ of sorts with the release of Java 7, that for the most part, in my opinion, went unnoticed and could possibly be the best part of the Java 7 release. The change I’m referring to is the addition of the java.nio.file package. The java.nio.file package added classes and interfaces that make working with files and directories in Java much easier. First and formost of these changes is the ability to copy or move files. I always found it frustrating that if you want to copy or move a file, you have to roll your own version of ‘copy’ or ‘move’. The utilities found in the Guava project com.google.common.io package provides these capabilities, but I feel that copy and move operations should be a core part of the language. Over the next few posts, I’ll be going into greater detail (with code examples) on the classes/interfaces discussed here and some others that have not been covered. This post serves as an introduction and overview of the new functionality in the java.nio.file package.

Breaking Out Responsibilities

If you take a look at the java.io package as it stands now, the vast majority of the classes are for input streams, output streams, readers or writers . There is only one class that defines operations for directly working with the file system, the File class. Some of the other classes in java.io will take a File object as an argument in a constructor, but all file and directory interaction is through the File class. In the java.nio.file package, functionality has been teased out into other classes/interfaces. The first ones to discuss are the Path interface and the Files class.

Path and Files

A Path object is some what analogous to a java.io.File object as it can represent a file or directory on the file system. A Path object is more abstract though, in that it is a sequence of names that represent a directory hierarchy (that may or may not include a file) on the file system. There are not methods in the Path interface that allow for working with directories or files. The methods defined are for working with or manipulating Path objects only, resolving one Path to another etc. (There is one method that can be used to obtain a java.io.File object from a Path, toFile. Likewise the java.io.File class now contains a toPath method.) To work with files and directories, Path objects are used in conjunction with the Files class. The Files class consists entirely of static methods for manipulating directories and files, including copy, move and functions for working with symbolic links. Another interesting method in the Files class is the newDirectoryStream method that returns a DirectoryStream object that can iterate over all the entries in a directory. Although the java.io.File class has the list method where you provide a FileFilter instance, the newDirectoryStream can take a String ‘glob’ like ‘*.txt’ to filter on.

FileStore

As previously mentioned, all interaction with the file system in the java.io package is through the File class. This includes getting information on used or available space in the file system. In java.nio.file there is a FileStore class that represents the storage type for the files whether it’s a device, partition or concreate file system. The FileStore class defines methods for getting information about the file storage such as getTotalSpace, getUsableSpace, getUnallocated space. A FileStore can be obtained by calling the Files.getFileStore(Path path) method that will return the FileStore for that particular file.

FileSystem and FileSystems

A FileSystem, as the name implies, provides access to the file system and is a factory for other objects in the file system. For example, the FileSystem class defines a getPath method that converts a string (/foo/bar) into a system dependent Path object that can be used for accessing a file or directory. The FileSystem class also provides a getFileStores method that returns an iterable of all FileStores in the FileSystem. The FileSystems class provides access to the FileSystem object with the static FactorySystems.getDefault method. There are also static methods for creating custom FileSystem objects.

Conclusion

This has been a fast, high level view of the new functionality for working with files provided by the java.nio.file package. There is much more information that has not been covered here, so take a look at the api docs Hopefully this post has been able get the reader interested in the improved file handling in Java 7. Be sure to stick around as we begin to explore in more detail what the java.nio.file package has to offer.

]]> http://codingjunkie.net/java7-file-revolution/feed/ 16 http://codingjunkie.net/guava-eventbus/ http://codingjunkie.net/guava-eventbus/#comments Fri, 06 Jan 2012 07:41:19 +0000 Bill B http://codingjunkie.net/?p=1597 It’s a given in any software application there are objects that need to share information in order to get work done. In Java applications, one way of achieving information sharing is to have event listeners, whose sole purpose is to take some action when a desired event occurs. For the most part this process works and most experienced Java developers are used to writing the requisite anonymous inner class that implements some event listener interface. This post is about taking a different approach to handling Java events using Guava’s EventBus. The EventBus allows for objects to subscribe for or publish events, without having explicit knowledge of each other. The EventBus is not meant to be a general purpose publish/subscribe system or support interprocess communication.

EventBus Class

The EventBus is very flexible and can be used as a singleton, or an application can have several instances to accomodate transferring events in different contexts. The EventBus will dispatch all events serially, so it is important to keep the event handling methods lightweight. If you need to do heavier processing in the event handlers, there is another flavor of the EventBus, AsyncEventBus. The AsyncEventBus is identical in functionality, but takes an ExecutorService as a constructor argument to allow for asynchronous dispatching of events.

Subscribing for Events

An object subscribes for events by taking the following steps:

1. Define a public method that takes a single argument of the desired event type and place a @Subscribe annotation on that method.
2. Register with the EventBus by passing an instance of the object to the EventBus.register method.

Here is a brief example with details omitted for clarity:

 public class PurchaseSubscriber {
    @Subscribe
    public void handlePurchaseEvent(PurchaseEvent event) {
       .....
    }
   .....

  EventBus eventBus = new EventBus();
  PurchaseSubscriber purchaseSubscriber = new PurchaseSubscriber();
  eventBus.register(purchaseSubscriber);

There is another annotation that can be used in conjunction with @Subscribe, and that is @AllowConcurrentEvents. The @AllowConcurrentEvents marks a handler method as thread safe so an EventBus (most likely an AsyncEventBus) could potentially call the event handler from simultaneous threads. One interesting thing I discovered in unit testing is that if a handler method does not have the @AllowConcurrentEvents annotation, it will invoke handlers for an event serially even when using the AsyncEventBus. It’s important to note that @AllowConcurrentEvents will not mark a method as an event handler, the @Subscribe annotation still needs to be present. Finally, the event handling method must have one and only one parameter or an IllegalArgumentException will be thrown when you register the object with the EventBus.

Publishing Events

Publishing events with the EventBus is equally straight forward. In the section of code where you want to send the notification of an event, call EventBus.post and all subscribers registered for that event object will be notified.

  public void handleTransaction(){
    purchaseService.purchase(item,amount);
    eventBus.post(new CashPurchaseEvent(item,amount));
    ....
  }

While it might be obvious, it’s important that the subscribing and publishing classes share the same EventBus instance, and it would make sense to use Guice or Spring to help manage the dependencies.

More About Event Handlers

A very powerful feature of the EventBus is that you can make your handlers as course or fine grained as needed. The EventBus will call the registered subscribers for all subtypes and implemented interfaces of a posted event object. For example, to handle any and all events one could create an event handler that takes a parameter of type Object. To handle only singular events, create a handler that is very type specific. To help illustrate, consider the following simple event hierarchy:

public abstract class PurchaseEvent {
        String item;
   public PurchaseEvent(String item){
       this.item = item;
   }
}

public class CashPurchaseEvent extends PurchaseEvent {
       int amount;
       public CashPurchaseEvent(String item, int amount){
          super(item);
          this.amount = amount;
       }
}

public class CreditPurchaseEvent extends PurchaseEvent {
       int amount;
       String cardNumber;
       public CreditPurchaseEvent(String item,int amount, String cardNumber){
          super(item);
          this.amount = amount;
          this.cardNumber = cardNumber;
       }
}

Here are the corresponding event handling classes:

//Would only be notified of Cash purchase events
 public class CashPurchaseSubscriber {
     @Subscribe
     public void handleCashPurchase(CashPurchaseEvent event){
      ...
     }
 }
 //Would only be notified of credit purchases
 public class CreditPurchaseSubscriber {
    @Subscribe
    public void handleCreditPurchase(CreditPurchaseEvent event) {
     ....
    }
}
//Notified of any purchase event
 public class PurchaseSubscriber {
    @Subscribe
    public void handlePurchaseEvent(PurchaseEvent event) {
       .....
    }
}

If there is a need to capture a broad range of event types, an alternative would be to have more than one event handling method in a class. Having multiple handlers could be a better solution as you would not have to do any “instanceof” checks on the event object parameter. Here’s a simple example from my unit test:

public class MultiHandlerSubscriber {

    List<CashPurchaseEvent> cashEvents = new ArrayList<ashPurchaseEvent>();
    List<CreditPurchaseEvent> creditEvents = new ArrayList<CreditPurchaseEvent>();
    List<SimpleEvent> simpleEvents = new ArrayList<SimpleEvent>();

    public MultiHandlerSubscriber(EventBus eventBus){
        eventBus.register(this);
    }

    @Subscribe
    public void handleCashEvents(CashPurchaseEvent event){
        cashEvents.add(event);
    }

    @Subscribe
    public void handleCreditEvents(CreditPurchaseEvent event){
        creditEvents.add(event);
    }

    @Subscribe
    public void handleSimpleEvents(SimpleEvent event){
        simpleEvents.add(event);
    }
....

Testing

Since the event handlers are just plain methods, they can easily be tested by instantiating an EventBus in the test case or simulate the EventBus by passing the appropriate event object. When working with the EventBus I found it was very easy to:

• Forget to register the subscribing object with the EventBus
• Neglect to add the @Subscribe annotation.

If it seems that an event handler is not being called, check for those two errors first.
A useful debugging technique is to subscribe to the DeadEvent class. An EventBus will wrap any published event with no handlers in a DeadEvent instance. DeadEvent provides the getEvent method that returns the original event object.

Conclusion

The Guava EventBus class provides an attractive and useful alternative to the standard Java event handling mechanism. It is hoped the reader will find the EventBus as useful as I do. As always comments and suggestions are welcomed.

Resources

• EventBus API
• Guava API
• Sample Code
• Event Collaboration
]]> http://codingjunkie.net/guava-eventbus/feed/ 15 http://codingjunkie.net/guava-functions-java-8-lambdas/ http://codingjunkie.net/guava-functions-java-8-lambdas/#comments Fri, 23 Dec 2011 05:48:58 +0000 Bill B http://codingjunkie.net/?p=1512 I recently read Brian Goetz’s The State of the Lambda and after reading that article I wanted to try using Java 8 lambda expressions. In his article, Brian goes on to describe interfaces that have one method as “functional” interfaces. Functional interfaces are almost always used as anonymous classes, with the ActionListener being the canonical example. These “functional” interfaces are the initial target for lambda expressions, with the main goal being removing much of the boilerplate or ceremony around their usage. As I have been writing a series on the Guava library, the Function interface immediately came to mind as a prime candiate for trying lambda expressions. My goal is simple, take the unit test from my Guava Futures blog (it makes heavy use of the Function interface) and convert it to use lambda expressions. While I will describe the structure of the lamba expression as it pertains to the example presented, this post is not a tutorial on Java 8 lambda expressions. Rather it’s documenting my first attempts at using lambda expressions in Java.

Lambda Expressions

The first example is a test for the Futures.chain method, which takes a Function as one of it’s arguments:

Function<List<String>, ListenableFuture<List<Person>>> queryFunction =
      new Function<List<String>, ListenableFuture<List<Person>>>() {
            @Override
            public ListenableFuture<List<Person>> apply(final List<String> ids) {
                 return dbService.getPersonsByIdAsync(ids);
            }
        };

ListenableFuture<List<String>> indexSearch =
      luceneSearcher.searchAsync("firstName:martin");
ListenableFuture<List<Person>> results =
      Futures.chain(indexSearch,queryFunction,executorService);

Using lamba expressions it would now look like:

Function<List<String>, ListenableFuture<List<Person>>> queryFunction =
      ids ->(dbService.getPersonsByIdAsync(ids));

  ListenableFuture<List<String>> indexSearch =
      luceneSearcher.searchAsync("firstName:martin");
  ListenableFuture<List<Person>>results =
      Futures.chain(indexSearch, queryFunction,executorService);

Keep in mind that from the code examples above the two highlighted sections are equivalent. Let’s take a look at line 2 and explain how this matches up with lines 1-7 from the first code example

1. ids is the input to the apply method and corresponds to the final List<String> ids parameter on line 4 in the first code example. The types of the lamba expression are being inferred from the context in which it is being used, so we don’t need to repeat them for the ids parameter.
2. Then there is the arrow (->) token which is part of the general syntax of Java 8 lambda expressions in their current form
3. Then we have the body of the lambda (dbService.getPersonsByIdAsync(ids)) which is a method call that returns a ListenableFuture that in turn yields a List of Person objects. Note that we did not have to put a return statement, as this is a single expression and is evaluated and returned.

The next example is a utility method from the test that returned ListenableFutures by passing anonymous Callable instances into an ExecutorService:

private ListenableFuture<List<Person>> getPersonsByFirstNameFuture(final String firstName, final boolean error) {
return executorService.submit(new Callable<List<Person>>() {
            @Override
            public List<Person> call() throws Exception {
                startSignal.await();
                if (error) {
                    throw new RuntimeException("Ooops!");
                }
                List<String> ids = luceneSearcher.search("firstName:" + firstName);
                return dbService.getPersonsById(ids);
            }
        });
}

And here is the equivalent using a lambda expression:

private ListenableFuture<List<Person>> getPersonsByFirstNameFuture(final String firstName, final boolean error) {
 return executorService.submit(() -> {startSignal.await();
                                       if (error) {
                                        throw new RuntimeException("Ooops!");
                                       }
                                     List<String> ids = luceneSearcher.search("firstName:" + firstName);
                                     return dbService.getPersonsById(ids);
                                  });
}

In this example there are no input arguments so the expression starts with empty parentheses () on line 2. There is the -> token, but in this example, the body contains multiple statements surrounded by { …}. Since there are multiple statements an explicit return statement is needed on line 7.

Environment to Run Java 8

My current laptop is a MacBook Pro, so I needed to set up an environment to run Java 8 with lambda support. Here are the steps I took:

1. Installed LinuxMint 12 on VirtualBox.
2. Created a directory and shared it with the LinuxMint guest
3. Installed the developer preview of java 8.
4. To get the source and test source from the existing maven project I ran mvn jar:jar jar:test-jar and put the resulting jar files in the shared directory
5. Placed all the dependencies in the shared directory (guava,lucene,h2 and junit)
6. Re-wrote the unit test to use lambdas and ran the new test from the command line

Conclusion

Although it will be some time before Java 8 with lambda support is released, what is available in the developer preview looks promising.Thanks for your time and as always comments and suggestions are welcomed

Resources

• State of the Lambda, Part 4
• Java 8 developer preview
• Source code for this post
• Guava Project Home
• Source Code for Guava blog series
]]> http://codingjunkie.net/guava-functions-java-8-lambdas/feed/ 8 http://codingjunkie.net/google-guava-cache/ http://codingjunkie.net/google-guava-cache/#comments Wed, 14 Dec 2011 05:44:37 +0000 Bill B http://codingjunkie.net/?p=1438 This Post is a continuation of my series on Google Guava, this time covering Guava Cache. Guava Cache offers more flexibility and power than either a HashMap or ConcurrentHashMap, but is not as heavy as using EHCache or Memcached (or robust for that matter, as Guava Cache operates solely in memory). The Cache interface has methods you would expect to see like ‘get’, and ‘invalidate’. A method you won’t find is ‘put’, because Guava Cache is ‘self-populating’, values that aren’t present when requested are fetched or calculated, then stored. This means a ‘get’ call will never return null. In all fairness, the previous statement is not %100 accurate. There is another method ‘asMap’ that exposes the entries in the cache as a thread safe map. Using ‘asMap’ will result in not having any of the self loading operations performed, so calls to ‘get’ will return null if the value is not present (What fun is that?). Although this is a post about Guava Cache, I am going to spend the bulk of the time talking about CacheLoader and CacheBuilder. CacheLoader specifies how to load values, and CacheBuilder is used to set the desired features and actually build the cache.

CacheLoader

CacheLoader is an abstract class that specifies how to calculate or load values, if not present. There are two ways to create an instance of a CacheLoader:

1. Extend the CacheLoader<K,V> class
2. Use the static factory method CacheLoader.from

If you extend CacheLoader you need to override the V load(K key) method, instructing how to generate the value for a given key. Using the static CacheLoader.from method you build a CacheLoader either by supplying a Function or Supplier interface. When supplying a Function object, the Function is applied to the key to calculate or retrieve the results. Using a Supplier interface the value is obtained independent of the key.

CacheBuilder

The CacheBuilder is used to construct cache instances. It uses the fluent style of building and gives you the option of setting the following properties on the cache:

• Cache Size limit (removals use a LRU algorithm)
• Wrapping keys in WeakReferences (Strong references used by default for keys)
• Wrapping values in either WeakReferences or SoftReferences (Strong references used by default)
• Time to expire entires after last access
• Time based expiration of entries after being written or updated
• Setting a RemovalListener that can recieve events once an entry is removed from the cache
• Concurrency Level of the cache (defaults to 4)

The concurrency level option is used to partition the table internally such that updates can occur without contention. The ideal setting would be the maximum number of threads that could potentially access the cache at one time. Here is an example of a possible usage scenario for Guava Cache.

public class PersonSearchServiceImpl implements SearchService<List<Person>> {

public PersonSearchServiceImpl(SampleLuceneSearcher luceneSearcher, SampleDBService dbService) {
        this.luceneSearcher = luceneSearcher;
        this.dbService = dbService;
        buildCache();
    }

    @Override
    public List<Person> search(String query) throws Exception {
        return cache.get(query);
    }

    private void buildCache() {
        cache = CacheBuilder.newBuilder().expireAfterWrite(10, TimeUnit.MINUTES)
                .maximumSize(1000)
                .build(new CacheLoader<String, List<Person>>() {
                    @Override
                    public List<Person> load(String queryKey) throws Exception {
                        List<String> ids = luceneSearcher.search(queryKey);
                        return dbService.getPersonsById(ids);
                    }
                });
    }
}

In this example, I am setting the cache entries to expire after 10 minutes of being written or updated in the cache, with a maximum amount of 1,000 entires. Note the usage of CacheLoader on line 15.

RemovalListener

The RemovalListener will receive notification of an item being removed from the cache. These notifications could be from manual invalidations or from a automatic one due to time expiration or garbage collection. The RemovalListener<K,V> parameters can be set to listen for specific type. To receive notifications for any key or value set them to use Object. It should be noted here that a RemovalListener will receive a RemovalNotification<K,V> object that implements the Map.Entry interface. The key or value could be null if either has already been garbage collected. Also the key and value object will be strong references, regardless of the type of references used by the cache.

CacheStats

There is also a very useful class CacheStats that can be retrieved via a call to Cache.stats(). The CacheStats object can give
insight into the effectiveness and performance of your cache by providing statistics such as:

• hit count
• miss count
• total load timme
• total requests

CacheStats provides many other counts in addition to the ones listed above.

Conclusion

The Guava Cache presents some very compelling functionality. The decision to use a Guava Cache really comes down to the tradeoff between memory availability/usage versus increases in performance. I have added a unit test CacheTest demonstrating the usages discussed here. As alway comments and suggestions are welcomed. Thanks for your time.

Resources

• Guava Project Home
• Cache API
• Source Code for blog series
]]> http://codingjunkie.net/google-guava-cache/feed/ 9 http://codingjunkie.net/simple-wordpress-backups/ http://codingjunkie.net/simple-wordpress-backups/#comments Wed, 07 Dec 2011 04:36:16 +0000 Bill B http://codingjunkie.net/?p=1404 Backing up your data is an important task. As we all know, it’s not a matter of if you are going to experience a crash or failure, but when. Blogs are no exception. I wanted to take a break from my regular style of posts to share my simple backup script. While I’m know there ar more sophisticated approaches, I really enjoy automating tasks, like remote backups, with bash scripts. My approach is simple. I create a gzipped tar of my WordPress install directory, and then run a mysqldump command. It’s all done locally from my MacBook Pro via ssh and the results end up in my Dropbox directory.

Setting up Password-less SSH

While this might be obvious, it is very helpful to set up your account to use password-less logins via public keys. If you have not generated your private/public ssh keys, you will need to open up a terminal session on your local computer and from the command line type:

ssh-keygen -t rsa

After you run this command you will be prompted to:

1. Enter the file where you want to store the key, hit return to accept the default (~/.ssh/id_rsa)
2. Enter a passphrase. It’s important to just hit return, then return again when asked to confirm the empty passphrase.

You now have a private key, id_rsa and a public key, id_rsa.pub in the ~/.ssh directory. It is very important that you do not share your private key with anyone. The next step is to copy the public key, id_rsa.pub to the ~/.ssh/authorized_keys file on the remote server. Run the following from the terminal:

ssh USERNAME@HOST cat < ~/.ssh/id_rsa.pub ">>" ~/.ssh/authorized_keys

With that step completed, it should possible to run remote commands via ssh without being prompted for a password.

Running the Backup

Now it’s time to run the backup script:

#! /bin/sh

. backup_config.sh

NOW=$(date +"%m%d%Y")
FILENAME="${BLOG}_${NOW}.tar.gz"
DATABASE_BAK="Database_${BLOG}_${NOW}.gz"

echo "Running remote tar backup of blog directory"
ssh $USER@$IP_ADDRESS "cd $BASE_DIR && tar -zcf ~/$FILENAME ."

echo "Begin remote copy"; scp $USER@$IP_ADDRESS:~/$FILENAME $DEST_DIR \
&& echo "Remote copy successful, removing backup on server"; ssh $USER@$IP_ADDRESS "rm ~/$FILENAME"

echo "Starting Mysql dump"
ssh $USER@$IP_ADDRESS "mysqldump -u$DBUSER -p$PASS $DATABASE | gzip -c" >  $DEST_DIR/$DATABASE_BAK
echo "Mysql dump completed"

echo "Created backup files:"
ls -la  $DEST_DIR | grep $NOW

Let’s go over the interesting parts of the backup script:

1. Line 3 sources the backup_config.sh file that sets all the variables.
2. Line 10 changes into the specified directory then makes a gzip compressed tar file of it’s contents, placing the resulting file in the home directory
3. Line 12 then copies the tar file from the remote server to your local machine. If the copy is successful, the remote tar file is deleted
4. Line 16 runs a mysqldump commmand (the contents and structure of the database) compressing the output with gzip then sends the contents over the wire to the locally specified database backup file.

As final convenience, I set an alias to the script so I am always just one short command away from running a backup.

Conclusion

So that is my basic, but effective backup script. The script has lots of room for improvement:

• There is no error handling.
• Copying the entire WordPress install is a little heavy.
• Automatic removal of old backups should be included.

If you want to try this out for yourself just edit the backup_config.sh file with the appropriate values. Hopefully others will be able to benefit from this simple backup script as well.

Resources

All sources for this post can be found at this gist.

]]> http://codingjunkie.net/simple-wordpress-backups/feed/ 0