Random Thoughts on Coding

Whatever comes to mind at the moment.

Sql for Lucene

A short time ago, I started a side project to learn the latest version of Antlr. I decided to do something that has always interested me, a sql parser for the Lucene search engine. Even though the parser is a learning exercise, I thought someone else could find this useful. This post will cover the functionality of the LuceneQueryParser. Building the parser using Antlr4 will be coming in later posts.

Introduction and Examples

The LuceneSqlParser supports a subset of standard sql. Here are some examples:

Sample sql query handled
Select last_name from '/path/to/index/' where first_name='Foo' and age <=30 and city='Boston' limit 25

Select * from 'path/index/' where age in (31, 30, 50)

Select first_name, last_name from '/path/index/' where city in ('Cincinatti', 'New York', 'Boyds')

Select first_name from '/path/index/' where age between 35 and 50 and first_name like 'Br*'
-- Also takes paths from Windows OS
Select first_name from 'C:/path/index/' where first_name='John' and (age<=45 and city not in ('New York', 'Boston', 'Atlanta'))

The LuceneSqlParser returns a BooleanQuery. The BooleanQuery will contain different types of lucene query objects depending on the predicates used. There is a class Searcher avaiable for use with the LuceneSqlParser. The Searcher abstracts away the opening of a lucene IndexSearcher, iterating over the ScoreDoc array and extracting results. Next, we’ll take a look at the rules used to parse the sql.

Java 8 Functional Interfaces and Checked Exceptions

The Java 8 lambda syntax and functional interfaces have been a productivity boost for Java developers. But there is one drawback to functional interfaces. None of the them as currently defined in Java 8 declare any checked exceptions. This leaves the developer at odds on how best to handle checked exceptions. This post will present one option for handling checked exceptions in functional interfaces. We will use use the Function in our example, but the pattern should apply to any of the functional interfaces.

Example of a Function with a Checked Exception

Here’s an example from a recent side-project using a Function to open a directory in Lucene. As expected, opening a directory for writing/searching throws an IOException:

Create a Lucene Directory
   private Function<Path, Directory> createDirectory =  path -> {
            return FSDirectory.open(path);
        }catch (IOException e){
            throw new RuntimeException(e);

I/O With Files That Aren’t Files

Recently at work I needed to search through our archived files and provide the results by the end of the day. Here’s the parameters of the request:

  1. The archive files are encrypted and stored in HDFS (Don’t ask why we store them in HDFS).
  2. The files vary in size form 3-9 GB.
  3. The total number of files to search was 300+
  4. It takes between 1 – 2 minutes to decrypt each file.

In the past there have been requests to search one archived file. In those cases we would copy the file out of HDFS to a server. Then run a shell script to decrypt the file and perform the search. The decrypting program requires 2 arguments: an encrypted file and a file to write the decrypted data to. This means the decrypted and encrypted file are on disk at the same time.

At an average rate of of 1.5 minutes to decrypt a single file, it was going to take 450 minutes (7.5 hours) for 300 files. To add to my dilema, there wasn’t enough time to write custom RecordReader. The only solution would be to stream the files in parallel. But there 2 problems with that approach:

  1. The server does not have enough space for 20 (10 encrypted and 10 decrypted) files at a time.
  2. The decrypting code does read from stdin or write to stdout.

What to do? Use named pipes of course!

Whats New in Java 8 - Date API Part II

This post is continues our review of the Date API that came with the release of Java 8. We are going to continue our concentration on classes that make working with dates/times very easy. Working with date objects in previous releases of Java was very challenging with respect to adding time or getting the difference between dates. Hopefully after looking at the classes we present here, your opinion of working with dates and times in Java will change. Specifically, we are going to take a look at the following classes:

  • Other classes to represent dates/times ZonedDateTime and OffsetDateTime
  • Getting the current snapshot in time with Instant
  • Using the Clock class to get system time but specify different time zones
  • Represent arbitrary number of days with the Period class
  • Represent arbitrary amount of hours with the Duration class

Blog Migrated to Octopress

After a long period of good intentions with no action, I finally moved the codingjunkie.net blog from Wordpress to Octopress. I have nothing bad to say about Wordpress, it’s been a great tool for me. It’s just that over time I’ve found myself wanting a simpler blogging platform. Aside from liking the overall looks of Ocotpress, the big draw for me was the decreased load time from serving up static HTML pages. Once I sat down and decided to pull the plug, it was really pretty simple. It all boiled down to just a few steps:

  1. Export my posts into the Wordpress XML format.
  2. Run exitwp on the exported XML.
  3. Some basic regex work to fix image tags.
  4. Configure the permalinks to match the form I already use.
  5. Spend 5 minutes reconfiguring my blog on the WebFaction control panel.
  6. Deploy the converted posts using rsync.

Overall, it went much easier than I had anticipated. Only time will tell if switching to the new platform will help me write better content!