Expert kdb+ programmer Ben Jeffery of Kx Labs recently presented his NLP in q talk at the Kx Community NYC Meetup. When Ben began this project he found there were no existing NLP libraries in q. He decided to focus on vector operations because q is especially suitable for these, rather than named entity recognition, part-of-speech tagging or co-reference resolution.
In his talk, Ben demonstrated clustering, finding groupings of entities in documents, like terms and proper nouns, as well as showing other features of NLP analytics in q. His examples included the Old and New Testaments of the Bible, Moby Dick and Jeff Skillings’ emails from his Enron days. Using his NLP program, Ben made easy work of deciphering the secret password for Jeff Skillings’ fraternity.