NLP Categorizer

Most of the time, people need to categorize the documents according to their context. It is very useful when people work with very large number of documents.

Therefore It is very easy to make NLP categorizer for above purpose. There are several algorithms  used categorize these documents. most of them are semi supervised algorithms, like maximum entropy, naive bayes and maximum entropy markov models. Today I'm going describe how to categorize documents using apache openNLP toolkit. Apache openNLP supports maximum entropy algorithm.

First we have create a training dataset. training data should include category and content. normally there should be more than 5000 training data set for get a fine model.

Other good morning /
Other good evening /
Other have you any update on negombo road till wattala /
Other perhaps the madness was always there but only the schools bring it out? /
Other sorry didn't notice geotag /
Feed high traffic in wattala /
Feed low traffic in negombo road /
Feed moving traffic in wattala /
Feed nawala bridge area clear /
Feed no traffic at all at ja-ela /

Then we need to train a nlp categorizer model according to the dataset. Therefore you can easily go through OpenNLP documentation and train you model.
This following code can be used to train categorizing model and testing. here I have used training parameters as 2 cutoff mark and 300 iterations.

import opennlp.tools.doccat.DoccatModel;
import opennlp.tools.doccat.DocumentCategorizerME;
import opennlp.tools.doccat.DocumentSample;
import opennlp.tools.doccat.DocumentSampleStream;
import opennlp.tools.util.ObjectStream;
import opennlp.tools.util.PlainTextByLineStream;
import org.apache.log4j.Logger;

import java.io.*;

public class FeedClassifierTrainer {

 private static final Logger log = Logger.getLogger(FeedClassifierTrainer.class);

 private static DoccatModel model = null;

 public static void main(String[] args) {
  log.debug("Model training started");
  //to train the model
 new FeedClassifierTrainer().train();
  //testing purpose
  String content = "due to train strike heavy traffic in maradhana ";
  try {
   //test the model
   new FeedClassifierTrainer().test(content);
  } catch (IOException e) {
   e.printStackTrace();
  }
 }

 /**
  * Training the models
  */
 public void train() {
  // model name you define your own name for the model
  String onlpModelPath = "en-doccat.bin";
  // training data set
  String trainingDataFilePath = "data.txt";

  InputStream dataInputStream = null;
  try {
   // Read training data file
   dataInputStream = new FileInputStream(trainingDataFilePath);
   // Read each training instance
   ObjectStream lineStream = new PlainTextByLineStream(dataInputStream, "UTF-8");
   // making sample Stream to train
   ObjectStream sampleStream = new DocumentSampleStream(lineStream);
   // Calculate the training model "en" means english, sampleStream is the training data, 2 cutoff, 300 iterations
   model = DocumentCategorizerME.train("en", sampleStream, 2, 300);
  } catch (IOException e) {
   log.error(e.getMessage());
  } finally {
   if (dataInputStream != null) {
    try {
     dataInputStream.close();
    } catch (IOException e) {
     log.error(e.getMessage());
    }
   }
  }


 // Now we are writing the calculated model to a file in order to use the
 // trained classifier in production

try {
   if (model != null) {
  //saving the file 
    model.serialize(new FileOutputStream(onlpModelPath));
   }
  } catch (IOException e) {
   log.error(e.getMessage());
  }
 }

 /*
  * Now we call the saved model and test it
  * Give it a new text document and the expected category
  */
 public void test(String text) throws IOException {
  String classificationModelFilePath = "en-doccat.bin";
  DocumentCategorizerME classificationME =
    new DocumentCategorizerME(
      new DoccatModel(
        new FileInputStream(
          classificationModelFilePath)));
  String documentContent = text;
  double[] classDistribution = classificationME.categorize(documentContent);
  // get the predicted model
  String predictedCategory = classificationME.getBestCategory(classDistribution);
  System.out.println("Model prediction : " + predictedCategory);

 }
}