Package smile.classification

Class DiscreteNaiveBayes

java.lang.Object

smile.classification.AbstractClassifier<int[]>

smile.classification.DiscreteNaiveBayes

All Implemented Interfaces:

Serializable, ToDoubleFunction<int[]>, ToIntFunction<int[]>, Classifier<int[]>

public class DiscreteNaiveBayes
extends AbstractClassifier<int[]>

Naive Bayes classifier for document classification in NLP. A naive Bayes classifier is a simple probabilistic classifier based on applying Bayes' theorem with strong (naive) independence assumptions. Depending on the precise nature of the probability model, naive Bayes classifiers can be trained very efficiently in a supervised learning setting.

In spite of their naive design and apparently over-simplified assumptions, naive Bayes classifiers have worked quite well in many complex real-world situations and are very popular in Natural Language Processing (NLP).

For document classification in NLP, there are two major different ways we can set up a naive Bayes classifier: multinomial model and Bernoulli model. The multinomial model generates one term from the vocabulary in each position of the document. The multivariate Bernoulli model or Bernoulli model generates an indicator for each term of the vocabulary, either indicating presence of the term in the document or indicating absence. Of the two models, the Bernoulli model is particularly sensitive to noise features. A Bernoulli naive Bayes classifier requires some form of feature selection or else its accuracy will be low.

The different generation models imply different estimation strategies and different classification rules. The Bernoulli model estimates as the fraction of documents of class that contain term. In contrast, the multinomial model estimates as the fraction of tokens or fraction of positions in documents of class that contain term. When classifying a test document, the Bernoulli model uses binary occurrence information, ignoring the number of occurrences, whereas the multinomial model keeps track of multiple occurrences. As a result, the Bernoulli model typically makes many mistakes when classifying long documents. However, it was reported that the Bernoulli model works better in sentiment analysis.

The models also differ in how non-occurring terms are used in classification. They do not affect the classification decision in the multinomial model; but in the Bernoulli model the probability of nonoccurrence is factored in when computing. This is because only the Bernoulli model models absence of terms explicitly.

A third setting is Polya Urn model which simply add twice for what is seen in training data instead of one time. See reference for more details.

References

1. Rennie, Jason D., et al. Tackling the poor assumptions of naive Bayes text classifiers. ICML, 2003.
2. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schutze. Introduction to Information Retrieval, Chapter 13, 2009.
3. Kevin P. Murphy. Machina Learning A Probability Perspective, Chapter 3, 2012.

See Also:

• Distribution
• LDA
• QDA
• RDA
• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	DiscreteNaiveBayes.Model	The generation models of naive Bayes classifier.

Nested classes/interfaces inherited from interface smile.classification.Classifier

Classifier.Trainer<T,M extends Classifier<T>>

Field Summary

Fields inherited from class smile.classification.AbstractClassifier

classes

Constructor Summary

Constructors

Constructor	Description
DiscreteNaiveBayes(DiscreteNaiveBayes.Model model, double[] priori, int p)	Constructor of naive Bayes classifier for document classification.
DiscreteNaiveBayes(DiscreteNaiveBayes.Model model, double[] priori, int p, double sigma, IntSet labels)	Constructor of naive Bayes classifier for document classification.
DiscreteNaiveBayes(DiscreteNaiveBayes.Model model, int k, int p)	Constructor of naive Bayes classifier for document classification.
DiscreteNaiveBayes(DiscreteNaiveBayes.Model model, int k, int p, double sigma, IntSet labels)	Constructor of naive Bayes classifier for document classification.

Method Summary

Modifier and Type	Method	Description
boolean	online()	Returns true if this is an online learner.
int	predict(int[] x)	Predict the class of an instance.
int	predict(int[] x, double[] posteriori)	Predict the class of an instance.
int	predict(SparseArray x)	Predict the class of an instance.
int	predict(SparseArray x, double[] posteriori)	Predict the class of an instance.
double[]	priori()	Returns a priori probabilities.
boolean	soft()	Returns true if this is a soft classifier that can estimate the posteriori probabilities of classification.
void	update(int[][] x, int[] y)	Batch learning of naive Bayes classifier on sequences, which are modeled as a bag of words.
void	update(int[] x, int y)	Online learning of naive Bayes classifier on a sequence, which is modeled as a bag of words.
void	update(SparseArray[] x, int[] y)	Batch learning of naive Bayes classifier on sequences, which are modeled as a bag of words.
void	update(SparseArray x, int y)	Online learning of naive Bayes classifier on a sequence, which is modeled as a bag of words.

Methods inherited from class smile.classification.AbstractClassifier

classes, numClasses

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface smile.classification.Classifier

applyAsDouble, applyAsInt, predict, predict, predict, predict, predict, predict, score, update

Constructor Details

DiscreteNaiveBayes

public DiscreteNaiveBayes(DiscreteNaiveBayes.Model model,
 int k,
 int p)

Constructor of naive Bayes classifier for document classification. The priori probability of each class will be learned from data. By default, we use add-one/Laplace smoothing, which simply adds one to each count to eliminate zeros. Add-one smoothing can be interpreted as a uniform prior (each term occurs once for each class) that is then updated as evidence from the training data comes in.

Parameters:

model - the generation model of naive Bayes classifier.

k - the number of classes.

p - the dimensionality of input space.

DiscreteNaiveBayes

public DiscreteNaiveBayes(DiscreteNaiveBayes.Model model,
 int k,
 int p,
 double sigma,
 IntSet labels)

Constructor of naive Bayes classifier for document classification. The priori probability of each class will be learned from data. Add-k smoothing.

Parameters:

model - the generation model of naive Bayes classifier.

k - the number of classes.

p - the dimensionality of input space.

sigma - the prior count of add-k smoothing of evidence.

labels - the class label encoder.

DiscreteNaiveBayes

public DiscreteNaiveBayes(DiscreteNaiveBayes.Model model,
 double[] priori,
 int p)

Constructor of naive Bayes classifier for document classification. By default, we use add-one/Laplace smoothing, which simply adds one to each count to eliminate zeros. Add-one smoothing can be interpreted as a uniform prior (each term occurs once for each class) that is then updated as evidence from the training data comes in.

Parameters:

model - the generation model of naive Bayes classifier.

priori - the priori probability of each class.

p - the dimensionality of input space.

DiscreteNaiveBayes

public DiscreteNaiveBayes(DiscreteNaiveBayes.Model model,
 double[] priori,
 int p,
 double sigma,
 IntSet labels)

Constructor of naive Bayes classifier for document classification. Add-k smoothing.

Parameters:

model - the generation model of naive Bayes classifier.

priori - the priori probability of each class.

p - the dimensionality of input space.

sigma - the prior count of add-k smoothing of evidence.

labels - the class label encoder.

Method Details

priori

public double[] priori()

Returns a priori probabilities.

Returns:

a priori probabilities.

update

public void update(int[] x,
 int y)

Online learning of naive Bayes classifier on a sequence, which is modeled as a bag of words. Note that this method is NOT applicable for naive Bayes classifier with general generation model.

Parameters:

x - training instance.

y - training label.

update

public void update(SparseArray x,
 int y)

Online learning of naive Bayes classifier on a sequence, which is modeled as a bag of words. Note that this method is NOT applicable for naive Bayes classifier with general generation model.

Parameters:

x - training instance in sparse format.

y - training label.

update

public void update(int[][] x,
 int[] y)

Batch learning of naive Bayes classifier on sequences, which are modeled as a bag of words. Note that this method is NOT applicable for naive Bayes classifier with general generation model.

Parameters:

x - training instances.

y - training labels.

update

public void update(SparseArray[] x,
 int[] y)

Batch learning of naive Bayes classifier on sequences, which are modeled as a bag of words. Note that this method is NOT applicable for naive Bayes classifier with general generation model.

Parameters:

x - training instances.

y - training labels.

soft

public boolean soft()

Description copied from interface: Classifier

Returns true if this is a soft classifier that can estimate the posteriori probabilities of classification.

Returns:

true if soft classifier.

online

public boolean online()

Description copied from interface: Classifier

Returns true if this is an online learner.

Returns:

true if online learner.

predict

public int predict(int[] x)

Predict the class of an instance.

Parameters:

x - the instance to be classified.

Returns:

the predicted class label.

predict

public int predict(int[] x,
 double[] posteriori)

Predict the class of an instance.

Parameters:

x - the instance to be classified.

posteriori - a posteriori probabilities on output.

Returns:

the predicted class label. If the instance is of all zeros, returns Integer.MIN_VALUE.

predict

public int predict(SparseArray x)

Predict the class of an instance.

Parameters:

x - the instance to be classified.

Returns:

the predicted class label. For MULTINOMIAL and BERNOULLI models, returns -1 if the instance does not contain any feature words.

predict

public int predict(SparseArray x,
 double[] posteriori)

Predict the class of an instance.

Parameters:

x - the instance to be classified.

posteriori - the array to store a posteriori probabilities on output.

Returns:

the predicted class label. If the instance is of all zeros, return returns Integer.MIN_VALUE.