Package weka.classifiers
Class Evaluation
- java.lang.Object
-
- weka.classifiers.Evaluation
-
- All Implemented Interfaces:
RevisionHandler
,Summarizable
public class Evaluation extends java.lang.Object implements Summarizable, RevisionHandler
Class for evaluating machine learning models. ------------------------------------------------------------------- General options when evaluating a learning scheme from the command-line: -t filename
Name of the file with the training data. (required) -T filename
Name of the file with the test data. If missing a cross-validation is performed. -c index
Index of the class attribute (1, 2, ...; default: last). -x number
The number of folds for the cross-validation (default: 10). -no-cv
No cross validation. If no test file is provided, no evaluation is done. -split-percentage percentage
Sets the percentage for the train/test set split, e.g., 66. -preserve-order
Preserves the order in the percentage split instead of randomizing the data first with the seed value ('-s'). -s seed
Random number seed for the cross-validation and percentage split (default: 1). -m filename
The name of a file containing a cost matrix. -l filename
Loads classifier from the given file. In case the filename ends with ".xml", a PMML file is loaded or, if that fails, options are loaded from XML. -d filename
Saves classifier built from the training data into the given file. In case the filename ends with ".xml" the options are saved XML, not the model. -v
Outputs no statistics for the training data. -o
Outputs statistics only, not the classifier. -i
Outputs information-retrieval statistics per class. -k
Outputs information-theoretic statistics. -p range
Outputs predictions for test instances (or the train instances if no test instances provided and -no-cv is used), along with the attributes in the specified range (and nothing else). Use '-p 0' if no attributes are desired. -distribution
Outputs the distribution instead of only the prediction in conjunction with the '-p' option (only nominal classes). -r
Outputs cumulative margin distribution (and nothing else). -g
Only for classifiers that implement "Graphable." Outputs the graph representation of the classifier (and nothing else). -xml filename | xml-string
Retrieves the options from the XML-data instead of the command line. -threshold-file file
The file to save the threshold data to. The format is determined by the extensions, e.g., '.arff' for ARFF format or '.csv' for CSV. -threshold-label label
The class label to determine the threshold data for (default is the first label) ------------------------------------------------------------------- Example usage as the main of a classifier (called FunkyClassifier):public static void main(String [] args) { runClassifier(new FunkyClassifier(), args); }
Instances trainInstances = ... instances got from somewhere Instances testInstances = ... instances got from somewhere Classifier scheme = ... scheme got from somewhere Evaluation evaluation = new Evaluation(trainInstances); evaluation.evaluateModel(scheme, testInstances); System.out.println(evaluation.toSummaryString());
- Version:
- $Revision: 10974 $
- Author:
- Eibe Frank (eibe@cs.waikato.ac.nz), Len Trigg (trigg@cs.waikato.ac.nz)
-
-
Constructor Summary
Constructors Constructor Description Evaluation(Instances data)
Initializes all the counters for the evaluation.Evaluation(Instances data, CostMatrix costMatrix)
Initializes all the counters for the evaluation and also takes a cost matrix as parameter.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description double
areaUnderROC(int classIndex)
Returns the area under ROC for those predictions that have been collected in the evaluateClassifier(Classifier, Instances) method.double
avgCost()
Gets the average cost, that is, total cost of misclassifications (incorrect plus unclassified) over the total number of instances.double[][]
confusionMatrix()
Returns a copy of the confusion matrix.double
correct()
Gets the number of instances correctly classified (that is, for which a correct prediction was made).double
correlationCoefficient()
Returns the correlation coefficient if the class is numeric.void
crossValidateModel(java.lang.String classifierString, Instances data, int numFolds, java.lang.String[] options, java.util.Random random)
Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances.void
crossValidateModel(Classifier classifier, Instances data, int numFolds, java.util.Random random, java.lang.Object... forPredictionsPrinting)
Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances.boolean
equals(java.lang.Object obj)
Tests whether the current evaluation object is equal to another evaluation objectdouble
errorRate()
Returns the estimated error rate or the root mean squared error (if the class is numeric).static java.lang.String
evaluateModel(java.lang.String classifierString, java.lang.String[] options)
Evaluates a classifier with the options given in an array of strings.static java.lang.String
evaluateModel(Classifier classifier, java.lang.String[] options)
Evaluates a classifier with the options given in an array of strings.double[]
evaluateModel(Classifier classifier, Instances data, java.lang.Object... forPredictionsPrinting)
Evaluates the classifier on a given set of instances.double
evaluateModelOnce(double[] dist, Instance instance)
Evaluates the supplied distribution on a single instance.void
evaluateModelOnce(double prediction, Instance instance)
Evaluates the supplied prediction on a single instance.double
evaluateModelOnce(Classifier classifier, Instance instance)
Evaluates the classifier on a single instance.double
evaluateModelOnceAndRecordPrediction(double[] dist, Instance instance)
Evaluates the supplied distribution on a single instance.double
evaluateModelOnceAndRecordPrediction(Classifier classifier, Instance instance)
Evaluates the classifier on a single instance and records the prediction (if the class is nominal).double
falseNegativeRate(int classIndex)
Calculate the false negative rate with respect to a particular class.double
falsePositiveRate(int classIndex)
Calculate the false positive rate with respect to a particular class.double
fMeasure(int classIndex)
Calculate the F-Measure with respect to a particular class.double[]
getClassPriors()
Get the current weighted class countsjava.lang.String
getRevision()
Returns the revision string.double
incorrect()
Gets the number of instances incorrectly classified (that is, for which an incorrect prediction was made).double
kappa()
Returns value of kappa statistic if class is nominal.double
KBInformation()
Return the total Kononenko & Bratko Information score in bitsdouble
KBMeanInformation()
Return the Kononenko & Bratko Information score in bits per instance.double
KBRelativeInformation()
Return the Kononenko & Bratko Relative Information scorestatic void
main(java.lang.String[] args)
A test method for this class.double
meanAbsoluteError()
Returns the mean absolute error.double
meanPriorAbsoluteError()
Returns the mean absolute error of the prior.double
numFalseNegatives(int classIndex)
Calculate number of false negatives with respect to a particular class.double
numFalsePositives(int classIndex)
Calculate number of false positives with respect to a particular class.double
numInstances()
Gets the number of test instances that had a known class value (actually the sum of the weights of test instances with known class value).double
numTrueNegatives(int classIndex)
Calculate the number of true negatives with respect to a particular class.double
numTruePositives(int classIndex)
Calculate the number of true positives with respect to a particular class.double
pctCorrect()
Gets the percentage of instances correctly classified (that is, for which a correct prediction was made).double
pctIncorrect()
Gets the percentage of instances incorrectly classified (that is, for which an incorrect prediction was made).double
pctUnclassified()
Gets the percentage of instances not classified (that is, for which no prediction was made by the classifier).double
precision(int classIndex)
Calculate the precision with respect to a particular class.FastVector
predictions()
Returns the predictions that have been collected.static void
printClassifications(Classifier classifier, Instances train, ConverterUtils.DataSource testSource, int classIndex, Range attributesToOutput, boolean printDistribution, java.lang.StringBuffer text)
Prints the predictions for the given dataset into a supplied StringBufferstatic void
printClassifications(Classifier classifier, Instances train, ConverterUtils.DataSource testSource, int classIndex, Range attributesToOutput, java.lang.StringBuffer predsText)
Prints the predictions for the given dataset into a String variable.double
priorEntropy()
Calculate the entropy of the prior distributiondouble
recall(int classIndex)
Calculate the recall with respect to a particular class.double
relativeAbsoluteError()
Returns the relative absolute error.double
rootMeanPriorSquaredError()
Returns the root mean prior squared error.double
rootMeanSquaredError()
Returns the root mean squared error.double
rootRelativeSquaredError()
Returns the root relative squared error if the class is numeric.void
setPriors(Instances train)
Sets the class prior probabilitiesdouble
SFEntropyGain()
Returns the total SF, which is the null model entropy minus the scheme entropy.double
SFMeanEntropyGain()
Returns the SF per instance, which is the null model entropy minus the scheme entropy, per instance.double
SFMeanPriorEntropy()
Returns the entropy per instance for the null modeldouble
SFMeanSchemeEntropy()
Returns the entropy per instance for the schemedouble
SFPriorEntropy()
Returns the total entropy for the null modeldouble
SFSchemeEntropy()
Returns the total entropy for the schemejava.lang.String
toClassDetailsString()
Generates a breakdown of the accuracy for each class (with default title), incorporating various information-retrieval statistics, such as true/false positive rate, precision/recall/F-Measure.java.lang.String
toClassDetailsString(java.lang.String title)
Generates a breakdown of the accuracy for each class, incorporating various information-retrieval statistics, such as true/false positive rate, precision/recall/F-Measure.java.lang.String
toCumulativeMarginDistributionString()
Output the cumulative margin distribution as a string suitable for input for gnuplot or similar package.java.lang.String
toMatrixString()
Calls toMatrixString() with a default title.java.lang.String
toMatrixString(java.lang.String title)
Outputs the performance statistics as a classification confusion matrix.java.lang.String
toSummaryString()
Calls toSummaryString() with no title and no complexity statsjava.lang.String
toSummaryString(boolean printComplexityStatistics)
Calls toSummaryString() with a default title.java.lang.String
toSummaryString(java.lang.String title, boolean printComplexityStatistics)
Outputs the performance statistics in summary form.double
totalCost()
Gets the total cost, that is, the cost of each prediction times the weight of the instance, summed over all instances.double
trueNegativeRate(int classIndex)
Calculate the true negative rate with respect to a particular class.double
truePositiveRate(int classIndex)
Calculate the true positive rate with respect to a particular class.double
unclassified()
Gets the number of instances not classified (that is, for which no prediction was made by the classifier).void
updatePriors(Instance instance)
Updates the class prior probabilities (when incrementally training)void
useNoPriors()
disables the use of priors, e.g., in case of de-serialized schemes that have no access to the original training set, but are evaluated on a set set.double
weightedAreaUnderROC()
Calculates the weighted (by class size) AUC.double
weightedFalseNegativeRate()
Calculates the weighted (by class size) false negative rate.double
weightedFalsePositiveRate()
Calculates the weighted (by class size) false positive rate.double
weightedFMeasure()
Calculates the weighted (by class size) F-Measure.double
weightedPrecision()
Calculates the weighted (by class size) false precision.double
weightedRecall()
Calculates the weighted (by class size) recall.double
weightedTrueNegativeRate()
Calculates the weighted (by class size) true negative rate.double
weightedTruePositiveRate()
Calculates the weighted (by class size) true positive rate.static java.lang.String
wekaStaticWrapper(Sourcable classifier, java.lang.String className)
Wraps a static classifier in enough source to test using the weka class libraries.
-
-
-
Constructor Detail
-
Evaluation
public Evaluation(Instances data) throws java.lang.Exception
Initializes all the counters for the evaluation. UseuseNoPriors()
if the dataset is the test set and you can't initialize with the priors from the training set viasetPriors(Instances)
.- Parameters:
data
- set of training instances, to get some header information and prior class distribution information- Throws:
java.lang.Exception
- if the class is not defined- See Also:
useNoPriors()
,setPriors(Instances)
-
Evaluation
public Evaluation(Instances data, CostMatrix costMatrix) throws java.lang.Exception
Initializes all the counters for the evaluation and also takes a cost matrix as parameter. UseuseNoPriors()
if the dataset is the test set and you can't initialize with the priors from the training set viasetPriors(Instances)
.- Parameters:
data
- set of training instances, to get some header information and prior class distribution informationcostMatrix
- the cost matrix---if null, default costs will be used- Throws:
java.lang.Exception
- if cost matrix is not compatible with data, the class is not defined or the class is numeric- See Also:
useNoPriors()
,setPriors(Instances)
-
-
Method Detail
-
areaUnderROC
public double areaUnderROC(int classIndex)
Returns the area under ROC for those predictions that have been collected in the evaluateClassifier(Classifier, Instances) method. Returns Instance.missingValue() if the area is not available.- Parameters:
classIndex
- the index of the class to consider as "positive"- Returns:
- the area under the ROC curve or not a number
-
weightedAreaUnderROC
public double weightedAreaUnderROC()
Calculates the weighted (by class size) AUC.- Returns:
- the weighted AUC.
-
confusionMatrix
public double[][] confusionMatrix()
Returns a copy of the confusion matrix.- Returns:
- a copy of the confusion matrix as a two-dimensional array
-
crossValidateModel
public void crossValidateModel(Classifier classifier, Instances data, int numFolds, java.util.Random random, java.lang.Object... forPredictionsPrinting) throws java.lang.Exception
Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances. Now performs a deep copy of the classifier before each call to buildClassifier() (just in case the classifier is not initialized properly).- Parameters:
classifier
- the classifier with any options set.data
- the data on which the cross-validation is to be performednumFolds
- the number of folds for the cross-validationrandom
- random number generator for randomizationforPredictionsString
- varargs parameter that, if supplied, is expected to hold a StringBuffer to print predictions to, a Range of attributes to output and a Boolean (true if the distribution is to be printed)- Throws:
java.lang.Exception
- if a classifier could not be generated successfully or the class is not defined
-
crossValidateModel
public void crossValidateModel(java.lang.String classifierString, Instances data, int numFolds, java.lang.String[] options, java.util.Random random) throws java.lang.Exception
Performs a (stratified if class is nominal) cross-validation for a classifier on a set of instances.- Parameters:
classifierString
- a string naming the class of the classifierdata
- the data on which the cross-validation is to be performednumFolds
- the number of folds for the cross-validationoptions
- the options to the classifier. Any optionsrandom
- the random number generator for randomizing the data accepted by the classifier will be removed from this array.- Throws:
java.lang.Exception
- if a classifier could not be generated successfully or the class is not defined
-
evaluateModel
public static java.lang.String evaluateModel(java.lang.String classifierString, java.lang.String[] options) throws java.lang.Exception
Evaluates a classifier with the options given in an array of strings. Valid options are: -t filename
Name of the file with the training data. (required) -T filename
Name of the file with the test data. If missing a cross-validation is performed. -c index
Index of the class attribute (1, 2, ...; default: last). -x number
The number of folds for the cross-validation (default: 10). -no-cv
No cross validation. If no test file is provided, no evaluation is done. -split-percentage percentage
Sets the percentage for the train/test set split, e.g., 66. -preserve-order
Preserves the order in the percentage split instead of randomizing the data first with the seed value ('-s'). -s seed
Random number seed for the cross-validation and percentage split (default: 1). -m filename
The name of a file containing a cost matrix. -l filename
Loads classifier from the given file. In case the filename ends with ".xml",a PMML file is loaded or, if that fails, options are loaded from XML. -d filename
Saves classifier built from the training data into the given file. In case the filename ends with ".xml" the options are saved XML, not the model. -v
Outputs no statistics for the training data. -o
Outputs statistics only, not the classifier. -i
Outputs detailed information-retrieval statistics per class. -k
Outputs information-theoretic statistics. -p range
Outputs predictions for test instances (or the train instances if no test instances provided and -no-cv is used), along with the attributes in the specified range (and nothing else). Use '-p 0' if no attributes are desired. -distribution
Outputs the distribution instead of only the prediction in conjunction with the '-p' option (only nominal classes). -r
Outputs cumulative margin distribution (and nothing else). -g
Only for classifiers that implement "Graphable." Outputs the graph representation of the classifier (and nothing else). -xml filename | xml-string
Retrieves the options from the XML-data instead of the command line. -threshold-file file
The file to save the threshold data to. The format is determined by the extensions, e.g., '.arff' for ARFF format or '.csv' for CSV. -threshold-label label
The class label to determine the threshold data for (default is the first label)- Parameters:
classifierString
- class of machine learning classifier as a stringoptions
- the array of string containing the options- Returns:
- a string describing the results
- Throws:
java.lang.Exception
- if model could not be evaluated successfully
-
main
public static void main(java.lang.String[] args)
A test method for this class. Just extracts the first command line argument as a classifier class name and calls evaluateModel.- Parameters:
args
- an array of command line arguments, the first of which must be the class name of a classifier.
-
evaluateModel
public static java.lang.String evaluateModel(Classifier classifier, java.lang.String[] options) throws java.lang.Exception
Evaluates a classifier with the options given in an array of strings. Valid options are: -t name of training file
Name of the file with the training data. (required) -T name of test file
Name of the file with the test data. If missing a cross-validation is performed. -c class index
Index of the class attribute (1, 2, ...; default: last). -x number of folds
The number of folds for the cross-validation (default: 10). -no-cv
No cross validation. If no test file is provided, no evaluation is done. -split-percentage percentage
Sets the percentage for the train/test set split, e.g., 66. -preserve-order
Preserves the order in the percentage split instead of randomizing the data first with the seed value ('-s'). -s seed
Random number seed for the cross-validation and percentage split (default: 1). -m file with cost matrix
The name of a file containing a cost matrix. -l filename
Loads classifier from the given file. In case the filename ends with ".xml",a PMML file is loaded or, if that fails, options are loaded from XML. -d filename
Saves classifier built from the training data into the given file. In case the filename ends with ".xml" the options are saved XML, not the model. -v
Outputs no statistics for the training data. -o
Outputs statistics only, not the classifier. -i
Outputs detailed information-retrieval statistics per class. -k
Outputs information-theoretic statistics. -p range
Outputs predictions for test instances (or the train instances if no test instances provided and -no-cv is used), along with the attributes in the specified range (and nothing else). Use '-p 0' if no attributes are desired. -distribution
Outputs the distribution instead of only the prediction in conjunction with the '-p' option (only nominal classes). -r
Outputs cumulative margin distribution (and nothing else). -g
Only for classifiers that implement "Graphable." Outputs the graph representation of the classifier (and nothing else). -xml filename | xml-string
Retrieves the options from the XML-data instead of the command line.- Parameters:
classifier
- machine learning classifieroptions
- the array of string containing the options- Returns:
- a string describing the results
- Throws:
java.lang.Exception
- if model could not be evaluated successfully
-
evaluateModel
public double[] evaluateModel(Classifier classifier, Instances data, java.lang.Object... forPredictionsPrinting) throws java.lang.Exception
Evaluates the classifier on a given set of instances. Note that the data must have exactly the same format (e.g. order of attributes) as the data used to train the classifier! Otherwise the results will generally be meaningless.- Parameters:
classifier
- machine learning classifierdata
- set of test instances for evaluationforPredictionsString
- varargs parameter that, if supplied, is expected to hold a StringBuffer to print predictions to, a Range of attributes to output and a Boolean (true if the distribution is to be printed)- Returns:
- the predictions
- Throws:
java.lang.Exception
- if model could not be evaluated successfully
-
evaluateModelOnceAndRecordPrediction
public double evaluateModelOnceAndRecordPrediction(Classifier classifier, Instance instance) throws java.lang.Exception
Evaluates the classifier on a single instance and records the prediction (if the class is nominal).- Parameters:
classifier
- machine learning classifierinstance
- the test instance to be classified- Returns:
- the prediction made by the clasifier
- Throws:
java.lang.Exception
- if model could not be evaluated successfully or the data contains string attributes
-
evaluateModelOnce
public double evaluateModelOnce(Classifier classifier, Instance instance) throws java.lang.Exception
Evaluates the classifier on a single instance.- Parameters:
classifier
- machine learning classifierinstance
- the test instance to be classified- Returns:
- the prediction made by the clasifier
- Throws:
java.lang.Exception
- if model could not be evaluated successfully or the data contains string attributes
-
evaluateModelOnce
public double evaluateModelOnce(double[] dist, Instance instance) throws java.lang.Exception
Evaluates the supplied distribution on a single instance.- Parameters:
dist
- the supplied distributioninstance
- the test instance to be classified- Returns:
- the prediction
- Throws:
java.lang.Exception
- if model could not be evaluated successfully
-
evaluateModelOnceAndRecordPrediction
public double evaluateModelOnceAndRecordPrediction(double[] dist, Instance instance) throws java.lang.Exception
Evaluates the supplied distribution on a single instance.- Parameters:
dist
- the supplied distributioninstance
- the test instance to be classified- Returns:
- the prediction
- Throws:
java.lang.Exception
- if model could not be evaluated successfully
-
evaluateModelOnce
public void evaluateModelOnce(double prediction, Instance instance) throws java.lang.Exception
Evaluates the supplied prediction on a single instance.- Parameters:
prediction
- the supplied predictioninstance
- the test instance to be classified- Throws:
java.lang.Exception
- if model could not be evaluated successfully
-
predictions
public FastVector predictions()
Returns the predictions that have been collected.- Returns:
- a reference to the FastVector containing the predictions that have been collected. This should be null if no predictions have been collected (e.g. if the class is numeric).
-
wekaStaticWrapper
public static java.lang.String wekaStaticWrapper(Sourcable classifier, java.lang.String className) throws java.lang.Exception
Wraps a static classifier in enough source to test using the weka class libraries.- Parameters:
classifier
- a Sourcable ClassifierclassName
- the name to give to the source code class- Returns:
- the source for a static classifier that can be tested with weka libraries.
- Throws:
java.lang.Exception
- if code-generation fails
-
numInstances
public final double numInstances()
Gets the number of test instances that had a known class value (actually the sum of the weights of test instances with known class value).- Returns:
- the number of test instances with known class
-
incorrect
public final double incorrect()
Gets the number of instances incorrectly classified (that is, for which an incorrect prediction was made). (Actually the sum of the weights of these instances)- Returns:
- the number of incorrectly classified instances
-
pctIncorrect
public final double pctIncorrect()
Gets the percentage of instances incorrectly classified (that is, for which an incorrect prediction was made).- Returns:
- the percent of incorrectly classified instances (between 0 and 100)
-
totalCost
public final double totalCost()
Gets the total cost, that is, the cost of each prediction times the weight of the instance, summed over all instances.- Returns:
- the total cost
-
avgCost
public final double avgCost()
Gets the average cost, that is, total cost of misclassifications (incorrect plus unclassified) over the total number of instances.- Returns:
- the average cost.
-
correct
public final double correct()
Gets the number of instances correctly classified (that is, for which a correct prediction was made). (Actually the sum of the weights of these instances)- Returns:
- the number of correctly classified instances
-
pctCorrect
public final double pctCorrect()
Gets the percentage of instances correctly classified (that is, for which a correct prediction was made).- Returns:
- the percent of correctly classified instances (between 0 and 100)
-
unclassified
public final double unclassified()
Gets the number of instances not classified (that is, for which no prediction was made by the classifier). (Actually the sum of the weights of these instances)- Returns:
- the number of unclassified instances
-
pctUnclassified
public final double pctUnclassified()
Gets the percentage of instances not classified (that is, for which no prediction was made by the classifier).- Returns:
- the percent of unclassified instances (between 0 and 100)
-
errorRate
public final double errorRate()
Returns the estimated error rate or the root mean squared error (if the class is numeric). If a cost matrix was given this error rate gives the average cost.- Returns:
- the estimated error rate (between 0 and 1, or between 0 and maximum cost)
-
kappa
public final double kappa()
Returns value of kappa statistic if class is nominal.- Returns:
- the value of the kappa statistic
-
correlationCoefficient
public final double correlationCoefficient() throws java.lang.Exception
Returns the correlation coefficient if the class is numeric.- Returns:
- the correlation coefficient
- Throws:
java.lang.Exception
- if class is not numeric
-
meanAbsoluteError
public final double meanAbsoluteError()
Returns the mean absolute error. Refers to the error of the predicted values for numeric classes, and the error of the predicted probability distribution for nominal classes.- Returns:
- the mean absolute error
-
meanPriorAbsoluteError
public final double meanPriorAbsoluteError()
Returns the mean absolute error of the prior.- Returns:
- the mean absolute error
-
relativeAbsoluteError
public final double relativeAbsoluteError() throws java.lang.Exception
Returns the relative absolute error.- Returns:
- the relative absolute error
- Throws:
java.lang.Exception
- if it can't be computed
-
rootMeanSquaredError
public final double rootMeanSquaredError()
Returns the root mean squared error.- Returns:
- the root mean squared error
-
rootMeanPriorSquaredError
public final double rootMeanPriorSquaredError()
Returns the root mean prior squared error.- Returns:
- the root mean prior squared error
-
rootRelativeSquaredError
public final double rootRelativeSquaredError()
Returns the root relative squared error if the class is numeric.- Returns:
- the root relative squared error
-
priorEntropy
public final double priorEntropy() throws java.lang.Exception
Calculate the entropy of the prior distribution- Returns:
- the entropy of the prior distribution
- Throws:
java.lang.Exception
- if the class is not nominal
-
KBInformation
public final double KBInformation() throws java.lang.Exception
Return the total Kononenko & Bratko Information score in bits- Returns:
- the K&B information score
- Throws:
java.lang.Exception
- if the class is not nominal
-
KBMeanInformation
public final double KBMeanInformation() throws java.lang.Exception
Return the Kononenko & Bratko Information score in bits per instance.- Returns:
- the K&B information score
- Throws:
java.lang.Exception
- if the class is not nominal
-
KBRelativeInformation
public final double KBRelativeInformation() throws java.lang.Exception
Return the Kononenko & Bratko Relative Information score- Returns:
- the K&B relative information score
- Throws:
java.lang.Exception
- if the class is not nominal
-
SFPriorEntropy
public final double SFPriorEntropy()
Returns the total entropy for the null model- Returns:
- the total null model entropy
-
SFMeanPriorEntropy
public final double SFMeanPriorEntropy()
Returns the entropy per instance for the null model- Returns:
- the null model entropy per instance
-
SFSchemeEntropy
public final double SFSchemeEntropy()
Returns the total entropy for the scheme- Returns:
- the total scheme entropy
-
SFMeanSchemeEntropy
public final double SFMeanSchemeEntropy()
Returns the entropy per instance for the scheme- Returns:
- the scheme entropy per instance
-
SFEntropyGain
public final double SFEntropyGain()
Returns the total SF, which is the null model entropy minus the scheme entropy.- Returns:
- the total SF
-
SFMeanEntropyGain
public final double SFMeanEntropyGain()
Returns the SF per instance, which is the null model entropy minus the scheme entropy, per instance.- Returns:
- the SF per instance
-
toCumulativeMarginDistributionString
public java.lang.String toCumulativeMarginDistributionString() throws java.lang.Exception
Output the cumulative margin distribution as a string suitable for input for gnuplot or similar package.- Returns:
- the cumulative margin distribution
- Throws:
java.lang.Exception
- if the class attribute is nominal
-
toSummaryString
public java.lang.String toSummaryString()
Calls toSummaryString() with no title and no complexity stats- Specified by:
toSummaryString
in interfaceSummarizable
- Returns:
- a summary description of the classifier evaluation
-
toSummaryString
public java.lang.String toSummaryString(boolean printComplexityStatistics)
Calls toSummaryString() with a default title.- Parameters:
printComplexityStatistics
- if true, complexity statistics are returned as well- Returns:
- the summary string
-
toSummaryString
public java.lang.String toSummaryString(java.lang.String title, boolean printComplexityStatistics)
Outputs the performance statistics in summary form. Lists number (and percentage) of instances classified correctly, incorrectly and unclassified. Outputs the total number of instances classified, and the number of instances (if any) that had no class value provided.- Parameters:
title
- the title for the statisticsprintComplexityStatistics
- if true, complexity statistics are returned as well- Returns:
- the summary as a String
-
toMatrixString
public java.lang.String toMatrixString() throws java.lang.Exception
Calls toMatrixString() with a default title.- Returns:
- the confusion matrix as a string
- Throws:
java.lang.Exception
- if the class is numeric
-
toMatrixString
public java.lang.String toMatrixString(java.lang.String title) throws java.lang.Exception
Outputs the performance statistics as a classification confusion matrix. For each class value, shows the distribution of predicted class values.- Parameters:
title
- the title for the confusion matrix- Returns:
- the confusion matrix as a String
- Throws:
java.lang.Exception
- if the class is numeric
-
toClassDetailsString
public java.lang.String toClassDetailsString() throws java.lang.Exception
Generates a breakdown of the accuracy for each class (with default title), incorporating various information-retrieval statistics, such as true/false positive rate, precision/recall/F-Measure. Should be useful for ROC curves, recall/precision curves.- Returns:
- the statistics presented as a string
- Throws:
java.lang.Exception
- if class is not nominal
-
toClassDetailsString
public java.lang.String toClassDetailsString(java.lang.String title) throws java.lang.Exception
Generates a breakdown of the accuracy for each class, incorporating various information-retrieval statistics, such as true/false positive rate, precision/recall/F-Measure. Should be useful for ROC curves, recall/precision curves.- Parameters:
title
- the title to prepend the stats string with- Returns:
- the statistics presented as a string
- Throws:
java.lang.Exception
- if class is not nominal
-
numTruePositives
public double numTruePositives(int classIndex)
Calculate the number of true positives with respect to a particular class. This is defined ascorrectly classified positives
- Parameters:
classIndex
- the index of the class to consider as "positive"- Returns:
- the true positive rate
-
truePositiveRate
public double truePositiveRate(int classIndex)
Calculate the true positive rate with respect to a particular class. This is defined ascorrectly classified positives ------------------------------ total positives
- Parameters:
classIndex
- the index of the class to consider as "positive"- Returns:
- the true positive rate
-
weightedTruePositiveRate
public double weightedTruePositiveRate()
Calculates the weighted (by class size) true positive rate.- Returns:
- the weighted true positive rate.
-
numTrueNegatives
public double numTrueNegatives(int classIndex)
Calculate the number of true negatives with respect to a particular class. This is defined ascorrectly classified negatives
- Parameters:
classIndex
- the index of the class to consider as "positive"- Returns:
- the true positive rate
-
trueNegativeRate
public double trueNegativeRate(int classIndex)
Calculate the true negative rate with respect to a particular class. This is defined ascorrectly classified negatives ------------------------------ total negatives
- Parameters:
classIndex
- the index of the class to consider as "positive"- Returns:
- the true positive rate
-
weightedTrueNegativeRate
public double weightedTrueNegativeRate()
Calculates the weighted (by class size) true negative rate.- Returns:
- the weighted true negative rate.
-
numFalsePositives
public double numFalsePositives(int classIndex)
Calculate number of false positives with respect to a particular class. This is defined asincorrectly classified negatives
- Parameters:
classIndex
- the index of the class to consider as "positive"- Returns:
- the false positive rate
-
falsePositiveRate
public double falsePositiveRate(int classIndex)
Calculate the false positive rate with respect to a particular class. This is defined asincorrectly classified negatives -------------------------------- total negatives
- Parameters:
classIndex
- the index of the class to consider as "positive"- Returns:
- the false positive rate
-
weightedFalsePositiveRate
public double weightedFalsePositiveRate()
Calculates the weighted (by class size) false positive rate.- Returns:
- the weighted false positive rate.
-
numFalseNegatives
public double numFalseNegatives(int classIndex)
Calculate number of false negatives with respect to a particular class. This is defined asincorrectly classified positives
- Parameters:
classIndex
- the index of the class to consider as "positive"- Returns:
- the false positive rate
-
falseNegativeRate
public double falseNegativeRate(int classIndex)
Calculate the false negative rate with respect to a particular class. This is defined asincorrectly classified positives -------------------------------- total positives
- Parameters:
classIndex
- the index of the class to consider as "positive"- Returns:
- the false positive rate
-
weightedFalseNegativeRate
public double weightedFalseNegativeRate()
Calculates the weighted (by class size) false negative rate.- Returns:
- the weighted false negative rate.
-
recall
public double recall(int classIndex)
Calculate the recall with respect to a particular class. This is defined ascorrectly classified positives ------------------------------ total positives
(Which is also the same as the truePositiveRate.)- Parameters:
classIndex
- the index of the class to consider as "positive"- Returns:
- the recall
-
weightedRecall
public double weightedRecall()
Calculates the weighted (by class size) recall.- Returns:
- the weighted recall.
-
precision
public double precision(int classIndex)
Calculate the precision with respect to a particular class. This is defined ascorrectly classified positives ------------------------------ total predicted as positive
- Parameters:
classIndex
- the index of the class to consider as "positive"- Returns:
- the precision
-
weightedPrecision
public double weightedPrecision()
Calculates the weighted (by class size) false precision.- Returns:
- the weighted precision.
-
fMeasure
public double fMeasure(int classIndex)
Calculate the F-Measure with respect to a particular class. This is defined as2 * recall * precision ---------------------- recall + precision
- Parameters:
classIndex
- the index of the class to consider as "positive"- Returns:
- the F-Measure
-
weightedFMeasure
public double weightedFMeasure()
Calculates the weighted (by class size) F-Measure.- Returns:
- the weighted F-Measure.
-
setPriors
public void setPriors(Instances train) throws java.lang.Exception
Sets the class prior probabilities- Parameters:
train
- the training instances used to determine the prior probabilities- Throws:
java.lang.Exception
- if the class attribute of the instances is not set
-
getClassPriors
public double[] getClassPriors()
Get the current weighted class counts- Returns:
- the weighted class counts
-
updatePriors
public void updatePriors(Instance instance) throws java.lang.Exception
Updates the class prior probabilities (when incrementally training)- Parameters:
instance
- the new training instance seen- Throws:
java.lang.Exception
- if the class of the instance is not set
-
useNoPriors
public void useNoPriors()
disables the use of priors, e.g., in case of de-serialized schemes that have no access to the original training set, but are evaluated on a set set.
-
equals
public boolean equals(java.lang.Object obj)
Tests whether the current evaluation object is equal to another evaluation object- Overrides:
equals
in classjava.lang.Object
- Parameters:
obj
- the object to compare against- Returns:
- true if the two objects are equal
-
printClassifications
public static void printClassifications(Classifier classifier, Instances train, ConverterUtils.DataSource testSource, int classIndex, Range attributesToOutput, java.lang.StringBuffer predsText) throws java.lang.Exception
Prints the predictions for the given dataset into a String variable.- Parameters:
classifier
- the classifier to usetrain
- the training datatestSource
- the test setclassIndex
- the class index (1-based), if -1 ot does not override the class index is stored in the data file (by using the last attribute)attributesToOutput
- the indices of the attributes to output- Throws:
java.lang.Exception
- if test file cannot be opened
-
printClassifications
public static void printClassifications(Classifier classifier, Instances train, ConverterUtils.DataSource testSource, int classIndex, Range attributesToOutput, boolean printDistribution, java.lang.StringBuffer text) throws java.lang.Exception
Prints the predictions for the given dataset into a supplied StringBuffer- Parameters:
classifier
- the classifier to usetrain
- the training datatestSource
- the test setclassIndex
- the class index (1-based), if -1 ot does not override the class index is stored in the data file (by using the last attribute)attributesToOutput
- the indices of the attributes to outputprintDistribution
- prints the complete distribution for nominal classes, not just the predicted valuetext
- StringBuffer to hold the printed predictions- Throws:
java.lang.Exception
- if test file cannot be opened
-
getRevision
public java.lang.String getRevision()
Returns the revision string.- Specified by:
getRevision
in interfaceRevisionHandler
- Returns:
- the revision
-
-