Package weka.experiment
Class PairedTTester
- java.lang.Object
-
- weka.experiment.PairedTTester
-
- All Implemented Interfaces:
java.io.Serializable
,OptionHandler
,RevisionHandler
,Tester
- Direct Known Subclasses:
PairedCorrectedTTester
public class PairedTTester extends java.lang.Object implements OptionHandler, Tester, RevisionHandler
Calculates T-Test statistics on data stored in a set of instances. Valid options are:-D <index,index2-index4,...> Specify list of columns that specify a unique dataset. First and last are valid indexes. (default none)
-R <index> Set the index of the column containing the run number
-F <index> Set the index of the column containing the fold number
-G <index1,index2-index4,...> Specify list of columns that specify a unique 'result generator' (eg: classifier name and options). First and last are valid indexes. (default none)
-S <significance level> Set the significance level for comparisons (default 0.05)
-V Show standard deviations
-L Produce table comparisons in Latex table format
-csv Produce table comparisons in CSV table format
-html Produce table comparisons in HTML table format
-significance Produce table comparisons with only the significance values
-gnuplot Produce table comparisons output suitable for GNUPlot
- Version:
- $Revision: 6431 $
- Author:
- Len Trigg (trigg@cs.waikato.ac.nz)
- See Also:
- Serialized Form
-
-
Constructor Summary
Constructors Constructor Description PairedTTester()
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description void
assign(Tester tester)
retrieves all the settings from the given TesterPairedStats
calculateStatistics(Instance datasetSpecifier, int resultset1Index, int resultset2Index, int comparisonColumn)
Computes a paired t-test comparison for a specified dataset between two resultsets.boolean
displayResultset(int index)
Checks whether the resultset with the given index shall be displayed.Range
getDatasetKeyColumns()
Get the value of DatasetKeyColumns.int[]
getDisplayedResultsets()
Gets the indices of the the datasets that are displayed (ifnull
then all are displayed).java.lang.String
getDisplayName()
returns the name of the testerint
getFoldColumn()
Get the value of FoldColumn.Instances
getInstances()
Get the value of Instances.int
getNumDatasets()
Gets the number of datasets in the resultsetsint
getNumResultsets()
Gets the number of resultsets in the data.java.lang.String[]
getOptions()
Gets current settings of the PairedTTester.ResultMatrix
getResultMatrix()
Gets the instance that produces the output.Range
getResultsetKeyColumns()
Get the value of ResultsetKeyColumns.java.lang.String
getResultsetName(int index)
Gets a string descriptive of the specified resultset.java.lang.String
getRevision()
Returns the revision string.int
getRunColumn()
Get the value of RunColumn.boolean
getShowStdDevs()
Returns true if standard deviations have been requested.double
getSignificanceLevel()
Get the value of SignificanceLevel.int
getSortColumn()
Returns the column to sort on, -1 means the default sorting.java.lang.String
getSortColumnName()
Returns the name of the column to sort on.java.lang.String
getToolTipText()
returns a string that is displayed as tooltip on the "perform test" button in the experimenterjava.lang.String
header(int comparisonColumn)
Creates a "header" string describing the current resultsets.java.util.Enumeration
listOptions()
Lists options understood by this object.static void
main(java.lang.String[] args)
Test the class from the command line.java.lang.String
multiResultsetFull(int baseResultset, int comparisonColumn)
Creates a comparison table where a base resultset is compared to the other resultsets.java.lang.String
multiResultsetRanking(int comparisonColumn)
returns a ranking of the resultsetsjava.lang.String
multiResultsetSummary(int comparisonColumn)
Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.int[][]
multiResultsetWins(int comparisonColumn, int[][] nonSigWin)
Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.java.lang.String
resultsetKey()
Creates a key that maps resultset numbers to their descriptions.void
setDatasetKeyColumns(Range newDatasetKeyColumns)
Set the value of DatasetKeyColumns.void
setDisplayedResultsets(int[] cols)
Sets the indicies of the datasets to display (null
means all).void
setFoldColumn(int newFoldColumn)
Set the value of FoldColumn.void
setInstances(Instances newInstances)
Set the value of Instances.void
setOptions(java.lang.String[] options)
Parses a given list of options.void
setResultMatrix(ResultMatrix matrix)
Sets the matrix to use to produce the output.void
setResultsetKeyColumns(Range newResultsetKeyColumns)
Set the value of ResultsetKeyColumns.void
setRunColumn(int newRunColumn)
Set the value of RunColumn.void
setShowStdDevs(boolean s)
Set whether standard deviations are displayed or not.void
setSignificanceLevel(double newSignificanceLevel)
Set the value of SignificanceLevel.void
setSortColumn(int newSortColumn)
Set the column to sort on, -1 means the default sorting.
-
-
-
Method Detail
-
setResultMatrix
public void setResultMatrix(ResultMatrix matrix)
Sets the matrix to use to produce the output.- Specified by:
setResultMatrix
in interfaceTester
- Parameters:
matrix
- the instance to use to produce the output- See Also:
ResultMatrix
-
getResultMatrix
public ResultMatrix getResultMatrix()
Gets the instance that produces the output.- Specified by:
getResultMatrix
in interfaceTester
- Returns:
- the instance to produce the output
-
setShowStdDevs
public void setShowStdDevs(boolean s)
Set whether standard deviations are displayed or not.- Specified by:
setShowStdDevs
in interfaceTester
- Parameters:
s
- true if standard deviations are to be displayed
-
getShowStdDevs
public boolean getShowStdDevs()
Returns true if standard deviations have been requested.- Specified by:
getShowStdDevs
in interfaceTester
- Returns:
- true if standard deviations are to be displayed.
-
getNumDatasets
public int getNumDatasets()
Gets the number of datasets in the resultsets- Specified by:
getNumDatasets
in interfaceTester
- Returns:
- the number of datasets in the resultsets
-
getNumResultsets
public int getNumResultsets()
Gets the number of resultsets in the data.- Specified by:
getNumResultsets
in interfaceTester
- Returns:
- the number of resultsets in the data
-
getResultsetName
public java.lang.String getResultsetName(int index)
Gets a string descriptive of the specified resultset.- Specified by:
getResultsetName
in interfaceTester
- Parameters:
index
- the index of the resultset- Returns:
- a descriptive string for the resultset
-
displayResultset
public boolean displayResultset(int index)
Checks whether the resultset with the given index shall be displayed.- Specified by:
displayResultset
in interfaceTester
- Parameters:
index
- the index of the resultset to check whether it shall be displayed- Returns:
- whether the specified resultset is displayed
-
calculateStatistics
public PairedStats calculateStatistics(Instance datasetSpecifier, int resultset1Index, int resultset2Index, int comparisonColumn) throws java.lang.Exception
Computes a paired t-test comparison for a specified dataset between two resultsets.- Specified by:
calculateStatistics
in interfaceTester
- Parameters:
datasetSpecifier
- the dataset specifierresultset1Index
- the index of the first resultsetresultset2Index
- the index of the second resultsetcomparisonColumn
- the column containing values to compare- Returns:
- the results of the paired comparison
- Throws:
java.lang.Exception
- if an error occurs
-
resultsetKey
public java.lang.String resultsetKey()
Creates a key that maps resultset numbers to their descriptions.- Specified by:
resultsetKey
in interfaceTester
- Returns:
- a value of type 'String'
-
header
public java.lang.String header(int comparisonColumn)
Creates a "header" string describing the current resultsets.
-
multiResultsetWins
public int[][] multiResultsetWins(int comparisonColumn, int[][] nonSigWin) throws java.lang.Exception
Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.- Specified by:
multiResultsetWins
in interfaceTester
- Parameters:
comparisonColumn
- the index of the comparison columnnonSigWin
- for storing the non-significant wins- Returns:
- a 2d array where element [i][j] is the number of times resultset j performed significantly better than resultset i.
- Throws:
java.lang.Exception
- if an error occurs
-
multiResultsetSummary
public java.lang.String multiResultsetSummary(int comparisonColumn) throws java.lang.Exception
Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other. The results are summarized in a table.- Specified by:
multiResultsetSummary
in interfaceTester
- Parameters:
comparisonColumn
- the index of the comparison column- Returns:
- the results in a string
- Throws:
java.lang.Exception
- if an error occurs
-
multiResultsetRanking
public java.lang.String multiResultsetRanking(int comparisonColumn) throws java.lang.Exception
returns a ranking of the resultsets- Specified by:
multiResultsetRanking
in interfaceTester
- Parameters:
comparisonColumn
- the column to compare with- Returns:
- the ranking
- Throws:
java.lang.Exception
- if something goes wrong
-
multiResultsetFull
public java.lang.String multiResultsetFull(int baseResultset, int comparisonColumn) throws java.lang.Exception
Creates a comparison table where a base resultset is compared to the other resultsets. Results are presented for every dataset.- Specified by:
multiResultsetFull
in interfaceTester
- Parameters:
baseResultset
- the index of the base resultsetcomparisonColumn
- the index of the column to compare over- Returns:
- the comparison table string
- Throws:
java.lang.Exception
- if an error occurs
-
listOptions
public java.util.Enumeration listOptions()
Lists options understood by this object.- Specified by:
listOptions
in interfaceOptionHandler
- Returns:
- an enumeration of Options.
-
setOptions
public void setOptions(java.lang.String[] options) throws java.lang.Exception
Parses a given list of options. Valid options are:-D <index,index2-index4,...> Specify list of columns that specify a unique dataset. First and last are valid indexes. (default none)
-R <index> Set the index of the column containing the run number
-F <index> Set the index of the column containing the fold number
-G <index1,index2-index4,...> Specify list of columns that specify a unique 'result generator' (eg: classifier name and options). First and last are valid indexes. (default none)
-S <significance level> Set the significance level for comparisons (default 0.05)
-V Show standard deviations
-L Produce table comparisons in Latex table format
-csv Produce table comparisons in CSV table format
-html Produce table comparisons in HTML table format
-significance Produce table comparisons with only the significance values
-gnuplot Produce table comparisons output suitable for GNUPlot
- Specified by:
setOptions
in interfaceOptionHandler
- Parameters:
options
- an array containing options to set.- Throws:
java.lang.Exception
- if invalid options are given
-
getOptions
public java.lang.String[] getOptions()
Gets current settings of the PairedTTester.- Specified by:
getOptions
in interfaceOptionHandler
- Returns:
- an array of strings containing current options.
-
getResultsetKeyColumns
public Range getResultsetKeyColumns()
Get the value of ResultsetKeyColumns.- Specified by:
getResultsetKeyColumns
in interfaceTester
- Returns:
- Value of ResultsetKeyColumns.
-
setResultsetKeyColumns
public void setResultsetKeyColumns(Range newResultsetKeyColumns)
Set the value of ResultsetKeyColumns.- Specified by:
setResultsetKeyColumns
in interfaceTester
- Parameters:
newResultsetKeyColumns
- Value to assign to ResultsetKeyColumns.
-
getDisplayedResultsets
public int[] getDisplayedResultsets()
Gets the indices of the the datasets that are displayed (ifnull
then all are displayed). The base is always displayed.- Specified by:
getDisplayedResultsets
in interfaceTester
- Returns:
- the indices of the datasets to display
-
setDisplayedResultsets
public void setDisplayedResultsets(int[] cols)
Sets the indicies of the datasets to display (null
means all). The base is always displayed.- Specified by:
setDisplayedResultsets
in interfaceTester
- Parameters:
cols
- the indices of the datasets to display
-
getSignificanceLevel
public double getSignificanceLevel()
Get the value of SignificanceLevel.- Specified by:
getSignificanceLevel
in interfaceTester
- Returns:
- Value of SignificanceLevel.
-
setSignificanceLevel
public void setSignificanceLevel(double newSignificanceLevel)
Set the value of SignificanceLevel.- Specified by:
setSignificanceLevel
in interfaceTester
- Parameters:
newSignificanceLevel
- Value to assign to SignificanceLevel.
-
getDatasetKeyColumns
public Range getDatasetKeyColumns()
Get the value of DatasetKeyColumns.- Specified by:
getDatasetKeyColumns
in interfaceTester
- Returns:
- Value of DatasetKeyColumns.
-
setDatasetKeyColumns
public void setDatasetKeyColumns(Range newDatasetKeyColumns)
Set the value of DatasetKeyColumns.- Specified by:
setDatasetKeyColumns
in interfaceTester
- Parameters:
newDatasetKeyColumns
- Value to assign to DatasetKeyColumns.
-
getRunColumn
public int getRunColumn()
Get the value of RunColumn.- Specified by:
getRunColumn
in interfaceTester
- Returns:
- Value of RunColumn.
-
setRunColumn
public void setRunColumn(int newRunColumn)
Set the value of RunColumn.- Specified by:
setRunColumn
in interfaceTester
- Parameters:
newRunColumn
- Value to assign to RunColumn.
-
getFoldColumn
public int getFoldColumn()
Get the value of FoldColumn.- Specified by:
getFoldColumn
in interfaceTester
- Returns:
- Value of FoldColumn.
-
setFoldColumn
public void setFoldColumn(int newFoldColumn)
Set the value of FoldColumn.- Specified by:
setFoldColumn
in interfaceTester
- Parameters:
newFoldColumn
- Value to assign to FoldColumn.
-
getSortColumnName
public java.lang.String getSortColumnName()
Returns the name of the column to sort on.- Specified by:
getSortColumnName
in interfaceTester
- Returns:
- the name of the column to sort on.
-
getSortColumn
public int getSortColumn()
Returns the column to sort on, -1 means the default sorting.- Specified by:
getSortColumn
in interfaceTester
- Returns:
- the column to sort on.
-
setSortColumn
public void setSortColumn(int newSortColumn)
Set the column to sort on, -1 means the default sorting.- Specified by:
setSortColumn
in interfaceTester
- Parameters:
newSortColumn
- the new sort column.
-
getInstances
public Instances getInstances()
Get the value of Instances.- Specified by:
getInstances
in interfaceTester
- Returns:
- Value of Instances.
-
setInstances
public void setInstances(Instances newInstances)
Set the value of Instances.- Specified by:
setInstances
in interfaceTester
- Parameters:
newInstances
- Value to assign to Instances.
-
assign
public void assign(Tester tester)
retrieves all the settings from the given Tester
-
getToolTipText
public java.lang.String getToolTipText()
returns a string that is displayed as tooltip on the "perform test" button in the experimenter- Specified by:
getToolTipText
in interfaceTester
- Returns:
- the tool tip
-
getDisplayName
public java.lang.String getDisplayName()
returns the name of the tester- Specified by:
getDisplayName
in interfaceTester
- Returns:
- the display name
-
getRevision
public java.lang.String getRevision()
Returns the revision string.- Specified by:
getRevision
in interfaceRevisionHandler
- Returns:
- the revision
-
main
public static void main(java.lang.String[] args)
Test the class from the command line.- Parameters:
args
- contains options for the instance ttests
-
-