SKEWNESS OUTLIER TEST

SKEWNESS OUTLIER TEST

Analysis Command

Perform the skewness test for univariate outliers from a normal distribution.

The ASTM E178-16a standard for detecting outliers from a univariate normal distribution includes the skewness outlier test.

The test statistic is the adjusted Fisher-Pearson skewness coefficient

\[ g_1 = \frac{n \sum_{i=1}^{n}{(x_{i} - \bar{x})^2}} {(n-1) (n-2) s^2} \]

with n, \( \bar{x} \) and s denoting the sample size, the sample mean and the sample standard deviation, respectively.

The critical values are obtained via simulation. The ASTM standard provides table values for n = 3 to 50 and \( \alpha \) levels of 0.10, 0.05 and 0.01. Linear interpolation is used for values of n not given in the table. Alternatively, you can perform a dynamic simulation to obtain the critical values.

To specify the method used to compute the critical value, enter one of the following commands (the default is ASTM)

SET SKEW OUTLIER TEST CRITICAL VALUES ASTM
SET SKEW OUTLIER TEST CRITICAL VALUES SIMULATION


If n > 50, the simulation method will be used.

SKEWNESS OUTLIER TEST <y>
                        <SUBSET/EXCEPT/FOR qualification>
where <y> is the response variable being tested;
and where the <SUBSET/EXCEPT/FOR qualification> is optional.

MULTIPLE SKEWNESS OUTLIER TEST <y1> ... <yk>
                        <SUBSET/EXCEPT/FOR qualification>
where <y1> ... <yk> is a list of up to k response variables;
and where the <SUBSET/EXCEPT/FOR qualification> is optional.

This syntax performs the skewness outlier test on <y1>, then on <y2>, and so on. Up to 30 response variables can be specified.

Note that the syntax

MULTIPLE SKEWNESS OUTLIER TEST Y1 TO Y4

is supported. This is equivalent to

MULTIPLE SKEWNESS OUTLIER TEST Y1 Y2 Y3 Y4

REPLICATED SKEWNESS OUTLIER TEST <y> <x1> ... <xk>
                        <SUBSET/EXCEPT/FOR qualification>
where <y> is the response variable;
            <x1> ... <xk> is a list of up to k group-id variables;
and where the <SUBSET/EXCEPT/FOR qualification> is optional.

This syntax performs a cross-tabulation of <x1> ... <xk> and performs a skewness outlier test for each unique combination of cross-tabulated values. For example, if X1 has 3 levels and X2 has 2 levels, there will be a total of 6 skewness outlier tests performed.

Up to six group-id variables can be specified.

Note that the syntax

REPLICATED SKEWNESS OUTLIER TEST Y X1 TO X4

is supported. This is equivalent to

REPLICATED SKEWNESS OUTLIER TEST Y X1 X2 X3 X4

SKEWNESS OUTLIER TEST Y1
MULTIPLE SKEWNESS OUTLIER TEST Y1 Y2 Y3
REPLICATED SKEWNESS OUTLIER TEST Y X1 X2
SKEWNESS OUTLIER TEST Y1 SUBSET TAG > 2

Tests for outliers are dependent on knowing the distribution of the data. The skewness outlier test assumes that the data come from an approximately normal distribution. For this reason, it is strongly recommended that the skewness outlier test be complemented with a normal probability test. If the data are not approximately normally distributed, then the skewness outlier test may be detecting the non-normality of the data rather than the presence of an outlier.

You can specify the number of digits in the skewness outlier test output with the command
SET WRITE DECIMALS <value>

The SKEWNESS OUTLIER TEST command automatically saves the following parameters:

STATVAL	=	the value of the test statistic
STATDCF	=	the CDF value of the test statistic
PVALUE	=	the p-value of the test statistic
CUTOFF80	=	the 80 percent point of the reference distribution
CUTOFF90	=	the 90 percent point of the reference distribution
CUTOFF95	=	the 95 percent point of the reference distribution
CUTOF975	=	the 97.5 percent point of the reference distribution
CUTOFF99	=	= the 99 percent point of the reference distribution

The STATCDF and PVALUE are only saved when the simulation method is used to obtain critical values. If the ASTM method is used to obtain critical values, the CUTOFF80 and CUTOF975 values are not saved.

If the MULTIPLE or REPLICATED option is used, these values will be written to the file "dpst1f.dat" instead.

In addition to the SKEWNESS OUTLIER TEST command, the following commands can also be used:
LET A = SKEWNESS OUTLIER TEST Y
LET A = SKEWNESS OUTLIER TEST CDF Y
LET A = SKEWNESS OUTLIER TEST PVALUE Y
LET A = SKEWNESS OUTLIER TEST INDEX Y

LET ALPHA = <value>
LET A = SKEWNESS OUTLIER TEST CRITICAL VALUE Y


The SKEWNESS OUTLIER TEST, SKEWNESS OUTLIER TEST CDF, and SKEWNESS OUTLIER TEST PVALUE return the values of the test statistic, the cdf of the test statistic and the pvalue of the test statistic, respectively. For the SKEWNESS OUTLIER TEST CDF and SKEWNESS OUTLIER TEST PVALUE commands, the simulation method will be used. Otherwise, the method specified by the SET SKEWNESS OUTLIER TEST CRITICAL VALUE command will be used.

The SKEWNESS OUTLIER TEST INDEX returns the row index of the most extreme value in the response variable. The most extreme value is defined as the value furtherest from the mean.

The SKEWNESS OUTLIER TEST CRITICAL VALUE returns the critical value for the specified value of ALPHA. If ALPHA is not specified, it will be set to 0.05. Note that if the ASTM method is specified for the critical values, only a few select values for alpha are supported (0.01, 0.05 and 0.10).

In addition to the above LET command, built-in statistics are supported for 30+ different commands (enter HELP STATISTICS for details).

The ASTM method is used to obtain critical values

None

KURTOSIS OUTLIER TEST	=	Perform the kurtosis outlier test.
DAVID TEST	=	Perform the David outlier test.
GRUBBS TEST	=	Perform the Grubbs outlier test.
TIETJEN-MOORE TEST	=	Perform a Tietjen-Moore outlier test.
EXTREME STUDENTIZED DEVIATE TEST	=	Perform a extreme studentized deviate outlier test.
DIXON TEST	=	Perform a Dixon outlier test.
GOODNESS OF FIT TEST	=	Perform a goodness of fit test (Anderson-Darling, Kolmogorov-Smirnov, chi-square, PPCC)
WILKS SHAPIRO NORMALITY TEST	=	Perform a Wilks Shapiro normality test.
HISTOGRAM	=	Generate a histogram.
PROBABILITY PLOT	=	Generates a probability plot.
BOX PLOT	=	Generate a box plot.

E178 - 16A (2016), "Standard Practice for Dealing with Outlying Observations", ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959, USA.

Ferguson, T.S. (1961), "On the Rejection of Outliers," Fourth Berkeley Symposium on Mathematical Statistics and Probability, edited by Jerzy Neyman, University of California Press, Berkeley and Los Angeles, CA.

Ferguson, T.S. (1961), "Rules for Rejection of Outliers," Revue Inst. Int. de Stat., RINSA, Vol. 29, No. 3, pp. 29-43.

Outlier Detection

2019/10

 
. Step 1:   Read the data (from ASTM E-178 document)
.
read y
3.73
3.59
3.94
4.13
3.04
2.22
3.23
4.05
4.11
2.02
end of data
set write decimals 3
.
. Step 2:   Compute the statistics
.
let stat = skew outlier test y
set skew outlier test critical values astm
let cv1 = skew outlier test critical value y
set skew outlier test critical values simulation
let cv2 = skew outlier test critical value y
.
let pval = skew outlier test pvalue y
let statcdf = skew outlier test cdf y
let iindx = skew outlier test index y
.
print stat cv1 cv2 pval statcdf iindx
.
set skew outlier test critical values astm
skewness outlier test y
set skew outlier test critical values simulation
skewness outlier test y
    

The following output is generated

 PARAMETERS AND CONSTANTS--

    STAT    --         -0.969
    CV1     --         -1.131
    CV2     --         -1.139
    PVAL    --          0.079
    STATCDF --          0.922
    IINDX   --         10.000
 
THE FORTRAN COMMON CHARACTER VARIABLE SKEWOUTL HAS JUST BEEN SET TO ASTM
 
            Skewness Test for Outliers
             (Assumption: Normality)
 
Response Variable: Y
 
H0: The most extreme point is not
    an outlier
Ha: The most extreme point is not
    an outlier
Potential outlier value tested:                   2.020
ID for potential outlier:                            10
 
Summary Statistics:
Number of Observations:                              10
Sample Minimum:                                   2.020
Sample Maximum:                                   4.130
Sample Mean:                                      3.406
Sample SD:                                        0.771
Sample Adjusted Skewness:                        -0.969
 
Skewness Outlier Test Statistic Value:           -0.969
 
 
Conclusions (Upper 1-Tailed Test)
-------------------------------------------------------------
  Alpha    CDF      Statistic   Critical Value     Conclusion
-------------------------------------------------------------
    10%    90%         -0.969           -0.862      Accept H0
     5%    95%         -0.969           -1.131      Reject H0
     1%    99%         -0.969           -1.668      Reject H0
 
 
 
Critical Values Based on ASTM E-178 Tables
 
 
THE FORTRAN COMMON CHARACTER VARIABLE SKEWOUTL HAS JUST BEEN SET TO SIMU
 
            Skewness Test for Outliers
             (Assumption: Normality)
 
Response Variable: Y
 
H0: The most extreme point is not
    an outlier
Ha: The most extreme point is not
    an outlier
Potential outlier value tested:                   2.020
ID for potential outlier:                            10
 
Summary Statistics:
Number of Observations:                              10
Sample Minimum:                                   2.020
Sample Maximum:                                   4.130
Sample Mean:                                      3.406
Sample SD:                                        0.771
Sample Adjusted Skewness:                        -0.969
 
Skewness Outlier Test Statistic Value:           -0.969
CDF Value:                                        0.923
P-Value                                           0.077
 
 
 
Conclusions (Upper 1-Tailed Test)
-------------------------------------------------------------
  Alpha    CDF      Statistic   Critical Value     Conclusion
-------------------------------------------------------------
    20%    80%         -0.969           -0.557      Accept H0
    10%    90%         -0.969           -0.862      Accept H0
     5%    95%         -0.969           -1.133      Reject H0
   2.5%  97.5%         -0.969           -1.385      Reject H0
     1%    99%         -0.969           -1.671      Reject H0
   0.5%  99.5%         -0.969           -1.864      Reject H0
 
 
 
Critical Values Based on 50,000 Simulations