Dataplot Vol 1 Vol 2

MEDIAN CONFIDENCE LIMITS

Name:
MEDIAN CONFIDENCE LIMITS
Type:
Analysis Command
Purpose:
Generates a median based confidence interval for the location of a variable.
Description:
Mosteller and Tukey (see Reference section below) define two types of robustness:

1. resistance means that changing a small part, even by a large amount, of the data does not cause a large change in the estimate

2. robustness of efficiency means that the statistic has high efficiency in a variety of situations rather than in any one situation. Efficiency means that the estimate is close to optimal estimate given that we distribution that the data comes from. A useful measure of efficiency is:

Efficiency = (lowest variance feasible)/ (actual variance)

Standard confidence intervals are based on the mean and variance. These are the optimal estimators if the data are in fact from a Gaussian population. However, the mean lacks both resistance and robustness of efficiency. The median is less affected by outliers (i.e., resistance) than the mean. However, the median is not particularly robust with regards to efficiency.

Dataplot generates confidence intervals for the median using the following two methods:

1. Method 1 is the Hettmansperger-Sheather interpolation method. The steps in this method are:
• Suppose W is a binomial random variable with n trials and a probability of success p = 0.5. For any integer, k, between 0 and [n/2], let $$\gamma_{k} = P(k \le W \le n - k)$$.

A 95% confidence interval for the median is:

(Xk, Xn-k+1)

with X denoting the sorted observations.

• Determine k such that $$\gamma_{k+1} < 1 - \alpha < \gamma_{k}$$.

• Compute

$$I = \frac{\gamma_k -1 - \alpha}{\gamma_k - \gamma_{k+1}}$$

and

$$\lambda = \frac{(n-k)I}{k + (n - 2k)I}$$

• An approximate (1-$$\alpha$$) confidence interval is

$$LCL = \lambda X_{k+1} + (1 - \lambda) X_k$$
$$UCL = \lambda X_{n-k} + (1 - \lambda) X_{n-k+1}$$

2. Method 2, based on a method given by Wilcox (see Reference below) on page 87, is based on the Maritz-Jarrett estimate of the standard error for a quantile. Specifically,

$$\hat{x}_{q} \pm \Phi^{-1}(1-\alpha/2)\hat{\sigma}_{mj}$$

where

q = the desired quantile (q = 0.5 for the median)

$$\hat{x}$$ = the estimated sample quantile

$$\Phi^{-1}$$ = the percent point function of the standard normal distribution

$$\alpha$$ = the significance level

$$\hat{\sigma}_{mj}$$ = the quantile standard error based on the Maritz-Jarrett method

Note that this method can be applied to quantiles other than the median. However, the accuracy of this method has not been studied for quantiles other than 0.5.

Syntax 1:
MEDIAN CONFIDENCE LIMITS <y>       <SUBSET/EXCEPT/FOR qualification>
where <y> is the response variable,
and where the <SUBSET/EXCEPT/FOR qualification> is optional.
Syntax 2:
QUANTILE CONFIDENCE LIMITS <y>
<SUBSET/EXCEPT/FOR qualification>
where <y> is the response variable, and where the <SUBSET/EXCEPT/FOR qualification> is optional.
Examples:
MEDIAN CONFIDENCE LIMITS Y1

LET P100 = 0.25
QUANTILE CONFIDENCE LIMITS Y1 SUBSET TAG = 2

Note:
For quantiles other than the median, the desired quantile is specified with the LET command. Specifically, define the parameter P100. For example,

LET P100 = 0.25

Only the method based on the Maritz-Jarrett standard error is supported for quantiles other than the median.

Note:
A table of confidence intervals is printed for alpha levels of 50.0, 75.0, 90.0, 95.0, 99.0, 99.9, 99.99, and 99.999.
Note:
Alternative methods for generating confidence intervals for medians or quantiles are available.

1. You can use the BOOTSTRAP MEDIAN PLOT or the BOOTSTRAP QUANTILE PLOT command. For example,
LET Y = CAUCHY RANDOM NUMBERS FOR I = 1 1 100
BOOTSTRAP MEDIAN PLOT Y
LET LCL = B025
LET UCL = B975
.
LET XQ = 0.75
BOOTSTRAP QUANTILE PLOT Y
LET LCL = B025
LET UCL = B975

2. Wilcox suggests the following method for quantiles (the median is a special case with the quantile = 0.5) on page 86.

$$\hat{\theta} = \pm \; \hat{c} \; \hat{\sigma}_{hd}$$

where

$$\hat{\theta}$$ = the Herrell-Davis quantile estimate

$$\hat{\sigma}_{hd}$$ = the bootstrap estimate of the Herrell-Davis quantile standard error

$$\hat{c}$$ = 0.5064*N**(-0.25) + 1.96 for N >= 11 and 0.3 <= q <= 0.7
for N > 21 and 0.2 <= q <= 0.8
for N > 41 0.1 ≤ q ≤ 0.9
= -6.23*(1/N) + 5.01 for 11 ≤ N ≤ 21, q = 0.2, 0.8
= 36.2*(1/N) + 1.31 for N > 41, q = 0.1, 0.9

This can be coded in the following Dataplot macro:

SET QUANTILE METHOD HERRELL DAVIS
LET P100 = 0.5
LET THETAHAT = QUANTILE Y
BOOTSTRAP QUANTILE STANDARD ERROR PLOT Y
LET SIGMAHAT = B50
LET N = SIZE Y
IF N < 11
QUIT
END OF IF
LET C = 0.5064*N**(-0.25) + 1.96
LET IQFLAG = 1
IF P100 <= 0.19
IF N > 41
LET C = 36.2*(1/N) + 1.31
END OF IF
ELSEIF P100 <= 0.29
IF N <= 21
LET C = -6.23*(1/N) + 5.01
END OF IF
ELSE IF P100 >= 0.81
IF N > 41
LET C = 36.2*(1/N) + 1.31
END OF IF
ELSE IF P100 >= 0.71
IF N <= 21
LET C = -6.23*(1/N) + 5.01
END OF IF
ENDIF
LET LOWLIMIT = THETAHAT - C*B50
LET UPPLIMIT = THETAHAT + C*B50

Default:
None
Synonyms:
MEDIAN CONFIDENCE INTERVAL
Related Commands:
 MEDIAN = Compute the median. MEDIAN PLOT = Generate a median (versus subset) plot. BOOTSTRAP PLOT = Generate a bootstrap plot. CONFIDENCE LIMITS = Compute a Gaussian based confidence limit. BIWEIGHT CONFIDENCE LIMITS = Compute a biweight location based confidence limit. TRIMMED MEAN CONFIDENCE LIMITS = Compute a trimmed mean based confidence limit. T-TEST = Perform a t-test.
Reference:
Rand Wilcox (1997), "Introduction to Robust Estimation and Hypothesis Testing," Academic Press, p. 87.

T. P. Hettmansperger and S. J. Sheather (1986), "Confidence Interval Based on Interpolated Order Statistics," Statistical Probability Letters 4, pp. 75-79.

Applications:
Robust Data Analysis
Implementation Date:
2003/2
Program:

LET Y1 = NORMAL RANDOM NUMBERS FOR I = 1 1 100
LET Y2 = LOGISTIC RANDOM NUMBERS FOR I = 1 1 100
LET Y3 = CAUCHY RANDOM NUMBERS FOR I = 1 1 100
LET Y4 = DOUBLE EXPONENTIAL RANDOM NUMBERS FOR I = 1 1 100
SET WRITE DECIMALS 4
MEDIAN CONFIDENCE LIMITS Y1 TO Y4

Dataplot generates the following output:
             Confidence Limits for the Median
(Based on Maritz-Jarrett Standard Error for Quantiles)

Response Variable: Y1

Summary Statistics:
Number of Observations:                  100
Sample Minimum:                          -3.4580
Sample Maximum:                          2.0059
Sample Median:                           0.0015
Sample Quantile Standard Error:          0.1119

-----------------------------------------------------------------
Confidence       Z      Z-Value X          Lower          Upper
Value (%)   Value         StdErr          Limit          Limit
-----------------------------------------------------------------
50.000   0.674         0.0755        -0.0739         0.0770
75.000   1.150         0.1287        -0.1272         0.1302
90.000   1.645         0.1840        -0.1825         0.1856
95.000   1.960         0.2193        -0.2177         0.2208
99.000   2.576         0.2882        -0.2866         0.2897
99.900   3.291         0.3682        -0.3666         0.3697
99.990   3.891         0.4353        -0.4337         0.4368
99.999   4.417         0.4942        -0.4927         0.4957

Hettmansperger-Sheater Median Confidence Limits

-----------------------------------
Confidence   Lower          Upper
Value (%)   Limit          Limit
-----------------------------------
50.000  -0.064         0.0410
75.000  -0.102         0.1053
90.000  -0.132         0.2318
95.000  -0.187         0.2382
99.000  -0.352         0.2681
99.900  -0.383         0.3789
99.990  -0.446         0.4064
99.999  -0.467         0.4250

Confidence Limits for the Median
(Based on Maritz-Jarrett Standard Error for Quantiles)

Response Variable: Y2

Summary Statistics:
Number of Observations:                  100
Sample Minimum:                          -5.0249
Sample Maximum:                          5.3818
Sample Median:                           0.1507
Sample Quantile Standard Error:          0.2162

-----------------------------------------------------------------
Confidence       Z      Z-Value X          Lower          Upper
Value (%)   Value         StdErr          Limit          Limit
-----------------------------------------------------------------
50.000   0.674         0.1458         0.0048         0.2965
75.000   1.150         0.2487        -0.0981         0.3994
90.000   1.645         0.3556        -0.2050         0.5063
95.000   1.960         0.4238        -0.2731         0.5744
99.000   2.576         0.5569        -0.4063         0.7076
99.900   3.291         0.7115        -0.5608         0.8621
99.990   3.891         0.8412        -0.6905         0.9919
99.999   4.417         0.9551        -0.8044         1.1057

Hettmansperger-Sheater Median Confidence Limits

-----------------------------------
Confidence   Lower          Upper
Value (%)   Limit          Limit
-----------------------------------
50.000  -0.032         0.3550
75.000  -0.048         0.5086
90.000  -0.069         0.5268
95.000  -0.105         0.5454
99.000  -0.139         0.5622
99.900  -0.543         0.7184
99.990  -0.779         0.8084
99.999  -1.021         0.9848

Confidence Limits for the Median
(Based on Maritz-Jarrett Standard Error for Quantiles)

Response Variable: Y3

Summary Statistics:
Number of Observations:                  100
Sample Minimum:                          -27.0517
Sample Maximum:                          8.6177
Sample Median:                           0.0212
Sample Quantile Standard Error:          0.1866

-----------------------------------------------------------------
Confidence       Z      Z-Value X          Lower          Upper
Value (%)   Value         StdErr          Limit          Limit
-----------------------------------------------------------------
50.000   0.674         0.1258        -0.1046         0.1470
75.000   1.150         0.2146        -0.1934         0.2358
90.000   1.645         0.3069        -0.2856         0.3281
95.000   1.960         0.3656        -0.3444         0.3869
99.000   2.576         0.4805        -0.4593         0.5018
99.900   3.291         0.6139        -0.5927         0.6351
99.990   3.891         0.7258        -0.7046         0.7470
99.999   4.417         0.8241        -0.8028         0.8453

Hettmansperger-Sheater Median Confidence Limits

-----------------------------------
Confidence   Lower          Upper
Value (%)   Limit          Limit
-----------------------------------
50.000  -0.086         0.1580
75.000  -0.158         0.2898
90.000  -0.225         0.3791
95.000  -0.333         0.4389
99.000  -0.380         0.4515
99.900  -0.412         0.6212
99.990  -0.485         0.8956
99.999  -0.683         0.9482

Confidence Limits for the Median
(Based on Maritz-Jarrett Standard Error for Quantiles)

Response Variable: Y4

Summary Statistics:
Number of Observations:                  100
Sample Minimum:                          -9.6504
Sample Maximum:                          3.0304
Sample Median:                           0.0233
Sample Quantile Standard Error:          0.0698

-----------------------------------------------------------------
Confidence       Z      Z-Value X          Lower          Upper
Value (%)   Value         StdErr          Limit          Limit
-----------------------------------------------------------------
50.000   0.674         0.0471        -0.0238         0.0703
75.000   1.150         0.0803        -0.0570         0.1036
90.000   1.645         0.1148        -0.0915         0.1381
95.000   1.960         0.1368        -0.1135         0.1601
99.000   2.576         0.1798        -0.1565         0.2030
99.900   3.291         0.2296        -0.2064         0.2529
99.990   3.891         0.2715        -0.2482         0.2948
99.999   4.417         0.3083        -0.2850         0.3315

Hettmansperger-Sheater Median Confidence Limits

-----------------------------------
Confidence   Lower          Upper
Value (%)   Limit          Limit
-----------------------------------
50.000  -0.032         0.0478
75.000  -0.063         0.0691
90.000  -0.083         0.1003
95.000  -0.137         0.1559
99.000  -0.213         0.2189
99.900  -0.312         0.3046
99.990  -0.389         0.4273
99.999  -0.479         0.4552


NIST is an agency of the U.S. Commerce Department.

Date created: 02/26/2003
Last updated: 11/04/2015