Exploratory Data Analysis
Estimating the Parameters of a Distribution
Maximum likelihood estimation begins with the mathematical
expression known as a likelihood function of the sample data.
Loosely speaking, the likelihood of a set of data is the probability
of obtaining that particular set of data given the chosen probability
model. This expression contains the unknown parameters. Those
values of the parameter that maximize the sample likelihood are known
as the maximum likelihood estimates.
The reliability chapter
contains some examples of the likelihood functions for a few of the
commonly used distributions in reliability analysis.
The advantages of this method are:
- Maximum likelihood provides a consistent approach to
parameter estimation problems. This means that maximum
likelihood estimates can be developed for a large variety
of estimation situations. For example, they can be applied
in reliability analysis to censored data under various
- Maximum likelihood methods have desirable mathematical
and optimality properties. Specifically,
- They become minimum variance unbiased estimators as
the sample size increases. By unbiased, we mean
that if we take (a very large number of) random samples
with replacement from a population, the average value of
the parameter estimates will be theoretically exactly
equal to the population value. By minimum variance, we
mean that the estimator has the smallest variance, and
thus the narrowest confidence interval, of all estimators
of that type.
- They have approximate normal distributions and
approximate sample variances that can be used to
generate confidence bounds and hypothesis tests
for the parameters.
- Several popular statistical software packages provide
excellent algorithms for maximum likelihood estimates
for many of the commonly used distributions. This helps
mitigate the computational complexity of maximum likelihood
The disadvantages of this method are:
- The likelihood equations need to be specifically worked
out for a given distribution and estimation problem. The
mathematics is often non-trivial, particularly if confidence
intervals for the parameters are desired.
- The numerical estimation is usually non-trivial. Except for
a few cases where the maximum likelihood formulas are
in fact simple, it is generally best to rely on
high quality statistical software to obtain maximum likelihood
estimates. Fortunately, high quality maximum likelihood
software is becoming increasingly common.
- Maximum likelihood estimates can be heavily biased for small
samples. The optimality properties may not apply for small
- Maximum likelihood can be sensitive to the choice of starting
Most general purpose statistical software programs support
maximum likelihood estimation (MLE) in some form. MLE
estimation can be supported in two ways.
The advantage of function minimization software is that it can be
applied to many different MLE problems. The drawback is that
you have to specify the maximum likelihood equations to the
software. As the functions can be non-trivial, there is potential
for error in entering the equations.
- A software program may provide a generic function
minimization (or equivalently, maximization) capability.
This is also referred to as function optimization.
Maximum likelihood estimation is essentially a function
This type of capability is particularly common in mathematical
- A software program may provide MLE computations for a
specific problem. For example, it may generate ML estimates
for the parameters of a Weibull distribution.
Statistical software programs will often provide ML estimates
for many specific problems even when they do not support
general function optimization.
The advantage of the specific MLE procedures is that greater
efficiency and better numerical stability can often be obtained
by taking advantage of the properties of the specific estimation
problem. The specific methods often return explicit confidence
intervals. In addition, you do not have to know or specify the
likelihood equations to the software. The disadvantage is that each
MLE problem must be specifically coded.