Data Analysis for Process Modeling
How are estimates of the unknown parameters obtained?
Parameter Estimation in General
After selecting the basic form of the functional part of the model, the next
step in the model-building process is estimation of the unknown parameters in
the function. In general, this is accomplished by solving an optimization
problem in which the objective function (the function being minimized or
maximized) relates the response variable and the functional part of the model
containing the unknown parameters in a way that will produce parameter
estimates that will be close to the true, unknown parameter values. The
unknown parameters are, loosely speaking, treated as variables to be solved
for in the optimization, and the data serve as known coefficients of the
objective function in this stage of the modeling process.
In theory, there are as many different ways of estimating parameters as
there are objective functions to be minimized or maximized. However, a few
principles have dominated because they result in parameter estimators that have
good statistical properties. The two major methods of parameter estimation
for process models are maximum likelihood and least squares. Both of these
methods provide parameter estimators that have many good properties.
Both maximum likelihood and least squares are sensitive to the presence of
outliers, however. There are also many newer methods of parameter estimation,
called robust methods, that try to balance the efficiency and desirable
properties of least squares and maximum likelihood with a lower sensitivity
Overview of Section 4.3
Although robust techniques are valuable, they are not as well developed as the
more traditional methods and often require specialized software that is not
readily available. Maximum likelihood also requires specialized algorithms in
general, although there are important special cases that do not have such a
requirement. For example, for data with normally distributed random errors,
the least squares and maximum likelihood parameter estimators are identical.
As a result of these software and developmental issues, and the coincidence of
maximum likelihood and least squares in many applications, this section
currently focuses on parameter estimation only by least squares methods. The
remainder of this section offers some intuition into how least squares works
and illustrates the effectiveness of this method.
Contents of Section 4.3
- Least Squares
- Weighted Least Squares