Data Collection for Process Modeling
How can I tell if a particular experimental design is good for my application?
Assess Relative to the Six Design Principles
If you have a design, generated by whatever method, in hand,
how can you assess its after-the-fact goodness? Such checks can
potentially parallel the list of the
six general design principles.
The design can be assessed relative to each of these six principles.
For example, does it have capacity for the primary model, does it
have capacity for an alternative model, etc.
Some of these checks are quantitative and complicated; other checks
are simpler and graphical. The graphical checks are the most easily
done and yet are among the most informative. We include two such
graphical checks and one quantitative check.
Graphically Check for Univariate Balance
If you have a design that claims to be globally good in \(k\)
then generally that design should be locally good in each of the
factors. Checking high-dimensional global goodness is
difficult, but checking low-dimensional local goodness is easy.
counts plots, with the levels of factors \(x_i\)
plotted on the horizontal axis of each plot and the number of
design points for each level in factor \(x_i\)
on the vertical axis. For most good designs, these counts should be
about the same (equal balance) for all levels of a factor.
Exceptions exist, but such balance is a low-level characteristic of most
Graphically Check for Bivariate Balance
If you have a design that is purported to be globally good in \(k\)
factors, then generally that design should be locally good
in all pairs of the individual \(k\)
factors. Graphically check for such 2-way balance by generating
plots for all pairs of factors, where the horizontal axis of a
given plot is \(x_i\)
and the vertical axis is \(x_j\).
The response variable \(y\)
does NOT come into play in these plots. We are only interested in
characteristics of the design, and so only the \(x\)
variables are involved. The 2-way plots of most good designs
have a certain symmetric and balanced look about them--all
combination points should be covered and each combination point
should have about the same number of points.
Check for Minimal Variation
For optimal designs, metrics exist (D-efficiency, A-efficiency, etc.)
that can be computed and that reflect the quality of the design.
Further, relative ratios of standard deviations of the coefficient
estimators and relative ratios of predicted values can be computed
and compared for such designs. Such calculations are commonly
performed in computer packages which specialize in the generation of