Design matrix

In statistics, a design matrix (also known as regressor matrix or model matrix) is a matrix of values of explanatory variables of a set of objects, often denoted by X. Each row represents an individual object, with the successive columns corresponding to the variables and their specific values for that object. The design matrix is used in certain statistical models, e.g., the general linear model.^[1]^[2]^[3] It can contain indicator variables (ones and zeros) that indicate group membership in an ANOVA, or it can contain values of continuous variables.

The design matrix contains data on the independent variables (also called explanatory variables) in statistical models which attempt to explain observed data on a response variable (often called a dependent variable) in terms of the explanatory variables. The theory relating to such models makes substantial use of matrix manipulations involving the design matrix: see for example linear regression. A notable feature of the concept of a design matrix is that it is able to represent a number of different experimental designs and statistical models, e.g., ANOVA, ANCOVA, and linear regression.

Definition

The design matrix is defined to be a matrix X such that the j^th column of the i^th row of X represents the value of the j^th variable associated with the i^th object.

A regression model which is a linear combination of the explanatory variables may therefore be represented via matrix multiplication as

y=X\beta ,

where X is the design matrix, $\beta$ is a vector of the model's coefficients (one for each variable), and y is the vector of predicted outputs for each object.

Examples

Simple Regression

This section gives an example of simple linear regression—that is, regression with only a single explanatory variable—with seven observations. The seven data points are {y_i, x_i}, for i = 1, 2, …, 7. The simple linear regression model is

y_{i}=\beta _{0}+\beta _{1}x_{i}+\epsilon _{i},\,

where $\beta _{0}$ is the y-intercept and $\beta _{1}$ is the slope of the regression line. This model can be represented in matrix form as

{\begin{bmatrix}y_{1}\\y_{2}\\y_{3}\\y_{4}\\y_{5}\\y_{6}\\y_{7}\end{bmatrix}}={\begin{bmatrix}1&x_{1}\\1&x_{2}\\1&x_{3}\\1&x_{4}\\1&x_{5}\\1&x_{6}\\1&x_{7}\end{bmatrix}}{\begin{bmatrix}\beta _{0}\\\beta _{1}\end{bmatrix}}+{\begin{bmatrix}\epsilon _{1}\\\epsilon _{2}\\\epsilon _{3}\\\epsilon _{4}\\\epsilon _{5}\\\epsilon _{6}\\\epsilon _{7}\end{bmatrix}}

where the first column of ones in the design matrix allows estimation of the y-intercept while the second column contains the x-values associated with the corresponding y-values.

Multiple Regression

This section contains an example of multiple regression with two covariates (explanatory variables): w and x. Again suppose that the data consist of seven observations, and that for each observed value to be predicted ( $y_{i}$ ), values w_i and x_i of the two covariates are also observed. The model to be considered is

y_{i}=\beta _{0}+\beta _{1}w_{i}+\beta _{2}x_{i}+\epsilon _{i}

This model can be written in matrix terms as

{\begin{bmatrix}y_{1}\\y_{2}\\y_{3}\\y_{4}\\y_{5}\\y_{6}\\y_{7}\end{bmatrix}}={\begin{bmatrix}1&w_{1}&x_{1}\\1&w_{2}&x_{2}\\1&w_{3}&x_{3}\\1&w_{4}&x_{4}\\1&w_{5}&x_{5}\\1&w_{6}&x_{6}\\1&w_{7}&x_{7}\end{bmatrix}}{\begin{bmatrix}\beta _{0}\\\beta _{1}\\\beta _{2}\end{bmatrix}}+{\begin{bmatrix}\epsilon _{1}\\\epsilon _{2}\\\epsilon _{3}\\\epsilon _{4}\\\epsilon _{5}\\\epsilon _{6}\\\epsilon _{7}\end{bmatrix}}

Here the 7×3 matrix on the right side is the design matrix.

One-way ANOVA (Cell Means Model)

This section contains an example with a one-way analysis of variance (ANOVA) with three groups and seven observations. The given data set has the first three observations belonging to the first group, the following two observations belonging to the second group and the final two observations belonging to the third group. If the model to be fit is just the mean of each group, then the model is

y_{ij}=\mu _{i}+\epsilon _{ij}

which can be written

{\begin{bmatrix}y_{1}\\y_{2}\\y_{3}\\y_{4}\\y_{5}\\y_{6}\\y_{7}\end{bmatrix}}={\begin{bmatrix}1&0&0\\1&0&0\\1&0&0\\0&1&0\\0&1&0\\0&0&1\\0&0&1\end{bmatrix}}{\begin{bmatrix}\mu _{1}\\\mu _{2}\\\mu _{3}\end{bmatrix}}+{\begin{bmatrix}\epsilon _{1}\\\epsilon _{2}\\\epsilon _{3}\\\epsilon _{4}\\\epsilon _{5}\\\epsilon _{6}\\\epsilon _{7}\end{bmatrix}}

It should be emphasized that in this model $\mu _{i}$ represents the mean of the $i$ th group.

One-way ANOVA (offset from reference group)

The ANOVA model could be equivalently written as each group parameter $\tau _{i}$ being an offset from some overall reference. Typically this reference point is taken to be one of the groups under consideration. This makes sense in the context of comparing multiple treatment groups to a control group and the control group is considered the "reference". In this example, group 1 was chosen to be the reference group. As such the model to be fit is

y_{ij}=\mu +\tau _{i}+\epsilon _{ij}

with the constraint that $\tau _{1}$ is zero.

{\begin{bmatrix}y_{1}\\y_{2}\\y_{3}\\y_{4}\\y_{5}\\y_{6}\\y_{7}\end{bmatrix}}={\begin{bmatrix}1&0&0\\1&0&0\\1&0&0\\1&1&0\\1&1&0\\1&0&1\\1&0&1\end{bmatrix}}{\begin{bmatrix}\mu \\\tau _{2}\\\tau _{3}\end{bmatrix}}+{\begin{bmatrix}\epsilon _{1}\\\epsilon _{2}\\\epsilon _{3}\\\epsilon _{4}\\\epsilon _{5}\\\epsilon _{6}\\\epsilon _{7}\end{bmatrix}}

In this model $\mu$ is the mean of the reference group and $\tau _{i}$ is the difference from group $i$ to the reference group. $\tau _{1}$ is not included in the matrix because its difference from the reference group (itself) is necessarily zero.

References

↑ Everitt, B. S. (2002). Cambridge Dictionary of Statistics (2nd ed.). Cambridge, UK: Cambridge University Press. ISBN 0-521-81099-X.
↑ Box, G. E. P.; Tiao, G. C. (1992) [1973]. Bayesian Inference in Statistical Analysis. New York: John Wiley and Sons. ISBN 0-471-57428-7. (Section 8.1.1)
↑ Timm, Neil H. (2007). Applied Multivariate Analysis. Springer Science & Business Media. p. 107.

Matrix classes
Explicitly constrained entries	(0,1) Alternant Anti-diagonal Anti-Hermitian Anti-symmetric Arrowhead Band Bidiagonal Binary Bisymmetric Block-diagonal Block Block tridiagonal Boolean Cauchy Centrosymmetric Conference Complex Hadamard Copositive Diagonally dominant Diagonal Discrete Fourier Transform Elementary Equivalent Frobenius Generalized permutation Hadamard Hankel Hermitian Hessenberg Hollow Integer Logical Markov Metzler Monomial Moore Nonnegative Partitioned Parisi Pentadiagonal Permutation Persymmetric Polynomial Positive Quaternionic Sign Signature Skew-Hermitian Skew-symmetric Skyline Sparse Sylvester Symmetric Toeplitz Triangular Tridiagonal Unitary Vandermonde Walsh Z
Constant	Exchange Hilbert Identity Lehmer Of ones Pascal Pauli Redheffer Shift Zero
Conditions on eigenvalues or eigenvectors	Companion Convergent Defective Diagonalizable Hurwitz Positive-definite Stability Stieltjes
Satisfying conditions on products or inverses	Congruent Idempotent or Projection Invertible Involutory Nilpotent Normal Orthogonal Orthonormal Singular Unimodular Unipotent Totally unimodular Weighing
With specific applications	Adjugate Alternating sign Augmented Bézout Carleman Cartan Circulant Cofactor Commutation Coxeter Derogatory Distance Duplication Elimination Euclidean distance Fundamental (linear differential equation) Generator Gramian Hessian Householder Jacobian Moment Payoff Pick Random Rotation Seifert Shear Similarity Symplectic Totally positive Transformation Wedderburn X–Y–Z
Used in statistics	Bernoulli Centering Correlation Covariance Design Dispersion Doubly stochastic Fisher information Hat Precision Stochastic Transition
Used in graph theory	Adjacency Biadjacency Degree Edmonds Incidence Laplacian Seidel adjacency Skew-adjacency Tutte
Used in science and engineering	Cabibbo–Kobayashi–Maskawa Density Fundamental (computer vision) Fuzzy associative Gamma Gell-Mann Hamiltonian Irregular Overlap S State transition Substitution Z (chemistry)
Related terms	Jordan canonical form Linear independence Matrix exponential Matrix representation of conic sections Perfect matrix Pseudoinverse Quaternionic matrix Row echelon form Wronskian
List of matrices Category:Matrices

This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.