Constrained least squares

In constrained least squares one solves a linear least squares problem with an additional constraint on the solution. This means, the unconstrained equation 𝐗𝜷=𝐲 must be fit as closely as possible (in the least squares sense) while ensuring that some other property of 𝜷 is maintained.

There are often special-purpose algorithms for solving such problems efficiently. Some examples of constraints are given below:

If the constraint only applies to some of the variables, the mixed problem may be solved using separable least squares by letting 𝐗=[𝐗𝟏𝐗𝟐] and 𝜷T=[𝜷𝟏T𝜷𝟐T] represent the unconstrained (1) and constrained (2) components. Then substituting the least-squares solution for 𝜷𝟏, i.e.

𝜷̂1=𝐗1+(𝐲−𝐗2𝜷2)

(where + indicates the Moore–Penrose pseudoinverse) back into the original expression gives (following some rearrangement) an equation that can be solved as a purely constrained problem in 𝜷2.

𝐏𝐗2𝜷2=𝐏𝐲,

where 𝐏:=𝐈−𝐗1𝐗1+ is a projection matrix. Following the constrained estimation of 𝜷̂2 the vector 𝜷̂1 is obtained from the expression above.

See also