Skip to main content

Section 5.3 \(PA=LU\) Decomposition

An application of the Gaussian elimination algorithm without its Jordan component is involved in a procedure for solving multiple matrix equations. Consider that there is a matrix \(A\in\Mats{n,n}\) and a set of vectors
\begin{equation*} B=\set{\vec{b}_k:k\in\set{0,1,\dotsc,\ell-1}} \end{equation*}
such that all the solutions to \(A\vv=\vec{b}_k\) must be determined for all \(k\text{.}\) In order to solve all of these problems using Gauss-Jordan elimination in a naive manner, it would be necessary to perform the GJE process \(\ell\) times. A technique which allows GJE to be performed only once would be of great benefit.

Subsection 5.3.1 Special factors

In general, any representation of a matrix \(A\) of the form \(A=BC\) is referred to as a decomposition or factorization of \(A\text{;}\) the type of decomposition depends upon the desired properties of the factors. We begin with a definition of two interesting types of square matrices: upper and lower triangular matrices.

Definition 5.3.1. Triangular matrices.

A matrix \(A\in\Mats{m,n}\) is square if and only if \(m=n\text{.}\) A square matrix \(A\in\Mats{n,n}\) is
Upper triangular
if and only if \(\entry{A}{r,c} = 0\) whenever \(r\gt c\text{,}\)
Lower triangular
if and only if \(\entry{A}{r,c}=0\) whenever \(r\lt c\text{,}\) and is
Diagonal
if and only if \(\entry{A}{r,c}=0\) whenever \(r\neq c\text{.}\)
Every square matrix which is in row-echelon form is necessarily upper triangular, and a careful application of the Gaussian elimination algorithm to a matrix \(A\) will produce matrices \(P\text{,}\) \(L\text{,}\) and \(U\) such that
  • \(P\) is a permutation matrix obtained as a product of \(S_{k,l}\) row-swapping matrices,
  • \(L\) is lower triangular, and
  • \(U\) is upper triangular and row-equivalent to \(PA\text{.}\)

Proof.

We will apply the Gaussian elimination algorithm with partial pivoting (GEPP) but limiting the eligible row operations only to swapping rows via \(\rswap{k}{\ell}\) and elimination via \(\relim{\alpha}{k}{\ell}\text{.}\) Initially let \(P=I_n\text{,}\) the identity matrix in \(\Mats{n,n}\text{;}\) we will modify a matrix \(M\) which is initally equal to \(A\text{.}\)
In the process of applying the GEPP algorithm to \(M\text{,}\) whenever an operation \(M\xmapsto{\rswap{k}{\ell}}M\) occurs, also perform \(P\xmapsto{\rswap{k}{\ell}}\text{.}\) Moreover, when the operation \(M\xmapsto{\relim{\alpha}{k}{\ell}}\) is applied to eliminate a value in row index \(\ell\) and column index \(k\text{,}\) rather than writing \(\entry{M}{\ell}{k}=0\) put \(\entry{M}{\ell}{k} = -\alpha\text{.}\)
When the GEPP algorithm completes, the matrix \(P\) is the correct permutation matrix. The matrix \(M\) contains all of the necessary elements of both matrices \(L\) and \(U\text{,}\) which meed to be extracted into the triangular matrices. Specifically we let \(L\in\Mats{n,n}\) be the lower triangular matrix satisfying
\begin{equation*} \entry{L}{r,c} = \begin{cases} 0\amp\text{if }r \lt c \\ 1\amp\text{if }r = c \\ \entry{M}{r,c}\amp\text{if }r\gt c. \end{cases} \end{equation*}
Further, \(U\in\Mats{n,n}\) is the upper triangular matrix satisfying
\begin{equation*} \entry{U}{r,c} = \begin{cases} 0 \amp\text{if} r\gt c \\ \entry{M}{r,c}\amp\text{if }r\lt c. \end{cases} \end{equation*}
Since this matrix \(L\) is the product of the inverse matrices \(E_{\alpha,k,\ell}^{-1}\) corresponding to each of the eliminations mentioned previously, and \(U\) is the result of those eliminations, it is necessarily the case that \(PA=LU\text{.}\)

Example 5.3.3. A worked \(PA=LU\) decomposition example.

Suppose that
\begin{equation*} A = \begin{bmatrix} 1 \amp 4 \amp 9 \amp 7 \\ 5 \amp 6 \amp 8 \amp 3 \\ 8 \amp 7 \amp 2 \amp 7 \\ 5 \amp 4 \amp 1 \amp 2 \end{bmatrix}\text{.} \end{equation*}
Then the sequence of row operations and resulting matrices \(M\) are as follows below. Notice that the value of the permutation matrix \(P\) is only given when it changes, and entries of \(M\) enclosed in boxes are those which are recorded to be extracted into \(L\text{.}\)
\begin{align*} M = \begin{bmatrix} 1 \amp 4 \amp 9 \amp 7 \\ 5 \amp 6 \amp 8 \amp 3 \\ 8 \amp 7 \amp 2 \amp 7 \\ 5 \amp 4 \amp 1 \amp 2 \end{bmatrix} \xto{\rswap{0}{2}} \amp \begin{bmatrix} 8 \amp 7 \amp 2 \amp 7 \\ 5 \amp 6 \amp 8 \amp 3 \\ 1 \amp 4 \amp 9 \amp 7 \\ 5 \amp 4 \amp 1 \amp 2 \end{bmatrix}, P=\begin{bmatrix} 0 \amp 0 \amp 1 \amp 0 \\ 0 \amp 1 \amp 0 \amp 0 \\ 1 \amp 0 \amp 0 \amp 0 \\ 0 \amp 0 \amp 0 \amp 1 \end{bmatrix}\\ \xto{\begin{smallmatrix} \relim{-5/8}{0}{1}\\ \relim{-1/8}{0}{2} \\ \relim{-5/8}{0}{3} \end{smallmatrix}} \amp \begin{bmatrix} 8 \amp 7 \amp 2 \amp 7 \\ \fbox{$5/8$} \amp 13/8 \amp 27/4 \amp -11/8 \\ \fbox{$1/8$} \amp 25/8 \amp 35/4 \amp 49/8 \\ \fbox{$5/8$} \amp -3/8 \amp -1/4 \amp -19/8 \end{bmatrix}\\ \xto{\rswap{1}{2}} \amp \begin{bmatrix} 8 \amp 7 \amp 2 \amp 7 \\ \fbox{$1/8$} \amp 25/8 \amp 35/4 \amp 49/8 \\ \fbox{$5/8$} \amp 13/8 \amp 27/4 \amp -11/8 \\ \fbox{$5/8$} \amp -3/8 \amp -1/4 \amp -19/8 \end{bmatrix}, P = \begin{bmatrix} 0 \amp 0 \amp 1 \amp 0 \\ 1 \amp 0 \amp 0 \amp 0 \\ 0 \amp 1 \amp 0 \amp 0 \\ 0 \amp 0 \amp 0 \amp 1 \end{bmatrix}\\ \xto{\begin{smallmatrix} \relim{-13/25}{1}{2} \\ \relim{3/25}{1}{3} \end{smallmatrix}} \amp \begin{bmatrix} 8 \amp 7 \amp 2 \amp 7 \\ \fbox{$1/8$} \amp 25/8 \amp 35/4 \amp 49/8 \\ \fbox{$5/8$} \amp \fbox{$13/25$} \amp 11/5 \amp -114/25 \\ \fbox{$5/8$} \amp \fbox{$-3/25$} \amp 4/5 \amp -41/25 \end{bmatrix}\\ \xto{\relim{-4/11}{2}{3}} \amp \begin{bmatrix} 8 \amp 7 \amp 2 \amp 7 \\ \fbox{$1/8$} \amp 25/8 \amp 35/4 \amp 49/8 \\ \fbox{$5/8$} \amp \fbox{$13/25$} \amp 11/5 \amp -114/25 \\ \fbox{$5/8$} \amp \fbox{$-3/25$} \amp \fbox{$4/11$} \amp 1/55 \end{bmatrix} \end{align*}
Extracting \(L\) and \(U\) from this matrix, we obtain
\begin{equation*} P = \left[~\begin{smallmatrix} 0 \amp 0 \amp 1 \amp 0 \\ 1 \amp 0 \amp 0 \amp 0 \\ 0 \amp 1 \amp 0 \amp 0 \\ 0 \amp 0 \amp 0 \amp 1 \end{smallmatrix}~\right]\qquad L = \left[~\begin{smallmatrix} 1 \amp 0 \amp 0 \amp 0 \\ 1/8 \amp 1 \amp 0 \amp 0 \\ 5/8 \amp 13/25 \amp 1 \amp 0 \\ 5/8 \amp -3/25 \amp 4/11 \amp 1 \end{smallmatrix}~\right]\qquad U = \left[~\begin{smallmatrix} 8 \amp 7 \amp 2 \amp 7 \\ 0 \amp 25/8 \amp 35/4 \amp 49/8 \\ 0 \amp 0 \amp 11/5 \amp -114/25 \\ 0 \amp 0 \amp 0 \amp 1/55 \end{smallmatrix}~\right] \end{equation*}

Subsection 5.3.2 Solving systems via \(LU\) decomposition

Now we can put this decomposition to work using basic algebra properties of matrices to solve a simple matrix equation. Beginning from \(A\vec{x} = \vec{b}\) when \(PA=LU\text{,}\) we observe that \(PA\vec{x} = P\vec{b}\text{,}\) so \(LU\vec{x} = P\vec{b}\text{.}\) Introducing an intermediate equation \(U\vec{x} = \vec{y}\) and writing \(\vec{b}' = P\vec{b}\text{,}\) we have to solve \(L\vec{y} = \vec{b}'\) and then \(U\vec{x} = \vec{y}\text{.}\) These are easy to solve via substitution.

Subsection 5.3.3 Algorithms for \(LU\) decomposition

The algorithm for \(PA=LU\) decomposition is little different from any other Gaussian elimination algorithm. In order to operate efficiently and not require the same row operations to be performed on multiple matrices, we will carefully work in a list of lists and convert the final result to a tuple of three AlgoMatrix objects.
A crucial and easy-to-miss part of the \(PA=LU\) algorithm is Step (2.b.iii); in order to store the values for \(L\) in place of the unnecessary zeroes which normally appear below the diagonal of a matrix in row-echelon form, it is essential that only the values to the right of the diagonal be recomputed during the elimination steps. The elimination which actually is known to send values to zero must not do so, otherwise the lower triangular matrix \(L\) is lost. The following is a method implementing our algorithm, suitable fo inclusion in the AlgoMatrix class.
def palu(self, debug=False):
    m,n = self.dims()
    if m!=n: raise ValueError("Cannot PA=LU decompose nonsquare matrix.")
    # Initialize grids
    grid = [[entry for entry in row] for row in self._data] # Skim the data
    pmat = [[1 if r==c else 0 for c in range(n)] for r in range(m)]
    debug_out = []
    # Start at upper left
    r,c = 0,0
    while (r < m) and (c < n):
        # Check if rows need swapped
        debug_out.append( str(grid) )
        cur_max = abs(grid[r][c])
        piv = r
        for r_prime in range(r+1, m):
            if abs(grid[r_prime][c]) > cur_max:
                cur_max = abs(grid[r_prime][c])
                piv = r_prime
        # if cur_max == 0, shift right.
        if cur_max == 0:
            c += 1
            continue
            # otherwise check if rows need swapped
        elif piv != r:
            debug_out.append(f"Swap R({r})<->R({r_prime})")
            # interchange elements of lists
            grid[piv],grid[r] = grid[r],grid[piv]
            pmat[piv],pmat[r] = pmat[r],pmat[piv]
        # No scaling
        # Elimination
        for r_prime in range(r+1, m):
            if grid[r_prime][c] == 0: continue
            alpha = grid[r_prime][c] / grid[r][c]
            # Since we want to maintain the lower-triangular part for L, we only alter
            # columns c+1 through n of grid[r_prime]
            grid[r_prime][c+1:] = [-alpha*grid[r][c_prime] + grid[r_prime][c_prime] for c_prime in range(c+1,n)]
            # Then store the alpha value in grid[r_prime][c] to pull out for L later
            grid[r_prime][c] = alpha
            debug_out.append(f"Elim ({-alpha})*R({r}) + R({r_prime})")
        r += 1
        c += 1
    P = AlgoMatrix(pmat)
    L = AlgoMatrix([[1 if r==c else (grid[r][c] if r > c else 0) for c in range(n)] for r in range(m)])
    U = AlgoMatrix([[grid[r][c] if r <= c else 0 for c in range(n)] for r in range(m)])
    if debug: print("\n".join(debug_out))
    return P,L,U
Listing 5.3.7. An implementation of the \(PA=LU\) algorithm using partial pivoting.