An application of the Gaussian elimination algorithm without its Jordan component is involved in a procedure for solving multiple matrix equations. Consider that there is a matrix \(A\in\Mats{n,n}\) and a set of vectors
such that all the solutions to \(A\vv=\vec{b}_k\) must be determined for all \(k\text{.}\) In order to solve all of these problems using Gauss-Jordan elimination in a naive manner, it would be necessary to perform the GJE process \(\ell\) times. A technique which allows GJE to be performed only once would be of great benefit.
Subsection5.3.1Special factors
In general, any representation of a matrix \(A\) of the form \(A=BC\) is referred to as a decomposition or factorization of \(A\text{;}\) the type of decomposition depends upon the desired properties of the factors. We begin with a definition of two interesting types of square matrices: upper and lower triangular matrices.
Definition5.3.1.Triangular matrices.
A matrix \(A\in\Mats{m,n}\) is square if and only if \(m=n\text{.}\) A square matrix \(A\in\Mats{n,n}\) is
Upper triangular
if and only if \(\entry{A}{r,c} = 0\) whenever \(r\gt c\text{,}\)
Lower triangular
if and only if \(\entry{A}{r,c}=0\) whenever \(r\lt c\text{,}\) and is
Diagonal
if and only if \(\entry{A}{r,c}=0\) whenever \(r\neq c\text{.}\)
Every square matrix which is in row-echelon form is necessarily upper triangular, and a careful application of the Gaussian elimination algorithm to a matrix \(A\) will produce matrices \(P\text{,}\)\(L\text{,}\) and \(U\) such that
\(P\) is a permutation matrix obtained as a product of \(S_{k,l}\) row-swapping matrices,
\(L\) is lower triangular, and
\(U\) is upper triangular and row-equivalent to \(PA\text{.}\)
Theorem5.3.2.\(PA=LU\) decomposition of \(A\).
Suppose that \(A\in\Mats{n,n}\) is an invertible matrix. Then there is a permutation matrix \(P\) and a pair of lower and upper triangular matrices \(L\) and \(U\) such that \(PA=LU\text{.}\)
Proof.
We will apply the Gaussian elimination algorithm with partial pivoting (GEPP) but limiting the eligible row operations only to swapping rows via \(\rswap{k}{\ell}\) and elimination via \(\relim{\alpha}{k}{\ell}\text{.}\) Initially let \(P=I_n\text{,}\) the identity matrix in \(\Mats{n,n}\text{;}\) we will modify a matrix \(M\) which is initally equal to \(A\text{.}\)
In the process of applying the GEPP algorithm to \(M\text{,}\) whenever an operation \(M\xmapsto{\rswap{k}{\ell}}M\) occurs, also perform \(P\xmapsto{\rswap{k}{\ell}}\text{.}\) Moreover, when the operation \(M\xmapsto{\relim{\alpha}{k}{\ell}}\) is applied to eliminate a value in row index \(\ell\) and column index \(k\text{,}\) rather than writing \(\entry{M}{\ell}{k}=0\) put \(\entry{M}{\ell}{k} = -\alpha\text{.}\)
When the GEPP algorithm completes, the matrix \(P\) is the correct permutation matrix. The matrix \(M\) contains all of the necessary elements of both matrices \(L\) and \(U\text{,}\) which meed to be extracted into the triangular matrices. Specifically we let \(L\in\Mats{n,n}\) be the lower triangular matrix satisfying
\begin{equation*}
\entry{L}{r,c} = \begin{cases}
0\amp\text{if }r \lt c \\
1\amp\text{if }r = c \\
\entry{M}{r,c}\amp\text{if }r\gt c.
\end{cases}
\end{equation*}
Further, \(U\in\Mats{n,n}\) is the upper triangular matrix satisfying
\begin{equation*}
\entry{U}{r,c} = \begin{cases}
0 \amp\text{if} r\gt c \\
\entry{M}{r,c}\amp\text{if }r\lt c.
\end{cases}
\end{equation*}
Since this matrix \(L\) is the product of the inverse matrices \(E_{\alpha,k,\ell}^{-1}\) corresponding to each of the eliminations mentioned previously, and \(U\) is the result of those eliminations, it is necessarily the case that \(PA=LU\text{.}\)
Example5.3.3.A worked \(PA=LU\) decomposition example.
Then the sequence of row operations and resulting matrices \(M\) are as follows below. Notice that the value of the permutation matrix \(P\) is only given when it changes, and entries of \(M\) enclosed in boxes are those which are recorded to be extracted into \(L\text{.}\)
Subsection5.3.2Solving systems via \(LU\) decomposition
Now we can put this decomposition to work using basic algebra properties of matrices to solve a simple matrix equation. Beginning from \(A\vec{x} = \vec{b}\) when \(PA=LU\text{,}\) we observe that \(PA\vec{x} = P\vec{b}\text{,}\) so \(LU\vec{x} = P\vec{b}\text{.}\) Introducing an intermediate equation \(U\vec{x} = \vec{y}\) and writing \(\vec{b}' = P\vec{b}\text{,}\) we have to solve \(L\vec{y} = \vec{b}'\) and then \(U\vec{x} = \vec{y}\text{.}\) These are easy to solve via substitution.
Theorem5.3.4.Solution via forward substitution.
Given an invertible lower triangular matrix \(L\in\Mats{n,n}\) and a vector \(\vec{b}\text{,}\) the solution \(\vec{y}\) to the matrix-vector product equation \(L\vec{y}=\vec{b}\) satisfies
and as such solving for \(\entry{y}{r}\) in the order of \(r = 0,1,2,\dotsc, n-2,n-1\) involves only direct substitution.
Theorem5.3.5.Solution via back substitution.
Given an invertible upper triangular matrix \(U\in\Mats{n,n}\) and a vector \(\vec{y}\text{,}\) the solution \(\vec{x}\) to the matrix-vector product equation \(U\vec{x}=\vec{y}\) satisfies
and as such solving for \(\entry{x}{r}\) in the order of \(r = n-1,n-2,\dotsc,2,1,0\) involves only direct substitution.
Subsection5.3.3Algorithms for \(LU\) decomposition
The algorithm for \(PA=LU\) decomposition is little different from any other Gaussian elimination algorithm. In order to operate efficiently and not require the same row operations to be performed on multiple matrices, we will carefully work in a list of lists and convert the final result to a tuple of three AlgoMatrix objects.
Algorithm5.3.6.\(PA=LU\) decomposition.
Suppose that \(A\in\Mats{n,n}\) be a matrix, and initialize \(P=I_n\) to be the identity matrix of size \(n\text{.}\)
Let \(r=0\) and \(c=0\)
While \(r\lt n\) and \(c\lt n\text{,}\) repeat the following:
Partial pivoting.
If all \(\entry{A}{r',c}=0\) for \(r'\in\set{r+1,\dotsc,n-1}\text{,}\) increase \(c\) by 1 and return to step (2).
If \(p\neq r\text{,}\) then \(A\xmapsto{\rswap{r}{p}} A\) and \(P\xmapsto{\rswap{r}{p}} P\text{;}\) otherwise proceed to step (2.b).
Elimination.
For each \(r'\in\set{r+1,\dotsc,m-1}\) do the following:
If \(\entry{A}{r',c}=0\text{,}\) continue to the next value of \(r'\text{.}\)
Otherwise, let \(\alpha = \entry{A}{r',c}/\entry{A}{r,c}\text{.}\)
For each \(c'\in\set{c+1,\dotsc,n-1}\text{,}\) update the value of \(\entry{A}{r',c'}\) by setting it equal to \(-\alpha\entry{A}{r,c'} + \entry{A}{r',c'}\) via the elimination operation.
Reset \(\entry{A}{r',c} = \alpha\text{.}\)
Increase \(r\) and \(c\) each by 1.
When the while loop in step (2) terminates, the matrix \(P\) is the correct permutation matrix, and we may simply extract the values of \(L\) and \(U\) from \(A\text{.}\)
Let \(L\in\Mats{n,n}\) be the matrix satisfying
\begin{equation*}
\entry{L}{r,c} = \begin{cases}
0,\amp\text{if }r\lt c \\
1,\amp\text{if }r=c\\
\entry{A}{r,c},\amp\text{if }r\gt c
\end{cases}
\end{equation*}
A crucial and easy-to-miss part of the \(PA=LU\) algorithm is Step (2.b.iii); in order to store the values for \(L\) in place of the unnecessary zeroes which normally appear below the diagonal of a matrix in row-echelon form, it is essential that only the values to the right of the diagonal be recomputed during the elimination steps. The elimination which actually is known to send values to zero must not do so, otherwise the lower triangular matrix \(L\) is lost. The following is a method implementing our algorithm, suitable fo inclusion in the AlgoMatrix class.
def palu(self, debug=False):
m,n = self.dims()
if m!=n: raise ValueError("Cannot PA=LU decompose nonsquare matrix.")
# Initialize grids
grid = [[entry for entry in row] for row in self._data] # Skim the data
pmat = [[1 if r==c else 0 for c in range(n)] for r in range(m)]
debug_out = []
# Start at upper left
r,c = 0,0
while (r < m) and (c < n):
# Check if rows need swapped
debug_out.append( str(grid) )
cur_max = abs(grid[r][c])
piv = r
for r_prime in range(r+1, m):
if abs(grid[r_prime][c]) > cur_max:
cur_max = abs(grid[r_prime][c])
piv = r_prime
# if cur_max == 0, shift right.
if cur_max == 0:
c += 1
continue
# otherwise check if rows need swapped
elif piv != r:
debug_out.append(f"Swap R({r})<->R({r_prime})")
# interchange elements of lists
grid[piv],grid[r] = grid[r],grid[piv]
pmat[piv],pmat[r] = pmat[r],pmat[piv]
# No scaling
# Elimination
for r_prime in range(r+1, m):
if grid[r_prime][c] == 0: continue
alpha = grid[r_prime][c] / grid[r][c]
# Since we want to maintain the lower-triangular part for L, we only alter
# columns c+1 through n of grid[r_prime]
grid[r_prime][c+1:] = [-alpha*grid[r][c_prime] + grid[r_prime][c_prime] for c_prime in range(c+1,n)]
# Then store the alpha value in grid[r_prime][c] to pull out for L later
grid[r_prime][c] = alpha
debug_out.append(f"Elim ({-alpha})*R({r}) + R({r_prime})")
r += 1
c += 1
P = AlgoMatrix(pmat)
L = AlgoMatrix([[1 if r==c else (grid[r][c] if r > c else 0) for c in range(n)] for r in range(m)])
U = AlgoMatrix([[grid[r][c] if r <= c else 0 for c in range(n)] for r in range(m)])
if debug: print("\n".join(debug_out))
return P,L,U
Listing5.3.7.An implementation of the \(PA=LU\) algorithm using partial pivoting.