Lecture 4: Recurrences and Strassen’s Algorithm

(CLRS 4.1-4.2, 28.1+28.2)

May 21st, 2002

1 Summation review

• Last time we computed a number of sum’s using:

– Splitting and bounding terms ideas

– Induction (!)

• Arithmetic sum: – ∑_{k=1}^{n} k = \frac{n(n+1)}{2} = Θ(n^{2})

– ∑_{k=1}^{n} k^{d} = Θ(n^{d+1})

• Geometric sum: – ∑_{k=0}^{n} x^{k} = \frac{x^{n+1}−1}{x−1} = Θ(x^{n})

• Harmonic sum: – ∑_{i=1}^{n} \frac{1}{k} = Θ(log n)

2 Recurrences

• Last time we started discussing how to solve recurrences.

– Recurrences often needs to be solved in order to analyze divide-and-conquer algorithms.

• We saw how to solve the recurrence T (n) = 2T (n/2) + n using the substitution method – Idea in substitution method is to make good guess and prove by induction.

2.1 Substitution method

• Solution to T (n) = 2T (n/2) + n using substitution – Guess T (n) ≤ cn log n for some constant c (that is, T (n) = O(n log n))

– Proof:

∗ Basis: Function constant for small constant n ∗ Induction:

Assume holds for n/2: T (n/2) ≤ c\frac{n}{2} log \frac{n}{2} Show holds for n: T (n) ≤ cn log n Proof:

T (n) = 2T (n/2) + n
≤ 2(c\frac{n}{2} log \frac{n}{2}) + n
= cn log \frac{n}{2} + n
= cn log n − cn log 2 + n
= cn log n − cn + n

So ok if c ≥ 1

• The hard part of the substitution method is often to make a good guess.

2.2 Iteration/recursion-tree method

• In the iteration method we iteratively “unfold” the recurrence until we “see the pattern”.

The iteration method does not require making a good guess like the substitution method (but it is often more involved than using induction).

– Example: Solve T (n) = 8T (n/2) + n^{2} (T (1) = 1)

T (n) = n^{2} + 8T (n/2)
= n^{2} + 8(8T (\frac{n}{2^{2}} ) + (\frac{n}{2})^{2})
= n^{2} + 8^{2}T (\frac{n}{2^{2}} ) + 8(\frac{n^{2}}{4} ))
= n^{2} + 2n^{2} + 8^{2}T (\frac{n}{2^{2}} )
= n^{2} + 2n^{2} + 8^{2}(8T (\frac{n}{2^{3}} ) + (\frac{n}{2^{2}} )^{2})
= n^{2} + 2n^{2} + 8^{3}T (\frac{n}{2^{3}} ) + 8^{2}(\frac{n^{2}}{4^{2}} ))
= n^{2} + 2n^{2} + 2^{2}n^{2} + 8^{3}T (\frac{n}{2^{3}} )
= …
= n^{2} + 2n^{2} + 2^{2}n^{2} + 2^{3}n^{2} + 2^{4}n^{2} + …

– How long does it continue? i times where \frac{n}{2^{i}} = 1 ⇒ i = log n

– What is the last term? 8^{i}T (1) = 8^{log n}

T (n) = n^{2} + 2n^{2} + 2^{2}n^{2} + 2^{3}n^{2} + 2^{4}n^{2} + … + 2^{log n−1}n^{2} + 8^{log n}
= ∑_{k=0}^{log n−1} 2^{k}n^{2} + 8^{log n}

= n^{2} ∑_{k=0}^{log n−1} 2^{k} + (2^{3})^{log n}

– Now ∑_{k=0}^{log n−1} 2^{k} is a geometric sum so we have ∑_{k=0}^{log n−1} 2^{k} = Θ(2^{log n−1}) = Θ(n)

– (2^{3})^{log n} = (2^{log n})^{3} = n^{3}

T (n) = n^{2} · Θ(n) + n^{3}
= Θ(n^{3})

• The book discuss a different way of looking at the iteration method: the recursion-tree method

– we draw out the recursion tree with cost of single call in each node—running time is sum of costs in all nodes (like we discussed when analyzing merge-sort).

– really the same as iterating.

– Example: T (n) = 8T (n/2) + n^{2} (T (1) = 1)

T (n) = n^{2} + 2n^{2} + 2^{2}n^{2} + 2^{3}n^{2} + 2^{4}n^{2} + … + 2^{log n−1}n^{2} + 8^{log n}

3 Matrix Multiplication

• Let X and Y be n × n matrices

X = \{ \begin{matrix}x_{11} & x_{12} & ⋯ & x_{1n} \\ x_{21} & x_{22} & ⋯ & x_{1n} \\ x_{31} & x_{32} & ⋯ & x_{1n} \\ ⋯ & ⋯ & ⋯ & ⋯ \\ x_{n1} & x_{n2} & ⋯ & x_{nn} \end{matrix} \}

• We want to compute Z = X · Y – z_{ij} = ∑_{k=1}^{n} X_{ik} · Y_{kj}

• Naive method uses ⇒ n^{2} · n = Θ(n^{3}) operations

• Divide-and-conquer solution:

Z = \{ \begin{matrix}A & B \\ C & D \end{matrix} \} · \{ \begin{matrix}E & F \\ G & H \end{matrix} \} = \{ \begin{matrix}(A · E + B · G) & (A · F + B · H) \\ (C · E + D · G) & (C · F + D · H) \end{matrix} \}

– The above naturally leads to divide-and-conquer solution:

∗ Divide X and Y into 8 sub-matrices A, B, C, and D.

∗ Do 8 matrix multiplications recursively.

∗ Compute Z by combining results (doing 4 matrix additions).

– Lets assume n = 2^{c} for some constant c and let A, B, C and D be n/2 × n/2 matrices ∗ Running time of algorithm is T (n) = 8T (n/2) + Θ(n^{2}) ⇒ T (n) = Θ(n^{3})

– But we already discussed a (simpler/naive) O(n^{3}) algorithm! Can we do better?

3.1 Strassen’s Algorithm

• Strassen observed the following:

Z = \{ \begin{matrix}A & B \\ C & D \end{matrix} \} · \{ \begin{matrix}E & F \\ G & H \end{matrix} \} = \{ \begin{matrix}(S_{1} + S_{2} − S_{4} + S_{6}) & (S_{4} + S_{5}) \\ (S_{6} + S_{7}) & (S_{2} + S_{3} + S_{5} − S_{7}) \end{matrix} \}

where

S_{1} = (B − D) · (G + H)
S_{2} = (A + D) · (E + H)
S_{3} = (A − C) · (E + F)
S_{4} = (A + B) · H
S_{5} = A · (F − H)
S_{6} = D · (G − E)
S_{7} = (C + D) · E

– Lets test that S_{6} + S_{7} is really C · E + D · G

S_{6} + S_{7} = D · (G − E) + (C + D) · E
= DG − DE + CE + DE
= DG + CE

• This leads to a divide-and-conquer algorithm with running time T (n) = 7T (n/2) + Θ(n^{2}) – We only need to perform 7 multiplications recursively.

– Division/Combination can still be performed in Θ(n^{2}) time.

• Lets solve the recurrence using the iteration method

T (n) = 7T (n/2) + n^{2}
= n^{2} + 7(7T (\frac{n}{2^{2}} ) + (\frac{n}{2})^{2})
= n^{2} + (\frac{7}{2^{2}} )n^{2} + 7^{2}T (\frac{n}{2^{2}} )
= n^{2} + (\frac{7}{2^{2}} )n^{2} + 7^{2}(7T (\frac{n}{2^{3}} ) + (\frac{n}{2^{2}} )^{2})
= n^{2} + (\frac{7}{2^{2}} )n^{2} + (\frac{7}{2^{2}} )^{2} · n^{2} + 7^{3}T (\frac{n}{2^{3}} )
= n^{2} + (\frac{7}{2^{2}} )n^{2} + (\frac{7}{2^{2}} )^{2}n^{2} + (\frac{7}{2^{2}} )^{3}n^{2}.... + (\frac{7}{2^{2}} )^{log n−1}n^{2} + 7^{log n}
= ∑_{i=0}^{log n−1}(\frac{7}{2^{2}} )^{i}n^{2} + 7^{log n}
= n^{2} · Θ((\frac{7}{2^{2}} )^{log n−1}) + 7^{log n}
= n^{2} · Θ(\frac{7^{log n}}{(2^{2})^{log n}}) + 7^{log n}
= n^{2} · Θ(\frac{7^{log n}}{n^{2}} ) + 7^{log n}
= Θ(7^{log n})

– Now we have the following:

7^{log n} = 7^{\frac{log_{7} n}{log_{7} 2}}
= (7^{log_{7} n})^{(1/ log_{7} 2)}
= n^{(1/ log_{7} 2)}
= n^{\frac{log_{2} 7}{log_{2} 2}}
= n^{log 7}

– Or in general: a^{log_{k} n} = n^{log_{k} a}

So the solution is T (n) = Θ(n^{log 7}) = Θ(n^{2.81...})

• Note:

– We are ’hiding’ a much bigger constant in Θ() than before.

– Currently best known bound is O(n^{2.376..}) (another method).

– Lower bound is (trivially) Ω(n^{2}).

– Book present Strassen’s algorithm in a somewhat strange way.