### The recurrence relation techique for analyzing running time of algorithms

• Introduction

• A commonly used technique to study the running times of algorithms is recurrence relations

• A recurrence relations is a relation between values in a common set that are expressed in terms of other elements of that set

Example: Fibonacci numbers

 ``` F0 = 1 F1 = 1 FN = FN-1 + FN-2 for N >= 2 ^^^^^^^^^^^^^^^ recurrence relation ```

• We will look at

 How to set up a recurrence relation in running times analysis Two techniques to solve a recurrence relation

• Example 1: Setting up a recurrence relation for running time analysis

• Consider the following program fragment:

 ``` int N; for ( i = 0; i < N; i++ ) do_marker(); // Marker operation ```

• Define:

 Cn = total # times that the marker operation do_marker() is executed when N = n

(Clearly, Cn = n

What we want to do now is to find Cn using a recurrence relation)

• To set up a recurrence relation, we compare the execution of the program when N = n and when N = n+1:

 ``` i = 0 1 2 3 ... n # do_marker(): 1 1 1 1 ... 1 | | +-----------------------+ amount of work done = Cn i = 0 1 2 3 ... n n+1 # do_marker(): 1 1 1 1 ... 1 1 | | +---------------------------+ amount of work done = Cn+1 ```

• The amount of work done for N = n+1 is one more do_marker() operation compared to the amount of work done for N = n.

So we have that:

 Cn+1 = Cn + 1

• Initial condition

• Additional information to solve a recurrence relation:

 In order to solve a recurrence relation, you always need some initial conditions

Typical initial conditions:

 C0 = some constant C1 = some constant etc., etc.

• The number of initial conditions needed depends on the number of unknowns (variables):

 If there are N unknowns (variables), you need N-1 initial conditions

Example: to solve

 Cn+1 = Cn + 1

you need one initial condition because there are 2 unknowns: Cn and Cn+1

• How to discover initial conditions

 Initial conditions are discovered by computing the running time for the (small) initial value(s)

Example:

 ``` int N = 1; for ( i = 0; i < N; i++ ) do_marker(); // Marker operation ```

The running time (= # times that do_marker() is executed) when N = 1 is: 1

Therefore:

 C1 = 1

• The complete recurrence relation is then:

 Cn+1 = Cn + 1     C1 = 1

• Rewriting a recurrence relation

• It is common to write a recurrence relation as:

 Cn = ....

rather than:

 Cn+1 = ....

• A recurrence relation can be rewritten using a substitution

Example:

 ``` Cn+1 = Cn + 1 Substitution: n = m - 1 Result: C(m-1)+1 = C(m-1) + 1 Cm = Cm-1 + 1 And after replacing (renaming) m by n: Cn = Cn-1 + 1 ```

Therefore, the following set of recurrence relations are equivalent:

Recurrence relation 1         Recurrence relation 2
• Cn+1 = Cn + 1
• C1 = 1
• Cn = Cn-1 + 1
• C1 = 1

 Substitute n by m-1 And then rename m with n

We can substitute n with n-1 directly.

Example:

 ``` Cn+1 = Cn + 1 Substitution: n => n - 1 Result: C(n-1)+1 = C(n-1) + 1 Cn = Cn-1 + 1 ```

• Example 2: Setting up a recurrence relation for running time analysis

• Consider the following program fragment:

 ``` int N; for ( i = 0; i < N; i++ ) for ( j = 0; j < N; j++ ) do_marker(); // Marker operation ```

• Define:

 Cn = total # times that the marker operation do_marker() is executed when N = n

(Clearly, Cn = n2

What we want to do now is to find Cn using a recurrence relation)

• To set up a recurrence relation, we again compare the execution of the program when N = n and when N = n+1:

 ``` i = 0 1 2 3 ... n n+1 j 0 0 0 0 ... 0 n+1 1 1 1 1 ... 1 n+1 | 2 2 2 2 ... 2 n+1 | .... V n n n n ... n n+1 n+1 n+1 n+1 n+1 ... n+1 n+1 ```

The red indices are the times that do_marker() is executed when N = n.

The magenta indices are the additional times that do_marker() is executed when N = n+1.

• We can see that:

 Cn+1 = Cn + n + (n+1)        Cn+1 = Cn + 2n + 1

• Rewrite using n => n-1:

 ``` Cn+1 = Cn + 2n + 1 n => n-1 ==> C(n-1)+1 = C(n-1) + 2(n-1) + 1 <==> Cn = Cn-1 + 2n - 2 + 1 <==> Cn = Cn-1 + 2n - 1 ```

• Initial condition:

There are two unknowns (Cn and Cn+1)

Therefore, we need to discover one initial condition

Compute the running time for N = 1:

 ``` int N = 1; for ( i = 0; i < N; i++ ) for ( j = 0; j < N; j++ ) do_marker(); // Marker operation ```

We find that:

 C1 = 1

• The complete recurrence relation:

 Cn = Cn-1 + 2n - 1     C1 = 1

• Example 3: Setting up a recurrence relation for running time analysis

• The following algorithm is the well-known binary search algorithm to find a value in an sorted array

You can take advantage of the fact that the item in the array are sorted to speed up the search

Example:

1. Compare the value x with the middle value of the array

2. If x is equal to that element, stop (we have found x

Otherwise:

if x is less to that element, then search x in the first half of the array, else search x in the second half of the array

• The binary search algorithm in "psuedo code":

 ``` input: int item[N]; int left, right; left = 0; right = N-1; while ( left <= right ) { int middle = (left + right)/2; if ( item[middle] == x ) // found return( middle ); /* ------------------------------------------ Not found, continue search... ------------------------------------------ */ if ( x < item[middle] ) { right = middle - 1; // Search in first half of remaining array } else { left = middle + 1; // Search in second half of remaining array } // Marker operation <--------------------- } ```

• Counting loop execution with constant number of statements:

Notice that are no loop statement inside the while-loop (while ( left <= right ))

In other words:

 The number of instructions executed in each loop is constant In fact, each time through the while-loop, the program executes 3 statements (1 assignment statement and 2 if-statements)

So if the while-loop is executed p times, then the running time is 3×p

As you know, 3×p is O(p)

Rule of thumb:

• Analysis of a loop statement where a constant number of statement is executed in one iteration is simplified to:

 ``` loop-statement { do_marker(); } ```

I.e., we just count the constant number as one.

It is a good enough analysis; as this will give us the order O(fN) of the running time of the loop.

• To set up a recurrence relation, we consider the structure of the algorithm:

 Let Cn = number of times the while loop is executed when there are n unsearched elements in the array The binary search algorithm: executes one (compare) statement and reduces the number of unsearched elements in the array to n/2 According to the definition above, the number of times the while loop is executed when there are n/2 unsearched elements in the array is equal to: Cn/2

Therefore:

 ``` Cn = 1 + Cn/2 ```

(Number of statements needed to find a value in an array of n unsearched values = one (compare) statement plus the number of statements needed to find a value in an array of n/2 unsearched values)

• Initial condition:

There are two unknowns (Cn and Cn/2)

Therefore, we need to discover one initial condition

Compute the running time for N = 1:

 ``` input: int item[N]; int left, right; left = 0; right = 0; // Only item[0] needs to be tested while ( left <= right ) { int middle = (left + right)/2 ==> (0+0)/2 = 0 if ( item[middle] == x ) // found return( middle ); /* ------------------------------------------ Not found, continue search... ------------------------------------------ */ if ( x < item[middle] ) { right = middle - 1; // right = -1, loop will end } else { left = middle + 1; // left = -1, loop will end } // Marker operation <--------------------- } ```

We find that:

 C1 = 1

• The complete recurrence relation:

 Cn = Cn/2 + 1     C1 = 1

• Solving recurrence relations

• Obviously, we must solve the recurrence relation to find an expression for Cn

• We will do that in a subsequent web page....

• Before we can solve the recurrence relation, we need to study some background material (functions, serie sums, etc) that are commonly encountered when solving recurrence relations