Introduction to Algorithms

Third Edition

Thomas H. Cormen · Charles E. Leiserson · Ronald L. Rivest · Clifford Stein

The MIT Press, Cambridge, Massachusetts / London, England

© 2009 Massachusetts Institute of Technology. All rights reserved. ISBN 978-0-262-03384-8 (hardcover), 978-0-262-53305-8 (paperback). Library of Congress call number QA76.6.I58582009, 005.1, LCCN 2009008593.

Preface (p. xiii)

Part I: Foundations (Introduction, p. 3)

1 The Role of Algorithms in Computing (5)
2 Getting Started (16)
3 Growth of Functions (43)
4 Divide-and-Conquer (65)
5 Probabilistic Analysis and Randomized Algorithms (114)

Part II: Sorting and Order Statistics (Introduction, p. 147)

6 Heapsort (151)
7 Quicksort (170)
8 Sorting in Linear Time (191)
9 Medians and Order Statistics (213)

Part III: Data Structures (Introduction, p. 229)

10 Elementary Data Structures (232)
11 Hash Tables (253)
12 Binary Search Trees (286)
13 Red-Black Trees (308)
14 Augmenting Data Structures (339)

Part IV: Advanced Design and Analysis Techniques (Introduction, p. 357)

15 Dynamic Programming (359)
16 Greedy Algorithms (414)
17 Amortized Analysis (451)

Part V: Advanced Data Structures (Introduction, p. 481)

18 B-Trees (484)
19 Fibonacci Heaps (505)
20 van Emde Boas Trees (531)
21 Data Structures for Disjoint Sets (561)

Part VI: Graph Algorithms (Introduction, p. 587)

22 Elementary Graph Algorithms (589)
23 Minimum Spanning Trees (624)
24 Single-Source Shortest Paths (643)
25 All-Pairs Shortest Paths (684)
26 Maximum Flow (708)

Part VII: Selected Topics (Introduction, p. 769)

27 Multithreaded Algorithms (772)
28 Matrix Operations (813)
29 Linear Programming (843)
30 Polynomials and the FFT (898)
31 Number-Theoretic Algorithms (926)
32 String Matching (985)
33 Computational Geometry (1014)
34 NP-Completeness (1048)
35 Approximation Algorithms (1106)

Part VIII: Appendix: Mathematical Background (Introduction, p. 1143)

A Summations (1145)
B Sets, Etc. (1158)
C Counting and Probability (1183)
D Matrices (1217)

Bibliography (1231) · Index (1251)

Sections marked with a star in the full table of contents (for example 4.6, 5.4, 11.5, 12.4, 16.4, 16.5, 21.4, 26.4, 26.5, 31.8, 31.9, 32.4, C.5) are the ones more suited to graduate students.

Preface

Big picture. Imagine a recipe book where every recipe is not just “what to cook” but also “how long it will take and why this way beats the others.” That is what this book is for algorithms. Before you dive in, the authors tell you who the book is for, how to travel through it, and what changed since the last edition. Read this as a map of the territory ahead.

Before there were computers, there were algorithms. But now that there are computers, there are even more algorithms, and algorithms lie at the heart of computing.

This book provides a comprehensive introduction to the modern study of computer algorithms. It presents many algorithms and covers them in considerable depth, yet makes their design and analysis accessible to all levels of readers. We have tried to keep explanations elementary without sacrificing depth of coverage or mathematical rigor.

Each chapter presents an algorithm, a design technique, an application area, or a related topic. Algorithms are described in English and in a pseudocode designed to be readable by anyone who has done a little programming. The book contains 244 figures (many with multiple parts) illustrating how the algorithms work. Since we emphasize efficiency as a design criterion, we include careful analyses of the running times of all our algorithms.

The text is intended primarily for use in undergraduate or graduate courses in algorithms or data structures. Because it discusses engineering issues in algorithm design, as well as mathematical aspects, it is equally well suited for self-study by technical professionals.

In this, the third edition, we have once again updated the entire book. The changes cover a broad spectrum, including new chapters, revised pseudocode, and a more active writing style.

To the teacher

We have designed this book to be both versatile and complete. You should find it useful for a variety of courses, from an undergraduate course in data structures up through a graduate course in algorithms. Because we have provided considerably more material than can fit in a typical one-term course, you should consider the book to be a “buffet” or “smorgasbord” from which you can pick and choose the material that best supports the course you wish to teach.

You should find it easy to organize your course around just the chapters you need. We have made chapters relatively self-contained, so that you need not worry about an unexpected and unnecessary dependence of one chapter on another. Each chapter presents the easier material first and the more difficult material later, with section boundaries marking natural stopping points. In an undergraduate course, you might use only the earlier sections from a chapter; in a graduate course, you might cover the entire chapter.

We have included 957 exercises and 158 problems. Each section ends with exercises, and each chapter ends with problems. The exercises are generally short questions that test basic mastery of the material. Some are simple self-check thought exercises, whereas others are more substantial and are suitable as assigned homework. The problems are more elaborate case studies that often introduce new material; they often consist of several questions that lead the student through the steps required to arrive at a solution.

Departing from our practice in previous editions of this book, we have made publicly available solutions to some, but by no means all, of the problems and exercises. Our Web site, http://mitpress.mit.edu/algorithms/, links to these solutions. You will want to check this site to make sure that it does not contain the solution to an exercise or problem that you plan to assign. We expect the set of solutions that we post to grow slowly over time, so you will need to check it each time you teach the course.

We have starred ( $⋆$ ) the sections and exercises that are more suitable for graduate students than for undergraduates. A starred section is not necessarily more difficult than an unstarred one, but it may require an understanding of more advanced mathematics. Likewise, starred exercises may require an advanced background or more than average creativity.

To the student

We hope that this textbook provides you with an enjoyable introduction to the field of algorithms. We have attempted to make every algorithm accessible and interesting. To help you when you encounter unfamiliar or difficult algorithms, we describe each one in a step-by-step manner. We also provide careful explanations of the mathematics needed to understand the analysis of the algorithms. If you already have some familiarity with a topic, you will find the chapters organized so that you can skim introductory sections and proceed quickly to the more advanced material.

This is a large book, and your class will probably cover only a portion of its material. We have tried, however, to make this a book that will be useful to you now as a course textbook and also later in your career as a mathematical desk reference or an engineering handbook.

What are the prerequisites for reading this book?

You should have some programming experience. In particular, you should understand recursive procedures and simple data structures such as arrays and linked lists.
You should have some facility with mathematical proofs, and especially proofs by mathematical induction. A few portions of the book rely on some knowledge of elementary calculus. Beyond that, Parts I and VIII of this book teach you all the mathematical techniques you will need.

We have heard, loud and clear, the call to supply solutions to problems and exercises. Our Web site, http://mitpress.mit.edu/algorithms/, links to solutions for a few of the problems and exercises. Feel free to check your solutions against ours. We ask, however, that you do not send your solutions to us.

Think about it. Why might checking your answer after wrestling with a problem teach you more than reading the solution first?

To the professional

The wide range of topics in this book makes it an excellent handbook on algorithms. Because each chapter is relatively self-contained, you can focus in on the topics that most interest you.

Most of the algorithms we discuss have great practical utility. We therefore address implementation concerns and other engineering issues. We often provide practical alternatives to the few algorithms that are primarily of theoretical interest.

If you wish to implement any of the algorithms, you should find the translation of our pseudocode into your favorite programming language to be a fairly straightforward task. We have designed the pseudocode to present each algorithm clearly and succinctly. Consequently, we do not address error-handling and other software-engineering issues that require specific assumptions about your programming environment. We attempt to present each algorithm simply and directly without allowing the idiosyncrasies of a particular programming language to obscure its essence.

We understand that if you are using this book outside of a course, then you might be unable to check your solutions to problems and exercises against solutions provided by an instructor. Our Web site, http://mitpress.mit.edu/algorithms/, links to solutions for some of the problems and exercises so that you can check your work. Please do not send your solutions to us.

To our colleagues

We have supplied an extensive bibliography and pointers to the current literature. Each chapter ends with a set of chapter notes that give historical details and references. The chapter notes do not provide a complete reference to the whole field of algorithms, however. Though it may be hard to believe for a book of this size, space constraints prevented us from including many interesting algorithms.

Despite myriad requests from students for solutions to problems and exercises, we have chosen as a matter of policy not to supply references for problems and exercises, to remove the temptation for students to look up a solution rather than to find it themselves.

Changes for the third edition

What has changed between the second and third editions of this book? The magnitude of the changes is on a par with the changes between the first and second editions. As we said about the second-edition changes, depending on how you look at it, the book changed either not much or quite a bit.

A quick look at the table of contents shows that most of the second-edition chapters and sections appear in the third edition. We removed two chapters and one section, but we have added three new chapters and two new sections apart from these new chapters.

We kept the hybrid organization from the first two editions. Rather than organizing chapters by only problem domains or according only to techniques, this book has elements of both. It contains technique-based chapters on divide-and-conquer, dynamic programming, greedy algorithms, amortized analysis, NP-Completeness, and approximation algorithms. But it also has entire parts on sorting, on data structures for dynamic sets, and on algorithms for graph problems. We find that although you need to know how to apply techniques for designing and analyzing algorithms, problems seldom announce to you which techniques are most amenable to solving them.

Here is a summary of the most significant changes for the third edition:

We added new chapters on van Emde Boas trees and multithreaded algorithms, and we have broken out material on matrix basics into its own appendix chapter.
We revised the chapter on recurrences to more broadly cover the divide-and-conquer technique, and its first two sections apply divide-and-conquer to solve two problems. The second section of this chapter presents Strassen’s algorithm for matrix multiplication, which we have moved from the chapter on matrix operations.
We removed two chapters that were rarely taught: binomial heaps and sorting networks. One key idea in the sorting networks chapter, the 0-1 principle, appears in this edition within Problem 8-7 as the 0-1 sorting lemma for compare-exchange algorithms. The treatment of Fibonacci heaps no longer relies on binomial heaps as a precursor.
We revised our treatment of dynamic programming and greedy algorithms. Dynamic programming now leads off with a more interesting problem, rod cutting, than the assembly-line scheduling problem from the second edition. Furthermore, we emphasize memoization a bit more than we did in the second edition, and we introduce the notion of the subproblem graph as a way to understand the running time of a dynamic-programming algorithm. In our opening example of greedy algorithms, the activity-selection problem, we get to the greedy algorithm more directly than we did in the second edition.
The way we delete a node from binary search trees (which includes red-black trees) now guarantees that the node requested for deletion is the node that is actually deleted. In the first two editions, in certain cases, some other node would be deleted, with its contents moving into the node passed to the deletion procedure. With our new way to delete nodes, if other components of a program maintain pointers to nodes in the tree, they will not mistakenly end up with stale pointers to nodes that have been deleted.
The material on flow networks now bases flows entirely on edges. This approach is more intuitive than the net flow used in the first two editions.
With the material on matrix basics and Strassen’s algorithm moved to other chapters, the chapter on matrix operations is smaller than in the second edition.
We have modified our treatment of the Knuth-Morris-Pratt string-matching algorithm.
We corrected several errors. Most of these errors were posted on our Web site of second-edition errata, but a few were not.
Based on many requests, we changed the syntax (as it were) of our pseudocode. We now use " $=$ " to indicate assignment and " $==$ " to test for equality, just as C, C++, Java, and Python do. Likewise, we have eliminated the keywords do and then and adopted ”//” as our comment-to-end-of-line symbol. We also now use dot-notation to indicate object attributes. Our pseudocode remains procedural, rather than object-oriented. In other words, rather than running methods on objects, we simply call procedures, passing objects as parameters.
We added 100 new exercises and 28 new problems. We also updated many bibliography entries and added several new ones.
Finally, we went through the entire book and rewrote sentences, paragraphs, and sections to make the writing clearer and more active.

Website

You can use our Web site, http://mitpress.mit.edu/algorithms/, to obtain supplementary information and to communicate with us. The Web site links to a list of known errors, solutions to selected exercises and problems, and (of course) a list explaining the corny professor jokes, as well as other content that we might add. The Web site also tells you how to report errors or make suggestions.

How we produced this book

Like the second edition, the third edition was produced in $L A T E X 2_{ε}$ . We used the Times font with mathematics typeset using the MathTime Pro 2 fonts. We thank Michael Spivak from Publish or Perish, Inc., Lance Carnes from Personal TeX, Inc., and Tim Tregubov from Dartmouth College for technical support. As in the previous two editions, we compiled the index using Windex, a C program that we wrote, and the bibliography was produced with BibTeX. The PDF files for this book were created on a MacBook running OS 10.5.

We drew the illustrations for the third edition using MacDraw Pro, with some of the mathematical expressions in illustrations laid in with the psfrag package for $L A T E X 2_{ε}$ . Unfortunately, MacDraw Pro is legacy software, having not been marketed for over a decade now. Happily, we still have a couple of Macintoshes that can run the Classic environment under OS 10.4, and hence they can run MacDraw Pro (mostly). Even under the Classic environment, we find MacDraw Pro to be far easier to use than any other drawing software for the types of illustrations that accompany computer-science text, and it produces beautiful output.¹ Who knows how long our pre-Intel Macs will continue to run, so if anyone from Apple is listening: Please create an OS X-compatible version of MacDraw Pro!

Acknowledgments for the third edition

We have been working with the MIT Press for over two decades now, and what a terrific relationship it has been! We thank Ellen Faran, Bob Prior, Ada Brunstein, and Mary Reilly for their help and support.

We were geographically distributed while producing the third edition, working in the Dartmouth College Department of Computer Science, the MIT Computer Science and Artificial Intelligence Laboratory, and the Columbia University Department of Industrial Engineering and Operations Research. We thank our respective universities and colleagues for providing such supportive and stimulating environments.

Julie Sussman, P.P.A., once again bailed us out as the technical copyeditor. Time and again, we were amazed at the errors that eluded us, but that Julie caught. She also helped us improve our presentation in several places. If there is a Hall of Fame for technical copyeditors, Julie is a sure-fire, first-ballot inductee. She is nothing short of phenomenal. Thank you, thank you, thank you, Julie! Priya Natarajan also found some errors that we were able to correct before this book went to press. Any errors that remain (and undoubtedly, some do) are the responsibility of the authors (and probably were inserted after Julie read the material).

The treatment for van Emde Boas trees derives from Erik Demaine’s notes, which were in turn influenced by Michael Bender. We also incorporated ideas from Javed Aslam, Bradley Kuszmaul, and Hui Zha into this edition.

The chapter on multithreading was based on notes originally written jointly with Harald Prokop. The material was influenced by several others working on the Cilk project at MIT, including Bradley Kuszmaul and Matteo Frigo. The design of the multithreaded pseudocode took its inspiration from the MIT Cilk extensions to C and by Cilk Arts’s Cilk++ extensions to C++.

We also thank the many readers of the first and second editions who reported errors or submitted suggestions for how to improve this book. We corrected all the bona fide errors that were reported, and we incorporated as many suggestions as we could. We rejoice that the number of such contributors has grown so great that we must regret that it has become impractical to list them all.

Finally, we thank our wives (Nicole Cormen, Wendy Leiserson, Gail Rivest, and Rebecca Ivry) and our children (Ricky, Will, Debby, and Katie Leiserson; Alex and Christopher Rivest; and Molly, Noah, and Benjamin Stein) for their love and support while we prepared this book. The patience and encouragement of our families made this project possible. We affectionately dedicate this book to them.

THOMAS H. CORMEN, Lebanon, New Hampshire CHARLES E. LEISERSON, Cambridge, Massachusetts RONALD L. RIVEST, Cambridge, Massachusetts CLIFFORD STEIN, New York, New York

February 2009

Part I: Foundations

Introduction

Big picture. Before a builder pours concrete, they learn how to read a blueprint and how to test whether a design will hold. Part I is your blueprint-reading course for algorithms: how to write them down, how to design them, and how to judge how fast they are. Everything later in the book is built on this base.

This part will start you thinking about designing and analyzing algorithms. It is intended to be a gentle introduction to how we specify algorithms, some of the design strategies we will use throughout this book, and many of the fundamental ideas used in algorithm analysis. Later parts of this book will build upon this base.

Chapter 1 provides an overview of algorithms and their place in modern computing systems. This chapter defines what an algorithm is and lists some examples. It also makes a case that we should consider algorithms as a technology, alongside technologies such as fast hardware, graphical user interfaces, object-oriented systems, and networks.

In Chapter 2, we see our first algorithms, which solve the problem of sorting a sequence of $n$ numbers. They are written in a pseudocode which, although not directly translatable to any conventional programming language, conveys the structure of the algorithm clearly enough that you should be able to implement it in the language of your choice. The sorting algorithms we examine are insertion sort, which uses an incremental approach, and merge sort, which uses a recursive technique known as “divide-and-conquer.” Although the time each requires increases with the value of $n$ , the rate of increase differs between the two algorithms. We determine these running times in Chapter 2, and we develop a useful notation to express them.

Chapter 3 precisely defines this notation, which we call asymptotic notation. It starts by defining several asymptotic notations, which we use for bounding algorithm running times from above and/or below. The rest of Chapter 3 is primarily a presentation of mathematical notation, more to ensure that your use of notation matches that in this book than to teach you new mathematical concepts.

Chapter 4 delves further into the divide-and-conquer method introduced in Chapter 2. It provides additional examples of divide-and-conquer algorithms, including Strassen’s surprising method for multiplying two square matrices. Chapter 4 contains methods for solving recurrences, which are useful for describing the running times of recursive algorithms. One powerful technique is the “master method,” which we often use to solve recurrences that arise from divide-and-conquer algorithms. Although much of Chapter 4 is devoted to proving the correctness of the master method, you may skip this proof yet still employ the master method.

Chapter 5 introduces probabilistic analysis and randomized algorithms. We typically use probabilistic analysis to determine the running time of an algorithm in cases in which, due to the presence of an inherent probability distribution, the running time may differ on different inputs of the same size. In some cases, we assume that the inputs conform to a known probability distribution, so that we are averaging the running time over all possible inputs. In other cases, the probability distribution comes not from the inputs but from random choices made during the course of the algorithm. An algorithm whose behavior is determined not only by its input but by the values produced by a random-number generator is a randomized algorithm. We can use randomized algorithms to enforce a probability distribution on the inputs, thereby ensuring that no particular input always causes poor performance, or even to bound the error rate of algorithms that are allowed to produce incorrect results on a limited basis.

Appendices A-D contain other mathematical material that you will find helpful as you read this book. You are likely to have seen much of the material in the appendix chapters before having read this book (although the specific definitions and notational conventions we use may differ in some cases from what you have seen in the past), and so you should think of the Appendices as reference material. On the other hand, you probably have not already seen most of the material in Part I. All the chapters in Part I and the Appendices are written with a tutorial flavor.

1 The Role of Algorithms in Computing

Big picture. A recipe turns ingredients into a meal; an algorithm turns input into output. But two recipes for the same dish can differ wildly: one takes minutes, the other takes a week. This chapter asks what an algorithm really is, where algorithms hide in the world around you, and why choosing a good one can matter more than buying a faster machine.

What are algorithms? Why is the study of algorithms worthwhile? What is the role of algorithms relative to other technologies used in computers? In this chapter, we will answer these questions.

1.1 Algorithms

Informally, an algorithm is any well-defined computational procedure that takes some value, or set of values, as input and produces some value, or set of values, as output. An algorithm is thus a sequence of computational steps that transform the input into the output.

We can also view an algorithm as a tool for solving a well-specified computational problem. The statement of the problem specifies in general terms the desired input/output relationship. The algorithm describes a specific computational procedure for achieving that input/output relationship.

For example, we might need to sort a sequence of numbers into nondecreasing order. This problem arises frequently in practice and provides fertile ground for introducing many standard design techniques and analysis tools. Here is how we formally define the sorting problem:

Input: A sequence of $n$ numbers $⟨ a_{1}, a_{2}, \dots, a_{n} ⟩$ .
Output: A permutation (reordering) $⟨ a_{1}^{'}, a_{2}^{'}, \dots, a_{n}^{'} ⟩$ of the input sequence such that $a_{1}^{'} \leq a_{2}^{'} \leq \dots \leq a_{n}^{'} .$

For example, given the input sequence $⟨ 31, 41, 59, 26, 41, 58 ⟩$ , a sorting algorithm returns as output the sequence $⟨ 26, 31, 41, 41, 58, 59 ⟩$ . Such an input sequence is called an instance of the sorting problem. In general, an instance of a problem consists of the input (satisfying whatever constraints are imposed in the problem statement) needed to compute a solution to the problem.

Because many programs use it as an intermediate step, sorting is a fundamental operation in computer science. As a result, we have a large number of good sorting algorithms at our disposal. Which algorithm is best for a given application depends on, among other factors, the number of items to be sorted, the extent to which the items are already somewhat sorted, possible restrictions on the item values, the architecture of the computer, and the kind of storage devices to be used: main memory, disks, or even tapes.

An algorithm is said to be correct if, for every input instance, it halts with the correct output. We say that a correct algorithm solves the given computational problem. An incorrect algorithm might not halt at all on some input instances, or it might halt with an incorrect answer. Contrary to what you might expect, incorrect algorithms can sometimes be useful, if we can control their error rate. We shall see an example of an algorithm with a controllable error rate in Chapter 31 when we study algorithms for finding large prime numbers. Ordinarily, however, we shall be concerned only with correct algorithms.

An algorithm can be specified in English, as a computer program, or even as a hardware design. The only requirement is that the specification must provide a precise description of the computational procedure to be followed.

What kinds of problems are solved by algorithms?

Sorting is by no means the only computational problem for which algorithms have been developed. (You probably suspected as much when you saw the size of this book.) Practical applications of algorithms are ubiquitous and include the following examples:

The Human Genome Project has made great progress toward the goals of identifying all the 100,000 genes in human DNA, determining the sequences of the 3 billion chemical base pairs that make up human DNA, storing this information in databases, and developing tools for data analysis. Each of these steps requires sophisticated algorithms. Although the solutions to the various problems involved are beyond the scope of this book, many methods to solve these biological problems use ideas from several of the chapters in this book, thereby enabling scientists to accomplish tasks while using resources efficiently. The savings are in time, both human and machine, and in money, as more information can be extracted from laboratory techniques.
The Internet enables people all around the world to quickly access and retrieve large amounts of information. With the aid of clever algorithms, sites on the Internet are able to manage and manipulate this large volume of data. Examples of problems that make essential use of algorithms include finding good routes on which the data will travel (techniques for solving such problems appear in Chapter 24), and using a search engine to quickly find pages on which particular information resides (related techniques are in Chapters 11 and 32).
Electronic commerce enables goods and services to be negotiated and exchanged electronically, and it depends on the privacy of personal information such as credit card numbers, passwords, and bank statements. The core technologies used in electronic commerce include public-key cryptography and digital signatures (covered in Chapter 31), which are based on numerical algorithms and number theory.
Manufacturing and other commercial enterprises often need to allocate scarce resources in the most beneficial way. An oil company may wish to know where to place its wells in order to maximize its expected profit. A political candidate may want to determine where to spend money buying campaign advertising in order to maximize the chances of winning an election. An airline may wish to assign crews to flights in the least expensive way possible, making sure that each flight is covered and that government regulations regarding crew scheduling are met. An Internet service provider may wish to determine where to place additional resources in order to serve its customers more effectively. All of these are examples of problems that can be solved using linear programming, which we shall study in Chapter 29.

Although some of the details of these examples are beyond the scope of this book, we do give underlying techniques that apply to these problems and problem areas. We also show how to solve many specific problems, including the following:

We are given a road map on which the distance between each pair of adjacent intersections is marked, and we wish to determine the shortest route from one intersection to another. The number of possible routes can be huge, even if we disallow routes that cross over themselves. How do we choose which of all possible routes is the shortest? Here, we model the road map (which is itself a model of the actual roads) as a graph (which we will meet in Part VI and Appendix B), and we wish to find the shortest path from one vertex to another in the graph. We shall see how to solve this problem efficiently in Chapter 24.
We are given two ordered sequences of symbols, $X = ⟨ x_{1}, x_{2}, \dots, x_{m} ⟩$ and $Y = ⟨ y_{1}, y_{2}, \dots, y_{n} ⟩$ , and we wish to find a longest common subsequence of $X$ and $Y$ . A subsequence of $X$ is just $X$ with some (or possibly all or none) of its elements removed. For example, one subsequence of $⟨ A, B, C, D, E, F, G ⟩$ would be $⟨ B, C, E, G ⟩$ . The length of a longest common subsequence of $X$ and $Y$ gives one measure of how similar these two sequences are. For example, if the two sequences are base pairs in DNA strands, then we might consider them similar if they have a long common subsequence. If $X$ has $m$ symbols and $Y$ has $n$ symbols, then $X$ and $Y$ have $2^{m}$ and $2^{n}$ possible subsequences, respectively. Selecting all possible subsequences of $X$ and $Y$ and matching them up could take a prohibitively long time unless $m$ and $n$ are very small. We shall see in Chapter 15 how to use a general technique known as dynamic programming to solve this problem much more efficiently.
We are given a mechanical design in terms of a library of parts, where each part may include instances of other parts, and we need to list the parts in order so that each part appears before any part that uses it. If the design comprises $n$ parts, then there are $n!$ possible orders, where $n!$ denotes the factorial function. Because the factorial function grows faster than even an exponential function, we cannot feasibly generate each possible order and then verify that, within that order, each part appears before the parts using it (unless we have only a few parts). This problem is an instance of topological sorting, and we shall see in Chapter 22 how to solve this problem efficiently.
We are given $n$ points in the plane, and we wish to find the convex hull of these points. The convex hull is the smallest convex polygon containing the points. Intuitively, we can think of each point as being represented by a nail sticking out from a board. The convex hull would be represented by a tight rubber band that surrounds all the nails. Each nail around which the rubber band makes a turn is a vertex of the convex hull. (See Figure 33.6 on page 1029 for an example.) Any of the $2^{n}$ subsets of the points might be the vertices of the convex hull. Knowing which points are vertices of the convex hull is not quite enough, either, since we also need to know the order in which they appear. There are many choices, therefore, for the vertices of the convex hull. Chapter 33 gives two good methods for finding the convex hull.

Think about it. Notice the pattern: $2^{m}$ subsequences, $n!$ orderings, $2^{n}$ subsets. In every case the list of candidates explodes, so brute force collapses long before the problem gets big. Cleverness has to replace patience.

These lists are far from exhaustive (as you again have probably surmised from this book’s heft), but exhibit two characteristics that are common to many interesting algorithmic problems:

They have many candidate solutions, the overwhelming majority of which do not solve the problem at hand. Finding one that does, or one that is “best,” can present quite a challenge.
They have practical applications. Of the problems in the above list, finding the shortest path provides the easiest examples. A transportation firm, such as a trucking or railroad company, has a financial interest in finding shortest paths through a road or rail network because taking shorter paths results in lower labor and fuel costs. Or a routing node on the Internet may need to find the shortest path through the network in order to route a message quickly. Or a person wishing to drive from New York to Boston may want to obtain driving directions from an appropriate Web site, or she may use her GPS while driving.

Not every problem solved by algorithms has an easily identified set of candidate solutions. For example, suppose we are given a set of numerical values representing samples of a signal, and we want to compute the discrete Fourier transform of these samples. The discrete Fourier transform converts the time domain to the frequency domain, producing a set of numerical coefficients, so that we can determine the strength of various frequencies in the sampled signal. In addition to lying at the heart of signal processing, discrete Fourier transforms have applications in data compression and multiplying large polynomials and integers. Chapter 30 gives an efficient algorithm, the fast Fourier transform (commonly called the FFT), for this problem, and the chapter also sketches out the design of a hardware circuit to compute the FFT.

Data structures

This book also contains several data structures. A data structure is a way to store and organize data in order to facilitate access and modifications. No single data structure works well for all purposes, and so it is important to know the strengths and limitations of several of them.

Technique

Although you can use this book as a “cookbook” for algorithms, you may someday encounter a problem for which you cannot readily find a published algorithm (many of the exercises and problems in this book, for example). This book will teach you techniques of algorithm design and analysis so that you can develop algorithms on your own, show that they give the correct answer, and understand their efficiency. Different chapters address different aspects of algorithmic problem solving. Some chapters address specific problems, such as finding medians and order statistics in Chapter 9, computing minimum spanning trees in Chapter 23, and determining a maximum flow in a network in Chapter 26. Other chapters address techniques, such as divide-and-conquer in Chapter 4, dynamic programming in Chapter 15, and amortized analysis in Chapter 17.

Hard problems

Most of this book is about efficient algorithms. Our usual measure of efficiency is speed, i.e., how long an algorithm takes to produce its result. There are some problems, however, for which no efficient solution is known. Chapter 34 studies an interesting subset of these problems, which are known as NP-complete.

Why are NP-complete problems interesting? First, although no efficient algorithm for an NP-complete problem has ever been found, nobody has ever proven that an efficient algorithm for one cannot exist. In other words, no one knows whether or not efficient algorithms exist for NP-complete problems. Second, the set of NP-complete problems has the remarkable property that if an efficient algorithm exists for any one of them, then efficient algorithms exist for all of them. This relationship among the NP-complete problems makes the lack of efficient solutions all the more tantalizing. Third, several NP-complete problems are similar, but not identical, to problems for which we do know of efficient algorithms. Computer scientists are intrigued by how a small change to the problem statement can cause a big change to the efficiency of the best known algorithm.

You should know about NP-complete problems because some of them arise surprisingly often in real applications. If you are called upon to produce an efficient algorithm for an NP-complete problem, you are likely to spend a lot of time in a fruitless search. If you can show that the problem is NP-complete, you can instead spend your time developing an efficient algorithm that gives a good, but not the best possible, solution.

As a concrete example, consider a delivery company with a central depot. Each day, it loads up each delivery truck at the depot and sends it around to deliver goods to several addresses. At the end of the day, each truck must end up back at the depot so that it is ready to be loaded for the next day. To reduce costs, the company wants to select an order of delivery stops that yields the lowest overall distance traveled by each truck. This problem is the well-known “traveling-salesman problem,” and it is NP-complete. It has no known efficient algorithm. Under certain assumptions, however, we know of efficient algorithms that give an overall distance which is not too far above the smallest possible. Chapter 35 discusses such “approximation algorithms.”

Parallelism

For many years, we could count on processor clock speeds increasing at a steady rate. Physical limitations present a fundamental roadblock to ever-increasing clock speeds, however: because power density increases superlinearly with clock speed, chips run the risk of melting once their clock speeds become high enough. In order to perform more computations per second, therefore, chips are being designed to contain not just one but several processing “cores.” We can liken these multicore computers to several sequential computers on a single chip; in other words, they are a type of “parallel computer.” In order to elicit the best performance from multicore computers, we need to design algorithms with parallelism in mind. Chapter 27 presents a model for “multithreaded” algorithms, which take advantage of multiple cores. This model has advantages from a theoretical standpoint, and it forms the basis of several successful computer programs, including a championship chess program.

Exercises 1.1-1 to 1.1-5

1.1-1 Give a real-world example that requires sorting or a real-world example that requires computing a convex hull.

1.1-2 Other than speed, what other measures of efficiency might one use in a real-world setting?

1.1-3 Select a data structure that you have seen previously, and discuss its strengths and limitations.

1.1-4 How are the shortest-path and traveling-salesman problems given above similar? How are they different?

1.1-5 Come up with a real-world problem in which only the best solution will do. Then come up with one in which a solution that is “approximately” the best is good enough.

1.2 Algorithms as a technology

Suppose computers were infinitely fast and computer memory was free. Would you have any reason to study algorithms? The answer is yes, if for no other reason than that you would still like to demonstrate that your solution method terminates and does so with the correct answer.

If computers were infinitely fast, any correct method for solving a problem would do. You would probably want your implementation to be within the bounds of good software engineering practice (for example, your implementation should be well designed and documented), but you would most often use whichever method was the easiest to implement.

Of course, computers may be fast, but they are not infinitely fast. And memory may be inexpensive, but it is not free. Computing time is therefore a bounded resource, and so is space in memory. You should use these resources wisely, and algorithms that are efficient in terms of time or space will help you do so.

Efficiency

Different algorithms devised to solve the same problem often differ dramatically in their efficiency. These differences can be much more significant than differences due to hardware and software.

As an example, in Chapter 2, we will see two algorithms for sorting. The first, known as insertion sort, takes time roughly equal to $c_{1} n^{2}$ to sort $n$ items, where $c_{1}$ is a constant that does not depend on $n$ . That is, it takes time roughly proportional to $n^{2}$ . The second, merge sort, takes time roughly equal to $c_{2} n l g n$ , where $l g n$ stands for $lo g_{2} n$ and $c_{2}$ is another constant that also does not depend on $n$ . Insertion sort typically has a smaller constant factor than merge sort, so that $c_{1} < c_{2}$ . We shall see that the constant factors can have far less of an impact on the running time than the dependence on the input size $n$ . Let’s write insertion sort’s running time as $c_{1} n \cdot n$ and merge sort’s running time as $c_{2} n \cdot l g n$ . Then we see that where insertion sort has a factor of $n$ in its running time, merge sort has a factor of $l g n$ , which is much smaller. (For example, when $n = 1000$ , $l g n$ is approximately $10$ , and when $n$ equals one million, $l g n$ is approximately only $20$ .) Although insertion sort usually runs faster than merge sort for small input sizes, once the input size $n$ becomes large enough, merge sort’s advantage of $l g n$ vs. $n$ will more than compensate for the difference in constant factors. No matter how much smaller $c_{1}$ is than $c_{2}$ , there will always be a crossover point beyond which merge sort is faster.

For a concrete example, let us pit a faster computer (computer A) running insertion sort against a slower computer (computer B) running merge sort. They each must sort an array of 10 million numbers. (Although 10 million numbers might seem like a lot, if the numbers are eight-byte integers, then the input occupies about 80 megabytes, which fits in the memory of even an inexpensive laptop computer many times over.) Suppose that computer A executes 10 billion instructions per second (faster than any single sequential computer at the time of this writing) and computer B executes only 10 million instructions per second, so that computer A is $1000$ times faster than computer B in raw computing power. To make the difference even more dramatic, suppose that the world’s craftiest programmer codes insertion sort in machine language for computer A, and the resulting code requires $2 n^{2}$ instructions to sort $n$ numbers. Suppose further that just an average programmer implements merge sort, using a high-level language with an inefficient compiler, with the resulting code taking $50 n l g n$ instructions. To sort 10 million numbers, computer A takes

$\frac{2 \cdot ( 1 0 ^{7} ) ^{2} instructions}{1 0 ^{10} instructions/second} = 20, 000 seconds (more than 5.5 hours),$

while computer B takes

$\frac{50 \cdot 1 0 ^{7} l g 1 0 ^{7} instructions}{1 0 ^{7} instructions/second} \approx 1163 seconds (less than 20 minutes) .$

Worked check. Let us verify each number step by step.

Computer A (insertion sort):
$\text{instructions} &= 2n^2 = 2 \cdot (10^7)^2 = 2 \cdot 10^{14} \\ \text{time} &= \frac{2 \cdot 10^{14}}{10^{10}} = 2 \cdot 10^{4} = 20{,}000 \text{ s} \\ \frac{20{,}000}{3600} &\approx 5.56 \text{ hours.} \end{aligned}$$ *Computer B (merge sort):* $$\begin{aligned} \lg 10^7 &= 7 \lg 10 \approx 7 \cdot 3.3219 \approx 23.25 \\ \text{instructions} &= 50 \cdot 10^7 \cdot 23.25 \approx 1.163 \cdot 10^{10} \\ \text{time} &= \frac{1.163 \cdot 10^{10}}{10^7} \approx 1163 \text{ s} \\ \frac{1163}{60} &\approx 19.4 \text{ minutes.} \end{aligned}$$ *Ratio:* $$\frac{20{,}000}{1163} \approx 17.2 .$$$

By using an algorithm whose running time grows more slowly, even with a poor compiler, computer B runs more than 17 times faster than computer A! The advantage of merge sort is even more pronounced when we sort 100 million numbers: where insertion sort takes more than 23 days, merge sort takes under four hours. In general, as the problem size increases, so does the relative advantage of merge sort.

Think about it. Computer A is $1000 \times$ faster, run by an expert, yet it loses. Would buying a computer $1000 \times$ faster again rescue insertion sort when $n$ grows another tenfold?

Algorithms and other technologies

The example above shows that we should consider algorithms, like computer hardware, as a technology. Total system performance depends on choosing efficient algorithms as much as on choosing fast hardware. Just as rapid advances are being made in other computer technologies, they are being made in algorithms as well.

You might wonder whether algorithms are truly that important on contemporary computers in light of other advanced technologies, such as

advanced computer architectures and fabrication technologies,
easy-to-use, intuitive, graphical user interfaces (GUIs),
object-oriented systems,
integrated Web technologies, and
fast networking, both wired and wireless.

The answer is yes. Although some applications do not explicitly require algorithmic content at the application level (such as some simple, Web-based applications), many do. For example, consider a Web-based service that determines how to travel from one location to another. Its implementation would rely on fast hardware, a graphical user interface, wide-area networking, and also possibly on object orientation. However, it would also require algorithms for certain operations, such as finding routes (probably using a shortest-path algorithm), rendering maps, and interpolating addresses.

Moreover, even an application that does not require algorithmic content at the application level relies heavily upon algorithms. Does the application rely on fast hardware? The hardware design used algorithms. Does the application rely on graphical user interfaces? The design of any GUI relies on algorithms. Does the application rely on networking? Routing in networks relies heavily on algorithms. Was the application written in a language other than machine code? Then it was processed by a compiler, interpreter, or assembler, all of which make extensive use of algorithms. Algorithms are at the core of most technologies used in contemporary computers.

Furthermore, with the ever-increasing capacities of computers, we use them to solve larger problems than ever before. As we saw in the above comparison between insertion sort and merge sort, it is at larger problem sizes that the differences in efficiency between algorithms become particularly prominent.

Having a solid base of algorithmic knowledge and technique is one characteristic that separates the truly skilled programmers from the novices. With modern computing technology, you can accomplish some tasks without knowing much about algorithms, but with a good background in algorithms, you can do much, much more.

Exercises 1.2-1 to 1.2-3

1.2-1 Give an example of an application that requires algorithmic content at the application level, and discuss the function of the algorithms involved.

1.2-2 Suppose we are comparing implementations of insertion sort and merge sort on the same machine. For inputs of size $n$ , insertion sort runs in $8 n^{2}$ steps, while merge sort runs in $64 n l g n$ steps. For which values of $n$ does insertion sort beat merge sort?

1.2-3 What is the smallest value of $n$ such that an algorithm whose running time is $100 n^{2}$ runs faster than an algorithm whose running time is $2^{n}$ on the same machine?

Problems

1-1 Comparison of running times

For each function $f (n)$ and time $t$ in the following table, determine the largest size $n$ of a problem that can be solved in time $t$ , assuming that the algorithm to solve the problem takes $f (n)$ microseconds.

$f (n)$	1 second	1 minute	1 hour	1 day	1 month	1 year	1 century
$l g n$
$n$
$n$
$n l g n$
$n^{2}$
$n^{3}$
$2^{n}$
$n!$

Chapter notes

There are many excellent texts on the general topic of algorithms, including those by Aho, Hopcroft, and Ullman [5, 6]; Baase and Van Gelder [28]; Brassard and Bratley [54]; Dasgupta, Papadimitriou, and Vazirani [82]; Goodrich and Tamassia [148]; Hofri [175]; Horowitz, Sahni, and Rajasekaran [181]; Johnsonbaugh and Schaefer [193]; Kingston [205]; Kleinberg and Tardos [208]; Knuth [209, 210, 211]; Kozen [220]; Levitin [235]; Manber [242]; Mehlhorn [249, 250, 251]; Purdom and Brown [287]; Reingold, Nievergelt, and Deo [293]; Sedgewick [306]; Sedgewick and Flajolet [307]; Skiena [318]; and Wilf [356]. Some of the more practical aspects of algorithm design are discussed by Bentley [42, 43] and Gonnet [145]. Surveys of the field of algorithms can also be found in the Handbook of Theoretical Computer Science, Volume A [342] and the CRC Algorithms and Theory of Computation Handbook [25]. Overviews of the algorithms used in computational biology can be found in textbooks by Gusfield [156], Pevzner [275], Setubal and Meidanis [310], and Waterman [350].

2 Getting Started

Big picture. Before you can build a skyscraper, you need to learn how to read a blueprint and how to check that the beams will hold. This chapter is your blueprint-reading lesson: we meet a simple sorting algorithm, learn the “language” (pseudocode) used to describe algorithms, learn to prove that an algorithm works, and learn to measure how much work it does. Ask yourself as you read: how would you convince a skeptic that your method always works, and how fast it is?

This chapter will familiarize you with the framework we shall use throughout the book to think about the design and analysis of algorithms. It is self-contained, but it does include several references to material that we introduce in Chapters 3 and 4. (It also contains several summations, which Appendix A shows how to solve.)

We begin by examining the insertion sort algorithm to solve the sorting problem introduced in Chapter 1. We define a “pseudocode” that should be familiar to you if you have done computer programming, and we use it to show how we shall specify our algorithms. Having specified the insertion sort algorithm, we then argue that it correctly sorts, and we analyze its running time. The analysis introduces a notation that focuses on how that time increases with the number of items to be sorted. Following our discussion of insertion sort, we introduce the divide-and-conquer approach to the design of algorithms and use it to develop an algorithm called merge sort. We end with an analysis of merge sort’s running time.

2.1 Insertion sort

Our first algorithm, insertion sort, solves the sorting problem introduced in Chapter 1:

Input: A sequence of $n$ numbers $⟨ a_{1}, a_{2}, \dots, a_{n} ⟩$ .
Output: A permutation (reordering) $⟨ a_{1}^{'}, a_{2}^{'}, \dots, a_{n}^{'} ⟩$ of the input sequence such that $a_{1}^{'} \leq a_{2}^{'} \leq \dots \leq a_{n}^{'}$ .

The numbers that we wish to sort are also known as the keys. Although conceptually we are sorting a sequence, the input comes to us in the form of an array with $n$ elements.

In this book, we shall typically describe algorithms as programs written in a pseudocode that is similar in many respects to C, C++, Java, Python, or Pascal. If you have been introduced to any of these languages, you should have little trouble reading our algorithms. What separates pseudocode from “real” code is that in pseudocode, we employ whatever expressive method is most clear and concise to specify a given algorithm. Sometimes, the clearest method is English, so do not be surprised if you come across an English phrase or sentence embedded within a section of “real” code. Another difference between pseudocode and real code is that pseudocode is not typically concerned with issues of software engineering. Issues of data abstraction, modularity, and error handling are often ignored in order to convey the essence of the algorithm more concisely.

We start with insertion sort, which is an efficient algorithm for sorting a small number of elements. Insertion sort works the way many people sort a hand of playing cards. We start with an empty left hand and the cards face down on the table. We then remove one card at a time from the table and insert it into the correct position in the left hand. To find the correct position for a card, we compare it with each of the cards already in the hand, from right to left, as illustrated in Figure 2.1. At all times, the cards held in the left hand are sorted, and these cards were originally the top cards of the pile on the table.

Figure 2.1: Sorting a hand of cards using insertion sort.

We present our pseudocode for insertion sort as a procedure called INSERTION-SORT, which takes as a parameter an array $A [1.. n]$ containing a sequence of length $n$ that is to be sorted. (In the code, the number $n$ of elements in $A$ is denoted by $A . l e n g t h$ .) The algorithm sorts the input numbers in place: it rearranges the numbers within the array $A$ , with at most a constant number of them stored outside the array at any time. The input array $A$ contains the sorted output sequence when the INSERTION-SORT procedure is finished.

INSERTION-SORT(A)
1  for j = 2 to A.length
2      key = A[j]
3      // Insert A[j] into the sorted sequence A[1..j-1].
4      i = j - 1
5      while i > 0 and A[i] > key
6          A[i+1] = A[i]
7          i = i - 1
8      A[i+1] = key

Figure 2.2: The operation of INSERTION-SORT on the array $A = ⟨ 5, 2, 4, 6, 1, 3 ⟩$ . Array indices appear above the rectangles, and values stored in the array positions appear within the rectangles. (a)-(e) The iterations of the for loop of lines 1-8. In each iteration, the black rectangle holds the key taken from $A [j]$ , which is compared with the values in shaded rectangles to its left in the test of line 5. Shaded arrows show array values moved one position to the right in line 6, and black arrows indicate where the key moves to in line 8. (f) The final sorted array.

The array states after each iteration of the for loop (the vertical bar separates the sorted prefix from the rest):

Stage	Array
(a) start	$5 ∣ 24613$
(b) after $j = 2$	$25 ∣ 4613$
(c) after $j = 3$	$245 ∣ 613$
(d) after $j = 4$	$2456 ∣ 13$
(e) after $j = 5$	$12456 ∣ 3$
(f) after $j = 6$	$123456$

Loop invariants and the correctness of insertion sort

Figure 2.2 shows how this algorithm works for $A = ⟨ 5, 2, 4, 6, 1, 3 ⟩$ . The index $j$ indicates the “current card” being inserted into the hand. At the beginning of each iteration of the for loop, which is indexed by $j$ , the subarray consisting of elements $A [1.. j - 1]$ constitutes the currently sorted hand, and the remaining subarray $A [j + 1.. n]$ corresponds to the pile of cards still on the table. In fact, elements $A [1.. j - 1]$ are the elements originally in positions $1$ through $j - 1$ , but now in sorted order. We state these properties of $A [1.. j - 1]$ formally as a loop invariant:

At the start of each iteration of the for loop of lines 1-8, the subarray $A [1.. j - 1]$ consists of the elements originally in $A [1.. j - 1]$ , but in sorted order.

We use loop invariants to help us understand why an algorithm is correct. We must show three things about a loop invariant:

Initialization: It is true prior to the first iteration of the loop.
Maintenance: If it is true before an iteration of the loop, it remains true before the next iteration.
Termination: When the loop terminates, the invariant gives us a useful property that helps show that the algorithm is correct.

When the first two properties hold, the loop invariant is true prior to every iteration of the loop. (Of course, we are free to use established facts other than the loop invariant itself to prove that the loop invariant remains true before each iteration.) Note the similarity to mathematical induction, where to prove that a property holds, you prove a base case and an inductive step. Here, showing that the invariant holds before the first iteration corresponds to the base case, and showing that the invariant holds from iteration to iteration corresponds to the inductive step.

The third property is perhaps the most important one, since we are using the loop invariant to show correctness. Typically, we use the loop invariant along with the condition that caused the loop to terminate. The termination property differs from how we usually use mathematical induction, in which we apply the inductive step infinitely; here, we stop the “induction” when the loop terminates.

Think about it. A loop invariant is like a domino rule: “every domino behind me has fallen.” Why does it matter that the dominoes eventually run out?

Let us see how these properties hold for insertion sort.

Proof (correctness of INSERTION-SORT via the loop invariant).

Initialization. We start by showing that the loop invariant holds before the first loop iteration, when $j = 2$ .¹
$j = 2 ⟹ A [1.. j - 1] = A [1..1] = ⟨ A [1]⟩ ⟹ the subarray consists of the single element A [1], which is the original A [1] .$
A one-element subarray is sorted (trivially, of course). So the loop invariant holds prior to the first iteration of the loop.

Maintenance. Next, we tackle the second property: showing that each iteration maintains the loop invariant. Informally, the body of the for loop works by moving $A [j - 1]$ , $A [j - 2]$ , $A [j - 3]$ , and so on by one position to the right until it finds the proper position for $A [j]$ (lines 4-7), at which point it inserts the value of $A [j]$ (line 8).
$Before the iteration: A [1.. j - 1] is sorted and holds the original A [1.. j - 1] . Lines 4-7: elements of A [1.. j - 1] larger than k ey shift one place right, order preserved. Line 8: k ey is placed in the gap left behind, after all elements \leq k ey and before all elements > k ey . ⟹ A [1.. j] holds the original A [1.. j] in sorted order.$
Incrementing $j$ for the next iteration of the for loop then preserves the loop invariant.

A more formal treatment of the second property would require us to state and show a loop invariant for the while loop of lines 5-7. At this point, however, we prefer not to get bogged down in such formalism, and so we rely on our informal analysis to show that the second property holds for the outer loop.

Termination. Finally, we examine what happens when the loop terminates. The condition causing the for loop to terminate is that $j > A . l e n g t h = n$ . Because each loop iteration increases $j$ by 1, we must have $j = n + 1$ at that time. Substituting $n + 1$ for $j$ in the wording of the loop invariant:
$j = n + 1 ⟹ A [1.. j - 1] = A [1.. n] ⟹ A [1.. n] consists of the elements originally in A [1.. n], but in sorted order.$
Observing that the subarray $A [1.. n]$ is the entire array, we conclude that the entire array is sorted. Hence, the algorithm is correct. $■$

¹ When the loop is a for loop, the moment at which we check the loop invariant just prior to the first iteration is immediately after the initial assignment to the loop-counter variable and just before the first test in the loop header. In the case of INSERTION-SORT, this time is after assigning $2$ to the variable $j$ but before the first test of whether $j \leq A . l e n g t h$ .

We shall use this method of loop invariants to show correctness later in this chapter and in other chapters as well.

Pseudocode conventions

We use the following conventions in our pseudocode.

Indentation indicates block structure. For example, the body of the for loop that begins on line 1 consists of lines 2-8, and the body of the while loop that begins on line 5 contains lines 6-7 but not line 8. Our indentation style applies to if-else statements² as well. Using indentation instead of conventional indicators of block structure, such as begin and end statements, greatly reduces clutter while preserving, or even enhancing, clarity.³
The looping constructs while, for, and repeat-until and the if-else conditional construct have interpretations similar to those in C, C++, Java, Python, and Pascal.⁴ In this book, the loop counter retains its value after exiting the loop, unlike some situations that arise in C++, Java, and Pascal. Thus, immediately after a for loop, the loop counter’s value is the value that first exceeded the for loop bound. We used this property in our correctness argument for insertion sort. The for loop header in line 1 is for j = 2 to A.length, and so when this loop terminates, $j = A . l e n g t h + 1$ (or, equivalently, $j = n + 1$ , since $n = A . l e n g t h$ ). We use the keyword to when a for loop increments its loop counter in each iteration, and we use the keyword downto when a for loop decrements its loop counter. When the loop counter changes by an amount greater than 1, the amount of change follows the optional keyword by.
The symbol “//” indicates that the remainder of the line is a comment.
A multiple assignment of the form $i = j = e$ assigns to both variables $i$ and $j$ the value of expression $e$ ; it should be treated as equivalent to the assignment $j = e$ followed by the assignment $i = j$ .
Variables (such as $i$ , $j$ , and $k ey$ ) are local to the given procedure. We shall not use global variables without explicit indication.
We access array elements by specifying the array name followed by the index in square brackets. For example, $A [i]$ indicates the $i$ th element of the array $A$ . The notation " $..$ " is used to indicate a range of values within an array. Thus, $A [1.. j]$ indicates the subarray of $A$ consisting of the $j$ elements $A [1], A [2], \dots, A [j]$ .
We typically organize compound data into objects, which are composed of attributes. We access a particular attribute using the syntax found in many object-oriented programming languages: the object name, followed by a dot, followed by the attribute name. For example, we treat an array as an object with the attribute $l e n g t h$ indicating how many elements it contains. To specify the number of elements in an array $A$ , we write $A . l e n g t h$ .
We treat a variable representing an array or object as a pointer to the data representing the array or object. For all attributes $f$ of an object $x$ , setting $y = x$ causes $y . f$ to equal $x . f$ . Moreover, if we now set $x . f = 3$ , then afterward not only does $x . f$ equal $3$ , but $y . f$ equals $3$ as well. In other words, $x$ and $y$ point to the same object after the assignment $y = x$ .

Our attribute notation can “cascade.” For example, suppose that the attribute $f$ is itself a pointer to some type of object that has an attribute $g$ . Then the notation $x . f . g$ is implicitly parenthesized as $(x . f) . g$ . In other words, if we had assigned $y = x . f$ , then $x . f . g$ is the same as $y . g$ .

Sometimes, a pointer will refer to no object at all. In this case, we give it the special value NIL.
We pass parameters to a procedure by value: the called procedure receives its own copy of the parameters, and if it assigns a value to a parameter, the change is not seen by the calling procedure. When objects are passed, the pointer to the data representing the object is copied, but the object’s attributes are not. For example, if $x$ is a parameter of a called procedure, the assignment $x = y$ within the called procedure is not visible to the calling procedure. The assignment $x . f = 3$ , however, is visible. Similarly, arrays are passed by pointer, so that a pointer to the array is passed, rather than the entire array, and changes to individual array elements are visible to the calling procedure.
A return statement immediately transfers control back to the point of call in the calling procedure. Most return statements also take a value to pass back to the caller. Our pseudocode differs from many programming languages in that we allow multiple values to be returned in a single return statement.
The boolean operators and and or are short circuiting. That is, when we evaluate the expression ” $x$ and $y$ ” we first evaluate $x$ . If $x$ evaluates to FALSE, then the entire expression cannot evaluate to TRUE, and so we do not evaluate $y$ . If, on the other hand, $x$ evaluates to TRUE, we must evaluate $y$ to determine the value of the entire expression. Similarly, in the expression ” $x$ or $y$ ” we evaluate the expression $y$ only if $x$ evaluates to FALSE. Short-circuiting operators allow us to write boolean expressions such as ” $x \neq = NIL$ and $x . f = y$ ” without worrying about what happens when we try to evaluate $x . f$ when $x$ is NIL.
The keyword error indicates that an error occurred because conditions were wrong for the procedure to have been called. The calling procedure is responsible for handling the error, and so we do not specify what action to take.

² In an if-else statement, we indent else at the same level as its matching if. Although we omit the keyword then, we occasionally refer to the portion executed when the test following if is true as a then clause. For multiway tests, we use elseif for tests after the first one.

³ Each pseudocode procedure in this book appears on one page so that you will not have to discern levels of indentation in code that is split across pages.

⁴ Most block-structured languages have equivalent constructs, though the exact syntax may differ. Python lacks repeat-until loops, and its for loops operate a little differently from the for loops in this book.

Exercises 2.1-1 to 2.1-4

2.1-1 Using Figure 2.2 as a model, illustrate the operation of INSERTION-SORT on the array $A = ⟨ 31, 41, 59, 26, 41, 58 ⟩$ .

2.1-2 Rewrite the INSERTION-SORT procedure to sort into nonincreasing instead of nondecreasing order.

2.1-3 Consider the searching problem:

Input: A sequence of $n$ numbers $A = ⟨ a_{1}, a_{2}, \dots, a_{n} ⟩$ and a value $v$ .
Output: An index $i$ such that $v = A [i]$ or the special value NIL if $v$ does not appear in $A$ .

Write pseudocode for linear search, which scans through the sequence, looking for $v$ . Using a loop invariant, prove that your algorithm is correct. Make sure that your loop invariant fulfills the three necessary properties.

2.1-4 Consider the problem of adding two $n$ -bit binary integers, stored in two $n$ -element arrays $A$ and $B$ . The sum of the two integers should be stored in binary form in an $(n + 1)$ -element array $C$ . State the problem formally and write pseudocode for adding the two integers.

2.2 Analyzing algorithms

Analyzing an algorithm has come to mean predicting the resources that the algorithm requires. Occasionally, resources such as memory, communication bandwidth, or computer hardware are of primary concern, but most often it is computational time that we want to measure. Generally, by analyzing several candidate algorithms for a problem, we can identify a most efficient one. Such analysis may indicate more than one viable candidate, but we can often discard several inferior algorithms in the process.

Before we can analyze an algorithm, we must have a model of the implementation technology that we will use, including a model for the resources of that technology and their costs. For most of this book, we shall assume a generic one-processor, random-access machine (RAM) model of computation as our implementation technology and understand that our algorithms will be implemented as computer programs. In the RAM model, instructions are executed one after another, with no concurrent operations.

Strictly speaking, we should precisely define the instructions of the RAM model and their costs. To do so, however, would be tedious and would yield little insight into algorithm design and analysis. Yet we must be careful not to abuse the RAM model. For example, what if a RAM had an instruction that sorts? Then we could sort in just one instruction. Such a RAM would be unrealistic, since real computers do not have such instructions. Our guide, therefore, is how real computers are designed. The RAM model contains instructions commonly found in real computers: arithmetic (such as add, subtract, multiply, divide, remainder, floor, ceiling), data movement (load, store, copy), and control (conditional and unconditional branch, subroutine call and return). Each such instruction takes a constant amount of time.

The data types in the RAM model are integer and floating point (for storing real numbers). Although we typically do not concern ourselves with precision in this book, in some applications precision is crucial. We also assume a limit on the size of each word of data. For example, when working with inputs of size $n$ , we typically assume that integers are represented by $c l g n$ bits for some constant $c \geq 1$ . We require $c \geq 1$ so that each word can hold the value of $n$ , enabling us to index the individual input elements, and we restrict $c$ to be a constant so that the word size does not grow arbitrarily. (If the word size could grow arbitrarily, we could store huge amounts of data in one word and operate on it all in constant time, a clearly unrealistic scenario.)

Real computers contain instructions not listed above, and such instructions represent a gray area in the RAM model. For example, is exponentiation a constant-time instruction? In the general case, no; it takes several instructions to compute $x^{y}$ when $x$ and $y$ are real numbers. In restricted situations, however, exponentiation is a constant-time operation. Many computers have a “shift left” instruction, which in constant time shifts the bits of an integer by $k$ positions to the left. In most computers, shifting the bits of an integer by one position to the left is equivalent to multiplication by $2$ , so that shifting the bits by $k$ positions to the left is equivalent to multiplication by $2^{k}$ . Therefore, such computers can compute $2^{k}$ in one constant-time instruction by shifting the integer $1$ by $k$ positions to the left, as long as $k$ is no more than the number of bits in a computer word. We will endeavor to avoid such gray areas in the RAM model, but we will treat computation of $2^{k}$ as a constant-time operation when $k$ is a small enough positive integer.

In the RAM model, we do not attempt to model the memory hierarchy that is common in contemporary computers. That is, we do not model caches or virtual memory. Several computational models attempt to account for memory-hierarchy effects, which are sometimes significant in real programs on real machines. A handful of problems in this book examine memory-hierarchy effects, but for the most part, the analyses in this book will not consider them. Models that include the memory hierarchy are quite a bit more complex than the RAM model, and so they can be difficult to work with. Moreover, RAM-model analyses are usually excellent predictors of performance on actual machines.

Analyzing even a simple algorithm in the RAM model can be a challenge. The mathematical tools required may include combinatorics, probability theory, algebraic dexterity, and the ability to identify the most significant terms in a formula. Because the behavior of an algorithm may be different for each possible input, we need a means for summarizing that behavior in simple, easily understood formulas.

Even though we typically select only one machine model to analyze a given algorithm, we still face many choices in deciding how to express our analysis. We would like a way that is simple to write and manipulate, shows the important characteristics of an algorithm’s resource requirements, and suppresses tedious details.

Analysis of insertion sort

The time taken by the INSERTION-SORT procedure depends on the input: sorting a thousand numbers takes longer than sorting three numbers. Moreover, INSERTION-SORT can take different amounts of time to sort two input sequences of the same size depending on how nearly sorted they already are. In general, the time taken by an algorithm grows with the size of the input, so it is traditional to describe the running time of a program as a function of the size of its input. To do so, we need to define the terms “running time” and “size of input” more carefully.

The best notion for input size depends on the problem being studied. For many problems, such as sorting or computing discrete Fourier transforms, the most natural measure is the number of items in the input, for example, the array size $n$ for sorting. For many other problems, such as multiplying two integers, the best measure of input size is the total number of bits needed to represent the input in ordinary binary notation. Sometimes, it is more appropriate to describe the size of the input with two numbers rather than one. For instance, if the input to an algorithm is a graph, the input size can be described by the numbers of vertices and edges in the graph. We shall indicate which input size measure is being used with each problem we study.

The running time of an algorithm on a particular input is the number of primitive operations or “steps” executed. It is convenient to define the notion of step so that it is as machine-independent as possible. For the moment, let us adopt the following view. A constant amount of time is required to execute each line of our pseudocode. One line may take a different amount of time than another line, but we shall assume that each execution of the $i$ th line takes time $c_{i}$ , where $c_{i}$ is a constant. This viewpoint is in keeping with the RAM model, and it also reflects how the pseudocode would be implemented on most actual computers.⁵

In the following discussion, our expression for the running time of INSERTION-SORT will evolve from a messy formula that uses all the statement costs $c_{i}$ to a much simpler notation that is more concise and more easily manipulated. This simpler notation will also make it easy to determine whether one algorithm is more efficient than another.

We start by presenting the INSERTION-SORT procedure with the time “cost” of each statement and the number of times each statement is executed. For each $j = 2, 3, \dots, n$ , where $n = A . l e n g t h$ , we let $t_{j}$ denote the number of times the while loop test in line 5 is executed for that value of $j$ . When a for or while loop exits in the usual way (i.e., due to the test in the loop header), the test is executed one time more than the loop body. We assume that comments are not executable statements, and so they take no time.

⁵ There are some subtleties here. Computational steps that we specify in English are often variants of a procedure that requires more than just a constant amount of time. For example, later in this book we might say “sort the points by $x$ -coordinate,” which, as we shall see, takes more than a constant amount of time. Also, note that a statement that calls a subroutine takes constant time, though the subroutine, once invoked, may take more. That is, we separate the process of calling the subroutine (passing parameters to it, etc.) from the process of executing the subroutine.

INSERTION-SORT(A)                                          cost   times
1  for j = 2 to A.length                                   c1     n
2      key = A[j]                                          c2     n-1
3      // Insert A[j] into the sorted sequence A[1..j-1].  0      n-1
4      i = j - 1                                           c4     n-1
5      while i > 0 and A[i] > key                          c5     sum_{j=2}^{n} t_j
6          A[i+1] = A[i]                                   c6     sum_{j=2}^{n} (t_j - 1)
7          i = i - 1                                       c7     sum_{j=2}^{n} (t_j - 1)
8      A[i+1] = key                                        c8     n-1

The running time of the algorithm is the sum of running times for each statement executed; a statement that takes $c_{i}$ steps to execute and executes $n$ times will contribute $c_{i} n$ to the total running time.⁶ To compute $T (n)$ , the running time of INSERTION-SORT on an input of $n$ values, we sum the products of the cost and times columns, obtaining

T (n) = c_{1} n + c_{2} (n - 1) + c_{4} (n - 1) + c_{5} j = 2 \sum n t_{j} + c_{6} j = 2 \sum n (t_{j} - 1) + c_{7} j = 2 \sum n (t_{j} - 1) + c_{8} (n - 1) .

⁶ This characteristic does not necessarily hold for a resource such as memory. A statement that references $m$ words of memory and is executed $n$ times does not necessarily reference $mn$ distinct words of memory.

Even for inputs of a given size, an algorithm’s running time may depend on which input of that size is given. For example, in INSERTION-SORT, the best case occurs if the array is already sorted.

Best-case analysis. For each $j = 2, 3, \dots, n$ , we find that $A [i] \leq k ey$ in line 5 when $i$ has its initial value of $j - 1$ . So the test fails at once and the loop body never runs.
$t_{j} j = 2 \sum n t_{j} j = 2 \sum n (t_{j} - 1) = 1 for j = 2, 3, \dots, n = n - 1 (there are n - 1 terms, each equal to 1) = 0 (each term is 0)$
Substituting into the formula for $T (n)$ :
$T (n) = c_{1} n + c_{2} (n - 1) + c_{4} (n - 1) + c_{5} (n - 1) + c_{6} \cdot 0 + c_{7} \cdot 0 + c_{8} (n - 1) = c_{1} n + (c_{2} + c_{4} + c_{5} + c_{8}) (n - 1) = (c_{1} + c_{2} + c_{4} + c_{5} + c_{8}) n - (c_{2} + c_{4} + c_{5} + c_{8}) .$
We can express this running time as $an + b$ for constants $a$ and $b$ that depend on the statement costs $c_{i}$ ; it is thus a linear function of $n$ .

If the array is in reverse sorted order, that is, in decreasing order, the worst case results.

Worst-case analysis. We must compare each element $A [j]$ with each element in the entire sorted subarray $A [1.. j - 1]$ , and so $t_{j} = j$ for $j = 2, 3, \dots, n$ . (The loop runs until $i$ reaches $0$ , so the test is evaluated for $i = j - 1, j - 2, \dots, 1, 0$ , which is $j$ times.)

We need two sums (see Appendix A for a review of how to solve these summations):
$j = 2 \sum n j j = 2 \sum n (j - 1) = j = 1 \sum n j - 1 = \frac{n ( n + 1 )}{2} - 1 = k = 1 \sum n - 1 k = \frac{n ( n - 1 )}{2} (substituting k = j - 1)$
Substituting $t_{j} = j$ into the formula for $T (n)$ :
$T (n) = c_{1} n + c_{2} (n - 1) + c_{4} (n - 1) + c_{5} (\frac{n ( n + 1 )}{2} - 1) + c_{6} (\frac{n ( n - 1 )}{2}) + c_{7} (\frac{n ( n - 1 )}{2}) + c_{8} (n - 1)$
Expand each product and collect powers of $n$ :
$c_{5} (\frac{n ( n + 1 )}{2} - 1) c_{6} \frac{n ( n - 1 )}{2} c_{7} \frac{n ( n - 1 )}{2} = \frac{c _{5}}{2} n^{2} + \frac{c _{5}}{2} n - c_{5} = \frac{c _{6}}{2} n^{2} - \frac{c _{6}}{2} n = \frac{c _{7}}{2} n^{2} - \frac{c _{7}}{2} n$
Adding all terms:
$T (n) = (\frac{c _{5}}{2} + \frac{c _{6}}{2} + \frac{c _{7}}{2}) n^{2} + (c_{1} + c_{2} + c_{4} + \frac{c _{5}}{2} - \frac{c _{6}}{2} - \frac{c _{7}}{2} + c_{8}) n - (c_{2} + c_{4} + c_{5} + c_{8}) .$
We can express this worst-case running time as $a n^{2} + bn + c$ for constants $a$ , $b$ , and $c$ that again depend on the statement costs $c_{i}$ ; it is thus a quadratic function of $n$ .

Typically, as in insertion sort, the running time of an algorithm is fixed for a given input, although in later chapters we shall see some interesting “randomized” algorithms whose behavior can vary even for a fixed input.

Think about it. Sorting a nearly sorted deck is fast; sorting a deck dealt in exactly the wrong order is slow. Is it fair to judge an algorithm by its luckiest day or its unluckiest day? The next subsection explains which one we usually pick.

Worst-case and average-case analysis

In our analysis of insertion sort, we looked at both the best case, in which the input array was already sorted, and the worst case, in which the input array was reverse sorted. For the remainder of this book, though, we shall usually concentrate on finding only the worst-case running time, that is, the longest running time for any input of size $n$ . We give three reasons for this orientation.

The worst-case running time of an algorithm gives us an upper bound on the running time for any input. Knowing it provides a guarantee that the algorithm will never take any longer. We need not make some educated guess about the running time and hope that it never gets much worse.
For some algorithms, the worst case occurs fairly often. For example, in searching a database for a particular piece of information, the searching algorithm’s worst case will often occur when the information is not present in the database.
The “average case” is often roughly as bad as the worst case. Suppose that we randomly choose $n$ numbers and apply insertion sort. How long does it take to determine where in subarray $A [1.. j - 1]$ to insert element $A [j]$ ? On average, half the elements in $A [1.. j - 1]$ are less than $A [j]$ , and half the elements are greater. On average, therefore, we check half of the subarray $A [1.. j - 1]$ , and so $t_{j}$ is about $j /2$ . The resulting average-case running time turns out to be a quadratic function of the input size, just like the worst-case running time.

In some particular cases, we shall be interested in the average-case running time of an algorithm; we shall see the technique of probabilistic analysis applied to various algorithms throughout this book. The scope of average-case analysis is limited, because it may not be apparent what constitutes an “average” input for a particular problem. Often, we shall assume that all inputs of a given size are equally likely. In practice, this assumption may be violated, but we can sometimes use a randomized algorithm, which makes random choices, to allow a probabilistic analysis and yield an expected running time. We explore randomized algorithms more in Chapter 5 and in several other subsequent chapters.

Order of growth

We used some simplifying abstractions to ease our analysis of the INSERTION-SORT procedure. First, we ignored the actual cost of each statement, using the constants $c_{i}$ to represent these costs. Then, we observed that even these constants give us more detail than we really need: we expressed the worst-case running time as $a n^{2} + bn + c$ for some constants $a$ , $b$ , and $c$ that depend on the statement costs $c_{i}$ . We thus ignored not only the actual statement costs, but also the abstract costs $c_{i}$ .

We shall now make one more simplifying abstraction: it is the rate of growth, or order of growth, of the running time that really interests us. We therefore consider only the leading term of a formula (e.g., $a n^{2}$ ), since the lower-order terms are relatively insignificant for large values of $n$ . We also ignore the leading term’s constant coefficient, since constant factors are less significant than the rate of growth in determining computational efficiency for large inputs. For insertion sort, when we ignore the lower-order terms and the leading term’s constant coefficient, we are left with the factor of $n^{2}$ from the leading term. We write that insertion sort has a worst-case running time of $Θ (n^{2})$ (pronounced “theta of $n$ -squared”). We shall use $Θ$ -notation informally in this chapter, and we will define it precisely in Chapter 3.

We usually consider one algorithm to be more efficient than another if its worst-case running time has a lower order of growth. Due to constant factors and lower-order terms, an algorithm whose running time has a higher order of growth might take less time for small inputs than an algorithm whose running time has a lower order of growth. But for large enough inputs, a $Θ (n^{2})$ algorithm, for example, will run more quickly in the worst case than a $Θ (n^{3})$ algorithm.

Exercises 2.2-1 to 2.2-4

2.2-1 Express the function $n^{3} /1000 - 100 n^{2} - 100 n + 3$ in terms of $Θ$ -notation.

2.2-2 Consider sorting $n$ numbers stored in array $A$ by first finding the smallest element of $A$ and exchanging it with the element in $A [1]$ . Then find the second smallest element of $A$ , and exchange it with $A [2]$ . Continue in this manner for the first $n - 1$ elements of $A$ . Write pseudocode for this algorithm, which is known as selection sort. What loop invariant does this algorithm maintain? Why does it need to run for only the first $n - 1$ elements, rather than for all $n$ elements? Give the best-case and worst-case running times of selection sort in $Θ$ -notation.

2.2-3 Consider linear search again (see Exercise 2.1-3). How many elements of the input sequence need to be checked on the average, assuming that the element being searched for is equally likely to be any element in the array? How about in the worst case? What are the average-case and worst-case running times of linear search in $Θ$ -notation? Justify your answers.

2.2-4 How can we modify almost any algorithm to have a good best-case running time?

2.3 Designing algorithms

We can choose from a wide range of algorithm design techniques. For insertion sort, we used an incremental approach: having sorted the subarray $A [1.. j - 1]$ , we inserted the single element $A [j]$ into its proper place, yielding the sorted subarray $A [1.. j]$ .

In this section, we examine an alternative design approach, known as “divide-and-conquer,” which we shall explore in more detail in Chapter 4. We’ll use divide-and-conquer to design a sorting algorithm whose worst-case running time is much less than that of insertion sort. One advantage of divide-and-conquer algorithms is that their running times are often easily determined using techniques that we will see in Chapter 4.

2.3.1 The divide-and-conquer approach

Many useful algorithms are recursive in structure: to solve a given problem, they call themselves recursively one or more times to deal with closely related subproblems. These algorithms typically follow a divide-and-conquer approach: they break the problem into several subproblems that are similar to the original problem but smaller in size, solve the subproblems recursively, and then combine these solutions to create a solution to the original problem.

The divide-and-conquer paradigm involves three steps at each level of the recursion:

Divide the problem into a number of subproblems that are smaller instances of the same problem.
Conquer the subproblems by solving them recursively. If the subproblem sizes are small enough, however, just solve the subproblems in a straightforward manner.
Combine the solutions to the subproblems into the solution for the original problem.

The merge sort algorithm closely follows the divide-and-conquer paradigm. Intuitively, it operates as follows.

Divide: Divide the $n$ -element sequence to be sorted into two subsequences of $n /2$ elements each.
Conquer: Sort the two subsequences recursively using merge sort.
Combine: Merge the two sorted subsequences to produce the sorted answer.

The recursion “bottoms out” when the sequence to be sorted has length 1, in which case there is no work to be done, since every sequence of length 1 is already in sorted order.

Think about it. Imagine sorting a huge pile of exam papers with a friend: each of you sorts half, then you zip the two sorted stacks together. Why is zipping two already sorted stacks so much cheaper than sorting from scratch?

The key operation of the merge sort algorithm is the merging of two sorted sequences in the “combine” step. We merge by calling an auxiliary procedure MERGE $(A, p, q, r)$ , where $A$ is an array and $p$ , $q$ , and $r$ are indices into the array such that $p \leq q < r$ . The procedure assumes that the subarrays $A [p .. q]$ and $A [q + 1.. r]$ are in sorted order. It merges them to form a single sorted subarray that replaces the current subarray $A [p .. r]$ .

Our MERGE procedure takes time $Θ (n)$ , where $n = r - p + 1$ is the total number of elements being merged, and it works as follows. Returning to our card-playing motif, suppose we have two piles of cards face up on a table. Each pile is sorted, with the smallest cards on top. We wish to merge the two piles into a single sorted output pile, which is to be face down on the table. Our basic step consists of choosing the smaller of the two cards on top of the face-up piles, removing it from its pile (which exposes a new top card), and placing this card face down onto the output pile. We repeat this step until one input pile is empty, at which time we just take the remaining input pile and place it face down onto the output pile. Computationally, each basic step takes constant time, since we are comparing just the two top cards. Since we perform at most $n$ basic steps, merging takes $Θ (n)$ time.

The following pseudocode implements the above idea, but with an additional twist that avoids having to check whether either pile is empty in each basic step. We place on the bottom of each pile a sentinel card, which contains a special value that we use to simplify our code. Here, we use $\infty$ as the sentinel value, so that whenever a card with $\infty$ is exposed, it cannot be the smaller card unless both piles have their sentinel cards exposed. But once that happens, all the nonsentinel cards have already been placed onto the output pile. Since we know in advance that exactly $r - p + 1$ cards will be placed onto the output pile, we can stop once we have performed that many basic steps.

MERGE(A, p, q, r)
1   n1 = q - p + 1
2   n2 = r - q
3   let L[1..n1+1] and R[1..n2+1] be new arrays
4   for i = 1 to n1
5       L[i] = A[p + i - 1]
6   for j = 1 to n2
7       R[j] = A[q + j]
8   L[n1 + 1] = infinity
9   R[n2 + 1] = infinity
10  i = 1
11  j = 1
12  for k = p to r
13      if L[i] <= R[j]
14          A[k] = L[i]
15          i = i + 1
16      else A[k] = R[j]
17          j = j + 1

In detail, the MERGE procedure works as follows. Line 1 computes the length $n_{1}$ of the subarray $A [p .. q]$ , and line 2 computes the length $n_{2}$ of the subarray $A [q + 1.. r]$ . We create arrays $L$ and $R$ (“left” and “right”), of lengths $n_{1} + 1$ and $n_{2} + 1$ , respectively, in line 3; the extra position in each array will hold the sentinel. The for loop of lines 4-5 copies the subarray $A [p .. q]$ into $L [1.. n_{1}]$ , and the for loop of lines 6-7 copies the subarray $A [q + 1.. r]$ into $R [1.. n_{2}]$ . Lines 8-9 put the sentinels at the ends of the arrays $L$ and $R$ . Lines 10-17, illustrated in Figure 2.3, perform the $r - p + 1$ basic steps by maintaining the following loop invariant:

At the start of each iteration of the for loop of lines 12-17, the subarray $A [p .. k - 1]$ contains the $k - p$ smallest elements of $L [1.. n_{1} + 1]$ and $R [1.. n_{2} + 1]$ , in sorted order. Moreover, $L [i]$ and $R [j]$ are the smallest elements of their arrays that have not been copied back into $A$ .

Figure 2.3: The operation of lines 10-17 in the call MERGE $(A, 9, 12, 16)$ , when the subarray $A [9..16]$ contains the sequence $⟨ 2, 4, 5, 7, 1, 2, 3, 6 ⟩$ . After copying and inserting sentinels, the array $L$ contains $⟨ 2, 4, 5, 7, \infty ⟩$ and the array $R$ contains $⟨ 1, 2, 3, 6, \infty ⟩$ . Lightly shaded positions in $A$ contain their final values, and lightly shaded positions in $L$ and $R$ contain values that have yet to be copied back into $A$ . Taken together, the lightly shaded positions always comprise the values originally in $A [9..16]$ , along with the two sentinels. Heavily shaded positions in $A$ contain values that will be copied over, and heavily shaded positions in $L$ and $R$ contain values that have already been copied back into $A$ . (a)-(h) The arrays $A$ , $L$ , and $R$ , and their respective indices $k$ , $i$ , and $j$ prior to each iteration of the loop of lines 12-17. (i) The arrays and indices at termination. At this point, the subarray in $A [9..16]$ is sorted, and the two sentinels in $L$ and $R$ are the only two elements in these arrays that have not been copied into $A$ .

Final state of the example, in sorted order: $A [9..16] = ⟨ 1, 2, 2, 3, 4, 5, 6, 7 ⟩$ .

We must show that this loop invariant holds prior to the first iteration of the for loop of lines 12-17, that each iteration of the loop maintains the invariant, and that the invariant provides a useful property to show correctness when the loop terminates.

Proof (correctness of MERGE via the loop invariant).

Initialization. Prior to the first iteration of the loop, we have $k = p$ .
$k = p i = j = 1 ⟹ A [p .. k - 1] = A [p .. p - 1] is empty ⟹ A [p .. k - 1] contains the k - p = 0 smallest elements of L and R (vacuously, and trivially sorted) ⟹ L [1] and R [1] are the first elements of their arrays$
Since $L$ and $R$ are sorted and nothing has yet been copied back, $L [i]$ and $R [j]$ are the smallest elements of their arrays that have not been copied back into $A$ .

Maintenance. To see that each iteration maintains the loop invariant, let us first suppose that $L [i] \leq R [j]$ .
$L [i] is the smallest element of L not yet copied, and L [i] \leq R [j] is the smallest of R not yet copied ⟹ L [i] is the smallest element not yet copied back into A A [p .. k - 1] holds the k - p smallest elements ⟹ after line 14 copies L [i] into A [k], A [p .. k] holds the k - p + 1 smallest elements, in sorted order$
Incrementing $k$ (in the for loop update) and $i$ (in line 15) reestablishes the loop invariant for the next iteration: $A [p .. k^{'} - 1]$ with $k^{'} = k + 1$ holds the $k^{'} - p$ smallest elements, and the new $L [i^{'}]$ with $i^{'} = i + 1$ is the smallest element of $L$ not yet copied, while $R [j]$ is unchanged.

If instead $L [i] > R [j]$ , then lines 16-17 perform the symmetric action: $R [j]$ is the smallest element not yet copied, line 16 copies it into $A [k]$ , and incrementing $k$ and $j$ (line 17) reestablishes the loop invariant.

Termination. At termination, $k = r + 1$ .
$A [p .. k - 1] = A [p .. r] ∣ L ∣ + ∣ R ∣ contains the k - p = r - p + 1 smallest elements of L [1.. n_{1} + 1] and R [1.. n_{2} + 1], in sorted order = (n_{1} + 1) + (n_{2} + 1) = n_{1} + n_{2} + 2 = r - p + 3$
The arrays $L$ and $R$ together contain $r - p + 3$ elements. All but the two largest have been copied back into $A$ (that is, $r - p + 1$ of them), and these two largest elements are the sentinels. Hence $A [p .. r]$ is sorted. $■$

To see that the MERGE procedure runs in $Θ (n)$ time, where $n = r - p + 1$ , observe that each of lines 1-3 and 8-11 takes constant time, the for loops of lines 4-7 take $Θ (n_{1} + n_{2}) = Θ (n)$ time,⁷ and there are $n$ iterations of the for loop of lines 12-17, each of which takes constant time.

Altogether, the running time of MERGE is $Θ (n)$ .

⁷ We shall see in Chapter 3 how to formally interpret equations containing $Θ$ -notation.

We can now use the MERGE procedure as a subroutine in the merge sort algorithm. The procedure MERGE-SORT $(A, p, r)$ sorts the elements in the subarray $A [p .. r]$ . If $p \geq r$ , the subarray has at most one element and is therefore already sorted. Otherwise, the divide step simply computes an index $q$ that partitions $A [p .. r]$ into two subarrays: $A [p .. q]$ , containing $⌈ n /2 ⌉$ elements, and $A [q + 1.. r]$ , containing $⌊ n /2 ⌋$ elements.⁸

⁸ The expression $⌈ x ⌉$ denotes the least integer greater than or equal to $x$ , and $⌊ x ⌋$ denotes the greatest integer less than or equal to $x$ . These notations are defined in Chapter 3. The easiest way to verify that setting $q$ to $⌊(p + r) /2 ⌋$ yields subarrays $A [p .. q]$ and $A [q + 1.. r]$ of sizes $⌈ n /2 ⌉$ and $⌊ n /2 ⌋$ , respectively, is to examine the four cases that arise depending on whether each of $p$ and $r$ is odd or even.

MERGE-SORT(A, p, r)
1  if p < r
2      q = floor((p + r) / 2)
3      MERGE-SORT(A, p, q)
4      MERGE-SORT(A, q + 1, r)
5      MERGE(A, p, q, r)

To sort the entire sequence $A = ⟨ A [1], A [2], \dots, A [n]⟩$ , we make the initial call MERGE-SORT $(A, 1, A . l e n g t h)$ , where once again $A . l e n g t h = n$ . Figure 2.4 illustrates the operation of the procedure bottom-up when $n$ is a power of 2. The algorithm consists of merging pairs of 1-item sequences to form sorted sequences of length 2, merging pairs of sequences of length 2 to form sorted sequences of length 4, and so on, until two sequences of length $n /2$ are merged to form the final sorted sequence of length $n$ .

Figure 2.4: The operation of merge sort on the array $A = ⟨ 5, 2, 4, 7, 1, 3, 2, 6 ⟩$ . The lengths of the sorted sequences being merged increase as the algorithm progresses from bottom to top.

The figure shows a binary tree of merges: the initial sequence at the bottom, successive levels of pairwise merges above it, and the final sorted sequence at the top.

Think about it. Trace the recursion on the eight numbers above. The first real work happens only when the recursion reaches subarrays of length 1 and starts returning. Where do the actual comparisons occur: on the way down or on the way up?

2.3.2 Analyzing divide-and-conquer algorithms

When an algorithm contains a recursive call to itself, we can often describe its running time by a recurrence equation or recurrence, which describes the overall running time on a problem of size $n$ in terms of the running time on smaller inputs. We can then use mathematical tools to solve the recurrence and provide bounds on the performance of the algorithm.

A recurrence for the running time of a divide-and-conquer algorithm falls out from the three steps of the basic paradigm. As before, we let $T (n)$ be the running time on a problem of size $n$ . If the problem size is small enough, say $n \leq c$ for some constant $c$ , the straightforward solution takes constant time, which we write as $Θ (1)$ . Suppose that our division of the problem yields $a$ subproblems, each of which is $1/ b$ the size of the original. (For merge sort, both $a$ and $b$ are 2, but we shall see many divide-and-conquer algorithms in which $a \neq = b$ .) It takes time $T (n / b)$ to solve one subproblem of size $n / b$ , and so it takes time $a T (n / b)$ to solve $a$ of them. If we take $D (n)$ time to divide the problem into subproblems and $C (n)$ time to combine the solutions to the subproblems into the solution to the original problem, we get the recurrence

T (n) = {Θ (1) a T (n / b) + D (n) + C (n) if n \leq c, otherwise.

In Chapter 4, we shall see how to solve common recurrences of this form.

Analysis of merge sort

Although the pseudocode for MERGE-SORT works correctly when the number of elements is not even, our recurrence-based analysis is simplified if we assume that the original problem size is a power of 2. Each divide step then yields two subsequences of size exactly $n /2$ . In Chapter 4, we shall see that this assumption does not affect the order of growth of the solution to the recurrence.

We reason as follows to set up the recurrence for $T (n)$ , the worst-case running time of merge sort on $n$ numbers. Merge sort on just one element takes constant time. When we have $n > 1$ elements, we break down the running time as follows.

Divide: The divide step just computes the middle of the subarray, which takes constant time. Thus, $D (n) = Θ (1)$ .
Conquer: We recursively solve two subproblems, each of size $n /2$ , which contributes $2 T (n /2)$ to the running time.
Combine: We have already noted that the MERGE procedure on an $n$ -element subarray takes time $Θ (n)$ , and so $C (n) = Θ (n)$ .

When we add the functions $D (n)$ and $C (n)$ for the merge sort analysis, we are adding a function that is $Θ (n)$ and a function that is $Θ (1)$ . This sum is a linear function of $n$ , that is, $Θ (n)$ . Adding it to the $2 T (n /2)$ term from the “conquer” step gives the recurrence for the worst-case running time $T (n)$ of merge sort:

T (n) = {Θ (1) 2 T (n /2) + Θ (n) if n = 1, if n > 1. (2.1)

In Chapter 4, we shall see the “master theorem,” which we can use to show that $T (n)$ is $Θ (n l g n)$ , where $l g n$ stands for $lo g_{2} n$ . Because the logarithm function grows more slowly than any linear function, for large enough inputs, merge sort, with its $Θ (n l g n)$ running time, outperforms insertion sort, whose running time is $Θ (n^{2})$ , in the worst case.

We do not need the master theorem to intuitively understand why the solution to the recurrence (2.1) is $T (n) = Θ (n l g n)$ . Let us rewrite recurrence (2.1) as

T (n) = {c 2 T (n /2) + c n if n = 1, if n > 1, (2.2)

where the constant $c$ represents the time required to solve problems of size 1 as well as the time per array element of the divide and combine steps.⁹

⁹ It is unlikely that the same constant exactly represents both the time to solve problems of size 1 and the time per array element of the divide and combine steps. We can get around this problem by letting $c$ be the greater of these times and understanding that our recurrence gives an upper bound on the running time, or by letting $c$ be the lesser of these times and understanding that our recurrence gives a lower bound on the running time. Both bounds are on the order of $n l g n$ and, taken together, give a $Θ (n l g n)$ running time.

Figure 2.5 shows how we can solve recurrence (2.2). For convenience, we assume that $n$ is an exact power of 2. Part (a) of the figure shows $T (n)$ , which we expand in part (b) into an equivalent tree representing the recurrence. The $c n$ term is the root (the cost incurred at the top level of recursion), and the two subtrees of the root are the two smaller recurrences $T (n /2)$ . Part (c) shows this process carried one step further by expanding $T (n /2)$ . The cost incurred at each of the two subnodes at the second level of recursion is $c n /2$ . We continue expanding each node in the tree by breaking it into its constituent parts as determined by the recurrence, until the problem sizes get down to 1, each with a cost of $c$ . Part (d) shows the resulting recursion tree.

Figure 2.5: How to construct a recursion tree for the recurrence $T (n) = 2 T (n /2) + c n$ . Part (a) shows $T (n)$ , which progressively expands in (b)-(d) to form the recursion tree. The fully expanded tree in part (d) has $l g n + 1$ levels (i.e., it has height $l g n$ ), and each level contributes a total cost of $c n$ . The total cost, therefore, is $c n l g n + c n$ , which is $Θ (n l g n)$ .

The figure shows the root $c n$ , then two children $c n /2$ , then four children $c n /4$ , and so on down to $n$ leaves each costing $c$ ; every level sums to $c n$ .

Next, we add the costs across each level of the tree.

Cost per level.
$top level: level 1: level 2: level i : bottom level: c n c (n /2) + c (n /2) = c n 4 \cdot c (n /4) = c n 2^{i} nodes, each of cost c (n / 2^{i}), so 2^{i} \cdot c \frac{n}{2 ^{i}} = c n n nodes, each of cost c, total c n$
At level $i$ there are $2^{i}$ nodes because each node has two children; each node there is a subproblem of size $n / 2^{i}$ and contributes $c (n / 2^{i})$ . The bottom level has $n$ leaves because subproblems of size 1 are reached when $n / 2^{i} = 1$ , that is, $2^{i} = n$ .

The total number of levels of the recursion tree in Figure 2.5 is $l g n + 1$ , where $n$ is the number of leaves, corresponding to the input size. An informal inductive argument justifies this claim.

Proof (number of levels is $l g n + 1$ ).

Base case. $n = 1$ . The tree has only one level (a single node).
$l g 1 + 1 = 0 + 1 = 1 (so the formula gives the correct number of levels)$
Inductive hypothesis. Assume that a recursion tree with $2^{i}$ leaves has $l g 2^{i} + 1 = i + 1$ levels, using $l g 2^{i} = i$ for any $i$ .

Inductive step. Because the input size is a power of 2, the next input size to consider is $2^{i + 1}$ . A tree with $n = 2^{i + 1}$ leaves has one more level than a tree with $2^{i}$ leaves (the new root splits into two subtrees, each with $2^{i}$ leaves). Therefore
$levels = (i + 1) + 1 (by the inductive hypothesis, plus one more level) = l g 2^{i + 1} + 1 (since l g 2^{i + 1} = i + 1) . ■$

Total cost. To compute the total cost represented by the recurrence (2.2), we simply add up the costs of all the levels.
$total = (number of levels) \cdot (cost per level) = (l g n + 1) \cdot c n = c n l g n + c n .$
Ignoring the low-order term $c n$ and the constant $c$ gives the desired result of $Θ (n l g n)$ .

Think about it. Each level of the tree costs the same, $c n$ , and halving can happen only about $l g n$ times before pieces reach size 1. So total work is “work per level” times “number of levels”. Can you see the same pattern in how a tournament bracket with $n$ players has $l g n$ rounds?

Exercises 2.3-1 to 2.3-7

2.3-1 Using Figure 2.4 as a model, illustrate the operation of merge sort on the array $A = ⟨ 3, 41, 52, 26, 38, 57, 9, 49 ⟩$ .

2.3-2 Rewrite the MERGE procedure so that it does not use sentinels, instead stopping once either array $L$ or $R$ has had all its elements copied back to $A$ and then copying the remainder of the other array back into $A$ .

2.3-3 Use mathematical induction to show that when $n$ is an exact power of 2, the solution of the recurrence

T (n) = {2 2 T (n /2) + n if n = 2, if n = 2^{k}, for k > 1

is $T (n) = n l g n$ .

2.3-4 We can express insertion sort as a recursive procedure as follows. In order to sort $A [1.. n]$ , we recursively sort $A [1.. n - 1]$ and then insert $A [n]$ into the sorted array $A [1.. n - 1]$ . Write a recurrence for the worst-case running time of this recursive version of insertion sort.

2.3-5 Referring back to the searching problem (see Exercise 2.1-3), observe that if the sequence $A$ is sorted, we can check the midpoint of the sequence against $v$ and eliminate half of the sequence from further consideration. The binary search algorithm repeats this procedure, halving the size of the remaining portion of the sequence each time. Write pseudocode, either iterative or recursive, for binary search. Argue that the worst-case running time of binary search is $Θ (l g n)$ .

2.3-6 Observe that the while loop of lines 5-7 of the INSERTION-SORT procedure in Section 2.1 uses a linear search to scan (backward) through the sorted subarray $A [1.. j - 1]$ . Can we use a binary search (see Exercise 2.3-5) instead to improve the overall worst-case running time of insertion sort to $Θ (n l g n)$ ?

2.3-7* Describe a $Θ (n l g n)$ -time algorithm that, given a set $S$ of $n$ integers and another integer $x$ , determines whether or not there exist two elements in $S$ whose sum is exactly $x$ .

Problems

2-1 Insertion sort on small arrays in merge sort

Although merge sort runs in $Θ (n l g n)$ worst-case time and insertion sort runs in $Θ (n^{2})$ worst-case time, the constant factors in insertion sort can make it faster in practice for small problem sizes on many machines. Thus, it makes sense to coarsen the leaves of the recursion by using insertion sort within merge sort when subproblems become sufficiently small. Consider a modification to merge sort in which $n / k$ sublists of length $k$ are sorted using insertion sort and then merged using the standard merging mechanism, where $k$ is a value to be determined.

a. Show that insertion sort can sort the $n / k$ sublists, each of length $k$ , in $Θ (nk)$ worst-case time.

b. Show how to merge the sublists in $Θ (n l g (n / k))$ worst-case time.

c. Given that the modified algorithm runs in $Θ (nk + n l g (n / k))$ worst-case time, what is the largest value of $k$ as a function of $n$ for which the modified algorithm has the same running time as standard merge sort, in terms of $Θ$ -notation?

d. How should we choose $k$ in practice?

2-2 Correctness of bubblesort

Bubblesort is a popular, but inefficient, sorting algorithm. It works by repeatedly swapping adjacent elements that are out of order.

BUBBLESORT(A)
1  for i = 1 to A.length - 1
2      for j = A.length downto i + 1
3          if A[j] < A[j - 1]
4              exchange A[j] with A[j - 1]

a. Let $A^{'}$ denote the output of BUBBLESORT $(A)$ . To prove that BUBBLESORT is correct, we need to prove that it terminates and that

A^{'} [1] \leq A^{'} [2] \leq \dots \leq A^{'} [n], (2.3)

where $n = A . l e n g t h$ . In order to show that BUBBLESORT actually sorts, what else do we need to prove?

The next two parts will prove inequality (2.3).

b. State precisely a loop invariant for the for loop in lines 2-4, and prove that this loop invariant holds. Your proof should use the structure of the loop invariant proof presented in this chapter.

c. Using the termination condition of the loop invariant proved in part (b), state a loop invariant for the for loop in lines 1-4 that will allow you to prove inequality (2.3). Your proof should use the structure of the loop invariant proof presented in this chapter.

d. What is the worst-case running time of bubblesort? How does it compare to the running time of insertion sort?

2-3 Correctness of Horner’s rule

The following code fragment implements Horner’s rule for evaluating a polynomial

P (x) = k = 0 \sum n a_{k} x^{k} = a_{0} + x (a_{1} + x (a_{2} + \dots + x (a_{n - 1} + x a_{n}) \dots)),

given the coefficients $a_{0}, a_{1}, \dots, a_{n}$ and a value for $x$ :

1  y = 0
2  for i = n downto 0
3      y = a_i + x * y

a. In terms of $Θ$ -notation, what is the running time of this code fragment for Horner’s rule?

b. Write pseudocode to implement the naive polynomial-evaluation algorithm that computes each term of the polynomial from scratch. What is the running time of this algorithm? How does it compare to Horner’s rule?

c. Consider the following loop invariant:

At the start of each iteration of the for loop of lines 2-3,
$y = k = 0 \sum n - (i + 1) a_{k + i + 1} x^{k} .$

Interpret a summation with no terms as equaling $0$ . Following the structure of the loop invariant proof presented in this chapter, use this loop invariant to show that, at termination, $y = \sum_{k = 0}^{n} a_{k} x^{k}$ .

d. Conclude by arguing that the given code fragment correctly evaluates a polynomial characterized by the coefficients $a_{0}, a_{1}, \dots, a_{n}$ .

2-4 Inversions

Let $A [1.. n]$ be an array of $n$ distinct numbers. If $i < j$ and $A [i] > A [j]$ , then the pair $(i, j)$ is called an inversion of $A$ .

a. List the five inversions of the array $⟨ 2, 3, 8, 6, 1 ⟩$ .

b. What array with elements from the set ${1, 2, \dots, n}$ has the most inversions? How many does it have?

c. What is the relationship between the running time of insertion sort and the number of inversions in the input array? Justify your answer.

d. Give an algorithm that determines the number of inversions in any permutation on $n$ elements in $Θ (n l g n)$ worst-case time. (Hint: Modify merge sort.)

Chapter notes

In 1968, Knuth published the first of three volumes with the general title The Art of Computer Programming [209, 210, 211]. The first volume ushered in the modern study of computer algorithms with a focus on the analysis of running time, and the full series remains an engaging and worthwhile reference for many of the topics presented here. According to Knuth, the word “algorithm” is derived from the name of “al-Khowârizmî,” a ninth-century Persian mathematician.

Aho, Hopcroft, and Ullman [5] advocated the asymptotic analysis of algorithms (using notations that Chapter 3 introduces, including $Θ$ -notation) as a means of comparing relative performance. They also popularized the use of recurrence relations to describe the running times of recursive algorithms.

Knuth [211] provides an encyclopedic treatment of many sorting algorithms. His comparison of sorting algorithms (page 381) includes exact step-counting analyses, like the one we performed here for insertion sort. Knuth’s discussion of insertion sort encompasses several variations of the algorithm. The most important of these is Shell’s sort, introduced by D. L. Shell, which uses insertion sort on periodic subsequences of the input to produce a faster sorting algorithm.

Merge sort is also described by Knuth. He mentions that a mechanical collator capable of merging two decks of punched cards in a single pass was invented in 1938. J. von Neumann, one of the pioneers of computer science, apparently wrote a program for merge sort on the EDVAC computer in 1945.

The early history of proving programs correct is described by Gries [153], who credits P. Naur with the first article in this field. Gries attributes loop invariants to R. W. Floyd. The textbook by Mitchell [256] describes more recent progress in proving programs correct.

3 Growth of Functions

Big picture. Imagine two runners: one sprints at a fixed speed, the other starts slowly but keeps accelerating. Over a short dash the sprinter wins, but over a marathon the accelerating runner wins no matter how big the sprinter’s head start. Asymptotic notation is the language we use to say “who wins the marathon” without arguing about head starts, shoes, or the weather (the constants and lower-order terms).

The order of growth of the running time of an algorithm, defined in Chapter 2, gives a simple characterization of the algorithm’s efficiency and also allows us to compare the relative performance of alternative algorithms. Once the input size $n$ becomes large enough, merge sort, with its $O (n l g n)$ worst-case running time, beats insertion sort, whose worst-case running time is $O (n^{2})$ . Although we can sometimes determine the exact running time of an algorithm, as we did for insertion sort in Chapter 2, the extra precision is not usually worth the effort of computing it. For large enough inputs, the multiplicative constants and lower-order terms of an exact running time are dominated by the effects of the input size itself.

When we look at input sizes large enough to make only the order of growth of the running time relevant, we are studying the asymptotic efficiency of algorithms. That is, we are concerned with how the running time of an algorithm increases with the size of the input in the limit, as the size of the input increases without bound. Usually, an algorithm that is asymptotically more efficient will be the best choice for all but very small inputs.

This chapter gives several standard methods for simplifying the asymptotic analysis of algorithms. The next section begins by defining several types of “asymptotic notation,” of which we have already seen an example in $O$ -notation. We then present several notational conventions used throughout this book, and finally we review the behavior of functions that commonly arise in the analysis of algorithms.

3.1 Asymptotic notation

The notations we use to describe the asymptotic running time of an algorithm are defined in terms of functions whose domains are the set of natural numbers $N = {0, 1, 2, \dots}$ . Such notations are convenient for describing the worst-case running-time function $T (n)$ , which usually is defined only on integer input sizes. We sometimes find it convenient, however, to abuse asymptotic notation in a variety of ways. For example, we might extend the notation to the domain of real numbers or, alternatively, restrict it to a subset of the natural numbers. We should make sure, however, to understand the precise meaning of the notation so that when we abuse, we do not misuse it. This section defines the basic asymptotic notations and also introduces some common abuses.

Asymptotic notation, functions, and running times

We will use asymptotic notation primarily to describe the running times of algorithms, as when we wrote that insertion sort’s worst-case running time is $O (n^{2})$ . Asymptotic notation actually applies to functions, however. Recall that we characterized insertion sort’s worst-case running time as $a n^{2} + bn + c$ , for some constants $a$ , $b$ , and $c$ . By writing that insertion sort’s running time is $Θ (n^{2})$ , we abstracted away some details of this function. Because asymptotic notation applies to functions, what we were writing as $Θ (n^{2})$ was the function $a n^{2} + bn + c$ , which in that case happened to characterize the worst-case running time of insertion sort.

In this book, the functions to which we apply asymptotic notation will usually characterize the running times of algorithms. But asymptotic notation can apply to functions that characterize some other aspect of algorithms (the amount of space they use, for example), or even to functions that have nothing whatsoever to do with algorithms.

Even when we use asymptotic notation to apply to the running time of an algorithm, we need to understand which running time we mean. Sometimes we are interested in the worst-case running time. Often, however, we wish to characterize the running time no matter what the input. In other words, we often wish to make a blanket statement that covers all inputs, not just the worst case. We shall see asymptotic notations that are well suited to characterizing running times no matter what the input.

$Θ$ -notation

In Chapter 2, we found that the worst-case running time of insertion sort is $T (n) = Θ (n^{2})$ . Let us define what this notation means. For a given function $g (n)$ , we denote by $Θ (g (n))$ the set of functions

Θ (g (n)) = {f (n) : there exist positive constants c_{1}, c_{2}, and n_{0} such that 0 \leq c_{1} g (n) \leq f (n) \leq c_{2} g (n) for all n \geq n_{0}} .^{1}

¹ Within set notation, a colon means “such that.”

Figure 3.1: Graphic examples of the $Θ$ , $O$ , and $Ω$ notations. In each part, the value of $n_{0}$ shown is the minimum possible value; any greater value would also work.

(a) $f (n) = Θ (g (n))$ : $Θ$ -notation bounds a function to within constant factors. We write $f (n) = Θ (g (n))$ if there exist positive constants $n_{0}$ , $c_{1}$ , and $c_{2}$ such that at and to the right of $n_{0}$ , the value of $f (n)$ always lies between $c_{1} g (n)$ and $c_{2} g (n)$ inclusive.
(b) $f (n) = O (g (n))$ : $O$ -notation gives an upper bound for a function to within a constant factor. We write $f (n) = O (g (n))$ if there are positive constants $n_{0}$ and $c$ such that at and to the right of $n_{0}$ , the value of $f (n)$ always lies on or below $c g (n)$ .
(c) $f (n) = Ω (g (n))$ : $Ω$ -notation gives a lower bound for a function to within a constant factor. We write $f (n) = Ω (g (n))$ if there are positive constants $n_{0}$ and $c$ such that at and to the right of $n_{0}$ , the value of $f (n)$ always lies on or above $c g (n)$ .

A function $f (n)$ belongs to the set $Θ (g (n))$ if there exist positive constants $c_{1}$ and $c_{2}$ such that it can be “sandwiched” between $c_{1} g (n)$ and $c_{2} g (n)$ , for sufficiently large $n$ . Because $Θ (g (n))$ is a set, we could write " $f (n) \in Θ (g (n))$ " to indicate that $f (n)$ is a member of $Θ (g (n))$ . Instead, we will usually write " $f (n) = Θ (g (n))$ " to express the same notion. You might be confused because we abuse equality in this way, but we shall see later in this section that doing so has its advantages.

Figure 3.1(a) gives an intuitive picture of functions $f (n)$ and $g (n)$ , where $f (n) = Θ (g (n))$ . For all values of $n$ at and to the right of $n_{0}$ , the value of $f (n)$ lies at or above $c_{1} g (n)$ and at or below $c_{2} g (n)$ . In other words, for all $n \geq n_{0}$ , the function $f (n)$ is equal to $g (n)$ to within a constant factor. We say that $g (n)$ is an asymptotically tight bound for $f (n)$ .

The definition of $Θ (g (n))$ requires that every member $f (n) \in Θ (g (n))$ be asymptotically nonnegative, that is, that $f (n)$ be nonnegative whenever $n$ is sufficiently large. (An asymptotically positive function is one that is positive for all sufficiently large $n$ .) Consequently, the function $g (n)$ itself must be asymptotically nonnegative, or else the set $Θ (g (n))$ is empty. We shall therefore assume that every function used within $Θ$ -notation is asymptotically nonnegative. This assumption holds for the other asymptotic notations defined in this chapter as well.

In Chapter 2, we introduced an informal notion of $Θ$ -notation that amounted to throwing away lower-order terms and ignoring the leading coefficient of the highest-order term. Let us briefly justify this intuition by using the formal definition to show that $\frac{1}{2} n^{2} - 3 n = Θ (n^{2})$ .

Example. Show that $\frac{1}{2} n^{2} - 3 n = Θ (n^{2})$ .

We must determine positive constants $c_{1}$ , $c_{2}$ , and $n_{0}$ such that

$c_{1} n^{2} \leq \frac{1}{2} n^{2} - 3 n \leq c_{2} n^{2}$

for all $n \geq n_{0}$ . Dividing by $n^{2}$ (which is positive) yields

$c_{1} \leq \frac{1}{2} - \frac{3}{n} \leq c_{2} .$

Right-hand inequality. Since $\frac{3}{n} > 0$ , we have $\frac{1}{2} - \frac{3}{n} < \frac{1}{2}$ for every $n \geq 1$ . So the right-hand inequality holds for any $n \geq 1$ by choosing any constant $c_{2} \geq 1/2$ .

Left-hand inequality. The quantity $\frac{1}{2} - \frac{3}{n}$ is increasing in $n$ , and at $n = 7$ it equals $\frac{1}{2} - \frac{3}{7} = \frac{7 - 6}{14} = \frac{1}{14}$ . Hence for every $n \geq 7$ we have $\frac{1}{2} - \frac{3}{n} \geq \frac{1}{14}$ , so the left-hand inequality holds for any $n \geq 7$ by choosing any constant $c_{1} \leq 1/14$ .

Thus, by choosing $c_{1} = 1/14$ , $c_{2} = 1/2$ , and $n_{0} = 7$ , we can verify that $\frac{1}{2} n^{2} - 3 n = Θ (n^{2})$ . Certainly, other choices for the constants exist, but the important thing is that some choice exists. Note that the constants depend on the function $\frac{1}{2} n^{2} - 3 n$ ; a different function belonging to $Θ (n^{2})$ would usually require different constants.

We can also use the formal definition to verify that $6 n^{3} \neq = Θ (n^{2})$ .

Example. Show that $6 n^{3} \neq = Θ (n^{2})$ .

Suppose, for the purpose of contradiction, that $c_{2}$ and $n_{0}$ exist such that $6 n^{3} \leq c_{2} n^{2}$ for all $n \geq n_{0}$ . Dividing by $n^{2}$ yields

$n \leq \frac{c _{2}}{6},$

which cannot possibly hold for arbitrarily large $n$ , since $c_{2}$ is constant. This is a contradiction.

Intuitively, the lower-order terms of an asymptotically positive function can be ignored in determining asymptotically tight bounds because they are insignificant for large $n$ . When $n$ is large, even a tiny fraction of the highest-order term suffices to dominate the lower-order terms. Thus, setting $c_{1}$ to a value that is slightly smaller than the coefficient of the highest-order term and setting $c_{2}$ to a value that is slightly larger permits the inequalities in the definition of $Θ$ -notation to be satisfied. The coefficient of the highest-order term can likewise be ignored, since it only changes $c_{1}$ and $c_{2}$ by a constant factor equal to the coefficient.

As an example, consider any quadratic function $f (n) = a n^{2} + bn + c$ , where $a$ , $b$ , and $c$ are constants and $a > 0$ . Throwing away the lower-order terms and ignoring the constant yields $f (n) = Θ (n^{2})$ . Formally, to show the same thing, we take the constants

$c_{1} = a /4, c_{2} = 7 a /4, n_{0} = 2 \cdot max (∣ b ∣/ a, ∣ c ∣/ a) .$

Verification. Let $n \geq n_{0}$ . Then $n \geq 2∣ b ∣/ a$ and $n \geq 2 ∣ c ∣/ a$ , so $∣ b ∣ \leq an /2$ and $∣ c ∣ \leq a n^{2} /4$ . Therefore
$a n^{2} + bn + c \geq a n^{2} - ∣ b ∣ n - ∣ c ∣ \geq a n^{2} - \frac{an}{2} \cdot n - \frac{a n ^{2}}{4} = \frac{a}{4} n^{2} = c_{1} n^{2},$
and
$a n^{2} + bn + c \leq a n^{2} + ∣ b ∣ n + ∣ c ∣ \leq a n^{2} + \frac{a n ^{2}}{2} + \frac{a n ^{2}}{4} = \frac{7 a}{4} n^{2} = c_{2} n^{2} .$
Hence $0 \leq c_{1} n^{2} \leq a n^{2} + bn + c \leq c_{2} n^{2}$ for all $n \geq n_{0}$ .

In general, for any polynomial $p (n) = \sum_{i = 0}^{d} a_{i} n^{i}$ , where the $a_{i}$ are constants and $a_{d} > 0$ , we have $p (n) = Θ (n^{d})$ (see Problem 3-1).

Since any constant is a degree-0 polynomial, we can express any constant function as $Θ (n^{0})$ , or $Θ (1)$ . This latter notation is a minor abuse, however, because the expression does not indicate what variable is tending to infinity.² We shall often use the notation $Θ (1)$ to mean either a constant or a constant function with respect to some variable.

² The real problem is that our ordinary notation for functions does not distinguish functions from values. In $λ$ -calculus, the parameters to a function are clearly specified: the function $n^{2}$ could be written as $λn . n^{2}$ , or even $λ r . r^{2}$ . Adopting a more rigorous notation, however, would complicate algebraic manipulations, and so we choose to tolerate the abuse.

$O$ -notation

The $Θ$ -notation asymptotically bounds a function from above and below. When we have only an asymptotic upper bound, we use $O$ -notation. For a given function $g (n)$ , we denote by $O (g (n))$ (pronounced “big-oh of $g$ of $n$ ” or sometimes just “oh of $g$ of $n$ ”) the set of functions

O (g (n)) = {f (n) : there exist positive constants c and n_{0} such that 0 \leq f (n) \leq c g (n) for all n \geq n_{0}} .

We use $O$ -notation to give an upper bound on a function, to within a constant factor. Figure 3.1(b) shows the intuition behind $O$ -notation. For all values $n$ at and to the right of $n_{0}$ , the value of the function $f (n)$ is on or below $c g (n)$ .

We write $f (n) = O (g (n))$ to indicate that a function $f (n)$ is a member of the set $O (g (n))$ . Note that $f (n) = Θ (g (n))$ implies $f (n) = O (g (n))$ , since $Θ$ -notation is a stronger notion than $O$ -notation. Written set-theoretically, we have $Θ (g (n)) \subseteq O (g (n))$ . Thus, our proof that any quadratic function $a n^{2} + bn + c$ , where $a > 0$ , is in $Θ (n^{2})$ also shows that any such quadratic function is in $O (n^{2})$ . What may be more surprising is that when $a > 0$ , any linear function $an + b$ is in $O (n^{2})$ , which is easily verified by taking $c = a + ∣ b ∣$ and $n_{0} = max (1, - b / a)$ .

Check. For $n \geq n_{0}$ we have $n \geq 1$ , so $n \leq n^{2}$ and $∣ b ∣ \leq ∣ b ∣ n^{2}$ . Also $n \geq - b / a$ gives $an + b \geq 0$ . Hence

$0 \leq an + b \leq an + ∣ b ∣ \leq a n^{2} + ∣ b ∣ n^{2} = (a + ∣ b ∣) n^{2} = c n^{2} .$

If you have seen $O$ -notation before, you might find it strange that we should write, for example, $n = O (n^{2})$ . In the literature, we sometimes find $O$ -notation informally describing asymptotically tight bounds, that is, what we have defined using $Θ$ -notation. In this book, however, when we write $f (n) = O (g (n))$ , we are merely claiming that some constant multiple of $g (n)$ is an asymptotic upper bound on $f (n)$ , with no claim about how tight an upper bound it is. Distinguishing asymptotic upper bounds from asymptotically tight bounds is standard in the algorithms literature.

Using $O$ -notation, we can often describe the running time of an algorithm merely by inspecting the algorithm’s overall structure. For example, the doubly nested loop structure of the insertion sort algorithm from Chapter 2 immediately yields an $O (n^{2})$ upper bound on the worst-case running time: the cost of each iteration of the inner loop is bounded from above by $O (1)$ (constant), the indices $i$ and $j$ are both at most $n$ , and the inner loop is executed at most once for each of the $n^{2}$ pairs of values for $i$ and $j$ .

Since $O$ -notation describes an upper bound, when we use it to bound the worst-case running time of an algorithm, we have a bound on the running time of the algorithm on every input, the blanket statement we discussed earlier. Thus, the $O (n^{2})$ bound on worst-case running time of insertion sort also applies to its running time on every input. The $Θ (n^{2})$ bound on the worst-case running time of insertion sort, however, does not imply a $Θ (n^{2})$ bound on the running time of insertion sort on every input. For example, we saw in Chapter 2 that when the input is already sorted, insertion sort runs in $Θ (n)$ time.

Technically, it is an abuse to say that the running time of insertion sort is $O (n^{2})$ , since for a given $n$ , the actual running time varies, depending on the particular input of size $n$ . When we say “the running time is $O (n^{2})$ ,” we mean that there is a function $f (n)$ that is $O (n^{2})$ such that for any value of $n$ , no matter what particular input of size $n$ is chosen, the running time on that input is bounded from above by the value $f (n)$ . Equivalently, we mean that the worst-case running time is $O (n^{2})$ .

Think about it. $O$ is a “no more than” promise: saying “I’m at most 10 km from home” is true even if you are standing in your kitchen. That is why $n = O (n^{2})$ is perfectly legal, just not very informative.

$Ω$ -notation

Just as $O$ -notation provides an asymptotic upper bound on a function, $Ω$ -notation provides an asymptotic lower bound. For a given function $g (n)$ , we denote by $Ω (g (n))$ (pronounced “big-omega of $g$ of $n$ ” or sometimes just “omega of $g$ of $n$ ”) the set of functions

Ω (g (n)) = {f (n) : there exist positive constants c and n_{0} such that 0 \leq c g (n) \leq f (n) for all n \geq n_{0}} .

Figure 3.1(c) shows the intuition behind $Ω$ -notation. For all values $n$ at or to the right of $n_{0}$ , the value of $f (n)$ is on or above $c g (n)$ .

From the definitions of the asymptotic notations we have seen thus far, it is easy to prove the following important theorem (see Exercise 3.1-5).

Theorem 3.1 For any two functions $f (n)$ and $g (n)$ , we have $f (n) = Θ (g (n))$ if and only if $f (n) = O (g (n))$ and $f (n) = Ω (g (n))$ .

As an example of the application of this theorem, our proof that $a n^{2} + bn + c = Θ (n^{2})$ for any constants $a$ , $b$ , and $c$ , where $a > 0$ , immediately implies that $a n^{2} + bn + c = Ω (n^{2})$ and $a n^{2} + bn + c = O (n^{2})$ . In practice, rather than using Theorem 3.1 to obtain asymptotic upper and lower bounds from asymptotically tight bounds, as we did for this example, we usually use it to prove asymptotically tight bounds from asymptotic upper and lower bounds.

When we say that the running time (no modifier) of an algorithm is $Ω (g (n))$ , we mean that no matter what particular input of size $n$ is chosen for each value of $n$ , the running time on that input is at least a constant times $g (n)$ , for sufficiently large $n$ . Equivalently, we are giving a lower bound on the best-case running time of an algorithm. For example, the best-case running time of insertion sort is $Ω (n)$ , which implies that the running time of insertion sort is $Ω (n)$ .

The running time of insertion sort therefore belongs to both $Ω (n)$ and $O (n^{2})$ , since it falls anywhere between a linear function of $n$ and a quadratic function of $n$ . Moreover, these bounds are asymptotically as tight as possible: for instance, the running time of insertion sort is not $Ω (n^{2})$ , since there exists an input for which insertion sort runs in $Θ (n)$ time (e.g., when the input is already sorted). It is not contradictory, however, to say that the worst-case running time of insertion sort is $Ω (n^{2})$ , since there exists an input that causes the algorithm to take $Ω (n^{2})$ time.

Asymptotic notation in equations and inequalities

We have already seen how asymptotic notation can be used within mathematical formulas. For example, in introducing $O$ -notation, we wrote ” $n = O (n^{2})$ .” We might also write $2 n^{2} + 3 n + 1 = 2 n^{2} + Θ (n)$ . How do we interpret such formulas?

When the asymptotic notation stands alone (that is, not within a larger formula) on the right-hand side of an equation (or inequality), as in $n = O (n^{2})$ , we have already defined the equal sign to mean set membership: $n \in O (n^{2})$ . In general, however, when asymptotic notation appears in a formula, we interpret it as standing for some anonymous function that we do not care to name. For example, the formula $2 n^{2} + 3 n + 1 = 2 n^{2} + Θ (n)$ means that $2 n^{2} + 3 n + 1 = 2 n^{2} + f (n)$ , where $f (n)$ is some function in the set $Θ (n)$ . In this case, we let $f (n) = 3 n + 1$ , which indeed is in $Θ (n)$ .

Using asymptotic notation in this manner can help eliminate inessential detail and clutter in an equation. For example, in Chapter 2 we expressed the worst-case running time of merge sort as the recurrence

$T (n) = 2 T (n /2) + Θ (n) .$

If we are interested only in the asymptotic behavior of $T (n)$ , there is no point in specifying all the lower-order terms exactly; they are all understood to be included in the anonymous function denoted by the term $Θ (n)$ .

The number of anonymous functions in an expression is understood to be equal to the number of times the asymptotic notation appears. For example, in the expression

$\sum_{i = 1}^{n} O (i),$

there is only a single anonymous function (a function of $i$ ). This expression is thus not the same as $O (1) + O (2) + \dots + O (n)$ , which doesn’t really have a clean interpretation.

In some cases, asymptotic notation appears on the left-hand side of an equation, as in

$2 n^{2} + Θ (n) = Θ (n^{2}) .$

We interpret such equations using the following rule: No matter how the anonymous functions are chosen on the left of the equal sign, there is a way to choose the anonymous functions on the right of the equal sign to make the equation valid. Thus, our example means that for any function $f (n) \in Θ (n)$ , there is some function $g (n) \in Θ (n^{2})$ such that $2 n^{2} + f (n) = g (n)$ for all $n$ . In other words, the right-hand side of an equation provides a coarser level of detail than the left-hand side.

We can chain together a number of such relationships, as in

2 n^{2} + 3 n + 1 = 2 n^{2} + Θ (n) = Θ (n^{2}) .

We can interpret each equation separately by the rules above. The first equation says that there is some function $f (n) \in Θ (n)$ such that $2 n^{2} + 3 n + 1 = 2 n^{2} + f (n)$ for all $n$ . The second equation says that for any function $g (n) \in Θ (n)$ (such as the $f (n)$ just mentioned), there is some function $h (n) \in Θ (n^{2})$ such that $2 n^{2} + g (n) = h (n)$ for all $n$ . Note that this interpretation implies that $2 n^{2} + 3 n + 1 = Θ (n^{2})$ , which is what the chaining of equations intuitively gives us.

$o$ -notation

The asymptotic upper bound provided by $O$ -notation may or may not be asymptotically tight. The bound $2 n^{2} = O (n^{2})$ is asymptotically tight, but the bound $2 n = O (n^{2})$ is not. We use $o$ -notation to denote an upper bound that is not asymptotically tight. We formally define $o (g (n))$ (“little-oh of $g$ of $n$ ”) as the set

o (g (n)) = {f (n) : for any positive constant c > 0, there exists a constant n_{0} > 0 such that 0 \leq f (n) < c g (n) for all n \geq n_{0}} .

For example, $2 n = o (n^{2})$ , but $2 n^{2} \neq = o (n^{2})$ .

Example. Show $2 n = o (n^{2})$ and $2 n^{2} \neq = o (n^{2})$ .

First claim. Given any $c > 0$ , we need an $n_{0}$ with $2 n < c n^{2}$ for all $n \geq n_{0}$ . Dividing by $n > 0$ , this is $2 < c n$ , i.e. $n > 2/ c$ . So take $n_{0} = ⌈ 2/ c ⌉ + 1$ ; then every $n \geq n_{0}$ satisfies $n > 2/ c$ , hence $2 n < c n^{2}$ .

Second claim. Take $c = 2$ . The requirement would be $2 n^{2} < 2 n^{2}$ for all large $n$ , which is false. So not every $c > 0$ works, and $2 n^{2} \neq = o (n^{2})$ .

The definitions of $O$ -notation and $o$ -notation are similar. The main difference is that in $f (n) = O (g (n))$ , the bound $0 \leq f (n) \leq c g (n)$ holds for some constant $c > 0$ , but in $f (n) = o (g (n))$ , the bound $0 \leq f (n) < c g (n)$ holds for all constants $c > 0$ . Intuitively, in $o$ -notation, the function $f (n)$ becomes insignificant relative to $g (n)$ as $n$ approaches infinity; that is,

$\lim_{n \to \infty} \frac{f(n)}{g(n)} = 0. \tag{3.1}$

Some authors use this limit as a definition of the $o$ -notation; the definition in this book also restricts the anonymous functions to be asymptotically nonnegative.

$ω$ -notation

By analogy, $ω$ -notation is to $Ω$ -notation as $o$ -notation is to $O$ -notation. We use $ω$ -notation to denote a lower bound that is not asymptotically tight. One way to define it is by

$f (n) \in ω (g (n)) if and only if g (n) \in o (f (n)) .$

Formally, however, we define $ω (g (n))$ (“little-omega of $g$ of $n$ ”) as the set

ω (g (n)) = {f (n) : for any positive constant c > 0, there exists a constant n_{0} > 0 such that 0 \leq c g (n) < f (n) for all n \geq n_{0}} .

For example, $n^{2} /2 = ω (n)$ , but $n^{2} /2 \neq = ω (n^{2})$ . The relation $f (n) = ω (g (n))$ implies that

$lim_{n \to \infty} \frac{f ( n )}{g ( n )} = \infty,$

if the limit exists. That is, $f (n)$ becomes arbitrarily large relative to $g (n)$ as $n$ approaches infinity.

Comparing functions

Many of the relational properties of real numbers apply to asymptotic comparisons as well. For the following, assume that $f (n)$ and $g (n)$ are asymptotically positive.

Transitivity:

f (n) = Θ (g (n)) and g (n) = Θ (h (n)) f (n) = O (g (n)) and g (n) = O (h (n)) f (n) = Ω (g (n)) and g (n) = Ω (h (n)) f (n) = o (g (n)) and g (n) = o (h (n)) f (n) = ω (g (n)) and g (n) = ω (h (n)) imply f (n) = Θ (h (n)), imply f (n) = O (h (n)), imply f (n) = Ω (h (n)), imply f (n) = o (h (n)), imply f (n) = ω (h (n)) .

Reflexivity:

f (n) f (n) f (n) = Θ (f (n)), = O (f (n)), = Ω (f (n)) .

Symmetry:

$f (n) = Θ (g (n)) if and only if g (n) = Θ (f (n)) .$

Transpose symmetry:

f (n) = O (g (n)) f (n) = o (g (n)) if and only if g (n) = Ω (f (n)), if and only if g (n) = ω (f (n)) .

Because these properties hold for asymptotic notations, we can draw an analogy between the asymptotic comparison of two functions $f$ and $g$ and the comparison of two real numbers $a$ and $b$ :

Asymptotic statement	Analogy
$f (n) = O (g (n))$	is like $a \leq b$
$f (n) = Ω (g (n))$	is like $a \geq b$
$f (n) = Θ (g (n))$	is like $a = b$
$f (n) = o (g (n))$	is like $a < b$
$f (n) = ω (g (n))$	is like $a > b$

We say that $f (n)$ is asymptotically smaller than $g (n)$ if $f (n) = o (g (n))$ , and $f (n)$ is asymptotically larger than $g (n)$ if $f (n) = ω (g (n))$ .

One property of real numbers, however, does not carry over to asymptotic notation:

Trichotomy: For any two real numbers $a$ and $b$ , exactly one of the following must hold: $a < b$ , $a = b$ , or $a > b$ .

Although any two real numbers can be compared, not all functions are asymptotically comparable. That is, for two functions $f (n)$ and $g (n)$ , it may be the case that neither $f (n) = O (g (n))$ nor $f (n) = Ω (g (n))$ holds. For example, we cannot compare the functions $n$ and $n^{1 + s i n n}$ using asymptotic notation, since the value of the exponent in $n^{1 + s i n n}$ oscillates between $0$ and $2$ , taking on all values in between.

Think about it. Two runners who keep overtaking each other forever, with no one ever staying ahead, cannot be ranked in the marathon. That is exactly what $n$ versus $n^{1 + s i n n}$ looks like.

Exercises 3.1-1 to 3.1-8

3.1-1 Let $f (n)$ and $g (n)$ be asymptotically nonnegative functions. Using the basic definition of $Θ$ -notation, prove that $max (f (n), g (n)) = Θ (f (n) + g (n))$ .

3.1-2 Show that for any real constants $a$ and $b$ , where $b > 0$ , $(n + a)^b = \Theta(n^b). \tag{3.2}$

3.1-3 Explain why the statement, “The running time of algorithm $A$ is at least $O (n^{2})$ ,” is meaningless.

3.1-4 Is $2^{n + 1} = O (2^{n})$ ? Is $2^{2 n} = O (2^{n})$ ?

3.1-5 Prove Theorem 3.1.

3.1-6 Prove that the running time of an algorithm is $Θ (g (n))$ if and only if its worst-case running time is $O (g (n))$ and its best-case running time is $Ω (g (n))$ .

3.1-7 Prove that $o (g (n)) \cap ω (g (n))$ is the empty set.

3.1-8 We can extend our notation to the case of two parameters $n$ and $m$ that can go to infinity independently at different rates. For a given function $g (n, m)$ , we denote by $O (g (n, m))$ the set of functions

O (g (n, m)) = {f (n, m) : there exist positive constants c, n_{0}, and m_{0} such that 0 \leq f (n, m) \leq c g (n, m) for all n \geq n_{0} or m \geq m_{0}} .

Give corresponding definitions for $Ω (g (n, m))$ and $Θ (g (n, m))$ .

3.2 Standard notations and common functions

This section reviews some standard mathematical functions and notations and explores the relationships among them. It also illustrates the use of the asymptotic notations.

Monotonicity

A function $f (n)$ is monotonically increasing if $m \leq n$ implies $f (m) \leq f (n)$ . Similarly, it is monotonically decreasing if $m \leq n$ implies $f (m) \geq f (n)$ . A function $f (n)$ is strictly increasing if $m < n$ implies $f (m) < f (n)$ and strictly decreasing if $m < n$ implies $f (m) > f (n)$ .

Floors and ceilings

For any real number $x$ , we denote the greatest integer less than or equal to $x$ by $⌊ x ⌋$ (read “the floor of $x$ ”) and the least integer greater than or equal to $x$ by $⌈ x ⌉$ (read “the ceiling of $x$ ”). For all real $x$ ,

$x - 1 < \lfloor x \rfloor \le x \le \lceil x \rceil < x + 1. \tag{3.3}$

For any integer $n$ ,

$⌈ n /2 ⌉ + ⌊ n /2 ⌋ = n,$

and for any real number $x \geq 0$ and integers $a, b > 0$ ,

$\left\lceil \frac{\lceil x/a \rceil}{b} \right\rceil = \left\lceil \frac{x}{ab} \right\rceil, \tag{3.4}$

$\left\lfloor \frac{\lfloor x/a \rfloor}{b} \right\rfloor = \left\lfloor \frac{x}{ab} \right\rfloor, \tag{3.5}$

$\left\lceil \frac{a}{b} \right\rceil \le \frac{a + (b-1)}{b}, \tag{3.6}$

$\left\lfloor \frac{a}{b} \right\rfloor \ge \frac{a - (b-1)}{b}. \tag{3.7}$

The floor function $f (x) = ⌊ x ⌋$ is monotonically increasing, as is the ceiling function $f (x) = ⌈ x ⌉$ .

Modular arithmetic

For any integer $a$ and any positive integer $n$ , the value $a mod n$ is the remainder (or residue) of the quotient $a / n$ :

$a \bmod n = a - n \lfloor a/n \rfloor. \tag{3.8}$

It follows that

$0 \le a \bmod n < n. \tag{3.9}$

Given a well-defined notion of the remainder of one integer when divided by another, it is convenient to provide special notation to indicate equality of remainders. If $(a mod n) = (b mod n)$ , we write $a \equiv b (mod n)$ and say that $a$ is equivalent to $b$ , modulo $n$ . In other words, $a \equiv b (mod n)$ if $a$ and $b$ have the same remainder when divided by $n$ . Equivalently, $a \equiv b (mod n)$ if and only if $n$ is a divisor of $b - a$ . We write $a \neq \equiv b (mod n)$ if $a$ is not equivalent to $b$ , modulo $n$ .

Polynomials

Given a nonnegative integer $d$ , a polynomial in $n$ of degree $d$ is a function $p (n)$ of the form

$p (n) = \sum_{i = 0}^{d} a_{i} n^{i},$

where the constants $a_{0}, a_{1}, \dots, a_{d}$ are the coefficients of the polynomial and $a_{d} \neq = 0$ . A polynomial is asymptotically positive if and only if $a_{d} > 0$ . For an asymptotically positive polynomial $p (n)$ of degree $d$ , we have $p (n) = Θ (n^{d})$ . For any real constant $a \geq 0$ , the function $n^{a}$ is monotonically increasing, and for any real constant $a \leq 0$ , the function $n^{a}$ is monotonically decreasing. We say that a function $f (n)$ is polynomially bounded if $f (n) = O (n^{k})$ for some constant $k$ .

Exponentials

For all real $a > 0$ , $m$ , and $n$ , we have the following identities:

a^{0} a^{1} a^{- 1} (a^{m})^{n} (a^{m})^{n} a^{m} a^{n} = 1, = a, = 1/ a, = a^{mn}, = (a^{n})^{m}, = a^{m + n} .

For all $n$ and $a \geq 1$ , the function $a^{n}$ is monotonically increasing in $n$ . When convenient, we shall assume $0^{0} = 1$ .

We can relate the rates of growth of polynomials and exponentials by the following fact. For all real constants $a$ and $b$ such that $a > 1$ ,

$\lim_{n \to \infty} \frac{n^b}{a^n} = 0, \tag{3.10}$

from which we can conclude that

$n^{b} = o (a^{n}) .$

Thus, any exponential function with a base strictly greater than 1 grows faster than any polynomial function.

Using $e$ to denote $2.71828 \dots$ , the base of the natural logarithm function, we have for all real $x$ ,

$e^x = 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \cdots = \sum_{i=0}^{\infty} \frac{x^i}{i!}, \tag{3.11}$

where " $!$ " denotes the factorial function defined later in this section. For all real $x$ , we have the inequality

$e^x \ge 1 + x, \tag{3.12}$

where equality holds only when $x = 0$ . When $∣ x ∣ \leq 1$ , we have the approximation

$1 + x \le e^x \le 1 + x + x^2. \tag{3.13}$

When $x \to 0$ , the approximation of $e^{x}$ by $1 + x$ is quite good:

$e^{x} = 1 + x + Θ (x^{2}) .$

(In this equation, the asymptotic notation is used to describe the limiting behavior as $x \to 0$ rather than as $x \to \infty$ .) We have for all $x$ ,

$\lim_{n \to \infty} \left(1 + \frac{x}{n}\right)^n = e^x. \tag{3.14}$

Think about it. Doubling a grain of rice on each square of a chessboard starts out trivial and ends up buried under a mountain. Polynomials are the patient hikers; exponentials are the avalanche.

Logarithms

We shall use the following notations:

l g n ln n l g^{k} n l g l g n = lo g_{2} n = lo g_{e} n = (l g n)^{k} = l g (l g n) (binary logarithm), (natural logarithm), (exponentiation), (composition) .

An important notational convention we shall adopt is that logarithm functions will apply only to the next term in the formula, so that $l g n + k$ will mean $(l g n) + k$ and not $l g (n + k)$ . If we hold $b > 1$ constant, then for $n > 0$ , the function $lo g_{b} n$ is strictly increasing.

For all real $a > 0$ , $b > 0$ , $c > 0$ , and $n$ ,

a lo g_{c} (ab) lo g_{b} a^{n} lo g_{b} a lo g_{b} (1/ a) lo g_{b} a a^{l o g_{b} c} = b^{l o g_{b} a}, = lo g_{c} a + lo g_{c} b, = n lo g_{b} a, = \frac{lo g _{c} a}{lo g _{c} b}, = - lo g_{b} a, = \frac{1}{lo g _{a} b}, = c^{l o g_{b} a}, (3.15) (3.16)

where, in each equation above, logarithm bases are not 1.

By equation (3.15), changing the base of a logarithm from one constant to another changes the value of the logarithm by only a constant factor, and so we shall often use the notation " $l g n$ " when we don’t care about constant factors, such as in $O$ -notation. Computer scientists find 2 to be the most natural base for logarithms because so many algorithms and data structures involve splitting a problem into two parts.

There is a simple series expansion for $ln (1 + x)$ when $∣ x ∣ < 1$ :

$ln (1 + x) = x - \frac{x ^{2}}{2} + \frac{x ^{3}}{3} - \frac{x ^{4}}{4} + \frac{x ^{5}}{5} - \dots .$

We also have the following inequalities for $x > - 1$ :

$\frac{x}{1+x} \le \ln(1+x) \le x, \tag{3.17}$

where equality holds only for $x = 0$ .

We say that a function $f (n)$ is polylogarithmically bounded if $f (n) = O (l g^{k} n)$ for some constant $k$ . We can relate the growth of polynomials and polylogarithms by substituting $l g n$ for $n$ and $2^{a}$ for $a$ in equation (3.10), yielding

$lim_{n \to \infty} \frac{l g ^{b} n}{( 2 ^{a} ) ^{l g n}} = lim_{n \to \infty} \frac{l g ^{b} n}{n ^{a}} = 0.$

Why the substitution works. Equation (3.10) holds for every base greater than 1. Put $n \to l g n$ and base $2^{a}$ (which exceeds 1 when $a > 0$ ). Then $(2^{a})^{l g n} = 2^{a l g n} = (2^{l g n})^{a} = n^{a}$ , which gives the second form of the limit.

From this limit, we can conclude that

$l g^{b} n = o (n^{a})$

for any constant $a > 0$ . Thus, any positive polynomial function grows faster than any polylogarithmic function.

Factorials

The notation $n!$ (read ” $n$ factorial”) is defined for integers $n \geq 0$ as

n! = {1 n \cdot (n - 1)! if n = 0, if n > 0.

Thus, $n! = 1 \cdot 2 \cdot 3 \dots n$ .

A weak upper bound on the factorial function is $n! \leq n^{n}$ , since each of the $n$ terms in the factorial product is at most $n$ . Stirling’s approximation,

$n! = \sqrt{2\pi n}\left(\frac{n}{e}\right)^n \left(1 + \Theta\!\left(\frac{1}{n}\right)\right), \tag{3.18}$

where $e$ is the base of the natural logarithm, gives us a tighter upper bound, and a lower bound as well. As Exercise 3.2-3 asks you to prove,

n! n! l g (n!) = o (n^{n}), = ω (2^{n}), = Θ (n l g n), (3.19)

where Stirling’s approximation is helpful in proving equation (3.19). The following equation also holds for all $n \geq 1$ :

$n! = \sqrt{2\pi n}\left(\frac{n}{e}\right)^n e^{\alpha_n}, \tag{3.20}$

where

$\frac{1}{12n + 1} < \alpha_n < \frac{1}{12n}. \tag{3.21}$

Functional iteration

We use the notation $f^{(i)} (n)$ to denote the function $f (n)$ iteratively applied $i$ times to an initial value of $n$ . Formally, let $f (n)$ be a function over the reals. For nonnegative integers $i$ , we recursively define

f^{(i)} (n) = {n f (f^{(i - 1)} (n)) if i = 0, if i > 0.

For example, if $f (n) = 2 n$ , then $f^{(i)} (n) = 2^{i} n$ .

Example. Check $f^{(i)} (n) = 2^{i} n$ for $f (n) = 2 n$ by induction on $i$ .

Base case $i = 0$ : $f^{(0)} (n) = n = 2^{0} n$ .

Inductive step. Assume $f^{(i - 1)} (n) = 2^{i - 1} n$ . Then
$f^{(i)} (n) = f (f^{(i - 1)} (n)) = f (2^{i - 1} n) = 2 \cdot 2^{i - 1} n = 2^{i} n . (definition, i > 0) (induction hypothesis) (definition of f)$

The iterated logarithm function

We use the notation $l g^{*} n$ (read “log star of $n$ ”) to denote the iterated logarithm, defined as follows. Let $l g^{(i)} n$ be as defined above, with $f (n) = l g n$ . Because the logarithm of a nonpositive number is undefined, $l g^{(i)} n$ is defined only if $l g^{(i - 1)} n > 0$ . Be sure to distinguish $l g^{(i)} n$ (the logarithm function applied $i$ times in succession, starting with argument $n$ ) from $l g^{i} n$ (the logarithm of $n$ raised to the $i$ th power). Then we define the iterated logarithm function as

$l g^{*} n = min {i \geq 0 : l g^{(i)} n \leq 1} .$

The iterated logarithm is a very slowly growing function:

l g^{*} 2 l g^{*} 4 l g^{*} 16 l g^{*} 65536 l g^{*} (2^{65536}) = 1, = 2, = 3, = 4, = 5.

Example. Verify $l g^{*} (2^{65536}) = 5$ by iterating the logarithm.
$l g^{(0)} (2^{65536}) l g^{(1)} l g^{(2)} l g^{(3)} l g^{(4)} l g^{(5)} = 2^{65536} > 1, = 65536 > 1, = 16 > 1, = 4 > 1, = 2 > 1, = 1 \leq 1.$
The first $i$ with $l g^{(i)} \leq 1$ is $i = 5$ , so $l g^{*} (2^{65536}) = 5$ .

Since the number of atoms in the observable universe is estimated to be about $1 0^{80}$ , which is much less than $2^{65536}$ , we rarely encounter an input size $n$ such that $l g^{*} n > 5$ .

Think about it. How many times must you press the “log” button on a calculator, starting from a number with more digits than there are atoms in the universe, before the display shows 1 or less? At most five. This is why $l g^{*}$ is, for practical purposes, constant.

Fibonacci numbers

We define the Fibonacci numbers by the following recurrence:

F_{0} F_{1} F_{i} = 0, = 1, = F_{i - 1} + F_{i - 2} for i \geq 2. (3.22)

Thus, each Fibonacci number is the sum of the two previous ones, yielding the sequence

$0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, \dots .$

Fibonacci numbers are related to the golden ratio $ϕ$ and to its conjugate $\hat{ϕ}$ , which are the two roots of the equation

$x^2 = x + 1 \tag{3.23}$

and are given by the following formulas (see Exercise 3.2-6):

ϕ \hat{ϕ} = \frac{1 + 5}{2} = 1.61803 \dots, = \frac{1 - 5}{2} = - 0.61803 \dots . (3.24)

Specifically, we have

$F_{i} = \frac{ϕ ^{i} - ϕ ^ ^{i}}{5},$

which we can prove by induction (Exercise 3.2-7). Since $∣ \hat{ϕ} ∣ < 1$ , we have

$\frac{∣ ϕ ^ ^{i} ∣}{5} < \frac{1}{5} < \frac{1}{2},$

which implies that

$F_i = \left\lfloor \frac{\phi^i}{\sqrt{5}} + \frac{1}{2} \right\rfloor, \tag{3.25}$

which is to say that the $i$ th Fibonacci number $F_{i}$ is equal to $ϕ^{i} / 5$ rounded to the nearest integer. Thus, Fibonacci numbers grow exponentially.

Why the rounding formula holds.
$F_{i} - \frac{ϕ ^{i}}{5} F_{i} - \frac{ϕ ^{i}}{5} = - \frac{ϕ ^ ^{i}}{5} = \frac{∣ ϕ ^ ∣ ^{i}}{5} < \frac{1}{5} < \frac{1}{2} . (from the closed form) (since ∣ \hat{ϕ} ∣ < 1 and 5 > 2)$
The integer $F_{i}$ is therefore within $1/2$ of the real number $ϕ^{i} / 5$ , so it is the nearest integer to it.

Exercises 3.2-1 to 3.2-8

3.2-1 Show that if $f (n)$ and $g (n)$ are monotonically increasing functions, then so are the functions $f (n) + g (n)$ and $f (g (n))$ , and if $f (n)$ and $g (n)$ are in addition nonnegative, then $f (n) \cdot g (n)$ is monotonically increasing.

3.2-2 Prove equation (3.16).

3.2-3 Prove equation (3.19). Also prove that $n! = ω (2^{n})$ and $n! = o (n^{n})$ .

3.2-4* Is the function $⌈ l g n ⌉!$ polynomially bounded? Is the function $⌈ l g l g n ⌉!$ polynomially bounded?

3.2-5* Which is asymptotically larger: $l g (l g^{*} n)$ or $l g^{*} (l g n)$ ?

3.2-6 Show that the golden ratio $ϕ$ and its conjugate $\hat{ϕ}$ both satisfy the equation $x^{2} = x + 1$ .

3.2-7 Prove by induction that the $i$ th Fibonacci number satisfies the equality $F_{i} = \frac{ϕ ^{i} - ϕ ^ ^{i}}{5},$ where $ϕ$ is the golden ratio and $\hat{ϕ}$ is its conjugate.

3.2-8 Show that $k ln k = Θ (n)$ implies $k = Θ (n / ln n)$ .

Problems

3-1 Asymptotic behavior of polynomials

Let

$p (n) = \sum_{i = 0}^{d} a_{i} n^{i},$

where $a_{d} > 0$ , be a degree- $d$ polynomial in $n$ , and let $k$ be a constant. Use the definitions of the asymptotic notations to prove the following properties.

a. If $k \geq d$ , then $p (n) = O (n^{k})$ .
b. If $k \leq d$ , then $p (n) = Ω (n^{k})$ .
c. If $k = d$ , then $p (n) = Θ (n^{k})$ .
d. If $k > d$ , then $p (n) = o (n^{k})$ .
e. If $k < d$ , then $p (n) = ω (n^{k})$ .

3-2 Relative asymptotic growths

Indicate, for each pair of expressions $(A, B)$ in the table below, whether $A$ is $O$ , $o$ , $Ω$ , $ω$ , or $Θ$ of $B$ . Assume that $k \geq 1$ , $ϵ > 0$ , and $c > 1$ are constants. Your answer should be in the form of the table with “yes” or “no” written in each box.

	$A$	$B$
a.	$l g^{k} n$	$n^{ϵ}$
b.	$n^{k}$	$c^{n}$
c.	$n$	$n^{s i n n}$
d.	$2^{n}$	$2^{n /2}$
e.	$n^{l g c}$	$c^{l g n}$
f.	$l g (n!)$	$l g (n^{n})$

3-3 Ordering by asymptotic growth rates

a. Rank the following functions by order of growth; that is, find an arrangement $g_{1}, g_{2}, \dots, g_{30}$ of the functions satisfying $g_{1} = Ω (g_{2})$ , $g_{2} = Ω (g_{3})$ , $\dots$ , $g_{29} = Ω (g_{30})$ . Partition your list into equivalence classes such that functions $f (n)$ and $g (n)$ are in the same class if and only if $f (n) = Θ (g (n))$ .

l g (l g^{*} n) (\frac{3}{2})^{n} ln ln n 2^{l g n} l g^{*} (l g n) 2^{l g^{*} n} n^{3} l g^{*} n (l g n)^{l g n} 2^{2 l g n} (2)^{l g n} l g^{2} n n \cdot 2^{n} e^{n} n n^{2} l g (n!) n^{l g l g n} 4^{l g n} 2^{n} n! 2^{2^{n}} ln n (n + 1)! n l g n (l g n)! n^{1/ l g n} 1 l g n 2^{2^{n + 1}}

b. Give an example of a single nonnegative function $f (n)$ such that for all functions $g_{i} (n)$ in part (a), $f (n)$ is neither $O (g_{i} (n))$ nor $Ω (g_{i} (n))$ .

3-4 Asymptotic notation properties

Let $f (n)$ and $g (n)$ be asymptotically positive functions. Prove or disprove each of the following conjectures.

a. $f (n) = O (g (n))$ implies $g (n) = O (f (n))$ .
b. $f (n) + g (n) = Θ (min (f (n), g (n)))$ .
c. $f (n) = O (g (n))$ implies $l g (f (n)) = O (l g (g (n)))$ , where $l g (g (n)) \geq 1$ and $f (n) \geq 1$ for all sufficiently large $n$ .
d. $f (n) = O (g (n))$ implies $2^{f (n)} = O (2^{g (n)})$ .
e. $f (n) = O ((f (n))^{2})$ .
f. $f (n) = O (g (n))$ implies $g (n) = Ω (f (n))$ .
g. $f (n) = Θ (f (n /2))$ .
h. $f (n) + o (f (n)) = Θ (f (n))$ .

3-5 Variations on $O$ and $Ω$

Some authors define $Ω$ in a slightly different way than we do; let’s use $Ω \infty$ (read “omega infinity”) for this alternative definition. We say that $f (n) = Ω \infty (g (n))$ if there exists a positive constant $c$ such that $f (n) \geq c g (n) \geq 0$ for infinitely many integers $n$ .

a. Show that for any two functions $f (n)$ and $g (n)$ that are asymptotically nonnegative, either $f (n) = O (g (n))$ or $f (n) = Ω \infty (g (n))$ or both, whereas this is not true if we use $Ω$ in place of $Ω \infty$ .
b. Describe the potential advantages and disadvantages of using $Ω \infty$ instead of $Ω$ to characterize the running times of programs.

Some authors also define $O$ in a slightly different manner; let’s use $O^{'}$ for the alternative definition. We say that $f (n) = O^{'} (g (n))$ if and only if $∣ f (n) ∣ = O (g (n))$ .

c. What happens to each direction of the “if and only if” in Theorem 3.1 if we substitute $O^{'}$ for $O$ but still use $Ω$ ?

Some authors define $O$ (read “soft-oh”) to mean $O$ with logarithmic factors ignored:

O (g (n)) = {f (n) : there exist positive constants c, k, and n_{0} such that 0 \leq f (n) \leq c g (n) l g^{k} (n) for all n \geq n_{0}} .

d. Define $Ω$ and $Θ$ in a similar manner. Prove the corresponding analog to Theorem 3.1.

3-6 Iterated functions

We can apply the iteration operator $^{*}$ used in the $l g^{*}$ function to any monotonically increasing function $f (n)$ over the reals. For a given constant $c \in R$ , we define the iterated function $f_{c}^{*}$ by

$f_{c}^{*} (n) = min {i \geq 0 : f^{(i)} (n) \leq c},$

which need not be well defined in all cases. In other words, the quantity $f_{c}^{*} (n)$ is the number of iterated applications of the function $f$ required to reduce its argument down to $c$ or less.

For each of the following functions $f (n)$ and constants $c$ , give as tight a bound as possible on $f_{c}^{*} (n)$ .

	$f (n)$	$c$
a.	$n - 1$	$0$
b.	$l g n$	$1$
c.	$n /2$	$1$
d.	$n /2$	$2$
e.	$n$	$2$
f.	$n$	$1$
g.	$n^{1/3}$	$2$
h.	$n / l g n$	$2$

Chapter notes

Knuth [209] traces the origin of the $O$ -notation to a number-theory text by P. Bachmann in 1892. The $o$ -notation was invented by E. Landau in 1909 for his discussion of the distribution of prime numbers. The $Ω$ and $Θ$ notations were advocated by Knuth [213] to correct the popular, but technically sloppy, practice in the literature of using $O$ -notation for both upper and lower bounds. Many people continue to use the $O$ -notation where the $Θ$ -notation is more technically precise. Further discussion of the history and development of asymptotic notations appears in works by Knuth [209, 213] and Brassard and Bratley [54].

Not all authors define the asymptotic notations in the same way, although the various definitions agree in most common situations. Some of the alternative definitions encompass functions that are not asymptotically nonnegative, as long as their absolute values are appropriately bounded.

Equation (3.20) is due to Robbins [297]. Other properties of elementary mathematical functions can be found in any good mathematical reference, such as Abramowitz and Stegun [1] or Zwillinger [362], or in a calculus book, such as Apostol [18] or Thomas et al. [334]. Knuth [209] and Graham, Knuth, and Patashnik [152] contain a wealth of material on discrete mathematics as used in computer science.

4 Divide-and-Conquer

Big picture. Imagine you must sort a gigantic pile of exam papers. Rather than wrestle with the whole pile, you split it in half, hand each half to a friend who splits it again, and so on, until each person holds a single paper. Then everyone merges their results back together. That is divide-and-conquer: shrink the problem, trust recursion, and stitch the answers together. In this chapter we learn both new algorithms in this style and the mathematical machinery (recurrences) for measuring how fast they run.

In Section 2.3.1, we saw how merge sort serves as an example of the divide-and-conquer paradigm. Recall that in divide-and-conquer, we solve a problem recursively, applying three steps at each level of the recursion:

Divide the problem into a number of subproblems that are smaller instances of the same problem.
Conquer the subproblems by solving them recursively. If the subproblem sizes are small enough, however, just solve the subproblems in a straightforward manner.
Combine the solutions to the subproblems into the solution for the original problem.

When the subproblems are large enough to solve recursively, we call that the recursive case. Once the subproblems become small enough that we no longer recurse, we say that the recursion “bottoms out” and that we have gotten down to the base case. Sometimes, in addition to subproblems that are smaller instances of the same problem, we have to solve subproblems that are not quite the same as the original problem. We consider solving such subproblems as part of the combine step.

In this chapter, we shall see more algorithms based on divide-and-conquer. The first one solves the maximum-subarray problem: it takes as input an array of numbers, and it determines the contiguous subarray whose values have the greatest sum. Then we shall see two divide-and-conquer algorithms for multiplying $n \times n$ matrices. One runs in $Θ (n^{3})$ time, which is no better than the straightforward method of multiplying square matrices. But the other, Strassen’s algorithm, runs in $O (n^{2.81})$ time, which beats the straightforward method asymptotically.

Recurrences

Recurrences go hand in hand with the divide-and-conquer paradigm, because they give us a natural way to characterize the running times of divide-and-conquer algorithms. A recurrence is an equation or inequality that describes a function in terms of its value on smaller inputs. For example, in Section 2.3.2 we described the worst-case running time $T (n)$ of the MERGE-SORT procedure by the recurrence

T (n) = {Θ (1) 2 T (n /2) + Θ (n) if n = 1, if n > 1, (4.1)

whose solution we claimed to be $T (n) = Θ (n l g n)$ .

Recurrences can take many forms. For example, a recursive algorithm might divide subproblems into unequal sizes, such as a 2/3-to-1/3 split. If the divide and combine steps take linear time, such an algorithm would give rise to the recurrence

$T (n) = T (2 n /3) + T (n /3) + Θ (n) .$

Subproblems are not necessarily constrained to being a constant fraction of the original problem size. For example, a recursive version of linear search (see Exercise 2.1-3) would create just one subproblem containing only one element fewer than the original problem. Each recursive call would take constant time plus the time for the recursive calls it makes, yielding the recurrence

$T (n) = T (n - 1) + Θ (1) .$

This chapter offers three methods for solving recurrences, that is, for obtaining asymptotic " $Θ$ " or " $O$ " bounds on the solution:

In the substitution method, we guess a bound and then use mathematical induction to prove our guess correct.
The recursion-tree method converts the recurrence into a tree whose nodes represent the costs incurred at various levels of the recursion. We use techniques for bounding summations to solve the recurrence.
The master method provides bounds for recurrences of the form

T (n) = a T (n / b) + f (n), (4.2)

where $a \geq 1$ , $b > 1$ , and $f (n)$ is a given function. Such recurrences arise frequently. A recurrence of the form in equation (4.2) characterizes a divide-and-conquer algorithm that creates $a$ subproblems, each of which is $1/ b$ the size of the original problem, and in which the divide and combine steps together take $f (n)$ time.

To use the master method, you will need to memorize three cases, but once you do that, you will easily be able to determine asymptotic bounds for many simple recurrences. We will use the master method to determine the running times of the divide-and-conquer algorithms for the maximum-subarray problem and for matrix multiplication, as well as for other algorithms based on divide-and-conquer elsewhere in this book.

Occasionally, we shall see recurrences that are not equalities but rather inequalities, such as $T (n) \leq 2 T (n /2) + Θ (n)$ . Because such a recurrence states only an upper bound on $T (n)$ , we will couch its solution using $O$ -notation rather than $Θ$ -notation. Similarly, if the inequality were reversed to $T (n) \geq 2 T (n /2) + Θ (n)$ , then because the recurrence gives only a lower bound on $T (n)$ , we would use $Ω$ -notation in its solution.

Technicalities in recurrences

In practice, we neglect certain technical details when we state and solve recurrences. For example, if we call MERGE-SORT on $n$ elements when $n$ is odd, we end up with subproblems of size $⌊ n /2 ⌋$ and $⌈ n /2 ⌉$ . Neither size is actually $n /2$ , because $n /2$ is not an integer when $n$ is odd. Technically, the recurrence describing the worst-case running time of MERGE-SORT is really

T (n) = {Θ (1) T (⌈ n /2 ⌉) + T (⌊ n /2 ⌋) + Θ (n) if n = 1, if n > 1. (4.3)

Boundary conditions represent another class of details that we typically ignore. Since the running time of an algorithm on a constant-sized input is a constant, the recurrences that arise from the running times of algorithms generally have $T (n) = Θ (1)$ for sufficiently small $n$ . Consequently, for convenience, we shall generally omit statements of the boundary conditions of recurrences and assume that $T (n)$ is constant for small $n$ . For example, we normally state recurrence (4.1) as

T (n) = 2 T (n /2) + Θ (n), (4.4)

without explicitly giving values for small $n$ . The reason is that although changing the value of $T (1)$ changes the exact solution to the recurrence, the solution typically doesn’t change by more than a constant factor, and so the order of growth is unchanged.

When we state and solve recurrences, we often omit floors, ceilings, and boundary conditions. We forge ahead without these details and later determine whether or not they matter. They usually do not, but you should know when they do. Experience helps, and so do some theorems stating that these details do not affect the asymptotic bounds of many recurrences characterizing divide-and-conquer algorithms (see Theorem 4.1). In this chapter, however, we shall address some of these details and illustrate the fine points of recurrence solution methods.

Think about it. Why might it be safe to ignore the floors and ceilings? Hint: rounding $n /2$ changes the subproblem size by at most $1$ , a tiny nudge compared with $n$ itself.

4.1 The maximum-subarray problem

Suppose that you have been offered the opportunity to invest in the Volatile Chemical Corporation. Like the chemicals the company produces, the stock price of the Volatile Chemical Corporation is rather volatile. You are allowed to buy one unit of stock only one time and then sell it at a later date, buying and selling after the close of trading for the day. To compensate for this restriction, you are allowed to learn what the price of the stock will be in the future. Your goal is to maximize your profit. Figure 4.1 shows the price of the stock over a 17-day period. You may buy the stock at any one time, starting after day 0, when the price is $100 per share. Of course, you would want to “buy low, sell high”: buy at the lowest possible price and later on sell at the highest possible price to maximize your profit. Unfortunately, you might not be able to buy at the lowest price and then sell at the highest price within a given period. In Figure 4.1, the lowest price occurs after day 7, which occurs after the highest price, after day 1.

You might think that you can always maximize profit by either buying at the lowest price or selling at the highest price. For example, in Figure 4.1, we would maximize profit by buying at the lowest price, after day 7. If this strategy always worked, then it would be easy to determine how to maximize profit: find the highest and lowest prices, and then work left from the highest price to find the lowest prior price, work right from the lowest price to find the highest later price, and take the pair with the greater difference. Figure 4.2 shows a simple counterexample, demonstrating that the maximum profit sometimes comes neither by buying at the lowest price nor by selling at the highest price.

Figure 4.1: Information about the price of stock in the Volatile Chemical Corporation after the close of trading over a period of 17 days. The horizontal axis of the chart indicates the day, and the vertical axis shows the price. The bottom row of the table gives the change in price from the previous day.

Day	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
Price	100	113	110	85	105	102	86	63	81	101	94	106	101	79	94	90	97
Change		13	-3	-25	20	-3	-16	-23	18	20	-7	12	-5	-22	15	-4	7

Figure 4.2: An example showing that the maximum profit does not always start at the lowest price or end at the highest price. Here, the maximum profit of $3 per share would be earned by buying after day 2 and selling after day 3. The price of $7 after day 2 is not the lowest price overall, and the price of $10 after day 3 is not the highest price overall.

Day	0	1	2	3	4
Price	10	11	7	10	6
Change		1	-4	3	-4

A brute-force solution

We can easily devise a brute-force solution to this problem: just try every possible pair of buy and sell dates in which the buy date precedes the sell date. A period of $n$ days has $(2 n)$ such pairs of dates. Since $(2 n)$ is $Θ (n^{2})$ , and the best we can hope for is to evaluate each pair of dates in constant time, this approach would take $Ω (n^{2})$ time. Can we do better?

A transformation

In order to design an algorithm with an $o (n^{2})$ running time, we will look at the input in a slightly different way. We want to find a sequence of days over which the net change from the first day to the last is maximum. Instead of looking at the daily prices, let us instead consider the daily change in price, where the change on day $i$ is the difference between the prices after day $i - 1$ and after day $i$ . The table in Figure 4.1 shows these daily changes in the bottom row. If we treat this row as an array $A$ , shown in Figure 4.3, we now want to find the nonempty, contiguous subarray of $A$ whose values have the largest sum. We call this contiguous subarray the maximum subarray. For example, in the array of Figure 4.3, the maximum subarray of $A [1..16]$ is $A [8..11]$ , with the sum $43$ . Thus, you would want to buy the stock just before day 8 (that is, after day 7) and sell it after day 11, earning a profit of $43 per share.

Figure 4.3: The change in stock prices as a maximum-subarray problem. Here, the subarray $A [8..11]$ , with sum $43$ , has the greatest sum of any contiguous subarray of array $A$ .

index	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
$A$	13	-3	-25	20	-3	-16	-23	18	20	-7	12	-5	-22	15	-4	7

(The bold entries $A [8..11]$ form the maximum subarray: $18 + 20 - 7 + 12 = 43$ .)

At first glance, this transformation does not help. We still need to check $(2 n - 1) + (n - 1) = Θ (n^{2})$ subarrays for a period of $n$ days. Exercise 4.1-2 asks you to show that although computing the cost of one subarray might take time proportional to the length of the subarray, when computing all $Θ (n^{2})$ subarray sums, we can organize the computation so that each subarray sum takes $O (1)$ time, given the values of previously computed subarray sums, so that the brute-force solution takes $Θ (n^{2})$ time.

So let us seek a more efficient solution to the maximum-subarray problem. When doing so, we will usually speak of “a” maximum subarray rather than “the” maximum subarray, since there could be more than one subarray that achieves the maximum sum.

The maximum-subarray problem is interesting only when the array contains some negative numbers. If all the array entries were nonnegative, then the maximum-subarray problem would present no challenge, since the entire array would give the greatest sum.

A solution using divide-and-conquer

Let’s think about how we might solve the maximum-subarray problem using the divide-and-conquer technique. Suppose we want to find a maximum subarray of the subarray $A [low .. high]$ . Divide-and-conquer suggests that we divide the subarray into two subarrays of as equal size as possible. That is, we find the midpoint, say $mid$ , of the subarray, and consider the subarrays $A [low .. mid]$ and $A [mid + 1.. high]$ . As Figure 4.4(a) shows, any contiguous subarray $A [i .. j]$ of $A [low .. high]$ must lie in exactly one of the following places:

entirely in the subarray $A [low .. mid]$ , so that $low \leq i \leq j \leq mid$ ,
entirely in the subarray $A [mid + 1.. high]$ , so that $mid < i \leq j \leq high$ , or
crossing the midpoint, so that $low \leq i \leq mid < j \leq high$ .

Therefore, a maximum subarray of $A [low .. high]$ must lie in exactly one of these places. In fact, a maximum subarray of $A [low .. high]$ must have the greatest sum over all subarrays entirely in $A [low .. mid]$ , entirely in $A [mid + 1.. high]$ , or crossing the midpoint. We can find maximum subarrays of $A [low .. mid]$ and $A [mid + 1.. high]$ recursively, because these two subproblems are smaller instances of the problem of finding a maximum subarray. Thus, all that is left to do is find a maximum subarray that crosses the midpoint, and take a subarray with the largest sum of the three.

Figure 4.4: (a) Possible locations of subarrays of $A [low .. high]$ : entirely in $A [low .. mid]$ , entirely in $A [mid + 1.. high]$ , or crossing the midpoint $mid$ . (b) Any subarray of $A [low .. high]$ crossing the midpoint comprises two subarrays $A [i .. mid]$ and $A [mid + 1.. j]$ , where $low \leq i \leq mid$ and $mid < j \leq high$ .

We can easily find a maximum subarray crossing the midpoint in time linear in the size of the subarray $A [low .. high]$ . This problem is not a smaller instance of our original problem, because it has the added restriction that the subarray it chooses must cross the midpoint. As Figure 4.4(b) shows, any subarray crossing the midpoint is itself made of two subarrays $A [i .. mid]$ and $A [mid + 1.. j]$ , where $low \leq i \leq mid$ and $mid < j \leq high$ . Therefore, we just need to find maximum subarrays of the form $A [i .. mid]$ and $A [mid + 1.. j]$ and then combine them. The procedure FIND-MAX-CROSSING-SUBARRAY takes as input the array $A$ and the indices $low$ , $mid$ , and $high$ , and it returns a tuple containing the indices demarcating a maximum subarray that crosses the midpoint, along with the sum of the values in a maximum subarray.

FIND-MAX-CROSSING-SUBARRAY(A, low, mid, high)
 1  left-sum = -∞
 2  sum = 0
 3  for i = mid downto low
 4      sum = sum + A[i]
 5      if sum > left-sum
 6          left-sum = sum
 7          max-left = i
 8  right-sum = -∞
 9  sum = 0
10  for j = mid + 1 to high
11      sum = sum + A[j]
12      if sum > right-sum
13          right-sum = sum
14          max-right = j
15  return (max-left, max-right, left-sum + right-sum)

This procedure works as follows. Lines 1-7 find a maximum subarray of the left half, $A [low .. mid]$ . Since this subarray must contain $A [mid]$ , the for loop of lines 3-7 starts the index $i$ at $mid$ and works down to $low$ , so that every subarray it considers is of the form $A [i .. mid]$ . Lines 1-2 initialize the variables $left - sum$ , which holds the greatest sum found so far, and $sum$ , holding the sum of the entries in $A [i .. mid]$ . Whenever we find, in line 5, a subarray $A [i .. mid]$ with a sum of values greater than $left - sum$ , we update $left - sum$ to this subarray’s sum in line 6, and in line 7 we update the variable $max - left$ to record this index $i$ . Lines 8-14 work analogously for the right half, $A [mid + 1.. high]$ . Here, the for loop of lines 10-14 starts the index $j$ at $mid + 1$ and works up to $high$ , so that every subarray it considers is of the form $A [mid + 1.. j]$ . Finally, line 15 returns the indices $max - left$ and $max - right$ that demarcate a maximum subarray crossing the midpoint, along with the sum $left - sum + right - sum$ of the values in the subarray $A [max - left .. max - right]$ .

Think about it. Why is it enough to optimize the left and right halves independently? Because a crossing subarray’s sum is just (left piece) + (right piece), and the two pieces do not interact: the best total comes from the best left piece plus the best right piece.

Analysis of FIND-MAX-CROSSING-SUBARRAY. If the subarray $A [low .. high]$ contains $n$ entries (so that $n = high - low + 1$ ), we claim that the call takes $Θ (n)$ time. Each iteration of each of the two for loops takes $O (1)$ time, so we just need to count the iterations. The loop of lines 3-7 makes $mid - low + 1$ iterations, and the loop of lines 10-14 makes $high - mid$ iterations, so the total number of iterations is
$(mid - low + 1) + (high - mid) = high - low + 1 = n .$

With a linear-time FIND-MAX-CROSSING-SUBARRAY procedure in hand, we can write pseudocode for a divide-and-conquer algorithm to solve the maximum-subarray problem:

FIND-MAXIMUM-SUBARRAY(A, low, high)
 1  if high == low
 2      return (low, high, A[low])      // base case: only one element
 3  else mid = ⌊(low + high)/2⌋
 4      (left-low, left-high, left-sum) =
            FIND-MAXIMUM-SUBARRAY(A, low, mid)
 5      (right-low, right-high, right-sum) =
            FIND-MAXIMUM-SUBARRAY(A, mid + 1, high)
 6      (cross-low, cross-high, cross-sum) =
            FIND-MAX-CROSSING-SUBARRAY(A, low, mid, high)
 7      if left-sum ≥ right-sum and left-sum ≥ cross-sum
 8          return (left-low, left-high, left-sum)
 9      elseif right-sum ≥ left-sum and right-sum ≥ cross-sum
10          return (right-low, right-high, right-sum)
11      else return (cross-low, cross-high, cross-sum)

The initial call FIND-MAXIMUM-SUBARRAY $(A, 1, A . length)$ will find a maximum subarray of $A [1.. n]$ .

Similar to FIND-MAX-CROSSING-SUBARRAY, the recursive procedure FIND-MAXIMUM-SUBARRAY returns a tuple containing the indices that demarcate a maximum subarray, along with the sum of the values in a maximum subarray. Line 1 tests for the base case, where the subarray has just one element. A subarray with just one element has only one subarray (itself), and so line 2 returns a tuple with the starting and ending indices of just the one element, along with its value. Lines 3-11 handle the recursive case. Line 3 does the divide part, computing the index $mid$ of the midpoint. Let’s refer to the subarray $A [low .. mid]$ as the left subarray and to $A [mid + 1.. high]$ as the right subarray. Because we know that the subarray $A [low .. high]$ contains at least two elements, each of the left and right subarrays must have at least one element. Lines 4 and 5 conquer by recursively finding maximum subarrays within the left and right subarrays, respectively. Lines 6-11 form the combine part. Line 6 finds a maximum subarray that crosses the midpoint. (Recall that because line 6 solves a subproblem that is not a smaller instance of the original problem, we consider it to be in the combine part.) Line 7 tests whether the left subarray contains a subarray with the maximum sum, and line 8 returns that maximum subarray. Otherwise, line 9 tests whether the right subarray contains a subarray with the maximum sum, and line 10 returns that maximum subarray. If neither the left nor right subarrays contain a subarray achieving the maximum sum, then a maximum subarray must cross the midpoint, and line 11 returns it.

Analyzing the divide-and-conquer algorithm

Next we set up a recurrence that describes the running time of the recursive FIND-MAXIMUM-SUBARRAY procedure. As we did when we analyzed merge sort in Section 2.3.2, we make the simplifying assumption that the original problem size is a power of 2, so that all subproblem sizes are integers. We denote by $T (n)$ the running time of FIND-MAXIMUM-SUBARRAY on a subarray of $n$ elements. For starters, line 1 takes constant time. The base case, when $n = 1$ , is easy: line 2 takes constant time, and so

T (1) = Θ (1) . (4.5)

The recursive case occurs when $n > 1$ . Lines 1 and 3 take constant time. Each of the subproblems solved in lines 4 and 5 is on a subarray of $n /2$ elements (our assumption that the original problem size is a power of 2 ensures that $n /2$ is an integer), and so we spend $T (n /2)$ time solving each of them. Because we have to solve two subproblems (for the left subarray and for the right subarray), the contribution to the running time from lines 4 and 5 comes to $2 T (n /2)$ . As we have already seen, the call to FIND-MAX-CROSSING-SUBARRAY in line 6 takes $Θ (n)$ time. Lines 7-11 take only $Θ (1)$ time. For the recursive case, therefore, we have

T (n) = Θ (1) + 2 T (n /2) + Θ (n) + Θ (1) = 2 T (n /2) + Θ (n) . (4.6)

Combining equations (4.5) and (4.6) gives us a recurrence for the running time $T (n)$ of FIND-MAXIMUM-SUBARRAY:

T (n) = {Θ (1) 2 T (n /2) + Θ (n) if n = 1, if n > 1. (4.7)

This recurrence is the same as recurrence (4.1) for merge sort. As we shall see from the master method in Section 4.5, this recurrence has the solution $T (n) = Θ (n l g n)$ . You might also revisit the recursion tree in Figure 2.5 to understand why the solution should be $T (n) = Θ (n l g n)$ .

Thus, we see that the divide-and-conquer method yields an algorithm that is asymptotically faster than the brute-force method. With merge sort and now the maximum-subarray problem, we begin to get an idea of how powerful the divide-and-conquer method can be. Sometimes it will yield the asymptotically fastest algorithm for a problem, and other times we can do even better. As Exercise 4.1-5 shows, there is in fact a linear-time algorithm for the maximum-subarray problem, and it does not use divide-and-conquer.

Exercises 4.1-1 to 4.1-5

4.1-1 What does FIND-MAXIMUM-SUBARRAY return when all elements of $A$ are negative?

4.1-2 Write pseudocode for the brute-force method of solving the maximum-subarray problem. Your procedure should run in $Θ (n^{2})$ time.

4.1-3 Implement both the brute-force and recursive algorithms for the maximum-subarray problem on your own computer. What problem size $n_{0}$ gives the crossover point at which the recursive algorithm beats the brute-force algorithm? Then, change the base case of the recursive algorithm to use the brute-force algorithm whenever the problem size is less than $n_{0}$ . Does that change the crossover point?

4.1-4 Suppose we change the definition of the maximum-subarray problem to allow the result to be an empty subarray, where the sum of the values of an empty subarray is $0$ . How would you change any of the algorithms that do not allow empty subarrays to permit an empty subarray to be the result?

4.1-5 Use the following ideas to develop a nonrecursive, linear-time algorithm for the maximum-subarray problem. Start at the left end of the array, and progress toward the right, keeping track of the maximum subarray seen so far. Knowing a maximum subarray of $A [1.. j]$ , extend the answer to find a maximum subarray ending at index $j + 1$ by using the following observation: a maximum subarray of $A [1.. j + 1]$ is either a maximum subarray of $A [1.. j]$ or a subarray $A [i .. j + 1]$ , for some $1 \leq i \leq j + 1$ . Determine a maximum subarray of the form $A [i .. j + 1]$ in constant time based on knowing a maximum subarray ending at index $j$ .

4.2 Strassen’s algorithm for matrix multiplication

Big picture. Multiplying two matrices the school way feels like it must take about $n^{3}$ work, because that many products appear in the definition. Strassen’s insight is a magician’s trick: with some clever bookkeeping you can get away with fewer multiplications, paying for them with extra (cheaper) additions.

If you have seen matrices before, then you probably know how to multiply them. (Otherwise, you should read Section D.1 in Appendix D.) If $A = (a_{ij})$ and $B = (b_{ij})$ are square $n \times n$ matrices, then in the product $C = A \cdot B$ , we define the entry $c_{ij}$ , for $i, j = 1, 2, \dots, n$ , by

c_{ij} = k = 1 \sum n a_{ik} \cdot b_{k j} . (4.8)

We must compute $n^{2}$ matrix entries, and each is the sum of $n$ values. The following procedure takes $n \times n$ matrices $A$ and $B$ and multiplies them, returning their $n \times n$ product $C$ . We assume that each matrix has an attribute rows, giving the number of rows in the matrix.

SQUARE-MATRIX-MULTIPLY(A, B)
1  n = A.rows
2  let C be a new n × n matrix
3  for i = 1 to n
4      for j = 1 to n
5          c_ij = 0
6          for k = 1 to n
7              c_ij = c_ij + a_ik · b_kj
8  return C

The SQUARE-MATRIX-MULTIPLY procedure works as follows. The for loop of lines 3-7 computes the entries of each row $i$ , and within a given row $i$ , the for loop of lines 4-7 computes each of the entries $c_{ij}$ , for each column $j$ . Line 5 initializes $c_{ij}$ to $0$ as we start computing the sum given in equation (4.8), and each iteration of the for loop of lines 6-7 adds in one more term of equation (4.8).

Because each of the triply-nested for loops runs exactly $n$ iterations, and each execution of line 7 takes constant time, the SQUARE-MATRIX-MULTIPLY procedure takes $Θ (n^{3})$ time.

You might at first think that any matrix multiplication algorithm must take $Ω (n^{3})$ time, since the natural definition of matrix multiplication requires that many multiplications. You would be incorrect, however: we have a way to multiply matrices in $o (n^{3})$ time. In this section, we shall see Strassen’s remarkable recursive algorithm for multiplying $n \times n$ matrices. It runs in $Θ (n^{l g 7})$ time, which we shall show in Section 4.5. Since $l g 7$ lies between $2.80$ and $2.81$ , Strassen’s algorithm runs in $O (n^{2.81})$ time, which is asymptotically better than the simple SQUARE-MATRIX-MULTIPLY procedure.

A simple divide-and-conquer algorithm

To keep things simple, when we use a divide-and-conquer algorithm to compute the matrix product $C = A \cdot B$ , we assume that $n$ is an exact power of $2$ in each of the $n \times n$ matrices. We make this assumption because in each divide step, we will divide $n \times n$ matrices into four $n /2 \times n /2$ matrices, and by assuming that $n$ is an exact power of $2$ , we are guaranteed that as long as $n \geq 2$ , the dimension $n /2$ is an integer.

Suppose that we partition each of $A$ , $B$ , and $C$ into four $n /2 \times n /2$ matrices:

A = (A_{11} A_{21} A_{12} A_{22}), B = (B_{11} B_{21} B_{12} B_{22}), C = (C_{11} C_{21} C_{12} C_{22}) . (4.9)

so that we rewrite the equation $C = A \cdot B$ as

(C_{11} C_{21} C_{12} C_{22}) = (A_{11} A_{21} A_{12} A_{22}) \cdot (B_{11} B_{21} B_{12} B_{22}) . (4.10)

Equation (4.10) corresponds to the four equations

C_{11} C_{12} C_{21} C_{22} = A_{11} \cdot B_{11} + A_{12} \cdot B_{21}, = A_{11} \cdot B_{12} + A_{12} \cdot B_{22}, = A_{21} \cdot B_{11} + A_{22} \cdot B_{21}, = A_{21} \cdot B_{12} + A_{22} \cdot B_{22} . (4.11) (4.12) (4.13) (4.14)

Each of these four equations specifies two multiplications of $n /2 \times n /2$ matrices and the addition of their $n /2 \times n /2$ products. We can use these equations to create a straightforward, recursive, divide-and-conquer algorithm:

SQUARE-MATRIX-MULTIPLY-RECURSIVE(A, B)
 1  n = A.rows
 2  let C be a new n × n matrix
 3  if n == 1
 4      c_11 = a_11 · b_11
 5  else partition A, B, and C as in equations (4.9)
 6      C_11 = SQUARE-MATRIX-MULTIPLY-RECURSIVE(A_11, B_11)
                + SQUARE-MATRIX-MULTIPLY-RECURSIVE(A_12, B_21)
 7      C_12 = SQUARE-MATRIX-MULTIPLY-RECURSIVE(A_11, B_12)
                + SQUARE-MATRIX-MULTIPLY-RECURSIVE(A_12, B_22)
 8      C_21 = SQUARE-MATRIX-MULTIPLY-RECURSIVE(A_21, B_11)
                + SQUARE-MATRIX-MULTIPLY-RECURSIVE(A_22, B_21)
 9      C_22 = SQUARE-MATRIX-MULTIPLY-RECURSIVE(A_21, B_12)
                + SQUARE-MATRIX-MULTIPLY-RECURSIVE(A_22, B_22)
10  return C

This pseudocode glosses over one subtle but important implementation detail. How do we partition the matrices in line 5? If we were to create $12$ new $n /2 \times n /2$ matrices, we would spend $Θ (n^{2})$ time copying entries. In fact, we can partition the matrices without copying entries. The trick is to use index calculations. We identify a submatrix by a range of row indices and a range of column indices of the original matrix. We end up representing a submatrix a little differently from how we represent the original matrix, which is the subtlety we are glossing over. The advantage is that, since we can specify submatrices by index calculations, executing line 5 takes only $Θ (1)$ time (although it turns out that it makes no difference asymptotically to the overall running time whether we copy or partition in place).

Now, we derive a recurrence to characterize the running time of SQUARE-MATRIX-MULTIPLY-RECURSIVE. Let $T (n)$ be the time to multiply two $n \times n$ matrices using this procedure. In the base case, when $n = 1$ , we perform just the one scalar multiplication in line 4, and so

T (1) = Θ (1) . (4.15)

The recursive case occurs when $n > 1$ . As discussed, partitioning the matrices in line 5 takes $Θ (1)$ time, using index calculations. In lines 6-9, we recursively call SQUARE-MATRIX-MULTIPLY-RECURSIVE a total of eight times. Because each recursive call multiplies two $n /2 \times n /2$ matrices, thereby contributing $T (n /2)$ to the overall running time, the time taken by all eight recursive calls is $8 T (n /2)$ . We also must account for the four matrix additions in lines 6-9. Each of these matrices contains $n^{2} /4$ entries, and so each of the four matrix additions takes $Θ (n^{2})$ time. Since the number of matrix additions is a constant, the total time spent adding matrices in lines 6-9 is $Θ (n^{2})$ . (Again, we use index calculations to place the results of the matrix additions into the correct positions of matrix $C$ , with an overhead of $Θ (1)$ time per entry.) The total time for the recursive case, therefore, is the sum of the partitioning time, the time for all the recursive calls, and the time to add the matrices resulting from the recursive calls:

T (n) = Θ (1) + 8 T (n /2) + Θ (n^{2}) = 8 T (n /2) + Θ (n^{2}) . (4.16)

Notice that if we implemented partitioning by copying matrices, which would cost $Θ (n^{2})$ time, the recurrence would not change, and hence the overall running time would increase by only a constant factor.

Combining equations (4.15) and (4.16) gives us the recurrence for the running time of SQUARE-MATRIX-MULTIPLY-RECURSIVE:

T (n) = {Θ (1) 8 T (n /2) + Θ (n^{2}) if n = 1, if n > 1. (4.17)

As we shall see from the master method in Section 4.5, recurrence (4.17) has the solution $T (n) = Θ (n^{3})$ . Thus, this simple divide-and-conquer approach is no faster than the straightforward SQUARE-MATRIX-MULTIPLY procedure.

Before we continue on to examining Strassen’s algorithm, let us review where the components of equation (4.16) came from. Partitioning each $n \times n$ matrix by index calculation takes $Θ (1)$ time, but we have two matrices to partition. Although you could say that partitioning the two matrices takes $Θ (2)$ time, the constant of $2$ is subsumed by the $Θ$ -notation. Adding two matrices, each with, say, $k$ entries, takes $Θ (k)$ time. Since the matrices we add each have $n^{2} /4$ entries, you could say that adding each pair takes $Θ (n^{2} /4)$ time. Again, however, the $Θ$ -notation subsumes the constant factor of $1/4$ , and we say that adding two $n /2 \times n /2$ matrices takes $Θ (n^{2})$ time. We have four such matrix additions, and once again, instead of saying that they take $Θ (4 n^{2} /4)$ time, we say that they take $Θ (n^{2})$ time. (Of course, you might observe that we could say that the four matrix additions take $Θ (4 n^{2} /4)$ time, and that $4 n^{2} /4 = n^{2}$ , but the point here is that $Θ$ -notation subsumes constant factors, whatever they are.) Thus, we end up with two terms of $Θ (n^{2})$ , which we can combine into one.

When we account for the eight recursive calls, however, we cannot just subsume the constant factor of $8$ . In other words, we must say that together they take $8 T (n /2)$ time, rather than just $T (n /2)$ time. You can get a feel for why by looking back at the recursion tree in Figure 2.5, for recurrence (2.1) (which is identical to recurrence (4.7)), with the recursive case $T (n) = 2 T (n /2) + Θ (n)$ . The factor of $2$ determined how many children each tree node had, which in turn determined how many terms contributed to the sum at each level of the tree. If we were to ignore the factor of $8$ in equation (4.16) or the factor of $2$ in recurrence (4.1), the recursion tree would just be linear, rather than “bushy,” and each level would contribute only one term to the sum.

Bear in mind, therefore, that although asymptotic notation subsumes constant multiplicative factors, recursive notation such as $T (n /2)$ does not.

Think about it. Eight half-size multiplications, each costing $T (n /2)$ , hide a factor of $8/ 2^{3} = 1$ in the exponent: halving $n$ shrinks the work of one multiplication by a factor of $8$ , but we make $8$ of them. The savings exactly cancel, which is why this approach lands on $n^{3}$ again. What would happen if we could make only $7$ calls?

Strassen’s method

The key to Strassen’s method is to make the recursion tree slightly less bushy. That is, instead of performing eight recursive multiplications of $n /2 \times n /2$ matrices, it performs only seven. The cost of eliminating one matrix multiplication will be several new additions of $n /2 \times n /2$ matrices, but still only a constant number of additions. As before, the constant number of matrix additions will be subsumed by $Θ$ -notation when we set up the recurrence equation to characterize the running time.

Strassen’s method is not at all obvious. (This might be the biggest understatement in this book.) It has four steps:

Divide the input matrices $A$ and $B$ and output matrix $C$ into $n /2 \times n /2$ submatrices, as in equation (4.9). This step takes $Θ (1)$ time by index calculation, just as in SQUARE-MATRIX-MULTIPLY-RECURSIVE.
Create $10$ matrices $S_{1}, S_{2}, \dots, S_{10}$ , each of which is $n /2 \times n /2$ and is the sum or difference of two matrices created in step 1. We can create all $10$ matrices in $Θ (n^{2})$ time.
Using the submatrices created in step 1 and the $10$ matrices created in step 2, recursively compute seven matrix products $P_{1}, P_{2}, \dots, P_{7}$ . Each matrix $P_{i}$ is $n /2 \times n /2$ .
Compute the desired submatrices $C_{11}, C_{12}, C_{21}, C_{22}$ of the result matrix $C$ by adding and subtracting various combinations of the $P_{i}$ matrices. We can compute all four submatrices in $Θ (n^{2})$ time.

We shall see the details of steps 2-4 in a moment, but we already have enough information to set up a recurrence for the running time of Strassen’s method. Let us assume that once the matrix size $n$ gets down to $1$ , we perform a simple scalar multiplication, just as in line 4 of SQUARE-MATRIX-MULTIPLY-RECURSIVE. When $n > 1$ , steps 1, 2, and 4 take a total of $Θ (n^{2})$ time, and step 3 requires us to perform seven multiplications of $n /2 \times n /2$ matrices. Hence, we obtain the following recurrence for the running time $T (n)$ of Strassen’s algorithm:

T (n) = {Θ (1) 7 T (n /2) + Θ (n^{2}) if n = 1, if n > 1. (4.18)

We have traded off one matrix multiplication for a constant number of matrix additions. Once we understand recurrences and their solutions, we shall see that this tradeoff actually leads to a lower asymptotic running time. By the master method in Section 4.5, recurrence (4.18) has the solution $T (n) = Θ (n^{l g 7})$ .

We now proceed to describe the details. In step 2, we create the following $10$ matrices:

S_{1} S_{2} S_{3} S_{4} S_{5} S_{6} S_{7} S_{8} S_{9} S_{10} = B_{12} - B_{22}, = A_{11} + A_{12}, = A_{21} + A_{22}, = B_{21} - B_{11}, = A_{11} + A_{22}, = B_{11} + B_{22}, = A_{12} - A_{22}, = B_{21} + B_{22}, = A_{11} - A_{21}, = B_{11} + B_{12} .

Since we must add or subtract $n /2 \times n /2$ matrices $10$ times, this step does indeed take $Θ (n^{2})$ time.

In step 3, we recursively multiply $n /2 \times n /2$ matrices seven times to compute the following $n /2 \times n /2$ matrices, each of which is the sum or difference of products of $A$ and $B$ submatrices:

P_{1} P_{2} P_{3} P_{4} P_{5} P_{6} P_{7} = A_{11} \cdot S_{1} = A_{11} \cdot B_{12} - A_{11} \cdot B_{22}, = S_{2} \cdot B_{22} = A_{11} \cdot B_{22} + A_{12} \cdot B_{22}, = S_{3} \cdot B_{11} = A_{21} \cdot B_{11} + A_{22} \cdot B_{11}, = A_{22} \cdot S_{4} = A_{22} \cdot B_{21} - A_{22} \cdot B_{11}, = S_{5} \cdot S_{6} = A_{11} \cdot B_{11} + A_{11} \cdot B_{22} + A_{22} \cdot B_{11} + A_{22} \cdot B_{22}, = S_{7} \cdot S_{8} = A_{12} \cdot B_{21} + A_{12} \cdot B_{22} - A_{22} \cdot B_{21} - A_{22} \cdot B_{22}, = S_{9} \cdot S_{10} = A_{11} \cdot B_{11} + A_{11} \cdot B_{12} - A_{21} \cdot B_{11} - A_{21} \cdot B_{12} .

Note that the only multiplications we need to perform are those in the middle column of the above equations. The right-hand column just shows what these products equal in terms of the original submatrices created in step 1.

Step 4 adds and subtracts the $P_{i}$ matrices created in step 3 to construct the four $n /2 \times n /2$ submatrices of the product $C$ . We start with

C_{11} = P_{5} + P_{4} - P_{2} + P_{6} .

Expanding out the right-hand side, with the expansion of each $P_{i}$ on its own line and vertically aligning terms that cancel out, we see that $C_{11}$ equals

Verification of $C_{11}$ .
$C_{11} = P_{5} + P_{4} - P_{2} + P_{6} = P_{5} A_{11} B_{11} + A_{11} B_{22} + A_{22} B_{11} + A_{22} B_{22} + P_{4} A_{22} B_{21} - A_{22} B_{11} - P_{2} - A_{11} B_{22} - A_{12} B_{22} + P_{6} A_{12} B_{21} + A_{12} B_{22} - A_{22} B_{21} - A_{22} B_{22} = A_{11} B_{11} + (A_{11} B_{22} - A_{11} B_{22}) + (A_{22} B_{11} - A_{22} B_{11}) + (A_{22} B_{22} - A_{22} B_{22}) + (A_{22} B_{21} - A_{22} B_{21}) + (A_{12} B_{22} - A_{12} B_{22}) + A_{12} B_{21} (grouping cancelling pairs) = A_{11} B_{11} + A_{12} B_{21},$
which corresponds to equation (4.11).

Similarly, we set

C_{12} = P_{1} + P_{2},

and so $C_{12}$ equals

Verification of $C_{12}$ .
$C_{12} = P_{1} + P_{2} = (A_{11} B_{12} - A_{11} B_{22}) + (A_{11} B_{22} + A_{12} B_{22}) = A_{11} B_{12} + (A_{11} B_{22} - A_{11} B_{22}) + A_{12} B_{22} (cancelling the middle terms) = A_{11} B_{12} + A_{12} B_{22},$
corresponding to equation (4.12).

Setting

C_{21} = P_{3} + P_{4}

makes $C_{21}$ equal

Verification of $C_{21}$ .
$C_{21} = P_{3} + P_{4} = (A_{21} B_{11} + A_{22} B_{11}) + (A_{22} B_{21} - A_{22} B_{11}) = A_{21} B_{11} + (A_{22} B_{11} - A_{22} B_{11}) + A_{22} B_{21} (cancelling the middle terms) = A_{21} B_{11} + A_{22} B_{21},$
corresponding to equation (4.13).

Finally, we set

C_{22} = P_{5} + P_{1} - P_{3} - P_{7},

so that $C_{22}$ equals

Verification of $C_{22}$ .
$C_{22} = P_{5} + P_{1} - P_{3} - P_{7} = P_{5} A_{11} B_{11} + A_{11} B_{22} + A_{22} B_{11} + A_{22} B_{22} + P_{1} A_{11} B_{12} - A_{11} B_{22} - P_{3} - A_{21} B_{11} - A_{22} B_{11} - P_{7} - A_{11} B_{11} - A_{11} B_{12} + A_{21} B_{11} + A_{21} B_{12} = (A_{11} B_{11} - A_{11} B_{11}) + (A_{11} B_{22} - A_{11} B_{22}) + (A_{22} B_{11} - A_{22} B_{11}) + A_{22} B_{22} + (A_{11} B_{12} - A_{11} B_{12}) + (A_{21} B_{11} - A_{21} B_{11}) + A_{21} B_{12} (grouping cancelling pairs) = A_{22} B_{22} + A_{21} B_{12},$
which corresponds to equation (4.14), since $A_{21} B_{12} + A_{22} B_{22}$ is the same sum in a different order.

Altogether, we add or subtract $n /2 \times n /2$ matrices eight times in step 4 (three for $C_{11}$ , one for $C_{12}$ , one for $C_{21}$ , three for $C_{22}$ ), and so this step indeed takes $Θ (n^{2})$ time.

Thus, we see that Strassen’s algorithm, comprising steps 1-4, produces the correct matrix product and that recurrence (4.18) characterizes its running time. Since we shall see in Section 4.5 that this recurrence has the solution $T (n) = Θ (n^{l g 7})$ , Strassen’s method is asymptotically faster than the straightforward SQUARE-MATRIX-MULTIPLY procedure. The notes at the end of this chapter discuss some of the practical aspects of Strassen’s algorithm.

Exercises 4.2-1 to 4.2-7

Note: Although Exercises 4.2-3, 4.2-4, and 4.2-5 are about variants on Strassen’s algorithm, you should read Section 4.5 before trying to solve them.

4.2-1 Use Strassen’s algorithm to compute the matrix product

(1735) (6482) .

Show your work.

4.2-2 Write pseudocode for Strassen’s algorithm.

4.2-3 How would you modify Strassen’s algorithm to multiply $n \times n$ matrices in which $n$ is not an exact power of $2$ ? Show that the resulting algorithm runs in time $Θ (n^{l g 7})$ .

4.2-4 What is the largest $k$ such that if you can multiply $3 \times 3$ matrices using $k$ multiplications (not assuming commutativity of multiplication), then you can multiply $n \times n$ matrices in time $o (n^{l g 7})$ ? What would the running time of this algorithm be?

4.2-5 V. Pan has discovered a way of multiplying $68 \times 68$ matrices using $132, 464$ multiplications, a way of multiplying $70 \times 70$ matrices using $143, 640$ multiplications, and a way of multiplying $72 \times 72$ matrices using $155, 424$ multiplications. Which method yields the best asymptotic running time when used in a divide-and-conquer matrix-multiplication algorithm? How does it compare to Strassen’s algorithm?

4.2-6 How quickly can you multiply a $k n \times n$ matrix by an $n \times k n$ matrix, using Strassen’s algorithm as a subroutine? Answer the same question with the order of the input matrices reversed.

4.2-7 Show how to multiply the complex numbers $a + bi$ and $c + d i$ using only three multiplications of real numbers. The algorithm should take $a$ , $b$ , $c$ , and $d$ as input and produce the real component $a c - b d$ and the imaginary component $a d + b c$ separately.

4.3 The substitution method for solving recurrences

Big picture (for the next three sections). A recurrence is a riddle: “the answer for $n$ is built from the answer for smaller inputs.” Solving it means finding a closed form. The substitution method works like a detective: first you name a suspect (guess the answer), then you prove, by induction, that the suspect is guilty.

Now that we have seen how recurrences characterize the running times of divide-and-conquer algorithms, we will learn how to solve recurrences. We start in this section with the “substitution” method.

The substitution method for solving recurrences comprises two steps:

Guess the form of the solution.
Use mathematical induction to find the constants and show that the solution works.

We substitute the guessed solution for the function when applying the inductive hypothesis to smaller values; hence the name “substitution method.” This method is powerful, but we must be able to guess the form of the answer in order to apply it.

We can use the substitution method to establish either upper or lower bounds on a recurrence. As an example, let us determine an upper bound on the recurrence

T (n) = 2 T (⌊ n /2 ⌋) + n, (4.19)

which is similar to recurrences (4.3) and (4.4). We guess that the solution is $T (n) = O (n l g n)$ . The substitution method requires us to prove that $T (n) \leq c n l g n$ for an appropriate choice of the constant $c > 0$ . We start by assuming that this bound holds for all positive $m < n$ , in particular for $m = ⌊ n /2 ⌋$ , yielding $T (⌊ n /2 ⌋) \leq c ⌊ n /2 ⌋ l g (⌊ n /2 ⌋)$ . Substituting into the recurrence yields

Proof (inductive step for (4.19)).
$T (n) \leq 2 (c ⌊ n /2 ⌋ l g (⌊ n /2 ⌋)) + n \leq c n l g (n /2) + n = c n l g n - c n l g 2 + n = c n l g n - c n + n \leq c n l g n, (by the inductive hypothesis with m = ⌊ n /2 ⌋) (since 2 ⌊ n /2 ⌋ \leq n and ⌊ n /2 ⌋ \leq n /2) (since l g (n /2) = l g n - l g 2) (since l g 2 = 1)$
where the last step holds as long as $c \geq 1$ , because then $- c n + n = (1 - c) n \leq 0$ .

Mathematical induction now requires us to show that our solution holds for the boundary conditions. Typically, we do so by showing that the boundary conditions are suitable as base cases for the inductive proof. For the recurrence (4.19), we must show that we can choose the constant $c$ large enough so that the bound $T (n) \leq c n l g n$ works for the boundary conditions as well. This requirement can sometimes lead to problems. Let us assume, for the sake of argument, that $T (1) = 1$ is the sole boundary condition of the recurrence. Then for $n = 1$ , the bound $T (n) \leq c n l g n$ yields $T (1) \leq c \cdot 1 \cdot l g 1 = 0$ , which is at odds with $T (1) = 1$ . Consequently, the base case of our inductive proof fails to hold.

We can overcome this obstacle in proving an inductive hypothesis for a specific boundary condition with only a little more effort. In the recurrence (4.19), for example, we take advantage of asymptotic notation requiring us only to prove $T (n) \leq c n l g n$ for $n \geq n_{0}$ , where $n_{0}$ is a constant that we get to choose. We keep the troublesome boundary condition $T (1) = 1$ , but remove it from consideration in the inductive proof. We do so by first observing that for $n > 3$ , the recurrence does not depend directly on $T (1)$ . Thus, we can replace $T (1)$ by $T (2)$ and $T (3)$ as the base cases in the inductive proof, letting $n_{0} = 2$ . Note that we make a distinction between the base case of the recurrence ( $n = 1$ ) and the base cases of the inductive proof ( $n = 2$ and $n = 3$ ). With $T (1) = 1$ , we derive from the recurrence that $T (2) = 4$ and $T (3) = 5$ . Now we can complete the inductive proof that $T (n) \leq c n l g n$ for some constant $c \geq 1$ by choosing $c$ large enough so that $T (2) \leq c \cdot 2 l g 2$ and $T (3) \leq c \cdot 3 l g 3$ . As it turns out, any choice of $c \geq 2$ suffices for the base cases of $n = 2$ and $n = 3$ to hold.

Check of the base cases.
$T (2) T (3) c \cdot 2 l g 2 c \cdot 3 l g 3 = 2 T (⌊ 2/2 ⌋) + 2 = 2 T (1) + 2 = 2 \cdot 1 + 2 = 4, = 2 T (⌊ 3/2 ⌋) + 3 = 2 T (1) + 3 = 2 \cdot 1 + 3 = 5, = 2 c \geq 4 when c \geq 2, \geq 2 \cdot 3 \cdot 1.58 > 9 > 5 when c \geq 2.$

For most of the recurrences we shall examine, it is straightforward to extend boundary conditions to make the inductive assumption work for small $n$ , and we shall not always explicitly work out the details.

Making a good guess

Unfortunately, there is no general way to guess the correct solutions to recurrences. Guessing a solution takes experience and, occasionally, creativity. Fortunately, though, you can use some heuristics to help you become a good guesser. You can also use recursion trees, which we shall see in Section 4.4, to generate good guesses.

If a recurrence is similar to one you have seen before, then guessing a similar solution is reasonable. As an example, consider the recurrence

T (n) = 2 T (⌊ n /2 ⌋ + 17) + n,

which looks difficult because of the added " $17$ " in the argument to $T$ on the right-hand side. Intuitively, however, this additional term cannot substantially affect the solution to the recurrence. When $n$ is large, the difference between $T (⌊ n /2 ⌋)$ and $T (⌊ n /2 ⌋ + 17)$ is not that large: both cut $n$ nearly evenly in half. Consequently, we make the guess that $T (n) = O (n l g n)$ , which you can verify as correct by using the substitution method (see Exercise 4.3-6).

Another way to make a good guess is to prove loose upper and lower bounds on the recurrence and then reduce the range of uncertainty. For example, we might start with a lower bound of $T (n) = Ω (n)$ for the recurrence (4.19), since we have the term $n$ in the recurrence, and we can prove an initial upper bound of $T (n) = O (n^{2})$ . Then, we can gradually lower the upper bound and raise the lower bound until we converge on the correct, asymptotically tight solution of $T (n) = Θ (n l g n)$ .

Subtleties

Sometimes you might correctly guess an asymptotic bound on the solution of a recurrence, but somehow the math fails to work out in the induction. The problem frequently turns out to be that the inductive assumption is not strong enough to prove the detailed bound. If you revise the guess by subtracting a lower-order term when you hit such a snag, the math often goes through.

Consider the recurrence

T (n) = T (⌊ n /2 ⌋) + T (⌈ n /2 ⌉) + 1.

We guess that the solution is $T (n) = O (n)$ , and we try to show that $T (n) \leq c n$ for an appropriate choice of the constant $c$ . Substituting our guess in the recurrence, we obtain

T (n) \leq c ⌊ n /2 ⌋ + c ⌈ n /2 ⌉ + 1 = c n + 1,

which does not imply $T (n) \leq c n$ for any choice of $c$ . We might be tempted to try a larger guess, say $T (n) = O (n^{2})$ . Although we can make this larger guess work, our original guess of $T (n) = O (n)$ is correct. In order to show that it is correct, however, we must make a stronger inductive hypothesis.

Intuitively, our guess is nearly right: we are off only by the constant $1$ , a lower-order term. Nevertheless, mathematical induction does not work unless we prove the exact form of the inductive hypothesis. We overcome our difficulty by subtracting a lower-order term from our previous guess. Our new guess is $T (n) \leq c n - d$ , where $d \geq 0$ is a constant. We now have

Proof (strengthened hypothesis).
$T (n) \leq (c ⌊ n /2 ⌋ - d) + (c ⌈ n /2 ⌉ - d) + 1 = c n - 2 d + 1 \leq c n - d, (by the inductive hypothesis, applied to both terms) (since ⌊ n /2 ⌋ + ⌈ n /2 ⌉ = n)$
as long as $d \geq 1$ , because then $- 2 d + 1 \leq - d$ is equivalent to $1 \leq d$ .

As before, we must choose the constant $c$ large enough to handle the boundary conditions.

You might find the idea of subtracting a lower-order term counterintuitive. After all, if the math does not work out, we should increase our guess, right? Not necessarily! When proving an upper bound by induction, it may actually be more difficult to prove that a weaker upper bound holds, because in order to prove the weaker bound, we must use the same weaker bound inductively in the proof. In our current example, when the recurrence has more than one recursive term, we get to subtract out the lower-order term of the proposed bound once per recursive term. In the above example, we subtracted out the constant $d$ twice, once for the $T (⌊ n /2 ⌋)$ term and once for the $T (⌈ n /2 ⌉)$ term. We ended up with the inequality $T (n) \leq c n - 2 d + 1$ , and it was easy to find values of $d$ to make $c n - 2 d + 1$ be less than or equal to $c n - d$ .

Avoiding pitfalls

It is easy to err in the use of asymptotic notation. For example, in the recurrence (4.19) we can falsely “prove” $T (n) = O (n)$ by guessing $T (n) \leq c n$ and then arguing

T (n) \leq 2 (c ⌊ n /2 ⌋) + n \leq c n + n = O (n), \Leftarrow wrong!!

since $c$ is a constant. The error is that we have not proved the exact form of the inductive hypothesis, that is, that $T (n) \leq c n$ . We therefore will explicitly prove that $T (n) \leq c n$ when we want to show that $T (n) = O (n)$ .

Think about it. Why is " $c n + n = O (n)$ " no help here? Because the induction needs to land back on the same constant $c$ ; if each step quietly inflates it to $c + 1$ , then after $l g n$ levels the constant has grown without bound.

Changing variables

Sometimes, a little algebraic manipulation can make an unknown recurrence similar to one you have seen before. As an example, consider the recurrence

T (n) = 2 T (⌊ n ⌋) + l g n,

which looks difficult. We can simplify this recurrence, though, with a change of variables. For convenience, we shall not worry about rounding off values, such as $n$ , to be integers. Renaming $m = l g n$ yields

T (2^{m}) = 2 T (2^{m /2}) + m .

We can now rename $S (m) = T (2^{m})$ to produce the new recurrence

S (m) = 2 S (m /2) + m,

which is very much like recurrence (4.19). Indeed, this new recurrence has the same solution: $S (m) = O (m l g m)$ . Changing back from $S (m)$ to $T (n)$ , we obtain

T (n) = T (2^{m}) = S (m) = O (m l g m) = O (l g n l g l g n) (since m = l g n) .

Exercises 4.3-1 to 4.3-9

4.3-1 Show that the solution of $T (n) = T (n - 1) + n$ is $O (n^{2})$ .

4.3-2 Show that the solution of $T (n) = T (⌈ n /2 ⌉) + 1$ is $O (l g n)$ .

4.3-3 We saw that the solution of $T (n) = 2 T (⌊ n /2 ⌋) + n$ is $O (n l g n)$ . Show that the solution of this recurrence is also $Ω (n l g n)$ . Conclude that the solution is $Θ (n l g n)$ .

4.3-4 Show that by making a different inductive hypothesis, we can overcome the difficulty with the boundary condition $T (1) = 1$ for recurrence (4.19) without adjusting the boundary conditions for the inductive proof.

4.3-5 Show that $Θ (n l g n)$ is the solution to the “exact” recurrence (4.3) for merge sort.

4.3-6 Show that the solution to $T (n) = 2 T (⌊ n /2 ⌋ + 17) + n$ is $O (n l g n)$ .

4.3-7 Using the master method in Section 4.5, you can show that the solution to the recurrence $T (n) = 4 T (n /3) + n$ is $T (n) = Θ (n^{l o g_{3} 4})$ . Show that a substitution proof with the assumption $T (n) \leq c n^{l o g_{3} 4}$ fails. Then show how to subtract off a lower-order term to make a substitution proof work.

4.3-8 Using the master method in Section 4.5, you can show that the solution to the recurrence $T (n) = 4 T (n /2) + n$ is $T (n) = Θ (n^{2})$ . Show that a substitution proof with the assumption $T (n) \leq c n^{2}$ fails. Then show how to subtract off a lower-order term to make a substitution proof work.

4.3-9 Solve the recurrence $T (n) = 3 T (n) + lo g n$ by making a change of variables. Your solution should be asymptotically tight. Do not worry about whether values are integral.

4.4 The recursion-tree method for solving recurrences

Although you can use the substitution method to provide a succinct proof that a solution to a recurrence is correct, you might have trouble coming up with a good guess. Drawing out a recursion tree, as we did in our analysis of the merge sort recurrence in Section 2.3.2, serves as a straightforward way to devise a good guess. In a recursion tree, each node represents the cost of a single subproblem somewhere in the set of recursive function invocations. We sum the costs within each level of the tree to obtain a set of per-level costs, and then we sum all the per-level costs to determine the total cost of all levels of the recursion.

A recursion tree is best used to generate a good guess, which you can then verify by the substitution method. When using a recursion tree to generate a good guess, you can often tolerate a small amount of “sloppiness,” since you will be verifying your guess later on. If you are very careful when drawing out a recursion tree and summing the costs, however, you can use a recursion tree as a direct proof of a solution to a recurrence. In this section, we will use recursion trees to generate good guesses, and in Section 4.6, we will use recursion trees directly to prove the theorem that forms the basis of the master method.

Think about it. A recursion tree is like auditing a company’s payroll: instead of asking “what does the whole company cost?”, you ask “what does each management layer cost?” and then add up the layers.

Example: $T (n) = 3 T (⌊ n /4 ⌋) + Θ (n^{2})$

For example, let us see how a recursion tree would provide a good guess for the recurrence $T (n) = 3 T (⌊ n /4 ⌋) + Θ (n^{2})$ . We start by focusing on finding an upper bound for the solution. Because we know that floors and ceilings usually do not matter when solving recurrences (here is an example of sloppiness that we can tolerate), we create a recursion tree for the recurrence $T (n) = 3 T (n /4) + c n^{2}$ , having written out the implied constant coefficient $c > 0$ .

Figure 4.5 shows how we derive the recursion tree for $T (n) = 3 T (n /4) + c n^{2}$ . For convenience, we assume that $n$ is an exact power of $4$ (another example of tolerable sloppiness) so that all subproblem sizes are integers. Part (a) of the figure shows $T (n)$ , which we expand in part (b) into an equivalent tree representing the recurrence. The $c n^{2}$ term at the root represents the cost at the top level of recursion, and the three subtrees of the root represent the costs incurred by the subproblems of size $n /4$ . Part (c) shows this process carried one step further by expanding each node with cost $T (n /4)$ from part (b). The cost for each of the three children of the root is $c (n /4)^{2}$ . We continue expanding each node in the tree by breaking it into its constituent parts as determined by the recurrence.

Figure 4.5: Constructing a recursion tree for the recurrence $T (n) = 3 T (n /4) + c n^{2}$ . Part (a) shows $T (n)$ , which progressively expands in (b)-(d) to form the recursion tree. The fully expanded tree in part (d) has height $lo g_{4} n$ (it has $lo g_{4} n + 1$ levels).

The figure shows a tree whose root costs $c n^{2}$ , whose three children each cost $c (n /4)^{2}$ , and whose nine grandchildren each cost $c (n /16)^{2}$ , and so on down to $n^{l o g_{4} 3}$ leaves of cost $T (1)$ . The per-level totals are $c n^{2}$ , $(3/16) c n^{2}$ , $(3/16)^{2} c n^{2}, \dots$ , the leaf level totals $Θ (n^{l o g_{4} 3})$ , and the overall total is labeled $O (n^{2})$ .

Because subproblem sizes decrease by a factor of $4$ each time we go down one level, we eventually must reach a boundary condition. How far from the root do we reach one? The subproblem size for a node at depth $i$ is $n / 4^{i}$ . Thus, the subproblem size hits $n = 1$ when $n / 4^{i} = 1$ or, equivalently, when $i = lo g_{4} n$ . Thus, the tree has $lo g_{4} n + 1$ levels (at depths $0, 1, 2, \dots, lo g_{4} n$ ).

Next we determine the cost at each level of the tree. Each level has three times more nodes than the level above, and so the number of nodes at depth $i$ is $3^{i}$ . Because subproblem sizes reduce by a factor of $4$ for each level we go down from the root, each node at depth $i$ , for $i = 0, 1, 2, \dots, lo g_{4} n - 1$ , has a cost of $c (n / 4^{i})^{2}$ . Multiplying, we see that the total cost over all nodes at depth $i$ , for $i = 0, 1, 2, \dots, lo g_{4} n - 1$ , is

3^{i} c (\frac{n}{4 ^{i}})^{2} = (\frac{3}{16})^{i} c n^{2} .

The bottom level, at depth $lo g_{4} n$ , has $3^{l o g_{4} n} = n^{l o g_{4} 3}$ nodes, each contributing cost $T (1)$ , for a total cost of $n^{l o g_{4} 3} T (1)$ , which is $Θ (n^{l o g_{4} 3})$ , since we assume that $T (1)$ is a constant.

Now we add up the costs over all levels to determine the cost for the entire tree:

Example (exact sum of the tree).
$T (n) = c n^{2} + \frac{3}{16} c n^{2} + (\frac{3}{16})^{2} c n^{2} + \dots + (\frac{3}{16})^{l o g_{4} n - 1} c n^{2} + Θ (n^{l o g_{4} 3}) = i = 0 \sum l o g_{4} n - 1 (\frac{3}{16})^{i} c n^{2} + Θ (n^{l o g_{4} 3}) = \frac{( 3/16 ) ^{l o g_{4} n} - 1}{( 3/16 ) - 1} c n^{2} + Θ (n^{l o g_{4} 3}) (by equation (A.5), the finite geometric sum) .$

This last formula looks somewhat messy until we realize that we can again take advantage of small amounts of sloppiness and use an infinite decreasing geometric series as an upper bound. Backing up one step and applying equation (A.6), we have

Example (upper bound by an infinite series).
$T (n) = i = 0 \sum l o g_{4} n - 1 (\frac{3}{16})^{i} c n^{2} + Θ (n^{l o g_{4} 3}) < i = 0 \sum \infty (\frac{3}{16})^{i} c n^{2} + Θ (n^{l o g_{4} 3}) = \frac{1}{1 - ( 3/16 )} c n^{2} + Θ (n^{l o g_{4} 3}) = \frac{16}{13} c n^{2} + Θ (n^{l o g_{4} 3}) = O (n^{2}) (all added terms are positive) (by equation (A.6), since 3/16 < 1) (since lo g_{4} 3 < 1 < 2) .$

Thus, we have derived a guess of $T (n) = O (n^{2})$ for our original recurrence $T (n) = 3 T (⌊ n /4 ⌋) + Θ (n^{2})$ . In this example, the coefficients of $c n^{2}$ form a decreasing geometric series and, by equation (A.6), the sum of these coefficients is bounded from above by the constant $16/13$ . Since the root’s contribution to the total cost is $c n^{2}$ , the root contributes a constant fraction of the total cost. In other words, the cost of the root dominates the total cost of the tree.

In fact, if $O (n^{2})$ is indeed an upper bound for the recurrence (as we shall verify in a moment), then it must be a tight bound. Why? The first recursive call contributes a cost of $Θ (n^{2})$ , and so $Ω (n^{2})$ must be a lower bound for the recurrence.

Now we can use the substitution method to verify that our guess was correct, that is, $T (n) = O (n^{2})$ is an upper bound for the recurrence $T (n) = 3 T (⌊ n /4 ⌋) + Θ (n^{2})$ . We want to show that $T (n) \leq d n^{2}$ for some constant $d > 0$ . Using the same constant $c > 0$ as before, we have

Proof (substitution check).
$T (n) \leq 3 T (⌊ n /4 ⌋) + c n^{2} \leq 3 d ⌊ n /4 ⌋^{2} + c n^{2} \leq 3 d (n /4)^{2} + c n^{2} = \frac{3}{16} d n^{2} + c n^{2} \leq d n^{2}, (by the inductive hypothesis) (since ⌊ n /4 ⌋ \leq n /4)$
where the last step holds as long as $d \geq (16/13) c$ , because $\frac{3}{16} d + c \leq d$ is equivalent to $c \leq \frac{13}{16} d$ .

Example: $T (n) = T (n /3) + T (2 n /3) + O (n)$

In another, more intricate, example, Figure 4.6 shows the recursion tree for

T (n) = T (n /3) + T (2 n /3) + O (n) .

(Again, we omit floor and ceiling functions for simplicity.) As before, we let $c$ represent the constant factor in the $O (n)$ term. When we add the values across the levels of the recursion tree shown in the figure, we get a value of $c n$ for every level.

Figure 4.6: A recursion tree for the recurrence $T (n) = T (n /3) + T (2 n /3) + c n$ .

The tree has root cost $c n$ , children of cost $c (n /3)$ and $c (2 n /3)$ , grandchildren of cost $c (n /9)$ , $c (2 n /9)$ , $c (2 n /9)$ , $c (4 n /9)$ , and so on; every complete level sums to $c n$ , the height is $lo g_{3/2} n$ , and the total is labeled $O (n l g n)$ .

The longest simple path from the root to a leaf is $n \to (2/3) n \to (2/3)^{2} n \to \dots \to 1$ . Since $(2/3)^{k} n = 1$ when $k = lo g_{3/2} n$ , the height of the tree is $lo g_{3/2} n$ .

Intuitively, we expect the solution to the recurrence to be at most the number of levels times the cost of each level, or $O (c n lo g_{3/2} n) = O (n l g n)$ . Figure 4.6 shows only the top levels of the recursion tree, however, and not every level in the tree contributes a cost of $c n$ . Consider the cost of the leaves. If this recursion tree were a complete binary tree of height $lo g_{3/2} n$ , there would be $2^{l o g_{3/2} n} = n^{l o g_{3/2} 2}$ leaves. Since the cost of each leaf is a constant, the total cost of all leaves would then be $Θ (n^{l o g_{3/2} 2})$ which, since $lo g_{3/2} 2$ is a constant strictly greater than $1$ , is $ω (n l g n)$ . This recursion tree is not a complete binary tree, however, and so it has fewer than $n^{l o g_{3/2} 2}$ leaves. Moreover, as we go down from the root, more and more internal nodes are absent. Consequently, levels toward the bottom of the recursion tree contribute less than $c n$ to the total cost. We could work out an accurate accounting of all costs, but remember that we are just trying to come up with a guess to use in the substitution method. Let us tolerate the sloppiness and attempt to show that a guess of $O (n l g n)$ for the upper bound is correct.

Indeed, we can use the substitution method to verify that $O (n l g n)$ is an upper bound for the solution to the recurrence. We show that $T (n) \leq d n l g n$ , where $d$ is a suitable positive constant. We have

Proof (substitution check).
$T (n) \leq T (n /3) + T (2 n /3) + c n \leq d (n /3) l g (n /3) + d (2 n /3) l g (2 n /3) + c n = (d (n /3) l g n - d (n /3) l g 3) + (d (2 n /3) l g n - d (2 n /3) l g (3/2)) + c n = d n l g n - d ((n /3) l g 3 + (2 n /3) l g (3/2)) + c n = d n l g n - d ((n /3) l g 3 + (2 n /3) l g 3 - (2 n /3) l g 2) + c n = d n l g n - d n (l g 3 - 2/3) + c n \leq d n l g n, (by the inductive hypothesis) (splitting each logarithm) (since \frac{n}{3} + \frac{2 n}{3} = n) (since l g (3/2) = l g 3 - l g 2) (since l g 2 = 1 and \frac{n}{3} + \frac{2 n}{3} = n)$
as long as $d \geq c / (l g 3 - 2/3)$ ; note that $l g 3 - 2/3 > 0$ , so such a $d$ exists, and then $- d n (l g 3 - 2/3) + c n \leq 0$ .

Thus, we did not need to perform a more accurate accounting of costs in the recursion tree.

Exercises 4.4-1 to 4.4-9

4.4-1 Use a recursion tree to determine a good asymptotic upper bound on the recurrence $T (n) = 3 T (⌊ n /2 ⌋) + n$ . Use the substitution method to verify your answer.

4.4-2 Use a recursion tree to determine a good asymptotic upper bound on the recurrence $T (n) = T (n /2) + n^{2}$ . Use the substitution method to verify your answer.

4.4-3 Use a recursion tree to determine a good asymptotic upper bound on the recurrence $T (n) = 4 T (n /2 + 2) + n$ . Use the substitution method to verify your answer.

4.4-4 Use a recursion tree to determine a good asymptotic upper bound on the recurrence $T (n) = 2 T (n - 1) + 1$ . Use the substitution method to verify your answer.

4.4-5 Use a recursion tree to determine a good asymptotic upper bound on the recurrence $T (n) = T (n - 1) + T (n /2) + n$ . Use the substitution method to verify your answer.

4.4-6 Argue that the solution to the recurrence $T (n) = T (n /3) + T (2 n /3) + c n$ , where $c$ is a constant, is $Ω (n l g n)$ by appealing to a recursion tree.

4.4-7 Draw the recursion tree for $T (n) = 4 T (⌊ n /2 ⌋) + c n$ , where $c$ is a constant, and provide a tight asymptotic bound on its solution. Verify your bound by the substitution method.

4.4-8 Use a recursion tree to give an asymptotically tight solution to the recurrence $T (n) = T (n - a) + T (a) + c n$ , where $a \geq 1$ and $c > 0$ are constants.

4.4-9 Use a recursion tree to give an asymptotically tight solution to the recurrence $T (n) = T (α n) + T ((1 - α) n) + c n$ , where $α$ is a constant in the range $0 < α < 1$ , and $c > 0$ is also a constant.

4.5 The master method for solving recurrences

Big picture. Substitution asks you to be a detective and recursion trees ask you to be an accountant. The master method asks only that you be a good reader of recipes: compare two functions, pick one of three cases, and read off the answer.

The master method provides a “cookbook” method for solving recurrences of the form

T (n) = a T (n / b) + f (n), (4.20)

where $a \geq 1$ and $b > 1$ are constants and $f (n)$ is an asymptotically positive function. To use the master method, you will need to memorize three cases, but then you will be able to solve many recurrences quite easily, often without pencil and paper.

The recurrence (4.20) describes the running time of an algorithm that divides a problem of size $n$ into $a$ subproblems, each of size $n / b$ , where $a$ and $b$ are positive constants. The $a$ subproblems are solved recursively, each in time $T (n / b)$ . The function $f (n)$ encompasses the cost of dividing the problem and combining the results of the subproblems. For example, the recurrence arising from Strassen’s algorithm has $a = 7$ , $b = 2$ , and $f (n) = Θ (n^{2})$ .

As a matter of technical correctness, the recurrence is not actually well defined, because $n / b$ might not be an integer. Replacing each of the $a$ terms $T (n / b)$ with either $T (⌊ n / b ⌋)$ or $T (⌈ n / b ⌉)$ will not affect the asymptotic behavior of the recurrence, however. (We will prove this assertion in the next section.) We normally find it convenient, therefore, to omit the floor and ceiling functions when writing divide-and-conquer recurrences of this form.

The master theorem

The master method depends on the following theorem.

Theorem 4.1 (Master theorem). Let $a \geq 1$ and $b > 1$ be constants, let $f (n)$ be a function, and let $T (n)$ be defined on the nonnegative integers by the recurrence

T (n) = a T (n / b) + f (n),

where we interpret $n / b$ to mean either $⌊ n / b ⌋$ or $⌈ n / b ⌉$ . Then $T (n)$ has the following asymptotic bounds:

If $f (n) = O (n^{l o g_{b} a - ϵ})$ for some constant $ϵ > 0$ , then $T (n) = Θ (n^{l o g_{b} a})$ .
If $f (n) = Θ (n^{l o g_{b} a})$ , then $T (n) = Θ (n^{l o g_{b} a} l g n)$ .
If $f (n) = Ω (n^{l o g_{b} a + ϵ})$ for some constant $ϵ > 0$ , and if $a f (n / b) \leq c f (n)$ for some constant $c < 1$ and all sufficiently large $n$ , then $T (n) = Θ (f (n))$ .

Before applying the master theorem to some examples, let us spend a moment trying to understand what it says. In each of the three cases, we compare the function $f (n)$ with the function $n^{l o g_{b} a}$ . Intuitively, the larger of the two functions determines the solution to the recurrence. If, as in case 1, the function $n^{l o g_{b} a}$ is the larger, then the solution is $T (n) = Θ (n^{l o g_{b} a})$ . If, as in case 3, the function $f (n)$ is the larger, then the solution is $T (n) = Θ (f (n))$ . If, as in case 2, the two functions are the same size, we multiply by a logarithmic factor, and the solution is $T (n) = Θ (n^{l o g_{b} a} l g n) = Θ (f (n) l g n)$ .

Beyond this intuition, you need to be aware of some technicalities. In the first case, not only must $f (n)$ be smaller than $n^{l o g_{b} a}$ , it must be polynomially smaller. That is, $f (n)$ must be asymptotically smaller than $n^{l o g_{b} a}$ by a factor of $n^{ϵ}$ for some constant $ϵ > 0$ . In the third case, not only must $f (n)$ be larger than $n^{l o g_{b} a}$ , it also must be polynomially larger and in addition satisfy the “regularity” condition that $a f (n / b) \leq c f (n)$ . This condition is satisfied by most of the polynomially bounded functions that we shall encounter.

Note that the three cases do not cover all the possibilities for $f (n)$ . There is a gap between cases 1 and 2 when $f (n)$ is smaller than $n^{l o g_{b} a}$ but not polynomially smaller. Similarly, there is a gap between cases 2 and 3 when $f (n)$ is larger than $n^{l o g_{b} a}$ but not polynomially larger. If the function $f (n)$ falls into one of these gaps, or if the regularity condition in case 3 fails to hold, you cannot use the master method to solve the recurrence.

Using the master method

To use the master method, we simply determine which case (if any) of the master theorem applies and write down the answer.

Example 1. As a first example, consider

T (n) = 9 T (n /3) + n .

Solution. For this recurrence, we have $a = 9$ , $b = 3$ , $f (n) = n$ , and thus we have that
$n^{l o g_{b} a} = n^{l o g_{3} 9} = Θ (n^{2}) .$
Since $f (n) = n = O (n^{l o g_{3} 9 - ϵ})$ , where $ϵ = 1$ , we can apply case 1 of the master theorem and conclude that the solution is $T (n) = Θ (n^{2})$ .

Example 2. Now consider

T (n) = T (2 n /3) + 1.

Solution. Here $a = 1$ , $b = 3/2$ , $f (n) = 1$ , and
$n^{l o g_{b} a} = n^{l o g_{3/2} 1} = n^{0} = 1.$
Case 2 applies, since $f (n) = Θ (n^{l o g_{b} a}) = Θ (1)$ , and thus the solution to the recurrence is $T (n) = Θ (l g n)$ .

Example 3. For the recurrence

T (n) = 3 T (n /4) + n l g n,

Solution. We have $a = 3$ , $b = 4$ , $f (n) = n l g n$ , and $n^{l o g_{b} a} = n^{l o g_{4} 3} = O (n^{0.793})$ . Since $f (n) = Ω (n^{l o g_{4} 3 + ϵ})$ , where $ϵ \approx 0.2$ , case 3 applies if we can show that the regularity condition holds for $f (n)$ . For sufficiently large $n$ , we have that
$a f (n / b) = 3 (n /4) l g (n /4) \leq (3/4) n l g n = c f (n) (since l g (n /4) \leq l g n) for c = 3/4.$
Consequently, by case 3, the solution to the recurrence is $T (n) = Θ (n l g n)$ .

Example 4 (a recurrence the master method cannot solve). The master method does not apply to the recurrence

T (n) = 2 T (n /2) + n l g n,

even though it has the proper form: $a = 2$ , $b = 2$ , $f (n) = n l g n$ , and $n^{l o g_{b} a} = n$ . You might mistakenly think that case 3 should apply, since $f (n) = n l g n$ is asymptotically larger than $n^{l o g_{b} a} = n$ . The problem is that it is not polynomially larger. The ratio $f (n) / n^{l o g_{b} a} = (n l g n) / n = l g n$ is asymptotically less than $n^{ϵ}$ for any positive constant $ϵ$ . Consequently, the recurrence falls into the gap between case 2 and case 3. (See Exercise 4.6-2 for a solution.)

Think about it. Why is $l g n$ “not polynomially larger”? Because it grows slower than $n^{0.0001}$ , which is the smallest polynomial growth we are willing to count.

The earlier recurrences. Let us use the master method to solve the recurrences we saw in Sections 4.1 and 4.2.

Recurrence (4.7),

T (n) = 2 T (n /2) + Θ (n),

characterizes the running times of the divide-and-conquer algorithm for both the maximum-subarray problem and merge sort. (As is our practice, we omit stating the base case in the recurrence.) Here, we have $a = 2$ , $b = 2$ , $f (n) = Θ (n)$ , and thus we have that $n^{l o g_{b} a} = n^{l o g_{2} 2} = n$ . Case 2 applies, since $f (n) = Θ (n)$ , and so we have the solution $T (n) = Θ (n l g n)$ .

Recurrence (4.17),

T (n) = 8 T (n /2) + Θ (n^{2}),

describes the running time of the first divide-and-conquer algorithm that we saw for matrix multiplication. Now we have $a = 8$ , $b = 2$ , and $f (n) = Θ (n^{2})$ , and so $n^{l o g_{b} a} = n^{l o g_{2} 8} = n^{3}$ . Since $n^{3}$ is polynomially larger than $f (n)$ (that is, $f (n) = O (n^{3 - ϵ})$ for $ϵ = 1$ ), case 1 applies, and $T (n) = Θ (n^{3})$ .

Finally, consider recurrence (4.18),

T (n) = 7 T (n /2) + Θ (n^{2}),

which describes the running time of Strassen’s algorithm. Here, we have $a = 7$ , $b = 2$ , $f (n) = Θ (n^{2})$ , and thus $n^{l o g_{b} a} = n^{l o g_{2} 7}$ . Rewriting $lo g_{2} 7$ as $l g 7$ and recalling that $2.80 < l g 7 < 2.81$ , we see that $f (n) = O (n^{l g 7 - ϵ})$ for $ϵ = 0.8$ . Again, case 1 applies, and we have the solution $T (n) = Θ (n^{l g 7})$ .

Exercises 4.5-1 to 4.5-5

4.5-1 Use the master method to give tight asymptotic bounds for the following recurrences.

a. $T (n) = 2 T (n /4) + 1$ .

b. $T (n) = 2 T (n /4) + n$ .

c. $T (n) = 2 T (n /4) + n$ .

d. $T (n) = 2 T (n /4) + n^{2}$ .

4.5-2 Professor Caesar wishes to develop a matrix-multiplication algorithm that is asymptotically faster than Strassen’s algorithm. His algorithm will use the divide-and-conquer method, dividing each matrix into pieces of size $n /4 \times n /4$ , and the divide and combine steps together will take $Θ (n^{2})$ time. He needs to determine how many subproblems his algorithm has to create in order to beat Strassen’s algorithm. If his algorithm creates $a$ subproblems, then the recurrence for the running time $T (n)$ becomes $T (n) = a T (n /4) + Θ (n^{2})$ . What is the largest integer value of $a$ for which Professor Caesar’s algorithm would be asymptotically faster than Strassen’s algorithm?

4.5-3 Use the master method to show that the solution to the binary-search recurrence $T (n) = T (n /2) + Θ (1)$ is $T (n) = Θ (l g n)$ . (See Exercise 2.3-5 for a description of binary search.)

4.5-4 Can the master method be applied to the recurrence $T (n) = 4 T (n /2) + n^{2} l g n$ ? Why or why not? Give an asymptotic upper bound for this recurrence.

4.5-5 $⋆$ Consider the regularity condition $a f (n / b) \leq c f (n)$ for some constant $c < 1$ , which is part of case 3 of the master theorem. Give an example of constants $a \geq 1$ and $b > 1$ and a function $f (n)$ that satisfies all the conditions in case 3 of the master theorem except the regularity condition.

★ 4.6 Proof of the master theorem

This section contains a proof of the master theorem (Theorem 4.1). You do not need to understand the proof in order to apply the master theorem.

The proof appears in two parts. The first part analyzes the master recurrence (4.20), under the simplifying assumption that $T (n)$ is defined only on exact powers of $b > 1$ , that is, for $n = 1, b, b^{2}, \dots$ . This part gives all the intuition needed to understand why the master theorem is true. The second part shows how to extend the analysis to all positive integers $n$ ; it applies mathematical technique to the problem of handling floors and ceilings.

In this section, we shall sometimes abuse our asymptotic notation slightly by using it to describe the behavior of functions that are defined only over exact powers of $b$ . Recall that the definitions of asymptotic notations require that bounds be proved for all sufficiently large numbers, not just those that are powers of $b$ . Since we could make new asymptotic notations that apply only to the set ${b^{i} : i = 0, 1, 2, \dots}$ , instead of to the nonnegative numbers, this abuse is minor.

Nevertheless, we must always be on guard when we use asymptotic notation over a limited domain lest we draw improper conclusions. For example, proving that $T (n) = O (n)$ when $n$ is an exact power of $2$ does not guarantee that $T (n) = O (n)$ . The function $T (n)$ could be defined as

T (n) = {n n^{2} if n = 1, 2, 4, 8, \dots, otherwise,

in which case the best upper bound that applies to all values of $n$ is $T (n) = O (n^{2})$ . Because of this sort of drastic consequence, we shall never use asymptotic notation over a limited domain without making it absolutely clear from the context that we are doing so.

4.6.1 The proof for exact powers

The first part of the proof of the master theorem analyzes the recurrence (4.20)

T (n) = a T (n / b) + f (n)

for the master method, under the assumption that $n$ is an exact power of $b > 1$ , where $b$ need not be an integer. We break the analysis into three lemmas. The first reduces the problem of solving the master recurrence to the problem of evaluating an expression that contains a summation. The second determines bounds on this summation. The third lemma puts the first two together to prove a version of the master theorem for the case in which $n$ is an exact power of $b$ .

Lemma 4.2. Let $a \geq 1$ and $b > 1$ be constants, and let $f (n)$ be a nonnegative function defined on exact powers of $b$ . Define $T (n)$ on exact powers of $b$ by the recurrence

T (n) = {Θ (1) a T (n / b) + f (n) if n = 1, if n = b^{i},

where $i$ is a positive integer. Then

T (n) = Θ (n^{l o g_{b} a}) + j = 0 \sum l o g_{b} n - 1 a^{j} f (n / b^{j}) . (4.21)

Proof. We use the recursion tree in Figure 4.7. The root of the tree has cost $f (n)$ , and it has $a$ children, each with cost $f (n / b)$ . (It is convenient to think of $a$ as being an integer, especially when visualizing the recursion tree, but the mathematics does not require it.) Each of these children has $a$ children, making $a^{2}$ nodes at depth $2$ , and each of the $a$ children has cost $f (n / b^{2})$ . In general, there are $a^{j}$ nodes at depth $j$ , and each has cost $f (n / b^{j})$ . The cost of each leaf is $T (1) = Θ (1)$ , and each leaf is at depth $lo g_{b} n$ , since $n / b^{l o g_{b} n} = 1$ . There are $a^{l o g_{b} n} = n^{l o g_{b} a}$ leaves in the tree.

The identity for the number of leaves holds because
$a^{l o g_{b} n} = (b^{l o g_{b} a})^{l o g_{b} n} = (b^{l o g_{b} n})^{l o g_{b} a} = n^{l o g_{b} a} (since a = b^{l o g_{b} a}) (exponents multiply in either order) (since b^{l o g_{b} n} = n) .$
We can obtain equation (4.21) by summing the costs of the nodes at each depth in the tree, as shown in the figure. The cost for all internal nodes at depth $j$ is $a^{j} f (n / b^{j})$ , and so the total cost of all internal nodes is
$j = 0 \sum l o g_{b} n - 1 a^{j} f (n / b^{j}) .$
In the underlying divide-and-conquer algorithm, this sum represents the costs of dividing problems into subproblems and then recombining the subproblems. The cost of all the leaves, which is the cost of doing all $n^{l o g_{b} a}$ subproblems of size $1$ , is $Θ (n^{l o g_{b} a})$ . Adding the internal cost to the leaf cost gives equation (4.21). $■$

Figure 4.7: The recursion tree generated by $T (n) = a T (n / b) + f (n)$ . The tree is a complete $a$ -ary tree with $n^{l o g_{b} a}$ leaves and height $lo g_{b} n$ . The cost of each level is shown at the right, and their sum is given in equation (4.21).

The tree has root cost $f (n)$ , level 1 total $a f (n / b)$ , level 2 total $a^{2} f (n / b^{2})$ , and so on, down to $n^{l o g_{b} a}$ leaves each of cost $Θ (1)$ ; the total is labeled $Θ (n^{l o g_{b} a}) + \sum_{j = 0}^{l o g_{b} n - 1} a^{j} f (n / b^{j})$ .

In terms of the recursion tree, the three cases of the master theorem correspond to cases in which the total cost of the tree is (1) dominated by the costs in the leaves, (2) evenly distributed among the levels of the tree, or (3) dominated by the cost of the root.

The summation in equation (4.21) describes the cost of the dividing and combining steps in the underlying divide-and-conquer algorithm. The next lemma provides asymptotic bounds on the summation’s growth.

Lemma 4.3. Let $a \geq 1$ and $b > 1$ be constants, and let $f (n)$ be a nonnegative function defined on exact powers of $b$ . A function $g (n)$ defined over exact powers of $b$ by

g (n) = j = 0 \sum l o g_{b} n - 1 a^{j} f (n / b^{j}) (4.22)

has the following asymptotic bounds for exact powers of $b$ :

If $f (n) = O (n^{l o g_{b} a - ϵ})$ for some constant $ϵ > 0$ , then $g (n) = O (n^{l o g_{b} a})$ .
If $f (n) = Θ (n^{l o g_{b} a})$ , then $g (n) = Θ (n^{l o g_{b} a} l g n)$ .
If $a f (n / b) \leq c f (n)$ for some constant $c < 1$ and for all sufficiently large $n$ , then $g (n) = Θ (f (n))$ .

We can now prove a version of the master theorem for the case in which $n$ is an exact power of $b$ .

Lemma 4.4. Let $a \geq 1$ and $b > 1$ be constants, and let $f (n)$ be a nonnegative function defined on exact powers of $b$ . Define $T (n)$ on exact powers of $b$ by the recurrence

T (n) = {Θ (1) a T (n / b) + f (n) if n = 1, if n = b^{i},

where $i$ is a positive integer. Then $T (n)$ has the following asymptotic bounds for exact powers of $b$ :

If $f (n) = O (n^{l o g_{b} a - ϵ})$ for some constant $ϵ > 0$ , then $T (n) = Θ (n^{l o g_{b} a})$ .
If $f (n) = Θ (n^{l o g_{b} a})$ , then $T (n) = Θ (n^{l o g_{b} a} l g n)$ .
If $f (n) = Ω (n^{l o g_{b} a + ϵ})$ for some constant $ϵ > 0$ , and if $a f (n / b) \leq c f (n)$ for some constant $c < 1$ and all sufficiently large $n$ , then $T (n) = Θ (f (n))$ .

Proof. We use the bounds in Lemma 4.3 to evaluate the summation (4.21) from Lemma 4.2, namely
$T (n) = Θ (n^{l o g_{b} a}) + g (n) .$
Case 1. Lemma 4.3 case 1 gives $g (n) = O (n^{l o g_{b} a})$ , so
$T (n) = Θ (n^{l o g_{b} a}) + O (n^{l o g_{b} a}) = Θ (n^{l o g_{b} a}) .$
Case 2. Lemma 4.3 case 2 gives $g (n) = Θ (n^{l o g_{b} a} l g n)$ , so
$T (n) = Θ (n^{l o g_{b} a}) + Θ (n^{l o g_{b} a} l g n) = Θ (n^{l o g_{b} a} l g n) .$
Case 3. Lemma 4.3 case 3 gives $g (n) = Θ (f (n))$ , so
$T (n) = Θ (n^{l o g_{b} a}) + Θ (f (n)) = Θ (f (n)),$
because $f (n) = Ω (n^{l o g_{b} a + ϵ})$ , so the leaf term $Θ (n^{l o g_{b} a})$ is absorbed into $Θ (f (n))$ . $■$

4.6.2 Floors and ceilings

To complete the proof of the master theorem, we must now extend our analysis to the situation in which floors and ceilings appear in the master recurrence, so that the recurrence is defined for all integers, not for just exact powers of $b$ . Obtaining a lower bound on

T (n) = a T (⌈ n / b ⌉) + f (n) (4.25)

and an upper bound on

T (n) = a T (⌊ n / b ⌋) + f (n) (4.26)

is routine, since we can push through the bound $⌈ n / b ⌉ \geq n / b$ in the first case to yield the desired result, and we can push through the bound $⌊ n / b ⌋ \leq n / b$ in the second case. We use much the same technique to lower-bound the recurrence (4.26) as to upper-bound the recurrence (4.25), and so we shall present only this latter bound.

We modify the recursion tree of Figure 4.7 to produce the recursion tree in Figure 4.8. As we go down in the recursion tree, we obtain a sequence of recursive invocations on the arguments

n, ⌈ n / b ⌉, ⌈⌈ n / b ⌉ / b ⌉, ⌈⌈⌈ n / b ⌉ / b ⌉ / b ⌉, \dots

Let us denote the $j$ th element in the sequence by $n_{j}$ , where

n_{j} = {n ⌈ n_{j - 1} / b ⌉ if j = 0, if j > 0. (4.27)

Figure 4.8: The recursion tree generated by $T (n) = a T (⌈ n / b ⌉) + f (n)$ . The recursive argument $n_{j}$ is given by equation (4.27).

The tree has root cost $f (n)$ , level 1 total $a f (n_{1})$ , level 2 total $a^{2} f (n_{2})$ , and so on, to a depth of $⌊ lo g_{b} n ⌋$ , with $Θ (n^{l o g_{b} a})$ leaves of cost $Θ (1)$ each; the total is labeled $Θ (n^{l o g_{b} a}) + \sum_{j = 0}^{⌊ l o g_{b} n ⌋ - 1} a^{j} f (n_{j})$ .

Our first goal is to determine the depth $k$ such that $n_{k}$ is a constant. Using the inequality $⌈ x ⌉ \leq x + 1$ , we obtain

n_{0} n_{1} n_{2} n_{3} \leq n, \leq \frac{n}{b} + 1, \leq \frac{n}{b ^{2}} + \frac{1}{b} + 1, \leq \frac{n}{b ^{3}} + \frac{1}{b ^{2}} + \frac{1}{b} + 1, ⋮

In general, we have

n_{j} \leq \frac{n}{b ^{j}} + i = 0 \sum j - 1 \frac{1}{b ^{i}} < \frac{n}{b ^{j}} + i = 0 \sum \infty \frac{1}{b ^{i}} = \frac{n}{b ^{j}} + \frac{b}{b - 1} .

Letting $j = ⌊ lo g_{b} n ⌋$ , we obtain

n_{⌊ l o g_{b} n ⌋} < \frac{n}{b ^{⌊ l o g_{b} n ⌋}} + \frac{b}{b - 1} \leq \frac{n}{b ^{l o g_{b} n - 1}} + \frac{b}{b - 1} = \frac{n}{n / b} + \frac{b}{b - 1} = b + \frac{b}{b - 1} = O (1), (since ⌊ x ⌋ > x - 1)

and thus we see that at depth $⌊ lo g_{b} n ⌋$ , the problem size is at most a constant.

From Figure 4.8, we see that

T (n) = Θ (n^{l o g_{b} a}) + j = 0 \sum ⌊ l o g_{b} n ⌋ - 1 a^{j} f (n_{j}), (4.28)

which is much the same as equation (4.21), except that $n$ is an arbitrary integer and not restricted to be an exact power of $b$ .

We can now evaluate the summation

g (n) = j = 0 \sum ⌊ l o g_{b} n ⌋ - 1 a^{j} f (n_{j}) (4.29)

from equation (4.28) in a manner analogous to the proof of Lemma 4.3.

Case 3. If $a f (⌈ n / b ⌉) \leq c f (n)$ for $n > b + b / (b - 1)$ , where $c < 1$ is a constant, then it follows that $a^{j} f (n_{j}) \leq c^{j} f (n)$ . Therefore, we can evaluate the sum in equation (4.29) just as in Lemma 4.3.

Case 2. We have $f (n) = Θ (n^{l o g_{b} a})$ . If we can show that $f (n_{j}) = O (n^{l o g_{b} a} / a^{j}) = O ((n / b^{j})^{l o g_{b} a})$ , then the proof for case 2 of Lemma 4.3 will go through. Observe that $j \leq ⌊ lo g_{b} n ⌋$ implies $b^{j} / n \leq 1$ . The bound $f (n) = O (n^{l o g_{b} a})$ implies that there exists a constant $c > 0$ such that for all sufficiently large $n_{j}$ ,
$f (n_{j}) \leq c (\frac{n}{b ^{j}} + \frac{b}{b - 1})^{l o g_{b} a} = c (\frac{n}{b ^{j}})^{l o g_{b} a} (1 + \frac{b ^{j}}{n} \cdot \frac{b}{b - 1})^{l o g_{b} a} \leq c (\frac{n}{b ^{j}})^{l o g_{b} a} (1 + \frac{b}{b - 1})^{l o g_{b} a} = O (\frac{n ^{l o g_{b} a}}{a ^{j}}), (since b^{j} / n \leq 1)$
since $c (1 + b / (b - 1))^{l o g_{b} a}$ is a constant, and $(n / b^{j})^{l o g_{b} a} = n^{l o g_{b} a} / a^{j}$ . Thus, we have proved case 2.

Case 1. The proof is almost identical. The key is to prove the bound $f (n_{j}) = O ((n / b^{j})^{l o g_{b} a - ϵ})$ , which is similar to the corresponding proof of case 2, though the algebra is more intricate.

We have now proved the upper bounds in the master theorem for all integers $n$ . The proof of the lower bounds is similar.

Exercises 4.6-1

$⋆$ Give a simple and exact expression for $n_{j}$ in equation (4.27) for the case in which $b$ is a positive integer instead of an arbitrary real number.

Exercises 4.6-2

$⋆$ Show that if $f (n) = Θ (n^{l o g_{b} a} l g^{k} n)$ , where $k \geq 0$ , then the master recurrence has solution $T (n) = Θ (n^{l o g_{b} a} l g^{k + 1} n)$ . For simplicity, confine your analysis to exact powers of $b$ .

Exercises 4.6-3

$⋆$ Show that case 3 of the master theorem is overstated, in the sense that the regularity condition $a f (n / b) \leq c f (n)$ for some constant $c < 1$ implies that there exists a constant $ϵ > 0$ such that $f (n) = Ω (n^{l o g_{b} a + ϵ})$ .

Problems

4-1 Recurrence examples

Give asymptotic upper and lower bounds for $T (n)$ in each of the following recurrences. Assume that $T (n)$ is constant for $n \leq 2$ . Make your bounds as tight as possible, and justify your answers.

a. $T (n) = 2 T (n /2) + n^{4}$ .

b. $T (n) = T (7 n /10) + n$ .

c. $T (n) = 16 T (n /4) + n^{2}$ .

d. $T (n) = 7 T (n /3) + n^{2}$ .

e. $T (n) = 7 T (n /2) + n^{2}$ .

f. $T (n) = 2 T (n /4) + n$ .

g. $T (n) = T (n - 2) + n^{2}$ .

4-2 Parameter-passing costs

Throughout this book, we assume that parameter passing during procedure calls takes constant time, even if an $N$ -element array is being passed. This assumption is valid in most systems because a pointer to the array is passed, not the array itself. This problem examines the implications of three parameter-passing strategies:

An array is passed by pointer. Time $= Θ (1)$ .
An array is passed by copying. Time $= Θ (N)$ , where $N$ is the size of the array.
An array is passed by copying only the subrange that might be accessed by the called procedure. Time $= Θ (q - p + 1)$ if the subarray $A [p .. q]$ is passed.

a. Consider the recursive binary search algorithm for finding a number in a sorted array (see Exercise 2.3-5). Give recurrences for the worst-case running times of binary search when arrays are passed using each of the three methods above, and give good upper bounds on the solutions of the recurrences. Let $N$ be the size of the original problem and $n$ be the size of a subproblem.

b. Redo part (a) for the MERGE-SORT algorithm from Section 2.3.1.

4-3 More recurrence examples

Give asymptotic upper and lower bounds for $T (n)$ in each of the following recurrences. Assume that $T (n)$ is constant for sufficiently small $n$ . Make your bounds as tight as possible, and justify your answers.

a. $T (n) = 4 T (n /3) + n l g n$ .

b. $T (n) = 3 T (n /3) + n / l g n$ .

c. $T (n) = 4 T (n /2) + n^{2} n$ .

d. $T (n) = 3 T (n /3 - 2) + n /2$ .

e. $T (n) = 2 T (n /2) + n / l g n$ .

f. $T (n) = T (n /2) + T (n /4) + T (n /8) + n$ .

g. $T (n) = T (n - 1) + 1/ n$ .

h. $T (n) = T (n - 1) + l g n$ .

i. $T (n) = T (n - 2) + 1/ l g n$ .

j. $T (n) = n T (n) + n$ .

4-4 Fibonacci numbers

This problem develops properties of the Fibonacci numbers, which are defined by recurrence (3.22). We shall use the technique of generating functions to solve the Fibonacci recurrence. Define the generating function (or formal power series) $F$ as

F (z) = i = 0 \sum \infty F_{i} z^{i} = 0 + z + z^{2} + 2 z^{3} + 3 z^{4} + 5 z^{5} + 8 z^{6} + 13 z^{7} + 21 z^{8} + \dots,

where $F_{i}$ is the $i$ th Fibonacci number.

a. Show that $F (z) = z + z F (z) + z^{2} F (z)$ .

b. Show that

F (z) = \frac{z}{1 - z - z ^{2}} = \frac{z}{( 1 - ϕ z ) ( 1 - ϕ ^ z )} = \frac{1}{5} (\frac{1}{1 - ϕ z} - \frac{1}{1 - ϕ ^ z}),

where

ϕ = \frac{1 + 5}{2} = 1.61803 \dots

and

\hat{ϕ} = \frac{1 - 5}{2} = - 0.61803 \dots .

c. Show that

F (z) = i = 0 \sum \infty \frac{1}{5} (ϕ^{i} - \hat{ϕ}^{i}) z^{i} .

d. Use part (c) to prove that $F_{i} = ϕ^{i} / 5$ for $i > 0$ , rounded to the nearest integer. (Hint: Observe that $∣ \hat{ϕ} ∣ < 1$ .)

4-5 Chip testing

Professor Diogenes has $n$ supposedly identical integrated-circuit chips that in principle are capable of testing each other. The professor’s test jig accommodates two chips at a time. When the jig is loaded, each chip tests the other and reports whether it is good or bad. A good chip always reports accurately whether the other chip is good or bad, but the professor cannot trust the answer of a bad chip. Thus, the four possible outcomes of a test are as follows:

Chip $A$ says	Chip $B$ says	Conclusion
$B$ is good	$A$ is good	both are good, or both are bad
$B$ is good	$A$ is bad	at least one is bad
$B$ is bad	$A$ is good	at least one is bad
$B$ is bad	$A$ is bad	at least one is bad

a. Show that if at least $n /2$ chips are bad, the professor cannot necessarily determine which chips are good using any strategy based on this kind of pairwise test. Assume that the bad chips can conspire to fool the professor.

b. Consider the problem of finding a single good chip from among $n$ chips, assuming that more than $n /2$ of the chips are good. Show that $⌊ n /2 ⌋$ pairwise tests are sufficient to reduce the problem to one of nearly half the size.

c. Show that the good chips can be identified with $Θ (n)$ pairwise tests, assuming that more than $n /2$ of the chips are good. Give and solve the recurrence that describes the number of tests.

4-6 Monge arrays

An $m \times n$ array $A$ of real numbers is a Monge array if for all $i$ , $j$ , $k$ , and $l$ such that $1 \leq i < k \leq m$ and $1 \leq j < l \leq n$ , we have

A [i, j] + A [k, l] \leq A [i, l] + A [k, j] .

In other words, whenever we pick two rows and two columns of a Monge array and consider the four elements at the intersections of the rows and the columns, the sum of the upper-left and lower-right elements is less than or equal to the sum of the lower-left and upper-right elements. For example, the following array is Monge:

1017241145367517222813443366131622632195128293417372153232324723634

a. Prove that an array is Monge if and only if for all $i = 1, 2, \dots, m - 1$ and $j = 1, 2, \dots, n - 1$ , we have

A [i, j] + A [i + 1, j + 1] \leq A [i, j + 1] + A [i + 1, j] .

(Hint: For the “if” part, use induction separately on rows and columns.)

b. The following array is not Monge. Change one element in order to make it Monge. (Hint: Use part (a).)

37215332432363413212273091532103168

c. Let $f (i)$ be the index of the column containing the leftmost minimum element of row $i$ . Prove that $f (1) \leq f (2) \leq \dots \leq f (m)$ for any $m \times n$ Monge array.

d. Here is a description of a divide-and-conquer algorithm that computes the leftmost minimum element in each row of an $m \times n$ Monge array $A$ :

Construct a submatrix $A^{'}$ of $A$ consisting of the even-numbered rows of $A$ . Recursively determine the leftmost minimum for each row of $A^{'}$ . Then compute the leftmost minimum in the odd-numbered rows of $A$ .

Explain how to compute the leftmost minimum in the odd-numbered rows of $A$ (given that the leftmost minimum of the even-numbered rows is known) in $O (m + n)$ time.

e. Write the recurrence describing the running time of the algorithm described in part (d). Show that its solution is $O (m + n lo g m)$ .

Chapter notes

Divide-and-conquer as a technique for designing algorithms dates back to at least 1962 in an article by Karatsuba and Ofman [194]. It might have been used well before then, however; according to Heideman, Johnson, and Burrus [163], C. F. Gauss devised the first fast Fourier transform algorithm in 1805, and Gauss’s formulation breaks the problem into smaller subproblems whose solutions are combined.

The maximum-subarray problem in Section 4.1 is a minor variation on a problem studied by Bentley [43, Chapter 7].

Strassen’s algorithm [325] caused much excitement when it was published in 1969. Before then, few imagined the possibility of an algorithm asymptotically faster than the basic SQUARE-MATRIX-MULTIPLY procedure. The asymptotic upper bound for matrix multiplication has been improved since then. The most asymptotically efficient algorithm for multiplying $n \times n$ matrices to date, due to Coppersmith and Winograd [78], has a running time of $O (n^{2.376})$ . The best lower bound known is just the obvious $Ω (n^{2})$ bound (obvious because we must fill in $n^{2}$ elements of the product matrix).

From a practical point of view, Strassen’s algorithm is often not the method of choice for matrix multiplication, for four reasons:

The constant factor hidden in the $Θ (n^{l g 7})$ running time of Strassen’s algorithm is larger than the constant factor in the $Θ (n^{3})$ -time SQUARE-MATRIX-MULTIPLY procedure.
When the matrices are sparse, methods tailored for sparse matrices are faster.
Strassen’s algorithm is not quite as numerically stable as SQUARE-MATRIX-MULTIPLY. In other words, because of the limited precision of computer arithmetic on noninteger values, larger errors accumulate in Strassen’s algorithm than in SQUARE-MATRIX-MULTIPLY.
The submatrices formed at the levels of recursion consume space.

The latter two reasons were mitigated around 1990. Higham [167] demonstrated that the difference in numerical stability had been overemphasized; although Strassen’s algorithm is too numerically unstable for some applications, it is within acceptable limits for others. Bailey, Lee, and Simon [32] discuss techniques for reducing the memory requirements for Strassen’s algorithm.

In practice, fast matrix-multiplication implementations for dense matrices use Strassen’s algorithm for matrix sizes above a “crossover point,” and they switch to a simpler method once the subproblem size reduces to below the crossover point. The exact value of the crossover point is highly system dependent. Analyses that count operations but ignore effects from caches and pipelining have produced crossover points as low as $n = 8$ (by Higham [167]) or $n = 12$ (by Huss-Lederman et al. [186]). D’Alberto and Nicolau [81] developed an adaptive scheme, which determines the crossover point by benchmarking when their software package is installed. They found crossover points on various systems ranging from $n = 400$ to $n = 2150$ , and they could not find a crossover point on a couple of systems.

Recurrences were studied as early as 1202 by L. Fibonacci, for whom the Fibonacci numbers are named. A. De Moivre introduced the method of generating functions (see Problem 4-4) for solving recurrences. The master method is adapted from Bentley, Haken, and Saxe [44], which provides the extended method justified by Exercise 4.6-2. Knuth [209] and Liu [237] show how to solve linear recurrences using the method of generating functions. Purdom and Brown [287] and Graham, Knuth, and Patashnik [152] contain extended discussions of recurrence solving.

Several researchers, including Akra and Bazzi [13], Roura [299], Verma [346], and Yap [360], have given methods for solving more general divide-and-conquer recurrences than are solved by the master method. We describe the result of Akra and Bazzi here, as modified by Leighton [228]. The Akra-Bazzi method works for recurrences of the form

T (x) = ⎩ ⎨ ⎧ Θ (1) i = 1 \sum k a_{i} T (b_{i} x) + f (x) if 1 \leq x \leq x_{0}, if x > x_{0}, (4.30)

where

$x \geq 1$ is a real number,
$x_{0}$ is a constant such that $x_{0} \geq 1/ b_{i}$ and $x_{0} \geq 1/ (1 - b_{i})$ for $i = 1, 2, \dots, k$ ,
$a_{i}$ is a positive constant for $i = 1, 2, \dots, k$ ,
$b_{i}$ is a constant in the range $0 < b_{i} < 1$ for $i = 1, 2, \dots, k$ ,
$k \geq 1$ is an integer constant, and
$f (x)$ is a nonnegative function that satisfies the polynomial-growth condition: there exist positive constants $c_{1}$ and $c_{2}$ such that for all $x \geq 1$ , for $i = 1, 2, \dots, k$ , and for all $u$ such that $b_{i} x \leq u \leq x$ , we have $c_{1} f (x) \leq f (u) \leq c_{2} f (x)$ . (If $∣ f^{'} (x) ∣$ is upper-bounded by some polynomial in $x$ , then $f (x)$ satisfies the polynomial-growth condition. For example, $f (x) = x^{α} l g^{β} x$ satisfies this condition for any real constants $α$ and $β$ .)

Although the master method does not apply to a recurrence such as $T (n) = T (⌊ n /3 ⌋) + T (⌊ 2 n /3 ⌋) + O (n)$ , the Akra-Bazzi method does. To solve the recurrence (4.30), we first find the unique real number $p$ such that $\sum_{i = 1}^{k} a_{i} b_{i}^{p} = 1$ . (Such a $p$ always exists.) The solution to the recurrence is then

T (n) = Θ (n^{p} (1 + \int_{1}^{n} \frac{f ( x )}{x ^{p + 1}} d x)) .

The Akra-Bazzi method can be somewhat difficult to use, but it serves in solving recurrences that model division of the problem into substantially unequally sized subproblems. The master method is simpler to use, but it applies only when subproblem sizes are equal.

5 Probabilistic Analysis and Randomized Algorithms

Big picture. Imagine you are a talent scout who must decide, one candidate at a time, whether to swap out your current star. How often will you end up swapping? If you knew nothing about the order of arrivals, you could only brace for the worst. But if you either know the arrivals are random, or make them random yourself, a surprisingly small, predictable number of swaps emerges. This chapter is about harnessing chance: first as something we assume about the input (probabilistic analysis), then as a tool we wield ourselves (randomized algorithms).

This chapter introduces probabilistic analysis and randomized algorithms. If you are unfamiliar with the basics of probability theory, you should read Appendix C, which reviews this material. We shall revisit probabilistic analysis and randomized algorithms several times throughout this book.

5.1 The hiring problem

Suppose that you need to hire a new office assistant. Your previous attempts at hiring have been unsuccessful, and you decide to use an employment agency. The employment agency sends you one candidate each day. You interview that person and then decide either to hire that person or not. You must pay the employment agency a small fee to interview an applicant. To actually hire an applicant is more costly, however, since you must fire your current office assistant and pay a substantial hiring fee to the employment agency. You are committed to having, at all times, the best possible person for the job. Therefore, you decide that, after interviewing each applicant, if that applicant is better qualified than the current office assistant, you will fire the current office assistant and hire the new applicant. You are willing to pay the resulting price of this strategy, but you wish to estimate what that price will be.

The procedure HIRE-ASSISTANT, given below, expresses this strategy for hiring in pseudocode. It assumes that the candidates for the office assistant job are numbered $1$ through $n$ . The procedure assumes that you are able to, after interviewing candidate $i$ , determine whether candidate $i$ is the best candidate you have seen so far. To initialize, the procedure creates a dummy candidate, numbered $0$ , who is less qualified than each of the other candidates.

HIRE-ASSISTANT(n)
1  best = 0          // candidate 0 is a least-qualified dummy candidate
2  for i = 1 to n
3      interview candidate i
4      if candidate i is better than candidate best
5          best = i
6          hire candidate i

The cost model for this problem differs from the model described in Chapter 2. We focus not on the running time of HIRE-ASSISTANT, but instead on the costs incurred by interviewing and hiring. On the surface, analyzing the cost of this algorithm may seem very different from analyzing the running time of, say, merge sort. The analytical techniques used, however, are identical whether we are analyzing cost or running time. In either case, we are counting the number of times certain basic operations are executed.

Interviewing has a low cost, say $c_{i}$ , whereas hiring is expensive, costing $c_{h}$ . Letting $m$ be the number of people hired, the total cost associated with this algorithm is $O (c_{i} n + c_{h} m)$ . No matter how many people we hire, we always interview $n$ candidates and thus always incur the cost $c_{i} n$ associated with interviewing. We therefore concentrate on analyzing $c_{h} m$ , the hiring cost. This quantity varies with each run of the algorithm.

This scenario serves as a model for a common computational paradigm. We often need to find the maximum or minimum value in a sequence by examining each element of the sequence and maintaining a current “winner.” The hiring problem models how often we update our notion of which element is currently winning.

Worst-case analysis

In the worst case, we actually hire every candidate that we interview. This situation occurs if the candidates come in strictly increasing order of quality, in which case we hire $n$ times, for a total hiring cost of $O (c_{h} n)$ .

Of course, the candidates do not always come in increasing order of quality. In fact, we have no idea about the order in which they arrive, nor do we have any control over this order. Therefore, it is natural to ask what we expect to happen in a typical or average case.

Probabilistic analysis

Probabilistic analysis is the use of probability in the analysis of problems. Most commonly, we use probabilistic analysis to analyze the running time of an algorithm. Sometimes we use it to analyze other quantities, such as the hiring cost in procedure HIRE-ASSISTANT. In order to perform a probabilistic analysis, we must use knowledge of, or make assumptions about, the distribution of the inputs. Then we analyze our algorithm, computing an average-case running time, where we take the average over the distribution of the possible inputs. Thus we are, in effect, averaging the running time over all possible inputs. When reporting such a running time, we will refer to it as the average-case running time.

We must be very careful in deciding on the distribution of inputs. For some problems, we may reasonably assume something about the set of all possible inputs, and then we can use probabilistic analysis as a technique for designing an efficient algorithm and as a means for gaining insight into a problem. For other problems, we cannot describe a reasonable input distribution, and in these cases we cannot use probabilistic analysis.

For the hiring problem, we can assume that the applicants come in a random order. What does that mean for this problem? We assume that we can compare any two candidates and decide which one is better qualified; that is, there is a total order on the candidates. (See Appendix B for the definition of a total order.) Thus, we can rank each candidate with a unique number from $1$ through $n$ , using $rank (i)$ to denote the rank of applicant $i$ , and adopt the convention that a higher rank corresponds to a better qualified applicant. The ordered list $⟨ rank (1), rank (2), \dots, rank (n)⟩$ is a permutation of the list $⟨ 1, 2, \dots, n ⟩$ . Saying that the applicants come in a random order is equivalent to saying that this list of ranks is equally likely to be any one of the $n!$ permutations of the numbers $1$ through $n$ . Alternatively, we say that the ranks form a uniform random permutation; that is, each of the possible $n!$ permutations appears with equal probability.

Section 5.2 contains a probabilistic analysis of the hiring problem.

Randomized algorithms

In order to use probabilistic analysis, we need to know something about the distribution of the inputs. In many cases, we know very little about the input distribution. Even if we do know something about the distribution, we may not be able to model this knowledge computationally. Yet we often can use probability and randomness as a tool for algorithm design and analysis, by making the behavior of part of the algorithm random.

In the hiring problem, it may seem as if the candidates are being presented to us in a random order, but we have no way of knowing whether or not they really are. Thus, in order to develop a randomized algorithm for the hiring problem, we must have greater control over the order in which we interview the candidates. We will, therefore, change the model slightly. We say that the employment agency has $n$ candidates, and they send us a list of the candidates in advance. On each day, we choose, randomly, which candidate to interview. Although we know nothing about the candidates (besides their names), we have made a significant change. Instead of relying on a guess that the candidates come to us in a random order, we have instead gained control of the process and enforced a random order.

More generally, we call an algorithm randomized if its behavior is determined not only by its input but also by values produced by a random-number generator. We shall assume that we have at our disposal a random-number generator RANDOM. A call to $RANDOM (a, b)$ returns an integer between $a$ and $b$ , inclusive, with each such integer being equally likely. For example, $RANDOM (0, 1)$ produces $0$ with probability $1/2$ , and it produces $1$ with probability $1/2$ . A call to $RANDOM (3, 7)$ returns either $3$ , $4$ , $5$ , $6$ , or $7$ , each with probability $1/5$ . Each integer returned by RANDOM is independent of the integers returned on previous calls. You may imagine RANDOM as rolling a $(b - a + 1)$ -sided die to obtain its output. (In practice, most programming environments offer a pseudorandom-number generator: a deterministic algorithm returning numbers that “look” statistically random.)

When analyzing the running time of a randomized algorithm, we take the expectation of the running time over the distribution of values returned by the random number generator. We distinguish these algorithms from those in which the input is random by referring to the running time of a randomized algorithm as an expected running time. In general, we discuss the average-case running time when the probability distribution is over the inputs to the algorithm, and we discuss the expected running time when the algorithm itself makes random choices.

Exercises 5.1-1 to 5.1-3

5.1-1 Show that the assumption that we are always able to determine which candidate is best, in line 4 of procedure HIRE-ASSISTANT, implies that we know a total order on the ranks of the candidates.

5.1-2* Describe an implementation of the procedure $RANDOM (a, b)$ that only makes calls to $RANDOM (0, 1)$ . What is the expected running time of your procedure, as a function of $a$ and $b$ ?

5.1-3* Suppose that you want to output $0$ with probability $1/2$ and $1$ with probability $1/2$ . At your disposal is a procedure BIASED-RANDOM, that outputs either $0$ or $1$ . It outputs $1$ with some probability $p$ and $0$ with probability $1 - p$ , where $0 < p < 1$ , but you do not know what $p$ is. Give an algorithm that uses BIASED-RANDOM as a subroutine, and returns an unbiased answer, returning $0$ with probability $1/2$ and $1$ with probability $1/2$ . What is the expected running time of your algorithm as a function of $p$ ?

5.2 Indicator random variables

In order to analyze many algorithms, including the hiring problem, we use indicator random variables. Indicator random variables provide a convenient method for converting between probabilities and expectations. Suppose we are given a sample space $S$ and an event $A$ . Then the indicator random variable $I {A}$ associated with event $A$ is defined as

I {A} = {10 if A occurs, if A does not occur . (5.1)

As a simple example, let us determine the expected number of heads that we obtain when flipping a fair coin. Our sample space is $S = {H, T}$ , with $Pr {H} = Pr {T} = 1/2$ . We can then define an indicator random variable $X_{H}$ , associated with the coin coming up heads, which is the event $H$ . This variable counts the number of heads obtained in this flip, and it is $1$ if the coin comes up heads and $0$ otherwise. We write

X_{H} = I {H} = {10 if H occurs, if T occurs .

The expected number of heads obtained in one flip of the coin is simply the expected value of our indicator variable $X_{H}$ :

E [X_{H}] = E [I {H}] = 1 \cdot Pr {H} + 0 \cdot Pr {T} = 1 \cdot (1/2) + 0 \cdot (1/2) = 1/2.

Thus the expected number of heads obtained by one flip of a fair coin is $1/2$ . As the following lemma shows, the expected value of an indicator random variable associated with an event $A$ is equal to the probability that $A$ occurs.

Lemma 5.1. Given a sample space $S$ and an event $A$ in the sample space $S$ , let $X_{A} = I {A}$ . Then $E [X_{A}] = Pr {A}$ .

Proof. By the definition of an indicator random variable from equation (5.1) and the definition of expected value, we have
$E [X_{A}] = E [I {A}] = 1 \cdot Pr {A} + 0 \cdot Pr {\overset{ˉ}{A}} = Pr {A},$
where $\overset{ˉ}{A}$ denotes $S - A$ , the complement of $A$ . $■$

Although indicator random variables may seem cumbersome for an application such as counting the expected number of heads on a flip of a single coin, they are useful for analyzing situations in which we perform repeated random trials. For example, indicator random variables give us a simple way to arrive at the result of equation (C.37). In this equation, we compute the number of heads in $n$ coin flips by considering separately the probability of obtaining $0$ heads, $1$ head, $2$ heads, etc. The simpler method proposed in equation (C.38) instead uses indicator random variables implicitly. Making this argument more explicit, we let $X_{i}$ be the indicator random variable associated with the event in which the $i$ th flip comes up heads: $X_{i} = I {the i th flip results in the event H}$ . Let $X$ be the random variable denoting the total number of heads in the $n$ coin flips, so that

X = i = 1 \sum n X_{i} .

We wish to compute the expected number of heads, and so we take the expectation of both sides of the above equation to obtain

E [X] = E [i = 1 \sum n X_{i}] .

The above equation gives the expectation of the sum of $n$ indicator random variables. By Lemma 5.1, we can easily compute the expectation of each of the random variables. By equation (C.21), linearity of expectation, it is easy to compute the expectation of the sum: it equals the sum of the expectations of the $n$ random variables. Linearity of expectation makes the use of indicator random variables a powerful analytical technique; it applies even when there is dependence among the random variables. We now can easily compute the expected number of heads:

E [X] = E [i = 1 \sum n X_{i}] = i = 1 \sum n E [X_{i}] = i = 1 \sum n 1/2 = n /2.

Thus, compared to the method used in equation (C.37), indicator random variables greatly simplify the calculation. We shall use indicator random variables throughout this book.

Think about it. Linearity of expectation works even when the variables depend on each other. Why is that such a gift? Because tracking how variables interact is usually the hard part, and here we get to skip it entirely.

Analysis of the hiring problem using indicator random variables

Returning to the hiring problem, we now wish to compute the expected number of times that we hire a new office assistant. In order to use a probabilistic analysis, we assume that the candidates arrive in a random order, as discussed in the previous section. (We shall see in Section 5.3 how to remove this assumption.) Let $X$ be the random variable whose value equals the number of times we hire a new office assistant. We could then apply the definition of expected value from equation (C.20) to obtain

E [X] = x = 1 \sum n x Pr {X = x},

but this calculation would be cumbersome. We shall instead use indicator random variables to greatly simplify the calculation.

To use indicator random variables, instead of computing $E [X]$ by defining one variable associated with the number of times we hire a new office assistant, we define $n$ variables related to whether or not each particular candidate is hired. In particular, we let $X_{i}$ be the indicator random variable associated with the event in which the $i$ th candidate is hired. Thus,

X_{i} = I {candidate i is hired} = {10 if candidate i is hired, if candidate i is not hired,

and

X = X_{1} + X_{2} + \dots + X_{n} . (5.2)

By Lemma 5.1, we have that

E [X_{i}] = Pr {candidate i is hired},

and we must therefore compute the probability that lines 5-6 of HIRE-ASSISTANT are executed.

Candidate $i$ is hired, in line 6, exactly when candidate $i$ is better than each of candidates $1$ through $i - 1$ . Because we have assumed that the candidates arrive in a random order, the first $i$ candidates have appeared in a random order. Any one of these first $i$ candidates is equally likely to be the best-qualified so far. Candidate $i$ has a probability of $1/ i$ of being better qualified than candidates $1$ through $i - 1$ and thus a probability of $1/ i$ of being hired. By Lemma 5.1, we conclude that

E [X_{i}] = 1/ i . (5.3)

Now we can compute $E [X]$ :

E [X] = E [i = 1 \sum n X_{i}] = i = 1 \sum n E [X_{i}] = i = 1 \sum n \frac{1}{i} = ln n + O (1) (by equation (5.2)) (by linearity of expectation) (by equation (5.3)) (by equation (A.7)) . (5.4) (5.5)

Even though we interview $n$ people, we actually hire only approximately $ln n$ of them, on average. We summarize this result in the following lemma.

Lemma 5.2. Assuming that the candidates are presented in a random order, algorithm HIRE-ASSISTANT has an average-case total hiring cost of $O (c_{h} ln n)$ .

Proof. The hiring cost is $c_{h}$ times the number of hires $m$ . By the analysis above, which uses equation (5.3) and equation (5.5), the expected number of hires is $ln n + O (1)$ , so the expected hiring cost is $c_{h} (ln n + O (1)) = O (c_{h} ln n)$ . $■$

The average-case hiring cost is a significant improvement over the worst-case hiring cost of $O (c_{h} n)$ .

Exercises 5.2-1 to 5.2-5

5.2-1 In HIRE-ASSISTANT, assuming that the candidates are presented in a random order, what is the probability that you hire exactly one time? What is the probability that you hire exactly $n$ times?

5.2-2 In HIRE-ASSISTANT, assuming that the candidates are presented in a random order, what is the probability that you hire exactly twice?

5.2-3 Use indicator random variables to compute the expected value of the sum of $n$ dice.

5.2-4 Use indicator random variables to solve the following problem, which is known as the hat-check problem. Each of $n$ customers gives a hat to a hat-check person at a restaurant. The hat-check person gives the hats back to the customers in a random order. What is the expected number of customers who get back their own hat?

5.2-5 Let $A [1.. n]$ be an array of $n$ distinct numbers. If $i < j$ and $A [i] > A [j]$ , then the pair $(i, j)$ is called an inversion of $A$ . (See Problem 2-4 for more on inversions.) Suppose that the elements of $A$ form a uniform random permutation of $⟨ 1, 2, \dots, n ⟩$ . Use indicator random variables to compute the expected number of inversions.

5.3 Randomized algorithms

In the previous section, we showed how knowing a distribution on the inputs can help us to analyze the average-case behavior of an algorithm. Many times, we do not have such knowledge, thus precluding an average-case analysis. As mentioned in Section 5.1, we may be able to use a randomized algorithm.

For a problem such as the hiring problem, in which it is helpful to assume that all permutations of the input are equally likely, a probabilistic analysis can guide the development of a randomized algorithm. Instead of assuming a distribution of inputs, we impose a distribution. In particular, before running the algorithm, we randomly permute the candidates in order to enforce the property that every permutation is equally likely. Although we have modified the algorithm, we still expect to hire a new office assistant approximately $ln n$ times. But now we expect this to be the case for any input, rather than for inputs drawn from a particular distribution.

Let us further explore the distinction between probabilistic analysis and randomized algorithms. In Section 5.2, we claimed that, assuming that the candidates arrive in a random order, the expected number of times we hire a new office assistant is about $ln n$ . Note that the algorithm here is deterministic; for any particular input, the number of times a new office assistant is hired is always the same. Furthermore, the number of times we hire a new office assistant differs for different inputs, and it depends on the ranks of the various candidates. Since this number depends only on the ranks of the candidates, we can represent a particular input by listing, in order, the ranks of the candidates, i.e., $⟨ rank (1), rank (2), \dots, rank (n)⟩$ . Given the rank list $A_{1} = ⟨ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ⟩$ , a new office assistant is always hired $10$ times, since each successive candidate is better than the previous one, and lines 5-6 are executed in each iteration. Given the list of ranks $A_{2} = ⟨ 10, 9, 8, 7, 6, 5, 4, 3, 2, 1 ⟩$ , a new office assistant is hired only once, in the first iteration. Given a list of ranks $A_{3} = ⟨ 5, 2, 1, 8, 4, 7, 10, 9, 3, 6 ⟩$ , a new office assistant is hired three times, upon interviewing the candidates with ranks $5$ , $8$ , and $10$ . Recalling that the cost of our algorithm depends on how many times we hire a new office assistant, we see that there are expensive inputs such as $A_{1}$ , inexpensive inputs such as $A_{2}$ , and moderately expensive inputs such as $A_{3}$ .

Consider, on the other hand, the randomized algorithm that first permutes the candidates and then determines the best candidate. In this case, we randomize in the algorithm, not in the input distribution. Given a particular input, say $A_{3}$ above, we cannot say how many times the maximum is updated, because this quantity differs with each run of the algorithm. The first time we run the algorithm on $A_{3}$ , it may produce the permutation $A_{1}$ and perform $10$ updates; but the second time we run the algorithm, we may produce the permutation $A_{2}$ and perform only one update. The third time we run it, we may perform some other number of updates. Each time we run the algorithm, the execution depends on the random choices made and is likely to differ from the previous execution of the algorithm. For this algorithm and many other randomized algorithms, no particular input elicits its worst-case behavior. Even your worst enemy cannot produce a bad input array, since the random permutation makes the input order irrelevant. The randomized algorithm performs badly only if the random-number generator produces an “unlucky” permutation.

For the hiring problem, the only change needed in the code is to randomly permute the array.

RANDOMIZED-HIRE-ASSISTANT(n)
1  randomly permute the list of candidates
2  best = 0          // candidate 0 is a least-qualified dummy candidate
3  for i = 1 to n
4      interview candidate i
5      if candidate i is better than candidate best
6          best = i
7          hire candidate i

With this simple change, we have created a randomized algorithm whose performance matches that obtained by assuming that the candidates were presented in a random order.

Lemma 5.3. The expected hiring cost of the procedure RANDOMIZED-HIRE-ASSISTANT is $O (c_{h} ln n)$ .

Proof. After permuting the input array, we have achieved a situation identical to that of the probabilistic analysis of HIRE-ASSISTANT. Hence Lemma 5.2’s computation applies verbatim: $E [X_{i}] = 1/ i$ , so $E [X] = ln n + O (1)$ and the expected hiring cost is $O (c_{h} ln n)$ . $■$

Comparing Lemmas 5.2 and 5.3 highlights the difference between probabilistic analysis and randomized algorithms. In Lemma 5.2, we make an assumption about the input. In Lemma 5.3, we make no such assumption, although randomizing the input takes some additional time. To remain consistent with our terminology, we couched Lemma 5.2 in terms of the average-case hiring cost and Lemma 5.3 in terms of the expected hiring cost. In the remainder of this section, we discuss some issues involved in randomly permuting inputs.

Randomly permuting arrays

Many randomized algorithms randomize the input by permuting the given input array. (There are other ways to use randomization.) Here, we shall discuss two methods for doing so. We assume that we are given an array $A$ which, without loss of generality, contains the elements $1$ through $n$ . Our goal is to produce a random permutation of the array.

One common method is to assign each element $A [i]$ of the array a random priority $P [i]$ , and then sort the elements of $A$ according to these priorities. For example, if our initial array is $A = ⟨ 1, 2, 3, 4 ⟩$ and we choose random priorities $P = ⟨ 36, 3, 62, 19 ⟩$ , we would produce an array $B = ⟨ 2, 4, 1, 3 ⟩$ , since the second priority is the smallest, followed by the fourth, then the first, and finally the third. We call this procedure PERMUTE-BY-SORTING:

PERMUTE-BY-SORTING(A)
1  n = A.length
2  let P[1..n] be a new array
3  for i = 1 to n
4      P[i] = RANDOM(1, n^3)
5  sort A, using P as sort keys

Line 4 chooses a random number between $1$ and $n^{3}$ . We use a range of $1$ to $n^{3}$ to make it likely that all the priorities in $P$ are unique. (Exercise 5.3-5 asks you to prove that the probability that all entries are unique is at least $1 - 1/ n$ , and Exercise 5.3-6 asks how to implement the algorithm even if two or more priorities are identical.) Let us assume that all the priorities are unique.

The time-consuming step in this procedure is the sorting in line 5. As we shall see in Chapter 8, if we use a comparison sort, sorting takes $Ω (n l g n)$ time. We can achieve this lower bound, since we have seen that merge sort takes $Θ (n l g n)$ time. (We shall see other comparison sorts that take $Θ (n l g n)$ time in Part II. Exercise 8.3-4 asks you to solve the very similar problem of sorting numbers in the range $0$ to $n^{3} - 1$ in $O (n)$ time.) After sorting, if $P [i]$ is the $j$ th smallest priority, then $A [i]$ lies in position $j$ of the output. In this manner we obtain a permutation. It remains to prove that the procedure produces a uniform random permutation, that is, that the procedure is equally likely to produce every permutation of the numbers $1$ through $n$ .

Lemma 5.4. Procedure PERMUTE-BY-SORTING produces a uniform random permutation of the input, assuming that all priorities are distinct.

Proof. We start by considering the particular permutation in which each element $A [i]$ receives the $i$ th smallest priority. We shall show that this permutation occurs with probability exactly $1/ n!$ . For $i = 1, 2, \dots, n$ , let $E_{i}$ be the event that element $A [i]$ receives the $i$ th smallest priority. Then we wish to compute the probability that for all $i$ , event $E_{i}$ occurs, which is
$Pr {E_{1} \cap E_{2} \cap E_{3} \cap \dots \cap E_{n - 1} \cap E_{n}} .$
Using Exercise C.2-5, this probability is equal to
$Pr {E_{1}} \cdot Pr {E_{2} ∣ E_{1}} \cdot Pr {E_{3} ∣ E_{2} \cap E_{1}} \cdot Pr {E_{4} ∣ E_{3} \cap E_{2} \cap E_{1}} \dots Pr {E_{i} ∣ E_{i - 1} \cap E_{i - 2} \cap \dots \cap E_{1}} \dots Pr {E_{n} ∣ E_{n - 1} \cap \dots \cap E_{1}} .$
We have that $Pr {E_{1}} = 1/ n$ because it is the probability that one priority chosen randomly out of a set of $n$ is the smallest priority. Next, we observe that $Pr {E_{2} ∣ E_{1}} = 1/ (n - 1)$ because given that element $A [1]$ has the smallest priority, each of the remaining $n - 1$ elements has an equal chance of having the second smallest priority. In general, for $i = 2, 3, \dots, n$ , we have that
$Pr {E_{i} ∣ E_{i - 1} \cap E_{i - 2} \cap \dots \cap E_{1}} = \frac{1}{n - i + 1},$
since, given that elements $A [1]$ through $A [i - 1]$ have the $i - 1$ smallest priorities (in order), each of the remaining $n - (i - 1)$ elements has an equal chance of having the $i$ th smallest priority. Thus, we have
$Pr {E_{1} \cap E_{2} \cap \dots \cap E_{n - 1} \cap E_{n}} = (\frac{1}{n}) (\frac{1}{n - 1}) \dots (\frac{1}{2}) (\frac{1}{1}) = \frac{1}{n !},$
and we have shown that the probability of obtaining the identity permutation is $1/ n!$ .

We can extend this proof to work for any permutation of priorities. Consider any fixed permutation $σ = ⟨ σ (1), σ (2), \dots, σ (n)⟩$ of the set ${1, 2, \dots, n}$ . Let us denote by $r_{i}$ the rank of the priority assigned to element $A [i]$ , where the element with the $j$ th smallest priority has rank $j$ . If we define $E_{i}$ as the event in which element $A [i]$ receives the $σ (i)$ th smallest priority, or $r_{i} = σ (i)$ , the same proof still applies. Therefore, if we calculate the probability of obtaining any particular permutation, the calculation is identical to the one above, so that the probability of obtaining this permutation is also $1/ n!$ . $■$

You might think that to prove that a permutation is a uniform random permutation, it suffices to show that, for each element $A [i]$ , the probability that the element winds up in position $j$ is $1/ n$ . Exercise 5.3-4 shows that this weaker condition is, in fact, insufficient.

A better method for generating a random permutation is to permute the given array in place. The procedure RANDOMIZE-IN-PLACE does so in $O (n)$ time. In its $i$ th iteration, it chooses the element $A [i]$ randomly from among elements $A [i]$ through $A [n]$ . Subsequent to the $i$ th iteration, $A [i]$ is never altered.

RANDOMIZE-IN-PLACE(A)
1  n = A.length
2  for i = 1 to n
3      swap A[i] with A[RANDOM(i, n)]

We shall use a loop invariant to show that procedure RANDOMIZE-IN-PLACE produces a uniform random permutation. A $k$ -permutation on a set of $n$ elements is a sequence containing $k$ of the $n$ elements, with no repetitions. (See Appendix C.) There are $n! / (n - k)!$ such possible $k$ -permutations.

Lemma 5.5. Procedure RANDOMIZE-IN-PLACE computes a uniform random permutation.

Proof. We use the following loop invariant:

Just prior to the $i$ th iteration of the for loop of lines 2-3, for each possible $(i - 1)$ -permutation of the $n$ elements, the subarray $A [1.. i - 1]$ contains this $(i - 1)$ -permutation with probability $(n - i + 1)! / n!$ .

We need to show that this invariant is true prior to the first loop iteration, that each iteration of the loop maintains the invariant, and that the invariant provides a useful property to show correctness when the loop terminates.

Initialization: Consider the situation just before the first loop iteration, so that $i = 1$ . The loop invariant says that for each possible $0$ -permutation, the subarray $A [1..0]$ contains this $0$ -permutation with probability
$\frac{( n - i + 1 )!}{n !} = \frac{n !}{n !} = 1.$
The subarray $A [1..0]$ is an empty subarray, and a $0$ -permutation has no elements. Thus, $A [1..0]$ contains any $0$ -permutation with probability $1$ , and the loop invariant holds prior to the first iteration.

Maintenance: We assume that just before the $i$ th iteration, each possible $(i - 1)$ -permutation appears in the subarray $A [1.. i - 1]$ with probability $(n - i + 1)! / n!$ , and we shall show that after the $i$ th iteration, each possible $i$ -permutation appears in the subarray $A [1.. i]$ with probability $(n - i)! / n!$ . Incrementing $i$ for the next iteration then maintains the loop invariant.

Let us examine the $i$ th iteration. Consider a particular $i$ -permutation, and denote the elements in it by $⟨ x_{1}, x_{2}, \dots, x_{i} ⟩$ . This permutation consists of an $(i - 1)$ -permutation $⟨ x_{1}, \dots, x_{i - 1} ⟩$ followed by the value $x_{i}$ that the algorithm places in $A [i]$ . Let $E_{1}$ denote the event in which the first $i - 1$ iterations have created the particular $(i - 1)$ -permutation $⟨ x_{1}, \dots, x_{i - 1} ⟩$ in $A [1.. i - 1]$ . By the loop invariant, $Pr {E_{1}} = (n - i + 1)! / n!$ . Let $E_{2}$ be the event that the $i$ th iteration puts $x_{i}$ in position $A [i]$ . The $i$ -permutation $⟨ x_{1}, \dots, x_{i} ⟩$ appears in $A [1.. i]$ precisely when both $E_{1}$ and $E_{2}$ occur, and so we wish to compute $Pr {E_{2} \cap E_{1}}$ . Using equation (C.14), we have
$Pr {E_{2} \cap E_{1}} = Pr {E_{2} ∣ E_{1}} Pr {E_{1}} .$
The probability $Pr {E_{2} ∣ E_{1}}$ equals $1/ (n - i + 1)$ because in line 3 the algorithm chooses $x_{i}$ randomly from the $n - i + 1$ values in positions $A [i .. n]$ . Thus, we have
$Pr {E_{2} \cap E_{1}} = Pr {E_{2} ∣ E_{1}} Pr {E_{1}} = \frac{1}{n - i + 1} \cdot \frac{( n - i + 1 )!}{n !} = \frac{( n - i )!}{n !} .$
Termination: At termination, $i = n + 1$ , and we have that the subarray $A [1.. n]$ is a given $n$ -permutation with probability
$\frac{( n - ( n + 1 ) + 1 )!}{n !} = \frac{0 !}{n !} = \frac{1}{n !} .$
Thus, RANDOMIZE-IN-PLACE produces a uniform random permutation. $■$

A randomized algorithm is often the simplest and most efficient way to solve a problem. We shall use randomized algorithms occasionally throughout this book.

Exercises 5.3-1 to 5.3-7

5.3-1 Professor Marceau objects to the loop invariant used in the proof of Lemma 5.5. He questions whether it is true prior to the first iteration. He reasons that we could just as easily declare that an empty subarray contains no $0$ -permutations. Therefore, the probability that an empty subarray contains a $0$ -permutation should be $0$ , thus invalidating the loop invariant prior to the first iteration. Rewrite the procedure RANDOMIZE-IN-PLACE so that its associated loop invariant applies to a nonempty subarray prior to the first iteration, and modify the proof of Lemma 5.5 for your procedure.

5.3-2 Professor Kelp decides to write a procedure that produces at random any permutation besides the identity permutation. He proposes the following procedure:

PERMUTE-WITHOUT-IDENTITY(A)
1  n = A.length
2  for i = 1 to n - 1
3      swap A[i] with A[RANDOM(i + 1, n)]

Does this code do what Professor Kelp intends?

5.3-3 Suppose that instead of swapping element $A [i]$ with a random element from the subarray $A [i .. n]$ , we swapped it with a random element from anywhere in the array:

PERMUTE-WITH-ALL(A)
1  n = A.length
2  for i = 1 to n
3      swap A[i] with A[RANDOM(1, n)]

Does this code produce a uniform random permutation? Why or why not?

5.3-4 Professor Armstrong suggests the following procedure for generating a uniform random permutation:

PERMUTE-BY-CYCLIC(A)
1  n = A.length
2  let B[1..n] be a new array
3  offset = RANDOM(1, n)
4  for i = 1 to n
5      dest = i + offset
6      if dest > n
7          dest = dest - n
8      B[dest] = A[i]
9  return B

Show that each element $A [i]$ has a $1/ n$ probability of winding up in any particular position in $B$ . Then show that Professor Armstrong is mistaken by showing that the resulting permutation is not uniformly random.

5.3-5* Prove that in the array $P$ in procedure PERMUTE-BY-SORTING, the probability that all elements are unique is at least $1 - 1/ n$ .

5.3-6 Explain how to implement the algorithm PERMUTE-BY-SORTING to handle the case in which two or more priorities are identical. That is, your algorithm should produce a uniform random permutation, even if two or more priorities are identical.

5.3-7 Suppose we want to create a random sample of the set ${1, 2, 3, \dots, n}$ , that is, an $m$ -element subset $S$ , where $0 \leq m \leq n$ , such that each $m$ -subset is equally likely to be created. One way would be to set $A [i] = i$ for $i = 1, 2, 3, \dots, n$ , call RANDOMIZE-IN-PLACE $(A)$ , and then take just the first $m$ array elements. This method would make $n$ calls to the RANDOM procedure. If $n$ is much larger than $m$ , we can create a random sample with fewer calls to RANDOM. Show that the following recursive procedure returns a random $m$ -subset $S$ of ${1, 2, 3, \dots, n}$ , in which each $m$ -subset is equally likely, while making only $m$ calls to RANDOM:

RANDOM-SAMPLE(m, n)
1  if m == 0
2      return ∅
3  else S = RANDOM-SAMPLE(m - 1, n - 1)
4      i = RANDOM(1, n)
5      if i ∈ S
6          S = S ∪ {n}
7      else S = S ∪ {i}
8  return S

5.4 Probabilistic analysis and further uses of indicator random variables $⋆$

This advanced section further illustrates probabilistic analysis by way of four examples. The first determines the probability that in a room of $k$ people, two of them share the same birthday. The second example examines what happens when we randomly toss balls into bins. The third investigates “streaks” of consecutive heads when we flip coins. The final example analyzes a variant of the hiring problem in which you have to make decisions without actually interviewing all the candidates.

5.4.1 The birthday paradox

Our first example is the birthday paradox. How many people must there be in a room before there is a $50%$ chance that two of them were born on the same day of the year? The answer is surprisingly few. The paradox is that it is in fact far fewer than the number of days in a year, or even half the number of days in a year, as we shall see.

Think about it. Before reading on, guess: how many people do you need? Most people guess close to $183$ . The truth is far smaller, because the number of pairs of people grows quadratically while the number of people grows only linearly.

To answer this question, we index the people in the room with the integers $1, 2, \dots, k$ , where $k$ is the number of people in the room. We ignore the issue of leap years and assume that all years have $n = 365$ days. For $i = 1, 2, \dots, k$ , let $b_{i}$ be the day of the year on which person $i$ ‘s birthday falls, where $1 \leq b_{i} \leq n$ . We also assume that birthdays are uniformly distributed across the $n$ days of the year, so that $Pr {b_{i} = r} = 1/ n$ for $i = 1, 2, \dots, k$ and $r = 1, 2, \dots, n$ .

The probability that two given people, say $i$ and $j$ , have matching birthdays depends on whether the random selection of birthdays is independent. We assume from now on that birthdays are independent, so that the probability that $i$ ‘s birthday and $j$ ‘s birthday both fall on day $r$ is

Pr {b_{i} = r and b_{j} = r} = Pr {b_{i} = r} Pr {b_{j} = r} = 1/ n^{2} .

Thus, the probability that they both fall on the same day is

Pr {b_{i} = b_{j}} = r = 1 \sum n Pr {b_{i} = r and b_{j} = r} = r = 1 \sum n (1/ n^{2}) = 1/ n . (5.6)

More intuitively, once $b_{i}$ is chosen, the probability that $b_{j}$ is chosen to be the same day is $1/ n$ . Thus, the probability that $i$ and $j$ have the same birthday is the same as the probability that the birthday of one of them falls on a given day. Notice, however, that this coincidence depends on the assumption that the birthdays are independent.

We can analyze the probability of at least $2$ out of $k$ people having matching birthdays by looking at the complementary event. The probability that at least two of the birthdays match is $1$ minus the probability that all the birthdays are different. The event that $k$ people have distinct birthdays is

B_{k} = i = 1 ⋂ k A_{i},

where $A_{i}$ is the event that person $i$ ‘s birthday is different from person $j$ ‘s for all $j < i$ . Since we can write $B_{k} = A_{k} \cap B_{k - 1}$ , we obtain from equation (C.16) the recurrence

Pr {B_{k}} = Pr {B_{k - 1}} Pr {A_{k} ∣ B_{k - 1}}, (5.7)

where we take $Pr {B_{1}} = Pr {A_{1}} = 1$ as an initial condition. In other words, the probability that $b_{1}, b_{2}, \dots, b_{k}$ are distinct birthdays is the probability that $b_{1}, b_{2}, \dots, b_{k - 1}$ are distinct birthdays times the probability that $b_{k} \neq = b_{i}$ for $i = 1, 2, \dots, k - 1$ , given that $b_{1}, b_{2}, \dots, b_{k - 1}$ are distinct.

If $b_{1}, b_{2}, \dots, b_{k - 1}$ are distinct, the conditional probability that $b_{k} \neq = b_{i}$ for $i = 1, 2, \dots, k - 1$ is $Pr {A_{k} ∣ B_{k - 1}} = (n - k + 1) / n$ , since out of the $n$ days, $n - (k - 1)$ days are not taken. We iteratively apply the recurrence (5.7) to obtain

Pr {B_{k}} = Pr {B_{k - 1}} Pr {A_{k} ∣ B_{k - 1}} = Pr {B_{k - 2}} Pr {A_{k - 1} ∣ B_{k - 2}} Pr {A_{k} ∣ B_{k - 1}} ⋮ = Pr {B_{1}} Pr {A_{2} ∣ B_{1}} Pr {A_{3} ∣ B_{2}} \dots Pr {A_{k} ∣ B_{k - 1}} = 1 \cdot (\frac{n - 1}{n}) (\frac{n - 2}{n}) \dots (\frac{n - k + 1}{n}) = 1 \cdot (1 - \frac{1}{n}) (1 - \frac{2}{n}) \dots (1 - \frac{k - 1}{n}) .

Inequality (3.12), $1 + x \leq e^{x}$ , gives us

Pr {B_{k}} \leq e^{- 1/ n} e^{- 2/ n} \dots e^{- (k - 1) / n} = e^{- \sum_{i = 1}^{k - 1} i / n} = e^{- k (k - 1) /2 n} \leq 1/2

when $- k (k - 1) /2 n \leq ln (1/2)$ . The probability that all $k$ birthdays are distinct is at most $1/2$ when $k (k - 1) \geq 2 n ln 2$ or, solving the quadratic equation, when

k \geq \frac{1 + 1 + ( 8 ln 2 ) n}{2} .

For $n = 365$ , we must have $k \geq 23$ . Thus, if at least $23$ people are in a room, the probability is at least $1/2$ that at least two people have the same birthday. On Mars, a year is $669$ Martian days long; it therefore takes $31$ Martians to get the same effect.

An analysis using indicator random variables

We can use indicator random variables to provide a simpler but approximate analysis of the birthday paradox. For each pair $(i, j)$ of the $k$ people in the room, we define the indicator random variable $X_{ij}$ , for $1 \leq i < j \leq k$ , by

X_{ij} = I {person i and person j have the same birthday} = {10 if person i and person j have the same birthday, otherwise .

By equation (5.6), the probability that two people have matching birthdays is $1/ n$ , and thus by Lemma 5.1, we have

E [X_{ij}] = Pr {person i and person j have the same birthday} = 1/ n .

Letting $X$ be the random variable that counts the number of pairs of individuals having the same birthday, we have

X = i = 1 \sum k j = i + 1 \sum k X_{ij} .

Taking expectations of both sides and applying linearity of expectation, we obtain

E [X] = E [i = 1 \sum k j = i + 1 \sum k X_{ij}] = i = 1 \sum k j = i + 1 \sum k E [X_{ij}] = (2 k) \frac{1}{n} = \frac{k ( k - 1 )}{2 n} . (by linearity of expectation) (there are (2 k) pairs, each with E [X_{ij}] = 1/ n)

When $k (k - 1) \geq 2 n$ , therefore, the expected number of pairs of people with the same birthday is at least $1$ . Thus, if we have at least $2 n + 1$ individuals in a room, we can expect at least two to have the same birthday. For $n = 365$ , if $k = 28$ , the expected number of pairs with the same birthday is $(28 \cdot 27) / (2 \cdot 365) \approx 1.0356$ . Thus, with at least $28$ people, we expect to find at least one matching pair of birthdays. On Mars, where a year is $669$ Martian days long, we need at least $38$ Martians.

The first analysis, which used only probabilities, determined the number of people required for the probability to exceed $1/2$ that a matching pair of birthdays exists, and the second analysis, which used indicator random variables, determined the number such that the expected number of matching birthdays is $1$ . Although the exact numbers of people differ for the two situations, they are the same asymptotically: $Θ (n)$ .

5.4.2 Balls and bins

Consider a process in which we randomly toss identical balls into $b$ bins, numbered $1, 2, \dots, b$ . The tosses are independent, and on each toss the ball is equally likely to end up in any bin. The probability that a tossed ball lands in any given bin is $1/ b$ . Thus, the ball-tossing process is a sequence of Bernoulli trials (see Appendix C.4) with a probability $1/ b$ of success, where success means that the ball falls in the given bin. This model is particularly useful for analyzing hashing (see Chapter 11), and we can answer a variety of interesting questions about the ball-tossing process. (Problem C-1 asks additional questions about balls and bins.)

How many balls fall in a given bin? The number of balls that fall in a given bin follows the binomial distribution $b (k; n, 1/ b)$ . If we toss $n$ balls, equation (C.37) tells us that the expected number of balls that fall in the given bin is $n / b$ .

How many balls must we toss, on the average, until a given bin contains a ball? The number of tosses until the given bin receives a ball follows the geometric distribution with probability $1/ b$ and, by equation (C.32), the expected number of tosses until success is $1/ (1/ b) = b$ .

How many balls must we toss until every bin contains at least one ball? Let us call a toss in which a ball falls into an empty bin a “hit.” We want to know the expected number $n$ of tosses required to get $b$ hits.

Using the hits, we can partition the $n$ tosses into stages. The $i$ th stage consists of the tosses after the $(i - 1)$ st hit until the $i$ th hit. The first stage consists of the first toss, since we are guaranteed to have a hit when all bins are empty. For each toss during the $i$ th stage, $i - 1$ bins contain balls and $b - i + 1$ bins are empty. Thus, for each toss in the $i$ th stage, the probability of obtaining a hit is $(b - i + 1) / b$ .

Let $n_{i}$ denote the number of tosses in the $i$ th stage. Thus, the number of tosses required to get $b$ hits is $n = \sum_{i = 1}^{b} n_{i}$ . Each random variable $n_{i}$ has a geometric distribution with probability of success $(b - i + 1) / b$ and thus, by equation (C.32), we have

E [n_{i}] = \frac{b}{b - i + 1} .

By linearity of expectation, we have

E [n] = E [i = 1 \sum b n_{i}] = i = 1 \sum b E [n_{i}] = i = 1 \sum b \frac{b}{b - i + 1} = b i = 1 \sum b \frac{1}{i} = b (ln b + O (1)) (by linearity of expectation) (by the geometric expectation above) (reindexing the sum, with i replaced by b - i + 1) (by equation (A.7)) .

It therefore takes approximately $b ln b$ tosses before we can expect that every bin has a ball. This problem is also known as the coupon collector’s problem, which says that a person trying to collect each of $b$ different coupons expects to acquire approximately $b ln b$ randomly obtained coupons in order to succeed.

5.4.3 Streaks

Suppose you flip a fair coin $n$ times. What is the longest streak of consecutive heads that you expect to see? The answer is $Θ (l g n)$ , as the following analysis shows.

We first prove that the expected length of the longest streak of heads is $O (l g n)$ . The probability that each coin flip is a head is $1/2$ . Let $A_{ik}$ be the event that a streak of heads of length at least $k$ begins with the $i$ th coin flip or, more precisely, the event that the $k$ consecutive coin flips $i, i + 1, \dots, i + k - 1$ yield only heads, where $1 \leq k \leq n$ and $1 \leq i \leq n - k + 1$ . Since coin flips are mutually independent, for any given event $A_{ik}$ , the probability that all $k$ flips are heads is

Pr {A_{ik}} = 1/ 2^{k} . (5.8)

For $k = 2 ⌈ l g n ⌉$ ,

Pr {A_{i, 2 ⌈ l g n ⌉}} = 1/ 2^{2 ⌈ l g n ⌉} \leq 1/ 2^{2 l g n} = 1/ n^{2},

and thus the probability that a streak of heads of length at least $2 ⌈ l g n ⌉$ begins in position $i$ is quite small. There are at most $n - 2 ⌈ l g n ⌉ + 1$ positions where such a streak can begin. The probability that a streak of heads of length at least $2 ⌈ l g n ⌉$ begins anywhere is therefore

Pr ⎩ ⎨ ⎧ i = 1 ⋃ n - 2 ⌈ l g n ⌉ + 1 A_{i, 2 ⌈ l g n ⌉} ⎭ ⎬ ⎫ \leq i = 1 \sum n - 2 ⌈ l g n ⌉ + 1 1/ n^{2} \leq i = 1 \sum n 1/ n^{2} = 1/ n, (5.9)

since by Boole’s inequality (C.19), the probability of a union of events is at most the sum of the probabilities of the individual events. (Note that Boole’s inequality holds even for events such as these that are not independent.)

We now use inequality (5.9) to bound the length of the longest streak. For $j = 0, 1, 2, \dots, n$ , let $L_{j}$ be the event that the longest streak of heads has length exactly $j$ , and let $L$ be the length of the longest streak. By the definition of expected value, we have

E [L] = j = 0 \sum n j Pr {L_{j}} . (5.10)

We could try to evaluate this sum using upper bounds on each $Pr {L_{j}}$ similar to those computed in inequality (5.9). Unfortunately, this method would yield weak bounds. We can use some intuition gained by the above analysis to obtain a good bound, however. Informally, we observe that for no individual term in the summation in equation (5.10) are both the factors $j$ and $Pr {L_{j}}$ large. Why? When $j \geq 2 ⌈ l g n ⌉$ , then $Pr {L_{j}}$ is very small, and when $j < 2 ⌈ l g n ⌉$ , then $j$ is fairly small. More formally, we note that the events $L_{j}$ for $j = 0, 1, \dots, n$ are disjoint, and so the probability that a streak of heads of length at least $2 ⌈ l g n ⌉$ begins anywhere is $\sum_{j = 2 ⌈ l g n ⌉}^{n} Pr {L_{j}} < 1/ n$ by inequality (5.9). Using this bound, we obtain

E [L] = j = 0 \sum n j Pr {L_{j}} = j = 0 \sum 2 ⌈ l g n ⌉ - 1 j Pr {L_{j}} + j = 2 ⌈ l g n ⌉ \sum n j Pr {L_{j}} < j = 0 \sum 2 ⌈ l g n ⌉ - 1 (2 ⌈ l g n ⌉) Pr {L_{j}} + j = 2 ⌈ l g n ⌉ \sum n n Pr {L_{j}} = 2 ⌈ l g n ⌉ j = 0 \sum 2 ⌈ l g n ⌉ - 1 Pr {L_{j}} + n j = 2 ⌈ l g n ⌉ \sum n Pr {L_{j}} < 2 ⌈ l g n ⌉ \cdot 1 + n \cdot (1/ n) = O (l g n) . (splitting the sum) (since j < 2 ⌈ l g n ⌉ in the first sum and j \leq n in the second) (factoring out constants) (the probabilities sum to at most 1; the tail sum is < 1/ n)

The probability that a streak of heads exceeds $r ⌈ l g n ⌉$ flips diminishes quickly with $r$ . For $r \geq 1$ , the probability that a streak of at least $r ⌈ l g n ⌉$ heads starts in position $i$ is

Pr {A_{i, r ⌈ l g n ⌉}} = 1/ 2^{r ⌈ l g n ⌉} \leq 1/ n^{r} .

Thus, the probability is at most $n / n^{r} = 1/ n^{r - 1}$ that the longest streak is at least $r ⌈ l g n ⌉$ , or equivalently, the probability is at least $1 - 1/ n^{r - 1}$ that the longest streak has length less than $r ⌈ l g n ⌉$ .

As an example, for $n = 1000$ coin flips, the probability of having a streak of at least $2 ⌈ l g n ⌉ = 20$ heads is at most $1/ n = 1/1000$ . The chance of having a streak longer than $3 ⌈ l g n ⌉ = 30$ heads is at most $1/ n^{2} = 1/1, 000, 000$ .

We now prove a complementary lower bound: the expected length of the longest streak of heads in $n$ coin flips is $Ω (l g n)$ . To prove this bound, we look for streaks of length $s$ by partitioning the $n$ flips into approximately $n / s$ groups of $s$ flips each. If we choose $s = ⌊(l g n) /2 ⌋$ , we can show that it is likely that at least one of these groups comes up all heads, and hence it is likely that the longest streak has length at least $s = Ω (l g n)$ . We then show that the longest streak has expected length $Ω (l g n)$ .

We partition the $n$ coin flips into at least $⌊ n / ⌊(l g n) /2 ⌋⌋$ groups of $⌊(l g n) /2 ⌋$ consecutive flips, and we bound the probability that no group comes up all heads. By equation (5.8), the probability that the group starting in position $i$ comes up all heads is

Pr {A_{i, ⌊(l g n) /2 ⌋}} = 1/ 2^{⌊(l g n) /2 ⌋} \geq 1/ n .

The probability that a streak of heads of length at least $⌊(l g n) /2 ⌋$ does not begin in position $i$ is therefore at most $1 - 1/ n$ . Since the $⌊ n / ⌊(l g n) /2 ⌋⌋$ groups are formed from mutually exclusive, independent coin flips, the probability that every one of these groups fails to be a streak of length $⌊(l g n) /2 ⌋$ is at most

(1 - 1/ n)^{⌊ n / ⌊(l g n) /2 ⌋⌋} \leq (1 - 1/ n)^{n / ⌊(l g n) /2 ⌋ - 1} \leq (1 - 1/ n)^{2 n / l g n - 1} \leq e^{- (2 n / l g n - 1) / n} = O (e^{- l g n}) = O (1/ n) . (by 1 + x \leq e^{x})

For this argument, we used inequality (3.12), $1 + x \leq e^{x}$ , and the fact, which you might want to verify, that $(2 n / l g n - 1) / n \geq l g n$ for sufficiently large $n$ .

Thus, the probability that the longest streak equals or exceeds $⌊(l g n) /2 ⌋$ is

j = ⌊(l g n) /2 ⌋ \sum n Pr {L_{j}} \geq 1 - O (1/ n) . (5.11)

We can now calculate a lower bound on the expected length of the longest streak, beginning with equation (5.10) and proceeding in a manner similar to our analysis of the upper bound:

E [L] = j = 0 \sum n j Pr {L_{j}} = j = 0 \sum ⌊(l g n) /2 ⌋ - 1 j Pr {L_{j}} + j = ⌊(l g n) /2 ⌋ \sum n j Pr {L_{j}} \geq j = 0 \sum ⌊(l g n) /2 ⌋ - 1 0 \cdot Pr {L_{j}} + j = ⌊(l g n) /2 ⌋ \sum n ⌊(l g n) /2 ⌋ Pr {L_{j}} = 0 + ⌊(l g n) /2 ⌋ j = ⌊(l g n) /2 ⌋ \sum n Pr {L_{j}} \geq ⌊(l g n) /2 ⌋ (1 - O (1/ n)) = Ω (l g n) . (splitting the sum) (since j \geq 0 in the first sum and j \geq ⌊(l g n) /2 ⌋ in the second) (by inequality (5.11))

As with the birthday paradox, we can obtain a simpler but approximate analysis using indicator random variables. We let $X_{ik} = I {A_{ik}}$ be the indicator random variable associated with a streak of heads of length at least $k$ beginning with the $i$ th coin flip. To count the total number of such streaks, we define

X = i = 1 \sum n - k + 1 X_{ik} .

Taking expectations and using linearity of expectation, we have

E [X] = E [i = 1 \sum n - k + 1 X_{ik}] = i = 1 \sum n - k + 1 E [X_{ik}] = i = 1 \sum n - k + 1 Pr {A_{ik}} = i = 1 \sum n - k + 1 1/ 2^{k} = \frac{n - k + 1}{2 ^{k}} . (by linearity of expectation) (by Lemma 5.1) (by equation (5.8))

By plugging in various values for $k$ , we can calculate the expected number of streaks of length $k$ . If this number is large (much greater than $1$ ), then we expect many streaks of length $k$ to occur and the probability that one occurs is high. If this number is small (much less than $1$ ), then we expect few streaks of length $k$ to occur and the probability that one occurs is low. If $k = c l g n$ , for some positive constant $c$ , we obtain

E [X] = \frac{n - c l g n + 1}{2 ^{c l g n}} = \frac{n - c l g n + 1}{n ^{c}} = \frac{1}{n ^{c - 1}} - \frac{c l g n - 1}{n ^{c}} = Θ (1/ n^{c - 1}) .

If $c$ is large, the expected number of streaks of length $c l g n$ is small, and we conclude that they are unlikely to occur. On the other hand, if $c = 1/2$ , then we obtain $E [X] = Θ (1/ n^{1/2 - 1}) = Θ (n^{1/2})$ , and we expect that there are a large number of streaks of length $(1/2) l g n$ . Therefore, one streak of such a length is likely to occur. From these rough estimates alone, we can conclude that the expected length of the longest streak is $Θ (l g n)$ .

5.4.4 The on-line hiring problem

As a final example, we consider a variant of the hiring problem. Suppose now that we do not wish to interview all the candidates in order to find the best one. We also do not wish to hire and fire as we find better and better applicants. Instead, we are willing to settle for a candidate who is close to the best, in exchange for hiring exactly once. We must obey one company requirement: after each interview we must either immediately offer the position to the applicant or immediately reject the applicant. What is the trade-off between minimizing the amount of interviewing and maximizing the quality of the candidate hired?

We can model this problem in the following way. After meeting an applicant, we are able to give each one a score; let $score (i)$ denote the score we give to the $i$ th applicant, and assume that no two applicants receive the same score. After we have seen $j$ applicants, we know which of the $j$ has the highest score, but we do not know whether any of the remaining $n - j$ applicants will receive a higher score. We decide to adopt the strategy of selecting a positive integer $k < n$ , interviewing and then rejecting the first $k$ applicants, and hiring the first applicant thereafter who has a higher score than all preceding applicants. If it turns out that the best-qualified applicant was among the first $k$ interviewed, then we hire the $n$ th applicant. We formalize this strategy in the procedure ON-LINE-MAXIMUM $(k, n)$ , which returns the index of the candidate we wish to hire.

ON-LINE-MAXIMUM(k, n)
1  bestscore = -∞
2  for i = 1 to k
3      if score(i) > bestscore
4          bestscore = score(i)
5  for i = k + 1 to n
6      if score(i) > bestscore
7          return i
8  return n

We wish to determine, for each possible value of $k$ , the probability that we hire the most qualified applicant. We then choose the best possible $k$ , and implement the strategy with that value. For the moment, assume that $k$ is fixed. Let $M (j) = max_{1 \leq i \leq j} {score (i)}$ denote the maximum score among applicants $1$ through $j$ . Let $S$ be the event that we succeed in choosing the best-qualified applicant, and let $S_{i}$ be the event that we succeed when the best-qualified applicant is the $i$ th one interviewed. Since the various $S_{i}$ are disjoint, we have that $Pr {S} = \sum_{i = 1}^{n} Pr {S_{i}}$ . Noting that we never succeed when the best-qualified applicant is one of the first $k$ , we have that $Pr {S_{i}} = 0$ for $i = 1, 2, \dots, k$ . Thus, we obtain

Pr {S} = i = k + 1 \sum n Pr {S_{i}} . (5.12)

We now compute $Pr {S_{i}}$ . In order to succeed when the best-qualified applicant is the $i$ th one, two things must happen. First, the best-qualified applicant must be in position $i$ , an event which we denote by $B_{i}$ . Second, the algorithm must not select any of the applicants in positions $k + 1$ through $i - 1$ , which happens only if, for each $j$ such that $k + 1 \leq j \leq i - 1$ , we find that $score (j) < bestscore$ in line 6. (Because scores are unique, we can ignore the possibility of $score (j) = bestscore$ .) In other words, all of the values $score (k + 1)$ through $score (i - 1)$ must be less than $M (k)$ ; if any are greater than $M (k)$ , we instead return the index of the first one that is greater. We use $O_{i}$ to denote the event that none of the applicants in position $k + 1$ through $i - 1$ are chosen. Fortunately, the two events $B_{i}$ and $O_{i}$ are independent. The event $O_{i}$ depends only on the relative ordering of the values in positions $1$ through $i - 1$ , whereas $B_{i}$ depends only on whether the value in position $i$ is greater than the values in all other positions. The ordering of the values in positions $1$ through $i - 1$ does not affect whether the value in position $i$ is greater than all of them, and the value in position $i$ does not affect the ordering of the values in positions $1$ through $i - 1$ . Thus we can apply equation (C.15) to obtain

Pr {S_{i}} = Pr {B_{i} \cap O_{i}} = Pr {B_{i}} Pr {O_{i}} .

The probability $Pr {B_{i}}$ is clearly $1/ n$ , since the maximum is equally likely to be in any one of the $n$ positions. For event $O_{i}$ to occur, the maximum value in positions $1$ through $i - 1$ , which is equally likely to be in any of these $i - 1$ positions, must be in one of the first $k$ positions. Consequently, $Pr {O_{i}} = k / (i - 1)$ and $Pr {S_{i}} = k / (n (i - 1))$ . Using equation (5.12), we have

Pr {S} = i = k + 1 \sum n Pr {S_{i}} = i = k + 1 \sum n \frac{k}{n ( i - 1 )} = \frac{k}{n} i = k + 1 \sum n \frac{1}{i - 1} = \frac{k}{n} i = k \sum n - 1 \frac{1}{i} (shifting the index) .

We approximate by integrals to bound this summation from above and below. By the inequalities (A.12), we have

\int_{k}^{n} \frac{1}{x} d x \leq i = k \sum n - 1 \frac{1}{i} \leq \int_{k - 1}^{n - 1} \frac{1}{x} d x .

Evaluating these definite integrals gives us the bounds

\frac{k}{n} (ln n - ln k) \leq Pr {S} \leq \frac{k}{n} (ln (n - 1) - ln (k - 1)),

which provide a rather tight bound for $Pr {S}$ . Because we wish to maximize our probability of success, let us focus on choosing the value of $k$ that maximizes the lower bound on $Pr {S}$ . (Besides, the lower-bound expression is easier to maximize than the upper-bound expression.) Differentiating the expression $(k / n) (ln n - ln k)$ with respect to $k$ , we obtain

\frac{d}{d k} [\frac{k}{n} (ln n - ln k)] = \frac{1}{n} (ln n - ln k) + \frac{k}{n} (- \frac{1}{k}) = \frac{1}{n} (ln n - ln k - 1) . (product rule)

Setting this derivative equal to $0$ , we see that we maximize the lower bound on the probability when $ln k = ln n - 1 = ln (n / e)$ or, equivalently, when $k = n / e$ . Thus, if we implement our strategy with $k = n / e$ , we succeed in hiring our best-qualified applicant with probability at least $1/ e$ .

Exercises 5.4-1 to 5.4-7

5.4-1 How many people must there be in a room before the probability that someone has the same birthday as you do is at least $1/2$ ? How many people must there be before the probability that at least two people have a birthday on July 4 is greater than $1/2$ ?

5.4-2 Suppose that we toss balls into $b$ bins until some bin contains two balls. Each toss is independent, and each ball is equally likely to end up in any bin. What is the expected number of ball tosses?

5.4-3* For the analysis of the birthday paradox, is it important that the birthdays be mutually independent, or is pairwise independence sufficient? Justify your answer.

5.4-4* How many people should be invited to a party in order to make it likely that there are three people with the same birthday?

5.4-5* What is the probability that a $k$ -string over a set of size $n$ forms a $k$ -permutation? How does this question relate to the birthday paradox?

5.4-6* Suppose that $n$ balls are tossed into $n$ bins, where each toss is independent and the ball is equally likely to end up in any bin. What is the expected number of empty bins? What is the expected number of bins with exactly one ball?

5.4-7* Sharpen the lower bound on streak length by showing that in $n$ flips of a fair coin, the probability is less than $1/ n$ that no streak longer than $l g n - 2 l g l g n$ consecutive heads occurs.

Problems

5-1 Probabilistic counting

With a $b$ -bit counter, we can ordinarily only count up to $2^{b} - 1$ . With R. Morris’s probabilistic counting, we can count up to a much larger value at the expense of some loss of precision.

We let a counter value of $i$ represent a count of $n_{i}$ for $i = 0, 1, \dots, 2^{b} - 1$ , where the $n_{i}$ form an increasing sequence of nonnegative values. We assume that the initial value of the counter is $0$ , representing a count of $n_{0} = 0$ . The INCREMENT operation works on a counter containing the value $i$ in a probabilistic manner. If $i = 2^{b} - 1$ , then the operation reports an overflow error. Otherwise, the INCREMENT operation increases the counter by $1$ with probability $1/ (n_{i + 1} - n_{i})$ , and it leaves the counter unchanged with probability $1 - 1/ (n_{i + 1} - n_{i})$ .

If we select $n_{i} = i$ for all $i \geq 0$ , then the counter is an ordinary one. More interesting situations arise if we select, say, $n_{i} = 2^{i - 1}$ for $i > 0$ or $n_{i} = F_{i}$ (the $i$ th Fibonacci number; see Section 3.2).

For this problem, assume that $n_{2^{b} - 1}$ is large enough that the probability of an overflow error is negligible.

a. Show that the expected value represented by the counter after $n$ INCREMENT operations have been performed is exactly $n$ .

b. The analysis of the variance of the count represented by the counter depends on the sequence of the $n_{i}$ . Let us consider a simple case: $n_{i} = 100 i$ for all $i \geq 0$ . Estimate the variance in the value represented by the register after $n$ INCREMENT operations have been performed.

5-2 Searching an unsorted array

This problem examines three algorithms for searching for a value $x$ in an unsorted array $A$ consisting of $n$ elements.

Consider the following randomized strategy: pick a random index $i$ into $A$ . If $A [i] = x$ , then we terminate; otherwise, we continue the search by picking a new random index into $A$ . We continue picking random indices into $A$ until we find an index $j$ such that $A [j] = x$ or until we have checked every element of $A$ . Note that we pick from the whole set of indices each time, so that we may examine a given element more than once.

a. Write pseudocode for a procedure RANDOM-SEARCH to implement the strategy above. Be sure that your algorithm terminates when all indices into $A$ have been picked.

b. Suppose that there is exactly one index $i$ such that $A [i] = x$ . What is the expected number of indices into $A$ that we must pick before we find $x$ and RANDOM-SEARCH terminates?

c. Generalizing your solution to part (b), suppose that there are $k \geq 1$ indices $i$ such that $A [i] = x$ . What is the expected number of indices into $A$ that we must pick before we find $x$ and RANDOM-SEARCH terminates? Your answer should be a function of $n$ and $k$ .

d. Suppose that there are no indices $i$ such that $A [i] = x$ . What is the expected number of indices into $A$ that we must pick before we have checked all elements of $A$ and RANDOM-SEARCH terminates?

Now consider a deterministic linear search algorithm, which we refer to as DETERMINISTIC-SEARCH. Specifically, the algorithm searches $A$ for $x$ in order, considering $A [1], A [2], A [3], \dots, A [n]$ until either it finds $A [i] = x$ or it reaches the end of the array. Assume that all possible permutations of the input array are equally likely.

e. Suppose that there is exactly one index $i$ such that $A [i] = x$ . What is the average-case running time of DETERMINISTIC-SEARCH? What is the worst-case running time of DETERMINISTIC-SEARCH?

f. Generalizing your solution to part (e), suppose that there are $k \geq 1$ indices $i$ such that $A [i] = x$ . What is the average-case running time of DETERMINISTIC-SEARCH? What is the worst-case running time of DETERMINISTIC-SEARCH? Your answer should be a function of $n$ and $k$ .

g. Suppose that there are no indices $i$ such that $A [i] = x$ . What is the average-case running time of DETERMINISTIC-SEARCH? What is the worst-case running time of DETERMINISTIC-SEARCH?

Finally, consider a randomized algorithm SCRAMBLE-SEARCH that works by first randomly permuting the input array and then running the deterministic linear search given above on the resulting permuted array.

h. Letting $k$ be the number of indices $i$ such that $A [i] = x$ , give the worst-case and expected running times of SCRAMBLE-SEARCH for the cases in which $k = 0$ and $k = 1$ . Generalize your solution to handle the case in which $k \geq 1$ .

i. Which of the three searching algorithms would you use? Explain your answer.

Chapter notes

Bollobás [53], Hofri [174], and Spencer [321] contain a wealth of advanced probabilistic techniques. The advantages of randomized algorithms are discussed and surveyed by Karp [200] and Rabin [288]. The textbook by Motwani and Raghavan [262] gives an extensive treatment of randomized algorithms.

Several variants of the hiring problem have been widely studied. These problems are more commonly referred to as “secretary problems.” An example of work in this area is the paper by Ajtai, Meggido, and Waarts [11].

Part II: Sorting and Order Statistics

Introduction

Big picture. Imagine a library where every book has been thrown into one giant pile. Finding anything is hopeless, yet once the books are shelved in order, a search takes seconds. Sorting is the art of paying a one-time cost to make the world easy to query. In this part we meet several ways of doing it, discover how fast it can possibly be done, and then ask a sneaky follow-up: if we only want the $i$ th smallest item, do we really need to sort everything?

This part presents several algorithms that solve the following sorting problem:

Input: A sequence of $n$ numbers $⟨ a_{1}, a_{2}, \dots, a_{n} ⟩$ .

Output: A permutation (reordering) $⟨ a_{1}^{'}, a_{2}^{'}, \dots, a_{n}^{'} ⟩$ of the input sequence such that $a_{1}^{'} \leq a_{2}^{'} \leq \dots \leq a_{n}^{'}$ .

The input sequence is usually an $n$ -element array, although it may be represented in some other fashion, such as a linked list.

The structure of the data

In practice, the numbers to be sorted are rarely isolated values. Each is usually part of a collection of data called a record. Each record contains a key, which is the value to be sorted. The remainder of the record consists of satellite data, which are usually carried around with the key. In practice, when a sorting algorithm permutes the keys, it must permute the satellite data as well. If each record includes a large amount of satellite data, we often permute an array of pointers to the records rather than the records themselves in order to minimize data movement.

In a sense, it is these implementation details that distinguish an algorithm from a full-blown program. A sorting algorithm describes the method by which we determine the sorted order, regardless of whether we are sorting individual numbers or large records containing many bytes of satellite data. Thus, when focusing on the problem of sorting, we typically assume that the input consists only of numbers. Translating an algorithm for sorting numbers into a program for sorting records is conceptually straightforward, although in a given engineering situation other subtleties may make the actual programming task a challenge.

Why sorting?

Many computer scientists consider sorting to be the most fundamental problem in the study of algorithms. There are several reasons:

Sometimes an application inherently needs to sort information. For example, in order to prepare customer statements, banks need to sort checks by check number.
Algorithms often use sorting as a key subroutine. For example, a program that renders graphical objects which are layered on top of each other might have to sort the objects according to an “above” relation so that it can draw these objects from bottom to top. We shall see numerous algorithms in this text that use sorting as a subroutine.
We can draw from among a wide variety of sorting algorithms, and they employ a rich set of techniques. In fact, many important techniques used throughout algorithm design appear in the body of sorting algorithms that have been developed over the years. In this way, sorting is also a problem of historical interest.
We can prove a nontrivial lower bound for sorting (as we shall do in Chapter 8). Our best upper bounds match the lower bound asymptotically, and so we know that our sorting algorithms are asymptotically optimal. Moreover, we can use the lower bound for sorting to prove lower bounds for certain other problems.
Many engineering issues come to the fore when implementing sorting algorithms. The fastest sorting program for a particular situation may depend on many factors, such as prior knowledge about the keys and satellite data, the memory hierarchy (caches and virtual memory) of the host computer, and the software environment. Many of these issues are best dealt with at the algorithmic level, rather than by “tweaking” the code.

Sorting algorithms

We introduced two algorithms that sort $n$ real numbers in Chapter 2. Insertion sort takes $Θ (n^{2})$ time in the worst case. Because its inner loops are tight, however, it is a fast in-place sorting algorithm for small input sizes. (Recall that a sorting algorithm sorts in place if only a constant number of elements of the input array are ever stored outside the array.) Merge sort has a better asymptotic running time, $Θ (n l g n)$ , but the MERGE procedure it uses does not operate in place.

In this part, we shall introduce two more algorithms that sort arbitrary real numbers. Heapsort, presented in Chapter 6, sorts $n$ numbers in place in $O (n l g n)$ time. It uses an important data structure, called a heap, with which we can also implement a priority queue.

Quicksort, in Chapter 7, also sorts $n$ numbers in place, but its worst-case running time is $Θ (n^{2})$ . Its expected running time is $Θ (n l g n)$ , however, and it generally outperforms heapsort in practice. Like insertion sort, quicksort has tight code, and so the hidden constant factor in its running time is small. It is a popular algorithm for sorting large input arrays.

Insertion sort, merge sort, heapsort, and quicksort are all comparison sorts: they determine the sorted order of an input array by comparing elements. Chapter 8 begins by introducing the decision-tree model in order to study the performance limitations of comparison sorts. Using this model, we prove a lower bound of $Ω (n l g n)$ on the worst-case running time of any comparison sort on $n$ inputs, thus showing that heapsort and merge sort are asymptotically optimal comparison sorts.

Chapter 8 then goes on to show that we can beat this lower bound of $Ω (n l g n)$ if we can gather information about the sorted order of the input by means other than comparing elements. The counting sort algorithm, for example, assumes that the input numbers are in the set ${0, 1, \dots, k}$ . By using array indexing as a tool for determining relative order, counting sort can sort $n$ numbers in $Θ (k + n)$ time. Thus, when $k = O (n)$ , counting sort runs in time that is linear in the size of the input array. A related algorithm, radix sort, can be used to extend the range of counting sort. If there are $n$ integers to sort, each integer has $d$ digits, and each digit can take on up to $k$ possible values, then radix sort can sort the numbers in $Θ (d (n + k))$ time. When $d$ is a constant and $k$ is $O (n)$ , radix sort runs in linear time. A third algorithm, bucket sort, requires knowledge of the probabilistic distribution of numbers in the input array. It can sort $n$ real numbers uniformly distributed in the half-open interval $[0, 1)$ in average-case $O (n)$ time.

The following table summarizes the running times of the sorting algorithms from Chapters 2 and 6-8. As usual, $n$ denotes the number of items to sort. For counting sort, the items to sort are integers in the set ${0, 1, \dots, k}$ . For radix sort, each item is a $d$ -digit number, where each digit takes on $k$ possible values. For bucket sort, we assume that the keys are real numbers uniformly distributed in the half-open interval $[0, 1)$ . The rightmost column gives the average-case or expected running time, indicating which it gives when it differs from the worst-case running time. We omit the average-case running time of heapsort because we do not analyze it in this book.

Algorithm	Worst-case running time	Average-case / expected running time
Insertion sort	$Θ (n^{2})$	$Θ (n^{2})$
Merge sort	$Θ (n l g n)$	$Θ (n l g n)$
Heapsort	$O (n l g n)$
Quicksort	$Θ (n^{2})$	$Θ (n l g n)$ (expected)
Counting sort	$Θ (k + n)$	$Θ (k + n)$
Radix sort	$Θ (d (n + k))$	$Θ (d (n + k))$
Bucket sort	$Θ (n^{2})$	$Θ (n)$ (average-case)

Order statistics

The $i$ th order statistic of a set of $n$ numbers is the $i$ th smallest number in the set. We can, of course, select the $i$ th order statistic by sorting the input and indexing the $i$ th element of the output. With no assumptions about the input distribution, this method runs in $Ω (n l g n)$ time, as the lower bound proved in Chapter 8 shows.

In Chapter 9, we show that we can find the $i$ th smallest element in $O (n)$ time, even when the elements are arbitrary real numbers. We present a randomized algorithm with tight pseudocode that runs in $Θ (n^{2})$ time in the worst case, but whose expected running time is $O (n)$ . We also give a more complicated algorithm that runs in $O (n)$ worst-case time.

Background

Although most of this part does not rely on difficult mathematics, some sections do require mathematical sophistication. In particular, analyses of quicksort, bucket sort, and the order-statistic algorithm use probability, which is reviewed in Appendix C, and the material on probabilistic analysis and randomized algorithms in Chapter 5. The analysis of the worst-case linear-time algorithm for order statistics involves somewhat more sophisticated mathematics than the other worst-case analyses in this part.

6 Heapsort

Big picture. Picture a tournament bracket in which the champion always sits at the top, and whenever the champion leaves, a quick round of “who is best now?” matches crowns a successor. A heap is exactly that bracket, squeezed into a plain array. Can we get merge sort’s speed and insertion sort’s frugal use of memory? Heapsort says yes.

In this chapter, we introduce another sorting algorithm: heapsort. Like merge sort, but unlike insertion sort, heapsort’s running time is $O (n l g n)$ . Like insertion sort, but unlike merge sort, heapsort sorts in place: only a constant number of array elements are stored outside the input array at any time. Thus, heapsort combines the better attributes of the two sorting algorithms we have already discussed.

Heapsort also introduces another algorithm design technique: using a data structure, in this case one we call a “heap,” to manage information. Not only is the heap data structure useful for heapsort, but it also makes an efficient priority queue. The heap data structure will reappear in algorithms in later chapters.

The term “heap” was originally coined in the context of heapsort, but it has since come to refer to “garbage-collected storage,” such as the programming languages Java and Lisp provide. Our heap data structure is not garbage-collected storage, and whenever we refer to heaps in this book, we shall mean a data structure rather than an aspect of garbage collection.

6.1 Heaps

The (binary) heap data structure is an array object that we can view as a nearly complete binary tree (see Section B.5.3), as shown in Figure 6.1. Each node of the tree corresponds to an element of the array. The tree is completely filled on all levels except possibly the lowest, which is filled from the left up to a point. An array $A$ that represents a heap is an object with two attributes: $A . length$ , which (as usual) gives the number of elements in the array, and $A . heap-size$ , which represents how many elements in the heap are stored within array $A$ . That is, although $A [1 \dots A . length]$ may contain numbers, only the elements in $A [1 \dots A . heap-size]$ , where $0 \leq A . heap-size \leq A . length$ , are valid elements of the heap. The root of the tree is $A [1]$ , and given the index $i$ of a node, we can easily compute the indices of its parent, left child, and right child:

PARENT(i)
1  return ⌊i/2⌋
 
LEFT(i)
1  return 2i
 
RIGHT(i)
1  return 2i + 1

Figure 6.1: A max-heap viewed as (a) a binary tree and (b) an array. The array is $A = ⟨ 16, 14, 10, 8, 7, 9, 3, 2, 4, 1 ⟩$ at indices $1, \dots, 10$ . The number within the circle at each node is the value stored there; the number above a node is its array index. Parents are always to the left of their children in the array. The tree has height three; the node at index 4 (with value 8) has height one.

On most computers, the LEFT procedure can compute $2 i$ in one instruction by simply shifting the binary representation of $i$ left by one bit position. Similarly, the RIGHT procedure can quickly compute $2 i + 1$ by shifting the binary representation of $i$ left by one bit position and then adding in a 1 as the low-order bit. The PARENT procedure can compute $⌊ i /2 ⌋$ by shifting $i$ right one bit position. Good implementations of heapsort often implement these procedures as “macros” or “inline” procedures.

There are two kinds of binary heaps: max-heaps and min-heaps. In both kinds, the values in the nodes satisfy a heap property, the specifics of which depend on the kind of heap. In a max-heap, the max-heap property is that for every node $i$ other than the root,

$A [PARENT (i)] \geq A [i],$

that is, the value of a node is at most the value of its parent. Thus, the largest element in a max-heap is stored at the root, and the subtree rooted at a node contains values no larger than that contained at the node itself. A min-heap is organized in the opposite way; the min-heap property is that for every node $i$ other than the root,

$A [PARENT (i)] \leq A [i] .$

The smallest element in a min-heap is at the root.

For the heapsort algorithm, we use max-heaps. Min-heaps commonly implement priority queues, which we discuss in Section 6.5. We shall be precise in specifying whether we need a max-heap or a min-heap for any particular application, and when properties apply to either max-heaps or min-heaps, we just use the term “heap.”

Viewing a heap as a tree, we define the height of a node in a heap to be the number of edges on the longest simple downward path from the node to a leaf, and we define the height of the heap to be the height of its root. Since a heap of $n$ elements is based on a complete binary tree, its height is $Θ (l g n)$ (see Exercise 6.1-2). We shall see that the basic operations on heaps run in time at most proportional to the height of the tree and thus take $O (l g n)$ time. The remainder of this chapter presents some basic procedures and shows how they are used in a sorting algorithm and a priority-queue data structure.

The MAX-HEAPIFY procedure, which runs in $O (l g n)$ time, is the key to maintaining the max-heap property.
The BUILD-MAX-HEAP procedure, which runs in linear time, produces a max-heap from an unordered input array.
The HEAPSORT procedure, which runs in $O (n l g n)$ time, sorts an array in place.
The MAX-HEAP-INSERT, HEAP-EXTRACT-MAX, HEAP-INCREASE-KEY, and HEAP-MAXIMUM procedures, which run in $O (l g n)$ time, allow the heap data structure to implement a priority queue.

Think about it. Why does the array layout work so well? Because the tree is “nearly complete,” there are no gaps, so index arithmetic ( $2 i$ , $2 i + 1$ , $⌊ i /2 ⌋$ ) replaces pointers entirely.

Exercises 6.1-1 to 6.1-7

6.1-1 What are the minimum and maximum numbers of elements in a heap of height $h$ ?

6.1-2 Show that an $n$ -element heap has height $⌊ l g n ⌋$ .

6.1-3 Show that in any subtree of a max-heap, the root of the subtree contains the largest value occurring anywhere in that subtree.

6.1-4 Where in a max-heap might the smallest element reside, assuming that all elements are distinct?

6.1-5 Is an array that is in sorted order a min-heap?

6.1-6 Is the array with values $⟨ 23, 17, 14, 6, 13, 10, 1, 5, 7, 12 ⟩$ a max-heap?

6.1-7 Show that, with the array representation for storing an $n$ -element heap, the leaves are the nodes indexed by $⌊ n /2 ⌋ + 1, ⌊ n /2 ⌋ + 2, \dots, n$ .

6.2 Maintaining the heap property

In order to maintain the max-heap property, we call the procedure MAX-HEAPIFY. Its inputs are an array $A$ and an index $i$ into the array. When it is called, MAX-HEAPIFY assumes that the binary trees rooted at LEFT( $i$ ) and RIGHT( $i$ ) are max-heaps, but that $A [i]$ might be smaller than its children, thus violating the max-heap property. MAX-HEAPIFY lets the value at $A [i]$ “float down” in the max-heap so that the subtree rooted at index $i$ obeys the max-heap property.

MAX-HEAPIFY(A, i)
1   l = LEFT(i)
2   r = RIGHT(i)
3   if l ≤ A.heap-size and A[l] > A[i]
4       largest = l
5   else largest = i
6   if r ≤ A.heap-size and A[r] > A[largest]
7       largest = r
8   if largest ≠ i
9       exchange A[i] with A[largest]
10      MAX-HEAPIFY(A, largest)

Figure 6.2 illustrates the action of MAX-HEAPIFY. At each step, the largest of the elements $A [i]$ , $A [LEFT (i)]$ , and $A [RIGHT (i)]$ is determined, and its index is stored in largest. If $A [i]$ is largest, then the subtree rooted at node $i$ is already a max-heap and the procedure terminates. Otherwise, one of the two children has the largest element, and $A [i]$ is swapped with $A [largest]$ , which causes node $i$ and its children to satisfy the max-heap property. The node indexed by largest, however, now has the original value $A [i]$ , and thus the subtree rooted at largest might violate the max-heap property. Consequently, we call MAX-HEAPIFY recursively on that subtree.

Figure 6.2: The action of MAX-HEAPIFY( $A$ , 2), where $A . heap-size = 10$ . (a) The initial configuration $A = ⟨ 16, 4, 10, 14, 7, 9, 3, 2, 8, 1 ⟩$ , with $A [2]$ at node $i = 2$ violating the max-heap property since it is not larger than both children. (b) The max-heap property is restored for node 2 by exchanging $A [2]$ with $A [4]$ , which destroys the max-heap property for node 4. The recursive call MAX-HEAPIFY( $A$ , 4) now has $i = 4$ . (c) After swapping $A [4]$ with $A [9]$ , node 4 is fixed up, and the recursive call MAX-HEAPIFY( $A$ , 9) yields no further change to the data structure.

Running time of MAX-HEAPIFY. On a subtree of size $n$ rooted at node $i$ , the cost is the $Θ (1)$ time to fix up the relationships among $A [i]$ , $A [LEFT (i)]$ , and $A [RIGHT (i)]$ , plus the time to run MAX-HEAPIFY on a subtree rooted at one of the children of node $i$ (assuming that the recursive call occurs). The children’s subtrees each have size at most $2 n /3$ (the worst case occurs when the bottom level of the tree is exactly half full), and therefore we can describe the running time by the recurrence

$T (n) \leq T (2 n /3) + Θ (1) .$

To solve it with the master theorem (Theorem 4.1), match $T (n) = a T (n / b) + f (n)$ :
$a &= 1, \quad b = 3/2, \quad f(n) = \Theta(1) \\ n^{\log_b a} &= n^{\log_{3/2} 1} = n^0 = 1 \\ f(n) &= \Theta(1) = \Theta\!\left(n^{\log_b a}\right) \quad \text{(case 2 applies)} \end{aligned}$$ Hence $$T(n) = \Theta(n^{\log_b a} \lg n) = \Theta(\lg n), \quad\text{so}\quad T(n) = O(\lg n).$$ Alternatively, we can characterize the running time of MAX-HEAPIFY on a node of height $h$ as $O(h)$.$

Exercises 6.2-1 to 6.2-6

6.2-1 Using Figure 6.2 as a model, illustrate the operation of MAX-HEAPIFY( $A$ , 3) on the array $A = ⟨ 27, 17, 3, 16, 13, 10, 1, 5, 7, 12, 4, 8, 9, 0 ⟩$ .

6.2-2 Starting with the procedure MAX-HEAPIFY, write pseudocode for the procedure MIN-HEAPIFY( $A$ , $i$ ), which performs the corresponding manipulation on a min-heap. How does the running time of MIN-HEAPIFY compare to that of MAX-HEAPIFY?

6.2-3 What is the effect of calling MAX-HEAPIFY( $A$ , $i$ ) when the element $A [i]$ is larger than its children?

6.2-4 What is the effect of calling MAX-HEAPIFY( $A$ , $i$ ) for $i > A . heap-size /2$ ?

6.2-5 The code for MAX-HEAPIFY is quite efficient in terms of constant factors, except possibly for the recursive call in line 10, which might cause some compilers to produce inefficient code. Write an efficient MAX-HEAPIFY that uses an iterative control construct (a loop) instead of recursion.

6.2-6 Show that the worst-case running time of MAX-HEAPIFY on a heap of size $n$ is $Ω (l g n)$ . (Hint: For a heap with $n$ nodes, give node values that cause MAX-HEAPIFY to be called recursively at every node on a simple path from the root down to a leaf.)

6.3 Building a heap

We can use the procedure MAX-HEAPIFY in a bottom-up manner to convert an array $A [1 \dots n]$ , where $n = A . length$ , into a max-heap. By Exercise 6.1-7, the elements in the subarray $A [(⌊ n /2 ⌋ + 1) \dots n]$ are all leaves of the tree, and so each is a 1-element heap to begin with. The procedure BUILD-MAX-HEAP goes through the remaining nodes of the tree and runs MAX-HEAPIFY on each one.

BUILD-MAX-HEAP(A)
1  A.heap-size = A.length
2  for i = ⌊A.length/2⌋ downto 1
3      MAX-HEAPIFY(A, i)

Figure 6.3 shows an example of the action of BUILD-MAX-HEAP.

Figure 6.3: The operation of BUILD-MAX-HEAP, showing the data structure before the call to MAX-HEAPIFY in line 3. (a) A 10-element input array $A = ⟨ 4, 1, 3, 2, 16, 9, 10, 14, 8, 7 ⟩$ and the binary tree it represents; the loop index $i$ refers to node 5 before the call MAX-HEAPIFY( $A$ , $i$ ). (b) The data structure that results; the loop index $i$ for the next iteration refers to node 4. (c)-(e) Subsequent iterations of the for loop. Observe that whenever MAX-HEAPIFY is called on a node, the two subtrees of that node are both max-heaps. (f) The max-heap after BUILD-MAX-HEAP finishes.

Correctness

To show why BUILD-MAX-HEAP works correctly, we use the following loop invariant:

Loop invariant. At the start of each iteration of the for loop of lines 2-3, each node $i + 1, i + 2, \dots, n$ is the root of a max-heap.

We need to show that this invariant is true prior to the first loop iteration, that each iteration of the loop maintains the invariant, and that the invariant provides a useful property to show correctness when the loop terminates.

Proof.

Initialization. Prior to the first iteration of the loop, $i = ⌊ n /2 ⌋$ . Each node $⌊ n /2 ⌋ + 1, ⌊ n /2 ⌋ + 2, \dots, n$ is a leaf (Exercise 6.1-7) and is thus the root of a trivial max-heap.

Maintenance. Observe that the children of node $i$ are numbered higher than $i$ . By the loop invariant, therefore, they are both roots of max-heaps. This is precisely the condition required for the call MAX-HEAPIFY( $A$ , $i$ ) to make node $i$ a max-heap root. Moreover, the MAX-HEAPIFY call preserves the property that nodes $i, i + 1, \dots, n$ are all roots of max-heaps. Decrementing $i$ in the for loop update reestablishes the loop invariant for the next iteration.

Termination. At termination, $i = 0$ . By the loop invariant, each node $1, 2, \dots, n$ is the root of a max-heap. In particular, node 1 is. $■$

Running time

We can compute a simple upper bound on the running time of BUILD-MAX-HEAP as follows. Each call to MAX-HEAPIFY costs $O (l g n)$ time, and BUILD-MAX-HEAP makes $O (n)$ such calls. Thus, the running time is $O (n l g n)$ . This upper bound, though correct, is not asymptotically tight.

We can derive a tighter bound by observing that the time for MAX-HEAPIFY to run at a node varies with the height of the node in the tree, and the heights of most nodes are small. Our tighter analysis relies on the properties that an $n$ -element heap has height $⌊ l g n ⌋$ (see Exercise 6.1-2) and at most $⌈ n / 2^{h + 1} ⌉$ nodes of any height $h$ (see Exercise 6.3-3).

Proof (BUILD-MAX-HEAP runs in $O (n)$ time). The time required by MAX-HEAPIFY when called on a node of height $h$ is $O (h)$ , and so the total cost of BUILD-MAX-HEAP is bounded from above by

$\sum_{h = 0}^{⌊ l g n ⌋} ⌈ \frac{n}{2 ^{h + 1}} ⌉ O (h) = O (n \sum_{h = 0}^{⌊ l g n ⌋} \frac{h}{2 ^{h}}) .$

We evaluate the last summation by substituting $x = 1/2$ in the formula (A.8), $\sum_{k = 0}^{\infty} k x^{k} = \frac{x}{( 1 - x ) ^{2}}$ for $∣ x ∣ < 1$ , yielding
$\sum_{h=0}^{\infty} \frac{h}{2^h} &= \frac{1/2}{(1 - 1/2)^2} \\ &= \frac{1/2}{1/4} \\ &= 2. \end{aligned}$$ Since the finite sum up to $\lfloor \lg n \rfloor$ is at most this infinite sum (all terms are nonnegative), we can bound the running time of BUILD-MAX-HEAP as $$\begin{aligned} O\!\left( n \sum_{h=0}^{\lfloor \lg n \rfloor} \frac{h}{2^h} \right) &\le O\!\left( n \sum_{h=0}^{\infty} \frac{h}{2^h} \right) \\ &= O(2n) \\ &= O(n). \end{aligned}$$ Hence, we can build a max-heap from an unordered array in linear time. $\blacksquare$$

Think about it. Why is the answer linear rather than $n l g n$ ? Half the nodes are leaves costing nothing, a quarter sit one level up costing one step, and so on: most of the work happens where the trees are tiny.

We can build a min-heap by the procedure BUILD-MIN-HEAP, which is the same as BUILD-MAX-HEAP but with the call to MAX-HEAPIFY in line 3 replaced by a call to MIN-HEAPIFY (see Exercise 6.2-2). BUILD-MIN-HEAP produces a min-heap from an unordered linear array in linear time.

Exercises 6.3-1 to 6.3-3

6.3-1 Using Figure 6.3 as a model, illustrate the operation of BUILD-MAX-HEAP on the array $A = ⟨ 5, 3, 17, 10, 84, 19, 6, 22, 9 ⟩$ .

6.3-2 Why do we want the loop index $i$ in line 2 of BUILD-MAX-HEAP to decrease from $⌊ A . length /2 ⌋$ to 1 rather than increase from 1 to $⌊ A . length /2 ⌋$ ?

6.3-3 Show that there are at most $⌈ n / 2^{h + 1} ⌉$ nodes of height $h$ in any $n$ -element heap.

6.4 The heapsort algorithm

The heapsort algorithm starts by using BUILD-MAX-HEAP to build a max-heap on the input array $A [1 \dots n]$ , where $n = A . length$ . Since the maximum element of the array is stored at the root $A [1]$ , we can put it into its correct final position by exchanging it with $A [n]$ . If we now discard node $n$ from the heap (and we can do so by simply decrementing $A . heap-size$ ), we observe that the children of the root remain max-heaps, but the new root element might violate the max-heap property. All we need to do to restore the max-heap property, however, is call MAX-HEAPIFY( $A$ , 1), which leaves a max-heap in $A [1 \dots n - 1]$ . The heapsort algorithm then repeats this process for the max-heap of size $n - 1$ down to a heap of size 2. (See Exercise 6.4-2 for a precise loop invariant.)

HEAPSORT(A)
1  BUILD-MAX-HEAP(A)
2  for i = A.length downto 2
3      exchange A[1] with A[i]
4      A.heap-size = A.heap-size - 1
5      MAX-HEAPIFY(A, 1)

Figure 6.4 shows an example of the operation of HEAPSORT after line 1 has built the initial max-heap. The figure shows the max-heap before the first iteration of the for loop of lines 2-5 and after each iteration.

Figure 6.4: The operation of HEAPSORT. (a) The max-heap data structure just after BUILD-MAX-HEAP has built it in line 1. (b)-(j) The max-heap just after each call of MAX-HEAPIFY in line 5, showing the value of $i$ at that time. Only lightly shaded nodes remain in the heap. (k) The resulting sorted array $A$ .

Running time. The HEAPSORT procedure takes time $O (n l g n)$ , since the call to BUILD-MAX-HEAP takes time $O (n)$ and each of the $n - 1$ calls to MAX-HEAPIFY takes time $O (l g n)$ :
$T(n) &= O(n) + (n-1)\cdot O(\lg n) \\ &= O(n) + O(n \lg n) \\ &= O(n \lg n). \end{aligned}$$$

Exercises 6.4-1 to 6.4-5

6.4-1 Using Figure 6.4 as a model, illustrate the operation of HEAPSORT on the array $A = ⟨ 5, 13, 2, 25, 7, 17, 20, 8, 4 ⟩$ .

6.4-2 Argue the correctness of HEAPSORT using the following loop invariant:

At the start of each iteration of the for loop of lines 2-5, the subarray $A [1 \dots i]$ is a max-heap containing the $i$ smallest elements of $A [1 \dots n]$ , and the subarray $A [i + 1 \dots n]$ contains the $n - i$ largest elements of $A [1 \dots n]$ , sorted.

6.4-3 What is the running time of HEAPSORT on an array $A$ of length $n$ that is already sorted in increasing order? What about decreasing order?

6.4-4 Show that the worst-case running time of HEAPSORT is $Ω (n l g n)$ .

6.4-5* Show that when all elements are distinct, the best-case running time of HEAPSORT is $Ω (n l g n)$ .

6.5 Priority queues

Heapsort is an excellent algorithm, but a good implementation of quicksort, presented in Chapter 7, usually beats it in practice. Nevertheless, the heap data structure itself has many uses. In this section, we present one of the most popular applications of a heap: as an efficient priority queue. As with heaps, priority queues come in two forms: max-priority queues and min-priority queues. We will focus here on how to implement max-priority queues, which are in turn based on max-heaps; Exercise 6.5-3 asks you to write the procedures for min-priority queues.

A priority queue is a data structure for maintaining a set $S$ of elements, each with an associated value called a key. A max-priority queue supports the following operations:

INSERT( $S$ , $x$ ) inserts the element $x$ into the set $S$ , which is equivalent to the operation $S = S \cup {x}$ .
MAXIMUM( $S$ ) returns the element of $S$ with the largest key.
EXTRACT-MAX( $S$ ) removes and returns the element of $S$ with the largest key.
INCREASE-KEY( $S$ , $x$ , $k$ ) increases the value of element $x$ ‘s key to the new value $k$ , which is assumed to be at least as large as $x$ ‘s current key value.

Among their other applications, we can use max-priority queues to schedule jobs on a shared computer. The max-priority queue keeps track of the jobs to be performed and their relative priorities. When a job is finished or interrupted, the scheduler selects the highest-priority job from among those pending by calling EXTRACT-MAX. The scheduler can add a new job to the queue at any time by calling INSERT.

Alternatively, a min-priority queue supports the operations INSERT, MINIMUM, EXTRACT-MIN, and DECREASE-KEY. A min-priority queue can be used in an event-driven simulator. The items in the queue are events to be simulated, each with an associated time of occurrence that serves as its key. The events must be simulated in order of their time of occurrence, because the simulation of an event can cause other events to be simulated in the future. The simulation program calls EXTRACT-MIN at each step to choose the next event to simulate. As new events are produced, the simulator inserts them into the min-priority queue by calling INSERT. We shall see other uses for min-priority queues, highlighting the DECREASE-KEY operation, in Chapters 23 and 24.

Not surprisingly, we can use a heap to implement a priority queue. In a given application, such as job scheduling or event-driven simulation, elements of a priority queue correspond to objects in the application. We often need to determine which application object corresponds to a given priority-queue element, and vice versa. When we use a heap to implement a priority queue, therefore, we often need to store a handle to the corresponding application object in each heap element. The exact makeup of the handle (such as a pointer or an integer) depends on the application. Similarly, we need to store a handle to the corresponding heap element in each application object. Here, the handle would typically be an array index. Because heap elements change locations within the array during heap operations, an actual implementation, upon relocating a heap element, would also have to update the array index in the corresponding application object. Because the details of accessing application objects depend heavily on the application and its implementation, we shall not pursue them here, other than noting that in practice, these handles do need to be correctly maintained.

Now we discuss how to implement the operations of a max-priority queue. The procedure HEAP-MAXIMUM implements the MAXIMUM operation in $Θ (1)$ time.

HEAP-MAXIMUM(A)
1  return A[1]

The procedure HEAP-EXTRACT-MAX implements the EXTRACT-MAX operation. It is similar to the for loop body (lines 3-5) of the HEAPSORT procedure.

HEAP-EXTRACT-MAX(A)
1  if A.heap-size < 1
2      error "heap underflow"
3  max = A[1]
4  A[1] = A[A.heap-size]
5  A.heap-size = A.heap-size - 1
6  MAX-HEAPIFY(A, 1)
7  return max

The running time of HEAP-EXTRACT-MAX is $O (l g n)$ , since it performs only a constant amount of work on top of the $O (l g n)$ time for MAX-HEAPIFY.

The procedure HEAP-INCREASE-KEY implements the INCREASE-KEY operation. An index $i$ into the array identifies the priority-queue element whose key we wish to increase. The procedure first updates the key of element $A [i]$ to its new value. Because increasing the key of $A [i]$ might violate the max-heap property, the procedure then, in a manner reminiscent of the insertion loop (lines 5-7) of INSERTION-SORT from Section 2.1, traverses a simple path from this node toward the root to find a proper place for the newly increased key. As HEAP-INCREASE-KEY traverses this path, it repeatedly compares an element to its parent, exchanging their keys and continuing if the element’s key is larger, and terminating if the element’s key is smaller, since the max-heap property now holds. (See Exercise 6.5-5 for a precise loop invariant.)

HEAP-INCREASE-KEY(A, i, key)
1  if key < A[i]
2      error "new key is smaller than current key"
3  A[i] = key
4  while i > 1 and A[PARENT(i)] < A[i]
5      exchange A[i] with A[PARENT(i)]
6      i = PARENT(i)

Figure 6.5 shows an example of a HEAP-INCREASE-KEY operation. The running time of HEAP-INCREASE-KEY on an $n$ -element heap is $O (l g n)$ , since the path traced from the node updated in line 3 to the root has length $O (l g n)$ .

Figure 6.5: The operation of HEAP-INCREASE-KEY. (a) The max-heap of Figure 6.4(a) with a node whose index is $i$ heavily shaded. (b) This node has its key increased to 15. (c) After one iteration of the while loop of lines 4-6, the node and its parent have exchanged keys, and the index $i$ moves up to the parent. (d) The max-heap after one more iteration of the while loop. At this point, $A [PARENT (i)] \geq A [i]$ . The max-heap property now holds and the procedure terminates.

The procedure MAX-HEAP-INSERT implements the INSERT operation. It takes as an input the key of the new element to be inserted into max-heap $A$ . The procedure first expands the max-heap by adding to the tree a new leaf whose key is $- \infty$ . Then it calls HEAP-INCREASE-KEY to set the key of this new node to its correct value and maintain the max-heap property.

MAX-HEAP-INSERT(A, key)
1  A.heap-size = A.heap-size + 1
2  A[A.heap-size] = -∞
3  HEAP-INCREASE-KEY(A, A.heap-size, key)

The running time of MAX-HEAP-INSERT on an $n$ -element heap is $O (l g n)$ .

In summary, a heap can support any priority-queue operation on a set of size $n$ in $O (l g n)$ time.

Think about it. An airport’s boarding queue is a priority queue: the person with the most urgent flight is called next, no matter when they arrived. What operation would model a passenger being upgraded to a higher priority?

Exercises 6.5-1 to 6.5-9

6.5-1 Illustrate the operation of HEAP-EXTRACT-MAX on the heap $A = ⟨ 15, 13, 9, 5, 12, 8, 7, 4, 0, 6, 2, 1 ⟩$ .

6.5-2 Illustrate the operation of MAX-HEAP-INSERT( $A$ , 10) on the heap $A = ⟨ 15, 13, 9, 5, 12, 8, 7, 4, 0, 6, 2, 1 ⟩$ .

6.5-3 Write pseudocode for the procedures HEAP-MINIMUM, HEAP-EXTRACT-MIN, HEAP-DECREASE-KEY, and MIN-HEAP-INSERT that implement a min-priority queue with a min-heap.

6.5-4 Why do we bother setting the key of the inserted node to $- \infty$ in line 2 of MAX-HEAP-INSERT when the next thing we do is increase its key to the desired value?

6.5-5 Argue the correctness of HEAP-INCREASE-KEY using the following loop invariant:

At the start of each iteration of the while loop of lines 4-6, $A [PARENT (i)] \geq A [LEFT (i)]$ and $A [PARENT (i)] \geq A [RIGHT (i)]$ , if these nodes exist, and the subarray $A [1 \dots A . heap-size]$ satisfies the max-heap property, except that there may be one violation: $A [i]$ may be larger than $A [PARENT (i)]$ .

You may assume that the subarray $A [1 \dots A . heap-size]$ satisfies the max-heap property at the time HEAP-INCREASE-KEY is called.

6.5-6 Each exchange operation on line 5 of HEAP-INCREASE-KEY typically requires three assignments. Show how to use the idea of the inner loop of INSERTION-SORT to reduce the three assignments down to just one assignment.

6.5-7 Show how to implement a first-in, first-out queue with a priority queue. Show how to implement a stack with a priority queue. (Queues and stacks are defined in Section 10.1.)

6.5-8 The operation HEAP-DELETE( $A$ , $i$ ) deletes the item in node $i$ from heap $A$ . Give an implementation of HEAP-DELETE that runs in $O (l g n)$ time for an $n$ -element max-heap.

6.5-9 Give an $O (n l g k)$ -time algorithm to merge $k$ sorted lists into one sorted list, where $n$ is the total number of elements in all the input lists. (Hint: Use a min-heap for $k$ -way merging.)

Problems

6-1 Building a heap using insertion

We can build a heap by repeatedly calling MAX-HEAP-INSERT to insert the elements into the heap. Consider the following variation on the BUILD-MAX-HEAP procedure:

BUILD-MAX-HEAP'(A)
1  A.heap-size = 1
2  for i = 2 to A.length
3      MAX-HEAP-INSERT(A, A[i])

a. Do the procedures BUILD-MAX-HEAP and BUILD-MAX-HEAP’ always create the same heap when run on the same input array? Prove that they do, or provide a counterexample.

b. Show that in the worst case, BUILD-MAX-HEAP’ requires $Θ (n l g n)$ time to build an $n$ -element heap.

6-2 Analysis of $d$ -ary heaps

A $d$ -ary heap is like a binary heap, but (with one possible exception) non-leaf nodes have $d$ children instead of 2 children.

a. How would you represent a $d$ -ary heap in an array?

b. What is the height of a $d$ -ary heap of $n$ elements in terms of $n$ and $d$ ?

c. Give an efficient implementation of EXTRACT-MAX in a $d$ -ary max-heap. Analyze its running time in terms of $d$ and $n$ .

d. Give an efficient implementation of INSERT in a $d$ -ary max-heap. Analyze its running time in terms of $d$ and $n$ .

e. Give an efficient implementation of INCREASE-KEY( $A$ , $i$ , $k$ ), which flags an error if $k < A [i]$ , but otherwise sets $A [i] = k$ and then updates the $d$ -ary max-heap structure appropriately. Analyze its running time in terms of $d$ and $n$ .

6-3 Young tableaus

An $m \times n$ Young tableau is an $m \times n$ matrix such that the entries of each row are in sorted order from left to right and the entries of each column are in sorted order from top to bottom. Some of the entries of a Young tableau may be $\infty$ , which we treat as nonexistent elements. Thus, a Young tableau can be used to hold $r \leq mn$ finite numbers.

a. Draw a $4 \times 4$ Young tableau containing the elements ${9, 16, 3, 2, 4, 8, 5, 14, 12}$ .

b. Argue that an $m \times n$ Young tableau $Y$ is empty if $Y [1, 1] = \infty$ . Argue that $Y$ is full (contains $mn$ elements) if $Y [m, n] < \infty$ .

c. Give an algorithm to implement EXTRACT-MIN on a nonempty $m \times n$ Young tableau that runs in $O (m + n)$ time. Your algorithm should use a recursive subroutine that solves an $m \times n$ problem by recursively solving either an $(m - 1) \times n$ or an $m \times (n - 1)$ subproblem. (Hint: Think about MAX-HEAPIFY.) Define $T (p)$ , where $p = m + n$ , to be the maximum running time of EXTRACT-MIN on any $m \times n$ Young tableau. Give and solve a recurrence for $T (p)$ that yields the $O (m + n)$ time bound.

d. Show how to insert a new element into a nonfull $m \times n$ Young tableau in $O (m + n)$ time.

e. Using no other sorting method as a subroutine, show how to use an $n \times n$ Young tableau to sort $n^{2}$ numbers in $O (n^{3})$ time.

f. Give an $O (m + n)$ -time algorithm to determine whether a given number is stored in a given $m \times n$ Young tableau.

Chapter notes

The heapsort algorithm was invented by Williams [357], who also described how to implement a priority queue with a heap. The BUILD-MAX-HEAP procedure was suggested by Floyd [106].

We use min-heaps to implement min-priority queues in Chapters 16, 23, and 24. We also give an implementation with improved time bounds for certain operations in Chapter 19 and, assuming that the keys are drawn from a bounded set of nonnegative integers, Chapter 20.

If the data are $b$ -bit integers, and the computer memory consists of addressable $b$ -bit words, Fredman and Willard [115] showed how to implement MINIMUM in $O (1)$ time and INSERT and EXTRACT-MIN in $O (l g n)$ time. Thorup [337] has improved the $O (l g n)$ bound to $O (l g l g n)$ time. This bound uses an amount of space unbounded in $n$ , but it can be implemented in linear space by using randomized hashing.

An important special case of priority queues occurs when the sequence of EXTRACT-MIN operations is monotone, that is, the values returned by successive EXTRACT-MIN operations are monotonically increasing over time. This case arises in several important applications, such as Dijkstra’s single-source shortest-paths algorithm, which we discuss in Chapter 24, and in discrete-event simulation. For Dijkstra’s algorithm it is particularly important that the DECREASE-KEY operation be implemented efficiently. For the monotone case, if the data are integers in the range $1, 2, \dots, C$ , Ahuja, Mehlhorn, Orlin, and Tarjan [8] describe how to implement EXTRACT-MIN and INSERT in $O (l g C)$ amortized time (see Chapter 17 for more on amortized analysis) and DECREASE-KEY in $O (1)$ time, using a data structure called a radix heap. The $O (l g C)$ bound can be improved to $O (l g C)$ using Fibonacci heaps (see Chapter 19) in conjunction with radix heaps. Cherkassky, Goldberg, and Silverstein [65] further improved the bound to $O (l g^{1/3 + ϵ} C)$ expected time by combining the multilevel bucketing structure of Denardo and Fox [85] with the heap of Thorup mentioned earlier. Raman [291] further improved these results to obtain a bound of $O (min (l g^{1/4 + ϵ} C, l g^{1/3 + ϵ} n))$ , for any fixed $ϵ > 0$ .

7 Quicksort

Big picture. Imagine sorting a messy bookshelf by grabbing one book, then shoving everything shorter to its left and everything taller to its right. That book is now exactly where it belongs, and you have two smaller messy shelves to repeat the trick on. That is quicksort. Its worst case is ugly, yet in practice it is often the fastest sorter around. The big question of this chapter: why does an algorithm with a terrible worst case behave so well almost all the time?

The quicksort algorithm has a worst-case running time of $Θ (n^{2})$ on an input array of $n$ numbers. Despite this slow worst-case running time, quicksort is often the best practical choice for sorting because it is remarkably efficient on the average: its expected running time is $Θ (n l g n)$ , and the constant factors hidden in the $Θ (n l g n)$ notation are quite small. It also has the advantage of sorting in place (see page 17), and it works well even in virtual-memory environments.

Section 7.1 describes the algorithm and an important subroutine used by quicksort for partitioning. Because the behavior of quicksort is complex, we start with an intuitive discussion of its performance in Section 7.2 and postpone its precise analysis to the end of the chapter. Section 7.3 presents a version of quicksort that uses random sampling. This algorithm has a good expected running time, and no particular input elicits its worst-case behavior. Section 7.4 analyzes the randomized algorithm, showing that it runs in $Θ (n^{2})$ time in the worst case and, assuming distinct elements, in expected $O (n l g n)$ time.

7.1 Description of quicksort

Quicksort, like merge sort, applies the divide-and-conquer paradigm introduced in Section 2.3.1. Here is the three-step divide-and-conquer process for sorting a typical subarray $A [p .. r]$ :

Divide: Partition (rearrange) the array $A [p .. r]$ into two (possibly empty) subarrays $A [p .. q - 1]$ and $A [q + 1.. r]$ such that each element of $A [p .. q - 1]$ is less than or equal to $A [q]$ , which is, in turn, less than or equal to each element of $A [q + 1.. r]$ . Compute the index $q$ as part of this partitioning procedure.
Conquer: Sort the two subarrays $A [p .. q - 1]$ and $A [q + 1.. r]$ by recursive calls to quicksort.
Combine: Because the subarrays are already sorted, no work is needed to combine them: the entire array $A [p .. r]$ is now sorted.

The following procedure implements quicksort:

QUICKSORT(A, p, r)
1  if p < r
2      q = PARTITION(A, p, r)
3      QUICKSORT(A, p, q - 1)
4      QUICKSORT(A, q + 1, r)

To sort an entire array $A$ , the initial call is QUICKSORT(A, 1, A.length).

Think about it. Merge sort does its hard work in the combine step. Quicksort does all its hard work in the divide step and gets the combine step for free. Where does the work hide in each?

Partitioning the array

The key to the algorithm is the PARTITION procedure, which rearranges the subarray $A [p .. r]$ in place.

PARTITION(A, p, r)
1  x = A[r]
2  i = p - 1
3  for j = p to r - 1
4      if A[j] <= x
5          i = i + 1
6          exchange A[i] with A[j]
7  exchange A[i + 1] with A[r]
8  return i + 1

Figure 7.1 shows how PARTITION works on an 8-element array. PARTITION always selects an element $x = A [r]$ as a pivot element around which to partition the subarray $A [p .. r]$ . As the procedure runs, it partitions the array into four (possibly empty) regions. At the start of each iteration of the for loop in lines 3-6, the regions satisfy certain properties, shown in Figure 7.2. We state these properties as a loop invariant:

Loop invariant. At the beginning of each iteration of the loop of lines 3-6, for any array index $k$ ,

If $p \leq k \leq i$ , then $A [k] \leq x$ .

If $i + 1 \leq k \leq j - 1$ , then $A [k] > x$ .

If $k = r$ , then $A [k] = x$ .

The indices between $j$ and $r - 1$ are not covered by any of the three cases, and the values in these entries have no particular relationship to the pivot $x$ .

Figure 7.1: The operation of PARTITION on a sample 8-element array with pivot $x = A [r]$ . Parts (a)-(i) show successive states of the array and of the indices $p$ , $i$ , $j$ , $r$ . Lightly shaded elements are in the first partition (values no greater than $x$ ), heavily shaded elements are in the second partition (values greater than $x$ ), unshaded elements have not yet been placed, and the final white element is the pivot. (a) The initial array and variable settings. (b) The value 2 is “swapped with itself” and put in the partition of smaller values. (c)-(d) The values 8 and 7 are added to the partition of larger values. (e) The values 1 and 8 are swapped, and the smaller partition grows. (f) The values 3 and 7 are swapped, and the smaller partition grows. (g)-(h) The larger partition grows to include 5 and 6, and the loop terminates. (i) In lines 7-8, the pivot element is swapped so that it lies between the two partitions.

Figure 7.2: The four regions maintained by PARTITION on a subarray $A [p .. r]$ . The values in $A [p .. i]$ are all less than or equal to $x$ , the values in $A [i + 1.. j - 1]$ are all greater than $x$ , and $A [r] = x$ . The subarray $A [j .. r - 1]$ can take on any values.

We need to show that this loop invariant is true prior to the first iteration, that each iteration of the loop maintains the invariant, and that the invariant provides a useful property to show correctness when the loop terminates.

Proof (loop invariant of PARTITION).

Initialization. Prior to the first iteration of the loop, $i = p - 1$ and $j = p$ .

No values lie between $p$ and $i$ (the range $p .. p - 1$ is empty), so condition 1 holds vacuously.

No values lie between $i + 1$ and $j - 1$ (the range $p .. p - 1$ is empty), so condition 2 holds vacuously.

The assignment in line 1 sets $x = A [r]$ , so condition 3 holds.

Maintenance. As Figure 7.3 shows, we consider two cases, depending on the outcome of the test in line 4.

Case $A [j] > x$ (Figure 7.3(a)). The only action in the loop is to increment $j$ .
$before: after j \leftarrow j + 1 : A [i + 1.. j - 1] > x A [i + 1.. (j + 1) - 1] = A [i + 1.. j] > x$
That is, condition 2 now holds for the old $A [j]$ (which becomes the new $A [j - 1]$ ). All other entries are unchanged, so conditions 1 and 3 still hold.

Case $A [j] \leq x$ (Figure 7.3(b)). The loop increments $i$ , swaps $A [i]$ and $A [j]$ , and then increments $j$ .

Because of the swap, we now have $A [i] \leq x$ , and condition 1 is satisfied for the enlarged range $p .. i$ .

Similarly, we have $A [j - 1] > x$ (with the new $j$ ), since the item that was swapped into this position is, by the loop invariant, the old $A [i]$ , which was greater than $x$ .

Condition 3 is unaffected, because index $r$ is never touched by the loop.

Termination. At termination, $j = r$ . Therefore, every entry in the array is in one of the three sets described by the invariant. We have partitioned the values in the array into three sets: those less than or equal to $x$ , those greater than $x$ , and a singleton set containing $x$ . $■$

The final two lines of PARTITION finish up by swapping the pivot element with the leftmost element greater than $x$ , thereby moving the pivot into its correct place in the partitioned array, and then returning the pivot’s new index. The output of PARTITION now satisfies the specifications given for the divide step. In fact, it satisfies a slightly stronger condition: after line 2 of QUICKSORT, $A [q]$ is strictly less than every element of $A [q + 1.. r]$ .

The running time of PARTITION on the subarray $A [p .. r]$ is $Θ (n)$ , where $n = r - p + 1$ (see Exercise 7.1-3).

Figure 7.3: The two cases for one iteration of procedure PARTITION. (a) If $A [j] > x$ , the only action is to increment $j$ , which maintains the loop invariant. (b) If $A [j] \leq x$ , index $i$ is incremented, $A [i]$ and $A [j]$ are swapped, and then $j$ is incremented. Again, the loop invariant is maintained.

Exercises 7.1-1 to 7.1-4

7.1-1 Using Figure 7.1 as a model, illustrate the operation of PARTITION on the array $A = ⟨ 13, 19, 9, 5, 12, 8, 7, 4, 21, 2, 6, 11 ⟩$ .

7.1-2 What value of $q$ does PARTITION return when all elements in the array $A [p .. r]$ have the same value? Modify PARTITION so that $q = ⌊(p + r) /2 ⌋$ when all elements in the array $A [p .. r]$ have the same value.

7.1-3 Give a brief argument that the running time of PARTITION on a subarray of size $n$ is $Θ (n)$ .

7.1-4 How would you modify QUICKSORT to sort into nonincreasing order?

7.2 Performance of quicksort

The running time of quicksort depends on whether the partitioning is balanced or unbalanced, which in turn depends on which elements are used for partitioning. If the partitioning is balanced, the algorithm runs asymptotically as fast as merge sort. If the partitioning is unbalanced, however, it can run asymptotically as slowly as insertion sort. In this section, we shall informally investigate how quicksort performs under the assumptions of balanced versus unbalanced partitioning.

Worst-case partitioning

The worst-case behavior for quicksort occurs when the partitioning routine produces one subproblem with $n - 1$ elements and one with $0$ elements. (We prove this claim in Section 7.4.1.) Let us assume that this unbalanced partitioning arises in each recursive call. The partitioning costs $Θ (n)$ time. Since the recursive call on an array of size $0$ just returns, $T (0) = Θ (1)$ , and the recurrence for the running time is

T (n) = T (n - 1) + T (0) + Θ (n) = T (n - 1) + Θ (n) .

Intuitively, if we sum the costs incurred at each level of the recursion, we get an arithmetic series (equation (A.2)), which evaluates to $Θ (n^{2})$ :

k = 1 \sum n k = \frac{n ( n + 1 )}{2} = Θ (n^{2}) .

Indeed, it is straightforward to use the substitution method to prove that the recurrence $T (n) = T (n - 1) + Θ (n)$ has the solution $T (n) = Θ (n^{2})$ . (See Exercise 7.2-1.)

Thus, if the partitioning is maximally unbalanced at every recursive level of the algorithm, the running time is $Θ (n^{2})$ . Therefore the worst-case running time of quicksort is no better than that of insertion sort. Moreover, the $Θ (n^{2})$ running time occurs when the input array is already completely sorted, a common situation in which insertion sort runs in $O (n)$ time.

Think about it. With pivot $A [r]$ and an already sorted array, the pivot is the maximum every time. Every “shelf” splits into “everything” and “nothing”. Does that feel like the worst possible luck?

Best-case partitioning

In the most even possible split, PARTITION produces two subproblems, each of size no more than $n /2$ , since one is of size $⌊ n /2 ⌋$ and one of size $⌈ n /2 ⌉ - 1$ . In this case, quicksort runs much faster. The recurrence for the running time is then

T (n) = 2 T (n /2) + Θ (n),

where we tolerate the sloppiness from ignoring the floor and ceiling and from subtracting 1. By case 2 of the master theorem (Theorem 4.1), this recurrence has the solution $T (n) = Θ (n l g n)$ . By equally balancing the two sides of the partition at every level of the recursion, we get an asymptotically faster algorithm.

Balanced partitioning

The average-case running time of quicksort is much closer to the best case than to the worst case, as the analyses in Section 7.4 will show. The key to understanding why is to understand how the balance of the partitioning is reflected in the recurrence that describes the running time.

Suppose, for example, that the partitioning algorithm always produces a 9-to-1 proportional split, which at first blush seems quite unbalanced. We then obtain the recurrence

T (n) = T (9 n /10) + T (n /10) + c n,

on the running time of quicksort, where we have explicitly included the constant $c$ hidden in the $Θ (n)$ term. Figure 7.4 shows the recursion tree for this recurrence.

Figure 7.4: A recursion tree for QUICKSORT in which PARTITION always produces a 9-to-1 split, yielding a running time of $O (n l g n)$ . Nodes show subproblem sizes ( $n$ ; then $\frac{1}{10} n$ and $\frac{9}{10} n$ ; then $\frac{1}{100} n$ , $\frac{9}{100} n$ , $\frac{9}{100} n$ , $\frac{81}{100} n$ ; and so on). Every full level costs $c n$ ; the shortest path ends at depth $lo g_{10} n$ , the longest at depth $lo g_{10/9} n$ , and levels beyond the shortest path cost at most $c n$ . Total: $O (n l g n)$ . The per-level costs include the constant $c$ implicit in the $Θ (n)$ term.

Notice that every level of the tree has cost $c n$ , until the recursion reaches a boundary condition at depth $lo g_{10} n = Θ (l g n)$ , and then the levels have cost at most $c n$ . The recursion terminates at depth $lo g_{10/9} n = Θ (l g n)$ . The total cost of quicksort is therefore $O (n l g n)$ . Thus, with a 9-to-1 proportional split at every level of recursion, which intuitively seems quite unbalanced, quicksort runs in $O (n l g n)$ time, asymptotically the same as if the split were right down the middle. Indeed, even a 99-to-1 split yields an $O (n l g n)$ running time. In fact, any split of constant proportionality yields a recursion tree of depth $Θ (l g n)$ , where the cost at each level is $O (n)$ . The running time is therefore $O (n l g n)$ whenever the split has constant proportionality.

Intuition for the average case

To develop a clear notion of the randomized behavior of quicksort, we must make an assumption about how frequently we expect to encounter the various inputs. The behavior of quicksort depends on the relative ordering of the values in the array elements given as the input, and not on the particular values in the array. As in our probabilistic analysis of the hiring problem in Section 5.2, we will assume for now that all permutations of the input numbers are equally likely.

When we run quicksort on a random input array, the partitioning is highly unlikely to happen in the same way at every level, as our informal analysis has assumed. We expect that some of the splits will be reasonably well balanced and that some will be fairly unbalanced. For example, Exercise 7.2-6 asks you to show that about 80 percent of the time PARTITION produces a split that is more balanced than 9 to 1, and about 20 percent of the time it produces a split that is less balanced than 9 to 1.

In the average case, PARTITION produces a mix of “good” and “bad” splits. In a recursion tree for an average-case execution of PARTITION, the good and bad splits are distributed randomly throughout the tree. Suppose, for the sake of intuition, that the good and bad splits alternate levels in the tree, and that the good splits are best-case splits and the bad splits are worst-case splits. Figure 7.5(a) shows the splits at two consecutive levels in the recursion tree. At the root of the tree, the cost is $n$ for partitioning, and the subarrays produced have sizes $n - 1$ and $0$ : the worst case. At the next level, the subarray of size $n - 1$ undergoes best-case partitioning into subarrays of size $(n - 1) /2 - 1$ and $(n - 1) /2$ . Let’s assume that the boundary-condition cost is 1 for the subarray of size 0.

Figure 7.5: (a) Two levels of a recursion tree for quicksort. The partitioning at the root costs $n$ and produces a “bad” split: two subarrays of sizes $0$ and $n - 1$ . The partitioning of the subarray of size $n - 1$ costs $n - 1$ and produces a “good” split: subarrays of size $(n - 1) /2 - 1$ and $(n - 1) /2$ . (b) A single level of a recursion tree that is very well balanced (cost $n$ , two subarrays of size $(n - 1) /2$ ). In both parts, the partitioning cost for the subproblems shown with elliptical shading is $Θ (n)$ . Yet the subproblems remaining to be solved in (a), shown with square shading, are no larger than the corresponding subproblems remaining to be solved in (b).

The combination of the bad split followed by the good split produces three subarrays of sizes $0$ , $(n - 1) /2 - 1$ , and $(n - 1) /2$ at a combined partitioning cost of $Θ (n) + Θ (n - 1) = Θ (n)$ . Certainly, this situation is no worse than that in Figure 7.5(b), namely a single level of partitioning that produces two subarrays of size $(n - 1) /2$ , at a cost of $Θ (n)$ . Yet this latter situation is balanced! Intuitively, the $Θ (n - 1)$ cost of the bad split can be absorbed into the $Θ (n)$ cost of the good split, and the resulting split is good. Thus, the running time of quicksort, when levels alternate between good and bad splits, is like the running time for good splits alone: still $O (n l g n)$ , but with a slightly larger constant hidden by the $O$ -notation. We shall give a rigorous analysis of the expected running time of a randomized version of quicksort in Section 7.4.2.

Think about it. A bad split is like a wasted move, but a good split right after it “pays off the debt”. Could you survive a few unlucky pivots if lucky ones keep arriving regularly?

Exercises 7.2-1 to 7.2-6

7.2-1 Use the substitution method to prove that the recurrence $T (n) = T (n - 1) + Θ (n)$ has the solution $T (n) = Θ (n^{2})$ , as claimed at the beginning of Section 7.2.

7.2-2 What is the running time of QUICKSORT when all elements of array $A$ have the same value?

7.2-3 Show that the running time of QUICKSORT is $Θ (n^{2})$ when the array $A$ contains distinct elements and is sorted in decreasing order.

7.2-4 Banks often record transactions on an account in order of the times of the transactions, but many people like to receive their bank statements with checks listed in order by check number. People usually write checks in order by check number, and merchants usually cash them with reasonable dispatch. The problem of converting time-of-transaction ordering to check-number ordering is therefore the problem of sorting almost-sorted input. Argue that the procedure INSERTION-SORT would tend to beat the procedure QUICKSORT on this problem.

7.2-5 Suppose that the splits at every level of quicksort are in the proportion $1 - α$ to $α$ , where $0 < α \leq 1/2$ is a constant. Show that the minimum depth of a leaf in the recursion tree is approximately $- l g n / l g α$ and the maximum depth is approximately $- l g n / l g (1 - α)$ . (Don’t worry about integer round-off.)

7.2-6* Argue that for any constant $0 < α \leq 1/2$ , the probability is approximately $1 - 2 α$ that on a random input array, PARTITION produces a split more balanced than $1 - α$ to $α$ .

7.3 A randomized version of quicksort

In exploring the average-case behavior of quicksort, we have made an assumption that all permutations of the input numbers are equally likely. In an engineering situation, however, we cannot always expect this assumption to hold. (See Exercise 7.2-4.) As we saw in Section 5.3, we can sometimes add randomization to an algorithm in order to obtain good expected performance over all inputs. Many people regard the resulting randomized version of quicksort as the sorting algorithm of choice for large enough inputs.

In Section 5.3, we randomized our algorithm by explicitly permuting the input. We could do so for quicksort also, but a different randomization technique, called random sampling, yields a simpler analysis. Instead of always using $A [r]$ as the pivot, we will select a randomly chosen element from the subarray $A [p .. r]$ . We do so by first exchanging element $A [r]$ with an element chosen at random from $A [p .. r]$ . By randomly sampling the range $p, \dots, r$ , we ensure that the pivot element $x = A [r]$ is equally likely to be any of the $r - p + 1$ elements in the subarray. Because we randomly choose the pivot element, we expect the split of the input array to be reasonably well balanced on average.

The changes to PARTITION and QUICKSORT are small. In the new partition procedure, we simply implement the swap before actually partitioning:

RANDOMIZED-PARTITION(A, p, r)
1  i = RANDOM(p, r)
2  exchange A[r] with A[i]
3  return PARTITION(A, p, r)

The new quicksort calls RANDOMIZED-PARTITION in place of PARTITION:

RANDOMIZED-QUICKSORT(A, p, r)
1  if p < r
2      q = RANDOMIZED-PARTITION(A, p, r)
3      RANDOMIZED-QUICKSORT(A, p, q - 1)
4      RANDOMIZED-QUICKSORT(A, q + 1, r)

We analyze this algorithm in the next section.

Exercises 7.3-1 to 7.3-2

7.3-1 Why do we analyze the expected running time of a randomized algorithm and not its worst-case running time?

7.3-2 When RANDOMIZED-QUICKSORT runs, how many calls are made to the random-number generator RANDOM in the worst case? How about in the best case? Give your answer in terms of $Θ$ -notation.

7.4 Analysis of quicksort

Section 7.2 gave some intuition for the worst-case behavior of quicksort and for why we expect it to run quickly. In this section, we analyze the behavior of quicksort more rigorously. We begin with a worst-case analysis, which applies to either QUICKSORT or RANDOMIZED-QUICKSORT, and conclude with an analysis of the expected running time of RANDOMIZED-QUICKSORT.

7.4.1 Worst-case analysis

We saw in Section 7.2 that a worst-case split at every level of recursion in quicksort produces a $Θ (n^{2})$ running time, which, intuitively, is the worst-case running time of the algorithm. We now prove this assertion.

Using the substitution method (see Section 4.3), we can show that the running time of quicksort is $O (n^{2})$ . Let $T (n)$ be the worst-case time for the procedure QUICKSORT on an input of size $n$ . We have the recurrence

T (n) = 0 \leq q \leq n - 1 max (T (q) + T (n - q - 1)) + Θ (n), (7.1)

where the parameter $q$ ranges from $0$ to $n - 1$ because the procedure PARTITION produces two subproblems with total size $n - 1$ . We guess that $T (n) \leq c n^{2}$ for some constant $c$ .

Proof ( $T (n) = O (n^{2})$ ).

Substitute the guess into recurrence (7.1), using $T (q) \leq c q^{2}$ and $T (n - q - 1) \leq c (n - q - 1)^{2}$ (induction hypothesis, since $q$ and $n - q - 1$ are both less than $n$ ):
$T (n) \leq 0 \leq q \leq n - 1 max (c q^{2} + c (n - q - 1)^{2}) + Θ (n) = c \cdot 0 \leq q \leq n - 1 max (q^{2} + (n - q - 1)^{2}) + Θ (n) (induction hypothesis) (factor out the constant c > 0)$
The expression $f (q) = q^{2} + (n - q - 1)^{2}$ achieves a maximum over the parameter’s range $0 \leq q \leq n - 1$ at either endpoint. To verify this claim, note that the second derivative of the expression with respect to $q$ is positive (see Exercise 7.4-3):
$f^{'} (q) f^{''} (q) = 2 q - 2 (n - q - 1) = 4 > 0 (so f is convex, and a convex function on an interval is maximized at an endpoint)$
Evaluate $f$ at both endpoints:
$f (0) f (n - 1) = 0^{2} + (n - 1)^{2} = (n - 1)^{2} = (n - 1)^{2} + 0^{2} = (n - 1)^{2}$
This observation gives the bound
$0 \leq q \leq n - 1 max (q^{2} + (n - q - 1)^{2}) \leq (n - 1)^{2} = n^{2} - 2 n + 1.$
Continuing with our bounding of $T (n)$ , we obtain
$T (n) \leq c (n^{2} - 2 n + 1) + Θ (n) = c n^{2} - c (2 n - 1) + Θ (n) \leq c n^{2} (the bound above) (distribute c) (see below)$
The last step holds since we can pick the constant $c$ large enough so that the $c (2 n - 1)$ term dominates the $Θ (n)$ term. Hence $T (n) = O (n^{2})$ . $■$

We saw in Section 7.2 a specific case in which quicksort takes $Ω (n^{2})$ time: when partitioning is unbalanced. Alternatively, Exercise 7.4-1 asks you to show that recurrence (7.1) has a solution of $T (n) = Ω (n^{2})$ . Thus, the (worst-case) running time of quicksort is $Θ (n^{2})$ .

7.4.2 Expected running time

We have already seen the intuition behind why the expected running time of RANDOMIZED-QUICKSORT is $O (n l g n)$ : if, in each level of recursion, the split induced by RANDOMIZED-PARTITION puts any constant fraction of the elements on one side of the partition, then the recursion tree has depth $Θ (l g n)$ , and $O (n)$ work is performed at each level. Even if we add a few new levels with the most unbalanced split possible between these levels, the total time remains $O (n l g n)$ . We can analyze the expected running time of RANDOMIZED-QUICKSORT precisely by first understanding how the partitioning procedure operates and then using this understanding to derive an $O (n l g n)$ bound on the expected running time. This upper bound on the expected running time, combined with the $Θ (n l g n)$ best-case bound we saw in Section 7.2, yields a $Θ (n l g n)$ expected running time. We assume throughout that the values of the elements being sorted are distinct.

Think about it. Imagine a tournament where players are compared only through a referee (the pivot). Once the referee separates two players into different rooms, they will never meet. Which pairs of players are most likely to meet? Keep that picture in mind for the analysis below.

Running time and comparisons

The QUICKSORT and RANDOMIZED-QUICKSORT procedures differ only in how they select pivot elements; they are the same in all other respects. We can therefore analyze RANDOMIZED-QUICKSORT by discussing the QUICKSORT and PARTITION procedures, but with the assumption that pivot elements are selected randomly from the subarray passed to RANDOMIZED-PARTITION.

The running time of QUICKSORT is dominated by the time spent in the PARTITION procedure. Each time the PARTITION procedure is called, it selects a pivot element, and this element is never included in any future recursive calls to QUICKSORT and PARTITION. Thus, there can be at most $n$ calls to PARTITION over the entire execution of the quicksort algorithm. One call to PARTITION takes $O (1)$ time plus an amount of time that is proportional to the number of iterations of the for loop in lines 3-6. Each iteration of this for loop performs a comparison in line 4, comparing the pivot element to another element of the array $A$ . Therefore, if we can count the total number of times that line 4 is executed, we can bound the total time spent in the for loop during the entire execution of QUICKSORT.

Lemma 7.1. Let $X$ be the number of comparisons performed in line 4 of PARTITION over the entire execution of QUICKSORT on an $n$ -element array. Then the running time of QUICKSORT is $O (n + X)$ .

Proof. By the discussion above, the algorithm makes at most $n$ calls to PARTITION, each of which does a constant amount of work and then executes the for loop some number of times. Each iteration of the for loop executes line 4.
$total time \leq constant work in at most n calls n \cdot O (1) + one line-4 comparison per iteration X \cdot O (1) = O (n + X) .$
$■$

Our goal, therefore, is to compute $X$ , the total number of comparisons performed in all calls to PARTITION. We will not attempt to analyze how many comparisons are made in each call to PARTITION. Rather, we will derive an overall bound on the total number of comparisons. To do so, we must understand when the algorithm compares two elements of the array and when it does not. For ease of analysis, we rename the elements of the array $A$ as $z_{1}, z_{2}, \dots, z_{n}$ , with $z_{i}$ being the $i$ th smallest element. We also define the set $Z_{ij} = {z_{i}, z_{i + 1}, \dots, z_{j}}$ to be the set of elements between $z_{i}$ and $z_{j}$ , inclusive.

When does the algorithm compare $z_{i}$ and $z_{j}$ ? To answer this question, we first observe that each pair of elements is compared at most once. Why? Elements are compared only to the pivot element and, after a particular call of PARTITION finishes, the pivot element used in that call is never again compared to any other elements.

Our analysis uses indicator random variables (see Section 5.2). We define

X_{ij} = I {z_{i} is compared to z_{j}},

where we are considering whether the comparison takes place at any time during the execution of the algorithm, not just during one iteration or one call of PARTITION. Since each pair is compared at most once, we can easily characterize the total number of comparisons performed by the algorithm:

X = i = 1 \sum n - 1 j = i + 1 \sum n X_{ij} .

Taking expectations of both sides, and then using linearity of expectation and Lemma 5.1, we obtain

E [X] = E [i = 1 \sum n - 1 j = i + 1 \sum n X_{ij}] = i = 1 \sum n - 1 j = i + 1 \sum n E [X_{ij}] = i = 1 \sum n - 1 j = i + 1 \sum n Pr {z_{i} is compared to z_{j}} . (linearity of expectation) (Lemma 5.1: E [I {A}] = Pr {A}) (7.2)

It remains to compute $Pr {z_{i} is compared to z_{j}}$ . Our analysis assumes that the RANDOMIZED-PARTITION procedure chooses each pivot randomly and independently.

Let us think about when two items are not compared. Consider an input to quicksort of the numbers 1 through 10 (in any order), and suppose that the first pivot element is 7. Then the first call to PARTITION separates the numbers into two sets: ${1, 2, 3, 4, 5, 6}$ and ${8, 9, 10}$ . In doing so, the pivot element 7 is compared to all other elements, but no number from the first set (e.g., 2) is or ever will be compared to any number from the second set (e.g., 9).

In general, because we assume that element values are distinct, once a pivot $x$ is chosen with $z_{i} < x < z_{j}$ , we know that $z_{i}$ and $z_{j}$ cannot be compared at any subsequent time. If, on the other hand, $z_{i}$ is chosen as a pivot before any other item in $Z_{ij}$ , then $z_{i}$ will be compared to each item in $Z_{ij}$ , except for itself. Similarly, if $z_{j}$ is chosen as a pivot before any other item in $Z_{ij}$ , then $z_{j}$ will be compared to each item in $Z_{ij}$ , except for itself. In our example, the values 7 and 9 are compared because 7 is the first item from $Z_{7, 9}$ to be chosen as a pivot. In contrast, 2 and 9 will never be compared because the first pivot element chosen from $Z_{2, 9}$ is 7. Thus, $z_{i}$ and $z_{j}$ are compared if and only if the first element to be chosen as a pivot from $Z_{ij}$ is either $z_{i}$ or $z_{j}$ .

We now compute the probability that this event occurs. Prior to the point at which an element from $Z_{ij}$ has been chosen as a pivot, the whole set $Z_{ij}$ is together in the same partition. Therefore, any element of $Z_{ij}$ is equally likely to be the first one chosen as a pivot. Because the set $Z_{ij}$ has $j - i + 1$ elements, and because pivots are chosen randomly and independently, the probability that any given element is the first one chosen as a pivot is $1/ (j - i + 1)$ . Thus, we have

Pr {z_{i} is compared to z_{j}} = Pr {z_{i} or z_{j} is first pivot chosen from Z_{ij}} = Pr {z_{i} is first pivot chosen from Z_{ij}} + Pr {z_{j} is first pivot chosen from Z_{ij}} = \frac{1}{j - i + 1} + \frac{1}{j - i + 1} = \frac{2}{j - i + 1} . (7.3)

The second line follows because the two events are mutually exclusive.

Combining equations (7.2) and (7.3), we get that

E [X] = i = 1 \sum n - 1 j = i + 1 \sum n \frac{2}{j - i + 1} .

We can evaluate this sum using a change of variables ( $k = j - i$ ) and the bound on the harmonic series in equation (A.7):

Derivation.
$E [X] = i = 1 \sum n - 1 j = i + 1 \sum n \frac{2}{j - i + 1} = i = 1 \sum n - 1 k = 1 \sum n - i \frac{2}{k + 1} < i = 1 \sum n - 1 k = 1 \sum n \frac{2}{k} = i = 1 \sum n - 1 2 H_{n} = i = 1 \sum n - 1 O (l g n) = O (n l g n) . (let k = j - i; as j runs from i + 1 to n, k runs from 1 to n - i) (since \frac{1}{k + 1} < \frac{1}{k}, and extending the inner range from n - i to n adds nonnegative terms) (H_{n} = \sum_{k = 1}^{n} 1/ k is the n th harmonic number) (harmonic series bound (A.7): H_{n} = ln n + O (1) = O (l g n)) (n - 1 terms, each O (l g n)) (7.4)$

Thus we conclude that, using RANDOMIZED-PARTITION, the expected running time of quicksort is $O (n l g n)$ when element values are distinct.

Wrapping up. By Lemma 7.1 the running time is $O (n + X)$ , so its expectation is $O (n + E [X]) = O (n + n l g n) = O (n l g n)$ . Combined with the best-case lower bound from Section 7.2, the expected running time is $Θ (n l g n)$ .

Exercises 7.4-1 to 7.4-6

7.4-1 Show that in the recurrence

T (n) = 0 \leq q \leq n - 1 max (T (q) + T (n - q - 1)) + Θ (n),

$T (n) = Ω (n^{2})$ .

7.4-2 Show that quicksort’s best-case running time is $Ω (n l g n)$ .

7.4-3 Show that the expression $q^{2} + (n - q - 1)^{2}$ achieves a maximum over $q = 0, 1, \dots, n - 1$ when $q = 0$ or $q = n - 1$ .

7.4-4 Show that RANDOMIZED-QUICKSORT’s expected running time is $Ω (n l g n)$ .

7.4-5 We can improve the running time of quicksort in practice by taking advantage of the fast running time of insertion sort when its input is “nearly” sorted. Upon calling quicksort on a subarray with fewer than $k$ elements, let it simply return without sorting the subarray. After the top-level call to quicksort returns, run insertion sort on the entire array to finish the sorting process. Argue that this sorting algorithm runs in $O (nk + n l g (n / k))$ expected time. How should we pick $k$ , both in theory and in practice?

7.4-6* Consider modifying the PARTITION procedure by randomly picking three elements from array $A$ and partitioning about their median (the middle value of the three elements). Approximate the probability of getting at worst an $α$ -to- $(1 - α)$ split, as a function of $α$ in the range $0 < α < 1$ .

Problems

7-1 Hoare partition correctness

The version of PARTITION given in this chapter is not the original partitioning algorithm. Here is the original partition algorithm, which is due to C. A. R. Hoare:

HOARE-PARTITION(A, p, r)
1  x = A[p]
2  i = p - 1
3  j = r + 1
4  while TRUE
5      repeat
6          j = j - 1
7      until A[j] <= x
8      repeat
9          i = i + 1
10     until A[i] >= x
11     if i < j
12         exchange A[i] with A[j]
13     else return j

a. Demonstrate the operation of HOARE-PARTITION on the array $A = ⟨ 13, 19, 9, 5, 12, 8, 7, 4, 11, 2, 6, 21 ⟩$ , showing the values of the array and auxiliary values after each iteration of the while loop in lines 4-13.

The next three questions ask you to give a careful argument that the procedure HOARE-PARTITION is correct. Assuming that the subarray $A [p .. r]$ contains at least two elements, prove the following:

b. The indices $i$ and $j$ are such that we never access an element of $A$ outside the subarray $A [p .. r]$ .

c. When HOARE-PARTITION terminates, it returns a value $j$ such that $p \leq j < r$ .

d. Every element of $A [p .. j]$ is less than or equal to every element of $A [j + 1.. r]$ when HOARE-PARTITION terminates.

The PARTITION procedure in Section 7.1 separates the pivot value (originally in $A [r]$ ) from the two partitions it forms. The HOARE-PARTITION procedure, on the other hand, always places the pivot value (originally in $A [p]$ ) into one of the two partitions $A [p .. j]$ and $A [j + 1.. r]$ . Since $p \leq j < r$ , this split is always nontrivial.

e. Rewrite the QUICKSORT procedure to use HOARE-PARTITION.

7-2 Quicksort with equal element values

The analysis of the expected running time of randomized quicksort in Section 7.4.2 assumes that all element values are distinct. In this problem, we examine what happens when they are not.

a. Suppose that all element values are equal. What would be randomized quicksort’s running time in this case?

b. The PARTITION procedure returns an index $q$ such that each element of $A [p .. q - 1]$ is less than or equal to $A [q]$ and each element of $A [q + 1.. r]$ is greater than $A [q]$ . Modify the PARTITION procedure to produce a procedure PARTITION $^{'} (A, p, r)$ , which permutes the elements of $A [p .. r]$ and returns two indices $q$ and $t$ , where $p \leq q \leq t \leq r$ , such that

all elements of $A [q .. t]$ are equal,
each element of $A [p .. q - 1]$ is less than $A [q]$ , and
each element of $A [t + 1.. r]$ is greater than $A [q]$ .

Like PARTITION, your PARTITION $^{'}$ procedure should take $Θ (r - p)$ time.

c. Modify the RANDOMIZED-PARTITION procedure to call PARTITION $^{'}$ , and name the new procedure RANDOMIZED-PARTITION $^{'}$ . Then modify the QUICKSORT procedure to produce a procedure QUICKSORT $^{'} (A, p, r)$ that calls RANDOMIZED-PARTITION $^{'}$ and recurses only on partitions of elements not known to be equal to each other.

d. Using QUICKSORT $^{'}$ , how would you adjust the analysis in Section 7.4.2 to avoid the assumption that all elements are distinct?

7-3 Alternative quicksort analysis

An alternative analysis of the running time of randomized quicksort focuses on the expected running time of each individual recursive call to RANDOMIZED-QUICKSORT, rather than on the number of comparisons performed.

a. Argue that, given an array of size $n$ , the probability that any particular element is chosen as the pivot is $1/ n$ . Use this to define indicator random variables $X_{i} = I {i th smallest element is chosen as the pivot}$ . What is $E [X_{i}]$ ?

b. Let $T (n)$ be a random variable denoting the running time of quicksort on an array of size $n$ . Argue that

E [T (n)] = E [q = 1 \sum n X_{q} (T (q - 1) + T (n - q) + Θ (n))] . (7.5)

c. Show that we can rewrite equation (7.5) as

E [T (n)] = \frac{2}{n} q = 2 \sum n - 1 E [T (q)] + Θ (n) . (7.6)

d. Show that

k = 2 \sum n - 1 k l g k \leq \frac{1}{2} n^{2} l g n - \frac{1}{8} n^{2} . (7.7)

(Hint: Split the summation into two parts, one for $k = 2, 3, \dots, ⌈ n /2 ⌉ - 1$ and one for $k = ⌈ n /2 ⌉, \dots, n - 1$ .)

e. Using the bound from equation (7.7), show that the recurrence in equation (7.6) has the solution $E [T (n)] = Θ (n l g n)$ . (Hint: Show, by substitution, that $E [T (n)] \leq an l g n$ for sufficiently large $n$ and for some positive constant $a$ .)

7-4 Stack depth for quicksort

The QUICKSORT algorithm of Section 7.1 contains two recursive calls to itself. After QUICKSORT calls PARTITION, it recursively sorts the left subarray and then it recursively sorts the right subarray. The second recursive call in QUICKSORT is not really necessary; we can avoid it by using an iterative control structure. This technique, called tail recursion, is provided automatically by good compilers. Consider the following version of quicksort, which simulates tail recursion:

TAIL-RECURSIVE-QUICKSORT(A, p, r)
1  while p < r
2      // Partition and sort left subarray.
3      q = PARTITION(A, p, r)
4      TAIL-RECURSIVE-QUICKSORT(A, p, q - 1)
5      p = q + 1

a. Argue that TAIL-RECURSIVE-QUICKSORT $(A, 1, A . l e n g t h)$ correctly sorts the array $A$ .

Compilers usually execute recursive procedures by using a stack that contains pertinent information, including the parameter values, for each recursive call. The information for the most recent call is at the top of the stack, and the information for the initial call is at the bottom. Upon calling a procedure, its information is pushed onto the stack; when it terminates, its information is popped. Since we assume that array parameters are represented by pointers, the information for each procedure call on the stack requires $O (1)$ stack space. The stack depth is the maximum amount of stack space used at any time during a computation.

b. Describe a scenario in which TAIL-RECURSIVE-QUICKSORT’s stack depth is $Θ (n)$ on an $n$ -element input array.

c. Modify the code for TAIL-RECURSIVE-QUICKSORT so that the worst-case stack depth is $Θ (l g n)$ . Maintain the $O (n l g n)$ expected running time of the algorithm.

7-5 Median-of-3 partition

One way to improve the RANDOMIZED-QUICKSORT procedure is to partition around a pivot that is chosen more carefully than by picking a random element from the subarray. One common approach is the median-of-3 method: choose the pivot as the median (middle element) of a set of 3 elements randomly selected from the subarray. (See Exercise 7.4-6.) For this problem, let us assume that the elements in the input array $A [1.. n]$ are distinct and that $n \geq 3$ . We denote the sorted output array by $A^{'} [1.. n]$ . Using the median-of-3 method to choose the pivot element $x$ , define $p_{i} = Pr {x = A^{'} [i]}$ .

a. Give an exact formula for $p_{i}$ as a function of $n$ and $i$ for $i = 2, 3, \dots, n - 1$ . (Note that $p_{1} = p_{n} = 0$ .)

b. By what amount have we increased the likelihood of choosing the pivot as $x = A^{'} [⌊(n + 1) /2 ⌋]$ , the median of $A [1.. n]$ , compared with the ordinary implementation? Assume that $n \to \infty$ , and give the limiting ratio of these probabilities.

c. If we define a “good” split to mean choosing the pivot as $x = A^{'} [i]$ , where $n /3 \leq i \leq 2 n /3$ , by what amount have we increased the likelihood of getting a good split compared with the ordinary implementation? (Hint: Approximate the sum by an integral.)

d. Argue that in the $Ω (n l g n)$ running time of quicksort, the median-of-3 method affects only the constant factor.

7-6 Fuzzy sorting of intervals

Consider a sorting problem in which we do not know the numbers exactly. Instead, for each number, we know an interval on the real line to which it belongs. That is, we are given $n$ closed intervals of the form $[a_{i}, b_{i}]$ , where $a_{i} \leq b_{i}$ . We wish to fuzzy-sort these intervals, i.e., to produce a permutation $⟨ i_{1}, i_{2}, \dots, i_{n} ⟩$ of the intervals such that for $j = 1, 2, \dots, n$ , there exist $c_{j} \in [a_{i_{j}}, b_{i_{j}}]$ satisfying $c_{1} \leq c_{2} \leq \dots \leq c_{n}$ .

a. Design a randomized algorithm for fuzzy-sorting $n$ intervals. Your algorithm should have the general structure of an algorithm that quicksorts the left endpoints (the $a_{i}$ values), but it should take advantage of overlapping intervals to improve the running time. (As the intervals overlap more and more, the problem of fuzzy-sorting the intervals becomes progressively easier. Your algorithm should take advantage of such overlapping, to the extent that it exists.)

b. Argue that your algorithm runs in expected time $Θ (n l g n)$ in general, but runs in expected time $Θ (n)$ when all of the intervals overlap (i.e., when there exists a value $x$ such that $x \in [a_{i}, b_{i}]$ for all $i$ ). Your algorithm should not be checking for this case explicitly; rather, its performance should naturally improve as the amount of overlap increases.

Chapter notes

The quicksort procedure was invented by Hoare [170]; Hoare’s version appears in Problem 7-1. The PARTITION procedure given in Section 7.1 is due to N. Lomuto. The analysis in Section 7.4 is due to Avrim Blum. Sedgewick [305] and Bentley [43] provide a good reference on the details of implementation and how they matter.

McIlroy [248] showed how to engineer a “killer adversary” that produces an array on which virtually any implementation of quicksort takes $Θ (n^{2})$ time. If the implementation is randomized, the adversary produces the array after seeing the random choices of the quicksort algorithm.

8 Sorting in Linear Time

Big picture. Imagine a librarian who may only ask one kind of question: “Which of these two books goes first?” No matter how clever she is, there is a hard speed limit on how fast she can shelve a whole library. But what if she is allowed to peek at the call number printed on each spine and drop the book straight into the right slot? The speed limit vanishes. This chapter is about both halves of that story: first proving the limit for comparison-only sorting, then sidestepping it by using what the keys actually look like.

We have now introduced several algorithms that can sort $n$ numbers in $O (n l g n)$ time. Merge sort and heapsort achieve this upper bound in the worst case; quicksort achieves it on average. Moreover, for each of these algorithms, we can produce a sequence of $n$ input numbers that causes the algorithm to run in $Ω (n l g n)$ time.

These algorithms share an interesting property: the sorted order they determine is based only on comparisons between the input elements. We call such sorting algorithms comparison sorts. All the sorting algorithms introduced thus far are comparison sorts.

In Section 8.1, we shall prove that any comparison sort must make $Ω (n l g n)$ comparisons in the worst case to sort $n$ elements. Thus, merge sort and heapsort are asymptotically optimal, and no comparison sort exists that is faster by more than a constant factor.

Sections 8.2, 8.3, and 8.4 examine three sorting algorithms (counting sort, radix sort, and bucket sort) that run in linear time. Of course, these algorithms use operations other than comparisons to determine the sorted order. Consequently, the $Ω (n l g n)$ lower bound does not apply to them.

8.1 Lower bounds for sorting

In a comparison sort, we use only comparisons between elements to gain order information about an input sequence $⟨ a_{1}, a_{2}, \dots, a_{n} ⟩$ . That is, given two elements $a_{i}$ and $a_{j}$ , we perform one of the tests $a_{i} < a_{j}$ , $a_{i} \leq a_{j}$ , $a_{i} = a_{j}$ , $a_{i} \geq a_{j}$ , or $a_{i} > a_{j}$ to determine their relative order. We may not inspect the values of the elements or gain order information about them in any other way.

In this section, we assume without loss of generality that all the input elements are distinct. Given this assumption, comparisons of the form $a_{i} = a_{j}$ are useless, so we can assume that no comparisons of this form are made. We also note that the comparisons $a_{i} \leq a_{j}$ , $a_{i} \geq a_{j}$ , $a_{i} > a_{j}$ , and $a_{i} < a_{j}$ are all equivalent in that they yield identical information about the relative order of $a_{i}$ and $a_{j}$ . We therefore assume that all comparisons have the form $a_{i} \leq a_{j}$ .

The decision-tree model

We can view comparison sorts abstractly in terms of decision trees. A decision tree is a full binary tree that represents the comparisons between elements that are performed by a particular sorting algorithm operating on an input of a given size. Control, data movement, and all other aspects of the algorithm are ignored. Figure 8.1 shows the decision tree corresponding to the insertion sort algorithm from Section 2.1 operating on an input sequence of three elements.

Figure 8.1: The decision tree for insertion sort operating on three elements. The root is the comparison 1:2; its children are 2:3 and 1:3, and so on down to six leaves. An internal node annotated by $i : j$ indicates a comparison between $a_{i}$ and $a_{j}$ . A leaf annotated by the permutation $⟨ π (1), π (2), \dots, π (n)⟩$ indicates the ordering $a_{π (1)} \leq a_{π (2)} \leq \dots \leq a_{π (n)}$ . The shaded path indicates the decisions made when sorting the input sequence $⟨ a_{1} = 6, a_{2} = 8, a_{3} = 5 ⟩$ ; the permutation $⟨ 3, 1, 2 ⟩$ at the leaf indicates that the sorted ordering is $a_{3} = 5 \leq a_{1} = 6 \leq a_{2} = 8$ . There are $3! = 6$ possible permutations of the input elements, and so the decision tree must have at least 6 leaves.

In a decision tree, we annotate each internal node by $i : j$ for some $i$ and $j$ in the range $1 \leq i, j \leq n$ , where $n$ is the number of elements in the input sequence. We also annotate each leaf by a permutation $⟨ π (1), π (2), \dots, π (n)⟩$ . (See Section C.1 for background on permutations.) The execution of the sorting algorithm corresponds to tracing a simple path from the root of the decision tree down to a leaf. Each internal node indicates a comparison $a_{i} \leq a_{j}$ . The left subtree then dictates subsequent comparisons once we know that $a_{i} \leq a_{j}$ , and the right subtree dictates subsequent comparisons knowing that $a_{i} > a_{j}$ . When we come to a leaf, the sorting algorithm has established the ordering $a_{π (1)} \leq a_{π (2)} \leq \dots \leq a_{π (n)}$ . Because any correct sorting algorithm must be able to produce each permutation of its input, each of the $n!$ permutations on $n$ elements must appear as one of the leaves of the decision tree for a comparison sort to be correct. Furthermore, each of these leaves must be reachable from the root by a downward path corresponding to an actual execution of the comparison sort. (We shall refer to such leaves as “reachable.”) Thus, we shall consider only decision trees in which each permutation appears as a reachable leaf.

A lower bound for the worst case

The length of the longest simple path from the root of a decision tree to any of its reachable leaves represents the worst-case number of comparisons that the corresponding sorting algorithm performs. Consequently, the worst-case number of comparisons for a given comparison sort algorithm equals the height of its decision tree. A lower bound on the heights of all decision trees in which each permutation appears as a reachable leaf is therefore a lower bound on the running time of any comparison sort algorithm. The following theorem establishes such a lower bound.

Theorem 8.1. Any comparison sort algorithm requires $Ω (n l g n)$ comparisons in the worst case.

Proof. From the preceding discussion, it suffices to determine the height of a decision tree in which each permutation appears as a reachable leaf.

Consider a decision tree of height $h$ with $l$ reachable leaves corresponding to a comparison sort on $n$ elements. Because each of the $n!$ permutations of the input appears as some leaf, we have $n! \leq l$ .

A binary tree of height $h$ has no more than $2^{h}$ leaves (each level at most doubles the number of nodes), so $l \leq 2^{h}$ . Combining,

$n! \leq l \leq 2^{h} .$

Taking base-2 logarithms of both sides:
$h &\ge \lg(n!) && \text{(since the } \lg \text{ function is monotonically increasing)} \\ &= \Omega(n \lg n) && \text{(by equation (3.19))}. \end{aligned}$$ To see why equation (3.19), $\lg(n!) = \Theta(n \lg n)$, gives this: by Stirling's approximation, $n! \ge (n/e)^n$, so $$\lg(n!) \ge n \lg n - n \lg e = \Omega(n \lg n). \qquad \blacksquare$$$

Corollary 8.2. Heapsort and merge sort are asymptotically optimal comparison sorts.

Proof. The $O (n l g n)$ upper bounds on the running times for heapsort and merge sort match the $Ω (n l g n)$ worst-case lower bound from Theorem 8.1. $■$

Think about it: The proof never looked at how the algorithm chooses its comparisons, only that it has to distinguish $n!$ outcomes using yes/no answers. Each question can at best halve the remaining possibilities. Does that remind you of the twenty-questions game?

Exercises 8.1-1 to 8.1-4

8.1-1 What is the smallest possible depth of a leaf in a decision tree for a comparison sort?

8.1-2 Obtain asymptotically tight bounds on $l g (n!)$ without using Stirling’s approximation. Instead, evaluate the summation $\sum_{k = 1}^{n} l g k$ using techniques from Section A.2.

8.1-3 Show that there is no comparison sort whose running time is linear for at least half of the $n!$ inputs of length $n$ . What about a fraction of $1/ n$ of the inputs of length $n$ ? What about a fraction $1/ 2^{n}$ ?

8.1-4 Suppose that you are given a sequence of $n$ elements to sort. The input sequence consists of $n / k$ subsequences, each containing $k$ elements. The elements in a given subsequence are all smaller than the elements in the succeeding subsequence and larger than the elements in the preceding subsequence. Thus, all that is needed to sort the whole sequence of length $n$ is to sort the $k$ elements in each of the $n / k$ subsequences. Show an $Ω (n l g k)$ lower bound on the number of comparisons needed to solve this variant of the sorting problem. (Hint: It is not rigorous to simply combine the lower bounds for the individual subsequences.)

8.2 Counting sort

Counting sort assumes that each of the $n$ input elements is an integer in the range $0$ to $k$ , for some integer $k$ . When $k = O (n)$ , the sort runs in $O (n)$ time.

Counting sort determines, for each input element $x$ , the number of elements less than $x$ . It uses this information to place element $x$ directly into its position in the output array. For example, if 17 elements are less than $x$ , then $x$ belongs in output position 18. We must modify this scheme slightly to handle the situation in which several elements have the same value, since we do not want to put them all in the same position.

In the code for counting sort, we assume that the input is an array $A [1.. n]$ , and thus $A . l e n g t h = n$ . We require two other arrays: the array $B [1.. n]$ holds the sorted output, and the array $C [0.. k]$ provides temporary working storage.

COUNTING-SORT(A, B, k)
1  let C[0..k] be a new array
2  for i = 0 to k
3      C[i] = 0
4  for j = 1 to A.length
5      C[A[j]] = C[A[j]] + 1
6  // C[i] now contains the number of elements equal to i.
7  for i = 1 to k
8      C[i] = C[i] + C[i - 1]
9  // C[i] now contains the number of elements less than or equal to i.
10 for j = A.length downto 1
11     B[C[A[j]]] = A[j]
12     C[A[j]] = C[A[j]] - 1

Figure 8.2: The operation of COUNTING-SORT on an input array $A [1..8]$ , where each element of $A$ is a nonnegative integer no larger than $k = 5$ . (a) The array $A$ and the auxiliary array $C$ after line 5. (b) The array $C$ after line 8. (c)-(e) The output array $B$ and the auxiliary array $C$ after one, two, and three iterations of the loop in lines 10-12, respectively; only the lightly shaded elements of $B$ have been filled in. (f) The final sorted output array $B$ . (In the book’s example, $A = ⟨ 2, 5, 3, 0, 2, 3, 0, 3 ⟩$ , $C$ after line 5 is $⟨ 2, 0, 2, 3, 0, 1 ⟩$ , and $C$ after line 8 is $⟨ 2, 2, 4, 7, 7, 8 ⟩$ .)

Figure 8.2 illustrates counting sort. After the for loop of lines 2-3 initializes the array $C$ to all zeros, the for loop of lines 4-5 inspects each input element. If the value of an input element is $i$ , we increment $C [i]$ . Thus, after line 5, $C [i]$ holds the number of input elements equal to $i$ for each integer $i = 0, 1, \dots, k$ . Lines 7-8 determine for each $i = 0, 1, \dots, k$ how many input elements are less than or equal to $i$ by keeping a running sum of the array $C$ .

Finally, the for loop of lines 10-12 places each element $A [j]$ into its correct sorted position in the output array $B$ . If all $n$ elements are distinct, then when we first enter line 10, for each $A [j]$ , the value $C [A [j]]$ is the correct final position of $A [j]$ in the output array, since there are $C [A [j]]$ elements less than or equal to $A [j]$ . Because the elements might not be distinct, we decrement $C [A [j]]$ each time we place a value $A [j]$ into the $B$ array. Decrementing $C [A [j]]$ causes the next input element with a value equal to $A [j]$ , if one exists, to go to the position immediately before $A [j]$ in the output array.

Running time

How much time does counting sort require?

The for loop of lines 2-3 takes time $Θ (k)$ .
The for loop of lines 4-5 takes time $Θ (n)$ .
The for loop of lines 7-8 takes time $Θ (k)$ .
The for loop of lines 10-12 takes time $Θ (n)$ .

Thus, the overall time is

$Θ (k + n) .$

In practice, we usually use counting sort when we have $k = O (n)$ , in which case the running time is $Θ (n)$ .

Counting sort beats the lower bound of $Ω (n l g n)$ proved in Section 8.1 because it is not a comparison sort. In fact, no comparisons between input elements occur anywhere in the code. Instead, counting sort uses the actual values of the elements to index into an array. The $Ω (n l g n)$ lower bound for sorting does not apply when we depart from the comparison sort model.

Stability

An important property of counting sort is that it is stable: numbers with the same value appear in the output array in the same order as they do in the input array. That is, it breaks ties between two numbers by the rule that whichever number appears first in the input array appears first in the output array. Normally, the property of stability is important only when satellite data are carried around with the element being sorted. Counting sort’s stability is important for another reason: counting sort is often used as a subroutine in radix sort. As we shall see in the next section, in order for radix sort to work correctly, counting sort must be stable.

Exercises 8.2-1 to 8.2-4

8.2-1 Using Figure 8.2 as a model, illustrate the operation of COUNTING-SORT on the array $A = ⟨ 6, 0, 2, 0, 1, 3, 4, 6, 1, 3, 2 ⟩$ .

8.2-2 Prove that COUNTING-SORT is stable.

8.2-3 Suppose that we were to rewrite the for loop header in line 10 of the COUNTING-SORT as

10 for j = 1 to A.length

Show that the algorithm still works properly. Is the modified algorithm stable?

8.2-4 Describe an algorithm that, given $n$ integers in the range $0$ to $k$ , preprocesses its input and then answers any query about how many of the $n$ integers fall into a range $[a .. b]$ in $O (1)$ time. Your algorithm should use $Θ (n + k)$ preprocessing time.

8.3 Radix sort

Radix sort is the algorithm used by the card-sorting machines you now find only in computer museums. The cards have 80 columns, and in each column a machine can punch a hole in one of 12 places. The sorter can be mechanically “programmed” to examine a given column of each card in a deck and distribute the card into one of 12 bins depending on which place has been punched. An operator can then gather the cards bin by bin, so that cards with the first place punched are on top of cards with the second place punched, and so on.

For decimal digits, each column uses only 10 places. (The other two places are reserved for encoding nonnumeric characters.) A $d$ -digit number would then occupy a field of $d$ columns. Since the card sorter can look at only one column at a time, the problem of sorting $n$ cards on a $d$ -digit number requires a sorting algorithm.

Intuitively, you might sort numbers on their most significant digit, sort each of the resulting bins recursively, and then combine the decks in order. Unfortunately, since the cards in 9 of the 10 bins must be put aside to sort each of the bins, this procedure generates many intermediate piles of cards that you would have to keep track of. (See Exercise 8.3-5.)

Radix sort solves the problem of card sorting, counterintuitively, by sorting on the least significant digit first. The algorithm then combines the cards into a single deck, with the cards in the 0 bin preceding the cards in the 1 bin preceding the cards in the 2 bin, and so on. Then it sorts the entire deck again on the second-least significant digit and recombines the deck in a like manner. The process continues until the cards have been sorted on all $d$ digits. Remarkably, at that point the cards are fully sorted on the $d$ -digit number. Thus, only $d$ passes through the deck are required to sort. Figure 8.3 shows how radix sort operates on a “deck” of seven 3-digit numbers.

In order for radix sort to work correctly, the digit sorts must be stable. The sort performed by a card sorter is stable, but the operator has to be wary about not changing the order of the cards as they come out of a bin, even though all the cards in a bin have the same digit in the chosen column.

Figure 8.3: The operation of radix sort on a list of seven 3-digit numbers.

input	after sort on digit 1	after sort on digit 2	after sort on digit 3
329	720	720	329
457	355	329	355
657	436	436	436
839	457	839	457
436	657	355	657
720	329	457	720
355	839	657	839

The leftmost column is the input. The remaining columns show the list after successive sorts on increasingly significant digit positions (ones, then tens, then hundreds). Shading in the book indicates the digit position sorted on to produce each list from the previous one.

Think about it: Why does sorting the least significant digit first work? Because each later, more significant pass is the “boss” and overrides earlier ones, while stability makes the earlier passes the tie-breakers. It is like sorting a class roster by first name, then by last name with a stable sort: the final list is by last name, with first names breaking ties.

In a typical computer, which is a sequential random-access machine, we sometimes use radix sort to sort records of information that are keyed by multiple fields. For example, we might wish to sort dates by three keys: year, month, and day. We could run a sorting algorithm with a comparison function that, given two dates, compares years, and if there is a tie, compares months, and if another tie occurs, compares days. Alternatively, we could sort the information three times with a stable sort: first on day, next on month, and finally on year.

The code for radix sort is straightforward. The following procedure assumes that each element in the $n$ -element array $A$ has $d$ digits, where digit 1 is the lowest-order digit and digit $d$ is the highest-order digit.

RADIX-SORT(A, d)
1  for i = 1 to d
2      use a stable sort to sort array A on digit i

Lemma 8.3. Given $n$ $d$ -digit numbers in which each digit can take on up to $k$ possible values, RADIX-SORT correctly sorts these numbers in $Θ (d (n + k))$ time if the stable sort it uses takes $Θ (n + k)$ time.

Proof. Correctness. It follows by induction on the column being sorted (see Exercise 8.3-3).

Running time. The analysis depends on the stable sort used as the intermediate sorting algorithm. When each digit is in the range $0$ to $k - 1$ (so that it can take on $k$ possible values), and $k$ is not too large, counting sort is the obvious choice. Each pass over $n$ $d$ -digit numbers then takes time $Θ (n + k)$ . There are $d$ passes, and so the total time for radix sort is

$d \cdot Θ (n + k) = Θ (d (n + k)) . ■$

When $d$ is constant and $k = O (n)$ , we can make radix sort run in linear time. More generally, we have some flexibility in how to break each key into digits.

Lemma 8.4. Given $n$ $b$ -bit numbers and any positive integer $r \leq b$ , RADIX-SORT correctly sorts these numbers in $Θ ((b / r) (n + 2^{r}))$ time if the stable sort it uses takes $Θ (n + k)$ time for inputs in the range $0$ to $k$ .

Proof. For a value $r \leq b$ , we view each key as having $d = ⌈ b / r ⌉$ digits of $r$ bits each. Each digit is an integer in the range $0$ to $2^{r} - 1$ , so that we can use counting sort with $k = 2^{r} - 1$ . (For example, we can view a 32-bit word as having four 8-bit digits, so that $b = 32$ , $r = 8$ , $k = 2^{r} - 1 = 255$ , and $d = b / r = 4$ .) Each pass of counting sort takes time $Θ (n + k) = Θ (n + 2^{r})$ and there are $d$ passes, for a total running time of

$Θ (d (n + 2^{r})) = Θ (\frac{b}{r} (n + 2^{r})) . ■$

Choosing the best $r$

For given values of $n$ and $b$ , we wish to choose the value of $r$ , with $r \leq b$ , that minimizes the expression $(b / r) (n + 2^{r})$ .

Case 1: $b < ⌊ l g n ⌋$ . For any value of $r \leq b$ , we have $2^{r} \leq 2^{b} < n$ , so $n + 2^{r} = Θ (n)$ . Thus, choosing $r = b$ yields a running time of

$\frac{b}{b} (n + 2^{b}) = Θ (n),$

which is asymptotically optimal (we must at least read the input).

Case 2: $b \geq ⌊ l g n ⌋$ . Choosing $r = ⌊ l g n ⌋$ gives the best time to within a constant factor, which we can see as follows.

Choosing $r = ⌊ l g n ⌋$ gives $2^{r} \leq n$ , so $n + 2^{r} = Θ (n)$ , and the running time is $Θ (bn / l g n)$ .
As we increase $r$ above $⌊ l g n ⌋$ , the $2^{r}$ term in the numerator increases faster than the $r$ term in the denominator, and so increasing $r$ above $⌊ l g n ⌋$ yields a running time of $Ω (bn / l g n)$ .
If instead we were to decrease $r$ below $⌊ l g n ⌋$ , then the $b / r$ term increases and the $n + 2^{r}$ term remains at $Θ (n)$ .

Radix sort versus comparison sorts

Is radix sort preferable to a comparison-based sorting algorithm, such as quicksort? If $b = O (l g n)$ , as is often the case, and we choose $r \approx l g n$ , then radix sort’s running time is $Θ (n)$ , which appears to be better than quicksort’s expected running time of $Θ (n l g n)$ . The constant factors hidden in the $Θ$ -notation differ, however. Although radix sort may make fewer passes than quicksort over the $n$ keys, each pass of radix sort may take significantly longer. Which sorting algorithm we prefer depends on the characteristics of the implementations, of the underlying machine (e.g., quicksort often uses hardware caches more effectively than radix sort), and of the input data. Moreover, the version of radix sort that uses counting sort as the intermediate stable sort does not sort in place, which many of the $Θ (n l g n)$ -time comparison sorts do. Thus, when primary memory storage is at a premium, we might prefer an in-place algorithm such as quicksort.

Exercises 8.3-1 to 8.3-5

8.3-1 Using Figure 8.3 as a model, illustrate the operation of RADIX-SORT on the following list of English words: COW, DOG, SEA, RUG, ROW, MOB, BOX, TAB, BAR, EAR, TAR, DIG, BIG, TEA, NOW, FOX.

8.3-2 Which of the following sorting algorithms are stable: insertion sort, merge sort, heapsort, and quicksort? Give a simple scheme that makes any sorting algorithm stable. How much additional time and space does your scheme entail?

8.3-3 Use induction to prove that radix sort works. Where does your proof need the assumption that the intermediate sort is stable?

8.3-4 Show how to sort $n$ integers in the range $0$ to $n^{2} - 1$ in $O (n)$ time.

8.3-5* In the first card-sorting algorithm in this section, exactly how many sorting passes are needed to sort $d$ -digit decimal numbers in the worst case? How many piles of cards would an operator need to keep track of in the worst case?

8.4 Bucket sort

Bucket sort assumes that the input is drawn from a uniform distribution and has an average-case running time of $O (n)$ . Like counting sort, bucket sort is fast because it assumes something about the input. Whereas counting sort assumes that the input consists of integers in a small range, bucket sort assumes that the input is generated by a random process that distributes elements uniformly and independently over the interval $[0, 1)$ . (See Section C.2 for a definition of uniform distribution.)

Bucket sort divides the interval $[0, 1)$ into $n$ equal-sized subintervals, or buckets, and then distributes the $n$ input numbers into the buckets. Since the inputs are uniformly and independently distributed over $[0, 1)$ , we do not expect many numbers to fall into each bucket. To produce the output, we simply sort the numbers in each bucket and then go through the buckets in order, listing the elements in each.

Our code for bucket sort assumes that the input is an $n$ -element array $A$ and that each element $A [i]$ in the array satisfies $0 \leq A [i] < 1$ . The code requires an auxiliary array $B [0.. n - 1]$ of linked lists (buckets) and assumes that there is a mechanism for maintaining such lists. (Section 10.2 describes how to implement basic operations on linked lists.)

BUCKET-SORT(A)
1  n = A.length
2  let B[0..n - 1] be a new array
3  for i = 0 to n - 1
4      make B[i] an empty list
5  for i = 1 to n
6      insert A[i] into list B[⌊n·A[i]⌋]
7  for i = 0 to n - 1
8      sort list B[i] with insertion sort
9  concatenate the lists B[0], B[1], ..., B[n - 1] together in order

Figure 8.4: The operation of BUCKET-SORT for $n = 10$ . (a) The input array $A [1..10]$ . (b) The array $B [0..9]$ of sorted lists (buckets) after line 8 of the algorithm. Bucket $i$ holds values in the half-open interval $[i /10, (i + 1) /10)$ . The sorted output consists of a concatenation in order of the lists $B [0], B [1], \dots, B [9]$ . (The input is $.78, .17, .39, .26, .72, .94, .21, .12, .23, .68$ ; for example, bucket 2 holds $.21 \to .23 \to .26$ .)

Figure 8.4 shows the operation of bucket sort on an input array of 10 numbers.

Correctness

To see that this algorithm works, consider two elements $A [i]$ and $A [j]$ . Assume without loss of generality that $A [i] \leq A [j]$ . Since $⌊ n A [i]⌋ \leq ⌊ n A [j]⌋$ , either element $A [i]$ goes into the same bucket as $A [j]$ or it goes into a bucket with a lower index.

If $A [i]$ and $A [j]$ go into the same bucket, then the for loop of lines 7-8 puts them into the proper order.
If $A [i]$ and $A [j]$ go into different buckets, then line 9 puts them into the proper order.

Therefore, bucket sort works correctly.

Running time

To analyze the running time, observe that all lines except line 8 take $O (n)$ time in the worst case. We need to analyze the total time taken by the $n$ calls to insertion sort in line 8.

To analyze the cost of the calls to insertion sort, let $n_{i}$ be the random variable denoting the number of elements placed in bucket $B [i]$ . Since insertion sort runs in quadratic time (see Section 2.2), the running time of bucket sort is

$T (n) = Θ (n) + \sum_{i = 0}^{n - 1} O (n_{i}^{2}) .$

We now analyze the average-case running time of bucket sort, by computing the expected value of the running time, where we take the expectation over the input distribution. Taking expectations of both sides and using linearity of expectation, we have

\begin{aligned} \text{comparisons} &= 3 \cdot \frac{n-1}{2} \ &= 3 \left\lfloor \frac{n}{2} \right\rfloor \qquad \text{(since $n$ is odd, } \lfloor n/2 \rfloor = (n-1)/2\text{)}. \end{aligned}

*Case $n$ even.* We use 1 comparison for the first 2 elements, leaving $n - 2$ elements, i.e. $(n-2)/2$ pairs, at 3 comparisons each:

\begin{aligned} \text{comparisons} &= 1 + 3 \cdot \frac{n-2}{2} \ &= 1 + \frac{3n}{2} - 3 \ &= \frac{3n}{2} - 2 . \end{aligned}

Thus, in either case, the total number of comparisons is at most $3\lfloor n/2 \rfloor$. ### Exercises 9.1-1 to 9.1-2 **9.1-1** Show that the second smallest of $n$ elements can be found with $n + \lceil \lg n \rceil - 2$ comparisons in the worst case. (*Hint:* Also find the smallest element.) **9.1-2\*** Prove the lower bound of $\lceil 3n/2 \rceil - 2$ comparisons in the worst case to find both the maximum and minimum of $n$ numbers. (*Hint:* Consider how many numbers are potentially either the maximum or minimum, and investigate how a comparison affects these counts.) --- ## 9.2 Selection in expected linear time The general selection problem appears more difficult than the simple problem of finding a minimum. Yet, surprisingly, the asymptotic running time for both problems is the same: $\Theta(n)$. In this section, we present a divide-and-conquer algorithm for the selection problem. The algorithm RANDOMIZED-SELECT is modeled after the quicksort algorithm of Chapter 7. As in quicksort, we partition the input array recursively. But unlike quicksort, which recursively processes both sides of the partition, RANDOMIZED-SELECT works on only one side of the partition. This difference shows up in the analysis: whereas quicksort has an expected running time of $\Theta(n \lg n)$, the expected running time of RANDOMIZED-SELECT is $\Theta(n)$, assuming that the elements are distinct. RANDOMIZED-SELECT uses the procedure RANDOMIZED-PARTITION introduced in Section 7.3. Thus, like RANDOMIZED-QUICKSORT, it is a randomized algorithm, since its behavior is determined in part by the output of a random-number generator. The following code for RANDOMIZED-SELECT returns the $i$th smallest element of the array $A[p \ldots r]$. ```text RANDOMIZED-SELECT(A, p, r, i) 1 if p == r 2 return A[p] 3 q = RANDOMIZED-PARTITION(A, p, r) 4 k = q - p + 1 5 if i == k // the pivot value is the answer 6 return A[q] 7 elseif i < k 8 return RANDOMIZED-SELECT(A, p, q - 1, i) 9 else return RANDOMIZED-SELECT(A, q + 1, r, i - k) ``` The RANDOMIZED-SELECT procedure works as follows. - **Line 1** checks for the base case of the recursion, in which the subarray $A[p \ldots r]$ consists of just one element. In this case, $i$ must equal 1, and we simply return $A[p]$ in line 2 as the $i$th smallest element. - **Line 3**: otherwise, the call to RANDOMIZED-PARTITION partitions the array $A[p \ldots r]$ into two (possibly empty) subarrays $A[p \ldots q-1]$ and $A[q+1 \ldots r]$ such that each element of $A[p \ldots q-1]$ is less than or equal to $A[q]$, which in turn is less than each element of $A[q+1 \ldots r]$. As in quicksort, we refer to $A[q]$ as the **pivot** element. - **Line 4** computes the number $k$ of elements in the subarray $A[p \ldots q]$, that is, the number of elements in the low side of the partition, plus one for the pivot element. - **Line 5** checks whether $A[q]$ is the $i$th smallest element. If it is, line 6 returns $A[q]$. - Otherwise, the algorithm determines in which of the two subarrays $A[p \ldots q-1]$ and $A[q+1 \ldots r]$ the $i$th smallest element lies. If $i < k$, then the desired element lies on the low side of the partition, and **line 8** recursively selects it from the subarray. If $i > k$, however, then the desired element lies on the high side of the partition. Since we already know $k$ values that are smaller than the $i$th smallest element of $A[p \ldots r]$ (namely, the elements of $A[p \ldots q]$), the desired element is the $(i-k)$th smallest element of $A[q+1 \ldots r]$, which **line 9** finds recursively. The code appears to allow recursive calls to subarrays with 0 elements, but Exercise 9.2-1 asks you to show that this situation cannot happen. The worst-case running time for RANDOMIZED-SELECT is $\Theta(n^2)$, even to find the minimum, because we could be extremely unlucky and always partition around the largest remaining element, and partitioning takes $\Theta(n)$ time. We will see that the algorithm has a linear expected running time, though, and because it is randomized, no particular input elicits the worst-case behavior. ### Analysis of the expected running time To analyze the expected running time of RANDOMIZED-SELECT, we let the running time on an input array $A[p \ldots r]$ of $n$ elements be a random variable that we denote by $T(n)$, and we obtain an upper bound on $E[T(n)]$ as follows. The procedure RANDOMIZED-PARTITION is equally likely to return any element as the pivot. Therefore, for each $k$ such that $1 \le k \le n$, the subarray $A[p \ldots q]$ has $k$ elements (all less than or equal to the pivot) with probability $1/n$. For $k = 1, 2, \ldots, n$, we define indicator random variables $X_k$ where $$X_k = I\{\text{the subarray } A[p \ldots q] \text{ has exactly } k \text{ elements}\},$$ and so, assuming that the elements are distinct, we have $$E[X_k] = 1/n . \tag{9.1}$$ When we call RANDOMIZED-SELECT and choose $A[q]$ as the pivot element, we do not know, a priori, if we will terminate immediately with the correct answer, recurse on the subarray $A[p \ldots q-1]$, or recurse on the subarray $A[q+1 \ldots r]$. This decision depends on where the $i$th smallest element falls relative to $A[q]$. Assuming that $T(n)$ is monotonically increasing, we can upper-bound the time needed for the recursive call by the time needed for the recursive call on the largest possible input. In other words, to obtain an upper bound, we assume that the $i$th element is always on the side of the partition with the greater number of elements. For a given call of RANDOMIZED-SELECT, the indicator random variable $X_k$ has the value 1 for exactly one value of $k$, and it is 0 for all other $k$. When $X_k = 1$, the two subarrays on which we might recurse have sizes $k - 1$ and $n - k$. Hence, we have the recurrence

\begin{aligned} T(n) &\le \sum_{k=1}^{n} X_k \cdot \bigl( T(\max(k-1, n-k)) + O(n) \bigr) \ &= \sum_{k=1}^{n} X_k \cdot T(\max(k-1, n-k)) + O(n) . \end{aligned}

T ak in g e x p ec t e d v a l u es, w e ha v e

\begin{aligned} E[T(n)] &\le E\left[ \sum_{k=1}^{n} X_k \cdot T(\max(k-1, n-k)) + O(n) \right] \ &= \sum_{k=1}^{n} E\bigl[ X_k \cdot T(\max(k-1, n-k)) \bigr] + O(n) && \text{(by linearity of expectation)} \ &= \sum_{k=1}^{n} E[X_k] \cdot E\bigl[ T(\max(k-1, n-k)) \bigr] + O(n) && \text{(by equation (C.24))} \ &= \sum_{k=1}^{n} \frac{1}{n} \cdot E\bigl[ T(\max(k-1, n-k)) \bigr] + O(n) && \text{(by equation (9.1))} . \end{aligned}

In order to apply equation (C.24), we rely on $X_k$ and $T(\max(k-1, n-k))$ being independent random variables. Exercise 9.2-2 asks you to justify this assertion. Let us consider the expression $\max(k-1, n-k)$. We have

\max(k-1, n-k) = \begin{cases} k - 1 & \text{if } k > \lceil n/2 \rceil , \ n - k & \text{if } k \le \lceil n/2 \rceil . \end{cases}

If $n$ is even, each term from $T(\lceil n/2 \rceil)$ up to $T(n-1)$ appears exactly twice in the summation, and if $n$ is odd, all these terms appear twice and $T(\lfloor n/2 \rfloor)$ appears once. Thus, we have $$E[T(n)] \le \frac{2}{n} \sum_{k=\lfloor n/2 \rfloor}^{n-1} E[T(k)] + O(n) .$$ ### Solving the recurrence by substitution We show that $E[T(n)] = O(n)$ by substitution. Assume that $E[T(n)] \le cn$ for some constant $c$ that satisfies the initial conditions of the recurrence. We assume that $T(n) = O(1)$ for $n$ less than some constant; we shall pick this constant later. We also pick a constant $a$ such that the function described by the $O(n)$ term above (which describes the non-recursive component of the running time of the algorithm) is bounded from above by $an$ for all $n > 0$. Using this inductive hypothesis, we have

\begin{aligned} E[T(n)] &\le \frac{2}{n} \sum_{k=\lfloor n/2 \rfloor}^{n-1} ck + an && \text{(by the inductive hypothesis)} \ &= \frac{2c}{n} \left( \sum_{k=1}^{n-1} k - \sum_{k=1}^{\lfloor n/2 \rfloor - 1} k \right) + an && \text{(split the sum)} \ &= \frac{2c}{n} \left( \frac{(n-1)n}{2} - \frac{(\lfloor n/2 \rfloor - 1)\lfloor n/2 \rfloor}{2} \right) + an && \text{(arithmetic series)} \ &\le \frac{2c}{n} \left( \frac{(n-1)n}{2} - \frac{(n/2 - 2)(n/2 - 1)}{2} \right) + an && \text{(since } \lfloor n/2 \rfloor \ge n/2 - 1\text{)} \ &= \frac{2c}{n} \left( \frac{n^2 - n}{2} - \frac{n^2/4 - 3n/2 + 2}{2} \right) + an && \text{(expand)} \ &= \frac{c}{n} \left( n^2 - n - \frac{n^2}{4} + \frac{3n}{2} - 2 \right) + an && \text{(factor the } 2 \text{ into the bracket)} \ &= \frac{c}{n} \left( \frac{3n^2}{4} + \frac{n}{2} - 2 \right) + an && \text{(combine like terms)} \ &= c \left( \frac{3n}{4} + \frac{1}{2} - \frac{2}{n} \right) + an && \text{(divide through by } n\text{)} \ &\le \frac{3cn}{4} + \frac{c}{2} + an && \text{(drop the negative term } -2c/n\text{)} \ &= cn - \left( \frac{cn}{4} - \frac{c}{2} - an \right) . \end{aligned}

(The fourth line uses that $x(x-1)$ is increasing for $x \ge 1$, so replacing $\lfloor n/2 \rfloor$ by the smaller value $n/2 - 1$ can only make the subtracted term smaller and the whole expression larger.) In order to complete the proof, we need to show that for sufficiently large $n$, this last expression is at most $cn$ or, equivalently, that $cn/4 - c/2 - an \ge 0$. If we add $c/2$ to both sides and factor out $n$, we get $n(c/4 - a) \ge c/2$. As long as we choose the constant $c$ so that $c/4 - a > 0$, i.e., $c > 4a$, we can divide both sides by $c/4 - a$, giving $$n \ge \frac{c/2}{c/4 - a} = \frac{2c}{c - 4a} .$$ Thus, if we assume that $T(n) = O(1)$ for $n < 2c/(c - 4a)$, then $E[T(n)] = O(n)$. We conclude that we can find any order statistic, and in particular the median, in expected linear time, assuming that the elements are distinct. > **Think about it.** Quicksort pays for both halves, which gives $n \lg n$. Selection throws away one half each round. If each round discards about a constant fraction, the work forms a geometric series $n + \tfrac{3}{4}n + (\tfrac{3}{4})^2 n + \cdots = O(n)$. That is the intuition the proof makes rigorous. ### Exercises 9.2-1 to 9.2-4 **9.2-1** Show that RANDOMIZED-SELECT never makes a recursive call to a 0-length array. **9.2-2** Argue that the indicator random variable $X_k$ and the value $T(\max(k-1, n-k))$ are independent. **9.2-3** Write an iterative version of RANDOMIZED-SELECT. **9.2-4** Suppose we use RANDOMIZED-SELECT to select the minimum element of the array $A = \langle 3, 2, 9, 0, 7, 5, 4, 8, 6, 1 \rangle$. Describe a sequence of partitions that results in a worst-case performance of RANDOMIZED-SELECT. --- ## 9.3 Selection in worst-case linear time We now examine a selection algorithm whose running time is $O(n)$ in the worst case. Like RANDOMIZED-SELECT, the algorithm SELECT finds the desired element by recursively partitioning the input array. Here, however, we *guarantee* a good split upon partitioning the array. SELECT uses the deterministic partitioning algorithm PARTITION from quicksort (see Section 7.1), but modified to take the element to partition around as an input parameter. The SELECT algorithm determines the $i$th smallest of an input array of $n > 1$ distinct elements by executing the following steps. (If $n = 1$, then SELECT merely returns its only input value as the $i$th smallest.) 1. Divide the $n$ elements of the input array into $\lfloor n/5 \rfloor$ groups of 5 elements each and at most one group made up of the remaining $n \bmod 5$ elements. 2. Find the median of each of the $\lceil n/5 \rceil$ groups by first insertion-sorting the elements of each group (of which there are at most 5) and then picking the median from the sorted list of group elements. 3. Use SELECT recursively to find the median $x$ of the $\lceil n/5 \rceil$ medians found in step 2. (If there are an even number of medians, then by our convention, $x$ is the lower median.) 4. Partition the input array around the median-of-medians $x$ using the modified version of PARTITION. Let $k$ be one more than the number of elements on the low side of the partition, so that $x$ is the $k$th smallest element and there are $n - k$ elements on the high side of the partition. 5. If $i = k$, then return $x$. Otherwise, use SELECT recursively to find the $i$th smallest element on the low side if $i < k$, or the $(i-k)$th smallest element on the high side if $i > k$. > **Think about it.** Why groups of 5 and a median of medians? Picking a pivot at random is a gamble. This scheme buys insurance: the pivot is guaranteed to be "pretty central", so neither side of the split can be huge. To analyze the running time of SELECT, we first determine a lower bound on the number of elements that are greater than the partitioning element $x$. Figure 9.1 helps us to visualize this bookkeeping. *Figure 9.1: Analysis of the algorithm SELECT. The $n$ elements are represented by small circles, and each group of 5 elements occupies a column. The medians of the groups are whitened, and the median-of-medians $x$ is labeled. (When finding the median of an even number of elements, we use the lower median.) Arrows go from larger elements to smaller, from which we can see that 3 out of every full group of 5 elements to the right of $x$ are greater than $x$, and 3 out of every group of 5 elements to the left of $x$ are less than $x$. The elements known to be greater than $x$ appear on a shaded background.* (Diagram of columns of 5 circles, group medians whitened, with $x$ marked as the median of those medians.) At least half of the medians found in step 2 are greater than or equal to the median-of-medians $x$.[^1] Thus, at least half of the $\lceil n/5 \rceil$ groups contribute at least 3 elements that are greater than $x$, except for the one group that has fewer than 5 elements if 5 does not divide $n$ exactly, and the one group containing $x$ itself. Discounting these two groups, it follows that the number of elements greater than $x$ is at least

\begin{aligned} 3\left( \left\lceil \frac{1}{2} \left\lceil \frac{n}{5} \right\rceil \right\rceil - 2 \right) &\ge 3 \left( \frac{1}{2} \cdot \frac{n}{5} - 2 \right) \ &= \frac{3n}{10} - 6 . \end{aligned}

Similarly, at least $3n/10 - 6$ elements are less than $x$. Thus, in the worst case, step 5 calls SELECT recursively on at most $n - (3n/10 - 6) = 7n/10 + 6$ elements. [^1]: Because of our assumption that the numbers are distinct, all medians except $x$ are either greater than or less than $x$. ### The recurrence We can now develop a recurrence for the worst-case running time $T(n)$ of the algorithm SELECT. Steps 1, 2, and 4 take $O(n)$ time. (Step 2 consists of $\Theta(n)$ calls of insertion sort on sets of size $\Theta(1)$.) Step 3 takes time $T(\lceil n/5 \rceil)$, and step 5 takes time at most $T(7n/10 + 6)$, assuming that $T$ is monotonically increasing. We make the assumption, which seems unmotivated at first, that any input of fewer than 140 elements requires $O(1)$ time; the origin of the magic constant 140 will be clear shortly. We can therefore obtain the recurrence

T(n) \le \begin{cases} O(1) & \text{if } n < 140 , \ T(\lceil n/5 \rceil) + T(7n/10 + 6) + O(n) & \text{if } n \ge 140 . \end{cases}

### Solving the recurrence by substitution We show that the running time is linear by substitution. More specifically, we will show that $T(n) \le cn$ for some suitably large constant $c$ and all $n > 0$. We begin by assuming that $T(n) \le cn$ for some suitably large constant $c$ and all $n < 140$; this assumption holds if $c$ is large enough. We also pick a constant $a$ such that the function described by the $O(n)$ term above (which describes the non-recursive component of the running time of the algorithm) is bounded above by $an$ for all $n > 0$. Substituting this inductive hypothesis into the right-hand side of the recurrence yields

\begin{aligned} T(n) &\le c \lceil n/5 \rceil + c(7n/10 + 6) + an && \text{(by the inductive hypothesis)} \ &\le cn/5 + c + 7cn/10 + 6c + an && \text{(since } \lceil n/5 \rceil \le n/5 + 1\text{)} \ &= 9cn/10 + 7c + an && \text{(combine like terms)} \ &= cn + \left( -cn/10 + 7c + an \right) , \end{aligned}

which is at most $cn$ if $$-cn/10 + 7c + an \le 0 . \tag{9.2}$$ Inequality (9.2) is equivalent to the inequality $c \ge 10a \cdot n/(n - 70)$ when $n > 70$. To see this:

\begin{aligned} -\frac{cn}{10} + 7c + an \le 0 &\iff 7c + an \le \frac{cn}{10} \ &\iff an \le c\left( \frac{n}{10} - 7 \right) = c \cdot \frac{n - 70}{10} \ &\iff c \ge \frac{10an}{n - 70} \qquad \text{(dividing by } (n-70)/10 > 0 \text{ when } n > 70\text{)} . \end{aligned}

Because we assume that $n \ge 140$, we have $n/(n - 70) \le 2$ (indeed $n \ge 140 \Rightarrow n \le 2(n - 70)$), and so choosing $c \ge 20a$ will satisfy inequality (9.2). (Note that there is nothing special about the constant 140; we could replace it by any integer strictly greater than 70 and then choose $c$ accordingly.) The worst-case running time of SELECT is therefore linear. ### Why this beats the sorting bound As in a comparison sort (see Section 8.1), SELECT and RANDOMIZED-SELECT determine information about the relative order of elements only by comparing elements. Recall from Chapter 8 that sorting requires $\Omega(n \lg n)$ time in the comparison model, even on average (see Problem 8-1). The linear-time sorting algorithms in Chapter 8 make assumptions about the input. In contrast, the linear-time selection algorithms in this chapter do not require any assumptions about the input. They are not subject to the $\Omega(n \lg n)$ lower bound because they manage to solve the selection problem without sorting. Thus, solving the selection problem by sorting and indexing, as presented in the introduction to this chapter, is asymptotically inefficient. ### Exercises 9.3-1 to 9.3-9 **9.3-1** In the algorithm SELECT, the input elements are divided into groups of 5. Will the algorithm work in linear time if they are divided into groups of 7? Argue that SELECT does not run in linear time if groups of 3 are used. **9.3-2** Analyze SELECT to show that if $n \ge 140$, then at least $\lceil n/4 \rceil$ elements are greater than the median-of-medians $x$ and at least $\lceil n/4 \rceil$ elements are less than $x$. **9.3-3** Show how quicksort can be made to run in $O(n \lg n)$ time in the worst case, assuming that all elements are distinct. **9.3-4\*** Suppose that an algorithm uses only comparisons to find the $i$th smallest element in a set of $n$ elements. Show that it can also find the $i - 1$ smaller elements and the $n - i$ larger elements without performing any additional comparisons. **9.3-5** Suppose that you have a "black-box" worst-case linear-time median subroutine. Give a simple, linear-time algorithm that solves the selection problem for an arbitrary order statistic. **9.3-6** The $k$th *quantiles* of an $n$-element set are the $k - 1$ order statistics that divide the sorted set into $k$ equal-sized sets (to within 1). Give an $O(n \lg k)$-time algorithm to list the $k$th quantiles of a set. **9.3-7** Describe an $O(n)$-time algorithm that, given a set $S$ of $n$ distinct numbers and a positive integer $k \le n$, determines the $k$ numbers in $S$ that are closest to the median of $S$. **9.3-8** Let $X[1 \ldots n]$ and $Y[1 \ldots n]$ be two arrays, each containing $n$ numbers already in sorted order. Give an $O(\lg n)$-time algorithm to find the median of all $2n$ elements in arrays $X$ and $Y$. **9.3-9** Professor Olay is consulting for an oil company, which is planning a large pipeline running east to west through an oil field of $n$ wells. The company wants to connect a spur pipeline from each well directly to the main pipeline along a shortest route (either north or south), as shown in Figure 9.2. Given the $x$- and $y$-coordinates of the wells, how should the professor pick the optimal location of the main pipeline, which would be the one that minimizes the total length of the spurs? Show how to determine the optimal location in linear time. *Figure 9.2: Professor Olay needs to determine the position of the east-west oil pipeline that minimizes the total length of the north-south spurs.* (Diagram of wells scattered in the plane with an east-west main pipeline and north-south spurs connecting each well to it.) --- ## Problems ### 9-1 Largest $i$ numbers in sorted order Given a set of $n$ numbers, we wish to find the $i$ largest in sorted order using a comparison-based algorithm. Find the algorithm that implements each of the following methods with the best asymptotic worst-case running time, and analyze the running times of the algorithms in terms of $n$ and $i$. **a.** Sort the numbers, and list the $i$ largest. **b.** Build a max-priority queue from the numbers, and call EXTRACT-MAX $i$ times. **c.** Use an order-statistic algorithm to find the $i$th largest number, partition around that number, and sort the $i$ largest numbers. ### 9-2 Weighted median For $n$ distinct elements $x_1, x_2, \ldots, x_n$ with positive weights $w_1, w_2, \ldots, w_n$ such that $\sum_{i=1}^{n} w_i = 1$, the **weighted (lower) median** is the element $x_k$ satisfying $$\sum_{x_i < x_k} w_i < \frac{1}{2}$$ and $$\sum_{x_i > x_k} w_i \le \frac{1}{2} .$$ For example, if the elements are $0.1, 0.35, 0.05, 0.1, 0.15, 0.05, 0.2$ and each element equals its weight (that is, $w_i = x_i$ for $i = 1, 2, \ldots, 7$), then the median is $0.1$, but the weighted median is $0.2$. **a.** Argue that the median of $x_1, x_2, \ldots, x_n$ is the weighted median of the $x_i$ with weights $w_i = 1/n$ for $i = 1, 2, \ldots, n$. **b.** Show how to compute the weighted median of $n$ elements in $O(n \lg n)$ worst-case time using sorting. **c.** Show how to compute the weighted median in $\Theta(n)$ worst-case time using a linear-time median algorithm such as SELECT from Section 9.3. The **post-office location problem** is defined as follows. We are given $n$ points $p_1, p_2, \ldots, p_n$ with associated weights $w_1, w_2, \ldots, w_n$. We wish to find a point $p$ (not necessarily one of the input points) that minimizes the sum $\sum_{i=1}^{n} w_i\, d(p, p_i)$, where $d(a, b)$ is the distance between points $a$ and $b$. **d.** Argue that the weighted median is a best solution for the 1-dimensional post-office location problem, in which points are simply real numbers and the distance between points $a$ and $b$ is $d(a, b) = |a - b|$. **e.** Find the best solution for the 2-dimensional post-office location problem, in which the points are $(x, y)$ coordinate pairs and the distance between points $a = (x_1, y_1)$ and $b = (x_2, y_2)$ is the *Manhattan distance* given by $d(a, b) = |x_1 - x_2| + |y_1 - y_2|$. ### 9-3 Small order statistics We showed that the worst-case number $T(n)$ of comparisons used by SELECT to select the $i$th order statistic from $n$ numbers satisfies $T(n) = \Theta(n)$, but the constant hidden by the $\Theta$-notation is rather large. When $i$ is small relative to $n$, we can implement a different procedure that uses SELECT as a subroutine but makes fewer comparisons in the worst case. **a.** Describe an algorithm that uses $U_i(n)$ comparisons to find the $i$th smallest of $n$ elements, where

U_i(n) = \begin{cases} T(n) & \text{if } i \ge n/2 , \ \lfloor n/2 \rfloor + U_i(\lceil n/2 \rceil) + T(2i) & \text{otherwise.} \end{cases}

\begin{aligned} h_1(k) &= k \bmod m, \ h_2(k) &= 1 + (k \bmod m’), \end{aligned}

\begin{aligned} \Pr{X_{ik} = 1} &= \frac{(k-i-1)!}{(k-i+1)!} \ &= \frac{1}{(k-i+1)(k-i)}. \end{aligned}

* * h . * * S h o w t ha t

\begin{aligned} E[C] &= \sum_{i=1}^{k-1} \frac{1}{(k-i+1)(k-i)} \ &= 1 - \frac{1}{k}. \end{aligned}

w(i,j) = \sum_{l=i}^{j} p_l + \sum_{l=i-1}^{j} q_l . \tag{15.12}

Thus, if $k_r$ is the root of an optimal subtree containing keys $k_i, \ldots, k_j$, we have

e[i,j] = p_r + \bigl(e[i,r-1] + w(i,r-1)\bigr) + \bigl(e[r+1,j] + w(r+1,j)\bigr).

N o t in g t ha t

w(i,j) = w(i,r-1) + p_r + w(r+1,j),

we rewrite $e[i,j]$ as

\begin{aligned} e[i,j] &= p_r + e[i,r-1] + w(i,r-1) + e[r+1,j] + w(r+1,j) \ &= e[i,r-1] + e[r+1,j] + \bigl(w(i,r-1) + p_r + w(r+1,j)\bigr) \ &= e[i,r-1] + e[r+1,j] + w(i,j) . \end{aligned} \tag{15.13}

The recursive equation (15.13) assumes that we know which node $k_r$ to use as the root. We choose the root that gives the lowest expected search cost, giving us our final recursive formulation:

e[i,j] = \begin{cases} q_{i-1} & \text{if } j = i-1, \[4pt] \displaystyle\min_{i \le r \le j} {, e[i,r-1] + e[r+1,j] + w(i,j) ,} & \text{if } i \le j . \end{cases} \tag{15.14}

The $e[i,j]$ values give the expected search costs in optimal binary search trees. To help us keep track of the structure of optimal binary search trees, we define $root[i,j]$, for $1 \le i \le j \le n$, to be the index $r$ for which $k_r$ is the root of an optimal binary search tree containing keys $k_i, \ldots, k_j$. Although we will see how to compute the values of $root[i,j]$, we leave the construction of an optimal binary search tree from these values as Exercise 15.5-1. ### Step 3: Computing the expected search cost of an optimal binary search tree At this point, you may have noticed some similarities between our characterizations of optimal binary search trees and matrix-chain multiplication. For both problem domains, our subproblems consist of contiguous index subranges. A direct, recursive implementation of equation (15.14) would be as inefficient as a direct, recursive matrix-chain multiplication algorithm. Instead, we store the $e[i,j]$ values in a table $e[1..n+1,\, 0..n]$. The first index needs to run to $n+1$ rather than $n$ because in order to have a subtree containing only the dummy key $d_n$, we need to compute and store $e[n+1,n]$. The second index needs to start from $0$ because in order to have a subtree containing only the dummy key $d_0$, we need to compute and store $e[1,0]$. We use only the entries $e[i,j]$ for which $j \ge i-1$. We also use a table $root[i,j]$, for recording the root of the subtree containing keys $k_i, \ldots, k_j$. This table uses only the entries for which $1 \le i \le j \le n$. We will need one other table for efficiency. Rather than compute the value of $w(i,j)$ from scratch every time we are computing $e[i,j]$ (which would take $\Theta(j-i)$ additions), we store these values in a table $w[1..n+1,\, 0..n]$. For the base case, we compute $w[i,i-1] = q_{i-1}$ for $1 \le i \le n+1$. For $j \ge i$, we compute

w[i,j] = w[i,j-1] + p_j + q_j . \tag{15.15}

3 \cdot \text{cost(copy)} + \text{cost(replace)} + \text{cost(delete)} + 4 \cdot \text{cost(insert)} + \text{cost(twiddle)} + \text{cost(kill)} .

\begin{aligned} \sum_{i=0}^{k-1} \left\lfloor \frac{n}{2^i} \right\rfloor &< n \sum_{i=0}^{\infty} \frac{1}{2^i} && \text{(since } \lfloor x \rfloor \le x \text{, and extending the sum to infinity adds nonnegative terms)}\ &= 2n, && \text{(by equation (A.6), the infinite geometric series with ratio } 1/2 \text{)} \end{aligned}

The worst-case time for a sequence of $n$ INCREMENT operations on an initially zero counter is therefore $O(n)$. The average cost of each operation, and therefore the amortized cost per operation, is $O(n)/n = O(1)$. ### Exercises 17.1-1 to 17.1-3 **17.1-1** If the set of stack operations included a MULTIPUSH operation, which pushes $k$ items onto the stack, would the $O(1)$ bound on the amortized cost of stack operations continue to hold? **17.1-2** Show that if a DECREMENT operation were included in the $k$-bit counter example, $n$ operations could cost as much as $\Theta(nk)$ time. **17.1-3** Suppose we perform a sequence of $n$ operations on a data structure in which the $i$th operation costs $i$ if $i$ is an exact power of 2, and 1 otherwise. Use aggregate analysis to determine the amortized cost per operation. --- ## 17.2 The accounting method In the accounting method of amortized analysis, we assign differing charges to different operations, with some operations charged more or less than they actually cost. We call the amount we charge an operation its *amortized cost*. When an operation's amortized cost exceeds its actual cost, we assign the difference to specific objects in the data structure as *credit*. Credit can help pay for later operations whose amortized cost is less than their actual cost. Thus, we can view the amortized cost of an operation as being split between its actual cost and credit that is either deposited or used up. Different operations may have different amortized costs. This method differs from aggregate analysis, in which all operations have the same amortized cost. We must choose the amortized costs of operations carefully. If we want to show that in the worst case the average cost per operation is small by analyzing with amortized costs, we must ensure that the total amortized cost of a sequence of operations provides an upper bound on the total actual cost of the sequence. Moreover, as in aggregate analysis, this relationship must hold for all sequences of operations. If we denote the actual cost of the $i$th operation by $c_i$ and the amortized cost of the $i$th operation by $\hat{c}_i$, we require

\sum_{i=1}^{n} \hat{c}i \ge \sum{i=1}^{n} c_i \tag{17.1}

for all sequences of $n$ operations. The total credit stored in the data structure is the difference between the total amortized cost and the total actual cost, or $\sum_{i=1}^{n} \hat{c}_i - \sum_{i=1}^{n} c_i$. By inequality (17.1), the total credit associated with the data structure must be nonnegative at all times. If we ever were to allow the total credit to become negative (the result of undercharging early operations with the promise of repaying the account later on), then the total amortized costs incurred at that time would be below the total actual costs incurred; for the sequence of operations up to that time, the total amortized cost would not be an upper bound on the total actual cost. Thus, we must take care that the total credit in the data structure never becomes negative. ### Stack operations To illustrate the accounting method of amortized analysis, let us return to the stack example. Recall that the actual costs of the operations were

\begin{aligned} \text{PUSH} &: 1,\ \text{POP} &: 1,\ \text{MULTIPOP} &: \min(k, s), \end{aligned}

where $k$ is the argument supplied to MULTIPOP and $s$ is the stack size when it is called. Let us assign the following amortized costs:

\begin{aligned} \text{PUSH} &: 2,\ \text{POP} &: 0,\ \text{MULTIPOP} &: 0. \end{aligned}

Note that the amortized cost of MULTIPOP is a constant (0), whereas the actual cost is variable. Here, all three amortized costs are constant. In general, the amortized costs of the operations under consideration may differ from each other, and they may even differ asymptotically. We shall now show that we can pay for any sequence of stack operations by charging the amortized costs. Suppose we use a dollar bill to represent each unit of cost. We start with an empty stack. Recall the analogy of Section 10.1 between the stack data structure and a stack of plates in a cafeteria. When we push a plate on the stack, we use 1 dollar to pay the actual cost of the push and are left with a credit of 1 dollar (out of the 2 dollars charged), which we leave on top of the plate. At any point in time, every plate on the stack has a dollar of credit on it. The dollar stored on the plate serves as prepayment for the cost of popping it from the stack. When we execute a POP operation, we charge the operation nothing and pay its actual cost using the credit stored in the stack. To pop a plate, we take the dollar of credit off the plate and use it to pay the actual cost of the operation. Thus, by charging the PUSH operation a little bit more, we can charge the POP operation nothing. Moreover, we can also charge MULTIPOP operations nothing. To pop the first plate, we take the dollar of credit off the plate and use it to pay the actual cost of a POP operation. To pop a second plate, we again have a dollar of credit on the plate to pay for the POP operation, and so on. Thus, we have always charged enough up front to pay for MULTIPOP operations. In other words, since each plate on the stack has 1 dollar of credit on it, and the stack always has a nonnegative number of plates, we have ensured that the amount of credit is always nonnegative. Thus, for any sequence of $n$ PUSH, POP, and MULTIPOP operations, the total amortized cost is an upper bound on the total actual cost. Since the total amortized cost is $O(n)$, so is the total actual cost. > **Invariant, in symbols.** After every operation, the total credit equals the number of plates on the stack: > > $$\text{credit} = \sum_{i=1}^{n} \hat{c}_i - \sum_{i=1}^{n} c_i = s \ge 0 .$$ > > PUSH: charged $2$, actual $1$, so credit rises by $1$ and $s$ rises by $1$. POP: charged $0$, actual $1$, so credit falls by $1$ and $s$ falls by $1$. MULTIPOP: charged $0$, actual $\min(k,s)$, so credit falls by $\min(k,s)$ and $s$ falls by $\min(k,s)$. The invariant is preserved, and the stack size is never negative, so (17.1) holds. The total amortized cost is at most $2n$, which is $O(n)$. ### Incrementing a binary counter As another illustration of the accounting method, we analyze the INCREMENT operation on a binary counter that starts at zero. As we observed earlier, the running time of this operation is proportional to the number of bits flipped, which we shall use as our cost for this example. Let us once again use a dollar bill to represent each unit of cost (the flipping of a bit in this example). For the amortized analysis, let us charge an amortized cost of 2 dollars to set a bit to 1. When a bit is set, we use 1 dollar (out of the 2 dollars charged) to pay for the actual setting of the bit, and we place the other dollar on the bit as credit to be used later when we flip the bit back to 0. At any point in time, every 1 in the counter has a dollar of credit on it, and thus we can charge nothing to reset a bit to 0; we just pay for the reset with the dollar bill on the bit. Now we can determine the amortized cost of INCREMENT. The cost of resetting the bits within the while loop is paid for by the dollars on the bits that are reset. The INCREMENT procedure sets at most one bit, in line 6, and therefore the amortized cost of an INCREMENT operation is at most 2 dollars. The number of 1s in the counter never becomes negative, and thus the amount of credit stays nonnegative at all times. Thus, for $n$ INCREMENT operations, the total amortized cost is $O(n)$, which bounds the total actual cost. ### Exercises 17.2-1 to 17.2-3 **17.2-1** Suppose we perform a sequence of stack operations on a stack whose size never exceeds $k$. After every $k$ operations, we make a copy of the entire stack for backup purposes. Show that the cost of $n$ stack operations, including copying the stack, is $O(n)$ by assigning suitable amortized costs to the various stack operations. **17.2-2** Redo Exercise 17.1-3 using an accounting method of analysis. **17.2-3** Suppose we wish not only to increment a counter but also to reset it to zero (i.e., make all bits in it 0). Counting the time to examine or modify a bit as $\Theta(1)$, show how to implement a counter as an array of bits so that any sequence of $n$ INCREMENT and RESET operations takes time $O(n)$ on an initially zero counter. (*Hint:* Keep a pointer to the high-order 1.) --- ## 17.3 The potential method Instead of representing prepaid work as credit stored with specific objects in the data structure, the *potential method* of amortized analysis represents the prepaid work as "potential energy," or just "potential," that can be released to pay for future operations. We associate the potential with the data structure as a whole rather than with specific objects within the data structure. The potential method works as follows. We will perform $n$ operations, starting with an initial data structure $D_0$. For each $i = 1, 2, \ldots, n$, we let $c_i$ be the actual cost of the $i$th operation and $D_i$ be the data structure that results after applying the $i$th operation to data structure $D_{i-1}$. A *potential function* $\Phi$ maps each data structure $D_i$ to a real number $\Phi(D_i)$, which is the *potential* associated with data structure $D_i$. The *amortized cost* $\hat{c}_i$ of the $i$th operation with respect to potential function $\Phi$ is defined by

\hat{c}i = c_i + \Phi(D_i) - \Phi(D{i-1}) . \tag{17.2}

The amortized cost of each operation is therefore its actual cost plus the change in potential due to the operation. By equation (17.2), the total amortized cost of the $n$ operations is

\begin{aligned} \sum_{i=1}^{n} \hat{c}i &= \sum{i=1}^{n} \bigl( c_i + \Phi(D_i) - \Phi(D_{i-1}) \bigr)\ &= \sum_{i=1}^{n} c_i + \sum_{i=1}^{n} \bigl( \Phi(D_i) - \Phi(D_{i-1}) \bigr)\ &= \sum_{i=1}^{n} c_i + \Phi(D_n) - \Phi(D_0) . \end{aligned} \tag{17.3}

The last equality follows from equation (A.9) because the $\Phi(D_i)$ terms telescope: $\bigl(\Phi(D_1) - \Phi(D_0)\bigr) + \bigl(\Phi(D_2) - \Phi(D_1)\bigr) + \cdots + \bigl(\Phi(D_n) - \Phi(D_{n-1})\bigr) = \Phi(D_n) - \Phi(D_0)$. If we can define a potential function $\Phi$ so that $\Phi(D_n) \ge \Phi(D_0)$, then the total amortized cost $\sum_{i=1}^{n} \hat{c}_i$ gives an upper bound on the total actual cost $\sum_{i=1}^{n} c_i$. In practice, we do not always know how many operations might be performed. Therefore, if we require that $\Phi(D_i) \ge \Phi(D_0)$ for all $i$, then we guarantee, as in the accounting method, that we pay in advance. We usually just define $\Phi(D_0)$ to be 0 and then show that $\Phi(D_i) \ge 0$ for all $i$. (See Exercise 17.3-1 for an easy way to handle cases in which $\Phi(D_0) \ne 0$.) Intuitively, if the potential difference $\Phi(D_i) - \Phi(D_{i-1})$ of the $i$th operation is positive, then the amortized cost $\hat{c}_i$ represents an overcharge to the $i$th operation, and the potential of the data structure increases. If the potential difference is negative, then the amortized cost represents an undercharge to the $i$th operation, and the decrease in the potential pays for the actual cost of the operation. The amortized costs defined by equations (17.2) and (17.3) depend on the choice of the potential function $\Phi$. Different potential functions may yield different amortized costs yet still be upper bounds on the actual costs. We often find trade-offs that we can make in choosing a potential function; the best potential function to use depends on the desired time bounds. > **Think about it.** Think of $\Phi$ as the height of water in a reservoir above a mill. Cheap operations let the reservoir fill; an expensive operation opens the sluice and the stored water turns the wheel. You can never take out more than was put in, so $\Phi \ge 0$ is the rule that keeps the books honest. ### Stack operations To illustrate the potential method, we return once again to the example of the stack operations PUSH, POP, and MULTIPOP. We define the potential function $\Phi$ on a stack to be the number of objects in the stack. For the empty stack $D_0$ with which we start, we have $\Phi(D_0) = 0$. Since the number of objects in the stack is never negative, the stack $D_i$ that results after the $i$th operation has nonnegative potential, and thus

\begin{aligned} \Phi(D_i) &\ge 0\ &= \Phi(D_0) . \end{aligned}

The total amortized cost of $n$ operations with respect to $\Phi$ therefore represents an upper bound on the actual cost. Let us now compute the amortized costs of the various stack operations. If the $i$th operation on a stack containing $s$ objects is a PUSH operation, then the potential difference is

\begin{aligned} \Phi(D_i) - \Phi(D_{i-1}) &= (s+1) - s\ &= 1 . \end{aligned}

B y e q u a t i o n (17.2), t h e am or t i z e d cos t o f t hi s P U S H o p er a t i o ni s

\begin{aligned} \hat{c}i &= c_i + \Phi(D_i) - \Phi(D{i-1})\ &= 1 + 1\ &= 2 . \end{aligned}

Suppose that the $i$th operation on the stack is $\text{MULTIPOP}(S, k)$, which causes $k' = \min(k, s)$ objects to be popped off the stack. The actual cost of the operation is $k'$, and the potential difference is

\Phi(D_i) - \Phi(D_{i-1}) = -k’ .

T h u s, t h e am or t i z e d cos t o f t h e M U L T I P O P o p er a t i o ni s

\begin{aligned} \hat{c}i &= c_i + \Phi(D_i) - \Phi(D{i-1})\ &= k’ - k’\ &= 0 . \end{aligned}

Similarly, the amortized cost of an ordinary POP operation is 0 (actual cost 1, potential difference $-1$). The amortized cost of each of the three operations is $O(1)$, and thus the total amortized cost of a sequence of $n$ operations is $O(n)$. Since we have already argued that $\Phi(D_i) \ge \Phi(D_0)$, the total amortized cost of $n$ operations is an upper bound on the total actual cost. The worst-case cost of $n$ operations is therefore $O(n)$. ### Incrementing a binary counter As another example of the potential method, we again look at incrementing a binary counter. This time, we define the potential of the counter after the $i$th INCREMENT operation to be $b_i$, the number of 1s in the counter after the $i$th operation. Let us compute the amortized cost of an INCREMENT operation. Suppose that the $i$th INCREMENT operation resets $t_i$ bits. The actual cost of the operation is therefore at most $t_i + 1$, since in addition to resetting $t_i$ bits, it sets at most one bit to 1. If $b_i = 0$, then the $i$th operation resets all $k$ bits, and so $b_{i-1} = t_i = k$. If $b_i > 0$, then $b_i = b_{i-1} - t_i + 1$. In either case, $b_i \le b_{i-1} - t_i + 1$, and the potential difference is

\begin{aligned} \Phi(D_i) - \Phi(D_{i-1}) &\le (b_{i-1} - t_i + 1) - b_{i-1}\ &= 1 - t_i . \end{aligned}

T h e am or t i z e d cos t i s t h er e f or e

\begin{aligned} \hat{c}i &= c_i + \Phi(D_i) - \Phi(D{i-1})\ &\le (t_i + 1) + (1 - t_i)\ &= 2 . \end{aligned}

If the counter starts at zero, then $\Phi(D_0) = 0$. Since $\Phi(D_i) \ge 0$ for all $i$, the total amortized cost of a sequence of $n$ INCREMENT operations is an upper bound on the total actual cost, and so the worst-case cost of $n$ INCREMENT operations is $O(n)$. The potential method gives us an easy way to analyze the counter even when it does not start at zero. The counter starts with $b_0$ 1s, and after $n$ INCREMENT operations it has $b_n$ 1s, where $0 \le b_0, b_n \le k$. (Recall that $k$ is the number of bits in the counter.) We can rewrite equation (17.3) as

\sum_{i=1}^{n} c_i = \sum_{i=1}^{n} \hat{c}_i - \Phi(D_n) + \Phi(D_0) . \tag{17.4}

We have $\hat{c}_i \le 2$ for all $1 \le i \le n$. Since $\Phi(D_0) = b_0$ and $\Phi(D_n) = b_n$, the total actual cost of $n$ INCREMENT operations is

\begin{aligned} \sum_{i=1}^{n} c_i &\le \sum_{i=1}^{n} 2 - b_n + b_0\ &= 2n - b_n + b_0 . \end{aligned}

Note in particular that since $b_0 \le k$, as long as $k = O(n)$, the total actual cost is $O(n)$. In other words, if we execute at least $n = \Omega(k)$ INCREMENT operations, the total actual cost is $O(n)$, no matter what initial value the counter contains. ### Exercises 17.3-1 to 17.3-7 **17.3-1** Suppose we have a potential function $\Phi$ such that $\Phi(D_i) \ge \Phi(D_0)$ for all $i$, but $\Phi(D_0) \ne 0$. Show that there exists a potential function $\Phi'$ such that $\Phi'(D_0) = 0$, $\Phi'(D_i) \ge 0$ for all $i \ge 1$, and the amortized costs using $\Phi'$ are the same as the amortized costs using $\Phi$. **17.3-2** Redo Exercise 17.1-3 using a potential method of analysis. **17.3-3** Consider an ordinary binary min-heap data structure with $n$ elements supporting the instructions INSERT and EXTRACT-MIN in $O(\lg n)$ worst-case time. Give a potential function $\Phi$ such that the amortized cost of INSERT is $O(\lg n)$ and the amortized cost of EXTRACT-MIN is $O(1)$, and show that it works. **17.3-4** What is the total cost of executing $n$ of the stack operations PUSH, POP, and MULTIPOP, assuming that the stack begins with $s_0$ objects and finishes with $s_n$ objects? **17.3-5** Suppose that a counter begins at a number with $b$ 1s in its binary representation, rather than at 0. Show that the cost of performing $n$ INCREMENT operations is $O(n)$ if $n = \Omega(b)$. (Do not assume that $b$ is constant.) **17.3-6** Show how to implement a queue with two ordinary stacks (Exercise 10.1-6) so that the amortized cost of each ENQUEUE and each DEQUEUE operation is $O(1)$. **17.3-7** Design a data structure to support the following two operations for a dynamic multiset $S$ of integers, which allows duplicate values: - $\text{INSERT}(S, x)$ inserts $x$ into $S$. - $\text{DELETE-LARGER-HALF}(S)$ deletes the largest $\lceil |S|/2 \rceil$ elements from $S$. Explain how to implement this data structure so that any sequence of $m$ INSERT and DELETE-LARGER-HALF operations runs in $O(m)$ time. Your implementation should also include a way to output the elements of $S$ in $O(|S|)$ time. --- ## 17.4 Dynamic tables We do not always know in advance how many objects some applications will store in a table. We might allocate space for a table, only to find out later that it is not enough. We must then reallocate the table with a larger size and copy all objects stored in the original table over into the new, larger table. Similarly, if many objects have been deleted from the table, it may be worthwhile to reallocate the table with a smaller size. In this section, we study this problem of dynamically expanding and contracting a table. Using amortized analysis, we shall show that the amortized cost of insertion and deletion is only $O(1)$, even though the actual cost of an operation is large when it triggers an expansion or a contraction. Moreover, we shall see how to guarantee that the unused space in a dynamic table never exceeds a constant fraction of the total space. We assume that the dynamic table supports the operations TABLE-INSERT and TABLE-DELETE. TABLE-INSERT inserts into the table an item that occupies a single *slot*, that is, a space for one item. Likewise, TABLE-DELETE removes an item from the table, thereby freeing a slot. The details of the data-structuring method used to organize the table are unimportant; we might use a stack (Section 10.1), a heap (Chapter 6), or a hash table (Chapter 11). We might also use an array or collection of arrays to implement object storage, as we did in Section 10.3. We shall find it convenient to use a concept introduced in our analysis of hashing (Chapter 11). We define the *load factor* $\alpha(T)$ of a nonempty table $T$ to be the number of items stored in the table divided by the size (number of slots) of the table. We assign an empty table (one with no items) size 0, and we define its load factor to be 1. If the load factor of a dynamic table is bounded below by a constant, the unused space in the table is never more than a constant fraction of the total amount of space. We start by analyzing a dynamic table in which we only insert items. We then consider the more general case in which we both insert and delete items. > **Think about it.** A suitcase that is too small forces you to buy a bigger one and move every sock over. How big should the new suitcase be? Just one sock bigger, and you repack constantly; twice as big, and repacking becomes rare. That choice is the heart of this section. ### 17.4.1 Table expansion Let us assume that storage for a table is allocated as an array of slots. A table fills up when all slots have been used or, equivalently, when its load factor is 1.¹ In some software environments, upon attempting to insert an item into a full table, the only alternative is to abort with an error. We shall assume, however, that our software environment, like many modern ones, provides a memory-management system that can allocate and free blocks of storage on request. Thus, upon inserting an item into a full table, we can *expand* the table by allocating a new table with more slots than the old table had. Because we always need the table to reside in contiguous memory, we must allocate a new array for the larger table and then copy items from the old table into the new table. A common heuristic allocates a new table with twice as many slots as the old one. If the only table operations are insertions, then the load factor of the table is always at least $1/2$, and thus the amount of wasted space never exceeds half the total space in the table. In the following pseudocode, we assume that $T$ is an object representing the table. The attribute $T.table$ contains a pointer to the block of storage representing the table, $T.num$ contains the number of items in the table, and $T.size$ gives the total number of slots in the table. Initially, the table is empty: $T.num = T.size = 0$. ```text TABLE-INSERT(T, x) 1 if T.size == 0 2 allocate T.table with 1 slot 3 T.size = 1 4 if T.num == T.size 5 allocate new-table with 2 * T.size slots 6 insert all items in T.table into new-table 7 free T.table 8 T.table = new-table 9 T.size = 2 * T.size 10 insert x into T.table 11 T.num = T.num + 1 ``` ¹ In some situations, such as an open-address hash table, we may wish to consider a table to be full if its load factor equals some constant strictly less than 1. (See Exercise 17.4-1.) Notice that we have two "insertion" procedures here: the TABLE-INSERT procedure itself and the *elementary insertion* into a table in lines 6 and 10. We can analyze the running time of TABLE-INSERT in terms of the number of elementary insertions by assigning a cost of 1 to each elementary insertion. We assume that the actual running time of TABLE-INSERT is linear in the time to insert individual items, so that the overhead for allocating an initial table in line 2 is constant and the overhead for allocating and freeing storage in lines 5 and 7 is dominated by the cost of transferring items in line 6. We call the event in which lines 5-9 are executed an *expansion*. Let us analyze a sequence of $n$ TABLE-INSERT operations on an initially empty table. What is the cost $c_i$ of the $i$th operation? If the current table has room for the new item (or if this is the first operation), then $c_i = 1$, since we need only perform the one elementary insertion in line 10. If the current table is full, however, and an expansion occurs, then $c_i = i$: the cost is 1 for the elementary insertion in line 10 plus $i - 1$ for the items that we must copy from the old table to the new table in line 6. If we perform $n$ operations, the worst-case cost of an operation is $O(n)$, which leads to an upper bound of $O(n^2)$ on the total running time for $n$ operations. This bound is not tight, because we rarely expand the table in the course of $n$ TABLE-INSERT operations. Specifically, the $i$th operation causes an expansion only when $i - 1$ is an exact power of 2. The amortized cost of an operation is in fact $O(1)$, as we can show using aggregate analysis. The cost of the $i$th operation is

c_i = \begin{cases} i & \text{if } i - 1 \text{ is an exact power of 2},\ 1 & \text{otherwise}. \end{cases}

The total cost of $n$ TABLE-INSERT operations is therefore

\begin{aligned} \sum_{i=1}^{n} c_i &\le n + \sum_{j=0}^{\lfloor \lg n \rfloor} 2^j && \text{(at most } n \text{ operations cost 1; expansion costs are powers of 2)}\ &= n + \left( 2^{\lfloor \lg n \rfloor + 1} - 1 \right) && \text{(finite geometric series)}\ &< n + 2n && \text{(since } 2^{\lfloor \lg n \rfloor} \le n \text{)}\ &= 3n , \end{aligned}

because at most $n$ operations cost 1 and the costs of the remaining operations form a geometric series. Since the total cost of $n$ TABLE-INSERT operations is bounded by $3n$, the amortized cost of a single operation is at most 3. By using the accounting method, we can gain some feeling for why the amortized cost of a TABLE-INSERT operation should be 3. Intuitively, each item pays for 3 elementary insertions: inserting itself into the current table, moving itself when the table expands, and moving another item that has already been moved once when the table expands. For example, suppose that the size of the table is $m$ immediately after an expansion. Then the table holds $m/2$ items, and it contains no credit. We charge 3 dollars for each insertion. The elementary insertion that occurs immediately costs 1 dollar. We place another dollar as credit on the item inserted. We place the third dollar as credit on one of the $m/2$ items already in the table. The table will not fill again until we have inserted another $m/2 - 1$ items, and thus, by the time the table contains $m$ items and is full, we will have placed a dollar on each item to pay to reinsert it during the expansion. We can use the potential method to analyze a sequence of $n$ TABLE-INSERT operations, and we shall use it in Section 17.4.2 to design a TABLE-DELETE operation that has an $O(1)$ amortized cost as well. We start by defining a potential function $\Phi$ that is 0 immediately after an expansion but builds to the table size by the time the table is full, so that we can pay for the next expansion by the potential. The function

\Phi(T) = 2 \cdot T.num - T.size \tag{17.5}

is one possibility. Immediately after an expansion, we have $T.num = T.size/2$, and thus $\Phi(T) = 0$, as desired. Immediately before an expansion, we have $T.num = T.size$, and thus $\Phi(T) = T.num$, as desired. The initial value of the potential is 0, and since the table is always at least half full, $T.num \ge T.size/2$, which implies that $\Phi(T)$ is always nonnegative. Thus, the sum of the amortized costs of $n$ TABLE-INSERT operations gives an upper bound on the sum of the actual costs. To analyze the amortized cost of the $i$th TABLE-INSERT operation, we let $num_i$ denote the number of items stored in the table after the $i$th operation, $size_i$ denote the total size of the table after the $i$th operation, and $\Phi_i$ denote the potential after the $i$th operation. Initially, we have $num_0 = 0$, $size_0 = 0$, and $\Phi_0 = 0$. > **Proof (amortized cost of TABLE-INSERT is 3).** > > *Case 1: the $i$th operation does not trigger an expansion.* Then $size_i = size_{i-1}$, the actual cost is $c_i = 1$, and $num_i = num_{i-1} + 1$. The amortized cost is > > $$ > \begin{aligned} > \hat{c}_i &= c_i + \Phi_i - \Phi_{i-1} && \text{(definition (17.2))}\\ > &= 1 + (2 \cdot num_i - size_i) - (2 \cdot num_{i-1} - size_{i-1}) && \text{(by (17.5))}\\ > &= 1 + (2 \cdot num_i - size_i) - (2(num_i - 1) - size_i) && \text{(since } num_{i-1} = num_i - 1,\ size_{i-1} = size_i)\\ > &= 1 + 2 && \text{(the } size_i \text{ and } 2 \cdot num_i \text{ terms cancel)}\\ > &= 3 . > \end{aligned} > $$ > > *Case 2: the $i$th operation triggers an expansion.* Then $size_i = 2 \cdot size_{i-1}$, and the table was full, so $size_{i-1} = num_{i-1} = num_i - 1$. Hence $size_i = 2(num_i - 1)$. The actual cost is $c_i = num_i$ (one elementary insertion plus $num_i - 1$ copies). The potentials are > > $$ > \begin{aligned} > \Phi_i &= 2 \cdot num_i - 2(num_i - 1) = 2,\\ > \Phi_{i-1} &= 2(num_i - 1) - (num_i - 1) = num_i - 1 . > \end{aligned} > $$ > > Therefore the amortized cost is > > $$ > \begin{aligned} > \hat{c}_i &= c_i + \Phi_i - \Phi_{i-1}\\ > &= num_i + 2 - (num_i - 1)\\ > &= 3 . > \end{aligned} > $$ > > In both cases $\hat{c}_i = 3$, so the amortized cost of TABLE-INSERT is $O(1)$. *Figure 17.3: The effect of a sequence of $n$ TABLE-INSERT operations on the number $num_i$ of items in the table, the number $size_i$ of slots in the table, and the potential $\Phi_i = 2 \cdot num_i - size_i$, each being measured after the $i$th operation. The thin line shows $num_i$, the dashed line shows $size_i$, and the thick line shows $\Phi_i$. Notice that immediately before an expansion, the potential has built up to the number of items in the table, and therefore it can pay for moving all the items to the new table. Afterwards, the potential drops to 0, but it is immediately increased by 2 upon inserting the item that caused the expansion.* (Plot, horizontal axis $i$ from 0 to 32, vertical axis 0 to 32; not reproduced.) Figure 17.3 plots the values of $num_i$, $size_i$, and $\Phi_i$ against $i$. Notice how the potential builds to pay for expanding the table. ### 17.4.2 Table expansion and contraction To implement a TABLE-DELETE operation, it is simple enough to remove the specified item from the table. In order to limit the amount of wasted space, however, we might wish to *contract* the table when the load factor becomes too small. Table contraction is analogous to table expansion: when the number of items in the table drops too low, we allocate a new, smaller table and then copy the items from the old table into the new one. We can then free the storage for the old table by returning it to the memory-management system. Ideally, we would like to preserve two properties: - the load factor of the dynamic table is bounded below by a positive constant, and - the amortized cost of a table operation is bounded above by a constant. We assume that we measure the cost in terms of elementary insertions and deletions. You might think that we should double the table size upon inserting an item into a full table and halve the size when deleting an item would cause the table to become less than half full. This strategy would guarantee that the load factor of the table never drops below $1/2$, but unfortunately, it can cause the amortized cost of an operation to be quite large. Consider the following scenario. We perform $n$ operations on a table $T$, where $n$ is an exact power of 2. The first $n/2$ operations are insertions, which by our previous analysis cost a total of $\Theta(n)$. At the end of this sequence of insertions, $T.num = T.size = n/2$. For the second $n/2$ operations, we perform the following sequence:

\text{insert, delete, delete, insert, insert, delete, delete, insert, insert, } \ldots .

The first insertion causes the table to expand to size $n$. The two following deletions cause the table to contract back to size $n/2$. Two further insertions cause another expansion, and so forth. The cost of each expansion and contraction is $\Theta(n)$, and there are $\Theta(n)$ of them. Thus, the total cost of the $n$ operations is $\Theta(n^2)$, making the amortized cost of an operation $\Theta(n)$. The downside of this strategy is obvious: after expanding the table, we do not delete enough items to pay for a contraction. Likewise, after contracting the table, we do not insert enough items to pay for an expansion. We can improve upon this strategy by allowing the load factor of the table to drop below $1/2$. Specifically, we continue to double the table size upon inserting an item into a full table, but we halve the table size when deleting an item causes the table to become less than $1/4$ full, rather than $1/2$ full as before. The load factor of the table is therefore bounded below by the constant $1/4$. Intuitively, we would consider a load factor of $1/2$ to be ideal, and the table's potential would then be 0. As the load factor deviates from $1/2$, the potential increases so that by the time we expand or contract the table, the table has garnered sufficient potential to pay for copying all the items into the newly allocated table. Thus, we will need a potential function that has grown to $T.num$ by the time that the load factor has either increased to 1 or decreased to $1/4$. After either expanding or contracting the table, the load factor goes back to $1/2$ and the table's potential reduces back to 0. We omit the code for TABLE-DELETE, since it is analogous to TABLE-INSERT. For our analysis, we shall assume that whenever the number of items in the table drops to 0, we free the storage for the table. That is, if $T.num = 0$, then $T.size = 0$. We can now use the potential method to analyze the cost of a sequence of $n$ TABLE-INSERT and TABLE-DELETE operations. We start by defining a potential function $\Phi$ that is 0 immediately after an expansion or contraction and builds as the load factor increases to 1 or decreases to $1/4$. Let us denote the load factor of a nonempty table $T$ by $\alpha(T) = T.num / T.size$. Since for an empty table, $T.num = T.size = 0$ and $\alpha(T) = 1$, we always have $T.num = \alpha(T) \cdot T.size$, whether the table is empty or not. We shall use as our potential function

\Phi(T) = \begin{cases} 2 \cdot T.num - T.size & \text{if } \alpha(T) \ge 1/2,\ T.size/2 - T.num & \text{if } \alpha(T) < 1/2. \end{cases} \tag{17.6}

Observe that the potential of an empty table is 0 and that the potential is never negative. Thus, the total amortized cost of a sequence of operations with respect to $\Phi$ provides an upper bound on the actual cost of the sequence. *Figure 17.4: The effect of a sequence of $n$ TABLE-INSERT and TABLE-DELETE operations on the number $num_i$ of items in the table, the number $size_i$ of slots in the table, and the potential $\Phi_i = 2 \cdot num_i - size_i$ if $\alpha_i \ge 1/2$, and $\Phi_i = size_i/2 - num_i$ if $\alpha_i < 1/2$, each measured after the $i$th operation. The thin line shows $num_i$, the dashed line shows $size_i$, and the thick line shows $\Phi_i$. Notice that immediately before an expansion, the potential has built up to the number of items in the table, and therefore it can pay for moving all the items to the new table. Likewise, immediately before a contraction, the potential has built up to the number of items in the table.* (Plot, horizontal axis $i$ from 0 to 48, vertical axis 0 to 32; not reproduced.) Before proceeding with a precise analysis, we pause to observe some properties of the potential function, as illustrated in Figure 17.4. - When the load factor is $1/2$, the potential is 0. - When the load factor is 1, we have $T.size = T.num$, which implies $\Phi(T) = T.num$, and thus the potential can pay for an expansion if an item is inserted. - When the load factor is $1/4$, we have $T.size = 4 \cdot T.num$, which implies $\Phi(T) = T.size/2 - T.num = 2\,T.num - T.num = T.num$, and thus the potential can pay for a contraction if an item is deleted. To analyze a sequence of $n$ TABLE-INSERT and TABLE-DELETE operations, we let $c_i$ denote the actual cost of the $i$th operation, $\hat{c}_i$ denote its amortized cost with respect to $\Phi$, $num_i$ denote the number of items stored in the table after the $i$th operation, $size_i$ denote the total size of the table after the $i$th operation, $\alpha_i$ denote the load factor of the table after the $i$th operation, and $\Phi_i$ denote the potential after the $i$th operation. Initially, $num_0 = 0$, $size_0 = 0$, $\alpha_0 = 1$, and $\Phi_0 = 0$. > **Proof (amortized cost of every operation is bounded by a constant).** > > **The $i$th operation is TABLE-INSERT.** Here $num_i = num_{i-1} + 1$. > > *Case I1: $\alpha_{i-1} \ge 1/2$.* The analysis is identical to that for table expansion in Section 17.4.1. Whether the table expands or not, the amortized cost $\hat{c}_i$ of the operation is at most 3. > > If $\alpha_{i-1} < 1/2$, the table cannot expand as a result of the operation, since the table expands only when $\alpha_{i-1} = 1$. Then $size_i = size_{i-1}$ and $c_i = 1$. > > *Case I2: $\alpha_{i-1} < 1/2$ and $\alpha_i < 1/2$.* Both potentials use the second branch of (17.6): > > $$ > \begin{aligned} > \hat{c}_i &= c_i + \Phi_i - \Phi_{i-1}\\ > &= 1 + (size_i/2 - num_i) - (size_{i-1}/2 - num_{i-1})\\ > &= 1 + (size_i/2 - num_i) - (size_i/2 - (num_i - 1)) && \text{(since } size_{i-1} = size_i,\ num_{i-1} = num_i - 1)\\ > &= 1 - 1\\ > &= 0 . > \end{aligned} > $$ > > *Case I3: $\alpha_{i-1} < 1/2$ but $\alpha_i \ge 1/2$.* Now $\Phi_i$ uses the first branch and $\Phi_{i-1}$ the second: > > $$ > \begin{aligned} > \hat{c}_i &= 1 + (2 \cdot num_i - size_i) - (size_{i-1}/2 - num_{i-1})\\ > &= 1 + (2(num_{i-1} + 1) - size_{i-1}) - (size_{i-1}/2 - num_{i-1}) && \text{(substitute } num_i = num_{i-1} + 1,\ size_i = size_{i-1})\\ > &= 3 + 3 \cdot num_{i-1} - \tfrac{3}{2}\, size_{i-1} && \text{(collect terms)}\\ > &< 3 + \tfrac{3}{2}\, size_{i-1} - \tfrac{3}{2}\, size_{i-1} && \text{(since } \alpha_{i-1} < 1/2 \text{ means } num_{i-1} < size_{i-1}/2)\\ > &= 3 . > \end{aligned} > $$ > > Thus, the amortized cost of a TABLE-INSERT operation is at most 3. > > **The $i$th operation is TABLE-DELETE.** Here $num_i = num_{i-1} - 1$. If $\alpha_{i-1} < 1/2$, we must consider whether the operation causes the table to contract. > > *Case D1: $\alpha_{i-1} < 1/2$ and no contraction.* Then $size_i = size_{i-1}$ and $c_i = 1$, and the amortized cost is > > $$ > \begin{aligned} > \hat{c}_i &= 1 + (size_i/2 - num_i) - (size_{i-1}/2 - num_{i-1})\\ > &= 1 + (size_i/2 - num_i) - (size_i/2 - (num_i + 1)) && \text{(since } num_{i-1} = num_i + 1)\\ > &= 1 + 1\\ > &= 2 . > \end{aligned} > $$ > > *Case D2: $\alpha_{i-1} < 1/2$ and the operation triggers a contraction.* The actual cost is $c_i = num_i + 1$, since we delete one item and move $num_i$ items. The table contracts when it drops below $1/4$ full, so $size_{i-1}/4 = num_{i-1} = num_i + 1$, and after halving, $size_i/2 = size_{i-1}/4 = num_i + 1$. Then $size_{i-1}/2 = 2\,num_i + 2$, and > > $$ > \begin{aligned} > \hat{c}_i &= c_i + \Phi_i - \Phi_{i-1}\\ > &= (num_i + 1) + (size_i/2 - num_i) - (size_{i-1}/2 - num_{i-1})\\ > &= (num_i + 1) + ((num_i + 1) - num_i) - ((2\,num_i + 2) - (num_i + 1)) && \text{(substitute the identities above)}\\ > &= (num_i + 1) + 1 - (num_i + 1)\\ > &= 1 . > \end{aligned} > $$ > > *Case D3: $\alpha_{i-1} \ge 1/2$.* The amortized cost is also bounded above by a constant. We leave the analysis as Exercise 17.4-2. > > In summary, since the amortized cost of each operation is bounded above by a constant, the actual time for any sequence of $n$ operations on a dynamic table is $O(n)$. ### Exercises 17.4-1 to 17.4-3 **17.4-1** Suppose that we wish to implement a dynamic, open-address hash table. Why might we consider the table to be full when its load factor reaches some value $\alpha$ that is strictly less than 1? Describe briefly how to make insertion into a dynamic, open-address hash table run in such a way that the expected value of the amortized cost per insertion is $O(1)$. Why is the expected value of the actual cost per insertion not necessarily $O(1)$ for all insertions? **17.4-2** Show that if $\alpha_{i-1} \ge 1/2$ and the $i$th operation on a dynamic table is TABLE-DELETE, then the amortized cost of the operation with respect to the potential function (17.6) is bounded above by a constant. **17.4-3** Suppose that instead of contracting a table by halving its size when its load factor drops below $1/4$, we contract it by multiplying its size by $2/3$ when its load factor drops below $1/3$. Using the potential function

\Phi(T) = \left| 2 \cdot T.num - T.size \right| ,

show that the amortized cost of a TABLE-DELETE that uses this strategy is bounded above by a constant. --- ## Problems **17-1 Bit-reversed binary counter** Chapter 30 examines an important algorithm called the fast Fourier transform, or FFT. The first step of the FFT algorithm performs a *bit-reversal permutation* on an input array $A[0..n-1]$ whose length is $n = 2^k$ for some nonnegative integer $k$. This permutation swaps elements whose indices have binary representations that are the reverse of each other. We can express each index $a$ as a $k$-bit sequence $\langle a_{k-1}, a_{k-2}, \ldots, a_0 \rangle$, where $a = \sum_{i=0}^{k-1} a_i 2^i$. We define

\text{rev}k(\langle a{k-1}, a_{k-2}, \ldots, a_0 \rangle) = \langle a_0, a_1, \ldots, a_{k-1} \rangle ;

t h u s,

\text{rev}k(a) = \sum{i=0}^{k-1} a_{k-i-1} 2^i .

For example, if $n = 16$ (or, equivalently, $k = 4$), then $\text{rev}_k(3) = 12$, since the 4-bit representation of 3 is $0011$, which when reversed gives $1100$, the 4-bit representation of 12. **a.** Given a function $\text{rev}_k$ that runs in $\Theta(k)$ time, write an algorithm to perform the bit-reversal permutation on an array of length $n = 2^k$ in $O(nk)$ time. We can use an algorithm based on an amortized analysis to improve the running time of the bit-reversal permutation. We maintain a "bit-reversed counter" and a procedure BIT-REVERSED-INCREMENT that, when given a bit-reversed-counter value $a$, produces $\text{rev}_k(\text{rev}_k(a) + 1)$. If $k = 4$, for example, and the bit-reversed counter starts at 0, then successive calls to BIT-REVERSED-INCREMENT produce the sequence

0000, 1000, 0100, 1100, 0010, 1010, \ldots = 0, 8, 4, 12, 2, 10, \ldots .

**b.** Assume that the words in your computer store $k$-bit values and that in unit time, your computer can manipulate the binary values with operations such as shifting left or right by arbitrary amounts, bitwise-AND, bitwise-OR, etc. Describe an implementation of the BIT-REVERSED-INCREMENT procedure that allows the bit-reversal permutation on an $n$-element array to be performed in a total of $O(n)$ time. **c.** Suppose that you can shift a word left or right by only one bit in unit time. Is it still possible to implement an $O(n)$-time bit-reversal permutation? **17-2 Making binary search dynamic** Binary search of a sorted array takes logarithmic search time, but the time to insert a new element is linear in the size of the array. We can improve the time for insertion by keeping several sorted arrays. Specifically, suppose that we wish to support SEARCH and INSERT on a set of $n$ elements. Let $k = \lceil \lg(n+1) \rceil$, and let the binary representation of $n$ be $\langle n_{k-1}, n_{k-2}, \ldots, n_0 \rangle$. We have $k$ sorted arrays $A_0, A_1, \ldots, A_{k-1}$, where for $i = 0, 1, \ldots, k-1$, the length of array $A_i$ is $2^i$. Each array is either full or empty, depending on whether $n_i = 1$ or $n_i = 0$, respectively. The total number of elements held in all $k$ arrays is therefore $\sum_{i=0}^{k-1} n_i 2^i = n$. Although each individual array is sorted, elements in different arrays bear no particular relationship to each other. **a.** Describe how to perform the SEARCH operation for this data structure. Analyze its worst-case running time. **b.** Describe how to perform the INSERT operation. Analyze its worst-case and amortized running times. **c.** Discuss how to implement DELETE. **17-3 Amortized weight-balanced trees** Consider an ordinary binary search tree augmented by adding to each node $x$ the attribute $x.size$ giving the number of keys stored in the subtree rooted at $x$. Let $\alpha$ be a constant in the range $1/2 \le \alpha < 1$. We say that a given node $x$ is *$\alpha$-balanced* if $x.left.size \le \alpha \cdot x.size$ and $x.right.size \le \alpha \cdot x.size$. The tree as a whole is *$\alpha$-balanced* if every node in the tree is $\alpha$-balanced. The following amortized approach to maintaining weight-balanced trees was suggested by G. Varghese. **a.** A $1/2$-balanced tree is, in a sense, as balanced as it can be. Given a node $x$ in an arbitrary binary search tree, show how to rebuild the subtree rooted at $x$ so that it becomes $1/2$-balanced. Your algorithm should run in time $\Theta(x.size)$, and it can use $O(x.size)$ auxiliary storage. **b.** Show that performing a search in an $n$-node $\alpha$-balanced binary search tree takes $O(\lg n)$ worst-case time. For the remainder of this problem, assume that the constant $\alpha$ is strictly greater than $1/2$. Suppose that we implement INSERT and DELETE as usual for an $n$-node binary search tree, except that after every such operation, if any node in the tree is no longer $\alpha$-balanced, then we "rebuild" the subtree rooted at the highest such node in the tree so that it becomes $1/2$-balanced. We shall analyze this rebuilding scheme using the potential method. For a node $x$ in a binary search tree $T$, we define

\Delta(x) = \left| x.left.size - x.right.size \right| ,

and we define the potential of $T$ as

\Phi(T) = c \sum_{x \in T : \Delta(x) \ge 2} \Delta(x) ,

where $c$ is a sufficiently large constant that depends on $\alpha$. **c.** Argue that any binary search tree has nonnegative potential and that a $1/2$-balanced tree has potential 0. **d.** Suppose that $m$ units of potential can pay for rebuilding an $m$-node subtree. How large must $c$ be in terms of $\alpha$ in order for it to take $O(1)$ amortized time to rebuild a subtree that is not $\alpha$-balanced? **e.** Show that inserting a node into or deleting a node from an $n$-node $\alpha$-balanced tree costs $O(\lg n)$ amortized time. **17-4 The cost of restructuring red-black trees** There are four basic operations on red-black trees that perform *structural modifications*: node insertions, node deletions, rotations, and color changes. We have seen that RB-INSERT and RB-DELETE use only $O(1)$ rotations, node insertions, and node deletions to maintain the red-black properties, but they may make many more color changes. **a.** Describe a legal red-black tree with $n$ nodes such that calling RB-INSERT to add the $(n+1)$st node causes $\Omega(\lg n)$ color changes. Then describe a legal red-black tree with $n$ nodes for which calling RB-DELETE on a particular node causes $\Omega(\lg n)$ color changes. Although the worst-case number of color changes per operation can be logarithmic, we shall prove that any sequence of $m$ RB-INSERT and RB-DELETE operations on an initially empty red-black tree causes $O(m)$ structural modifications in the worst case. Note that we count each color change as a structural modification. **b.** Some of the cases handled by the main loop of the code of both RB-INSERT-FIXUP and RB-DELETE-FIXUP are *terminating*: once encountered, they cause the loop to terminate after a constant number of additional operations. For each of the cases of RB-INSERT-FIXUP and RB-DELETE-FIXUP, specify which are terminating and which are not. (*Hint:* Look at Figures 13.5, 13.6, and 13.7.) We shall first analyze the structural modifications when only insertions are performed. Let $T$ be a red-black tree, and define $\Phi(T)$ to be the number of red nodes in $T$. Assume that 1 unit of potential can pay for the structural modifications performed by any of the three cases of RB-INSERT-FIXUP. **c.** Let $T'$ be the result of applying Case 1 of RB-INSERT-FIXUP to $T$. Argue that $\Phi(T') = \Phi(T) - 1$. **d.** When we insert a node into a red-black tree using RB-INSERT, we can break the operation into three parts. List the structural modifications and potential changes resulting from lines 1-16 of RB-INSERT, from nonterminating cases of RB-INSERT-FIXUP, and from terminating cases of RB-INSERT-FIXUP. **e.** Using part (d), argue that the amortized number of structural modifications performed by any call of RB-INSERT is $O(1)$. We now wish to prove that there are $O(m)$ structural modifications when there are both insertions and deletions. Let us define, for each node $x$,

w(x) = \begin{cases} 0 & \text{if } x \text{ is red},\ 1 & \text{if } x \text{ is black and has no red children},\ 0 & \text{if } x \text{ is black and has one red child},\ 2 & \text{if } x \text{ is black and has two red children}. \end{cases}

Now we redefine the potential of a red-black tree $T$ as

\Phi(T) = \sum_{x \in T} w(x) ,

and let $T'$ be the tree that results from applying any nonterminating case of RB-INSERT-FIXUP or RB-DELETE-FIXUP to $T$. **f.** Show that $\Phi(T') \le \Phi(T) - 1$ for all nonterminating cases of RB-INSERT-FIXUP. Argue that the amortized number of structural modifications performed by any call of RB-INSERT-FIXUP is $O(1)$. **g.** Show that $\Phi(T') \le \Phi(T) - 1$ for all nonterminating cases of RB-DELETE-FIXUP. Argue that the amortized number of structural modifications performed by any call of RB-DELETE-FIXUP is $O(1)$. **h.** Complete the proof that in the worst case, any sequence of $m$ RB-INSERT and RB-DELETE operations performs $O(m)$ structural modifications. **17-5 Competitive analysis of self-organizing lists with move-to-front** A *self-organizing list* is a linked list of $n$ elements, in which each element has a unique key. When we search for an element in the list, we are given a key, and we want to find an element with that key. A self-organizing list has two important properties: 1. To find an element in the list, given its key, we must traverse the list from the beginning until we encounter the element with the given key. If that element is the $k$th element from the start of the list, then the cost to find the element is $k$. 2. We may reorder the list elements after any operation, according to a given rule with a given cost. We may choose any heuristic we like to decide how to reorder the list. Assume that we start with a given list of $n$ elements, and we are given an access sequence $\sigma = \langle \sigma_1, \sigma_2, \ldots, \sigma_m \rangle$ of keys to find, in order. The cost of the sequence is the sum of the costs of the individual accesses in the sequence. Out of the various possible ways to reorder the list after an operation, this problem focuses on transposing adjacent list elements (switching their positions in the list) with a unit cost for each transpose operation. You will show, by means of a potential function, that a particular heuristic for reordering the list, move-to-front, entails a total cost no worse than 4 times that of any other heuristic for maintaining the list order, even if the other heuristic knows the access sequence in advance! We call this type of analysis a *competitive analysis*. For a heuristic $H$ and a given initial ordering of the list, denote the access cost of sequence $\sigma$ by $C_H(\sigma)$. Let $m$ be the number of accesses in $\sigma$. **a.** Argue that if heuristic $H$ does not know the access sequence in advance, then the worst-case cost for $H$ on an access sequence $\sigma$ is $C_H(\sigma) = \Omega(mn)$. With the *move-to-front* heuristic, immediately after searching for an element $x$, we move $x$ to the first position on the list (i.e., the front of the list). Let $\text{rank}_L(x)$ denote the rank of element $x$ in list $L$, that is, the position of $x$ in list $L$. For example, if $x$ is the fourth element in $L$, then $\text{rank}_L(x) = 4$. Let $c_i$ denote the cost of access $\sigma_i$ using the move-to-front heuristic, which includes the cost of finding the element in the list and the cost of moving it to the front of the list by a series of transpositions of adjacent list elements. **b.** Show that if $\sigma_i$ accesses element $x$ in list $L$ using the move-to-front heuristic, then $c_i = 2 \cdot \text{rank}_L(x) - 1$. Now we compare move-to-front with any other heuristic $H$ that processes an access sequence according to the two properties above. Heuristic $H$ may transpose elements in the list in any way it wants, and it might even know the entire access sequence in advance. Let $L_i$ be the list after access $\sigma_i$ using move-to-front, and let $L_i^*$ be the list after access $\sigma_i$ using heuristic $H$. We denote the cost of access $\sigma_i$ by $c_i$ for move-to-front and by $c_i^*$ for heuristic $H$. Suppose that heuristic $H$ performs $t_i^*$ transpositions during access $\sigma_i$. **c.** In part (b), you showed that $c_i = 2 \cdot \text{rank}_{L_{i-1}}(x) - 1$. Now show that $c_i^* = \text{rank}_{L_{i-1}^*}(x) + t_i^*$. We define an *inversion* in list $L_i$ as a pair of elements $y$ and $z$ such that $y$ precedes $z$ in $L_i$ and $z$ precedes $y$ in list $L_i^*$. Suppose that list $L_i$ has $q_i$ inversions after processing the access sequence $\langle \sigma_1, \sigma_2, \ldots, \sigma_i \rangle$. Then, we define a potential function $\Phi$ that maps $L_i$ to a real number by $\Phi(L_i) = 2q_i$. For example, if $L_i$ has the elements $\langle e, c, a, d, b \rangle$ and $L_i^*$ has the elements $\langle c, a, b, d, e \rangle$, then $L_i$ has 5 inversions $((e,c), (e,a), (e,d), (e,b), (d,b))$, and so $\Phi(L_i) = 10$. Observe that $\Phi(L_i) \ge 0$ for all $i$ and that, if move-to-front and heuristic $H$ start with the same list $L_0$, then $\Phi(L_0) = 0$. **d.** Argue that a transposition either increases the potential by 2 or decreases the potential by 2. Suppose that access $\sigma_i$ finds the element $x$. To understand how the potential changes due to $\sigma_i$, let us partition the elements other than $x$ into four sets, depending on where they are in the lists just before the $i$th access: - Set $A$ consists of elements that precede $x$ in both $L_{i-1}$ and $L_{i-1}^*$. - Set $B$ consists of elements that precede $x$ in $L_{i-1}$ and follow $x$ in $L_{i-1}^*$. - Set $C$ consists of elements that follow $x$ in $L_{i-1}$ and precede $x$ in $L_{i-1}^*$. - Set $D$ consists of elements that follow $x$ in both $L_{i-1}$ and $L_{i-1}^*$. **e.** Argue that $\text{rank}_{L_{i-1}}(x) = |A| + |B| + 1$ and $\text{rank}_{L_{i-1}^*}(x) = |A| + |C| + 1$. **f.** Show that access $\sigma_i$ causes a change in potential of

\Phi(L_i) - \Phi(L_{i-1}) \le 2\left( |A| - |B| + t_i^* \right) ,

\begin{aligned} \Phi(H’) - \Phi(H) &= \bigl( (t(H) + 1) + 2m(H) \bigr) - \bigl( t(H) + 2m(H) \bigr) \ &= 1 . \end{aligned}

Since the actual cost is $O(1)$, the amortized cost is $O(1) + 1 = O(1)$. ### Finding the minimum node The minimum node of a Fibonacci heap $H$ is given by the pointer $H.min$, so we can find the minimum node in $O(1)$ actual time. Because the potential of $H$ does not change, the amortized cost of this operation is equal to its $O(1)$ actual cost. ### Uniting two Fibonacci heaps The following procedure unites Fibonacci heaps $H_1$ and $H_2$, destroying $H_1$ and $H_2$ in the process. It simply concatenates the root lists of $H_1$ and $H_2$ and then determines the new minimum node. Afterward, the objects representing $H_1$ and $H_2$ will never be used again. ```text FIB-HEAP-UNION(H1, H2) 1 H = MAKE-FIB-HEAP() 2 H.min = H1.min 3 concatenate the root list of H2 with the root list of H 4 if (H1.min == NIL) or (H2.min \neq NIL and H2.min.key < H1.min.key) 5 H.min = H2.min 6 H.n = H1.n + H2.n 7 return H ``` Lines 1-3 concatenate the root lists of $H_1$ and $H_2$ into a new root list $H$. Lines 2, 4, and 5 set the minimum node of $H$, and line 6 sets $H.n$ to the total number of nodes. Line 7 returns the resulting Fibonacci heap $H$. As in the FIB-HEAP-INSERT procedure, all roots remain roots. The change in potential is

\begin{aligned} &\Phi(H) - \bigl( \Phi(H_1) + \Phi(H_2) \bigr) \ &\quad = \bigl( t(H) + 2m(H) \bigr) - \bigl( (t(H_1) + 2m(H_1)) + (t(H_2) + 2m(H_2)) \bigr) \ &\quad = 0, \end{aligned}

F_k = \begin{cases} 0 & \text{if } k = 0, \ 1 & \text{if } k = 1, \ F_{k-1} + F_{k-2} & \text{if } k \ge 2. \end{cases}

T(n) = 2T\bigl(\lfloor \sqrt{n} \rfloor\bigr) + \lg n \tag{20.1}

has the solution $T(n) = O(\lg n \lg\lg n)$. Let's consider a similar, but simpler, recurrence:

T(u) = T(\sqrt{u}) + O(1). \tag{20.2}

> **Solving recurrence (20.2).** If we use the same technique, changing variables, we can show that recurrence (20.2) has the solution $T(u) = O(\lg\lg u)$. Let $m = \lg u$, so that $u = 2^m$ and we have > $$ > T(2^m) = T(2^{m/2}) + O(1). > $$ > Now we rename $S(m) = T(2^m)$, giving the new recurrence > $$ > S(m) = S(m/2) + O(1). > $$ > Here $a = 1$, $b = 2$, so $m^{\log_b a} = m^0 = 1$ and $f(m) = O(1) = \Theta(m^{\log_b a})$. By case 2 of the master method, this recurrence has the solution $S(m) = O(\lg m)$. We change back from $S(m)$ to $T(u)$: > $$ > \begin{aligned} > T(u) &= T(2^m) \\ > &= S(m) \\ > &= O(\lg m) \\ > &= O(\lg\lg u). > \end{aligned} > $$ Recurrence (20.2) will guide our search for a data structure. We will design a recursive data structure that shrinks by a factor of $\sqrt{u}$ in each level of its recursion. When an operation traverses this data structure, it will spend a constant amount of time at each level before recursing to the level below. Recurrence (20.2) will then characterize the running time of the operation. Here is another way to think of how the term $\lg\lg u$ ends up in the solution to recurrence (20.2). As we look at the universe size in each level of the recursive data structure, we see the sequence $u, u^{1/2}, u^{1/4}, u^{1/8}, \ldots$. If we consider how many bits we need to store the universe size at each level, we need $\lg u$ at the top level, and each level needs half the bits of the previous level. In general, if we start with $b$ bits and halve the number of bits at each level, then after $\lg b$ levels, we get down to just one bit. Since $b = \lg u$, we see that after $\lg\lg u$ levels, we have a universe size of 2. Looking back at the data structure in Figure 20.2, a given value $x$ resides in cluster number $\lfloor x/\sqrt{u} \rfloor$. If we view $x$ as a $\lg u$-bit binary integer, that cluster number, $\lfloor x/\sqrt{u} \rfloor$, is given by the most significant $(\lg u)/2$ bits of $x$. Within its cluster, $x$ appears in position $x \bmod \sqrt{u}$, which is given by the least significant $(\lg u)/2$ bits of $x$. We will need to index in this way, and so let us define some functions that will help us do so:

\begin{aligned} \operatorname{high}(x) &= \lfloor x/\sqrt{u} \rfloor, \ \operatorname{low}(x) &= x \bmod \sqrt{u}, \ \operatorname{index}(x, y) &= x\sqrt{u} + y. \end{aligned}

T(u) = 2T(\sqrt{u}) + O(1). \tag{20.3}

> **Solving recurrence (20.3).** Again, we use a change of variables to solve this recurrence, letting $m = \lg u$, which gives > $$ > T(2^m) = 2T(2^{m/2}) + O(1). > $$ > Renaming $S(m) = T(2^m)$ gives > $$ > S(m) = 2S(m/2) + O(1), > $$ > where $a = 2$, $b = 2$, $m^{\log_b a} = m$, and $f(m) = O(1) = O(m^{1-\epsilon})$ for, say, $\epsilon = 1$. So by case 1 of the master method, the solution is $S(m) = \Theta(m)$. By changing back from $S(m)$ to $T(u)$, we have > $$ > \begin{aligned} > T(u) &= T(2^m) \\ > &= S(m) \\ > &= \Theta(m) \\ > &= \Theta(\lg u). > \end{aligned} > $$ > Thus, we see that because of the second recursive call, PROTO-VEB-MINIMUM runs in $\Theta(\lg u)$ time rather than the desired $O(\lg\lg u)$ time. #### Finding the successor The SUCCESSOR operation is even worse. In the worst case, it makes two recursive calls, along with a call to PROTO-VEB-MINIMUM. The procedure PROTO-VEB-SUCCESSOR$(V, x)$ returns the smallest element in the proto-vEB structure $V$ that is greater than $x$, or NIL if no element in $V$ is greater than $x$. It does not require $x$ to be a member of the set, but it does assume that $0 \le x < V.u$. ```text PROTO-VEB-SUCCESSOR(V, x) 1 if V.u == 2 2 if x == 0 and V.A[1] == 1 3 return 1 4 else return NIL 5 else offset = PROTO-VEB-SUCCESSOR(V.cluster[high(x)], low(x)) 6 if offset != NIL 7 return index(high(x), offset) 8 else succ-cluster = PROTO-VEB-SUCCESSOR(V.summary, high(x)) 9 if succ-cluster == NIL 10 return NIL 11 else offset = PROTO-VEB-MINIMUM(V.cluster[succ-cluster]) 12 return index(succ-cluster, offset) ``` The PROTO-VEB-SUCCESSOR procedure works as follows. As usual, line 1 tests for the base case, which lines 2-4 handle by brute force: the only way that $x$ can have a successor within a proto-vEB(2) structure is when $x = 0$ and $A[1]$ is 1. Lines 5-12 handle the recursive case. Line 5 searches for a successor to $x$ within $x$'s cluster, assigning the result to $offset$. Line 6 determines whether $x$ has a successor within its cluster; if it does, then line 7 computes and returns the value of this successor. Otherwise, we have to search in other clusters. Line 8 assigns to $succ\text{-}cluster$ the number of the next nonempty cluster, using the summary information to find it. Line 9 tests whether $succ\text{-}cluster$ is NIL, with line 10 returning NIL if all succeeding clusters are empty. If $succ\text{-}cluster$ is non-NIL, line 11 assigns the first element within that cluster to $offset$, and line 12 computes and returns the minimum element in that cluster. > **Running time.** In the worst case, PROTO-VEB-SUCCESSOR calls itself recursively twice on proto-vEB$(\sqrt{u})$ structures, and it makes one call to PROTO-VEB-MINIMUM on a proto-vEB$(\sqrt{u})$ structure, which costs $\Theta(\lg\sqrt{u})$. Thus, the recurrence for the worst-case running time $T(u)$ of PROTO-VEB-SUCCESSOR is > $$ > \begin{aligned} > T(u) &= 2T(\sqrt{u}) + \Theta(\lg\sqrt{u}) \\ > &= 2T(\sqrt{u}) + \Theta(\lg u). > \end{aligned} > $$ > We can employ the same technique that we used for recurrence (20.1) to show that this recurrence has the solution $T(u) = \Theta(\lg u \lg\lg u)$. Concretely, with $m = \lg u$ and $S(m) = T(2^m)$ we get $S(m) = 2S(m/2) + \Theta(m)$, which is case 2 of the master method (since $m^{\log_2 2} = m$), giving $S(m) = \Theta(m \lg m)$, i.e., $T(u) = \Theta(\lg u \lg\lg u)$. Thus, PROTO-VEB-SUCCESSOR is asymptotically slower than PROTO-VEB-MINIMUM. #### Inserting an element To insert an element, we need to insert it into the appropriate cluster and also set the summary bit for that cluster to 1. The procedure PROTO-VEB-INSERT$(V, x)$ inserts the value $x$ into the proto-vEB structure $V$. ```text PROTO-VEB-INSERT(V, x) 1 if V.u == 2 2 V.A[x] = 1 3 else PROTO-VEB-INSERT(V.cluster[high(x)], low(x)) 4 PROTO-VEB-INSERT(V.summary, high(x)) ``` In the base case, line 2 sets the appropriate bit in the array $A$ to 1. In the recursive case, the recursive call in line 3 inserts $x$ into the appropriate cluster, and line 4 sets the summary bit for that cluster to 1. Because PROTO-VEB-INSERT makes two recursive calls in the worst case, recurrence (20.3) characterizes its running time. Hence, PROTO-VEB-INSERT runs in $\Theta(\lg u)$ time. #### Deleting an element The DELETE operation is more complicated than insertion. Whereas we can always set a summary bit to 1 when inserting, we cannot always reset the same summary bit to 0 when deleting. We need to determine whether any bit in the appropriate cluster is 1. As we have defined proto-vEB structures, we would have to examine all $\sqrt{u}$ bits within a cluster to determine whether any of them are 1. Alternatively, we could add an attribute $n$ to the proto-vEB structure, counting how many elements it has. We leave implementation of PROTO-VEB-DELETE as Exercises 20.2-2 and 20.2-3. Clearly, we need to modify the proto-vEB structure to get each operation down to making at most one recursive call. We will see in the next section how to do so. ### Exercises 20.2-1 Write pseudocode for the procedures PROTO-VEB-MAXIMUM and PROTO-VEB-PREDECESSOR. ### Exercises 20.2-2 Write pseudocode for PROTO-VEB-DELETE. It should update the appropriate summary bit by scanning the related bits within the cluster. What is the worst-case running time of your procedure? ### Exercises 20.2-3 Add the attribute $n$ to each proto-vEB structure, giving the number of elements currently in the set it represents, and write pseudocode for PROTO-VEB-DELETE that uses the attribute $n$ to decide when to reset summary bits to 0. What is the worst-case running time of your procedure? What other procedures need to change because of the new attribute? Do these changes affect their running times? ### Exercises 20.2-4 Modify the proto-vEB structure to support duplicate keys. ### Exercises 20.2-5 Modify the proto-vEB structure to support keys that have associated satellite data. ### Exercises 20.2-6 Write pseudocode for a procedure that creates a proto-vEB$(u)$ structure. ### Exercises 20.2-7 Argue that if line 9 of PROTO-VEB-MINIMUM is executed, then the proto-vEB structure is empty. ### Exercises 20.2-8 Suppose that we designed a proto-vEB structure in which each $cluster$ array had only $u^{1/4}$ elements. What would the running times of each operation be? ## 20.3 The van Emde Boas tree The proto-vEB structure of the previous section is close to what we need to achieve $O(\lg\lg u)$ running times. It falls short because we have to recurse too many times in most of the operations. In this section, we shall design a data structure that is similar to the proto-vEB structure but stores a little more information, thereby removing the need for some of the recursion. In Section 20.2, we observed that the assumption that we made about the universe size (that $u = 2^{2^k}$ for some integer $k$) is unduly restrictive, confining the possible values of $u$ to an overly sparse set. From this point on, therefore, we will allow the universe size $u$ to be any exact power of 2, and when $\sqrt{u}$ is not an integer (that is, if $u$ is an odd power of 2, $u = 2^{2k+1}$ for some integer $k \ge 0$) then we will divide the $\lg u$ bits of a number into the most significant $\lceil (\lg u)/2 \rceil$ bits and the least significant $\lfloor (\lg u)/2 \rfloor$ bits. For convenience, we denote $2^{\lceil (\lg u)/2 \rceil}$ (the "upper square root" of $u$) by $\uparrow\!\sqrt{u}$ and $2^{\lfloor (\lg u)/2 \rfloor}$ (the "lower square root" of $u$) by $\downarrow\!\sqrt{u}$, so that $u = \uparrow\!\sqrt{u} \cdot \downarrow\!\sqrt{u}$ and, when $u$ is an even power of 2 ($u = 2^{2k}$ for some integer $k$), $\uparrow\!\sqrt{u} = \downarrow\!\sqrt{u} = \sqrt{u}$. Because we now allow $u$ to be an odd power of 2, we must redefine our helpful functions from Section 20.2:

\begin{aligned} \operatorname{high}(x) &= \lfloor x/\downarrow!\sqrt{u} \rfloor, \ \operatorname{low}(x) &= x \bmod \downarrow!\sqrt{u}, \ \operatorname{index}(x, y) &= x,\downarrow!\sqrt{u} + y. \end{aligned}

### 20.3.1 van Emde Boas trees The **van Emde Boas tree**, or **vEB tree**, modifies the proto-vEB structure. We denote a vEB tree with a universe size of $u$ as vEB$(u)$ and, unless $u$ equals the base size of 2, the attribute $summary$ points to a vEB$(\uparrow\!\sqrt{u})$ tree and the array $cluster[0 \mathinner{.\,.} \uparrow\!\sqrt{u}-1]$ points to $\uparrow\!\sqrt{u}$ vEB$(\downarrow\!\sqrt{u})$ trees. As Figure 20.5 illustrates, a vEB tree contains two attributes not found in a proto-vEB structure: - $min$ stores the minimum element in the vEB tree, and - $max$ stores the maximum element in the vEB tree. *Figure 20.5: The information in a vEB$(u)$ tree when $u > 2$. The structure contains the universe size $u$, elements $min$ and $max$, a pointer $summary$ to a vEB$(\uparrow\!\sqrt{u})$ tree, and an array $cluster[0 \mathinner{.\,.} \uparrow\!\sqrt{u}-1]$ of $\uparrow\!\sqrt{u}$ pointers to vEB$(\downarrow\!\sqrt{u})$ trees.* Furthermore, the element stored in $min$ does not appear in any of the recursive vEB$(\downarrow\!\sqrt{u})$ trees that the $cluster$ array points to. The elements stored in a vEB$(u)$ tree $V$, therefore, are $V.min$ plus all the elements recursively stored in the vEB$(\downarrow\!\sqrt{u})$ trees pointed to by $V.cluster[0 \mathinner{.\,.} \uparrow\!\sqrt{u}-1]$. Note that when a vEB tree contains two or more elements, we treat $min$ and $max$ differently: the element stored in $min$ does not appear in any of the clusters, but unless the vEB tree contains just one element (so that the minimum and maximum elements are the same), the element stored in $max$ does. Since the base size is 2, a vEB(2) tree does not need the array $A$ that the corresponding proto-vEB(2) structure has. Instead, we can determine its elements from its $min$ and $max$ attributes. In a vEB tree with no elements, regardless of its universe size $u$, both $min$ and $max$ are NIL. *Figure 20.6: A vEB(16) tree corresponding to the proto-vEB tree in Figure 20.4. It stores the set $\{2,3,4,5,7,14,15\}$. Slashes indicate NIL values. The value stored in the $min$ attribute of a vEB tree does not appear in any of its clusters. Heavy shading serves the same purpose here as in Figure 20.4.* (The root has $min = 2$ and $max = 15$; the cluster 0 vEB(4) tree has $min = 3$; see the text.) Figure 20.6 shows a vEB(16) tree $V$ holding the set $\{2,3,4,5,7,14,15\}$. Because the smallest element is 2, $V.min$ equals 2, and even though $\operatorname{high}(2) = 0$, the element 2 does not appear in the vEB(4) tree pointed to by $V.cluster[0]$: notice that $V.cluster[0].min$ equals 3, and so 2 is not in this vEB tree. Similarly, since $V.cluster[0].min$ equals 3, and 2 and 3 are the only elements in $V.cluster[0]$, the vEB(2) clusters within $V.cluster[0]$ are empty. The $min$ and $max$ attributes will turn out to be key to reducing the number of recursive calls within the operations on vEB trees. These attributes will help us in four ways: 1. The MINIMUM and MAXIMUM operations do not even need to recurse, for they can just return the values of $min$ or $max$. 2. The SUCCESSOR operation can avoid making a recursive call to determine whether the successor of a value $x$ lies within $\operatorname{high}(x)$. That is because $x$'s successor lies within its cluster if and only if $x$ is strictly less than the $max$ attribute of its cluster. A symmetric argument holds for PREDECESSOR and $min$. 3. We can tell whether a vEB tree has no elements, exactly one element, or at least two elements in constant time from its $min$ and $max$ values. This ability will help in the INSERT and DELETE operations. If $min$ and $max$ are both NIL, then the vEB tree has no elements. If $min$ and $max$ are non-NIL but are equal to each other, then the vEB tree has exactly one element. Otherwise, both $min$ and $max$ are non-NIL but are unequal, and the vEB tree has two or more elements. 4. If we know that a vEB tree is empty, we can insert an element into it by updating only its $min$ and $max$ attributes. Hence, we can insert into an empty vEB tree in constant time. Similarly, if we know that a vEB tree has only one element, we can delete that element in constant time by updating only $min$ and $max$. These properties will allow us to cut short the chain of recursive calls. > **Think about it.** The $min$ trick is a bit like a hotel keeping its "first guest" in a special lobby room: the first arrival costs nothing to record, and it never needs to be announced to the floor directory. Why does this save a recursive call exactly when it matters? Even if the universe size $u$ is an odd power of 2, the difference in the sizes of the $summary$ vEB tree and the clusters will not turn out to affect the asymptotic running times of the vEB-tree operations. The recursive procedures that implement the vEB-tree operations will all have running times characterized by the recurrence

T(u) \le T(\uparrow!\sqrt{u}) + O(1). \tag{20.4}

P(u) = (\sqrt{u} + 1),P(\sqrt{u}) + \Theta(\sqrt{u}). \tag{20.5}

a_{ij} = \begin{cases} 1 & \text{if } (i, j) \in E ,, \ 0 & \text{otherwise} ,. \end{cases}

Figures 22.1(c) and 22.2(c) are the adjacency matrices of the undirected and directed graphs in Figures 22.1(a) and 22.2(a), respectively. The adjacency matrix of a graph requires $\Theta(V^2)$ memory, independent of the number of edges in the graph. Observe the symmetry along the main diagonal of the adjacency matrix in Figure 22.1(c). Since in an undirected graph, $(u, v)$ and $(v, u)$ represent the same edge, the adjacency matrix $A$ of an undirected graph is its own transpose: $A = A^{\mathsf T}$. In some applications, it pays to store only the entries on and above the diagonal of the adjacency matrix, thereby cutting the memory needed to store the graph almost in half. Like the adjacency-list representation of a graph, an adjacency matrix can represent a weighted graph. For example, if $G = (V, E)$ is a weighted graph with edge-weight function $w$, we can simply store the weight $w(u, v)$ of the edge $(u, v) \in E$ as the entry in row $u$ and column $v$ of the adjacency matrix. If an edge does not exist, we can store a $\text{NIL}$ value as its corresponding matrix entry, though for many problems it is convenient to use a value such as $0$ or $\infty$. Although the adjacency-list representation is asymptotically at least as space-efficient as the adjacency-matrix representation, adjacency matrices are simpler, and so we may prefer them when graphs are reasonably small. Moreover, adjacency matrices carry a further advantage for unweighted graphs: they require only one bit per entry. > **Think about it.** An adjacency list is like a contact card for each person (it lists only the people they actually know); an adjacency matrix is like a giant attendance grid with a cell for every possible pair. Which one wastes less paper at a party where almost nobody knows anybody? ### Representing attributes Most algorithms that operate on graphs need to maintain attributes for vertices and/or edges. We indicate these attributes using our usual notation, such as $v.d$ for an attribute $d$ of a vertex $v$. When we indicate edges as pairs of vertices, we use the same style of notation. For example, if edges have an attribute $f$, then we denote this attribute for edge $(u, v)$ by $(u, v).f$. For the purpose of presenting and understanding algorithms, our attribute notation suffices. Implementing vertex and edge attributes in real programs can be another story entirely. There is no one best way to store and access vertex and edge attributes. For a given situation, your decision will likely depend on the programming language you are using, the algorithm you are implementing, and how the rest of your program uses the graph. If you represent a graph using adjacency lists, one design represents vertex attributes in additional arrays, such as an array $d[1 \ldots |V|]$ that parallels the $Adj$ array. If the vertices adjacent to $u$ are in $Adj[u]$, then what we call the attribute $u.d$ would actually be stored in the array entry $d[u]$. Many other ways of implementing attributes are possible. For example, in an object-oriented programming language, vertex attributes might be represented as instance variables within a subclass of a `Vertex` class. ### Exercises 22.1-1 to 22.1-8 **22.1-1** Given an adjacency-list representation of a directed graph, how long does it take to compute the out-degree of every vertex? How long does it take to compute the in-degrees? **22.1-2** Give an adjacency-list representation for a complete binary tree on 7 vertices. Give an equivalent adjacency-matrix representation. Assume that vertices are numbered from 1 to 7 as in a binary heap. **22.1-3** The **transpose** of a directed graph $G = (V, E)$ is the graph $G^{\mathsf T} = (V, E^{\mathsf T})$, where $E^{\mathsf T} = \{(v, u) \in V \times V : (u, v) \in E\}$. Thus, $G^{\mathsf T}$ is $G$ with all its edges reversed. Describe efficient algorithms for computing $G^{\mathsf T}$ from $G$, for both the adjacency-list and adjacency-matrix representations of $G$. Analyze the running times of your algorithms. **22.1-4** Given an adjacency-list representation of a multigraph $G = (V, E)$, describe an $O(V + E)$-time algorithm to compute the adjacency-list representation of the "equivalent" undirected graph $G' = (V, E')$, where $E'$ consists of the edges in $E$ with all multiple edges between two vertices replaced by a single edge and with all self-loops removed. **22.1-5** The **square** of a directed graph $G = (V, E)$ is the graph $G^2 = (V, E^2)$ such that $(u, v) \in E^2$ if and only if $G$ contains a path with at most two edges between $u$ and $v$. Describe efficient algorithms for computing $G^2$ from $G$ for both the adjacency-list and adjacency-matrix representations of $G$. Analyze the running times of your algorithms. **22.1-6** Most graph algorithms that take an adjacency-matrix representation as input require time $\Omega(V^2)$, but there are some exceptions. Show how to determine whether a directed graph $G$ contains a **universal sink** (a vertex with in-degree $|V| - 1$ and out-degree $0$) in time $O(V)$, given an adjacency matrix for $G$. **22.1-7** The **incidence matrix** of a directed graph $G = (V, E)$ with no self-loops is a $|V| \times |E|$ matrix $B = (b_{ij})$ such that

b_{ij} = \begin{cases} -1 & \text{if edge } j \text{ leaves vertex } i ,, \ 1 & \text{if edge } j \text{ enters vertex } i ,, \ 0 & \text{otherwise} ,. \end{cases}

Describe what the entries of the matrix product $BB^{\mathsf T}$ represent, where $B^{\mathsf T}$ is the transpose of $B$. **22.1-8** Suppose that instead of a linked list, each array entry $Adj[u]$ is a hash table containing the vertices $v$ for which $(u, v) \in E$. If all edge lookups are equally likely, what is the expected time to determine whether an edge is in the graph? What disadvantages does this scheme have? Suggest an alternate data structure for each edge list that solves these problems. Does your alternative have disadvantages compared to the hash table? ## 22.2 Breadth-first search Breadth-first search is one of the simplest algorithms for searching a graph and the archetype for many important graph algorithms. Prim's minimum-spanning-tree algorithm (Section 23.2) and Dijkstra's single-source shortest-paths algorithm (Section 24.3) use ideas similar to those in breadth-first search. Given a graph $G = (V, E)$ and a distinguished **source** vertex $s$, breadth-first search systematically explores the edges of $G$ to "discover" every vertex that is reachable from $s$. It computes the distance (smallest number of edges) from $s$ to each reachable vertex. It also produces a "breadth-first tree" with root $s$ that contains all reachable vertices. For any vertex $v$ reachable from $s$, the simple path in the breadth-first tree from $s$ to $v$ corresponds to a "shortest path" from $s$ to $v$ in $G$, that is, a path containing the smallest number of edges. The algorithm works on both directed and undirected graphs. Breadth-first search is so named because it expands the frontier between discovered and undiscovered vertices uniformly across the breadth of the frontier. That is, the algorithm discovers all vertices at distance $k$ from $s$ before discovering any vertices at distance $k + 1$. > **Think about it.** Drop a pebble into a still pond: the ripples spread out as rings, each ring one step farther than the last. BFS is exactly that ripple, and the "ring number" of a vertex is its distance from $s$. To keep track of progress, breadth-first search colors each vertex white, gray, or black. All vertices start out white and may later become gray and then black. A vertex is **discovered** the first time it is encountered during the search, at which time it becomes nonwhite. Gray and black vertices, therefore, have been discovered, but breadth-first search distinguishes between them to ensure that the search proceeds in a breadth-first manner.¹ If $(u, v) \in E$ and vertex $u$ is black, then vertex $v$ is either gray or black; that is, all vertices adjacent to black vertices have been discovered. Gray vertices may have some adjacent white vertices; they represent the frontier between discovered and undiscovered vertices. Breadth-first search constructs a **breadth-first tree**, initially containing only its root, which is the source vertex $s$. Whenever the search discovers a white vertex $v$ in the course of scanning the adjacency list of an already discovered vertex $u$, the vertex $v$ and the edge $(u, v)$ are added to the tree. We say that $u$ is the **predecessor** or **parent** of $v$ in the breadth-first tree. Since a vertex is discovered at most once, it has at most one parent. Ancestor and descendant relationships in the breadth-first tree are defined relative to the root $s$ as usual: if $u$ is on the simple path in the tree from the root $s$ to vertex $v$, then $u$ is an ancestor of $v$ and $v$ is a descendant of $u$. ¹ We distinguish between gray and black vertices to help us understand how breadth-first search operates. In fact, as Exercise 22.2-3 shows, we would get the same result even if we did not distinguish between gray and black vertices. The breadth-first-search procedure BFS below assumes that the input graph $G = (V, E)$ is represented using adjacency lists. It attaches several additional attributes to each vertex in the graph. We store the color of each vertex $u \in V$ in the attribute $u.color$ and the predecessor of $u$ in the attribute $u.\pi$. If $u$ has no predecessor (for example, if $u = s$ or $u$ has not been discovered), then $u.\pi = \text{NIL}$. The attribute $u.d$ holds the distance from the source $s$ to vertex $u$ computed by the algorithm. The algorithm also uses a first-in, first-out queue $Q$ (see Section 10.1) to manage the set of gray vertices. ```text BFS(G, s) 1 for each vertex u ∈ G.V − {s} 2 u.color = WHITE 3 u.d = ∞ 4 u.π = NIL 5 s.color = GRAY 6 s.d = 0 7 s.π = NIL 8 Q = ∅ 9 ENQUEUE(Q, s) 10 while Q ≠ ∅ 11 u = DEQUEUE(Q) 12 for each v ∈ G.Adj[u] 13 if v.color == WHITE 14 v.color = GRAY 15 v.d = u.d + 1 16 v.π = u 17 ENQUEUE(Q, v) 18 u.color = BLACK ``` *Figure 22.3: The operation of BFS on an undirected graph. Tree edges are shown shaded as they are produced by BFS. The value of $u.d$ appears within each vertex $u$. The queue $Q$ is shown at the beginning of each iteration of the while loop of lines 10-18. Vertex distances appear below vertices in the queue.* (Nine panels (a)-(i) show successive states of the search.) Figure 22.3 illustrates the progress of BFS on a sample graph. The procedure BFS works as follows. With the exception of the source vertex $s$, lines 1-4 paint every vertex white, set $u.d$ to be infinity for each vertex $u$, and set the parent of every vertex to be NIL. Line 5 paints $s$ gray, since we consider it to be discovered as the procedure begins. Line 6 initializes $s.d$ to $0$, and line 7 sets the predecessor of the source to be NIL. Lines 8-9 initialize $Q$ to the queue containing just the vertex $s$. The **while** loop of lines 10-18 iterates as long as there remain gray vertices, which are discovered vertices that have not yet had their adjacency lists fully examined. This while loop maintains the following invariant: > At the test in line 10, the queue $Q$ consists of the set of gray vertices. Although we won't use this loop invariant to prove correctness, it is easy to see that it holds prior to the first iteration and that each iteration of the loop maintains the invariant. Prior to the first iteration, the only gray vertex, and the only vertex in $Q$, is the source vertex $s$. Line 11 determines the gray vertex $u$ at the head of the queue $Q$ and removes it from $Q$. The **for** loop of lines 12-17 considers each vertex $v$ in the adjacency list of $u$. If $v$ is white, then it has not yet been discovered, and the procedure discovers it by executing lines 14-17. The procedure paints vertex $v$ gray, sets its distance $v.d$ to $u.d + 1$, records $u$ as its parent $v.\pi$, and places it at the tail of the queue $Q$. Once the procedure has examined all the vertices on $u$'s adjacency list, it blackens $u$ in line 18. The loop invariant is maintained because whenever a vertex is painted gray (in line 14) it is also enqueued (in line 17), and whenever a vertex is dequeued (in line 11) it is also painted black (in line 18). The results of breadth-first search may depend upon the order in which the neighbors of a given vertex are visited in line 12: the breadth-first tree may vary, but the distances $d$ computed by the algorithm will not. (See Exercise 22.2-5.) ### Analysis Before proving the various properties of breadth-first search, we take on the somewhat easier job of analyzing its running time on an input graph $G = (V, E)$. We use aggregate analysis, as we saw in Section 17.1. > **Analysis of BFS.** > > 1. After initialization, breadth-first search never whitens a vertex, and thus the test in line 13 ensures that each vertex is enqueued at most once, and hence dequeued at most once. > 2. Enqueuing and dequeuing take $O(1)$ time, so the total time devoted to queue operations is $O(V)$. > 3. The procedure scans the adjacency list of each vertex only when the vertex is dequeued, so it scans each adjacency list at most once. > 4. The sum of the lengths of all the adjacency lists is $\Theta(E)$, so the total time spent scanning adjacency lists is $O(E)$. > 5. The overhead for initialization is $O(V)$. > > Adding these contributions, > > $$ > O(V) + O(E) + O(V) = O(V + E). > $$ > > Thus, breadth-first search runs in time linear in the size of the adjacency-list representation of $G$. ### Shortest paths At the beginning of this section, we claimed that breadth-first search finds the distance to each reachable vertex in a graph $G = (V, E)$ from a given source vertex $s \in V$. Define the **shortest-path distance** $\delta(s, v)$ from $s$ to $v$ as the minimum number of edges in any path from vertex $s$ to vertex $v$; if there is no path from $s$ to $v$, then $\delta(s, v) = \infty$. We call a path of length $\delta(s, v)$ from $s$ to $v$ a **shortest path**² from $s$ to $v$. Before showing that breadth-first search correctly computes shortest-path distances, we investigate an important property of shortest-path distances. ² In Chapters 24 and 25, we shall generalize our study of shortest paths to weighted graphs, in which every edge has a real-valued weight and the weight of a path is the sum of the weights of its constituent edges. The graphs considered in the present chapter are unweighted or, equivalently, all edges have unit weight. **Lemma 22.1.** Let $G = (V, E)$ be a directed or undirected graph, and let $s \in V$ be an arbitrary vertex. Then, for any edge $(u, v) \in E$,

\delta(s, v) \le \delta(s, u) + 1 ,.

> **Proof.** Consider two cases. > > **Case 1: $u$ is reachable from $s$.** Then $v$ is also reachable from $s$ (follow a shortest path to $u$, then the edge $(u, v)$). The shortest path from $s$ to $v$ cannot be longer than the shortest path from $s$ to $u$ followed by the edge $(u, v)$. That walk has length $\delta(s, u) + 1$, so > > $$ > \delta(s, v) \le \delta(s, u) + 1 \,. > $$ > > **Case 2: $u$ is not reachable from $s$.** Then $\delta(s, u) = \infty$, so the right-hand side is $\infty$, and the inequality holds trivially. $\blacksquare$ We want to show that BFS properly computes $v.d = \delta(s, v)$ for each vertex $v \in V$. We first show that $v.d$ bounds $\delta(s, v)$ from above. **Lemma 22.2.** Let $G = (V, E)$ be a directed or undirected graph, and suppose that BFS is run on $G$ from a given source vertex $s \in V$. Then upon termination, for each vertex $v \in V$, the value $v.d$ computed by BFS satisfies $v.d \ge \delta(s, v)$. > **Proof.** We use induction on the number of ENQUEUE operations. Our inductive hypothesis is that $v.d \ge \delta(s, v)$ for all $v \in V$. > > **Basis.** Immediately after enqueuing $s$ in line 9 of BFS, the hypothesis holds, because > > $$ > s.d = 0 = \delta(s, s) \quad\text{and}\quad v.d = \infty \ge \delta(s, v) \ \text{ for all } v \in V - \{s\} \,. > $$ > > **Inductive step.** Consider a white vertex $v$ that is discovered during the search from a vertex $u$. The inductive hypothesis implies that $u.d \ge \delta(s, u)$. From the assignment performed by line 15 and from Lemma 22.1, we obtain > > $$ > \begin{aligned} > v.d &= u.d + 1 && \text{(line 15)} \\ > &\ge \delta(s, u) + 1 && \text{(inductive hypothesis)} \\ > &\ge \delta(s, v) && \text{(Lemma 22.1)} \,. > \end{aligned} > $$ > > Vertex $v$ is then enqueued, and it is never enqueued again because it is also grayed and the **then** clause of lines 14-17 is executed only for white vertices. Thus, the value of $v.d$ never changes again, and the inductive hypothesis is maintained. $\blacksquare$ To prove that $v.d = \delta(s, v)$, we must first show more precisely how the queue $Q$ operates during the course of BFS. The next lemma shows that at all times, the queue holds at most two distinct $d$ values. **Lemma 22.3.** Suppose that during the execution of BFS on a graph $G = (V, E)$, the queue $Q$ contains the vertices $\langle v_1, v_2, \ldots, v_r \rangle$, where $v_1$ is the head of $Q$ and $v_r$ is the tail. Then, $v_r.d \le v_1.d + 1$ and $v_i.d \le v_{i+1}.d$ for $i = 1, 2, \ldots, r - 1$. > **Proof.** The proof is by induction on the number of queue operations. Initially, when the queue contains only $s$, the lemma certainly holds. > > For the inductive step, we must prove that the lemma holds after both dequeuing and enqueuing a vertex. > > **Dequeuing.** If the head $v_1$ of the queue is dequeued, $v_2$ becomes the new head. (If the queue becomes empty, then the lemma holds vacuously.) By the inductive hypothesis, $v_1.d \le v_2.d$. But then > > $$ > \begin{aligned} > v_r.d &\le v_1.d + 1 && \text{(inductive hypothesis)} \\ > &\le v_2.d + 1 && \text{(since } v_1.d \le v_2.d) \,, > \end{aligned} > $$ > > and the remaining inequalities are unaffected. Thus, the lemma follows with $v_2$ as the head. > > **Enqueuing.** When we enqueue a vertex $v$ in line 17 of BFS, it becomes $v_{r+1}$. At that time, we have already removed vertex $u$, whose adjacency list is currently being scanned, from the queue $Q$, and by the inductive hypothesis, the new head $v_1$ has $v_1.d \ge u.d$. Thus, > > $$ > \begin{aligned} > v_{r+1}.d &= v.d \\ > &= u.d + 1 && \text{(line 15)} \\ > &\le v_1.d + 1 && \text{(since } u.d \le v_1.d) \,. > \end{aligned} > $$ > > From the inductive hypothesis, we also have $v_r.d \le u.d + 1$, and so > > $$ > v_r.d \le u.d + 1 = v.d = v_{r+1}.d \,, > $$ > > and the remaining inequalities are unaffected. Thus, the lemma follows when $v$ is enqueued. $\blacksquare$ The following corollary shows that the $d$ values at the time that vertices are enqueued are monotonically increasing over time. **Corollary 22.4.** Suppose that vertices $v_i$ and $v_j$ are enqueued during the execution of BFS, and that $v_i$ is enqueued before $v_j$. Then $v_i.d \le v_j.d$ at the time that $v_j$ is enqueued. > **Proof.** Immediate from Lemma 22.3 and the property that each vertex receives a finite $d$ value at most once during the course of BFS. $\blacksquare$ We can now prove that breadth-first search correctly finds shortest-path distances. **Theorem 22.5 (Correctness of breadth-first search).** Let $G = (V, E)$ be a directed or undirected graph, and suppose that BFS is run on $G$ from a given source vertex $s \in V$. Then, during its execution, BFS discovers every vertex $v \in V$ that is reachable from the source $s$, and upon termination, $v.d = \delta(s, v)$ for all $v \in V$. Moreover, for any vertex $v \ne s$ that is reachable from $s$, one of the shortest paths from $s$ to $v$ is a shortest path from $s$ to $v.\pi$ followed by the edge $(v.\pi, v)$. > **Proof.** Assume, for the purpose of contradiction, that some vertex receives a $d$ value not equal to its shortest-path distance. Let $v$ be the vertex with minimum $\delta(s, v)$ that receives such an incorrect $d$ value; clearly $v \ne s$. By Lemma 22.2, $v.d \ge \delta(s, v)$, and thus we have that $v.d > \delta(s, v)$. Vertex $v$ must be reachable from $s$, for if it is not, then $\delta(s, v) = \infty \ge v.d$. Let $u$ be the vertex immediately preceding $v$ on a shortest path from $s$ to $v$, so that $\delta(s, v) = \delta(s, u) + 1$. Because $\delta(s, u) < \delta(s, v)$, and because of how we chose $v$, we have $u.d = \delta(s, u)$. Putting these properties together, we have > > $$ > v.d > \delta(s, v) = \delta(s, u) + 1 = u.d + 1 \,. \tag{22.1} > $$ > > Now consider the time when BFS chooses to dequeue vertex $u$ from $Q$ in line 11. At this time, vertex $v$ is either white, gray, or black. We shall show that in each of these cases, we derive a contradiction to inequality (22.1). > > - **$v$ is white.** Then line 15 sets $v.d = u.d + 1$, contradicting inequality (22.1). > - **$v$ is black.** Then it was already removed from the queue and, by Corollary 22.4, we have $v.d \le u.d$, again contradicting inequality (22.1). > - **$v$ is gray.** Then it was painted gray upon dequeuing some vertex $w$, which was removed from $Q$ earlier than $u$ and for which $v.d = w.d + 1$. By Corollary 22.4, however, $w.d \le u.d$, and so we have > > $$ > v.d = w.d + 1 \le u.d + 1 \,, > $$ > > once again contradicting inequality (22.1). > > Thus we conclude that $v.d = \delta(s, v)$ for all $v \in V$. All vertices $v$ reachable from $s$ must be discovered, for otherwise they would have $\infty = v.d > \delta(s, v)$. To conclude the proof of the theorem, observe that if $v.\pi = u$, then $v.d = u.d + 1$. Thus, we can obtain a shortest path from $s$ to $v$ by taking a shortest path from $s$ to $v.\pi$ and then traversing the edge $(v.\pi, v)$. $\blacksquare$ ### Breadth-first trees The procedure BFS builds a breadth-first tree as it searches the graph, as Figure 22.3 illustrates. The tree corresponds to the $\pi$ attributes. More formally, for a graph $G = (V, E)$ with source $s$, we define the **predecessor subgraph** of $G$ as $G_\pi = (V_\pi, E_\pi)$, where

V_\pi = {v \in V : v.\pi \ne \text{NIL}} \cup {s}

an d

E_\pi = {(v.\pi, v) : v \in V_\pi - {s}} ,.

The predecessor subgraph $G_\pi$ is a **breadth-first tree** if $V_\pi$ consists of the vertices reachable from $s$ and, for all $v \in V_\pi$, the subgraph $G_\pi$ contains a unique simple path from $s$ to $v$ that is also a shortest path from $s$ to $v$ in $G$. A breadth-first tree is in fact a tree, since it is connected and $|E_\pi| = |V_\pi| - 1$ (see Theorem B.2). We call the edges in $E_\pi$ **tree edges**. The following lemma shows that the predecessor subgraph produced by the BFS procedure is a breadth-first tree. **Lemma 22.6.** When applied to a directed or undirected graph $G = (V, E)$, procedure BFS constructs $\pi$ so that the predecessor subgraph $G_\pi = (V_\pi, E_\pi)$ is a breadth-first tree. > **Proof.** Line 16 of BFS sets $v.\pi = u$ if and only if $(u, v) \in E$ and $\delta(s, v) < \infty$ (that is, if $v$ is reachable from $s$), and thus $V_\pi$ consists of the vertices in $V$ reachable from $s$. Since $G_\pi$ forms a tree, by Theorem B.2, it contains a unique simple path from $s$ to each vertex in $V_\pi$. By applying Theorem 22.5 inductively, we conclude that every such path is a shortest path in $G$. $\blacksquare$ The following procedure prints out the vertices on a shortest path from $s$ to $v$, assuming that BFS has already computed a breadth-first tree: ```text PRINT-PATH(G, s, v) 1 if v == s 2 print s 3 elseif v.π == NIL 4 print "no path from" s "to" v "exists" 5 else PRINT-PATH(G, s, v.π) 6 print v ``` This procedure runs in time linear in the number of vertices in the path printed, since each recursive call is for a path one vertex shorter. ### Exercises 22.2-1 to 22.2-9 **22.2-1** Show the $d$ and $\pi$ values that result from running breadth-first search on the directed graph of Figure 22.2(a), using vertex $3$ as the source. **22.2-2** Show the $d$ and $\pi$ values that result from running breadth-first search on the undirected graph of Figure 22.3, using vertex $u$ as the source. **22.2-3** Show that using a single bit to store each vertex color suffices by arguing that the BFS procedure would produce the same result if line 18 were removed. **22.2-4** What is the running time of BFS if we represent its input graph by an adjacency matrix and modify the algorithm to handle this form of input? **22.2-5** Argue that in a breadth-first search, the value $u.d$ assigned to a vertex $u$ is independent of the order in which the vertices appear in each adjacency list. Using Figure 22.3 as an example, show that the breadth-first tree computed by BFS can depend on the ordering within adjacency lists. **22.2-6** Give an example of a directed graph $G = (V, E)$, a source vertex $s \in V$, and a set of tree edges $E_\pi \subseteq E$ such that for each vertex $v \in V$, the unique simple path in the graph $(V, E_\pi)$ from $s$ to $v$ is a shortest path in $G$, yet the set of edges $E_\pi$ cannot be produced by running BFS on $G$, no matter how the vertices are ordered in each adjacency list. **22.2-7** There are two types of professional wrestlers: "babyfaces" ("good guys") and "heels" ("bad guys"). Between any pair of professional wrestlers, there may or may not be a rivalry. Suppose we have $n$ professional wrestlers and we have a list of $r$ pairs of wrestlers for which there are rivalries. Give an $O(n + r)$-time algorithm that determines whether it is possible to designate some of the wrestlers as babyfaces and the remainder as heels such that each rivalry is between a babyface and a heel. If it is possible to perform such a designation, your algorithm should produce it. **22.2-8\*** The **diameter** of a tree $T = (V, E)$ is defined as $\max_{u, v \in V} \delta(u, v)$, that is, the largest of all shortest-path distances in the tree. Give an efficient algorithm to compute the diameter of a tree, and analyze the running time of your algorithm. **22.2-9** Let $G = (V, E)$ be a connected, undirected graph. Give an $O(V + E)$-time algorithm to compute a path in $G$ that traverses each edge in $E$ exactly once in each direction. Describe how you can find your way out of a maze if you are given a large supply of pennies. ## 22.3 Depth-first search The strategy followed by depth-first search is, as its name implies, to search "deeper" in the graph whenever possible. Depth-first search explores edges out of the most recently discovered vertex $v$ that still has unexplored edges leaving it. Once all of $v$'s edges have been explored, the search "backtracks" to explore edges leaving the vertex from which $v$ was discovered. This process continues until we have discovered all the vertices that are reachable from the original source vertex. If any undiscovered vertices remain, then depth-first search selects one of them as a new source, and it repeats the search from that source. The algorithm repeats this entire process until it has discovered every vertex.³ > **Think about it.** BFS is a ripple in a pond; DFS is a explorer in a maze with a ball of string. She walks down a corridor as far as she can, and only when stuck does she wind the string back to the last junction with an untried door. ³ It may seem arbitrary that breadth-first search is limited to only one source whereas depth-first search may search from multiple sources. Although conceptually, breadth-first search could proceed from multiple sources and depth-first search could be limited to one source, our approach reflects how the results of these searches are typically used. Breadth-first search usually serves to find shortest-path distances (and the associated predecessor subgraph) from a given source. Depth-first search is often a subroutine in another algorithm, as we shall see later in this chapter. As in breadth-first search, whenever depth-first search discovers a vertex $v$ during a scan of the adjacency list of an already discovered vertex $u$, it records this event by setting $v$'s predecessor attribute $v.\pi$ to $u$. Unlike breadth-first search, whose predecessor subgraph forms a tree, the predecessor subgraph produced by a depth-first search may be composed of several trees, because the search may repeat from multiple sources. Therefore, we define the predecessor subgraph of a depth-first search slightly differently from that of a breadth-first search: we let $G_\pi = (V, E_\pi)$, where

E_\pi = {(v.\pi, v) : v \in V \text{ and } v.\pi \ne \text{NIL}} ,.

The predecessor subgraph of a depth-first search forms a **depth-first forest** comprising several **depth-first trees**. The edges in $E_\pi$ are **tree edges**. As in breadth-first search, depth-first search colors vertices during the search to indicate their state. Each vertex is initially white, is grayed when it is **discovered** in the search, and is blackened when it is **finished**, that is, when its adjacency list has been examined completely. This technique guarantees that each vertex ends up in exactly one depth-first tree, so that these trees are disjoint. Besides creating a depth-first forest, depth-first search also timestamps each vertex. Each vertex $v$ has two timestamps: the first timestamp $v.d$ records when $v$ is first discovered (and grayed), and the second timestamp $v.f$ records when the search finishes examining $v$'s adjacency list (and blackens $v$). These timestamps provide important information about the structure of the graph and are generally helpful in reasoning about the behavior of depth-first search. The procedure DFS below records when it discovers vertex $u$ in the attribute $u.d$ and when it finishes vertex $u$ in the attribute $u.f$. These timestamps are integers between $1$ and $2|V|$, since there is one discovery event and one finishing event for each of the $|V|$ vertices. For every vertex $u$,

u.d < u.f ,. \tag{22.2}

Vertex $u$ is WHITE before time $u.d$, GRAY between time $u.d$ and time $u.f$, and BLACK thereafter. The following pseudocode is the basic depth-first-search algorithm. The input graph $G$ may be undirected or directed. The variable $time$ is a global variable that we use for timestamping. ```text DFS(G) 1 for each vertex u ∈ G.V 2 u.color = WHITE 3 u.π = NIL 4 time = 0 5 for each vertex u ∈ G.V 6 if u.color == WHITE 7 DFS-VISIT(G, u) DFS-VISIT(G, u) 1 time = time + 1 // white vertex u has just been discovered 2 u.d = time 3 u.color = GRAY 4 for each v ∈ G.Adj[u] // explore edge (u, v) 5 if v.color == WHITE 6 v.π = u 7 DFS-VISIT(G, v) 8 u.color = BLACK // blacken u; it is finished 9 time = time + 1 10 u.f = time ``` *Figure 22.4: The progress of the depth-first-search algorithm DFS on a directed graph (the graph of Figure 22.2). As edges are explored by the algorithm, they are shown as either shaded (if they are tree edges) or dashed (otherwise). Nontree edges are labeled B, C, or F according to whether they are back, cross, or forward edges. Timestamps within vertices indicate discovery time/finishing time.* (Sixteen panels (a)-(p) show successive states of the search.) Figure 22.4 illustrates the progress of DFS on the graph shown in Figure 22.2. Procedure DFS works as follows. Lines 1-3 paint all vertices white and initialize their $\pi$ attributes to NIL. Line 4 resets the global time counter. Lines 5-7 check each vertex in $V$ in turn and, when a white vertex is found, visit it using DFS-VISIT. Every time DFS-VISIT$(G, u)$ is called in line 7, vertex $u$ becomes the root of a new tree in the depth-first forest. When DFS returns, every vertex $u$ has been assigned a **discovery time** $u.d$ and a **finishing time** $u.f$. In each call DFS-VISIT$(G, u)$, vertex $u$ is initially white. Line 1 increments the global variable $time$, line 2 records the new value of $time$ as the discovery time $u.d$, and line 3 paints $u$ gray. Lines 4-7 examine each vertex $v$ adjacent to $u$ and recursively visit $v$ if it is white. As each vertex $v \in Adj[u]$ is considered in line 4, we say that edge $(u, v)$ is **explored** by the depth-first search. Finally, after every edge leaving $u$ has been explored, lines 8-10 paint $u$ black, increment $time$, and record the finishing time in $u.f$. Note that the results of depth-first search may depend upon the order in which line 5 of DFS examines the vertices and upon the order in which line 4 of DFS-VISIT visits the neighbors of a vertex. These different visitation orders tend not to cause problems in practice, as we can usually use any depth-first search result effectively, with essentially equivalent results. ### Analysis What is the running time of DFS? > **Analysis of DFS.** > > 1. The loops on lines 1-3 and lines 5-7 of DFS take time $\Theta(V)$, exclusive of the time to execute the calls to DFS-VISIT. > 2. As we did for breadth-first search, we use aggregate analysis. DFS-VISIT is called exactly once for each vertex $v \in V$, since the vertex $u$ on which DFS-VISIT is invoked must be white and the first thing DFS-VISIT does is paint vertex $u$ gray. > 3. During an execution of DFS-VISIT$(G, v)$, the loop on lines 4-7 executes $|Adj[v]|$ times. > 4. Since > > $$ > \sum_{v \in V} |Adj[v]| = \Theta(E) \,, > $$ > > the total cost of executing lines 4-7 of DFS-VISIT over all calls is $O(E)$. > 5. The running time of DFS is therefore > > $$ > \Theta(V) + O(E) = \Theta(V + E) \,. > $$ ### Properties of depth-first search Depth-first search yields valuable information about the structure of a graph. Perhaps the most basic property of depth-first search is that the predecessor subgraph $G_\pi$ does indeed form a forest of trees, since the structure of the depth-first trees exactly mirrors the structure of recursive calls of DFS-VISIT. That is, $u = v.\pi$ if and only if DFS-VISIT$(G, v)$ was called during a search of $u$'s adjacency list. Additionally, vertex $v$ is a descendant of vertex $u$ in the depth-first forest if and only if $v$ is discovered during the time in which $u$ is gray. Another important property of depth-first search is that discovery and finishing times have **parenthesis structure**. If we represent the discovery of vertex $u$ with a left parenthesis "$(u$" and represent its finishing by a right parenthesis "$u)$", then the history of discoveries and finishings makes a well-formed expression in the sense that the parentheses are properly nested. For example, the depth-first search of Figure 22.5(a) corresponds to the parenthesization shown in Figure 22.5(b). The following theorem provides another way to characterize the parenthesis structure. **Theorem 22.7 (Parenthesis theorem).** In any depth-first search of a (directed or undirected) graph $G = (V, E)$, for any two vertices $u$ and $v$, exactly one of the following three conditions holds: - the intervals $[u.d, u.f]$ and $[v.d, v.f]$ are entirely disjoint, and neither $u$ nor $v$ is a descendant of the other in the depth-first forest, - the interval $[u.d, u.f]$ is contained entirely within the interval $[v.d, v.f]$, and $u$ is a descendant of $v$ in a depth-first tree, or - the interval $[v.d, v.f]$ is contained entirely within the interval $[u.d, u.f]$, and $v$ is a descendant of $u$ in a depth-first tree. *Figure 22.5: Properties of depth-first search. (a) The result of a depth-first search of a directed graph. Vertices are timestamped and edge types are indicated as in Figure 22.4. (b) Intervals for the discovery time and finishing time of each vertex correspond to the parenthesization shown. Each rectangle spans the interval given by the discovery and finishing times of the corresponding vertex. Only tree edges are shown. If two intervals overlap, then one is nested within the other, and the vertex corresponding to the smaller interval is a descendant of the vertex corresponding to the larger. (c) The graph of part (a) redrawn with all tree and forward edges going down within a depth-first tree and all back edges going up from a descendant to an ancestor.* (Timestamps range over $1, \ldots, 16$; the parenthesization in (b) is a properly nested string of 16 parentheses.) > **Proof.** By symmetry in the roles of $u$ and $v$, we may assume $u.d < v.d$ (the case $v.d < u.d$ is the same with $u$ and $v$ exchanged; $u.d \ne v.d$ because each time value is used once). Split into two subcases according to whether $v.d < u.f$ or not. > > **Subcase 1: $v.d < u.f$.** > > 1. Then $u.d < v.d < u.f$, so $v$ was discovered while $u$ was still gray. > 2. Hence $v$ is a descendant of $u$. > 3. Since $v$ was discovered more recently than $u$, all of its outgoing edges are explored, and $v$ is finished, before the search returns to and finishes $u$ (the call DFS-VISIT$(G,v)$ is nested inside DFS-VISIT$(G,u)$). > 4. Therefore $v.f < u.f$, and > > $$ > u.d < v.d < v.f < u.f \,, > $$ > > so the interval $[v.d, v.f]$ is entirely contained within $[u.d, u.f]$. This is the third condition. > > **Subcase 2: $u.f < v.d$.** > > 1. By inequality (22.2), $v.d < v.f$. > 2. Combining with $u.d < u.f$ and $u.f < v.d$, > > $$ > u.d < u.f < v.d < v.f \,, > $$ > > so the intervals $[u.d, u.f]$ and $[v.d, v.f]$ are disjoint. > 3. Because the intervals are disjoint, neither vertex was discovered while the other was gray, and so neither vertex is a descendant of the other. This is the first condition. > > The case $v.d < u.d$ is similar, with the roles of $u$ and $v$ reversed in the above argument, and yields either the first condition or the second. Since in each case the conditions have mutually incompatible interval relationships, exactly one holds. $\blacksquare$ **Corollary 22.8 (Nesting of descendants' intervals).** Vertex $v$ is a proper descendant of vertex $u$ in the depth-first forest for a (directed or undirected) graph $G$ if and only if

u.d < v.d < v.f < u.f ,.

v.low = \min \left( {v.d} \cup {w.d : (u, w) \text{ is a back edge for some descendant } u \text{ of } v} \right) .

\begin{aligned} x_1 - x_2 &\le 1,\ x_1 - x_4 &\le -4,\ x_2 - x_3 &\le 2,\ x_2 - x_5 &\le 7,\ x_2 - x_6 &\le 5,\ x_3 - x_6 &\le 10,\ x_4 - x_2 &\le 2,\ x_5 - x_1 &\le -1,\ x_5 - x_4 &\le 3,\ x_6 - x_3 &\le -8. \end{aligned}

* * 24.4 - 2 * * F in d a f e a s ib l eso l u t i o n or d e t er min e t ha t n o f e a s ib l eso l u t i o n e x i s t s f or t h e f o l l o w in g sy s t e m o f d i f f er e n ceco n s t r ain t s :

\begin{aligned} x_1 - x_2 &\le 4,\ x_1 - x_5 &\le 5,\ x_2 - x_4 &\le -6,\ x_3 - x_2 &\le 1,\ x_4 - x_1 &\le 3,\ x_4 - x_3 &\le 5,\ x_4 - x_5 &\le 10,\ x_5 - x_3 &\le -4,\ x_5 - x_4 &\le -8. \end{aligned}

w_{ij} = \begin{cases} 0 & \text{if } i = j, \ \text{the weight of directed edge } (i, j) & \text{if } i \neq j \text{ and } (i, j) \in E, \ \infty & \text{if } i \neq j \text{ and } (i, j) \notin E. \end{cases} \tag{25.1}

We allow negative-weight edges, but we assume for the time being that the input graph contains no negative-weight cycles. The tabular output of the all-pairs shortest-paths algorithms presented in this chapter is an $n \times n$ matrix $D = (d_{ij})$, where entry $d_{ij}$ contains the weight of a shortest path from vertex $i$ to vertex $j$. That is, if we let $\delta(i, j)$ denote the shortest-path weight from vertex $i$ to vertex $j$ (as in Chapter 24), then $d_{ij} = \delta(i, j)$ at termination. To solve the all-pairs shortest-paths problem on an input adjacency matrix, we need to compute not only the shortest-path weights but also a **predecessor matrix** $\Pi = (\pi_{ij})$, where $\pi_{ij}$ is NIL if either $i = j$ or there is no path from $i$ to $j$, and otherwise $\pi_{ij}$ is the predecessor of $j$ on some shortest path from $i$. Just as the predecessor subgraph $G_\pi$ from Chapter 24 is a shortest-paths tree for a given source vertex, the subgraph induced by the $i$th row of the $\Pi$ matrix should be a shortest-paths tree with root $i$. For each vertex $i \in V$, we define the **predecessor subgraph** of $G$ for $i$ as $G_{\pi,i} = (V_{\pi,i}, E_{\pi,i})$, where $$V_{\pi,i} = \{ j \in V : \pi_{ij} \neq \text{NIL} \} \cup \{ i \}$$ and $$E_{\pi,i} = \{ (\pi_{ij}, j) : j \in V_{\pi,i} - \{ i \} \} .$$ If $G_{\pi,i}$ is a shortest-paths tree, then the following procedure, which is a modified version of the PRINT-PATH procedure from Chapter 22, prints a shortest path from vertex $i$ to vertex $j$. ```text PRINT-ALL-PAIRS-SHORTEST-PATH(Π, i, j) 1 if i == j 2 print i 3 elseif π_ij == NIL 4 print "no path from" i "to" j "exists" 5 else PRINT-ALL-PAIRS-SHORTEST-PATH(Π, i, π_ij) 6 print j ``` In order to highlight the essential features of the all-pairs algorithms in this chapter, we won't cover the creation and properties of predecessor matrices as extensively as we dealt with predecessor subgraphs in Chapter 24. Some of the exercises cover the basics. ### Chapter outline Section 25.1 presents a dynamic-programming algorithm based on matrix multiplication to solve the all-pairs shortest-paths problem. Using the technique of "repeated squaring," we can make this algorithm run in $\Theta(V^3 \lg V)$ time. Section 25.2 gives another dynamic-programming algorithm, the Floyd-Warshall algorithm, which runs in time $\Theta(V^3)$. Section 25.2 also covers the problem of finding the transitive closure of a directed graph, which is related to the all-pairs shortest-paths problem. Finally, Section 25.3 presents Johnson's algorithm, which solves the all-pairs shortest-paths problem in $O(V^2 \lg V + VE)$ time and is a good algorithm for large, sparse graphs. Before proceeding, we need to establish some conventions for adjacency-matrix representations. First, we shall generally assume that the input graph $G = (V, E)$ has $n$ vertices, so that $n = |V|$. Second, we shall use the convention of denoting matrices by uppercase letters, such as $W$, $L$, or $D$, and their individual elements by subscripted lowercase letters, such as $w_{ij}$, $l_{ij}$, or $d_{ij}$. Some matrices will have parenthesized superscripts, as in $L^{(m)} = (l_{ij}^{(m)})$ or $D^{(m)} = (d_{ij}^{(m)})$, to indicate iterates. Finally, for a given $n \times n$ matrix $A$, we shall assume that the value of $n$ is stored in the attribute $A.rows$. ## 25.1 Shortest paths and matrix multiplication This section presents a dynamic-programming algorithm for the all-pairs shortest-paths problem on a directed graph $G = (V, E)$. Each major loop of the dynamic program will invoke an operation that is very similar to matrix multiplication, so that the algorithm will look like repeated matrix multiplication. We shall start by developing a $\Theta(V^4)$-time algorithm for the all-pairs shortest-paths problem and then improve its running time to $\Theta(V^3 \lg V)$. Before proceeding, let us briefly recap the steps given in Chapter 15 for developing a dynamic-programming algorithm. 1. Characterize the structure of an optimal solution. 2. Recursively define the value of an optimal solution. 3. Compute the value of an optimal solution in a bottom-up fashion. We reserve the fourth step, constructing an optimal solution from computed information, for the exercises. ### The structure of a shortest path We start by characterizing the structure of an optimal solution. For the all-pairs shortest-paths problem on a graph $G = (V, E)$, we have proven (Lemma 24.1) that all subpaths of a shortest path are shortest paths. Suppose that we represent the graph by an adjacency matrix $W = (w_{ij})$. Consider a shortest path $p$ from vertex $i$ to vertex $j$, and suppose that $p$ contains at most $m$ edges. Assuming that there are no negative-weight cycles, $m$ is finite. If $i = j$, then $p$ has weight $0$ and no edges. If vertices $i$ and $j$ are distinct, then we decompose path $p$ into $i \overset{p'}{\leadsto} k \to j$, where path $p'$ now contains at most $m - 1$ edges. By Lemma 24.1, $p'$ is a shortest path from $i$ to $k$, and so $$\delta(i, j) = \delta(i, k) + w_{kj} .$$ ### A recursive solution to the all-pairs shortest-paths problem Now, let $l_{ij}^{(m)}$ be the minimum weight of any path from vertex $i$ to vertex $j$ that contains at most $m$ edges. When $m = 0$, there is a shortest path from $i$ to $j$ with no edges if and only if $i = j$. Thus,

l_{ij}^{(0)} = \begin{cases} 0 & \text{if } i = j, \ \infty & \text{if } i \neq j. \end{cases}

For $m \geq 1$, we compute $l_{ij}^{(m)}$ as the minimum of $l_{ij}^{(m-1)}$ (the weight of a shortest path from $i$ to $j$ consisting of at most $m - 1$ edges) and the minimum weight of any path from $i$ to $j$ consisting of at most $m$ edges, obtained by looking at all possible predecessors $k$ of $j$. Thus, we recursively define

\begin{aligned} l_{ij}^{(m)} &= \min \left( l_{ij}^{(m-1)}, \ \min_{1 \le k \le n} \left{ l_{ik}^{(m-1)} + w_{kj} \right} \right) \ &= \min_{1 \le k \le n} \left{ l_{ik}^{(m-1)} + w_{kj} \right} . \end{aligned} \tag{25.2}

The latter equality follows since $w_{jj} = 0$ for all $j$ (take $k = j$). What are the actual shortest-path weights $\delta(i, j)$? If the graph contains no negative-weight cycles, then for every pair of vertices $i$ and $j$ for which $\delta(i, j) < \infty$, there is a shortest path from $i$ to $j$ that is simple and thus contains at most $n - 1$ edges. A path from vertex $i$ to vertex $j$ with more than $n - 1$ edges cannot have lower weight than a shortest path from $i$ to $j$. The actual shortest-path weights are therefore given by $$\delta(i, j) = l_{ij}^{(n-1)} = l_{ij}^{(n)} = l_{ij}^{(n+1)} = \cdots . \tag{25.3}$$ ### Computing the shortest-path weights bottom up Taking as our input the matrix $W = (w_{ij})$, we now compute a series of matrices $L^{(1)}, L^{(2)}, \ldots, L^{(n-1)}$, where for $m = 1, 2, \ldots, n-1$, we have $L^{(m)} = (l_{ij}^{(m)})$. The final matrix $L^{(n-1)}$ contains the actual shortest-path weights. Observe that $l_{ij}^{(1)} = w_{ij}$ for all vertices $i, j \in V$, and so $L^{(1)} = W$. The heart of the algorithm is the following procedure, which, given matrices $L^{(m-1)}$ and $W$, returns the matrix $L^{(m)}$. That is, it extends the shortest paths computed so far by one more edge. ```text EXTEND-SHORTEST-PATHS(L, W) 1 n = L.rows 2 let L' = (l'_ij) be a new n × n matrix 3 for i = 1 to n 4 for j = 1 to n 5 l'_ij = ∞ 6 for k = 1 to n 7 l'_ij = min(l'_ij, l_ik + w_kj) 8 return L' ``` The procedure computes a matrix $L' = (l'_{ij})$, which it returns at the end. It does so by computing equation (25.2) for all $i$ and $j$, using $L$ for $L^{(m-1)}$ and $L'$ for $L^{(m)}$. (It is written without the superscripts to make its input and output matrices independent of $m$.) Its running time is $\Theta(n^3)$ due to the three nested **for** loops. Now we can see the relation to matrix multiplication. Suppose we wish to compute the matrix product $C = A \cdot B$ of two $n \times n$ matrices $A$ and $B$. Then, for $i, j = 1, 2, \ldots, n$, we compute $$c_{ij} = \sum_{k=1}^{n} a_{ik} \cdot b_{kj} . \tag{25.4}$$ Observe that if we make the substitutions

\begin{aligned} l^{(m-1)} &\to a , \ w &\to b , \ l^{(m)} &\to c , \ \min &\to + , \

&\to \cdot \end{aligned}

in equation (25.2), we obtain equation (25.4). Thus, if we make these changes to EXTEND-SHORTEST-PATHS and also replace $\infty$ (the identity for $\min$) by $0$ (the identity for $+$), we obtain the same $\Theta(n^3)$-time procedure for multiplying square matrices that we saw in Section 4.2: ```text SQUARE-MATRIX-MULTIPLY(A, B) 1 n = A.rows 2 let C be a new n × n matrix 3 for i = 1 to n 4 for j = 1 to n 5 c_ij = 0 6 for k = 1 to n 7 c_ij = c_ij + a_ik · b_kj 8 return C ``` > **Think about it.** Same loops, different "arithmetic": addition became $\min$, multiplication became $+$. Whenever you see a familiar algorithm, ask what *other* number system it would still work in. Returning to the all-pairs shortest-paths problem, we compute the shortest-path weights by extending shortest paths edge by edge. Letting $A \cdot B$ denote the matrix "product" returned by EXTEND-SHORTEST-PATHS$(A, B)$, we compute the sequence of $n - 1$ matrices

\begin{aligned} L^{(1)} &= L^{(0)} \cdot W = W , \ L^{(2)} &= L^{(1)} \cdot W = W^2 , \ L^{(3)} &= L^{(2)} \cdot W = W^3 , \ &\ \ \vdots \ L^{(n-1)} &= L^{(n-2)} \cdot W = W^{n-1} . \end{aligned}

As we argued above, the matrix $L^{(n-1)} = W^{n-1}$ contains the shortest-path weights. The following procedure computes this sequence in $\Theta(n^4)$ time. ```text SLOW-ALL-PAIRS-SHORTEST-PATHS(W) 1 n = W.rows 2 L^(1) = W 3 for m = 2 to n - 1 4 let L^(m) be a new n × n matrix 5 L^(m) = EXTEND-SHORTEST-PATHS(L^(m-1), W) 6 return L^(n-1) ``` Figure 25.1 shows a graph and the matrices $L^{(m)}$ computed by the procedure SLOW-ALL-PAIRS-SHORTEST-PATHS. *Figure 25.1: A directed graph (5 vertices) and the sequence of matrices $L^{(m)}$ computed by SLOW-ALL-PAIRS-SHORTEST-PATHS. You might want to verify that $L^{(5)}$, defined as $L^{(4)} \cdot W$, equals $L^{(4)}$, and thus $L^{(m)} = L^{(4)}$ for all $m \ge 4$.*

L^{(1)} = \begin{pmatrix} 0 & 3 & 8 & \infty & -4 \ \infty & 0 & \infty & 1 & 7 \ \infty & 4 & 0 & \infty & \infty \ 2 & \infty & -5 & 0 & \infty \ \infty & \infty & \infty & 6 & 0 \end{pmatrix} \qquad L^{(2)} = \begin{pmatrix} 0 & 3 & 8 & 2 & -4 \ 3 & 0 & -4 & 1 & 7 \ \infty & 4 & 0 & 5 & 11 \ 2 & -1 & -5 & 0 & -2 \ \infty & \infty & 1 & 6 & 0 \end{pmatrix}

L^{(3)} = \begin{pmatrix} 0 & 3 & -3 & 2 & -4 \ 3 & 0 & -4 & 1 & -1 \ 7 & 4 & 0 & 5 & 11 \ 2 & -1 & -5 & 0 & -2 \ 8 & 5 & 1 & 6 & 0 \end{pmatrix} \qquad L^{(4)} = \begin{pmatrix} 0 & 1 & -3 & 2 & -4 \ 3 & 0 & -4 & 1 & -1 \ 7 & 4 & 0 & 5 & 3 \ 2 & -1 & -5 & 0 & -2 \ 8 & 5 & 1 & 6 & 0 \end{pmatrix}

### Improving the running time Our goal, however, is not to compute *all* the $L^{(m)}$ matrices: we are interested only in matrix $L^{(n-1)}$. Recall that in the absence of negative-weight cycles, equation (25.3) implies $L^{(m)} = L^{(n-1)}$ for all integers $m \ge n - 1$. Just as traditional matrix multiplication is associative, so is matrix multiplication defined by the EXTEND-SHORTEST-PATHS procedure (see Exercise 25.1-4). Therefore, we can compute $L^{(n-1)}$ with only $\lceil \lg(n-1) \rceil$ matrix products by computing the sequence

\begin{aligned} L^{(1)} &= W , \ L^{(2)} &= W^2 = W \cdot W , \ L^{(4)} &= W^4 = W^2 \cdot W^2 , \ L^{(8)} &= W^8 = W^4 \cdot W^4 , \ &\ \ \vdots \ L^{(2^{\lceil \lg(n-1) \rceil})} &= W^{2^{\lceil \lg(n-1) \rceil}} = W^{2^{\lceil \lg(n-1) \rceil - 1}} \cdot W^{2^{\lceil \lg(n-1) \rceil - 1}} . \end{aligned}

Since $2^{\lceil \lg(n-1) \rceil} \ge n - 1$, the final product $L^{(2^{\lceil \lg(n-1) \rceil})}$ is equal to $L^{(n-1)}$. The following procedure computes the above sequence of matrices by using this technique of **repeated squaring**. *Figure 25.2: A weighted, directed graph for use in Exercises 25.1-1, 25.2-1, and 25.3-1.* ```text FASTER-ALL-PAIRS-SHORTEST-PATHS(W) 1 n = W.rows 2 L^(1) = W 3 m = 1 4 while m < n - 1 5 let L^(2m) be a new n × n matrix 6 L^(2m) = EXTEND-SHORTEST-PATHS(L^(m), L^(m)) 7 m = 2m 8 return L^(m) ``` In each iteration of the **while** loop of lines 4-7, we compute $L^{(2m)} = (L^{(m)})^2$, starting with $m = 1$. At the end of each iteration, we double the value of $m$. The final iteration computes $L^{(n-1)}$ by actually computing $L^{(2m)}$ for some $n - 1 \le 2m < 2n - 2$. By equation (25.3), $L^{(2m)} = L^{(n-1)}$. The next time the test in line 4 is performed, $m$ has been doubled, so now $m \ge n - 1$, the test fails, and the procedure returns the last matrix it computed. Because each of the $\lceil \lg(n-1) \rceil$ matrix products takes $\Theta(n^3)$ time, FASTER-ALL-PAIRS-SHORTEST-PATHS runs in $\Theta(n^3 \lg n)$ time. Observe that the code is tight, containing no elaborate data structures, and the constant hidden in the $\Theta$-notation is therefore small. ### Exercises 25.1-1 to 25.1-10 **25.1-1** Run SLOW-ALL-PAIRS-SHORTEST-PATHS on the weighted, directed graph of Figure 25.2, showing the matrices that result for each iteration of the loop. Then do the same for FASTER-ALL-PAIRS-SHORTEST-PATHS. **25.1-2** Why do we require that $w_{ii} = 0$ for all $1 \le i \le n$? **25.1-3** What does the matrix

L^{(0)} = \begin{pmatrix} 0 & \infty & \infty & \cdots & \infty \ \infty & 0 & \infty & \cdots & \infty \ \infty & \infty & 0 & \cdots & \infty \ \vdots & \vdots & \vdots & \ddots & \vdots \ \infty & \infty & \infty & \cdots & 0 \end{pmatrix}

used in the shortest-paths algorithms correspond to in regular matrix multiplication? **25.1-4** Show that matrix multiplication defined by EXTEND-SHORTEST-PATHS is associative. **25.1-5** Show how to express the single-source shortest-paths problem as a product of matrices and a vector. Describe how evaluating this product corresponds to a Bellman-Ford-like algorithm (see Section 24.1). **25.1-6** Suppose we also wish to compute the vertices on shortest paths in the algorithms of this section. Show how to compute the predecessor matrix $\Pi$ from the completed matrix $L$ of shortest-path weights in $O(n^3)$ time. **25.1-7** We can also compute the vertices on shortest paths as we compute the shortest-path weights. Define $\pi_{ij}^{(m)}$ as the predecessor of vertex $j$ on any minimum-weight path from $i$ to $j$ that contains at most $m$ edges. Modify EXTEND-SHORTEST-PATHS and SLOW-ALL-PAIRS-SHORTEST-PATHS to compute the matrices $\Pi^{(1)}, \Pi^{(2)}, \ldots, \Pi^{(n-1)}$ as the matrices $L^{(1)}, L^{(2)}, \ldots, L^{(n-1)}$ are computed. **25.1-8** The FASTER-ALL-PAIRS-SHORTEST-PATHS procedure, as written, requires us to store $\lceil \lg(n-1) \rceil$ matrices, each with $n^2$ elements, for a total space requirement of $\Theta(n^2 \lg n)$. Modify the procedure to require only $\Theta(n^2)$ space by using only two $n \times n$ matrices. **25.1-9** Modify FASTER-ALL-PAIRS-SHORTEST-PATHS so that it can determine whether the graph contains a negative-weight cycle. **25.1-10** Give an efficient algorithm to find the length (number of edges) of a minimum-length negative-weight cycle in a graph. ## 25.2 The Floyd-Warshall algorithm In this section, we shall use a different dynamic-programming formulation to solve the all-pairs shortest-paths problem on a directed graph $G = (V, E)$. The resulting algorithm, known as the **Floyd-Warshall algorithm**, runs in $\Theta(V^3)$ time. As before, negative-weight edges may be present, but we assume that there are no negative-weight cycles. As in Section 25.1, we follow the dynamic-programming process to develop the algorithm. After studying the resulting algorithm, we present a similar method for finding the transitive closure of a directed graph. > **Think about it.** Section 25.1 asked "how many *edges* may the path use?" Floyd-Warshall asks a cleverer question: "which *stopover cities* may the path use?" Picture a traveler who is allowed to change planes only at airports $1, \ldots, k$. What happens to the best itinerary when we unlock airport $k$? ### The structure of a shortest path In the Floyd-Warshall algorithm, we characterize the structure of a shortest path differently from how we characterized it in Section 25.1. The Floyd-Warshall algorithm considers the **intermediate** vertices of a shortest path, where an intermediate vertex of a simple path $p = \langle v_1, v_2, \ldots, v_l \rangle$ is any vertex of $p$ other than $v_1$ or $v_l$, that is, any vertex in the set $\{v_2, v_3, \ldots, v_{l-1}\}$. The Floyd-Warshall algorithm relies on the following observation. Under our assumption that the vertices of $G$ are $V = \{1, 2, \ldots, n\}$, let us consider a subset $\{1, 2, \ldots, k\}$ of vertices for some $k$. For any pair of vertices $i, j \in V$, consider all paths from $i$ to $j$ whose intermediate vertices are all drawn from $\{1, 2, \ldots, k\}$, and let $p$ be a minimum-weight path from among them. (Path $p$ is simple.) The Floyd-Warshall algorithm exploits a relationship between path $p$ and shortest paths from $i$ to $j$ with all intermediate vertices in the set $\{1, 2, \ldots, k-1\}$. The relationship depends on whether or not $k$ is an intermediate vertex of path $p$. - **Case 1: $k$ is not an intermediate vertex of $p$.** Then all intermediate vertices of path $p$ are in the set $\{1, 2, \ldots, k-1\}$. Thus, a shortest path from vertex $i$ to vertex $j$ with all intermediate vertices in the set $\{1, 2, \ldots, k-1\}$ is also a shortest path from $i$ to $j$ with all intermediate vertices in the set $\{1, 2, \ldots, k\}$. - **Case 2: $k$ is an intermediate vertex of $p$.** Then we decompose $p$ into $i \overset{p_1}{\leadsto} k \overset{p_2}{\leadsto} j$, as Figure 25.3 illustrates. By Lemma 24.1, $p_1$ is a shortest path from $i$ to $k$ with all intermediate vertices in the set $\{1, 2, \ldots, k\}$. In fact, we can make a slightly stronger statement. Because vertex $k$ is not an intermediate vertex of path $p_1$ (the path $p$ is simple, so $k$ appears only once), all intermediate vertices of $p_1$ are in the set $\{1, 2, \ldots, k-1\}$. Therefore, $p_1$ is a shortest path from $i$ to $k$ with all intermediate vertices in the set $\{1, 2, \ldots, k-1\}$. Similarly, $p_2$ is a shortest path from vertex $k$ to vertex $j$ with all intermediate vertices in the set $\{1, 2, \ldots, k-1\}$. *Figure 25.3: Path $p$ is a shortest path from vertex $i$ to vertex $j$, and $k$ is the highest-numbered intermediate vertex of $p$. Path $p_1$, the portion of path $p$ from vertex $i$ to vertex $k$, has all intermediate vertices in the set $\{1, 2, \ldots, k-1\}$. The same holds for path $p_2$ from vertex $k$ to vertex $j$. (The picture shows $i \leadsto k \leadsto j$, with $p_1$ and $p_2$ each labeled "all intermediate vertices in $\{1, 2, \ldots, k-1\}$" and the whole path $p$ labeled "all intermediate vertices in $\{1, 2, \ldots, k\}$".)* ### A recursive solution to the all-pairs shortest-paths problem Based on the above observations, we define a recursive formulation of shortest-path estimates that differs from the one in Section 25.1. Let $d_{ij}^{(k)}$ be the weight of a shortest path from vertex $i$ to vertex $j$ for which all intermediate vertices are in the set $\{1, 2, \ldots, k\}$. When $k = 0$, a path from vertex $i$ to vertex $j$ with no intermediate vertex numbered higher than $0$ has no intermediate vertices at all. Such a path has at most one edge, and hence $d_{ij}^{(0)} = w_{ij}$. Following the above discussion, we define $d_{ij}^{(k)}$ recursively by

d_{ij}^{(k)} = \begin{cases} w_{ij} & \text{if } k = 0, \[4pt] \min\left( d_{ij}^{(k-1)}, \ d_{ik}^{(k-1)} + d_{kj}^{(k-1)} \right) & \text{if } k \ge 1. \end{cases} \tag{25.5}

Because for any path, all intermediate vertices are in the set $\{1, 2, \ldots, n\}$, the matrix $D^{(n)} = (d_{ij}^{(n)})$ gives the final answer: $d_{ij}^{(n)} = \delta(i, j)$ for all $i, j \in V$. ### Computing the shortest-path weights bottom up Based on recurrence (25.5), we can use the following bottom-up procedure to compute the values $d_{ij}^{(k)}$ in order of increasing values of $k$. Its input is an $n \times n$ matrix $W$ defined as in equation (25.1). The procedure returns the matrix $D^{(n)}$ of shortest-path weights. ```text FLOYD-WARSHALL(W) 1 n = W.rows 2 D^(0) = W 3 for k = 1 to n 4 let D^(k) = (d_ij^(k)) be a new n × n matrix 5 for i = 1 to n 6 for j = 1 to n 7 d_ij^(k) = min(d_ij^(k-1), d_ik^(k-1) + d_kj^(k-1)) 8 return D^(n) ``` Figure 25.4 shows the matrices $D^{(k)}$ computed by the Floyd-Warshall algorithm for the graph in Figure 25.1. The running time of the Floyd-Warshall algorithm is determined by the triply nested **for** loops of lines 3-7. Because each execution of line 7 takes $O(1)$ time, the algorithm runs in time $\Theta(n^3)$. As in the final algorithm in Section 25.1, the code is tight, with no elaborate data structures, and so the constant hidden in the $\Theta$-notation is small. Thus, the Floyd-Warshall algorithm is quite practical for even moderate-sized input graphs. ### Constructing a shortest path There are a variety of different methods for constructing shortest paths in the Floyd-Warshall algorithm. One way is to compute the matrix $D$ of shortest-path weights and then construct the predecessor matrix $\Pi$ from the $D$ matrix. Exercise 25.1-6 asks you to implement this method so that it runs in $O(n^3)$ time. Given the predecessor matrix $\Pi$, the PRINT-ALL-PAIRS-SHORTEST-PATH procedure will print the vertices on a given shortest path. Alternatively, we can compute the predecessor matrix $\Pi$ *while* the algorithm computes the matrices $D^{(k)}$. Specifically, we compute a sequence of matrices $\Pi^{(0)}, \Pi^{(1)}, \ldots, \Pi^{(n)}$, where $\Pi = \Pi^{(n)}$ and we define $\pi_{ij}^{(k)}$ as the predecessor of vertex $j$ on a shortest path from vertex $i$ with all intermediate vertices in the set $\{1, 2, \ldots, k\}$. We can give a recursive formulation of $\pi_{ij}^{(k)}$. When $k = 0$, a shortest path from $i$ to $j$ has no intermediate vertices at all. Thus,

\pi_{ij}^{(0)} = \begin{cases} \text{NIL} & \text{if } i = j \text{ or } w_{ij} = \infty, \ i & \text{if } i \neq j \text{ and } w_{ij} < \infty. \end{cases} \tag{25.6}

For $k \ge 1$, if we take the path $i \leadsto k \leadsto j$, where $k \neq j$, then the predecessor of $j$ we choose is the same as the predecessor of $j$ we chose on a shortest path from $k$ with all intermediate vertices in the set $\{1, 2, \ldots, k-1\}$. Otherwise, we choose the same predecessor of $j$ that we chose on a shortest path from $i$ with all intermediate vertices in the set $\{1, 2, \ldots, k-1\}$. Formally, for $k \ge 1$,

\pi_{ij}^{(k)} = \begin{cases} \pi_{ij}^{(k-1)} & \text{if } d_{ij}^{(k-1)} \le d_{ik}^{(k-1)} + d_{kj}^{(k-1)}, \[4pt] \pi_{kj}^{(k-1)} & \text{if } d_{ij}^{(k-1)} > d_{ik}^{(k-1)} + d_{kj}^{(k-1)}. \end{cases} \tag{25.7}

We leave the incorporation of the $\Pi^{(k)}$ matrix computations into the FLOYD-WARSHALL procedure as Exercise 25.2-3. Figure 25.4 shows the sequence of $\Pi^{(k)}$ matrices that the resulting algorithm computes for the graph of Figure 25.1. The exercise also asks for the more difficult task of proving that the predecessor subgraph $G_{\pi,i}$ is a shortest-paths tree with root $i$. Exercise 25.2-7 asks for yet another way to reconstruct shortest paths. *Figure 25.4: The sequence of matrices $D^{(k)}$ and $\Pi^{(k)}$ computed by the Floyd-Warshall algorithm for the graph in Figure 25.1.*

D^{(0)} = \begin{pmatrix} 0 & 3 & 8 & \infty & -4 \ \infty & 0 & \infty & 1 & 7 \ \infty & 4 & 0 & \infty & \infty \ 2 & \infty & -5 & 0 & \infty \ \infty & \infty & \infty & 6 & 0 \end{pmatrix} \quad \Pi^{(0)} = \begin{pmatrix} \text{NIL} & 1 & 1 & \text{NIL} & 1 \ \text{NIL} & \text{NIL} & \text{NIL} & 2 & 2 \ \text{NIL} & 3 & \text{NIL} & \text{NIL} & \text{NIL} \ 4 & \text{NIL} & 4 & \text{NIL} & \text{NIL} \ \text{NIL} & \text{NIL} & \text{NIL} & 5 & \text{NIL} \end{pmatrix}

D^{(1)} = \begin{pmatrix} 0 & 3 & 8 & \infty & -4 \ \infty & 0 & \infty & 1 & 7 \ \infty & 4 & 0 & \infty & \infty \ 2 & 5 & -5 & 0 & -2 \ \infty & \infty & \infty & 6 & 0 \end{pmatrix} \quad \Pi^{(1)} = \begin{pmatrix} \text{NIL} & 1 & 1 & \text{NIL} & 1 \ \text{NIL} & \text{NIL} & \text{NIL} & 2 & 2 \ \text{NIL} & 3 & \text{NIL} & \text{NIL} & \text{NIL} \ 4 & 1 & 4 & \text{NIL} & 1 \ \text{NIL} & \text{NIL} & \text{NIL} & 5 & \text{NIL} \end{pmatrix}

D^{(2)} = \begin{pmatrix} 0 & 3 & 8 & 4 & -4 \ \infty & 0 & \infty & 1 & 7 \ \infty & 4 & 0 & 5 & 11 \ 2 & 5 & -5 & 0 & -2 \ \infty & \infty & \infty & 6 & 0 \end{pmatrix} \quad \Pi^{(2)} = \begin{pmatrix} \text{NIL} & 1 & 1 & 2 & 1 \ \text{NIL} & \text{NIL} & \text{NIL} & 2 & 2 \ \text{NIL} & 3 & \text{NIL} & 2 & 2 \ 4 & 1 & 4 & \text{NIL} & 1 \ \text{NIL} & \text{NIL} & \text{NIL} & 5 & \text{NIL} \end{pmatrix}

D^{(3)} = \begin{pmatrix} 0 & 3 & 8 & 4 & -4 \ \infty & 0 & \infty & 1 & 7 \ \infty & 4 & 0 & 5 & 11 \ 2 & -1 & -5 & 0 & -2 \ \infty & \infty & \infty & 6 & 0 \end{pmatrix} \quad \Pi^{(3)} = \begin{pmatrix} \text{NIL} & 1 & 1 & 2 & 1 \ \text{NIL} & \text{NIL} & \text{NIL} & 2 & 2 \ \text{NIL} & 3 & \text{NIL} & 2 & 2 \ 4 & 3 & 4 & \text{NIL} & 1 \ \text{NIL} & \text{NIL} & \text{NIL} & 5 & \text{NIL} \end{pmatrix}

D^{(4)} = \begin{pmatrix} 0 & 3 & -1 & 4 & -4 \ 3 & 0 & -4 & 1 & -1 \ 7 & 4 & 0 & 5 & 3 \ 2 & -1 & -5 & 0 & -2 \ 8 & 5 & 1 & 6 & 0 \end{pmatrix} \quad \Pi^{(4)} = \begin{pmatrix} \text{NIL} & 1 & 4 & 2 & 1 \ 4 & \text{NIL} & 4 & 2 & 1 \ 4 & 3 & \text{NIL} & 2 & 1 \ 4 & 3 & 4 & \text{NIL} & 1 \ 4 & 3 & 4 & 5 & \text{NIL} \end{pmatrix}

D^{(5)} = \begin{pmatrix} 0 & 1 & -3 & 2 & -4 \ 3 & 0 & -4 & 1 & -1 \ 7 & 4 & 0 & 5 & 3 \ 2 & -1 & -5 & 0 & -2 \ 8 & 5 & 1 & 6 & 0 \end{pmatrix} \quad \Pi^{(5)} = \begin{pmatrix} \text{NIL} & 3 & 4 & 5 & 1 \ 4 & \text{NIL} & 4 & 2 & 1 \ 4 & 3 & \text{NIL} & 2 & 1 \ 4 & 3 & 4 & \text{NIL} & 1 \ 4 & 3 & 4 & 5 & \text{NIL} \end{pmatrix}

### Transitive closure of a directed graph Given a directed graph $G = (V, E)$ with vertex set $V = \{1, 2, \ldots, n\}$, we might wish to determine whether $G$ contains a path from $i$ to $j$ for all vertex pairs $i, j \in V$. We define the **transitive closure** of $G$ as the graph $G^* = (V, E^*)$, where $$E^* = \{ (i, j) : \text{there is a path from vertex } i \text{ to vertex } j \text{ in } G \} .$$ One way to compute the transitive closure of a graph in $\Theta(n^3)$ time is to assign a weight of $1$ to each edge of $E$ and run the Floyd-Warshall algorithm. If there is a path from vertex $i$ to vertex $j$, we get $d_{ij} < n$. Otherwise, we get $d_{ij} = \infty$. There is another, similar way to compute the transitive closure of $G$ in $\Theta(n^3)$ time that can save time and space in practice. This method substitutes the logical operations $\lor$ (logical OR) and $\land$ (logical AND) for the arithmetic operations $\min$ and $+$ in the Floyd-Warshall algorithm. For $i, j, k = 1, 2, \ldots, n$, we define $t_{ij}^{(k)}$ to be $1$ if there exists a path in graph $G$ from vertex $i$ to vertex $j$ with all intermediate vertices in the set $\{1, 2, \ldots, k\}$, and $0$ otherwise. We construct the transitive closure $G^* = (V, E^*)$ by putting edge $(i, j)$ into $E^*$ if and only if $t_{ij}^{(n)} = 1$. A recursive definition of $t_{ij}^{(k)}$, analogous to recurrence (25.5), is

t_{ij}^{(0)} = \begin{cases} 0 & \text{if } i \neq j \text{ and } (i, j) \notin E, \ 1 & \text{if } i = j \text{ or } (i, j) \in E, \end{cases}

and for $k \ge 1$, $$t_{ij}^{(k)} = t_{ij}^{(k-1)} \lor \left( t_{ik}^{(k-1)} \land t_{kj}^{(k-1)} \right) . \tag{25.8}$$ As in the Floyd-Warshall algorithm, we compute the matrices $T^{(k)} = (t_{ij}^{(k)})$ in order of increasing $k$. ```text TRANSITIVE-CLOSURE(G) 1 n = |G.V| 2 let T^(0) = (t_ij^(0)) be a new n × n matrix 3 for i = 1 to n 4 for j = 1 to n 5 if i == j or (i, j) ∈ G.E 6 t_ij^(0) = 1 7 else t_ij^(0) = 0 8 for k = 1 to n 9 let T^(k) = (t_ij^(k)) be a new n × n matrix 10 for i = 1 to n 11 for j = 1 to n 12 t_ij^(k) = t_ij^(k-1) ∨ (t_ik^(k-1) ∧ t_kj^(k-1)) 13 return T^(n) ``` Figure 25.5 shows the matrices $T^{(k)}$ computed by the TRANSITIVE-CLOSURE procedure on a sample graph. The TRANSITIVE-CLOSURE procedure, like the Floyd-Warshall algorithm, runs in $\Theta(n^3)$ time. On some computers, though, logical operations on single-bit values execute faster than arithmetic operations on integer words of data. Moreover, because the direct transitive-closure algorithm uses only boolean values rather than integer values, its space requirement is less than the Floyd-Warshall algorithm's by a factor corresponding to the size of a word of computer storage. *Figure 25.5: A directed graph on vertices 1-4 (edges $2\to3$, $2\to4$, $3\to2$, $4\to1$, $4\to3$) and the matrices $T^{(k)}$ computed by the transitive-closure algorithm.*

T^{(0)} = T^{(1)} = \begin{pmatrix} 1 & 0 & 0 & 0 \ 0 & 1 & 1 & 1 \ 0 & 1 & 1 & 0 \ 1 & 0 & 1 & 1 \end{pmatrix} \quad T^{(2)} = \begin{pmatrix} 1 & 0 & 0 & 0 \ 0 & 1 & 1 & 1 \ 0 & 1 & 1 & 1 \ 1 & 0 & 1 & 1 \end{pmatrix}

T^{(3)} = \begin{pmatrix} 1 & 0 & 0 & 0 \ 0 & 1 & 1 & 1 \ 0 & 1 & 1 & 1 \ 1 & 1 & 1 & 1 \end{pmatrix} \quad T^{(4)} = \begin{pmatrix} 1 & 0 & 0 & 0 \ 1 & 1 & 1 & 1 \ 1 & 1 & 1 & 1 \ 1 & 1 & 1 & 1 \end{pmatrix}

### Exercises 25.2-1 to 25.2-9 **25.2-1** Run the Floyd-Warshall algorithm on the weighted, directed graph of Figure 25.2. Show the matrix $D^{(k)}$ that results for each iteration of the outer loop. **25.2-2** Show how to compute the transitive closure using the technique of Section 25.1. **25.2-3** Modify the FLOYD-WARSHALL procedure to compute the $\Pi^{(k)}$ matrices according to equations (25.6) and (25.7). Prove rigorously that for all $i \in V$, the predecessor subgraph $G_{\pi,i}$ is a shortest-paths tree with root $i$. (*Hint:* To show that $G_{\pi,i}$ is acyclic, first show that $\pi_{ij}^{(k)} = l$ implies $d_{ij}^{(k)} \ge d_{il}^{(k)} + w_{lj}$, according to the definition of $\pi_{ij}^{(k)}$. Then, adapt the proof of Lemma 24.16.) **25.2-4** As it appears above, the Floyd-Warshall algorithm requires $\Theta(n^3)$ space, since we compute $d_{ij}^{(k)}$ for $i, j, k = 1, 2, \ldots, n$. Show that the following procedure, which simply drops all the superscripts, is correct, and thus only $\Theta(n^2)$ space is required. ```text FLOYD-WARSHALL'(W) 1 n = W.rows 2 D = W 3 for k = 1 to n 4 for i = 1 to n 5 for j = 1 to n 6 d_ij = min(d_ij, d_ik + d_kj) 7 return D ``` **25.2-5** Suppose that we modify the way in which equation (25.7) handles equality:

\pi_{ij}^{(k)} = \begin{cases} \pi_{ij}^{(k-1)} & \text{if } d_{ij}^{(k-1)} < d_{ik}^{(k-1)} + d_{kj}^{(k-1)}, \[4pt] \pi_{kj}^{(k-1)} & \text{if } d_{ij}^{(k-1)} \ge d_{ik}^{(k-1)} + d_{kj}^{(k-1)}. \end{cases}

Is this alternative definition of the predecessor matrix $\Pi$ correct? **25.2-6** How can we use the output of the Floyd-Warshall algorithm to detect the presence of a negative-weight cycle? **25.2-7** Another way to reconstruct shortest paths in the Floyd-Warshall algorithm uses values $\phi_{ij}^{(k)}$ for $i, j, k = 1, 2, \ldots, n$, where $\phi_{ij}^{(k)}$ is the highest-numbered intermediate vertex of a shortest path from $i$ to $j$ in which all intermediate vertices are in the set $\{1, 2, \ldots, k\}$. Give a recursive formulation for $\phi_{ij}^{(k)}$, modify the FLOYD-WARSHALL procedure to compute the $\phi_{ij}^{(k)}$ values, and rewrite the PRINT-ALL-PAIRS-SHORTEST-PATH procedure to take the matrix $\Phi = (\phi_{ij}^{(n)})$ as an input. How is the matrix $\Phi$ like the $s$ table in the matrix-chain multiplication problem of Section 15.2? **25.2-8** Give an $O(VE)$-time algorithm for computing the transitive closure of a directed graph $G = (V, E)$. **25.2-9** Suppose that we can compute the transitive closure of a directed acyclic graph in $f(|V|, |E|)$ time, where $f$ is a monotonically increasing function of $|V|$ and $|E|$. Show that the time to compute the transitive closure $G^* = (V, E^*)$ of a general directed graph $G = (V, E)$ is then $f(|V|, |E|) + O(V + E^*)$. ## 25.3 Johnson's algorithm for sparse graphs Johnson's algorithm finds shortest paths between all pairs in $O(V^2 \lg V + VE)$ time. For sparse graphs, it is asymptotically faster than either repeated squaring of matrices or the Floyd-Warshall algorithm. The algorithm either returns a matrix of shortest-path weights for all pairs of vertices or reports that the input graph contains a negative-weight cycle. Johnson's algorithm uses as subroutines both Dijkstra's algorithm and the Bellman-Ford algorithm, which Chapter 24 describes. Johnson's algorithm uses the technique of **reweighting**, which works as follows. If all edge weights $w$ in a graph $G = (V, E)$ are nonnegative, we can find shortest paths between all pairs of vertices by running Dijkstra's algorithm once from each vertex; with the Fibonacci-heap min-priority queue, the running time of this all-pairs algorithm is $O(V^2 \lg V + VE)$. If $G$ has negative-weight edges but no negative-weight cycles, we simply compute a new set of nonnegative edge weights that allows us to use the same method. The new set of edge weights $\hat{w}$ must satisfy two important properties: 1. For all pairs of vertices $u, v \in V$, a path $p$ is a shortest path from $u$ to $v$ using weight function $w$ if and only if $p$ is also a shortest path from $u$ to $v$ using weight function $\hat{w}$. 2. For all edges $(u, v)$, the new weight $\hat{w}(u, v)$ is nonnegative. As we shall see in a moment, we can preprocess $G$ to determine the new weight function $\hat{w}$ in $O(VE)$ time. > **Think about it.** Adding a constant to every edge (the "obvious" fix) fails, because paths with more edges get penalized more. Reweighting must charge every path the *same* total adjustment, whatever its length. What kind of sum has that property? A telescoping one. ### Preserving shortest paths by reweighting The following lemma shows how easily we can reweight the edges to satisfy the first property above. We use $\delta$ to denote shortest-path weights derived from weight function $w$ and $\hat{\delta}$ to denote shortest-path weights derived from weight function $\hat{w}$. **Lemma 25.1 (Reweighting does not change shortest paths).** Given a weighted, directed graph $G = (V, E)$ with weight function $w : E \to \mathbb{R}$, let $h : V \to \mathbb{R}$ be any function mapping vertices to real numbers. For each edge $(u, v) \in E$, define $$\hat{w}(u, v) = w(u, v) + h(u) - h(v) . \tag{25.9}$$ Let $p = \langle v_0, v_1, \ldots, v_k \rangle$ be any path from vertex $v_0$ to vertex $v_k$. Then $p$ is a shortest path from $v_0$ to $v_k$ with weight function $w$ if and only if it is a shortest path with weight function $\hat{w}$. That is, $w(p) = \delta(v_0, v_k)$ if and only if $\hat{w}(p) = \hat{\delta}(v_0, v_k)$. Furthermore, $G$ has a negative-weight cycle using weight function $w$ if and only if $G$ has a negative-weight cycle using weight function $\hat{w}$. > **Proof.** We start by showing that > > $$\hat{w}(p) = w(p) + h(v_0) - h(v_k) . \tag{25.10}$$ > > We have > > $$ > \begin{aligned} > \hat{w}(p) &= \sum_{i=1}^{k} \hat{w}(v_{i-1}, v_i) && \text{(weight of a path is the sum of its edge weights)} \\ > &= \sum_{i=1}^{k} \bigl( w(v_{i-1}, v_i) + h(v_{i-1}) - h(v_i) \bigr) && \text{(by equation (25.9))} \\ > &= \sum_{i=1}^{k} w(v_{i-1}, v_i) + \sum_{i=1}^{k} \bigl( h(v_{i-1}) - h(v_i) \bigr) && \text{(split the sum)} \\ > &= \sum_{i=1}^{k} w(v_{i-1}, v_i) + h(v_0) - h(v_k) && \text{(because the second sum telescopes)} \\ > &= w(p) + h(v_0) - h(v_k) . && \text{(definition of } w(p)\text{)} > \end{aligned} > $$ > > The telescoping step is > $\bigl(h(v_0) - h(v_1)\bigr) + \bigl(h(v_1) - h(v_2)\bigr) + \cdots + \bigl(h(v_{k-1}) - h(v_k)\bigr) = h(v_0) - h(v_k)$, since every intermediate term appears once with each sign. > > **Shortest paths correspond.** Equation (25.10) holds for every path from $v_0$ to $v_k$, and the term $h(v_0) - h(v_k)$ does not depend on which path we choose. Hence, for any two paths $p$ and $p'$ from $v_0$ to $v_k$, > > $$ > \begin{aligned} > w(p) < w(p') > &\iff w(p) + h(v_0) - h(v_k) < w(p') + h(v_0) - h(v_k) && \text{(add the same constant to both sides)} \\ > &\iff \hat{w}(p) < \hat{w}(p') && \text{(by equation (25.10))}. > \end{aligned} > $$ > > So if one path from $v_0$ to $v_k$ is shorter than another using weight function $w$, then it is also shorter using $\hat{w}$, and conversely. Consequently, the minimum over all paths is attained by the same paths, and > > $$\hat{\delta}(v_0, v_k) = \delta(v_0, v_k) + h(v_0) - h(v_k) .$$ > > Thus, $w(p) = \delta(v_0, v_k)$ if and only if $\hat{w}(p) = \hat{\delta}(v_0, v_k)$. (If no path from $v_0$ to $v_k$ exists, both $\delta$ and $\hat{\delta}$ equal $\infty$, and the statement is vacuous.) > > **Negative-weight cycles correspond.** Finally, we show that $G$ has a negative-weight cycle using weight function $w$ if and only if $G$ has a negative-weight cycle using weight function $\hat{w}$. Consider any cycle $c = \langle v_0, v_1, \ldots, v_k \rangle$, where $v_0 = v_k$. By equation (25.10), > > $$ > \begin{aligned} > \hat{w}(c) &= w(c) + h(v_0) - h(v_k) && \text{(by equation (25.10))} \\ > &= w(c) , && \text{(because } v_0 = v_k\text{)} > \end{aligned} > $$ > > and thus $c$ has negative weight using $w$ if and only if it has negative weight using $\hat{w}$. $\blacksquare$ ### Producing nonnegative weights by reweighting Our next goal is to ensure that the second property holds: we want $\hat{w}(u, v)$ to be nonnegative for all edges $(u, v) \in E$. Given a weighted, directed graph $G = (V, E)$ with weight function $w : E \to \mathbb{R}$, we make a new graph $G' = (V', E')$, where $V' = V \cup \{s\}$ for some new vertex $s \notin V$ and $E' = E \cup \{(s, v) : v \in V\}$. We extend the weight function $w$ so that $w(s, v) = 0$ for all $v \in V$. Note that because $s$ has no edges that enter it, no shortest paths in $G'$, other than those with source $s$, contain $s$. Moreover, $G'$ has no negative-weight cycles if and only if $G$ has no negative-weight cycles. Figure 25.6(a) shows the graph $G'$ corresponding to the graph $G$ of Figure 25.1. Now suppose that $G$ and $G'$ have no negative-weight cycles. Let us define $h(v) = \delta(s, v)$ for all $v \in V'$. By the triangle inequality (Lemma 24.10), we have $h(v) \le h(u) + w(u, v)$ for all edges $(u, v) \in E'$. Thus, if we define the new weights $\hat{w}$ by reweighting according to equation (25.9), we have

\begin{aligned} \hat{w}(u, v) &= w(u, v) + h(u) - h(v) && \text{(equation (25.9))} \ &\ge 0 , && \text{(since } h(v) \le h(u) + w(u,v) \text{, i.e. } w(u,v) + h(u) - h(v) \ge 0\text{)} \end{aligned}

and we have satisfied the second property. Figure 25.6(b) shows the graph $G'$ from Figure 25.6(a) with reweighted edges. ### Computing all-pairs shortest paths Johnson's algorithm to compute all-pairs shortest paths uses the Bellman-Ford algorithm (Section 24.1) and Dijkstra's algorithm (Section 24.3) as subroutines. It assumes implicitly that the edges are stored in adjacency lists. The algorithm returns the usual $|V| \times |V|$ matrix $D = (d_{ij})$, where $d_{ij} = \delta(i, j)$, or it reports that the input graph contains a negative-weight cycle. As is typical for an all-pairs shortest-paths algorithm, we assume that the vertices are numbered from $1$ to $|V|$. *Figure 25.6: Johnson's all-pairs shortest-paths algorithm run on the graph of Figure 25.1. Vertex numbers appear outside the vertices. (a) The graph $G'$ with the original weight function $w$. The new vertex $s$ is black. Within each vertex $v$ is $h(v) = \delta(s, v)$. (b) After reweighting each edge $(u, v)$ with weight function $\hat{w}(u, v) = w(u, v) + h(u) - h(v)$. (c)-(g) The result of running Dijkstra's algorithm on each vertex of $G$ using weight function $\hat{w}$. In each part, the source vertex $u$ is black, and shaded edges are in the shortest-paths tree computed by the algorithm. Within each vertex $v$ are the values $\hat{\delta}(u, v)$ and $\delta(u, v)$, separated by a slash. The value $d_{uv} = \delta(u, v)$ is equal to $\hat{\delta}(u, v) + h(v) - h(u)$.* The figure consists of seven drawings (a)-(g) of the five-vertex graph of Figure 25.1, with the extra source vertex $s$ in (a) and (b); the numeric labels are not recoverable from the source text. ```text JOHNSON(G, w) 1 compute G', where G'.V = G.V ∪ {s}, G'.E = G.E ∪ {(s, v) : v ∈ G.V}, and w(s, v) = 0 for all v ∈ G.V 2 if BELLMAN-FORD(G', w, s) == FALSE 3 print "the input graph contains a negative-weight cycle" 4 else for each vertex v ∈ G'.V 5 set h(v) to the value of δ(s, v) computed by the Bellman-Ford algorithm 6 for each edge (u, v) ∈ G'.E 7 ŵ(u, v) = w(u, v) + h(u) - h(v) 8 let D = (d_uv) be a new n × n matrix 9 for each vertex u ∈ G.V 10 run DIJKSTRA(G, ŵ, u) to compute δ̂(u, v) for all v ∈ G.V 11 for each vertex v ∈ G.V 12 d_uv = δ̂(u, v) + h(v) - h(u) 13 return D ``` This code simply performs the actions we specified earlier. Line 1 produces $G'$. Line 2 runs the Bellman-Ford algorithm on $G'$ with weight function $w$ and source vertex $s$. If $G'$, and hence $G$, contains a negative-weight cycle, line 3 reports the problem. Lines 4-12 assume that $G'$ contains no negative-weight cycles. Lines 4-5 set $h(v)$ to the shortest-path weight $\delta(s, v)$ computed by the Bellman-Ford algorithm for all $v \in V'$. Lines 6-7 compute the new weights $\hat{w}$. For each pair of vertices $u, v \in V$, the **for** loop of lines 9-12 computes the shortest-path weight $\hat{\delta}(u, v)$ by calling Dijkstra's algorithm once from each vertex in $V$. Line 12 stores in matrix entry $d_{uv}$ the correct shortest-path weight $\delta(u, v)$, calculated using equation (25.10). Finally, line 13 returns the completed $D$ matrix. Figure 25.6 depicts the execution of Johnson's algorithm. Line 12 is correct because equation (25.10) gives $\hat{\delta}(u, v) = \delta(u, v) + h(u) - h(v)$, which rearranges to $$\delta(u, v) = \hat{\delta}(u, v) + h(v) - h(u) .$$ #### Running time If we implement the min-priority queue in Dijkstra's algorithm by a Fibonacci heap, Johnson's algorithm runs in $O(V^2 \lg V + VE)$ time. The simpler binary min-heap implementation yields a running time of $O(VE \lg V)$, which is still asymptotically faster than the Floyd-Warshall algorithm if the graph is sparse. To see the accounting for the Fibonacci-heap version:

\begin{aligned} \text{line 1} &: O(V) \ \text{line 2 (Bellman-Ford on } G’\text{, with } |V’| = V+1 \text{ and } |E’| = E+V) &: O(VE) \ \text{lines 6-7} &: O(E + V) \ \text{lines 9-12 (}V\text{ runs of Dijkstra, each } O(V \lg V + E)\text{, plus } O(V) \text{ per run)} &: O(V^2 \lg V + VE) \end{aligned}

c_f(u,v) = \begin{cases} c(u,v) - f(u,v) & \text{if } (u,v) \in E, \ f(v,u) & \text{if } (v,u) \in E, \ 0 & \text{otherwise.} \end{cases} \tag{26.2}

Because of our assumption that $(u,v) \in E$ implies $(v,u) \notin E$, exactly one case in equation (26.2) applies to each ordered pair of vertices. As an example of equation (26.2), if $c(u,v) = 16$ and $f(u,v) = 11$, then we can increase $f(u,v)$ by up to $c_f(u,v) = 16 - 11 = 5$ units before we exceed the capacity constraint on edge $(u,v)$. We also wish to allow an algorithm to return up to $11$ units of flow from $v$ to $u$, and hence $c_f(v,u) = 11$. Given a flow network $G = (V,E)$ and a flow $f$, the **residual network** of $G$ induced by $f$ is $G_f = (V, E_f)$, where $$E_f = \{(u,v) \in V \times V : c_f(u,v) > 0\}. \tag{26.3}$$ That is, as promised above, each edge of the residual network, or **residual edge**, can admit a flow that is greater than $0$. Figure 26.4(a) repeats the flow network $G$ and flow $f$ of Figure 26.1(b), and Figure 26.4(b) shows the corresponding residual network $G_f$. The edges in $E_f$ are either edges in $E$ or their reversals, and thus $$|E_f| \le 2|E|.$$ Observe that the residual network $G_f$ is similar to a flow network with capacities given by $c_f$. It does not satisfy our definition of a flow network because it may contain both an edge $(u,v)$ and its reversal $(v,u)$. Other than this difference, a residual network has the same properties as a flow network, and we can define a flow in the residual network as one that satisfies the definition of a flow, but with respect to capacities $c_f$ in the network $G_f$. A flow in a residual network provides a roadmap for adding flow to the original flow network. If $f$ is a flow in $G$ and $f'$ is a flow in the corresponding residual network $G_f$, we define $f \uparrow f'$, the **augmentation** of flow $f$ by $f'$, to be a function from $V \times V$ to $\mathbb{R}$, defined by

(f \uparrow f’)(u,v) = \begin{cases} f(u,v) + f’(u,v) - f’(v,u) & \text{if } (u,v) \in E, \ 0 & \text{otherwise.} \end{cases} \tag{26.4}

*Figure 26.4: (a) The flow network $G$ and flow $f$ of Figure 26.1(b). (b) The residual network $G_f$ with augmenting path $p$ shaded; its residual capacity is $c_f(p) = c_f(v_2,v_3) = 4$. Edges with residual capacity equal to $0$, such as $(v_1,v_3)$, are not shown, a convention we follow in the remainder of this section. (c) The flow in $G$ that results from augmenting along path $p$ by its residual capacity $4$. Edges carrying no flow, such as $(v_3,v_2)$, are labeled only by their capacity, another convention we follow throughout. (d) The residual network induced by the flow in (c).* The intuition behind this definition follows the definition of the residual network. We increase the flow on $(u,v)$ by $f'(u,v)$ but decrease it by $f'(v,u)$ because pushing flow on the reverse edge in the residual network signifies decreasing the flow in the original network. Pushing flow on the reverse edge in the residual network is also known as **cancellation**. For example, if we send $5$ crates of hockey pucks from $u$ to $v$ and send $2$ crates from $v$ to $u$, we could equivalently (from the perspective of the final result) just send $3$ crates from $u$ to $v$ and none from $v$ to $u$. Cancellation of this type is crucial for any maximum-flow algorithm. > **Think about it.** Why must a maximum-flow algorithm be allowed to "take back" flow? Imagine a greedy shipper who commits to a route early on and blocks a better combination. Reverse edges are the "undo button." > **Lemma 26.1.** Let $G = (V,E)$ be a flow network with source $s$ and sink $t$, and let $f$ be a flow in $G$. Let $G_f$ be the residual network of $G$ induced by $f$, and let $f'$ be a flow in $G_f$. Then the function $f \uparrow f'$ defined in equation (26.4) is a flow in $G$ with value $|f \uparrow f'| = |f| + |f'|$. > **Proof.** We first verify that $f \uparrow f'$ obeys the capacity constraint for each edge in $E$ and flow conservation at each vertex in $V - \{s,t\}$. > > **Capacity constraint, lower bound.** First observe that if $(u,v) \in E$, then $c_f(v,u) = f(u,v)$. Therefore, we have $f'(v,u) \le c_f(v,u) = f(u,v)$, and hence > $$ > \begin{aligned} > (f \uparrow f')(u,v) &= f(u,v) + f'(u,v) - f'(v,u) &&\text{(by equation (26.4))} \\ > &\ge f(u,v) + f'(u,v) - f(u,v) &&\text{(because } f'(v,u) \le f(u,v)\text{)} \\ > &= f'(u,v) \\ > &\ge 0. > \end{aligned} > $$ > > **Capacity constraint, upper bound.** In addition, > $$ > \begin{aligned} > (f \uparrow f')(u,v) &= f(u,v) + f'(u,v) - f'(v,u) &&\text{(by equation (26.4))} \\ > &\le f(u,v) + f'(u,v) &&\text{(because flows are nonnegative)} \\ > &\le f(u,v) + c_f(u,v) &&\text{(capacity constraint in } G_f\text{)} \\ > &= f(u,v) + c(u,v) - f(u,v) &&\text{(definition of } c_f\text{)} \\ > &= c(u,v). > \end{aligned} > $$ > > **Flow conservation.** Because both $f$ and $f'$ obey flow conservation, we have that for all $u \in V - \{s,t\}$, > $$ > \begin{aligned} > \sum_{v \in V} (f \uparrow f')(u,v) &= \sum_{v \in V} \bigl(f(u,v) + f'(u,v) - f'(v,u)\bigr) \\ > &= \sum_{v \in V} f(u,v) + \sum_{v \in V} f'(u,v) - \sum_{v \in V} f'(v,u) \\ > &= \sum_{v \in V} f(v,u) + \sum_{v \in V} f'(v,u) - \sum_{v \in V} f'(u,v) &&\text{(by flow conservation)} \\ > &= \sum_{v \in V} \bigl(f(v,u) + f'(v,u) - f'(u,v)\bigr) \\ > &= \sum_{v \in V} (f \uparrow f')(v,u), > \end{aligned} > $$ > where the third line follows from the second by flow conservation (applied to $f$ and to $f'$ at $u$). > > **Value.** Finally, we compute the value of $f \uparrow f'$. Recall that we disallow antiparallel edges in $G$ (but not in $G_f$), and hence for each vertex $v \in V$, we know that there can be an edge $(s,v)$ or $(v,s)$, but never both. We define $V_1 = \{v : (s,v) \in E\}$ to be the set of vertices with edges from $s$, and $V_2 = \{v : (v,s) \in E\}$ to be the set of vertices with edges to $s$. We have $V_1 \cup V_2 \subseteq V$ and, because we disallow antiparallel edges, $V_1 \cap V_2 = \emptyset$. We now compute > $$ > \begin{aligned} > |f \uparrow f'| &= \sum_{v \in V} (f \uparrow f')(s,v) - \sum_{v \in V} (f \uparrow f')(v,s) \\ > &= \sum_{v \in V_1} (f \uparrow f')(s,v) - \sum_{v \in V_2} (f \uparrow f')(v,s), > \end{aligned} > \tag{26.5} > $$ > where the second line follows because $(f \uparrow f')(w,x)$ is $0$ if $(w,x) \notin E$. We now apply the definition of $f \uparrow f'$ to equation (26.5), and then reorder and group terms to obtain > $$ > \begin{aligned} > |f \uparrow f'| &= \sum_{v \in V_1} \bigl(f(s,v) + f'(s,v) - f'(v,s)\bigr) - \sum_{v \in V_2} \bigl(f(v,s) + f'(v,s) - f'(s,v)\bigr) \\ > &= \sum_{v \in V_1} f(s,v) + \sum_{v \in V_1} f'(s,v) - \sum_{v \in V_1} f'(v,s) \\ > &\qquad - \sum_{v \in V_2} f(v,s) - \sum_{v \in V_2} f'(v,s) + \sum_{v \in V_2} f'(s,v) \\ > &= \sum_{v \in V_1} f(s,v) - \sum_{v \in V_2} f(v,s) \\ > &\qquad + \sum_{v \in V_1} f'(s,v) + \sum_{v \in V_2} f'(s,v) - \sum_{v \in V_1} f'(v,s) - \sum_{v \in V_2} f'(v,s) \\ > &= \sum_{v \in V_1} f(s,v) - \sum_{v \in V_2} f(v,s) + \sum_{v \in V_1 \cup V_2} f'(s,v) - \sum_{v \in V_1 \cup V_2} f'(v,s). > \end{aligned} > \tag{26.6} > $$ > In equation (26.6), we can extend all four summations to sum over $V$, since each additional term has value $0$. (Exercise 26.2-1 asks you to prove this formally.) We thus have > $$ > \begin{aligned} > |f \uparrow f'| &= \sum_{v \in V} f(s,v) - \sum_{v \in V} f(v,s) + \sum_{v \in V} f'(s,v) - \sum_{v \in V} f'(v,s) \\ > &= |f| + |f'|. > \end{aligned} > \tag{26.7} > $$ > $\blacksquare$ ### Augmenting paths Given a flow network $G = (V,E)$ and a flow $f$, an **augmenting path** $p$ is a simple path from $s$ to $t$ in the residual network $G_f$. By the definition of the residual network, we may increase the flow on an edge $(u,v)$ of an augmenting path by up to $c_f(u,v)$ without violating the capacity constraint on whichever of $(u,v)$ and $(v,u)$ is in the original flow network $G$. The shaded path in Figure 26.4(b) is an augmenting path. Treating the residual network $G_f$ in the figure as a flow network, we can increase the flow through each edge of this path by up to $4$ units without violating a capacity constraint, since the smallest residual capacity on this path is $c_f(v_2,v_3) = 4$. We call the maximum amount by which we can increase the flow on each edge in an augmenting path $p$ the **residual capacity** of $p$, given by $$c_f(p) = \min\{c_f(u,v) : (u,v) \text{ is on } p\}.$$ The following lemma, whose proof we leave as Exercise 26.2-7, makes the above argument more precise. > **Lemma 26.2.** Let $G = (V,E)$ be a flow network, let $f$ be a flow in $G$, and let $p$ be an augmenting path in $G_f$. Define a function $f_p : V \times V \to \mathbb{R}$ by > $$ > f_p(u,v) = > \begin{cases} > c_f(p) & \text{if } (u,v) \text{ is on } p, \\ > 0 & \text{otherwise.} > \end{cases} > \tag{26.8} > $$ > Then $f_p$ is a flow in $G_f$ with value $|f_p| = c_f(p) > 0$. The following corollary shows that if we augment $f$ by $f_p$, we get another flow in $G$ whose value is closer to the maximum. Figure 26.4(c) shows the result of augmenting the flow $f$ from Figure 26.4(a) by the flow $f_p$ in Figure 26.4(b), and Figure 26.4(d) shows the ensuing residual network. > **Corollary 26.3.** Let $G = (V,E)$ be a flow network, let $f$ be a flow in $G$, and let $p$ be an augmenting path in $G_f$. Let $f_p$ be defined as in equation (26.8), and suppose that we augment $f$ by $f_p$. Then the function $f \uparrow f_p$ is a flow in $G$ with value $|f \uparrow f_p| = |f| + |f_p| > |f|$. > **Proof.** By Lemma 26.2, $f_p$ is a flow in $G_f$ with $|f_p| = c_f(p) > 0$. Lemma 26.1 (with $f' = f_p$) then says that $f \uparrow f_p$ is a flow in $G$ and > $$ > \begin{aligned} > |f \uparrow f_p| &= |f| + |f_p| &&\text{(by Lemma 26.1)} \\ > &> |f| &&\text{(because } |f_p| > 0\text{).} > \end{aligned} > $$ > $\blacksquare$ ### Cuts of flow networks The Ford-Fulkerson method repeatedly augments the flow along augmenting paths until it has found a maximum flow. How do we know that when the algorithm terminates, we have actually found a maximum flow? The max-flow min-cut theorem, which we shall prove shortly, tells us that a flow is maximum if and only if its residual network contains no augmenting path. To prove this theorem, though, we must first explore the notion of a cut of a flow network. A **cut** $(S,T)$ of flow network $G = (V,E)$ is a partition of $V$ into $S$ and $T = V - S$ such that $s \in S$ and $t \in T$. (This definition is similar to the definition of "cut" that we used for minimum spanning trees in Chapter 23, except that here we are cutting a directed graph rather than an undirected graph, and we insist that $s \in S$ and $t \in T$.) If $f$ is a flow, then the **net flow** $f(S,T)$ across the cut $(S,T)$ is defined to be $$f(S,T) = \sum_{u \in S} \sum_{v \in T} f(u,v) - \sum_{u \in S} \sum_{v \in T} f(v,u). \tag{26.9}$$ The **capacity** of the cut $(S,T)$ is $$c(S,T) = \sum_{u \in S} \sum_{v \in T} c(u,v). \tag{26.10}$$ A **minimum cut** of a network is a cut whose capacity is minimum over all cuts of the network. *Figure 26.5: A cut $(S,T)$ in the flow network of Figure 26.1(b), where $S = \{s, v_1, v_2\}$ and $T = \{v_3, v_4, t\}$. The vertices in $S$ are black, and the vertices in $T$ are white. The net flow across $(S,T)$ is $f(S,T) = 19$, and the capacity is $c(S,T) = 26$.* The asymmetry between the definitions of flow and capacity of a cut is intentional and important. For capacity, we count only the capacities of edges going from $S$ to $T$, ignoring edges in the reverse direction. For flow, we consider the flow going from $S$ to $T$ minus the flow going in the reverse direction from $T$ to $S$. The reason for this difference will become clear later in this section. **Example.** Figure 26.5 shows the cut $(\{s, v_1, v_2\}, \{v_3, v_4, t\})$ in the flow network of Figure 26.1(b). The net flow across this cut is

\begin{aligned} f(v_1,v_3) + f(v_2,v_4) - f(v_3,v_2) &= 12 + 11 - 4 \ &= 19, \end{aligned}

an d t h ec a p a c i t y o f t hi sc u t i s

\begin{aligned} c(v_1,v_3) + c(v_2,v_4) &= 12 + 14 \ &= 26. \end{aligned}

\begin{aligned} F_0 &= 0, \ F_1 &= 1, \ F_i &= F_{i-1} + F_{i-2} \quad \text{for } i \ge 2. \end{aligned}

Here is a simple, recursive, serial algorithm to compute the $n$th Fibonacci number: ```text FIB(n) 1 if n \leq 1 2 return n 3 else x = FIB(n - 1) 4 y = FIB(n - 2) 5 return x + y ``` *Figure 27.1: The tree of recursive procedure instances when computing FIB(6).* The root FIB(6) has children FIB(5) and FIB(4); FIB(5) has children FIB(4) and FIB(3), and so on down to leaves FIB(1) and FIB(0). Each instance of FIB with the same argument does the same work to produce the same result, providing an inefficient but interesting way to compute Fibonacci numbers. You would not really want to compute large Fibonacci numbers this way, because this computation does much repeated work. Figure 27.1 shows the tree of recursive procedure instances that are created when computing $F_6$. For example, a call to FIB(6) recursively calls FIB(5) and then FIB(4). But, the call to FIB(5) also results in a call to FIB(4). Both instances of FIB(4) return the same result ($F_4 = 3$). Since the FIB procedure does not memoize, the second call to FIB(4) replicates the work that the first call performs. Let $T(n)$ denote the running time of FIB($n$). Since FIB($n$) contains two recursive calls plus a constant amount of extra work, we obtain the recurrence

T(n) = T(n-1) + T(n-2) + \Theta(1).

This recurrence has solution $T(n) = \Theta(F_n)$, which we can show using the substitution method. For an inductive hypothesis, assume that $T(n) \le aF_n - b$, where $a > 1$ and $b > 0$ are constants. Substituting, we obtain > **Proof (upper bound $T(n) = O(F_n)$).** > > $$ > \begin{aligned} > T(n) &\le (aF_{n-1} - b) + (aF_{n-2} - b) + \Theta(1) && \text{(by the inductive hypothesis for } n-1 \text{ and } n-2) \\ > &= a(F_{n-1} + F_{n-2}) - 2b + \Theta(1) && \text{(collect terms)} \\ > &= aF_n - b - \bigl(b - \Theta(1)\bigr) && \text{(since } F_n = F_{n-1}+F_{n-2}\text{, and } -2b = -b - b) \\ > &\le aF_n - b > \end{aligned} > $$ > > if we choose $b$ large enough to dominate the constant in the $\Theta(1)$, so that $b - \Theta(1) \ge 0$. We can then choose $a$ large enough to satisfy the initial (base-case) conditions. $\blacksquare$ The analytical bound

T(n) = \Theta(\phi^n), \tag{27.1}

T_P \ge T_1 / P. \tag{27.2}

- A $P$-processor ideal parallel computer cannot run any faster than a machine with an unlimited number of processors. Looked at another way, a machine with an unlimited number of processors can emulate a $P$-processor machine by using just $P$ of its processors. Thus, the **span law** follows:

T_P \ge T_\infty. \tag{27.3}

We define the **speedup** of a computation on $P$ processors by the ratio $T_1/T_P$, which says how many times faster the computation is on $P$ processors than on 1 processor. By the work law, we have $T_P \ge T_1/P$, which implies that $T_1/T_P \le P$. Thus, the speedup on $P$ processors can be at most $P$. When the speedup is linear in the number of processors, that is, when $T_1/T_P = \Theta(P)$, the computation exhibits **linear speedup**, and when $T_1/T_P = P$, we have **perfect linear speedup**. The ratio $T_1/T_\infty$ of the work to the span gives the **parallelism** of the multithreaded computation. We can view the parallelism from three perspectives. - As a ratio, the parallelism denotes the average amount of work that can be performed in parallel for each step along the critical path. - As an upper bound, the parallelism gives the maximum possible speedup that can be achieved on any number of processors. - Finally, and perhaps most important, the parallelism provides a limit on the possibility of attaining perfect linear speedup. Specifically, once the number of processors exceeds the parallelism, the computation cannot possibly achieve perfect linear speedup. To see this last point, suppose that $P > T_1/T_\infty$, in which case the span law implies that the speedup satisfies

\frac{T_1}{T_P} \le \frac{T_1}{T_\infty} < P.

Moreover, if the number $P$ of processors in the ideal parallel computer greatly exceeds the parallelism, that is, if $P \gg T_1/T_\infty$, then $T_1/T_P \ll P$, so that the speedup is much less than the number of processors. In other words, the more processors we use beyond the parallelism, the less perfect the speedup. **Example.** Consider the computation P-FIB(4) in Figure 27.2, and assume that each strand takes unit time. Since the work is $T_1 = 17$ and the span is $T_\infty = 8$, the parallelism is

\frac{T_1}{T_\infty} = \frac{17}{8} = 2.125.

Consequently, achieving much more than double the speedup is impossible, no matter how many processors we employ to execute the computation. For larger input sizes, however, we shall see that P-FIB($n$) exhibits substantial parallelism. We define the **(parallel) slackness** of a multithreaded computation executed on an ideal parallel computer with $P$ processors to be the ratio

\frac{T_1/T_\infty}{P} = \frac{T_1}{P,T_\infty},

which is the factor by which the parallelism of the computation exceeds the number of processors in the machine. Thus, if the slackness is less than 1, we cannot hope to achieve perfect linear speedup, because $T_1/(P\,T_\infty) < 1$ and the span law imply that the speedup on $P$ processors satisfies $T_1/T_P \le T_1/T_\infty < P$. Indeed, as the slackness decreases from 1 toward 0, the speedup of the computation diverges further and further from perfect linear speedup. If the slackness is greater than 1, however, the work per processor is the limiting constraint. As we shall see, as the slackness increases from 1, a good scheduler can achieve closer and closer to perfect linear speedup. > **Think about it.** Ten cooks, but the recipe has only a 3-step chain that cannot be reordered and little else. Would hiring an eleventh cook help? Parallelism answers exactly this question. ### Scheduling Good performance depends on more than just minimizing the work and span. The strands must also be scheduled efficiently onto the processors of the parallel machine. Our multithreaded programming model provides no way to specify which strands to execute on which processors. Instead, we rely on the concurrency platform's **scheduler** to map the dynamically unfolding computation to individual processors. In practice, the scheduler maps the strands to static threads, and the operating system schedules the threads on the processors themselves, but this extra level of indirection is unnecessary for our understanding of scheduling. We can just imagine that the concurrency platform's scheduler maps strands to processors directly. A multithreaded scheduler must schedule the computation with no advance knowledge of when strands will be spawned or when they will complete; it must operate **on-line**. Moreover, a good scheduler operates in a distributed fashion, where the threads implementing the scheduler cooperate to load-balance the computation. Provably good on-line, distributed schedulers exist, but analyzing them is complicated. Instead, to keep our analysis simple, we shall investigate an on-line centralized scheduler, which knows the global state of the computation at any given time. In particular, we shall analyze **greedy schedulers**, which assign as many strands to processors as possible in each time step. If at least $P$ strands are ready to execute during a time step, we say that the step is a **complete step**, and a greedy scheduler assigns any $P$ of the ready strands to processors. Otherwise, fewer than $P$ strands are ready to execute, in which case we say that the step is an **incomplete step**, and the scheduler assigns each ready strand to its own processor. From the work law, the best running time we can hope for on $P$ processors is $T_P = T_1/P$, and from the span law the best we can hope for is $T_P = T_\infty$. The following theorem shows that greedy scheduling is provably good in that it achieves the sum of these two lower bounds as an upper bound. **Theorem 27.1.** On an ideal parallel computer with $P$ processors, a greedy scheduler executes a multithreaded computation with work $T_1$ and span $T_\infty$ in time

T_P \le \frac{T_1}{P} + T_\infty. \tag{27.4}

> **Proof.** Every step is either complete or incomplete, so we bound the number of each kind separately. > > **Complete steps.** In each complete step, the $P$ processors together perform a total of $P$ units of work. Suppose for the purpose of contradiction that the number of complete steps is strictly greater than $\lfloor T_1/P \rfloor$. Then the total work of the complete steps is at least > > $$ > \begin{aligned} > P \cdot \bigl(\lfloor T_1/P \rfloor + 1\bigr) &= P\lfloor T_1/P \rfloor + P && \text{(distribute } P\text{)} \\ > &= T_1 - (T_1 \bmod P) + P && \text{(by equation (3.8))} \\ > &> T_1 && \text{(by inequality (3.9), } T_1 \bmod P < P\text{)}. > \end{aligned} > $$ > > Thus, we obtain the contradiction that the $P$ processors would perform more work than the computation requires. We conclude that the number of complete steps is at most $\lfloor T_1/P \rfloor$. > > **Incomplete steps.** Now consider an incomplete step. Let $G$ be the dag representing the entire computation, and without loss of generality, assume that each strand takes unit time. (We can replace each longer strand by a chain of unit-time strands.) Let $G'$ be the subgraph of $G$ that has yet to be executed at the start of the incomplete step, and let $G''$ be the subgraph remaining to be executed after the incomplete step. > > A longest path in a dag must necessarily start at a vertex with in-degree 0. Since an incomplete step of a greedy scheduler executes all strands with in-degree 0 in $G'$ (fewer than $P$ strands are ready, and all of them are assigned), the length of a longest path in $G''$ must be 1 less than the length of a longest path in $G'$. In other words, an incomplete step decreases the span of the unexecuted dag by 1. Hence, the number of incomplete steps is at most $T_\infty$. > > **Conclusion.** Since each step is either complete or incomplete, the total number of steps is at most > > $$ > T_P \le \left\lfloor \frac{T_1}{P} \right\rfloor + T_\infty \le \frac{T_1}{P} + T_\infty. \qquad \blacksquare > $$ The following corollary to Theorem 27.1 shows that a greedy scheduler always performs well. **Corollary 27.2.** The running time $T_P$ of any multithreaded computation scheduled by a greedy scheduler on an ideal parallel computer with $P$ processors is within a factor of 2 of optimal. > **Proof.** Let $T_P^*$ be the running time produced by an optimal scheduler on a machine with $P$ processors, and let $T_1$ and $T_\infty$ be the work and span of the computation, respectively. Since the work and span laws, inequalities (27.2) and (27.3), give us $T_P^* \ge \max(T_1/P,\, T_\infty)$, Theorem 27.1 implies that > > $$ > \begin{aligned} > T_P &\le \frac{T_1}{P} + T_\infty && \text{(by Theorem 27.1)} \\ > &\le 2 \cdot \max\!\left(\frac{T_1}{P},\, T_\infty\right) && \text{(a sum of two numbers is at most twice their maximum)} \\ > &\le 2\,T_P^* && \text{(by the work and span laws)}. \qquad \blacksquare > \end{aligned} > $$ The next corollary shows that, in fact, a greedy scheduler achieves near-perfect linear speedup on any multithreaded computation as the slackness grows. **Corollary 27.3.** Let $T_P$ be the running time of a multithreaded computation produced by a greedy scheduler on an ideal parallel computer with $P$ processors, and let $T_1$ and $T_\infty$ be the work and span of the computation, respectively. Then, if $P \ll T_1/T_\infty$, we have $T_P \approx T_1/P$, or equivalently, a speedup of approximately $P$. > **Proof.** Suppose that $P \ll T_1/T_\infty$. Then we also have $T_\infty \ll T_1/P$ (multiply both sides by $T_\infty/P$), and hence Theorem 27.1 gives us > > $$ > T_P \le \frac{T_1}{P} + T_\infty \approx \frac{T_1}{P}. > $$ > > Since the work law (27.2) dictates that $T_P \ge T_1/P$, we conclude that $T_P \approx T_1/P$, or equivalently, that the speedup is $T_1/T_P \approx P$. $\blacksquare$ The $\ll$ symbol denotes "much less," but how much is "much less"? As a rule of thumb, a slackness of at least 10 (that is, 10 times more parallelism than processors) generally suffices to achieve good speedup. Then, the span term in the greedy bound, inequality (27.4), is less than 10% of the work-per-processor term, which is good enough for most engineering situations. For example, if a computation runs on only 10 or 100 processors, it doesn't make sense to value parallelism of, say, 1,000,000 over parallelism of 10,000, even with the factor of 100 difference. As Problem 27-2 shows, sometimes by reducing extreme parallelism, we can obtain algorithms that are better with respect to other concerns and which still scale up well on reasonable numbers of processors. ### Analyzing multithreaded algorithms We now have all the tools we need to analyze multithreaded algorithms and provide good bounds on their running times on various numbers of processors. Analyzing the work is relatively straightforward, since it amounts to nothing more than analyzing the running time of an ordinary serial algorithm (namely, the serialization of the multithreaded algorithm), which you should already be familiar with, since that is what most of this textbook is about! Analyzing the span is more interesting, but generally no harder once you get the hang of it. We shall investigate the basic ideas using the P-FIB program. *Figure 27.3: The work and span of composed subcomputations. (a) When two subcomputations $A$ and $B$ are joined in series, the work of the composition is the sum of their work, and the span of the composition is the sum of their spans: $T_1(A \cup B) = T_1(A) + T_1(B)$ and $T_\infty(A \cup B) = T_\infty(A) + T_\infty(B)$. (b) When two subcomputations are joined in parallel, the work of the composition remains the sum of their work, but the span of the composition is only the maximum of their spans: $T_1(A \cup B) = T_1(A) + T_1(B)$ and $T_\infty(A \cup B) = \max(T_\infty(A), T_\infty(B))$.* Analyzing the work $T_1(n)$ of P-FIB($n$) poses no hurdles, because we've already done it. The original FIB procedure is essentially the serialization of P-FIB, and hence $T_1(n) = T(n) = \Theta(\phi^n)$ from equation (27.1). Figure 27.3 illustrates how to analyze the span. If two subcomputations are joined in series, their spans add to form the span of their composition, whereas if they are joined in parallel, the span of their composition is the maximum of the spans of the two subcomputations. For P-FIB($n$), the spawned call to P-FIB($n-1$) in line 3 runs in parallel with the call to P-FIB($n-2$) in line 4. Hence, we can express the span of P-FIB($n$) as the recurrence

\begin{aligned} T_\infty(n) &= \max\bigl(T_\infty(n-1),, T_\infty(n-2)\bigr) + \Theta(1) \ &= T_\infty(n-1) + \Theta(1), \end{aligned}

which has solution $T_\infty(n) = \Theta(n)$. (The maximum is $T_\infty(n-1)$ because the span is nondecreasing in $n$. Unrolling the recurrence adds a constant $n$ times.) The parallelism of P-FIB($n$) is

\frac{T_1(n)}{T_\infty(n)} = \Theta!\left(\frac{\phi

which grows dramatically as $n$ gets large. Thus, on even the largest parallel computers, a modest value for $n$ suffices to achieve near perfect linear speedup for P-FIB($n$), because this procedure exhibits considerable parallel slackness. ### Parallel loops Many algorithms contain loops all of whose iterations can operate in parallel. As we shall see, we can parallelize such loops using the **spawn** and **sync** keywords, but it is much more convenient to specify directly that the iterations of such loops can run concurrently. Our pseudocode provides this functionality via the **parallel** concurrency keyword, which precedes the **for** keyword in a **for** loop statement. As an example, consider the problem of multiplying an $n \times n$ matrix $A = (a_{ij})$ by an $n$-vector $x = (x_j)$. The resulting $n$-vector $y = (y_i)$ is given by the equation

y_i = \sum_{j=1}^{n} a_{ij} x_j,

for $i = 1, 2, \ldots, n$. We can perform matrix-vector multiplication by computing all the entries of $y$ in parallel as follows: ```text MAT-VEC(A, x) 1 n = A.rows 2 let y be a new vector of length n 3 parallel for i = 1 to n 4 y_i = 0 5 parallel for i = 1 to n 6 for j = 1 to n 7 y_i = y_i + a_ij x_j 8 return y ``` In this code, the **parallel for** keywords in lines 3 and 5 indicate that the iterations of the respective loops may be run concurrently. A compiler can implement each **parallel for** loop as a divide-and-conquer subroutine using nested parallelism. For example, the **parallel for** loop in lines 5-7 can be implemented with the call MAT-VEC-MAIN-LOOP($A, x, y, n, 1, n$), where the compiler produces the auxiliary subroutine MAT-VEC-MAIN-LOOP as follows: ```text MAT-VEC-MAIN-LOOP(A, x, y, n, i, i') 1 if i == i' 2 for j = 1 to n 3 y_i = y_i + a_ij x_j 4 else mid = ⌊(i + i') / 2⌋ 5 spawn MAT-VEC-MAIN-LOOP(A, x, y, n, i, mid) 6 MAT-VEC-MAIN-LOOP(A, x, y, n, mid + 1, i') 7 sync ``` This code recursively spawns the first half of the iterations of the loop to execute in parallel with the second half of the iterations and then executes a **sync**, thereby creating a binary tree of execution where the leaves are individual loop iterations, as shown in Figure 27.4. *Figure 27.4: A dag representing the computation of MAT-VEC-MAIN-LOOP($A, x, y, 8, 1, 8$). The two numbers within each rounded rectangle give the values of the last two parameters ($i$ and $i'$ in the procedure header) in the invocation (spawn or call) of the procedure. The black circles represent strands corresponding to either the base case or the part of the procedure up to the spawn of MAT-VEC-MAIN-LOOP in line 5; the shaded circles represent strands corresponding to the part of the procedure that calls MAT-VEC-MAIN-LOOP in line 6 up to the sync in line 7, where it suspends until the spawned subroutine in line 5 returns; and the white circles represent strands corresponding to the (negligible) part of the procedure after the sync up to the point where it returns.* The rounded rectangles form a binary tree with root (1,8), children (1,4) and (5,8), then (1,2), (3,4), (5,6), (7,8), and leaves (1,1) through (8,8). **Work of MAT-VEC.** To calculate the work $T_1(n)$ of MAT-VEC on an $n \times n$ matrix, we simply compute the running time of its serialization, which we obtain by replacing the **parallel for** loops with ordinary **for** loops. Thus, we have $T_1(n) = \Theta(n^2)$, because the quadratic running time of the doubly nested loops in lines 5-7 dominates. This analysis seems to ignore the overhead for recursive spawning in implementing the parallel loops, however. In fact, the overhead of recursive spawning does increase the work of a parallel loop compared with that of its serialization, but not asymptotically. To see why, observe that since the tree of recursive procedure instances is a full binary tree, the number of internal nodes is 1 fewer than the number of leaves (see Exercise B.5-3). Each internal node performs constant work to divide the iteration range, and each leaf corresponds to an iteration of the loop, which takes at least constant time ($\Theta(n)$ time in this case). Thus, we can amortize the overhead of recursive spawning against the work of the iterations, contributing at most a constant factor to the overall work. As a practical matter, dynamic-multithreading concurrency platforms sometimes **coarsen** the leaves of the recursion by executing several iterations in a single leaf, either automatically or under programmer control, thereby reducing the overhead of recursive spawning. This reduced overhead comes at the expense of also reducing the parallelism, however, but if the computation has sufficient parallel slackness, near-perfect linear speedup need not be sacrificed. **Span of a parallel loop.** We must also account for the overhead of recursive spawning when analyzing the span of a parallel-loop construct. Since the depth of recursive calling is logarithmic in the number of iterations, for a parallel loop with $n$ iterations in which the $i$th iteration has span $\mathit{iter}_\infty(i)$, the span is

T_\infty(n) = \Theta(\lg n) + \max_{1 \le i \le n} \mathit{iter}_\infty(i).

**Span of MAT-VEC.** For example, for MAT-VEC on an $n \times n$ matrix: - The parallel initialization loop in lines 3-4 has span $\Theta(\lg n)$, because the recursive spawning dominates the constant-time work of each iteration. - The span of the doubly nested loops in lines 5-7 is $\Theta(n)$, because each iteration of the outer **parallel for** loop contains $n$ iterations of the inner (serial) **for** loop. - The span of the remaining code in the procedure is constant. Thus the span is dominated by the doubly nested loops, yielding an overall span of $\Theta(n)$ for the whole procedure. Since the work is $\Theta(n^2)$, the parallelism is

\frac{\Theta(n^2)}{\Theta(n)} = \Theta(n).

(Exercise 27.1-6 asks you to provide an implementation with even more parallelism.) ### Race conditions A multithreaded algorithm is **deterministic** if it always does the same thing on the same input, no matter how the instructions are scheduled on the multicore computer. It is **nondeterministic** if its behavior might vary from run to run. Often, a multithreaded algorithm that is intended to be deterministic fails to be, because it contains a "determinacy race." Race conditions are the bane of concurrency. Famous race bugs include the Therac-25 radiation therapy machine, which killed three people and injured several others, and the North American Blackout of 2003, which left over 50 million people without power. These pernicious bugs are notoriously hard to find. You can run tests in the lab for days without a failure only to discover that your software sporadically crashes in the field. A **determinacy race** occurs when two logically parallel instructions access the same memory location and at least one of the instructions performs a write. The following procedure illustrates a race condition: ```text RACE-EXAMPLE() 1 x = 0 2 parallel for i = 1 to 2 3 x = x + 1 4 print x ``` After initializing $x$ to 0 in line 1, RACE-EXAMPLE creates two parallel strands, each of which increments $x$ in line 3. Although it might seem that RACE-EXAMPLE should always print the value 2 (its serialization certainly does), it could instead print the value 1. Let's see how this anomaly might occur. When a processor increments $x$, the operation is not indivisible, but is composed of a sequence of instructions: 1. Read $x$ from memory into one of the processor's registers. 2. Increment the value in the register. 3. Write the value in the register back into $x$ in memory. Figure 27.5(a) illustrates a computation dag representing the execution of RACE-EXAMPLE, with the strands broken down to individual instructions. Recall that since an ideal parallel computer supports sequential consistency, we can view the parallel execution of a multithreaded algorithm as an interleaving of instructions that respects the dependencies in the dag. Part (b) of the figure shows the values in an execution of the computation that elicits the anomaly. The value $x$ is stored in memory, and $r_1$ and $r_2$ are processor registers. *Figure 27.5: Illustration of the determinacy race in RACE-EXAMPLE. (a) A computation dag showing the dependencies among individual instructions. The processor registers are $r_1$ and $r_2$. Instructions unrelated to the race, such as the implementation of loop control, are omitted. (b) An execution sequence that elicits the bug, showing the values of $x$ in memory and registers $r_1$ and $r_2$ for each step in the execution sequence.* The dag begins with `x = 0`; then two parallel chains, `r1 = x; incr r1; x = r1` and `r2 = x; incr r2; x = r2`; both join into `print x`. The execution sequence in (b), reconstructed from the text, is: | step | instruction | $x$ | $r_1$ | $r_2$ | |:---:|---|:---:|:---:|:---:| | 1 | `x = 0` | 0 | | | | 2 | `r1 = x` | 0 | 0 | | | 3 | `incr r1` | 0 | 1 | | | 4 | `r2 = x` | 0 | 1 | 0 | | 5 | `incr r2` | 0 | 1 | 1 | | 6 | `x = r2` | 1 | 1 | 1 | | 7 | `x = r1` | 1 | 1 | 1 | | 8 | `print x` | 1 | 1 | 1 | **Trace of the anomaly.** - In step 1, one of the processors sets $x$ to 0. - In steps 2 and 3, processor 1 reads $x$ from memory into its register $r_1$ and increments it, producing the value 1 in $r_1$. - At that point, processor 2 comes into the picture, executing instructions 4-6. Processor 2 reads $x$ from memory into register $r_2$; increments it, producing the value 1 in $r_2$; and then stores this value into $x$, setting $x$ to 1. - Now, processor 1 resumes with step 7, storing the value 1 in $r_1$ into $x$, which leaves the value of $x$ unchanged. - Therefore, step 8 prints the value 1, rather than 2, as the serialization would print. We can see what has happened. If the effect of the parallel execution were that processor 1 executed all its instructions before processor 2, the value 2 would be printed. Conversely, if the effect were that processor 2 executed all its instructions before processor 1, the value 2 would still be printed. When the instructions of the two processors execute at the same time, however, it is possible, as in this example execution, that one of the updates to $x$ is lost. Of course, many executions do not elicit the bug. For example, if the execution order were $\langle 1, 2, 3, 7, 4, 5, 6, 8 \rangle$ or $\langle 1, 4, 5, 6, 2, 3, 7, 8 \rangle$, we would get the correct result. That's the problem with determinacy races. Generally, most orderings produce correct results, such as any in which the instructions on the left execute before the instructions on the right, or vice versa. But some orderings generate improper results when the instructions interleave. Consequently, races can be extremely hard to test for. You can run tests for days and never see the bug, only to experience a catastrophic system crash in the field when the outcome is critical. > **Think about it.** Two people both read the balance "100" from a shared ledger, each adds 1, and each writes back "101". Two deposits were made; one vanished. Where in RACE-EXAMPLE is the ledger? Although we can cope with races in a variety of ways, including using mutual-exclusion locks and other methods of synchronization, for our purposes, we shall simply ensure that strands that operate in parallel are **independent**: they have no determinacy races among them. Thus, in a **parallel for** construct, all the iterations should be independent. Between a **spawn** and the corresponding **sync**, the code of the spawned child should be independent of the code of the parent, including code executed by additional spawned or called children. Note that arguments to a spawned child are evaluated in the parent before the actual spawn occurs, and thus the evaluation of arguments to a spawned subroutine is in series with any accesses to those arguments after the spawn. As an example of how easy it is to generate code with races, here is a faulty implementation of multithreaded matrix-vector multiplication that achieves a span of $\Theta(\lg n)$ by parallelizing the inner **for** loop: ```text MAT-VEC-WRONG(A, x) 1 n = A.rows 2 let y be a new vector of length n 3 parallel for i = 1 to n 4 y_i = 0 5 parallel for i = 1 to n 6 parallel for j = 1 to n 7 y_i = y_i + a_ij x_j 8 return y ``` This procedure is, unfortunately, incorrect due to races on updating $y_i$ in line 7, which executes concurrently for all $n$ values of $j$. Exercise 27.1-6 asks you to give a correct implementation with $\Theta(\lg n)$ span. A multithreaded algorithm with races can sometimes be correct. As an example, two parallel threads might store the same value into a shared variable, and it wouldn't matter which stored the value first. Generally, however, we shall consider code with races to be illegal. ### A chess lesson We close this section with a true story that occurred during the development of the world-class multithreaded chess-playing program ⋆Socrates [80], although the timings below have been simplified for exposition. The program was prototyped on a 32-processor computer but was ultimately to run on a supercomputer with 512 processors. At one point, the developers incorporated an optimization into the program that reduced its running time on an important benchmark on the 32-processor machine from $T_{32} = 65$ seconds to $T'_{32} = 40$ seconds. Yet, the developers used the work and span performance measures to conclude that the optimized version, which was faster on 32 processors, would actually be slower than the original version on 512 processors. As a result, they abandoned the "optimization." Here is their analysis. The original version of the program had work $T_1 = 2048$ seconds and span $T_\infty = 1$ second. If we treat inequality (27.4) as an equation, $T_P = T_1/P + T_\infty$, and use it as an approximation to the running time on $P$ processors, we see that indeed

T_{32} = \frac{2048}{32} + 1 = 65.

With the optimization, the work became $T'_1 = 1024$ seconds and the span became $T'_\infty = 8$ seconds. Again using our approximation, we get

T’_{32} = \frac{1024}{32} + 8 = 40.

T h er e l a t i v es p ee d so f t h e tw o v er s i o n ss w i t c h w h e n w ec a l c u l a t e t h er u nnin g t im eso n 512 p r ocessor s, h o w e v er . I n p a r t i c u l a r, w e ha v e

\begin{aligned} T_{512} &= \frac{2048}{512} + 1 = 5 \text{ seconds}, \ T’_{512} &= \frac{1024}{512} + 8 = 10 \text{ seconds}. \end{aligned}

The optimization that sped up the program on 32 processors would have made the program twice as slow on 512 processors! The optimized version's span of 8, which was not the dominant term in the running time on 32 processors, became the dominant term on 512 processors, nullifying the advantage from using more processors. The moral of the story is that work and span can provide a better means of extrapolating performance than can measured running times. ### Exercises 27.1 **27.1-1** Suppose that we spawn P-FIB($n-2$) in line 4 of P-FIB, rather than calling it as is done in the code. What is the impact on the asymptotic work, span, and parallelism? **27.1-2** Draw the computation dag that results from executing P-FIB(5). Assuming that each strand in the computation takes unit time, what are the work, span, and parallelism of the computation? Show how to schedule the dag on 3 processors using greedy scheduling by labeling each strand with the time step in which it is executed. **27.1-3** Prove that a greedy scheduler achieves the following time bound, which is slightly stronger than the bound proven in Theorem 27.1:

T_P \le \frac{T_1 - T_\infty}{P} + T_\infty. \tag{27.5}

**27.1-4** Construct a computation dag for which one execution of a greedy scheduler can take nearly twice the time of another execution of a greedy scheduler on the same number of processors. Describe how the two executions would proceed. **27.1-5** Professor Karan measures her deterministic multithreaded algorithm on 4, 10, and 64 processors of an ideal parallel computer using a greedy scheduler. She claims that the three runs yielded $T_4 = 80$ seconds, $T_{10} = 42$ seconds, and $T_{64} = 10$ seconds. Argue that the professor is either lying or incompetent. (*Hint:* Use the work law (27.2), the span law (27.3), and inequality (27.5) from Exercise 27.1-3.) **27.1-6** Give a multithreaded algorithm to multiply an $n \times n$ matrix by an $n$-vector that achieves $\Theta(n^2 / \lg n)$ parallelism while maintaining $\Theta(n^2)$ work. **27.1-7** Consider the following multithreaded pseudocode for transposing an $n \times n$ matrix $A$ in place: ```text P-TRANSPOSE(A) 1 n = A.rows 2 parallel for j = 2 to n 3 parallel for i = 1 to j - 1 4 exchange a_ij with a_ji ``` Analyze the work, span, and parallelism of this algorithm. **27.1-8** Suppose that we replace the **parallel for** loop in line 3 of P-TRANSPOSE (see Exercise 27.1-7) with an ordinary **for** loop. Analyze the work, span, and parallelism of the resulting algorithm. **27.1-9** For how many processors do the two versions of the chess program run equally fast, assuming that $T_P = T_1/P + T_\infty$? ## 27.2 Multithreaded matrix multiplication In this section, we examine how to multithread matrix multiplication, a problem whose serial running time we studied in Section 4.2. We'll look at multithreaded algorithms based on the standard triply nested loop, as well as divide-and-conquer algorithms. > **Big picture.** Matrix multiplication is a friendly test bed for parallelism: every entry of the product is an independent job. The real question is how much of that independence we can expose without creating races, and what it costs in span. ### Multithreaded matrix multiplication The first algorithm we study is the straightforward algorithm based on parallelizing the loops in the procedure SQUARE-MATRIX-MULTIPLY on page 75: ```text P-SQUARE-MATRIX-MULTIPLY(A, B) 1 n = A.rows 2 let C be a new n × n matrix 3 parallel for i = 1 to n 4 parallel for j = 1 to n 5 c_ij = 0 6 for k = 1 to n 7 c_ij = c_ij + a_ik · b_kj 8 return C ``` > **Analysis.** > > - **Work.** The serialization of the algorithm is the same as SQUARE-MATRIX-MULTIPLY, so $T_1(n) = \Theta(n^3)$, the same as the running time of SQUARE-MATRIX-MULTIPLY. > - **Span.** A longest path goes down the recursion tree of the **parallel for** loop in line 3, then down the recursion tree of the **parallel for** loop in line 4, and then executes all $n$ iterations of the ordinary **for** loop in line 6. Hence > > $$ > \begin{aligned} > T_\infty(n) &= \Theta(\lg n) + \Theta(\lg n) + \Theta(n) \quad \text{(outer tree, inner tree, serial loop)} \\ > &= \Theta(n) \quad \text{(the linear term dominates)}. > \end{aligned} > $$ > > - **Parallelism.** > > $$ > \frac{T_1(n)}{T_\infty(n)} = \frac{\Theta(n^3)}{\Theta(n)} = \Theta(n^2). > $$ Exercise 27.2-3 asks you to parallelize the inner loop to obtain a parallelism of $\Theta(n^3 / \lg n)$, which you cannot do straightforwardly using **parallel for**, because you would create races. ### A divide-and-conquer multithreaded algorithm for matrix multiplication As we learned in Section 4.2, we can multiply $n \times n$ matrices serially in time $\Theta(n^{\lg 7}) = O(n^{2.81})$ using Strassen's divide-and-conquer strategy, which motivates us to look at multithreading such an algorithm. We begin, as we did in Section 4.2, with multithreading a simpler divide-and-conquer algorithm. Recall from page 77 that the SQUARE-MATRIX-MULTIPLY-RECURSIVE procedure, which multiplies two $n \times n$ matrices $A$ and $B$ to produce the $n \times n$ matrix $C$, relies on partitioning each of the three matrices into four $n/2 \times n/2$ submatrices:

A = \begin{pmatrix} A_{11} & A_{12} \ A_{21} & A_{22} \end{pmatrix}, \quad B = \begin{pmatrix} B_{11} & B_{12} \ B_{21} & B_{22} \end{pmatrix}, \quad C = \begin{pmatrix} C_{11} & C_{12} \ C_{21} & C_{22} \end{pmatrix}.

T h e n, w ec an w r i t e t h e ma t r i x p r o d u c t a s

\begin{pmatrix} C_{11} & C_{12} \ C_{21} & C_{22} \end{pmatrix} = \begin{pmatrix} A_{11} & A_{12} \ A_{21} & A_{22} \end{pmatrix} \begin{pmatrix} B_{11} & B_{12} \ B_{21} & B_{22} \end{pmatrix} = \begin{pmatrix} A_{11}B_{11} & A_{11}B_{12} \ A_{21}B_{11} & A_{21}B_{12} \end{pmatrix}

\begin{pmatrix} A_{12}B_{21} & A_{12}B_{22} \ A_{22}B_{21} & A_{22}B_{22} \end{pmatrix}. \tag{27.6}

Thus, to multiply two $n \times n$ matrices, we perform eight multiplications of $n/2 \times n/2$ matrices and one addition of $n \times n$ matrices. The following pseudocode implements this divide-and-conquer strategy using nested parallelism. Unlike the SQUARE-MATRIX-MULTIPLY-RECURSIVE procedure on which it is based, P-MATRIX-MULTIPLY-RECURSIVE takes the output matrix as a parameter to avoid allocating matrices unnecessarily. ```text P-MATRIX-MULTIPLY-RECURSIVE(C, A, B) 1 n = A.rows 2 if n == 1 3 c_11 = a_11 b_11 4 else let T be a new n × n matrix 5 partition A, B, C, and T into n/2 × n/2 submatrices A11, A12, A21, A22; B11, B12, B21, B22; C11, C12, C21, C22; and T11, T12, T21, T22; respectively 6 spawn P-MATRIX-MULTIPLY-RECURSIVE(C11, A11, B11) 7 spawn P-MATRIX-MULTIPLY-RECURSIVE(C12, A11, B12) 8 spawn P-MATRIX-MULTIPLY-RECURSIVE(C21, A21, B11) 9 spawn P-MATRIX-MULTIPLY-RECURSIVE(C22, A21, B12) 10 spawn P-MATRIX-MULTIPLY-RECURSIVE(T11, A12, B21) 11 spawn P-MATRIX-MULTIPLY-RECURSIVE(T12, A12, B22) 12 spawn P-MATRIX-MULTIPLY-RECURSIVE(T21, A22, B21) 13 P-MATRIX-MULTIPLY-RECURSIVE(T22, A22, B22) 14 sync 15 parallel for i = 1 to n 16 parallel for j = 1 to n 17 c_ij = c_ij + t_ij ``` Line 3 handles the base case, where we are multiplying $1 \times 1$ matrices. We handle the recursive case in lines 4-17. We allocate a temporary matrix $T$ in line 4, and line 5 partitions each of the matrices $A$, $B$, $C$, and $T$ into $n/2 \times n/2$ submatrices. (As with SQUARE-MATRIX-MULTIPLY-RECURSIVE on page 77, we gloss over the minor issue of how to use index calculations to represent submatrix sections of a matrix.) - The recursive call in line 6 sets the submatrix $C_{11}$ to the submatrix product $A_{11}B_{11}$, so that $C_{11}$ equals the first of the two terms that form its sum in equation (27.6). - Similarly, lines 7-9 set $C_{12}$, $C_{21}$, and $C_{22}$ to the first of the two terms that form their respective sums. - Line 10 sets $T_{11}$ to the submatrix product $A_{12}B_{21}$, so that $T_{11}$ equals the second of the two terms that form $C_{11}$'s sum. - Lines 11-13 set $T_{12}$, $T_{21}$, and $T_{22}$ to the second of the two terms that form the sums of $C_{12}$, $C_{21}$, and $C_{22}$, respectively. The first seven recursive calls are spawned, and the last one runs in the main strand. The **sync** statement in line 14 ensures that all the submatrix products in lines 6-13 have been computed, after which we add the products from $T$ into $C$ using the doubly nested **parallel for** loops in lines 15-17. > **Proof (work of P-MATRIX-MULTIPLY-RECURSIVE).** Let $M_1(n)$ be the work on $n \times n$ matrices. We echo the serial analysis of its progenitor SQUARE-MATRIX-MULTIPLY-RECURSIVE. > > - In the recursive case we partition in $\Theta(1)$ time (index calculations), perform eight recursive multiplications of $n/2 \times n/2$ matrices, and finish with $\Theta(n^2)$ work for adding two $n \times n$ matrices (the doubly nested loop has $n^2$ constant-time iterations, plus lower-order spawning overhead). > - Hence > > $$ > M_1(n) = 8 M_1(n/2) + \Theta(n^2). > $$ > > - Compare $f(n) = \Theta(n^2)$ with $n^{\log_2 8} = n^3$. Since $f(n) = O(n^{3-\epsilon})$ for $\epsilon = 1$, case 1 of the master theorem applies: > > $$ > \begin{aligned} > M_1(n) &= \Theta(n^{\log_2 8}) \quad \text{(case 1 of the master theorem)} \\ > &= \Theta(n^3). > \end{aligned} > $$ In other words, the work of our multithreaded algorithm is asymptotically the same as the running time of the procedure SQUARE-MATRIX-MULTIPLY in Section 4.2, with its triply nested loops. > **Proof (span of P-MATRIX-MULTIPLY-RECURSIVE).** Let $M_\infty(n)$ be the span. > > - The span for partitioning is $\Theta(1)$, which is dominated by the $\Theta(\lg n)$ span of the doubly nested **parallel for** loops in lines 15-17. > - Because the eight recursive calls all run in parallel on matrices of the same size, the maximum span over the recursive calls is just the span of any one, namely $M_\infty(n/2)$. > - Hence the recurrence for the span is > > $$ > M_\infty(n) = M_\infty(n/2) + \Theta(\lg n). \tag{27.7} > $$ > > - This recurrence does not fall under any of the cases of the master theorem, but it does meet the condition of Exercise 4.6-2. By Exercise 4.6-2, the solution is $M_\infty(n) = \Theta(\lg^2 n)$. > - Sanity check by unrolling: with $\Theta(\lg n)$ replaced by $c\lg n$, the $i$th level contributes $c\lg(n/2^i) = c(\lg n - i)$, so > > $$ > \begin{aligned} > M_\infty(n) &\approx \sum_{i=0}^{\lg n} c(\lg n - i) \quad \text{(sum the contribution of every level)} \\ > &= c\,\frac{\lg n\,(\lg n + 1)}{2} \quad \text{(arithmetic series)} \\ > &= \Theta(\lg^2 n). > \end{aligned} > $$ Now that we know the work and span of P-MATRIX-MULTIPLY-RECURSIVE, we can compute its parallelism as

\frac{M_1(n)}{M_\infty(n)} = \Theta!\left(\frac{n^3}{\lg^2 n}\right),

PM_\infty(n) = PM_\infty(3n/4) + \Theta(\lg n). \tag{27.8}

(For the base case, the span is $\Theta(1)$, since lines 1-8 execute in constant time.) This recurrence does not fall under any of the cases of the master theorem, but it meets the condition of Exercise 4.6-2. Therefore, the solution to recurrence (27.8) is $PM_\infty(n) = \Theta(\lg^2 n)$. > **Sanity check.** Unrolling, the argument shrinks by a factor $3/4$ each level, so there are $\log_{4/3} n = \Theta(\lg n)$ levels, each costing at most $c\lg n$. Thus the span is at most $\Theta(\lg n)\cdot c\lg n = O(\lg^2 n)$, and since at least $\Theta(\lg n)$ of the levels cost $\Omega(\lg n)$ each, it is also $\Omega(\lg^2 n)$. **Work.** We now analyze the work $PM_1(n)$ of P-MERGE on $n$ elements, which turns out to be $\Theta(n)$. Since each of the $n$ elements must be copied from array $T$ to array $A$, we have $PM_1(n) = \Omega(n)$. Thus, it remains only to show that $PM_1(n) = O(n)$. We shall first derive a recurrence for the worst-case work. The binary search in line 10 costs $\Theta(\lg n)$ in the worst case, which dominates the other work outside of the recursive calls. For the recursive calls, observe that although the recursive calls in lines 13 and 14 might merge different numbers of elements, together the two recursive calls merge at most $n$ elements (actually $n - 1$ elements, since $T[q_1]$ does not participate in either recursive call). Moreover, as we saw in analyzing the span, a recursive call operates on at most $3n/4$ elements. We therefore obtain the recurrence

PM_1(n) = PM_1(\alpha n) + PM_1((1 - \alpha) n) + \Theta(\lg n), \tag{27.9}

where $\alpha$ lies in the range $1/4 \le \alpha \le 3/4$, and where we understand that the actual value of $\alpha$ may vary for each level of recursion. > **Proof (recurrence (27.9) has solution $PM_1(n) = O(n)$, by substitution).** Assume that $PM_1(n) \le c_1 n - c_2 \lg n$ for some positive constants $c_1$ and $c_2$. Substituting gives us > > $$ > \begin{aligned} > PM_1(n) &\le (c_1 \alpha n - c_2 \lg(\alpha n)) + (c_1 (1-\alpha) n - c_2 \lg((1-\alpha) n)) + \Theta(\lg n) \quad \text{(inductive hypothesis on both subproblems)} \\ > &= c_1 (\alpha + (1-\alpha)) n - c_2 \big(\lg(\alpha n) + \lg((1-\alpha) n)\big) + \Theta(\lg n) \quad \text{(collect terms)} \\ > &= c_1 n - c_2 \big(\lg \alpha + \lg n + \lg(1-\alpha) + \lg n\big) + \Theta(\lg n) \quad \text{(since } \lg(ab) = \lg a + \lg b\text{)} \\ > &= c_1 n - c_2 \lg n - \big(c_2(\lg n + \lg(\alpha(1-\alpha))) - \Theta(\lg n)\big) \quad \text{(pull out one } c_2\lg n\text{)} \\ > &\le c_1 n - c_2 \lg n \quad \text{(see below)}. > \end{aligned} > $$ > > - The last step holds because we can pick $c_2$ large enough that $c_2(\lg n + \lg(\alpha(1-\alpha)))$ dominates the $\Theta(\lg n)$ term. This is possible: for $1/4 \le \alpha \le 3/4$, $\alpha(1-\alpha) \ge 3/16$, so $\lg(\alpha(1-\alpha))$ is a bounded negative constant, and $\lg n$ eventually exceeds it. > - Furthermore, we can choose $c_1$ large enough to satisfy the base conditions of the recurrence. > - Since $PM_1(n) = O(n)$ and $PM_1(n) = \Omega(n)$, we have $PM_1(n) = \Theta(n)$. **Parallelism.** The parallelism of P-MERGE is

\frac{PM_1(n)}{PM_\infty(n)} = \Theta!\left(\frac{n}{\lg^2 n}\right).

### Multithreaded merge sort Now that we have a nicely parallelized multithreaded merging procedure, we can incorporate it into a multithreaded merge sort. This version of merge sort is similar to the MERGE-SORT' procedure we saw earlier, but unlike MERGE-SORT', it takes as an argument an output subarray $B$, which will hold the sorted result. In particular, the call P-MERGE-SORT($A, p, r, B, s$) sorts the elements in $A[p..r]$ and stores them in $B[s..s + r - p]$. ```text P-MERGE-SORT(A, p, r, B, s) 1 n = r - p + 1 2 if n == 1 3 B[s] = A[p] 4 else let T[1..n] be a new array 5 q = ⌊(p + r) / 2⌋ 6 q' = q - p + 1 7 spawn P-MERGE-SORT(A, p, q, T, 1) 8 P-MERGE-SORT(A, q + 1, r, T, q' + 1) 9 sync 10 P-MERGE(T, 1, q', q' + 1, n, B, s) ``` After line 1 computes the number $n$ of elements in the input subarray $A[p..r]$, lines 2-3 handle the base case when the array has only 1 element. Lines 4-6 set up for the recursive spawn in line 7 and call in line 8. In particular, line 4 allocates a temporary array $T$ with $n$ elements to store the results of the recursive merge sorting. Line 5 calculates the index $q$ of $A[p..r]$ to divide the elements into the two subarrays $A[p..q]$ and $A[q+1..r]$ that will be sorted recursively, and line 6 goes on to compute the number $q'$ of elements in the first subarray $A[p..q]$, which line 8 uses to determine the starting index in $T$ of where to store the sorted result of $A[q+1..r]$. At that point, the spawn and recursive call are made, followed by the **sync** in line 9, which forces the procedure to wait until the spawned procedure is done. Finally, line 10 calls P-MERGE to merge the sorted subarrays, now in $T[1..q']$ and $T[q'+1..n]$, into the output subarray $B[s..s+r-p]$. ### Analysis of multithreaded merge sort **Work.** We start by analyzing the work $PMS_1(n)$ of P-MERGE-SORT, which is considerably easier than analyzing the work of P-MERGE. Indeed, the work is given by the recurrence

\begin{aligned} PMS_1(n) &= 2,PMS_1(n/2) + PM_1(n) \ &= 2,PMS_1(n/2) + \Theta(n). \end{aligned}

This recurrence is the same as the recurrence (4.4) for ordinary MERGE-SORT from Section 2.3.1 and has solution $PMS_1(n) = \Theta(n \lg n)$ by case 2 of the master theorem. **Span.** We now derive and analyze a recurrence for the worst-case span $PMS_\infty(n)$. Because the two recursive calls to P-MERGE-SORT on lines 7 and 8 operate logically in parallel, we can ignore one of them, obtaining the recurrence

\begin{aligned} PMS_\infty(n) &= PMS_\infty(n/2) + PM_\infty(n) \ &= PMS_\infty(n/2) + \Theta(\lg^2 n). \end{aligned} \tag{27.10}

As for recurrence (27.8), the master theorem does not apply to recurrence (27.10), but Exercise 4.6-2 does. The solution is $PMS_\infty(n) = \Theta(\lg^3 n)$, and so the span of P-MERGE-SORT is $\Theta(\lg^3 n)$. > **Sanity check.** There are $\lg n$ levels of recursion along the critical path. At the level with subproblem size $m$, the merge contributes $\Theta(\lg^2 m) \le \Theta(\lg^2 n)$. Summing gives $O(\lg n \cdot \lg^2 n) = O(\lg^3 n)$. Half of the levels have $m \ge \sqrt n$, so each contributes $\Omega(\lg^2 n)$, giving the matching lower bound. **Parallelism.** Parallel merging gives P-MERGE-SORT a significant parallelism advantage over MERGE-SORT'. Recall that the parallelism of MERGE-SORT', which calls the serial MERGE procedure, is only $\Theta(\lg n)$. For P-MERGE-SORT, the parallelism is

\begin{aligned} \frac{PMS_1(n)}{PMS_\infty(n)} &= \frac{\Theta(n \lg n)}{\Theta(\lg^3 n)} \ &= \Theta!\left(\frac{n}{\lg^2 n}\right), \end{aligned}

which is much better both in theory and in practice. A good implementation in practice would sacrifice some parallelism by coarsening the base case in order to reduce the constants hidden by the asymptotic notation. The straightforward way to coarsen the base case is to switch to an ordinary serial sort, perhaps quicksort, when the size of the array is sufficiently small. ### Exercises 27.3-1 Explain how to coarsen the base case of P-MERGE. ### Exercises 27.3-2 Instead of finding a median element in the larger subarray, as P-MERGE does, consider a variant that finds a median element of all the elements in the two sorted subarrays using the result of Exercise 9.3-8. Give pseudocode for an efficient multithreaded merging procedure that uses this median-finding procedure. Analyze your algorithm. ### Exercises 27.3-3 Give an efficient multithreaded algorithm for partitioning an array around a pivot, as is done by the PARTITION procedure on page 171. You need not partition the array in place. Make your algorithm as parallel as possible. Analyze your algorithm. (*Hint:* You may need an auxiliary array and may need to make more than one pass over the input elements.) ### Exercises 27.3-4 Give a multithreaded version of RECURSIVE-FFT on page 911. Make your implementation as parallel as possible. Analyze your algorithm. ### Exercises 27.3-5 $\star$ Give a multithreaded version of RANDOMIZED-SELECT on page 216. Make your implementation as parallel as possible. Analyze your algorithm. (*Hint:* Use the partitioning algorithm from Exercise 27.3-3.) ### Exercises 27.3-6 $\star$ Show how to multithread SELECT from Section 9.3. Make your implementation as parallel as possible. Analyze your algorithm. ## Problems ### 27-1 Implementing parallel loops using nested parallelism Consider the following multithreaded algorithm for performing pairwise addition on $n$-element arrays $A[1..n]$ and $B[1..n]$, storing the sums in $C[1..n]$: ```text SUM-ARRAYS(A, B, C) 1 parallel for i = 1 to A.length 2 C[i] = A[i] + B[i] ``` **a.** Rewrite the parallel loop in SUM-ARRAYS using nested parallelism (**spawn** and **sync**) in the manner of MAT-VEC-MAIN-LOOP. Analyze the parallelism of your implementation. Consider the following alternative implementation of the parallel loop, which contains a value *grain-size* to be specified: ```text SUM-ARRAYS'(A, B, C) 1 n = A.length 2 grain-size = ? // to be determined 3 r = ⌈n / grain-size⌉ 4 for k = 0 to r - 1 5 spawn ADD-SUBARRAY(A, B, C, k · grain-size + 1, min((k + 1) · grain-size, n)) 6 sync ``` ```text ADD-SUBARRAY(A, B, C, i, j) 1 for k = i to j 2 C[k] = A[k] + B[k] ``` **b.** Suppose that we set $\textit{grain-size} = 1$. What is the parallelism of this implementation? **c.** Give a formula for the span of SUM-ARRAYS' in terms of $n$ and *grain-size*. Derive the best value for *grain-size* to maximize parallelism. ### 27-2 Saving temporary space in matrix multiplication The P-MATRIX-MULTIPLY-RECURSIVE procedure has the disadvantage that it must allocate a temporary matrix $T$ of size $n \times n$, which can adversely affect the constants hidden by the $\Theta$-notation. The P-MATRIX-MULTIPLY-RECURSIVE procedure does have high parallelism, however. For example, ignoring the constants in the $\Theta$-notation, the parallelism for multiplying $1000 \times 1000$ matrices comes to approximately $1000^3/10^2 = 10^7$, since $\lg 1000 \approx 10$. Most parallel computers have far fewer than 10 million processors. **a.** Describe a recursive multithreaded algorithm that eliminates the need for the temporary matrix $T$ at the cost of increasing the span to $\Theta(n)$. (*Hint:* Compute $C = C + AB$ following the general strategy of P-MATRIX-MULTIPLY-RECURSIVE, but initialize $C$ in parallel and insert a **sync** in a judiciously chosen location.) **b.** Give and solve recurrences for the work and span of your implementation. **c.** Analyze the parallelism of your implementation. Ignoring the constants in the $\Theta$-notation, estimate the parallelism on $1000 \times 1000$ matrices. Compare with the parallelism of P-MATRIX-MULTIPLY-RECURSIVE. ### 27-3 Multithreaded matrix algorithms **a.** Parallelize the LU-DECOMPOSITION procedure on page 821 by giving pseudocode for a multithreaded version of this algorithm. Make your implementation as parallel as possible, and analyze its work, span, and parallelism. **b.** Do the same for LUP-DECOMPOSITION on page 824. **c.** Do the same for LUP-SOLVE on page 817. **d.** Do the same for a multithreaded algorithm based on equation (28.13) for inverting a symmetric positive-definite matrix. ### 27-4 Multithreading reductions and prefix computations A $\otimes$-*reduction* of an array $x[1..n]$, where $\otimes$ is an associative operator, is the value

y = x[1] \otimes x[2] \otimes \cdots \otimes x[n].

The following procedure computes the $\otimes$-reduction of a subarray $x[i..j]$ serially. ```text REDUCE(x, i, j) 1 y = x[i] 2 for k = i + 1 to j 3 y = y ⊗ x[k] 4 return y ``` **a.** Use nested parallelism to implement a multithreaded algorithm P-REDUCE, which performs the same function with $\Theta(n)$ work and $\Theta(\lg n)$ span. Analyze your algorithm. A related problem is that of computing a $\otimes$-*prefix computation*, sometimes called a $\otimes$-*scan*, on an array $x[1..n]$, where $\otimes$ is once again an associative operator. The $\otimes$-scan produces the array $y[1..n]$ given by

\begin{aligned} y[1] &= x[1], \ y[2] &= x[1] \otimes x[2], \ y[3] &= x[1] \otimes x[2] \otimes x[3], \ &;;\vdots \ y[n] &= x[1] \otimes x[2] \otimes x[3] \otimes \cdots \otimes x[n], \end{aligned}

that is, all prefixes of the array $x$ "summed" using the $\otimes$ operator. The following serial procedure SCAN performs a $\otimes$-prefix computation: ```text SCAN(x) 1 n = x.length 2 let y[1..n] be a new array 3 y[1] = x[1] 4 for i = 2 to n 5 y[i] = y[i - 1] ⊗ x[i] 6 return y ``` Unfortunately, multithreading SCAN is not straightforward. For example, changing the **for** loop to a **parallel for** loop would create races, since each iteration of the loop body depends on the previous iteration. The following procedure P-SCAN-1 performs the $\otimes$-prefix computation in parallel, albeit inefficiently: ```text P-SCAN-1(x) 1 n = x.length 2 let y[1..n] be a new array 3 P-SCAN-1-AUX(x, y, 1, n) 4 return y ``` ```text P-SCAN-1-AUX(x, y, i, j) 1 parallel for l = i to j 2 y[l] = P-REDUCE(x, 1, l) ``` **b.** Analyze the work, span, and parallelism of P-SCAN-1. By using nested parallelism, we can obtain a more efficient $\otimes$-prefix computation: ```text P-SCAN-2(x) 1 n = x.length 2 let y[1..n] be a new array 3 P-SCAN-2-AUX(x, y, 1, n) 4 return y ``` ```text P-SCAN-2-AUX(x, y, i, j) 1 if i == j 2 y[i] = x[i] 3 else k = ⌊(i + j) / 2⌋ 4 spawn P-SCAN-2-AUX(x, y, i, k) 5 P-SCAN-2-AUX(x, y, k + 1, j) 6 sync 7 parallel for l = k + 1 to j 8 y[l] = y[k] ⊗ y[l] ``` **c.** Argue that P-SCAN-2 is correct, and analyze its work, span, and parallelism. We can improve on both P-SCAN-1 and P-SCAN-2 by performing the $\otimes$-prefix computation in two distinct passes over the data. On the first pass, we gather the terms for various contiguous subarrays of $x$ into a temporary array $t$, and on the second pass we use the terms in $t$ to compute the final result $y$. The following pseudocode implements this strategy, but certain expressions have been omitted: ```text P-SCAN-3(x) 1 n = x.length 2 let y[1..n] and t[1..n] be new arrays 3 y[1] = x[1] 4 if n > 1 5 P-SCAN-UP(x, t, 2, n) 6 P-SCAN-DOWN(x[1], x, t, y, 2, n) 7 return y ``` ```text P-SCAN-UP(x, t, i, j) 1 if i == j 2 return x[i] 3 else 4 k = ⌊(i + j) / 2⌋ 5 t[k] = spawn P-SCAN-UP(x, t, i, k) 6 right = P-SCAN-UP(x, t, k + 1, j) 7 sync 8 return ____ // fill in the blank ``` ```text P-SCAN-DOWN(v, x, t, y, i, j) 1 if i == j 2 y[i] = v ⊗ x[i] 3 else 4 k = ⌊(i + j) / 2⌋ 5 spawn P-SCAN-DOWN(____, x, t, y, i, k) // fill in the blank 6 P-SCAN-DOWN(____, x, t, y, k + 1, j) // fill in the blank 7 sync ``` **d.** Fill in the three missing expressions in line 8 of P-SCAN-UP and lines 5 and 6 of P-SCAN-DOWN. Argue that with expressions you supplied, P-SCAN-3 is correct. (*Hint:* Prove that the value $v$ passed to P-SCAN-DOWN($v, x, t, y, i, j$) satisfies $v = x[1] \otimes x[2] \otimes \cdots \otimes x[i-1]$.) **e.** Analyze the work, span, and parallelism of P-SCAN-3. ### 27-5 Multithreading a simple stencil calculation Computational science is replete with algorithms that require the entries of an array to be filled in with values that depend on the values of certain already computed neighboring entries, along with other information that does not change over the course of the computation. The pattern of neighboring entries does not change during the computation and is called a *stencil*. For example, Section 15.4 presents a stencil algorithm to compute a longest common subsequence, where the value in entry $c[i, j]$ depends only on the values in $c[i-1, j]$, $c[i, j-1]$, and $c[i-1, j-1]$, as well as the elements $x_i$ and $y_j$ within the two sequences given as inputs. The input sequences are fixed, but the algorithm fills in the two-dimensional array $c$ so that it computes entry $c[i, j]$ after computing all three entries $c[i-1, j]$, $c[i, j-1]$, and $c[i-1, j-1]$. In this problem, we examine how to use nested parallelism to multithread a simple stencil calculation on an $n \times n$ array $A$ in which, of the values in $A$, the value placed into entry $A[i, j]$ depends only on values in $A[i', j']$, where $i' \le i$ and $j' \le j$ (and of course, $i' \ne i$ or $j' \ne j$). In other words, the value in an entry depends only on values in entries that are above it and/or to its left, along with static information outside of the array. Furthermore, we assume throughout this problem that once we have filled in the entries upon which $A[i, j]$ depends, we can fill in $A[i, j]$ in $O(1)$ time (as in the LCS-LENGTH procedure of Section 15.4). We can partition the $n \times n$ array $A$ into four $n/2 \times n/2$ subarrays as follows:

A = \begin{pmatrix} A_{11} & A_{12} \ A_{21} & A_{22} \end{pmatrix}. \tag{27.11}

Observe now that we can fill in subarray $A_{11}$ recursively, since it does not depend on the entries of the other three subarrays. Once $A_{11}$ is complete, we can continue to fill in $A_{12}$ and $A_{21}$ recursively in parallel, because although they both depend on $A_{11}$, they do not depend on each other. Finally, we can fill in $A_{22}$ recursively. **a.** Give multithreaded pseudocode that performs this simple stencil calculation using a divide-and-conquer algorithm SIMPLE-STENCIL based on the decomposition (27.11) and the discussion above. (Don't worry about the details of the base case, which depends on the specific stencil.) Give and solve recurrences for the work and span of this algorithm in terms of $n$. What is the parallelism? **b.** Modify your solution to part (a) to divide an $n \times n$ array into nine $n/3 \times n/3$ subarrays, again recursing with as much parallelism as possible. Analyze this algorithm. How much more or less parallelism does this algorithm have compared with the algorithm from part (a)? **c.** Generalize your solutions to parts (a) and (b) as follows. Choose an integer $b \ge 2$. Divide an $n \times n$ array into $b^2$ subarrays, each of size $n/b \times n/b$, recursing with as much parallelism as possible. In terms of $n$ and $b$, what are the work, span, and parallelism of your algorithm? Argue that, using this approach, the parallelism must be $o(n)$ for any choice of $b \ge 2$. (*Hint:* For this last argument, show that the exponent of $n$ in the parallelism is strictly less than 1 for any choice of $b \ge 2$.) **d.** Give pseudocode for a multithreaded algorithm for this simple stencil calculation that achieves $\Theta(n/\lg n)$ parallelism. Argue using notions of work and span that the problem, in fact, has $\Theta(n)$ inherent parallelism. As it turns out, the divide-and-conquer nature of our multithreaded pseudocode does not let us achieve this maximal parallelism. ### 27-6 Randomized multithreaded algorithms Just as with ordinary serial algorithms, we sometimes want to implement randomized multithreaded algorithms. This problem explores how to adapt the various performance measures in order to handle the expected behavior of such algorithms. It also asks you to design and analyze a multithreaded algorithm for randomized quicksort. **a.** Explain how to modify the work law (27.2), span law (27.3), and greedy scheduler bound (27.4) to work with expectations when $T_P$, $T_1$, and $T_\infty$ are all random variables. **b.** Consider a randomized multithreaded algorithm for which 1% of the time we have $T_1 = 10^4$ and $T_{10{,}000} = 1$, but for 99% of the time we have $T_1 = T_{10{,}000} = 10^9$. Argue that the speedup of a randomized multithreaded algorithm should be defined as $E[T_1]/E[T_P]$, rather than $E[T_1/T_P]$. **c.** Argue that the parallelism of a randomized multithreaded algorithm should be defined as the ratio $E[T_1]/E[T_\infty]$. **d.** Multithread the RANDOMIZED-QUICKSORT algorithm on page 179 by using nested parallelism. (Do not parallelize RANDOMIZED-PARTITION.) Give the pseudocode for your P-RANDOMIZED-QUICKSORT algorithm. **e.** Analyze your multithreaded algorithm for randomized quicksort. (*Hint:* Review the analysis of RANDOMIZED-SELECT on page 216.) ## Chapter notes Parallel computers, models for parallel computers, and algorithmic models for parallel programming have been around in various forms for years. Prior editions of this book included material on sorting networks and the PRAM (Parallel Random-Access Machine) model. The data-parallel model [48, 168] is another popular algorithmic programming model, which features operations on vectors and matrices as primitives. Graham [149] and Brent [55] showed that there exist schedulers achieving the bound of Theorem 27.1. Eager, Zahorjan, and Lazowska [98] showed that any greedy scheduler achieves this bound and proposed the methodology of using work and span (although not by those names) to analyze parallel algorithms. Blelloch [47] developed an algorithmic programming model based on work and span (which he called the "depth" of the computation) for data-parallel programming. Blumofe and Leiserson [52] gave a distributed scheduling algorithm for dynamic multithreading based on randomized "work-stealing" and showed that it achieves the bound $E[T_P] \le T_1/P + O(T_\infty)$. Arora, Blumofe, and Plaxton [19] and Blelloch, Gibbons, and Matias [49] also provided provably good algorithms for scheduling dynamic multithreaded computations. The multithreaded pseudocode and programming model were heavily influenced by the Cilk [51, 118] project at MIT and the Cilk++ [71] extensions to C++ distributed by Cilk Arts, Inc. Many of the multithreaded algorithms in this chapter appeared in unpublished lecture notes by C. E. Leiserson and H. Prokop and have been implemented in Cilk or Cilk++. The multithreaded merge-sorting algorithm was inspired by an algorithm of Akl [12]. The notion of sequential consistency is due to Lamport [223]. # 28 Matrix Operations > **Big picture.** Imagine a huge web of dials, each one tied to every other dial through a fixed rule, and you must find the one setting of all the dials that satisfies every rule at once. That is a system of linear equations, and matrices are the language that lets us handle the whole web in a single breath. In this chapter we ask: how can we solve such systems quickly and reliably, how is inverting a matrix secretly the same job as multiplying two matrices, and how can the special family of symmetric positive-definite matrices help us draw the best-fitting curve through noisy data? Because operations on matrices lie at the heart of scientific computing, efficient algorithms for working with matrices have many practical applications. This chapter focuses on how to multiply matrices and solve sets of simultaneous linear equations. Appendix D reviews the basics of matrices. Section 28.1 shows how to solve a set of linear equations using LUP decompositions. Then, Section 28.2 explores the close relationship between multiplying and inverting matrices. Finally, Section 28.3 discusses the important class of symmetric positive-definite matrices and shows how we can use them to find a least-squares solution to an overdetermined set of linear equations. One important issue that arises in practice is **numerical stability**. Due to the limited precision of floating-point representations in actual computers, round-off errors in numerical computations may become amplified over the course of a computation, leading to incorrect results; we call such computations **numerically unstable**. Although we shall briefly consider numerical stability on occasion, we do not focus on it in this chapter. We refer you to the excellent book by Golub and Van Loan [144] for a thorough discussion of stability issues. ## 28.1 Solving systems of linear equations Numerous applications need to solve sets of simultaneous linear equations. We can formulate a linear system as a matrix equation in which each matrix or vector element belongs to a field, typically the real numbers $\mathbb{R}$. This section discusses how to solve a system of linear equations using a method called LUP decomposition. We start with a set of linear equations in $n$ unknowns $x_1, x_2, \ldots, x_n$:

\begin{aligned} a_{11}x_1 + a_{12}x_2 + \cdots + a_{1n}x_n &= b_1 ,\ a_{21}x_1 + a_{22}x_2 + \cdots + a_{2n}x_n &= b_2 ,\ &;;\vdots\ a_{n1}x_1 + a_{n2}x_2 + \cdots + a_{nn}x_n &= b_n . \end{aligned} \tag{28.1}

A **solution** to the equations (28.1) is a set of values for $x_1, x_2, \ldots, x_n$ that satisfy all of the equations simultaneously. In this section, we treat only the case in which there are exactly $n$ equations in $n$ unknowns. We can conveniently rewrite equations (28.1) as the matrix-vector equation

\begin{pmatrix} a_{11} & a_{12} & \cdots & a_{1n}\ a_{21} & a_{22} & \cdots & a_{2n}\ \vdots & \vdots & \ddots & \vdots\ a_{n1} & a_{n2} & \cdots & a_{nn} \end{pmatrix} \begin{pmatrix} x_1\ x_2\ \vdots\ x_n \end{pmatrix}

\begin{pmatrix} b_1\ b_2\ \vdots\ b_n \end{pmatrix}

or, equivalently, letting $A = (a_{ij})$, $x = (x_i)$, and $b = (b_i)$, as $$Ax = b . \tag{28.2}$$ If $A$ is nonsingular, it possesses an inverse $A^{-1}$, and $$x = A^{-1}b \tag{28.3}$$ is the solution vector. We can prove that $x$ is the unique solution to equation (28.2) as follows. If there are two solutions, $x$ and $x'$, then $Ax = Ax' = b$ and, letting $I$ denote an identity matrix,

\begin{aligned} x &= Ix\ &= (A^{-1}A)x\ &= A^{-1}(Ax)\ &= A^{-1}(Ax’)\ &= (A^{-1}A)x’\ &= x’ . \end{aligned}

In this section, we shall be concerned predominantly with the case in which $A$ is nonsingular or, equivalently (by Theorem D.1), the rank of $A$ is equal to the number $n$ of unknowns. There are other possibilities, however, which merit a brief discussion. If the number of equations is less than the number $n$ of unknowns, or, more generally, if the rank of $A$ is less than $n$, then the system is **underdetermined**. An underdetermined system typically has infinitely many solutions, although it may have no solutions at all if the equations are inconsistent. If the number of equations exceeds the number $n$ of unknowns, the system is **overdetermined**, and there may not exist any solutions. Section 28.3 addresses the important problem of finding good approximate solutions to overdetermined systems of linear equations. Let us return to our problem of solving the system $Ax = b$ of $n$ equations in $n$ unknowns. We could compute $A^{-1}$ and then, using equation (28.3), multiply $b$ by $A^{-1}$, yielding $x = A^{-1}b$. This approach suffers in practice from numerical instability. Fortunately, another approach, LUP decomposition, is numerically stable and has the further advantage of being faster in practice. ### Overview of LUP decomposition The idea behind LUP decomposition is to find three $n \times n$ matrices $L$, $U$, and $P$ such that $$PA = LU , \tag{28.4}$$ where - $L$ is a unit lower-triangular matrix, - $U$ is an upper-triangular matrix, and - $P$ is a permutation matrix. We call matrices $L$, $U$, and $P$ satisfying equation (28.4) an **LUP decomposition** of the matrix $A$. We shall show that every nonsingular matrix $A$ possesses such a decomposition. Computing an LUP decomposition for the matrix $A$ has the advantage that we can more easily solve linear systems when they are triangular, as is the case for both matrices $L$ and $U$. Once we have found an LUP decomposition for $A$, we can solve equation (28.2), $Ax = b$, by solving only triangular linear systems, as follows. Multiplying both sides of $Ax = b$ by $P$ yields the equivalent equation $PAx = Pb$, which, by Exercise D.1-4, amounts to permuting the equations (28.1). Using our decomposition (28.4), we obtain $$LUx = Pb .$$ We can now solve this equation by solving two triangular linear systems. Let us define $y = Ux$, where $x$ is the desired solution vector. First, we solve the lower-triangular system $$Ly = Pb \tag{28.5}$$ for the unknown vector $y$ by a method called "forward substitution." Having solved for $y$, we then solve the upper-triangular system $$Ux = y \tag{28.6}$$ for the unknown $x$ by a method called "back substitution." Because the permutation matrix $P$ is invertible (Exercise D.2-3), multiplying both sides of equation (28.4) by $P^{-1}$ gives $P^{-1}PA = P^{-1}LU$, so that $$A = P^{-1}LU . \tag{28.7}$$ Hence, the vector $x$ is our solution to $Ax = b$:

\begin{aligned} Ax &= P^{-1}LUx &&\text{(by equation (28.7))}\ &= P^{-1}Ly &&\text{(by equation (28.6))}\ &= P^{-1}Pb &&\text{(by equation (28.5))}\ &= b . \end{aligned}

Our next step is to show how forward and back substitution work and then attack the problem of computing the LUP decomposition itself. *Think about it:* why is a triangular system so pleasant? Because the first (or last) equation has only one unknown, and once you know it, the next equation has only one unknown left, like a row of dominoes. ### Forward and back substitution Forward substitution can solve the lower-triangular system (28.5) in $\Theta(n^2)$ time, given $L$, $P$, and $b$. For convenience, we represent the permutation $P$ compactly by an array $\pi[1..n]$. For $i = 1, 2, \ldots, n$, the entry $\pi[i]$ indicates that $P_{i,\pi[i]} = 1$ and $P_{ij} = 0$ for $j \neq \pi[i]$. Thus, $PA$ has $a_{\pi[i],j}$ in row $i$ and column $j$, and $Pb$ has $b_{\pi[i]}$ as its $i$th element. Since $L$ is unit lower-triangular, we can rewrite equation (28.5) as

\begin{aligned} y_1 &= b_{\pi[1]} ,\ l_{21}y_1 + y_2 &= b_{\pi[2]} ,\ l_{31}y_1 + l_{32}y_2 + y_3 &= b_{\pi[3]} ,\ &;;\vdots\ l_{n1}y_1 + l_{n2}y_2 + l_{n3}y_3 + \cdots + y_n &= b_{\pi[n]} . \end{aligned}

The first equation tells us that $y_1 = b_{\pi[1]}$. Knowing the value of $y_1$, we can substitute it into the second equation, yielding $$y_2 = b_{\pi[2]} - l_{21}y_1 .$$ Now, we can substitute both $y_1$ and $y_2$ into the third equation, obtaining $$y_3 = b_{\pi[3]} - (l_{31}y_1 + l_{32}y_2) .$$ In general, we substitute $y_1, y_2, \ldots, y_{i-1}$ "forward" into the $i$th equation to solve for $y_i$: $$y_i = b_{\pi[i]} - \sum_{j=1}^{i-1} l_{ij}y_j .$$ Having solved for $y$, we solve for $x$ in equation (28.6) using **back substitution**, which is similar to forward substitution. Here, we solve the $n$th equation first and work backward to the first equation. Like forward substitution, this process runs in $\Theta(n^2)$ time. Since $U$ is upper-triangular, we can rewrite the system (28.6) as

\begin{aligned} u_{11}x_1 + u_{12}x_2 + \cdots + u_{1,n-2}x_{n-2} + u_{1,n-1}x_{n-1} + u_{1n}x_n &= y_1 ,\ u_{22}x_2 + \cdots + u_{2,n-2}x_{n-2} + u_{2,n-1}x_{n-1} + u_{2n}x_n &= y_2 ,\ &;;\vdots\ u_{n-2,n-2}x_{n-2} + u_{n-2,n-1}x_{n-1} + u_{n-2,n}x_n &= y_{n-2} ,\ u_{n-1,n-1}x_{n-1} + u_{n-1,n}x_n &= y_{n-1} ,\ u_{n,n}x_n &= y_n . \end{aligned}

Thus, we can solve for $x_n, x_{n-1}, \ldots, x_1$ successively as follows:

\begin{aligned} x_n &= y_n / u_{n,n} ,\ x_{n-1} &= (y_{n-1} - u_{n-1,n}x_n) / u_{n-1,n-1} ,\ x_{n-2} &= \bigl(y_{n-2} - (u_{n-2,n-1}x_{n-1} + u_{n-2,n}x_n)\bigr) / u_{n-2,n-2} ,\ &;;\vdots \end{aligned}

or, in general, $$x_i = \left( y_i - \sum_{j=i+1}^{n} u_{ij}x_j \right) \Big/ u_{ii} .$$ Given $P$, $L$, $U$, and $b$, the procedure LUP-SOLVE solves for $x$ by combining forward and back substitution. The pseudocode assumes that the dimension $n$ appears in the attribute $L.rows$ and that the permutation matrix $P$ is represented by the array $\pi$. ```text LUP-SOLVE(L, U, π, b) 1 n = L.rows 2 let x and y be new vectors of length n 3 for i = 1 to n 4 y_i = b_π[i] - Σ_{j=1}^{i-1} l_ij y_j 5 for i = n downto 1 6 x_i = (y_i - Σ_{j=i+1}^{n} u_ij x_j) / u_ii 7 return x ``` Procedure LUP-SOLVE solves for $y$ using forward substitution in lines 3-4, and then it solves for $x$ using back substitution in lines 5-6. Since the summation within each of the **for** loops includes an implicit loop, the running time is $\Theta(n^2)$. **Example.** As an example of these methods, consider the system of linear equations defined by

\begin{pmatrix} 1 & 2 & 0\ 3 & 4 & 4\ 5 & 6 & 3 \end{pmatrix} \begin{pmatrix} x_1\ x_2\ x_3 \end{pmatrix}

\begin{pmatrix} 3\ 7\ 8 \end{pmatrix},

w h er e

A = \begin{pmatrix} 1 & 2 & 0\ 3 & 4 & 4\ 5 & 6 & 3 \end{pmatrix},\quad b = \begin{pmatrix} 3\ 7\ 8 \end{pmatrix},

and we wish to solve for the unknown $x$. The LUP decomposition is

L = \begin{pmatrix} 1 & 0 & 0\ 0.2 & 1 & 0\ 0.6 & 0.5 & 1 \end{pmatrix},\quad U = \begin{pmatrix} 5 & 6 & 3\ 0 & 0.8 & -0.6\ 0 & 0 & 2.5 \end{pmatrix},\quad P = \begin{pmatrix} 0 & 0 & 1\ 1 & 0 & 0\ 0 & 1 & 0 \end{pmatrix}.

(You might want to verify that $PA = LU$.) Here $\pi = \langle 3, 1, 2 \rangle$, so $Pb = (8, 3, 7)^T$. Using forward substitution, we solve $Ly = Pb$ for $y$:

\begin{pmatrix} 1 & 0 & 0\ 0.2 & 1 & 0\ 0.6 & 0.5 & 1 \end{pmatrix} \begin{pmatrix} y_1\ y_2\ y_3 \end{pmatrix}

\begin{pmatrix} 8\ 3\ 7 \end{pmatrix},

o b t ainin g

y = \begin{pmatrix} 8\ 1.4\ 1.5 \end{pmatrix}

by computing first $y_1$, then $y_2$, and finally $y_3$:

\begin{aligned} y_1 &= 8 ,\ y_2 &= 3 - 0.2 \cdot 8 = 1.4 ,\ y_3 &= 7 - 0.6 \cdot 8 - 0.5 \cdot 1.4 = 7 - 4.8 - 0.7 = 1.5 . \end{aligned}

Using back substitution, we solve $Ux = y$ for $x$:

\begin{pmatrix} 5 & 6 & 3\ 0 & 0.8 & -0.6\ 0 & 0 & 2.5 \end{pmatrix} \begin{pmatrix} x_1\ x_2\ x_3 \end{pmatrix}

\begin{pmatrix} 8\ 1.4\ 1.5 \end{pmatrix},

t h er e b y o b t ainin g t h e d es i r e d an s w er

x = \begin{pmatrix} -1.4\ 2.2\ 0.6 \end{pmatrix}

by computing first $x_3$, then $x_2$, and finally $x_1$:

\begin{aligned} x_3 &= 1.5 / 2.5 = 0.6 ,\ x_2 &= (1.4 + 0.6 \cdot 0.6) / 0.8 = 1.76 / 0.8 = 2.2 ,\ x_1 &= (8 - 6 \cdot 2.2 - 3 \cdot 0.6) / 5 = (8 - 13.2 - 1.8) / 5 = -1.4 . \end{aligned}

### Computing an LU decomposition We have now shown that if we can create an LUP decomposition for a nonsingular matrix $A$, then forward and back substitution can solve the system $Ax = b$ of linear equations. Now we show how to efficiently compute an LUP decomposition for $A$. We start with the case in which $A$ is an $n \times n$ nonsingular matrix and $P$ is absent (or, equivalently, $P = I_n$). In this case, we factor $A = LU$. We call the two matrices $L$ and $U$ an **LU decomposition** of $A$. We use a process known as **Gaussian elimination** to create an LU decomposition. We start by subtracting multiples of the first equation from the other equations in order to remove the first variable from those equations. Then, we subtract multiples of the second equation from the third and subsequent equations so that now the first and second variables are removed from them. We continue this process until the system that remains has an upper-triangular form; in fact, it is the matrix $U$. The matrix $L$ is made up of the row multipliers that cause variables to be eliminated. Our algorithm to implement this strategy is recursive. We wish to construct an LU decomposition for an $n \times n$ nonsingular matrix $A$. If $n = 1$, then we are done, since we can choose $L = I_1$ and $U = A$. For $n > 1$, we break $A$ into four parts:

A = \begin{pmatrix} a_{11} & a_{12} & \cdots & a_{1n}\ a_{21} & a_{22} & \cdots & a_{2n}\ \vdots & \vdots & \ddots & \vdots\ a_{n1} & a_{n2} & \cdots & a_{nn} \end{pmatrix}

\begin{pmatrix} a_{11} & w^T\ v & A’ \end{pmatrix},

where $v = (a_{21}, a_{31}, \ldots, a_{n1})^T$ is a column $(n-1)$-vector, $w^T = (a_{12}, a_{13}, \ldots, a_{1n})$ is a row $(n-1)$-vector, and $A'$ is an $(n-1) \times (n-1)$ matrix. Then, using matrix algebra (verify the equations by simply multiplying through), we can factor $A$ as

\begin{aligned} A &= \begin{pmatrix} a_{11} & w^T\ v & A’ \end{pmatrix}\ &= \begin{pmatrix} 1 & 0\ v/a_{11} & I_{n-1} \end{pmatrix} \begin{pmatrix} a_{11} & w^T\ 0 & A’ - vw^T/a_{11} \end{pmatrix} . \end{aligned} \tag{28.8}

The $0$s in the first and second matrices of equation (28.8) are row and column $(n-1)$-vectors, respectively. The term $vw^T/a_{11}$, formed by taking the outer product of $v$ and $w$ and dividing each element of the result by $a_{11}$, is an $(n-1) \times (n-1)$ matrix, which conforms in size to the matrix $A'$ from which it is subtracted. The resulting $(n-1) \times (n-1)$ matrix $$A' - vw^T/a_{11} \tag{28.9}$$ is called the **Schur complement** of $A$ with respect to $a_{11}$. > **Verification of (28.8).** Multiplying the two factors block by block: > $$ > \begin{aligned} > \begin{pmatrix} 1 & 0\\ v/a_{11} & I_{n-1} \end{pmatrix} > \begin{pmatrix} a_{11} & w^T\\ 0 & A' - vw^T/a_{11} \end{pmatrix} > &= \begin{pmatrix} a_{11} & w^T\\ (v/a_{11})a_{11} & (v/a_{11})w^T + A' - vw^T/a_{11} \end{pmatrix}\\ > &= \begin{pmatrix} a_{11} & w^T\\ v & A' \end{pmatrix} . > \end{aligned} > $$ We claim that if $A$ is nonsingular, then the Schur complement is nonsingular, too. Why? > **Proof.** Suppose that the Schur complement, which is $(n-1) \times (n-1)$, is singular. Then by Theorem D.1, it has row rank strictly less than $n-1$. Because the bottom $n-1$ entries in the first column of the matrix > $$\begin{pmatrix} a_{11} & w^T\\ 0 & A' - vw^T/a_{11} \end{pmatrix}$$ > are all $0$, the bottom $n-1$ rows of this matrix must have row rank strictly less than $n-1$. The row rank of the entire matrix, therefore, is strictly less than $n$. Applying Exercise D.2-8 to equation (28.8), $A$ has rank strictly less than $n$, and from Theorem D.1 we derive the contradiction that $A$ is singular. Because the Schur complement is nonsingular, we can now recursively find an LU decomposition for it. Let us say that $$A' - vw^T/a_{11} = L'U' ,$$ where $L'$ is unit lower-triangular and $U'$ is upper-triangular. Then, using matrix algebra, we have

\begin{aligned} A &= \begin{pmatrix} 1 & 0\ v/a_{11} & I_{n-1} \end{pmatrix} \begin{pmatrix} a_{11} & w^T\ 0 & A’ - vw^T/a_{11} \end{pmatrix}\ &= \begin{pmatrix} 1 & 0\ v/a_{11} & I_{n-1} \end{pmatrix} \begin{pmatrix} a_{11} & w^T\ 0 & L’U’ \end{pmatrix}\ &= \begin{pmatrix} 1 & 0\ v/a_{11} & L’ \end{pmatrix} \begin{pmatrix} a_{11} & w^T\ 0 & U’ \end{pmatrix}\ &= LU , \end{aligned}

thereby providing our LU decomposition. (Note that because $L'$ is unit lower-triangular, so is $L$, and because $U'$ is upper-triangular, so is $U$.) Of course, if $a_{11} = 0$, this method doesn't work, because it divides by $0$. It also doesn't work if the upper leftmost entry of the Schur complement $A' - vw^T/a_{11}$ is $0$, since we divide by it in the next step of the recursion. The elements by which we divide during LU decomposition are called **pivots**, and they occupy the diagonal elements of the matrix $U$. The reason we include a permutation matrix $P$ during LUP decomposition is that it allows us to avoid dividing by $0$. When we use permutations to avoid division by $0$ (or by small numbers, which would contribute to numerical instability), we are **pivoting**. An important class of matrices for which LU decomposition always works correctly is the class of symmetric positive-definite matrices. Such matrices require no pivoting, and thus we can employ the recursive strategy outlined above without fear of dividing by $0$. We shall prove this result, as well as several others, in Section 28.3. Our code for LU decomposition of a matrix $A$ follows the recursive strategy, except that an iteration loop replaces the recursion. (This transformation is a standard optimization for a "tail-recursive" procedure, one whose last operation is a recursive call to itself. See Problem 7-4.) It assumes that the attribute $A.rows$ gives the dimension of $A$. We initialize the matrix $U$ with $0$s below the diagonal and matrix $L$ with $1$s on its diagonal and $0$s above the diagonal. Each iteration works on a square submatrix, using its upper leftmost element as the pivot to compute the $v$ and $w$ vectors and the Schur complement, which becomes the square submatrix worked on by the next iteration. ```text LU-DECOMPOSITION(A) 1 n = A.rows 2 let L and U be new n x n matrices 3 initialize U with 0s below the diagonal 4 initialize L with 1s on the diagonal and 0s above the diagonal 5 for k = 1 to n 6 u_kk = a_kk 7 for i = k + 1 to n 8 l_ik = a_ik / a_kk // a_ik holds v_i 9 u_ki = a_ki // a_ki holds w_i 10 for i = k + 1 to n 11 for j = k + 1 to n 12 a_ij = a_ij - l_ik u_kj 13 return L and U ``` The outer **for** loop beginning in line 5 iterates once for each recursive step. Within this loop, line 6 determines the pivot to be $u_{kk} = a_{kk}$. The **for** loop in lines 7-9 (which does not execute when $k = n$) uses the $v$ and $w$ vectors to update $L$ and $U$. Line 8 determines the below-diagonal elements of $L$, storing $v_i/a_{kk}$ in $l_{ik}$, and line 9 computes the above-diagonal elements of $U$, storing $w_i^T$ in $u_{ki}$. Finally, lines 10-12 compute the elements of the Schur complement and store them back into the matrix $A$. (We don't need to divide by $a_{kk}$ in line 12 because we already did so when we computed $l_{ik}$ in line 8.) Because line 12 is triply nested, LU-DECOMPOSITION runs in time $\Theta(n^3)$. *Figure 28.1: The operation of LU-DECOMPOSITION. (a) The matrix $A$. (b) The element $a_{11} = 2$ in the black circle is the pivot, the shaded column is $v/a_{11}$, and the shaded row is $w^T$. The elements of $U$ computed thus far are above the horizontal line, and the elements of $L$ are to the left of the vertical line. The Schur complement matrix $A' - vw^T/a_{11}$ occupies the lower right. (c) We now operate on the Schur complement matrix produced from part (b). The element $a_{22} = 4$ in the black circle is the pivot, and the shaded column and row are $v/a_{22}$ and $w^T$ (in the partitioning of the Schur complement), respectively. Lines divide the matrix into the elements of $U$ computed so far (above), the elements of $L$ computed so far (left), and the new Schur complement (lower right). (d) After the next step, the matrix $A$ is factored. (The element $3$ in the new Schur complement becomes part of $U$ when the recursion terminates.) (e) The factorization $A = LU$.* The figure shows a $4 \times 4$ example with final factorization

A = \begin{pmatrix} 2 & 3 & 1 & 5\ 6 & 13 & 5 & 19\ 2 & 19 & 10 & 23\ 4 & 10 & 11 & 31 \end{pmatrix},; L = \begin{pmatrix} 1 & 0 & 0 & 0\ 3 & 1 & 0 & 0\ 1 & 4 & 1 & 0\ 2 & 1 & 7 & 1 \end{pmatrix},; U = \begin{pmatrix} 2 & 3 & 1 & 5\ 0 & 4 & 2 & 4\ 0 & 0 & 1 & 2\ 0 & 0 & 0 & 3 \end{pmatrix}.

Figure 28.1 illustrates the operation of LU-DECOMPOSITION. It shows a standard optimization of the procedure in which we store the significant elements of $L$ and $U$ in place in the matrix $A$. That is, we can set up a correspondence between each element $a_{ij}$ and either $l_{ij}$ (if $i > j$) or $u_{ij}$ (if $i \le j$) and update the matrix $A$ so that it holds both $L$ and $U$ when the procedure terminates. To obtain the pseudocode for this optimization from the above pseudocode, just replace each reference to $l$ or $u$ by $a$; you can easily verify that this transformation preserves correctness. ### Computing an LUP decomposition Generally, in solving a system of linear equations $Ax = b$, we must pivot on off-diagonal elements of $A$ to avoid dividing by $0$. Dividing by $0$ would, of course, be disastrous. But we also want to avoid dividing by a small value, even if $A$ is nonsingular, because numerical instabilities can result. We therefore try to pivot on a large value. The mathematics behind LUP decomposition is similar to that of LU decomposition. Recall that we are given an $n \times n$ nonsingular matrix $A$, and we wish to find a permutation matrix $P$, a unit lower-triangular matrix $L$, and an upper-triangular matrix $U$ such that $PA = LU$. Before we partition the matrix $A$, as we did for LU decomposition, we move a nonzero element, say $a_{k1}$, from somewhere in the first column to the $(1,1)$ position of the matrix. For numerical stability, we choose $a_{k1}$ as the element in the first column with the greatest absolute value. (The first column cannot contain only $0$s, for then $A$ would be singular, because its determinant would be $0$, by Theorems D.4 and D.5.) In order to preserve the set of equations, we exchange row $1$ with row $k$, which is equivalent to multiplying $A$ by a permutation matrix $Q$ on the left (Exercise D.1-4). Thus, we can write $QA$ as

QA = \begin{pmatrix} a_{k1} & w^T\ v & A’ \end{pmatrix},

where $v = (a_{21}, a_{31}, \ldots, a_{11}, \ldots, a_{n1})^T$, except that $a_{11}$ replaces $a_{k1}$; $w^T = (a_{k2}, a_{k3}, \ldots, a_{kn})$; and $A'$ is an $(n-1) \times (n-1)$ matrix. Since $a_{k1} \neq 0$, we can now perform much the same linear algebra as for LU decomposition, but now guaranteeing that we do not divide by $0$:

QA = \begin{pmatrix} a_{k1} & w^T\ v & A’ \end{pmatrix} = \begin{pmatrix} 1 & 0\ v/a_{k1} & I_{n-1} \end{pmatrix} \begin{pmatrix} a_{k1} & w^T\ 0 & A’ - vw^T/a_{k1} \end{pmatrix} .

As we saw for LU decomposition, if $A$ is nonsingular, then the Schur complement $A' - vw^T/a_{k1}$ is nonsingular, too. Therefore, we can recursively find an LUP decomposition for it, with unit lower-triangular matrix $L'$, upper-triangular matrix $U'$, and permutation matrix $P'$, such that $$P'(A' - vw^T/a_{k1}) = L'U' .$$ Define

P = \begin{pmatrix} 1 & 0\ 0 & P’ \end{pmatrix} Q ,

w hi c hi s a p er m u t a t i o nma t r i x, s in ce i t i s t h e p r o d u c t o f tw o p er m u t a t i o nma t r i ces (E x er c i seD .1 - 4) . W e n o w ha v e

\begin{aligned} PA &= \begin{pmatrix} 1 & 0\ 0 & P’ \end{pmatrix} QA\ &= \begin{pmatrix} 1 & 0\ 0 & P’ \end{pmatrix} \begin{pmatrix} 1 & 0\ v/a_{k1} & I_{n-1} \end{pmatrix} \begin{pmatrix} a_{k1} & w^T\ 0 & A’ - vw^T/a_{k1} \end{pmatrix}\ &= \begin{pmatrix} 1 & 0\ P’v/a_{k1} & P’ \end{pmatrix} \begin{pmatrix} a_{k1} & w^T\ 0 & A’ - vw^T/a_{k1} \end{pmatrix}\ &= \begin{pmatrix} 1 & 0\ P’v/a_{k1} & I_{n-1} \end{pmatrix} \begin{pmatrix} a_{k1} & w^T\ 0 & P’(A’ - vw^T/a_{k1}) \end{pmatrix}\ &= \begin{pmatrix} 1 & 0\ P’v/a_{k1} & I_{n-1} \end{pmatrix} \begin{pmatrix} a_{k1} & w^T\ 0 & L’U’ \end{pmatrix}\ &= \begin{pmatrix} 1 & 0\ P’v/a_{k1} & L’ \end{pmatrix} \begin{pmatrix} a_{k1} & w^T\ 0 & U’ \end{pmatrix}\ &= LU , \end{aligned}

yielding the LUP decomposition. Because $L'$ is unit lower-triangular, so is $L$, and because $U'$ is upper-triangular, so is $U$. Notice that in this derivation, unlike the one for LU decomposition, we must multiply both the column vector $v/a_{k1}$ and the Schur complement $A' - vw^T/a_{k1}$ by the permutation matrix $P'$. Here is the pseudocode for LUP decomposition: ```text LUP-DECOMPOSITION(A) 1 n = A.rows 2 let π[1..n] be a new array 3 for i = 1 to n 4 π[i] = i 5 for k = 1 to n 6 p = 0 7 for i = k to n 8 if |a_ik| > p 9 p = |a_ik| 10 k' = i 11 if p == 0 12 error "singular matrix" 13 exchange π[k] with π[k'] 14 for i = 1 to n 15 exchange a_ki with a_k'i 16 for i = k + 1 to n 17 a_ik = a_ik / a_kk 18 for j = k + 1 to n 19 a_ij = a_ij - a_ik a_kj ``` Like LU-DECOMPOSITION, our LUP-DECOMPOSITION procedure replaces the recursion with an iteration loop. As an improvement over a direct implementation of the recursion, we dynamically maintain the permutation matrix $P$ as an array $\pi$, where $\pi[i] = j$ means that the $i$th row of $P$ contains a $1$ in column $j$. We also implement the code to compute $L$ and $U$ "in place" in the matrix $A$. Thus, when the procedure terminates,

a_{ij} = \begin{cases} l_{ij} & \text{if } i > j ,\ u_{ij} & \text{if } i \le j . \end{cases}

Figure 28.2 illustrates how LUP-DECOMPOSITION factors a matrix. Lines 3-4 initialize the array $\pi$ to represent the identity permutation. The outer **for** loop beginning in line 5 implements the recursion. Each time through the outer loop, lines 6-10 determine the element $a_{k'k}$ with largest absolute value of those in the current first column (column $k$) of the $(n-k+1) \times (n-k+1)$ matrix whose LUP decomposition we are finding. If all elements in the current first column are zero, lines 11-12 report that the matrix is singular. To pivot, we exchange $\pi[k']$ with $\pi[k]$ in line 13 and exchange the $k$th and $k'$th rows of $A$ in lines 14-15, thereby making the pivot element $a_{kk}$. (The entire rows are swapped because in the derivation of the method above, not only is $A' - vw^T/a_{k1}$ multiplied by $P'$, but so is $v/a_{k1}$.) Finally, the Schur complement is computed by lines 16-19 in much the same way as it is computed by lines 7-12 of LU-DECOMPOSITION, except that here the operation is written to work in place. Because of its triply nested loop structure, LUP-DECOMPOSITION has a running time of $\Theta(n^3)$, which is the same as that of LU-DECOMPOSITION. Thus, pivoting costs us at most a constant factor in time. *Figure 28.2: The operation of LUP-DECOMPOSITION. (a) The input matrix $A$ with the identity permutation of the rows on the left. The first step of the algorithm determines that the element $5$ in the black circle in the third row is the pivot for the first column. (b) Rows 1 and 3 are swapped and the permutation is updated. The shaded column and row represent $v$ and $w^T$. (c) The vector $v$ is replaced by $v/5$, and the lower right of the matrix is updated with the Schur complement. Lines divide the matrix into three regions: elements of $U$ (above), elements of $L$ (left), and elements of the Schur complement (lower right). (d)-(f) The second step. (g)-(i) The third step. No further changes occur on the fourth (final) step. (j) The LUP decomposition $PA = LU$.* The figure's final result is

P = \begin{pmatrix} 0&0&1&0\ 1&0&0&0\ 0&0&0&1\ 0&1&0&0 \end{pmatrix},; A = \begin{pmatrix} 2&0&2&0.6\ 3&3&4&-2\ 5&5&4&2\ -1&-2&3.4&-1 \end{pmatrix},; L = \begin{pmatrix} 1&0&0&0\ 0.4&1&0&0\ -0.2&0.5&1&0\ 0.6&0&0.4&1 \end{pmatrix},; U = \begin{pmatrix} 5&5&4&2\ 0&-2&0.4&-0.2\ 0&0&4&-0.5\ 0&0&0&-3 \end{pmatrix}.

### Exercises 28.1-1 to 28.1-7 **28.1-1** Solve the equation

\begin{pmatrix} 1 & 0 & 0\ 4 & 1 & 0\ -6 & 5 & 1 \end{pmatrix} \begin{pmatrix} x_1\ x_2\ x_3 \end{pmatrix}

\begin{pmatrix} 3\ 14\ -7 \end{pmatrix}

b y u s in g f or w a r d s u b s t i t u t i o n . * * 28.1 - 2 * * F in d an LU d eco m p os i t i o n o f t h e ma t r i x

\begin{pmatrix} 4 & -5 & 6\ 8 & -6 & 7\ 12 & -7 & 12 \end{pmatrix}.

* * 28.1 - 3 * * S o l v e t h ee q u a t i o n

\begin{pmatrix} 1 & 5 & 4\ 2 & 0 & 3\ 5 & 8 & 2 \end{pmatrix} \begin{pmatrix} x_1\ x_2\ x_3 \end{pmatrix}

\begin{pmatrix} 12\ 9\ 5 \end{pmatrix}

A = \begin{pmatrix} f_1(x_1) & f_2(x_1) & \cdots & f_n(x_1)\ f_1(x_2) & f_2(x_2) & \cdots & f_n(x_2)\ \vdots & \vdots & \ddots & \vdots\ f_1(x_m) & f_2(x_m) & \cdots & f_n(x_m) \end{pmatrix}

denote the matrix of values of the basis functions at the given points; that is, $a_{ij} = f_j(x_i)$. Let $c = (c_k)$ denote the desired $n$-vector of coefficients. Then,

Ac = \begin{pmatrix} f_1(x_1) & f_2(x_1) & \cdots & f_n(x_1)\ f_1(x_2) & f_2(x_2) & \cdots & f_n(x_2)\ \vdots & \vdots & \ddots & \vdots\ f_1(x_m) & f_2(x_m) & \cdots & f_n(x_m) \end{pmatrix} \begin{pmatrix} c_1\ c_2\ \vdots\ c_n \end{pmatrix} = \begin{pmatrix} F(x_1)\ F(x_2)\ \vdots\ F(x_m) \end{pmatrix}

is the $m$-vector of "predicted values" for $y$. Thus, $$\eta = Ac - y$$ is the $m$-vector of **approximation errors**. To minimize approximation errors, we choose to minimize the norm of the error vector $\eta$, which gives us a **least-squares solution**, since $$\|\eta\| = \left( \sum_{i=1}^{m} \eta_i^2 \right)^{1/2} .$$ Because

|\eta|^2 = |Ac - y|^2 = \sum_{i=1}^{m} \left( \sum_{j=1}^{n} a_{ij}c_j - y_i \right)^2 ,

we can minimize $\|\eta\|$ by differentiating $\|\eta\|^2$ with respect to each $c_k$ and then setting the result to $0$:

\frac{d|\eta|^2}{dc_k} = \sum_{i=1}^{m} 2 \left( \sum_{j=1}^{n} a_{ij}c_j - y_i \right) a_{ik} = 0 . \tag{28.18}

The $n$ equations (28.18) for $k = 1, 2, \ldots, n$ are equivalent to the single matrix equation $$(Ac - y)^TA = 0$$ or, equivalently (using Exercise D.1-2), to $$A^T(Ac - y) = 0 ,$$ which implies $$A^TAc = A^Ty . \tag{28.19}$$ In statistics, this is called the **normal equation**. The matrix $A^TA$ is symmetric by Exercise D.1-2, and if $A$ has full column rank, then by Theorem D.6, $A^TA$ is positive-definite as well. Hence, $(A^TA)^{-1}$ exists, and the solution to equation (28.19) is

\begin{aligned} c &= \bigl((A^TA)^{-1}A^T\bigr)y\ &= A^+y , \end{aligned} \tag{28.20}

where the matrix $A^+ = (A^TA)^{-1}A^T$ is the **pseudoinverse** of the matrix $A$. The pseudoinverse naturally generalizes the notion of a matrix inverse to the case in which $A$ is not square. (Compare equation (28.20) as the approximate solution to $Ac = y$ with the solution $A^{-1}b$ as the exact solution to $Ax = b$.) **Example.** As an example of producing a least-squares fit, suppose that we have five data points

\begin{aligned} (x_1, y_1) &= (-1, 2) ,\ (x_2, y_2) &= (1, 1) ,\ (x_3, y_3) &= (2, 1) ,\ (x_4, y_4) &= (3, 0) ,\ (x_5, y_5) &= (5, 3) , \end{aligned}

shown as black dots in Figure 28.3. We wish to fit these points with a quadratic polynomial $$F(x) = c_1 + c_2x + c_3x^2 .$$ We start with the matrix of basis-function values

A = \begin{pmatrix} 1 & x_1 & x_1^2\ 1 & x_2 & x_2^2\ 1 & x_3 & x_3^2\ 1 & x_4 & x_4^2\ 1 & x_5 & x_5^2 \end{pmatrix} = \begin{pmatrix} 1 & -1 & 1\ 1 & 1 & 1\ 1 & 2 & 4\ 1 & 3 & 9\ 1 & 5 & 25 \end{pmatrix},

w h ose p se u d o in v er se i s

A^+ = \begin{pmatrix} 0.500 & 0.300 & 0.200 & 0.100 & -0.100\ -0.388 & 0.093 & 0.190 & 0.193 & -0.088\ 0.060 & -0.036 & -0.048 & -0.036 & 0.060 \end{pmatrix}.

Multiplying $y$ by $A^+$, we obtain the coefficient vector

c = \begin{pmatrix} 1.200\ -0.757\ 0.214 \end{pmatrix},

which corresponds to the quadratic polynomial $$F(x) = 1.200 - 0.757x + 0.214x^2$$ as the closest-fitting quadratic to the given data, in a least-squares sense. *Figure 28.3: The least-squares fit of a quadratic polynomial to the set of five data points $\{(-1,2), (1,1), (2,1), (3,0), (5,3)\}$. The black dots are the data points, and the white dots are their estimated values predicted by the polynomial $F(x) = 1.2 - 0.757x + 0.214x^2$, the quadratic polynomial that minimizes the sum of the squared errors. Each shaded line shows the error for one data point.* As a practical matter, we solve the normal equation (28.19) by multiplying $y$ by $A^T$ and then finding an LU decomposition of $A^TA$. If $A$ has full rank, the matrix $A^TA$ is guaranteed to be nonsingular, because it is symmetric and positive-definite. (See Exercise D.1-2 and Theorem D.6.) *Think about it:* the fitted curve does not pass through any of the five dots, yet it is the "fairest" compromise. What would go wrong if we insisted on a degree-4 polynomial that hits all five points exactly? ### Exercises 28.3-1 to 28.3-7 **28.3-1** Prove that every diagonal element of a symmetric positive-definite matrix is positive. **28.3-2** Let $A = \begin{pmatrix} a & b\\ b & c \end{pmatrix}$ be a $2 \times 2$ symmetric positive-definite matrix. Prove that its determinant $ac - b^2$ is positive by "completing the square" in a manner similar to that used in the proof of Lemma 28.5. **28.3-3** Prove that the maximum element in a symmetric positive-definite matrix lies on the diagonal. **28.3-4** Prove that the determinant of each leading submatrix of a symmetric positive-definite matrix is positive. **28.3-5** Let $A_k$ denote the $k$th leading submatrix of a symmetric positive-definite matrix $A$. Prove that $\det(A_k)/\det(A_{k-1})$ is the $k$th pivot during LU decomposition, where, by convention, $\det(A_0) = 1$. **28.3-6** Find the function of the form $$F(x) = c_1 + c_2x\lg x + c_3e^x$$ that is the best least-squares fit to the data points $$(1, 1),\; (2, 1),\; (3, 3),\; (4, 8) .$$ **28.3-7** Show that the pseudoinverse $A^+$ satisfies the following four equations:

\begin{aligned} AA^+A &= A ,\ A^+AA^+ &= A^+ ,\ (AA^+)^T &= AA^+ ,\ (A^+A)^T &= A^+A . \end{aligned}

## Problems **28-1 Tridiagonal systems of linear equations** Consider the tridiagonal matrix

A = \begin{pmatrix} 1 & -1 & 0 & 0 & 0\ -1 & 2 & -1 & 0 & 0\ 0 & -1 & 2 & -1 & 0\ 0 & 0 & -1 & 2 & -1\ 0 & 0 & 0 & -1 & 2 \end{pmatrix}.

**a.** Find an LU decomposition of $A$. **b.** Solve the equation $Ax = \begin{pmatrix} 1 & 1 & 1 & 1 & 1 \end{pmatrix}^T$ by using forward and back substitution. **c.** Find the inverse of $A$. **d.** Show how, for any $n \times n$ symmetric positive-definite, tridiagonal matrix $A$ and any $n$-vector $b$, to solve the equation $Ax = b$ in $O(n)$ time by performing an LU decomposition. Argue that any method based on forming $A^{-1}$ is asymptotically more expensive in the worst case. **e.\*** Show how, for any $n \times n$ nonsingular, tridiagonal matrix $A$ and any $n$-vector $b$, to solve the equation $Ax = b$ in $O(n)$ time by performing an LUP decomposition. **28-2 Splines** A practical method for interpolating a set of points with a curve is to use **cubic splines**. We are given a set $\{(x_i, y_i) : i = 0, 1, \ldots, n\}$ of $n+1$ point-value pairs, where $x_0 < x_1 < \cdots < x_n$. We wish to fit a piecewise-cubic curve (spline) $f(x)$ to the points. That is, the curve $f(x)$ is made up of $n$ cubic polynomials $f_i(x) = a_i + b_ix + c_ix^2 + d_ix^3$ for $i = 0, 1, \ldots, n-1$, where if $x$ falls in the range $x_i \le x \le x_{i+1}$, then the value of the curve is given by $f(x) = f_i(x - x_i)$. The points $x_i$ at which the cubic polynomials are "pasted" together are called **knots**. For simplicity, we shall assume that $x_i = i$ for $i = 0, 1, \ldots, n$. To ensure continuity of $f(x)$, we require that

\begin{aligned} f(x_i) &= f_i(0) = y_i ,\ f(x_{i+1}) &= f_i(1) = y_{i+1} \end{aligned}

for $i = 0, 1, \ldots, n-1$. To ensure that $f(x)$ is sufficiently smooth, we also insist that the first derivative be continuous at each knot: $$f'(x_{i+1}) = f_i'(1) = f_{i+1}'(0)$$ for $i = 0, 1, \ldots, n-2$. **a.** Suppose that for $i = 0, 1, \ldots, n$, we are given not only the point-value pairs $\{(x_i, y_i)\}$ but also the first derivatives $D_i = f'(x_i)$ at each knot. Express each coefficient $a_i$, $b_i$, $c_i$, and $d_i$ in terms of the values $y_i$, $y_{i+1}$, $D_i$, and $D_{i+1}$. (Remember that $x_i = i$.) How quickly can we compute the $4n$ coefficients from the point-value pairs and first derivatives? The question remains of how to choose the first derivatives of $f(x)$ at the knots. One method is to require the second derivatives to be continuous at the knots: $$f''(x_{i+1}) = f_i''(1) = f_{i+1}''(0)$$ for $i = 0, 1, \ldots, n-2$. At the first and last knots, we assume that $f''(x_0) = f_0''(0) = 0$ and $f''(x_n) = f_{n-1}''(1) = 0$; these assumptions make $f(x)$ a **natural** cubic spline. **b.** Use the continuity constraints on the second derivative to show that for $i = 1, 2, \ldots, n-1$, $$D_{i-1} + 4D_i + D_{i+1} = 3(y_{i+1} - y_{i-1}) . \tag{28.21}$$ **c.** Show that

\begin{aligned} 2D_0 + D_1 &= 3(y_1 - y_0) ,\ D_{n-1} + 2D_n &= 3(y_n - y_{n-1}) . \end{aligned} \tag{28.22, 28.23}

**d.** Rewrite equations (28.21)-(28.23) as a matrix equation involving the vector $D = \langle D_0, D_1, \ldots, D_n \rangle$ of unknowns. What attributes does the matrix in your equation have? **e.** Argue that a natural cubic spline can interpolate a set of $n+1$ point-value pairs in $O(n)$ time (see Problem 28-1). **f.** Show how to determine a natural cubic spline that interpolates a set of $n+1$ points $(x_i, y_i)$ satisfying $x_0 < x_1 < \cdots < x_n$, even when $x_i$ is not necessarily equal to $i$. What matrix equation must your method solve, and how quickly does your algorithm run? ## Chapter notes Many excellent texts describe numerical and scientific computation in much greater detail than we have room for here. The following are especially readable: George and Liu [132], Golub and Van Loan [144], Press, Teukolsky, Vetterling, and Flannery [283, 284], and Strang [323, 324]. Golub and Van Loan [144] discuss numerical stability. They show why $\det(A)$ is not necessarily a good indicator of the stability of a matrix $A$, proposing instead to use $\|A\|_\infty \|A^{-1}\|_\infty$, where $\|A\|_\infty = \max_{1 \le i \le n} \sum_{j=1}^{n} |a_{ij}|$. They also address the question of how to compute this value without actually computing $A^{-1}$. Gaussian elimination, upon which the LU and LUP decompositions are based, was the first systematic method for solving linear systems of equations. It was also one of the earliest numerical algorithms. Although it was known earlier, its discovery is commonly attributed to C. F. Gauss (1777-1855). In his famous paper [325], Strassen showed that an $n \times n$ matrix can be inverted in $O(n^{\lg 7})$ time. Winograd [358] originally proved that matrix multiplication is no harder than matrix inversion, and the converse is due to Aho, Hopcroft, and Ullman [5]. Another important matrix decomposition is the **singular value decomposition**, or SVD. The SVD factors an $m \times n$ matrix $A$ into $A = Q_1 \Sigma Q_2^T$, where $\Sigma$ is an $m \times n$ matrix with nonzero values only on the diagonal, $Q_1$ is $m \times m$ with mutually orthonormal columns, and $Q_2$ is $n \times n$, also with mutually orthonormal columns. Two vectors are **orthonormal** if their inner product is $0$ and each vector has a norm of $1$. The books by Strang [323, 324] and Golub and Van Loan [144] contain good treatments of the SVD. Strang [324] has an excellent presentation of symmetric positive-definite matrices and of linear algebra in general. # 29 Linear Programming > **Big picture.** Imagine you are packing a backpack, running a campaign, or routing water through pipes: in every case you want the *best* outcome while a pile of limits pushes back. Linear programming is the art of describing "best" and "limits" with nothing but straight lines (and flat planes), and then letting a systematic algorithm walk to the answer. Ask yourself as you read: why would the best answer always be hiding in a corner? Many problems take the form of maximizing or minimizing an objective, given limited resources and competing constraints. If we can specify the objective as a linear function of certain variables, and if we can specify the constraints on resources as equalities or inequalities on those variables, then we have a **linear-programming problem**. Linear programs arise in a variety of practical applications. We begin by studying an application in electoral politics. ## A political problem Suppose that you are a politician trying to win an election. Your district has three different types of areas: urban, suburban, and rural. These areas have, respectively, 100,000, 200,000, and 50,000 registered voters. Although not all the registered voters actually go to the polls, you decide that to govern effectively, you would like at least half the registered voters in each of the three regions to vote for you. You are honorable and would never consider supporting policies in which you do not believe. You realize, however, that certain issues may be more effective in winning votes in certain places. Your primary issues are building more roads, gun control, farm subsidies, and a gasoline tax dedicated to improved public transit. According to your campaign staff's research, you can estimate how many votes you win or lose from each population segment by spending \$1,000 on advertising on each issue. This information appears in the table of Figure 29.1. In this table, each entry indicates the number of thousands of either urban, suburban, or rural voters who would be won over by spending \$1,000 on advertising in support of a particular issue. Negative entries denote votes that would be lost. Your task is to figure out the minimum amount of money that you need to spend in order to win 50,000 urban votes, 100,000 suburban votes, and 25,000 rural votes. You could, by trial and error, devise a strategy that wins the required number of votes, but the strategy you come up with might not be the least expensive one. For example, you could devote \$20,000 of advertising to building roads, \$0 to gun control, \$4,000 to farm subsidies, and \$9,000 to a gasoline tax. In this case, you would win

20(-2) + 0(8) + 4(0) + 9(10) = 50

thousand urban votes, $20(5) + 0(2) + 4(0) + 9(0) = 100$ thousand suburban votes, and $20(3) + 0(-5) + 4(10) + 9(-2) = 82$ thousand rural votes. You would win the exact number of votes desired in the urban and suburban areas and more than enough votes in the rural area. (In fact, in the rural area, you would receive more votes than there are voters.) In order to garner these votes, you would have paid for $20 + 0 + 4 + 9 = 33$ thousand dollars of advertising. | policy | urban | suburban | rural | |---|---|---|---| | build roads | $-2$ | $5$ | $3$ | | gun control | $8$ | $2$ | $-5$ | | farm subsidies | $0$ | $0$ | $10$ | | gasoline tax | $10$ | $0$ | $-2$ | *Figure 29.1: The effects of policies on voters. Each entry describes the number of thousands of urban, suburban, or rural voters who could be won over by spending \$1,000 on advertising support of a policy on a particular issue. Negative entries denote votes that would be lost.* Naturally, you may wonder whether this strategy is the best possible. That is, could you achieve your goals while spending less on advertising? Additional trial and error might help you to answer this question, but wouldn't you rather have a systematic method for answering such questions? In order to develop one, we shall formulate this question mathematically. We introduce 4 variables: - $x_1$ is the number of thousands of dollars spent on advertising on building roads, - $x_2$ is the number of thousands of dollars spent on advertising on gun control, - $x_3$ is the number of thousands of dollars spent on advertising on farm subsidies, and - $x_4$ is the number of thousands of dollars spent on advertising on a gasoline tax. We can write the requirement that we win at least 50,000 urban votes as

-2x_1 + 8x_2 + 0x_3 + 10x_4 \ge 50 . \tag{29.1}

S imi l a r l y, w ec an w r i t e t h er e q u i r e m e n t s t ha tw e w ina tl e a s t 100, 000 s u b u r ban v o t es an d 25, 000 r u r a l v o t es a s

5x_1 + 2x_2 + 0x_3 + 0x_4 \ge 100 \tag{29.2}

an d

3x_1 - 5x_2 + 10x_3 - 2x_4 \ge 25 . \tag{29.3}

Any setting of the variables $x_1, x_2, x_3, x_4$ that satisfies inequalities (29.1)-(29.3) yields a strategy that wins a sufficient number of each type of vote. In order to keep costs as small as possible, you would like to minimize the amount spent on advertising. That is, you want to minimize the expression

x_1 + x_2 + x_3 + x_4 . \tag{29.4}

A l t h o ug hn e g a t i v e a d v er t i s in g o f t e n occ u r s in p o l i t i c a l c am p ai g n s, t h er e i s n os u c h t hin g a s n e g a t i v e - cos t a d v er t i s in g . C o n se q u e n tl y, w er e q u i r e t ha t

x_1 \ge 0,; x_2 \ge 0,; x_3 \ge 0,; \text{and } x_4 \ge 0 . \tag{29.5}

C o mbinin g in e q u a l i t i es (29.1) - (29.3) an d (29.5) w i t h t h eo bj ec t i v eo f minimi z in g (29.4), w eo b t ain w ha t i s k n o w na s a " l in e a r p r o g r am ." W ec an f or ma tt hi s p r o b l e m t ab u l a r l y a s

\begin{aligned} \text{minimize}\quad & x_1 + x_2 + x_3 + x_4 && (29.6)\ \text{subject to}\quad & \ & -2x_1 + 8x_2 + 0x_3 + 10x_4 \ge 50 && (29.7)\ & 5x_1 + 2x_2 + 0x_3 + 0x_4 \ge 100 && (29.8)\ & 3x_1 - 5x_2 + 10x_3 - 2x_4 \ge 25 && (29.9)\ & x_1, x_2, x_3, x_4 \ge 0 && (29.10) \end{aligned}

The solution of this linear program yields your optimal strategy. ## General linear programs In the general linear-programming problem, we wish to optimize a linear function subject to a set of linear inequalities. Given a set of real numbers $a_1, a_2, \ldots, a_n$ and a set of variables $x_1, x_2, \ldots, x_n$, we define a **linear function** $f$ on those variables by

f(x_1, x_2, \ldots, x_n) = a_1 x_1 + a_2 x_2 + \cdots + a_n x_n = \sum_{j=1}^{n} a_j x_j .

If $b$ is a real number and $f$ is a linear function, then the equation

f(x_1, x_2, \ldots, x_n) = b

i s a * * l in e a r e q u a l i t y * * an d t h e in e q u a l i t i es

f(x_1, x_2, \ldots, x_n) \le b

an d

f(x_1, x_2, \ldots, x_n) \ge b

are **linear inequalities**. We use the general term **linear constraints** to denote either linear equalities or linear inequalities. In linear programming, we do not allow strict inequalities. Formally, a **linear-programming problem** is the problem of either minimizing or maximizing a linear function subject to a finite set of linear constraints. If we are to minimize, then we call the linear program a **minimization linear program**, and if we are to maximize, then we call the linear program a **maximization linear program**. The remainder of this chapter covers how to formulate and solve linear programs. Although several polynomial-time algorithms for linear programming have been developed, we will not study them in this chapter. Instead, we shall study the simplex algorithm, which is the oldest linear-programming algorithm. The simplex algorithm does not run in polynomial time in the worst case, but it is fairly efficient and widely used in practice. ## An overview of linear programming In order to describe properties of and algorithms for linear programs, we find it convenient to express them in canonical forms. We shall use two forms, *standard* and *slack*, in this chapter. We will define them precisely in Section 29.1. Informally, a linear program in standard form is the maximization of a linear function subject to linear inequalities, whereas a linear program in slack form is the maximization of a linear function subject to linear equalities. We shall typically use standard form for expressing linear programs, but we find it more convenient to use slack form when we describe the details of the simplex algorithm. For now, we restrict our attention to maximizing a linear function on $n$ variables subject to a set of $m$ linear inequalities. Let us first consider the following linear program with two variables:

\begin{aligned} \text{maximize}\quad & x_1 + x_2 && (29.11)\ \text{subject to}\quad & \ & 4x_1 - x_2 \le 8 && (29.12)\ & 2x_1 + x_2 \le 10 && (29.13)\ & 5x_1 - 2x_2 \ge -2 && (29.14)\ & x_1, x_2 \ge 0 . && (29.15) \end{aligned}

\begin{aligned} \text{maximize}\quad & \sum_{j=1}^{n} c_j x_j && (29.16)\ \text{subject to}\quad & \ & \sum_{j=1}^{n} a_{ij} x_j \le b_i \quad \text{for } i = 1, 2, \ldots, m && (29.17)\ & x_j \ge 0 \quad \text{for } j = 1, 2, \ldots, n . && (29.18) \end{aligned}

Generalizing the terminology we introduced for the two-variable linear program, we call expression (29.16) the **objective function** and the $n + m$ inequalities in lines (29.17) and (29.18) the **constraints**. The $n$ constraints in line (29.18) are the **nonnegativity constraints**. An arbitrary linear program need not have nonnegativity constraints, but standard form requires them. Sometimes we find it convenient to express a linear program in a more compact form. If we create an $m \times n$ matrix $A = (a_{ij})$, an $m$-vector $b = (b_i)$, an $n$-vector $c = (c_j)$, and an $n$-vector $x = (x_j)$, then we can rewrite the linear program defined in (29.16)-(29.18) as

\begin{aligned} \text{maximize}\quad & c^{\mathsf T} x && (29.19)\ \text{subject to}\quad & \ & Ax \le b && (29.20)\ & x \ge 0 . && (29.21) \end{aligned}

In line (29.19), $c^{\mathsf T} x$ is the inner product of two vectors. In inequality (29.20), $Ax$ is a matrix-vector product, and in inequality (29.21), $x \ge 0$ means that each entry of the vector $x$ must be nonnegative. We see that we can specify a linear program in standard form by a tuple $(A, b, c)$, and we shall adopt the convention that $A$, $b$, and $c$ always have the dimensions given above. We now introduce terminology to describe solutions to linear programs. We used some of this terminology in the earlier example of a two-variable linear program. We call a setting of the variables $\bar{x}$ that satisfies all the constraints a **feasible solution**, whereas a setting of the variables $\bar{x}$ that fails to satisfy at least one constraint is an **infeasible solution**. We say that a solution $\bar{x}$ has **objective value** $c^{\mathsf T}\bar{x}$. A feasible solution $\bar{x}$ whose objective value is maximum over all feasible solutions is an **optimal solution**, and we call its objective value $c^{\mathsf T}\bar{x}$ the **optimal objective value**. If a linear program has no feasible solutions, we say that the linear program is **infeasible**; otherwise it is **feasible**. If a linear program has some feasible solutions but does not have a finite optimal objective value, we say that the linear program is **unbounded**. Exercise 29.1-9 asks you to show that a linear program can have a finite optimal objective value even if the feasible region is not bounded. ### Converting linear programs into standard form It is always possible to convert a linear program, given as minimizing or maximizing a linear function subject to linear constraints, into standard form. A linear program might not be in standard form for any of four possible reasons: 1. The objective function might be a minimization rather than a maximization. 2. There might be variables without nonnegativity constraints. 3. There might be equality constraints, which have an equal sign rather than a less-than-or-equal-to sign. 4. There might be inequality constraints, but instead of having a less-than-or-equal-to sign, they have a greater-than-or-equal-to sign. When converting one linear program $L$ into another linear program $L'$, we would like the property that an optimal solution to $L'$ yields an optimal solution to $L$. To capture this idea, we say that two maximization linear programs $L$ and $L'$ are **equivalent** if for each feasible solution $\bar{x}$ to $L$ with objective value $z$, there is a corresponding feasible solution $\bar{x}'$ to $L'$ with objective value $z$, and for each feasible solution $\bar{x}'$ to $L'$ with objective value $z$, there is a corresponding feasible solution $\bar{x}$ to $L$ with objective value $z$. (This definition does not imply a one-to-one correspondence between feasible solutions.) A minimization linear program $L$ and a maximization linear program $L'$ are equivalent if for each feasible solution $\bar{x}$ to $L$ with objective value $z$, there is a corresponding feasible solution $\bar{x}'$ to $L'$ with objective value $-z$, and for each feasible solution $\bar{x}'$ to $L'$ with objective value $z$, there is a corresponding feasible solution $\bar{x}$ to $L$ with objective value $-z$. We now show how to remove, one by one, each of the possible problems in the list above. After removing each one, we shall argue that the new linear program is equivalent to the old one. **Step 1: minimization to maximization.** To convert a minimization linear program $L$ into an equivalent maximization linear program $L'$, we simply negate the coefficients in the objective function. Since $L$ and $L'$ have identical sets of feasible solutions and, for any feasible solution, the objective value in $L$ is the negative of the objective value in $L'$, these two linear programs are equivalent. For example, if we have the linear program

\begin{aligned} \text{minimize}\quad & -2x_1 + 3x_2\ \text{subject to}\quad & \ & x_1 + x_2 = 7\ & x_1 - 2x_2 \le 4\ & x_1 \ge 0, \end{aligned}

an d w e n e g a t e t h ecoe f f i c i e n t so f t h eo bj ec t i v e f u n c t i o n, w eo b t ain

\begin{aligned} \text{maximize}\quad & 2x_1 - 3x_2\ \text{subject to}\quad & \ & x_1 + x_2 = 7\ & x_1 - 2x_2 \le 4\ & x_1 \ge 0 . \end{aligned}

**Step 2: free variables.** Next, we show how to convert a linear program in which some of the variables do not have nonnegativity constraints into one in which each variable has a nonnegativity constraint. Suppose that some variable $x_j$ does not have a nonnegativity constraint. Then, we replace each occurrence of $x_j$ by $x_j' - x_j''$, and add the nonnegativity constraints $x_j' \ge 0$ and $x_j'' \ge 0$. Thus, if the objective function has a term $c_j x_j$, we replace it by $c_j x_j' - c_j x_j''$, and if constraint $i$ has a term $a_{ij} x_j$, we replace it by $a_{ij} x_j' - a_{ij} x_j''$. Any feasible solution $\hat{x}$ to the new linear program corresponds to a feasible solution $\bar{x}$ to the original linear program with $\bar{x}_j = \hat{x}_j' - \hat{x}_j''$ and with the same objective value. Also, any feasible solution $\bar{x}$ to the original linear program corresponds to a feasible solution $\hat{x}$ to the new linear program with $\hat{x}_j' = \bar{x}_j$ and $\hat{x}_j'' = 0$ if $\bar{x}_j \ge 0$, or with $\hat{x}_j'' = -\bar{x}_j$ and $\hat{x}_j' = 0$ if $\bar{x}_j < 0$. The two linear programs have the same objective value regardless of the sign of $x_j$. Thus, the two linear programs are equivalent. We apply this conversion scheme to each variable that does not have a nonnegativity constraint to yield an equivalent linear program in which all variables have nonnegativity constraints. Continuing the example, we want to ensure that each variable has a corresponding nonnegativity constraint. Variable $x_1$ has such a constraint, but variable $x_2$ does not. Therefore, we replace $x_2$ by two variables $x_2'$ and $x_2''$, and we modify the linear program to obtain

\begin{aligned} \text{maximize}\quad & 2x_1 - 3x_2’ + 3x_2”\ \text{subject to}\quad & \ & x_1 + x_2’ - x_2” = 7 && (29.22)\ & x_1 - 2x_2’ + 2x_2” \le 4\ & x_1, x_2’, x_2” \ge 0 . \end{aligned}

**Step 3: equalities to inequalities.** Next, we convert equality constraints into inequality constraints. Suppose that a linear program has an equality constraint $f(x_1, x_2, \ldots, x_n) = b$. Since $x = y$ if and only if both $x \ge y$ and $x \le y$, we can replace this equality constraint by the pair of inequality constraints $f(x_1, x_2, \ldots, x_n) \le b$ and $f(x_1, x_2, \ldots, x_n) \ge b$. Repeating this conversion for each equality constraint yields a linear program in which all constraints are inequalities. **Step 4: flipping $\ge$ to $\le$.** Finally, we can convert the greater-than-or-equal-to constraints to less-than-or-equal-to constraints by multiplying these constraints through by $-1$. That is, any inequality of the form

\sum_{j=1}^{n} a_{ij} x_j \ge b_i

i se q u i v a l e n tt o

\sum_{j=1}^{n} -a_{ij} x_j \le -b_i .

Thus, by replacing each coefficient $a_{ij}$ by $-a_{ij}$ and each value $b_i$ by $-b_i$, we obtain an equivalent less-than-or-equal-to constraint. Finishing our example, we replace the equality in constraint (29.22) by two inequalities, obtaining

\begin{aligned} \text{maximize}\quad & 2x_1 - 3x_2’ + 3x_2”\ \text{subject to}\quad & \ & x_1 + x_2’ - x_2” \le 7\ & x_1 + x_2’ - x_2” \ge 7 && (29.23)\ & x_1 - 2x_2’ + 2x_2” \le 4\ & x_1, x_2’, x_2” \ge 0 . \end{aligned}

Finally, we negate constraint (29.23). For consistency in variable names, we rename $x_2'$ to $x_2$ and $x_2''$ to $x_3$, obtaining the standard form

\begin{aligned} \text{maximize}\quad & 2x_1 - 3x_2 + 3x_3 && (29.24)\ \text{subject to}\quad & \ & x_1 + x_2 - x_3 \le 7 && (29.25)\ & -x_1 - x_2 + x_3 \le -7 && (29.26)\ & x_1 - 2x_2 + 2x_3 \le 4 && (29.27)\ & x_1, x_2, x_3 \ge 0 . && (29.28) \end{aligned}

### Converting linear programs into slack form To efficiently solve a linear program with the simplex algorithm, we prefer to express it in a form in which some of the constraints are equality constraints. More precisely, we shall convert it into a form in which the nonnegativity constraints are the only inequality constraints, and the remaining constraints are equalities. Let

\sum_{j=1}^{n} a_{ij} x_j \le b_i \tag{29.29}

be an inequality constraint. We introduce a new variable $s$ and rewrite inequality (29.29) as the two constraints

s = b_i - \sum_{j=1}^{n} a_{ij} x_j , \tag{29.30}

s \ge 0 . \tag{29.31}

We call $s$ a **slack variable** because it measures the slack, or difference, between the left-hand and right-hand sides of inequality (29.29). (We shall soon see why we find it convenient to write the constraint with only the slack variable on the left-hand side.) Because inequality (29.29) is true if and only if both equation (29.30) and inequality (29.31) are true, we can convert each inequality constraint of a linear program in this way to obtain an equivalent linear program in which the only inequality constraints are the nonnegativity constraints. When converting from standard to slack form, we shall use $x_{n+i}$ (instead of $s$) to denote the slack variable associated with the $i$th inequality. The $i$th constraint is therefore

x_{n+i} = b_i - \sum_{j=1}^{n} a_{ij} x_j , \tag{29.32}

along with the nonnegativity constraint $x_{n+i} \ge 0$. By converting each constraint of a linear program in standard form, we obtain a linear program in a different form. For example, for the linear program described in (29.24)-(29.28), we introduce slack variables $x_4$, $x_5$, and $x_6$, obtaining

\begin{aligned} \text{maximize}\quad & 2x_1 - 3x_2 + 3x_3 && (29.33)\ \text{subject to}\quad & \ & x_4 = 7 - x_1 - x_2 + x_3 && (29.34)\ & x_5 = -7 + x_1 + x_2 - x_3 && (29.35)\ & x_6 = 4 - x_1 + 2x_2 - 2x_3 && (29.36)\ & x_1, x_2, x_3, x_4, x_5, x_6 \ge 0 . && (29.37) \end{aligned}

In this linear program, all the constraints except for the nonnegativity constraints are equalities, and each variable is subject to a nonnegativity constraint. We write each equality constraint with one of the variables on the left-hand side of the equality and all others on the right-hand side. Furthermore, each equation has the same set of variables on the right-hand side, and these variables are also the only ones that appear in the objective function. We call the variables on the left-hand side of the equalities **basic variables** and those on the right-hand side **nonbasic variables**. For linear programs that satisfy these conditions, we shall sometimes omit the words "maximize" and "subject to," as well as the explicit nonnegativity constraints. We shall also use the variable $z$ to denote the value of the objective function. We call the resulting format **slack form**. If we write the linear program given in (29.33)-(29.37) in slack form, we obtain

\begin{aligned} z &= \phantom{-7}; 2x_1 - 3x_2 + 3x_3 && (29.38)\ x_4 &= \phantom{-}7 - x_1 - x_2 + x_3 && (29.39)\ x_5 &= -7 + x_1 + x_2 - x_3 && (29.40)\ x_6 &= \phantom{-}4 - x_1 + 2x_2 - 2x_3 . && (29.41) \end{aligned}

As with standard form, we find it convenient to have a more concise notation for describing a slack form. As we shall see in Section 29.3, the sets of basic and nonbasic variables will change as the simplex algorithm runs. We use $N$ to denote the set of indices of the nonbasic variables and $B$ to denote the set of indices of the basic variables. We always have that $|N| = n$, $|B| = m$, and $N \cup B = \{1, 2, \ldots, n + m\}$. The equations are indexed by the entries of $B$, and the variables on the right-hand sides are indexed by the entries of $N$. As in standard form, we use $b_i$, $c_j$, and $a_{ij}$ to denote constant terms and coefficients. We also use $v$ to denote an optional constant term in the objective function. (We shall see a little later that including the constant term in the objective function makes it easy to determine the value of the objective function.) Thus we can concisely define a slack form by a tuple $(N, B, A, b, c, v)$, denoting the slack form

\begin{aligned} z &= v + \sum_{j \in N} c_j x_j && (29.42)\ x_i &= b_i - \sum_{j \in N} a_{ij} x_j \quad \text{for } i \in B , && (29.43) \end{aligned}

in which all variables $x$ are constrained to be nonnegative. Because we subtract the sum $\sum_{j \in N} a_{ij} x_j$ in (29.43), the values $a_{ij}$ are actually the negatives of the coefficients as they appear in the slack form. For example, in the slack form

\begin{aligned} z &= 28 - \frac{x_3}{6} - \frac{x_5}{6} - \frac{2x_6}{3}\ x_1 &= 8 + \frac{x_3}{6} + \frac{x_5}{6} - \frac{x_6}{3}\ x_2 &= 4 - \frac{8x_3}{3} - \frac{2x_5}{3} + \frac{x_6}{3}\ x_4 &= 18 - \frac{x_3}{2} + \frac{x_5}{2} , \end{aligned}

we have $B = \{1, 2, 4\}$, $N = \{3, 5, 6\}$,

A = \begin{pmatrix} a_{13} & a_{15} & a_{16}\ a_{23} & a_{25} & a_{26}\ a_{43} & a_{45} & a_{46} \end{pmatrix} = \begin{pmatrix} -1/6 & -1/6 & 1/3\ 8/3 & 2/3 & -1/3\ 1/2 & -1/2 & 0 \end{pmatrix}, \qquad b = \begin{pmatrix} b_1\ b_2\ b_4 \end{pmatrix} = \begin{pmatrix} 8\ 4\ 18 \end{pmatrix},

$c = (c_3\;\; c_5\;\; c_6)^{\mathsf T} = (-1/6\;\; -1/6\;\; -2/3)^{\mathsf T}$, and $v = 28$. Note that the indices into $A$, $b$, and $c$ are not necessarily sets of contiguous integers; they depend on the index sets $B$ and $N$. As an example of the entries of $A$ being the negatives of the coefficients as they appear in the slack form, observe that the equation for $x_1$ includes the term $x_3/6$, yet the coefficient $a_{13}$ is actually $-1/6$ rather than $+1/6$. > **Think about it.** Slack literally means "wiggle room." If a slack variable equals zero, what does that say about its original inequality? (It is *tight*: the point sits right on that boundary.) ### Exercises 29.1-1 to 29.1-9 **29.1-1** If we express the linear program in (29.24)-(29.28) in the compact notation of (29.19)-(29.21), what are $n$, $m$, $A$, $b$, and $c$? **29.1-2** Give three feasible solutions to the linear program in (29.24)-(29.28). What is the objective value of each one? **29.1-3** For the slack form in (29.38)-(29.41), what are $N$, $B$, $A$, $b$, $c$, and $v$? **29.1-4** Convert the following linear program into standard form:

\begin{aligned} \text{minimize}\quad & 2x_1 + 7x_2 + x_3\ \text{subject to}\quad & \ & x_1 - x_3 = 7\ & 3x_1 + x_2 \ge 24\ & x_2 \ge 0\ & x_3 \le 0 \end{aligned}

* * 29.1 - 5 * * C o n v er tt h e f o l l o w in g l in e a r p r o g r amin t os l a c k f or m :

\begin{aligned} \text{maximize}\quad & 2x_1 - 6x_3\ \text{subject to}\quad & \ & x_1 + x_2 - x_3 \le 7\ & 3x_1 - x_2 \ge 8\ & -x_1 + 2x_2 + 2x_3 \ge 0\ & x_1, x_2, x_3 \ge 0 \end{aligned}

W ha t a r e t h e ba s i c an d n o nba s i c v a r iab l es ? * * 29.1 - 6 * * S h o w t ha tt h e f o l l o w in g l in e a r p r o g r ami s in f e a s ib l e :

\begin{aligned} \text{maximize}\quad & 3x_1 - 2x_2\ \text{subject to}\quad & \ & x_1 + x_2 \le 2\ & -2x_1 - 2x_2 \le -10\ & x_1, x_2 \ge 0 \end{aligned}

* * 29.1 - 7 * * S h o w t ha tt h e f o l l o w in g l in e a r p r o g r ami s u nb o u n d e d :

\begin{aligned} \text{maximize}\quad & x_1 - x_2\ \text{subject to}\quad & \ & -2x_1 + x_2 \le -1\ & -x_1 - 2x_2 \le -2\ & x_1, x_2 \ge 0 \end{aligned}

**29.1-8** Suppose that we have a general linear program with $n$ variables and $m$ constraints, and suppose that we convert it into standard form. Give an upper bound on the number of variables and constraints in the resulting linear program. **29.1-9** Give an example of a linear program for which the feasible region is not bounded, but the optimal objective value is finite. ## 29.2 Formulating problems as linear programs Although we shall focus on the simplex algorithm in this chapter, it is also important to be able to recognize when we can formulate a problem as a linear program. Once we cast a problem as a polynomial-sized linear program, we can solve it in polynomial time by the ellipsoid algorithm or interior-point methods. Several linear-programming software packages can solve problems efficiently, so that once the problem is in the form of a linear program, such a package can solve it. We shall look at several concrete examples of linear-programming problems. We start with two problems that we have already studied: the single-source shortest-paths problem (see Chapter 24) and the maximum-flow problem (see Chapter 26). We then describe the minimum-cost-flow problem. Although the minimum-cost-flow problem has a polynomial-time algorithm that is not based on linear programming, we won't describe the algorithm. Finally, we describe the multicommodity-flow problem, for which the only known polynomial-time algorithm is based on linear programming. When we solved graph problems in Part VI, we used attribute notation, such as $v.d$ and $(u,v).f$. Linear programs typically use subscripted variables rather than objects with attached attributes, however. Therefore, when we express variables in linear programs, we shall indicate vertices and edges through subscripts. For example, we denote the shortest-path weight for vertex $v$ not by $v.d$ but by $d_v$. And we denote the flow from vertex $u$ to vertex $v$ not by $(u,v).f$ but by $f_{uv}$. For quantities that are given as inputs to problems, such as edge weights or capacities, we shall continue to use notations such as $w(u,v)$ and $c(u,v)$. > **Think about it.** Each of these graph problems already has a famous dedicated algorithm. What do we gain by re-expressing them as linear programs? (Hint: the power to bolt on *new* constraints.) ### Shortest paths We can formulate the single-source shortest-paths problem as a linear program. In this section, we shall focus on how to formulate the single-pair shortest-path problem, leaving the extension to the more general single-source shortest-paths problem as Exercise 29.2-3. In the single-pair shortest-path problem, we are given a weighted, directed graph $G = (V, E)$, with weight function $w : E \to \mathbb{R}$ mapping edges to real-valued weights, a source vertex $s$, and destination vertex $t$. We wish to compute the value $d_t$, which is the weight of a shortest path from $s$ to $t$. To express this problem as a linear program, we need to determine a set of variables and constraints that define when we have a shortest path from $s$ to $t$. Fortunately, the Bellman-Ford algorithm does exactly this. When the Bellman-Ford algorithm terminates, it has computed, for each vertex $v$, a value $d_v$ (using subscript notation here rather than attribute notation) such that for each edge $(u,v) \in E$, we have $d_v \le d_u + w(u,v)$. The source vertex initially receives a value $d_s = 0$, which never changes. Thus we obtain the following linear program to compute the shortest-path weight from $s$ to $t$:

\begin{aligned} \text{maximize}\quad & d_t && (29.44)\ \text{subject to}\quad & \ & d_v \le d_u + w(u,v) \quad \text{for each edge } (u,v) \in E , && (29.45)\ & d_s = 0 . && (29.46) \end{aligned}

You might be surprised that this linear program maximizes an objective function when it is supposed to compute shortest paths. We do not want to minimize the objective function, since then setting $d_v = 0$ for all $v \in V$ would yield an optimal solution to the linear program without solving the shortest-paths problem. We maximize because an optimal solution to the shortest-paths problem sets each $d_v$ to $\min_{u : (u,v) \in E} \{ d_u + w(u,v) \}$, so that $d_v$ is the largest value that is less than or equal to all of the values in the set $\{ d_u + w(u,v) \}$. We want to maximize $d_v$ for all vertices $v$ on a shortest path from $s$ to $t$ subject to these constraints on all vertices $v$, and maximizing $d_t$ achieves this goal. This linear program has $|V|$ variables $d_v$, one for each vertex $v \in V$. It also has $|E| + 1$ constraints: one for each edge, plus the additional constraint that the source vertex's shortest-path weight always has the value 0. ### Maximum flow Next, we express the maximum-flow problem as a linear program. Recall that we are given a directed graph $G = (V, E)$ in which each edge $(u,v) \in E$ has a nonnegative capacity $c(u,v) \ge 0$, and two distinguished vertices: a source $s$ and a sink $t$. As defined in Section 26.1, a flow is a nonnegative real-valued function $f : V \times V \to \mathbb{R}$ that satisfies the capacity constraint and flow conservation. A maximum flow is a flow that satisfies these constraints and maximizes the flow value, which is the total flow coming out of the source minus the total flow into the source. A flow, therefore, satisfies linear constraints, and the value of a flow is a linear function. Recalling also that we assume that $c(u,v) = 0$ if $(u,v) \notin E$ and that there are no antiparallel edges, we can express the maximum-flow problem as a linear program:

\begin{aligned} \text{maximize}\quad & \sum_{v \in V} f_{sv} - \sum_{v \in V} f_{vs} && (29.47)\ \text{subject to}\quad & \ & f_{uv} \le c(u,v) \quad \text{for each } u, v \in V , && (29.48)\ & \sum_{v \in V} f_{vu} = \sum_{v \in V} f_{uv} \quad \text{for each } u \in V - {s, t} , && (29.49)\ & f_{uv} \ge 0 \quad \text{for each } u, v \in V . && (29.50) \end{aligned}

This linear program has $|V|^2$ variables, corresponding to the flow between each pair of vertices, and it has $2|V|^2 + |V| - 2$ constraints. It is usually more efficient to solve a smaller-sized linear program. The linear program in (29.47)-(29.50) has, for ease of notation, a flow and capacity of 0 for each pair of vertices $u, v$ with $(u,v) \notin E$. It would be more efficient to rewrite the linear program so that it has $O(V + E)$ constraints. Exercise 29.2-5 asks you to do so. ### Minimum-cost flow In this section, we have used linear programming to solve problems for which we already knew efficient algorithms. In fact, an efficient algorithm designed specifically for a problem, such as Dijkstra's algorithm for the single-source shortest-paths problem, or the push-relabel method for maximum flow, will often be more efficient than linear programming, both in theory and in practice. The real power of linear programming comes from the ability to solve new problems. Recall the problem faced by the politician in the beginning of this chapter. The problem of obtaining a sufficient number of votes, while not spending too much money, is not solved by any of the algorithms that we have studied in this book, yet we can solve it by linear programming. Books abound with such real-world problems that linear programming can solve. Linear programming is also particularly useful for solving variants of problems for which we may not already know of an efficient algorithm. Consider, for example, the following generalization of the maximum-flow problem. Suppose that, in addition to a capacity $c(u,v)$ for each edge $(u,v)$, we are given a real-valued cost $a(u,v)$. As in the maximum-flow problem, we assume that $c(u,v) = 0$ if $(u,v) \notin E$, and that there are no antiparallel edges. If we send $f_{uv}$ units of flow over edge $(u,v)$, we incur a cost of $a(u,v) f_{uv}$. We are also given a flow demand $d$. We wish to send $d$ units of flow from $s$ to $t$ while minimizing the total cost $\sum_{(u,v) \in E} a(u,v) f_{uv}$ incurred by the flow. This problem is known as the **minimum-cost-flow problem**. Figure 29.3(a) shows an example of the minimum-cost-flow problem. We wish to send 4 units of flow from $s$ to $t$ while incurring the minimum total cost. Any particular legal flow, that is, a function $f$ satisfying constraints (29.48)-(29.50), incurs a total cost of $\sum_{(u,v) \in E} a(u,v) f_{uv}$. We wish to find the particular 4-unit flow that minimizes this cost. Figure 29.3(b) shows an optimal solution, with total cost

\sum_{(u,v) \in E} a(u,v) f_{uv} = (2 \cdot 2) + (5 \cdot 2) + (3 \cdot 1) + (7 \cdot 1) + (1 \cdot 3) = 27 .

*Figure 29.3: (a) An example of a minimum-cost-flow problem. We denote the capacities by $c$ and the costs by $a$. Vertex $s$ is the source and vertex $t$ is the sink, and we wish to send 4 units of flow from $s$ to $t$. (b) A solution to the minimum-cost flow problem in which 4 units of flow are sent from $s$ to $t$. For each edge, the flow and capacity are written as flow/capacity.* (Two small flow networks on source $s$ and sink $t$; edges in (a) are labeled with capacity $c$ and cost $a$, and in (b) with flow/capacity.) There are polynomial-time algorithms specifically designed for the minimum-cost-flow problem, but they are beyond the scope of this book. We can, however, express the minimum-cost-flow problem as a linear program. The linear program looks similar to the one for the maximum-flow problem with the additional constraint that the value of the flow be exactly $d$ units, and with the new objective function of minimizing the cost:

\begin{aligned} \text{minimize}\quad & \sum_{(u,v) \in E} a(u,v) f_{uv} && (29.51)\ \text{subject to}\quad & \ & f_{uv} \le c(u,v) \quad \text{for each } u, v \in V ,\ & \sum_{v \in V} f_{vu} - \sum_{v \in V} f_{uv} = 0 \quad \text{for each } u \in V - {s, t} ,\ & \sum_{v \in V} f_{sv} - \sum_{v \in V} f_{vs} = d ,\ & f_{uv} \ge 0 \quad \text{for each } u, v \in V . && (29.52) \end{aligned}

### Multicommodity flow As a final example, we consider another flow problem. Suppose that the Lucky Puck company from Section 26.1 decides to diversify its product line and ship not only hockey pucks, but also hockey sticks and hockey helmets. Each piece of equipment is manufactured in its own factory, has its own warehouse, and must be shipped, each day, from factory to warehouse. The sticks are manufactured in Vancouver and must be shipped to Saskatoon, and the helmets are manufactured in Edmonton and must be shipped to Regina. The capacity of the shipping network does not change, however, and the different items, or commodities, must share the same network. This example is an instance of a **multicommodity-flow problem**. In this problem, we are again given a directed graph $G = (V, E)$ in which each edge $(u,v) \in E$ has a nonnegative capacity $c(u,v) \ge 0$. As in the maximum-flow problem, we implicitly assume that $c(u,v) = 0$ for $(u,v) \notin E$, and that the graph has no antiparallel edges. In addition, we are given $k$ different commodities, $K_1, K_2, \ldots, K_k$, where we specify commodity $i$ by the triple $K_i = (s_i, t_i, d_i)$. Here, vertex $s_i$ is the source of commodity $i$, vertex $t_i$ is the sink of commodity $i$, and $d_i$ is the demand for commodity $i$, which is the desired flow value for the commodity from $s_i$ to $t_i$. We define a flow for commodity $i$, denoted by $f_i$ (so that $f_{iuv}$ is the flow of commodity $i$ from vertex $u$ to vertex $v$) to be a real-valued function that satisfies the flow-conservation and capacity constraints. We now define $f_{uv}$, the **aggregate flow**, to be $\sum_{i=1}^{k} f_{iuv}$, the sum of the flows of the various commodities along edge $(u,v)$. The aggregate flow on edge $(u,v)$ must be no more than the capacity of edge $(u,v)$. We are not trying to minimize any objective function in this problem; we need only determine whether such a flow exists. Thus, we write a linear program with a "null" objective function:

\begin{aligned} \text{minimize}\quad & 0\ \text{subject to}\quad & \ & \sum_{i=1}^{k} f_{iuv} \le c(u,v) \quad \text{for each } u, v \in V ,\ & \sum_{v \in V} f_{iuv} - \sum_{v \in V} f_{ivu} = 0 \quad \text{for each } i = 1, 2, \ldots, k \text{ and for each } u \in V - {s_i, t_i} ,\ & \sum_{v \in V} f_{i s_i v} - \sum_{v \in V} f_{i v s_i} = d_i \quad \text{for each } i = 1, 2, \ldots, k ,\ & f_{iuv} \ge 0 \quad \text{for each } u, v \in V \text{ and for each } i = 1, 2, \ldots, k . \end{aligned}

The only known polynomial-time algorithm for this problem expresses it as a linear program and then solves it with a polynomial-time linear-programming algorithm. ### Exercises 29.2-1 to 29.2-7 **29.2-1** Put the single-pair shortest-path linear program from (29.44)-(29.46) into standard form. **29.2-2** Write out explicitly the linear program corresponding to finding the shortest path from node $s$ to node $y$ in Figure 24.2(a). **29.2-3** In the single-source shortest-paths problem, we want to find the shortest-path weights from a source vertex $s$ to all vertices $v \in V$. Given a graph $G$, write a linear program for which the solution has the property that $d_v$ is the shortest-path weight from $s$ to $v$ for each vertex $v \in V$. **29.2-4** Write out explicitly the linear program corresponding to finding the maximum flow in Figure 26.1(a). **29.2-5** Rewrite the linear program for maximum flow (29.47)-(29.50) so that it uses only $O(V + E)$ constraints. **29.2-6** Write a linear program that, given a bipartite graph $G = (V, E)$, solves the maximum-bipartite-matching problem. **29.2-7** In the **minimum-cost multicommodity-flow problem**, we are given directed graph $G = (V, E)$ in which each edge $(u,v) \in E$ has a nonnegative capacity $c(u,v) \ge 0$ and a cost $a(u,v)$. As in the multicommodity-flow problem, we are given $k$ different commodities, $K_1, K_2, \ldots, K_k$, where we specify commodity $i$ by the triple $K_i = (s_i, t_i, d_i)$. We define the flow $f_i$ for commodity $i$ and the aggregate flow $f_{uv}$ on edge $(u,v)$ as in the multicommodity-flow problem. A feasible flow is one in which the aggregate flow on each edge $(u,v)$ is no more than the capacity of edge $(u,v)$. The cost of a flow is $\sum_{u,v \in V} a(u,v) f_{uv}$, and the goal is to find the feasible flow of minimum cost. Express this problem as a linear program. ## 29.3 The simplex algorithm The simplex algorithm is the classical method for solving linear programs. In contrast to most of the other algorithms in this book, its running time is not polynomial in the worst case. It does yield insight into linear programs, however, and is often remarkably fast in practice. In addition to having a geometric interpretation, described earlier in this chapter, the simplex algorithm bears some similarity to Gaussian elimination, discussed in Section 28.1. Gaussian elimination begins with a system of linear equalities whose solution is unknown. In each iteration, we rewrite this system in an equivalent form that has some additional structure. After some number of iterations, we have rewritten the system so that the solution is simple to obtain. The simplex algorithm proceeds in a similar manner, and we can view it as Gaussian elimination for inequalities. We now describe the main idea behind an iteration of the simplex algorithm. Associated with each iteration will be a "basic solution" that we can easily obtain from the slack form of the linear program: set each nonbasic variable to 0 and compute the values of the basic variables from the equality constraints. An iteration converts one slack form into an equivalent slack form. The objective value of the associated basic feasible solution will be no less than that at the previous iteration, and usually greater. To achieve this increase in the objective value, we choose a nonbasic variable such that if we were to increase that variable's value from 0, then the objective value would increase, too. The amount by which we can increase the variable is limited by the other constraints. In particular, we raise it until some basic variable becomes 0. We then rewrite the slack form, exchanging the roles of that basic variable and the chosen nonbasic variable. Although we have used a particular setting of the variables to guide the algorithm, and we shall use it in our proofs, the algorithm does not explicitly maintain this solution. It simply rewrites the linear program until an optimal solution becomes "obvious." ### An example of the simplex algorithm We begin with an extended example. Consider the following linear program in standard form:

\begin{aligned} \text{maximize}\quad & 3x_1 + x_2 + 2x_3 && (29.53)\ \text{subject to}\quad & \ & x_1 + x_2 + 3x_3 \le 30 && (29.54)\ & 2x_1 + 2x_2 + 5x_3 \le 24 && (29.55)\ & 4x_1 + x_2 + 2x_3 \le 36 && (29.56)\ & x_1, x_2, x_3 \ge 0 . && (29.57) \end{aligned}

In order to use the simplex algorithm, we must convert the linear program into slack form; we saw how to do so in Section 29.1. In addition to being an algebraic manipulation, slack is a useful algorithmic concept. Recalling from Section 29.1 that each variable has a corresponding nonnegativity constraint, we say that an equality constraint is **tight** for a particular setting of its nonbasic variables if they cause the constraint's basic variable to become 0. Similarly, a setting of the nonbasic variables that would make a basic variable become negative **violates** that constraint. Thus, the slack variables explicitly maintain how far each constraint is from being tight, and so they help to determine how much we can increase values of nonbasic variables without violating any constraints. Associating the slack variables $x_4$, $x_5$, and $x_6$ with inequalities (29.54)-(29.56), respectively, and putting the linear program into slack form, we obtain

\begin{aligned} z &= 3x_1 + x_2 + 2x_3 && (29.58)\ x_4 &= 30 - x_1 - x_2 - 3x_3 && (29.59)\ x_5 &= 24 - 2x_1 - 2x_2 - 5x_3 && (29.60)\ x_6 &= 36 - 4x_1 - x_2 - 2x_3 . && (29.61) \end{aligned}

The system of constraints (29.59)-(29.61) has 3 equations and 6 variables. Any setting of the variables $x_1$, $x_2$, and $x_3$ defines values for $x_4$, $x_5$, and $x_6$; therefore, we have an infinite number of solutions to this system of equations. A solution is feasible if all of $x_1, x_2, \ldots, x_6$ are nonnegative, and there can be an infinite number of feasible solutions as well. The infinite number of possible solutions to a system such as this one will be useful in later proofs. We focus on the **basic solution**: set all the (nonbasic) variables on the right-hand side to 0 and then compute the values of the (basic) variables on the left-hand side. In this example, the basic solution is $(x_1, x_2, \ldots, x_6) = (0, 0, 0, 30, 24, 36)$ and it has objective value

z = (3 \cdot 0) + (1 \cdot 0) + (2 \cdot 0) = 0 .

Observe that this basic solution sets $x_i = b_i$ for each $i \in B$. An iteration of the simplex algorithm rewrites the set of equations and the objective function so as to put a different set of variables on the right-hand side. Thus, a different basic solution is associated with the rewritten problem. We emphasize that the rewrite does not in any way change the underlying linear-programming problem; the problem at one iteration has the identical set of feasible solutions as the problem at the previous iteration. The problem does, however, have a different basic solution than that of the previous iteration. If a basic solution is also feasible, we call it a **basic feasible solution**. As we run the simplex algorithm, the basic solution is almost always a basic feasible solution. We shall see in Section 29.5, however, that for the first few iterations of the simplex algorithm, the basic solution might not be feasible. Our goal, in each iteration, is to reformulate the linear program so that the basic solution has a greater objective value. We select a nonbasic variable $x_e$ whose coefficient in the objective function is positive, and we increase the value of $x_e$ as much as possible without violating any of the constraints. The variable $x_e$ becomes basic, and some other variable $x_l$ becomes nonbasic. The values of other basic variables and of the objective function may also change. **Example (first pivot).** Let us think about increasing the value of $x_1$. As we increase $x_1$ (keeping $x_2 = x_3 = 0$), the values of $x_4$, $x_5$, and $x_6$ all decrease. Because we have a nonnegativity constraint for each variable, we cannot allow any of them to become negative. Each equation gives a bound:

\begin{aligned} x_4 = 30 - x_1 \ge 0 &;\Longrightarrow; x_1 \le 30 ,\ x_5 = 24 - 2x_1 \ge 0 &;\Longrightarrow; x_1 \le 12 ,\ x_6 = 36 - 4x_1 \ge 0 &;\Longrightarrow; x_1 \le 9 . \end{aligned}

If $x_1$ increases above 30, then $x_4$ becomes negative, and $x_5$ and $x_6$ become negative when $x_1$ increases above 12 and 9, respectively. The third constraint (29.61) is the tightest constraint, and it limits how much we can increase $x_1$. Therefore, we switch the roles of $x_1$ and $x_6$. We solve equation (29.61) for $x_1$ and obtain

x_1 = 9 - \frac{x_2}{4} - \frac{x_3}{2} - \frac{x_6}{4} . \tag{29.62}

To rewrite the other equations with $x_6$ on the right-hand side, we substitute for $x_1$ using equation (29.62). Doing so for equation (29.59), we obtain

\begin{aligned} x_4 &= 30 - x_1 - x_2 - 3x_3\ &= 30 - \left( 9 - \frac{x_2}{4} - \frac{x_3}{2} - \frac{x_6}{4} \right) - x_2 - 3x_3 && \text{(substitute (29.62))}\ &= 21 + \frac{x_2}{4} - x_2 + \frac{x_3}{2} - 3x_3 + \frac{x_6}{4} && \text{(distribute the minus sign)}\ &= 21 - \frac{3x_2}{4} - \frac{5x_3}{2} + \frac{x_6}{4} . && (29.63) \end{aligned}

S imi l a r l y, w eco mbin ee q u a t i o n (29.62) w i t h co n s t r ain t (29.60) :

\begin{aligned} x_5 &= 24 - 2x_1 - 2x_2 - 5x_3\ &= 24 - 2\left( 9 - \frac{x_2}{4} - \frac{x_3}{2} - \frac{x_6}{4} \right) - 2x_2 - 5x_3\ &= 6 + \frac{x_2}{2} - 2x_2 + x_3 - 5x_3 + \frac{x_6}{2}\ &= 6 - \frac{3x_2}{2} - 4x_3 + \frac{x_6}{2} , \end{aligned}

an d w i t h o bj ec t i v e f u n c t i o n (29.58) :

\begin{aligned} z &= 3x_1 + x_2 + 2x_3\ &= 3\left( 9 - \frac{x_2}{4} - \frac{x_3}{2} - \frac{x_6}{4} \right) + x_2 + 2x_3\ &= 27 - \frac{3x_2}{4} - \frac{3x_3}{2} - \frac{3x_6}{4} + x_2 + 2x_3\ &= 27 + \frac{x_2}{4} + \frac{x_3}{2} - \frac{3x_6}{4} . \end{aligned}

T h u s w er e w r i t eo u r l in e a r p r o g r amin t h e f o l l o w in g f or m :

\begin{aligned} z &= 27 + \frac{x_2}{4} + \frac{x_3}{2} - \frac{3x_6}{4} && (29.64)\ x_1 &= 9 - \frac{x_2}{4} - \frac{x_3}{2} - \frac{x_6}{4} && (29.65)\ x_4 &= 21 - \frac{3x_2}{4} - \frac{5x_3}{2} + \frac{x_6}{4} && (29.66)\ x_5 &= 6 - \frac{3x_2}{2} - 4x_3 + \frac{x_6}{2} . && (29.67) \end{aligned}

We call this operation a **pivot**. As demonstrated above, a pivot chooses a nonbasic variable $x_e$, called the **entering variable**, and a basic variable $x_l$, called the **leaving variable**, and exchanges their roles. The linear program described in equations (29.64)-(29.67) is equivalent to the linear program described in equations (29.58)-(29.61). We perform two operations in the simplex algorithm: rewrite equations so that variables move between the left-hand side and the right-hand side, and substitute one equation into another. The first operation trivially creates an equivalent problem, and the second, by elementary linear algebra, also creates an equivalent problem. (See Exercise 29.3-3.) To demonstrate this equivalence, observe that our original basic solution $(0, 0, 0, 30, 24, 36)$ satisfies the new equations (29.65)-(29.67) and has objective value $27 + (1/4)\cdot 0 + (1/2)\cdot 0 - (3/4)\cdot 36 = 0$. The basic solution associated with the new linear program sets the nonbasic values to 0 and is $(9, 0, 0, 21, 6, 0)$, with objective value $z = 27$. Simple arithmetic verifies that this solution also satisfies equations (29.59)-(29.61) and, when plugged into objective function (29.58), has objective value $(3 \cdot 9) + (1 \cdot 0) + (2 \cdot 0) = 27$. > **Think about it.** A pivot is like trading seats at a table: one guest (a nonbasic variable) sits down at the head, and one who was seated (a basic variable) steps away to zero. The meal (the linear program) stays exactly the same. Continuing the example, we wish to find a new variable whose value we wish to increase. We do not want to increase $x_6$, since as its value increases, the objective value decreases. We can attempt to increase either $x_2$ or $x_3$; let us choose $x_3$. How far can we increase $x_3$ without violating any of the constraints? Setting $x_2 = x_6 = 0$ in (29.65)-(29.67), each constraint gives a bound:

\begin{aligned} x_1 = 9 - \frac{x_3}{2} \ge 0 &;\Longrightarrow; x_3 \le 18 ,\ x_4 = 21 - \frac{5x_3}{2} \ge 0 &;\Longrightarrow; x_3 \le \frac{42}{5} ,\ x_5 = 6 - 4x_3 \ge 0 &;\Longrightarrow; x_3 \le \frac{3}{2} . \end{aligned}

Constraint (29.65) limits it to 18, constraint (29.66) limits it to $42/5$, and constraint (29.67) limits it to $3/2$. The third constraint is again the tightest one, and therefore we rewrite the third constraint so that $x_3$ is on the left-hand side and $x_5$ is on the right-hand side. ### The example, finishing the pivots *Big picture: imagine rolling a ball along the edges of a many-sided gem, always to a neighbouring corner that sits higher, until no neighbour is higher. That is simplex. Each "pivot" is one step from corner to corner, carried out as pure algebra.* We continue the worked example. Solve for the entering variable's equation and substitute: from $x_3 = 3/2 - 3x_2/8 - x_5/4 + x_6/8$ we substitute into equations (29.64)-(29.66) and obtain the new, equivalent system

\begin{aligned} z &= \frac{111}{4} + \frac{x_2}{16} - \frac{x_5}{8} - \frac{11x_6}{16} && (29.68)\ x_1 &= \frac{33}{4} - \frac{x_2}{16} + \frac{x_5}{8} - \frac{5x_6}{16} && (29.69)\ x_3 &= \frac{3}{2} - \frac{3x_2}{8} - \frac{x_5}{4} + \frac{x_6}{8} && (29.70)\ x_4 &= \frac{69}{4} + \frac{3x_2}{16} + \frac{5x_5}{8} - \frac{x_6}{16} && (29.71) \end{aligned}

This system has the associated basic solution $(33/4, 0, 3/2, 69/4, 0, 0)$ with objective value $111/4$. Now the only way to increase the objective value is to increase $x_2$. The three constraints give upper bounds of $132$, $4$, and $\infty$, respectively. (We get $\infty$ from constraint (29.71) because, as we increase $x_2$, the basic variable $x_4$ increases too, so this constraint places no restriction.) We increase $x_2$ to $4$, and it becomes basic. Then we solve equation (29.70) for $x_2$ and substitute in the other equations to obtain

\begin{aligned} z &= 28 - \frac{x_3}{6} - \frac{x_5}{6} - \frac{2x_6}{3} && (29.72)\ x_1 &= 8 + \frac{x_3}{6} + \frac{x_5}{6} - \frac{x_6}{3} && (29.73)\ x_2 &= 4 - \frac{8x_3}{3} - \frac{2x_5}{3} + \frac{x_6}{3} && (29.74)\ x_4 &= 18 - \frac{x_3}{2} + \frac{x_5}{2} && (29.75) \end{aligned}

\begin{aligned} z &= x_1 + x_2 + x_3\ x_4 &= 8 - x_1 - x_2\ x_5 &= x_2 - x_3 . \end{aligned}

Suppose that we choose $x_1$ as the entering variable and $x_4$ as the leaving variable. After pivoting, we obtain

\begin{aligned} z &= 8 + x_3 - x_4\ x_1 &= 8 - x_2 - x_4\ x_5 &= x_2 - x_3 . \end{aligned}

At this point, our only choice is to pivot with $x_3$ entering and $x_5$ leaving. Since $b_5 = 0$, the objective value of $8$ remains unchanged after pivoting:

\begin{aligned} z &= 8 + x_2 - x_4 - x_5\ x_1 &= 8 - x_2 - x_4\ x_3 &= x_2 - x_5 . \end{aligned}

The objective value has not changed, but our slack form has. Fortunately, if we pivot again, with $x_2$ entering and $x_1$ leaving, the objective value increases (to $16$), and the simplex algorithm can continue. Degeneracy can prevent the simplex algorithm from terminating, because it can lead to a phenomenon known as **cycling**: the slack forms at two different iterations of SIMPLEX are identical. Because of degeneracy, SIMPLEX could choose a sequence of pivot operations that leave the objective value unchanged but repeat a slack form within the sequence. Since SIMPLEX is a deterministic algorithm, if it cycles, then it will cycle through the same series of slack forms forever, never terminating. *Think about it: walking on a perfectly flat plateau, how would you know whether you are making progress or circling the same spot?* To show that cycling is the only way SIMPLEX can fail to terminate, we must first develop some additional machinery. At each iteration, SIMPLEX maintains $A$, $b$, $c$, and $v$ in addition to the sets $N$ and $B$. Although we need to explicitly maintain $A$, $b$, $c$, and $v$ in order to implement the simplex algorithm efficiently, we can get by without maintaining them. In other words, the sets of basic and nonbasic variables suffice to uniquely determine the slack form. Before proving this fact, we prove a useful algebraic lemma. > **Lemma 29.3.** Let $I$ be a set of indices. For each $j \in I$, let $\alpha_j$ and $\beta_j$ be real numbers, and let $x_j$ be a real-valued variable. Let $\gamma$ be any real number. Suppose that for any settings of the $x_j$, we have > $$\sum_{j \in I} \alpha_j x_j = \gamma + \sum_{j \in I} \beta_j x_j. \qquad (29.78)$$ > Then $\alpha_j = \beta_j$ for each $j \in I$, and $\gamma = 0$. > **Proof.** Since equation (29.78) holds for any values of the $x_j$, we can use particular values to draw conclusions about $\alpha$, $\beta$, and $\gamma$. > - If we let $x_j = 0$ for each $j \in I$, we conclude that $\gamma = 0$. > - Now pick an arbitrary index $j \in I$, and set $x_j = 1$ and $x_k = 0$ for all $k \ne j$. Then we must have $\alpha_j = \beta_j$. > > Since we picked $j$ as any index in $I$, we conclude that $\alpha_j = \beta_j$ for each $j \in I$. $\blacksquare$ A particular linear program has many different slack forms; recall that each slack form has the same set of feasible and optimal solutions as the original linear program. We now show that the slack form of a linear program is uniquely determined by the set of basic variables. That is, given the set of basic variables, a unique slack form (unique set of coefficients and right-hand sides) is associated with those basic variables. > **Lemma 29.4.** Let $(A, b, c)$ be a linear program in standard form. Given a set $B$ of basic variables, the associated slack form is uniquely determined. > **Proof.** Assume for the purpose of contradiction that there are two different slack forms with the same set $B$ of basic variables. The slack forms must also have identical sets $N = \{1, 2, \ldots, n+m\} - B$ of nonbasic variables. We write the first slack form as > $$ > \begin{aligned} > z &= v + \sum_{j \in N} c_j x_j && (29.79)\\ > x_i &= b_i - \sum_{j \in N} a_{ij}x_j \quad \text{for } i \in B, && (29.80) > \end{aligned} > $$ > and the second as > $$ > \begin{aligned} > z &= v' + \sum_{j \in N} c'_j x_j && (29.81)\\ > x_i &= b'_i - \sum_{j \in N} a'_{ij}x_j \quad \text{for } i \in B. && (29.82) > \end{aligned} > $$ > Consider the system of equations formed by subtracting each equation in line (29.82) from the corresponding equation in line (29.80). The resulting system is > $$0 = (b_i - b'_i) - \sum_{j \in N}(a_{ij} - a'_{ij})x_j \quad \text{for } i \in B$$ > or, equivalently, > $$\sum_{j \in N} a_{ij}x_j = (b_i - b'_i) + \sum_{j \in N} a'_{ij}x_j \quad \text{for } i \in B.$$ > Now, for each $i \in B$, apply Lemma 29.3 with $\alpha_j = a_{ij}$, $\beta_j = a'_{ij}$, $\gamma = b_i - b'_i$, and $I = N$. Since $\alpha_j = \beta_j$, we have that $a_{ij} = a'_{ij}$ for each $j \in N$, and since $\gamma = 0$, we have that $b_i = b'_i$. Thus, for the two slack forms, $A$ and $b$ are identical to $A'$ and $b'$. Using a similar argument, Exercise 29.3-1 shows that it must also be the case that $c = c'$ and $v = v'$, and hence that the slack forms must be identical. $\blacksquare$ We can now show that cycling is the only possible reason that SIMPLEX might not terminate. > **Lemma 29.5.** If SIMPLEX fails to terminate in at most $\binom{n+m}{m}$ iterations, then it cycles. > **Proof.** By Lemma 29.4, the set $B$ of basic variables uniquely determines a slack form. There are $n + m$ variables and $|B| = m$, and therefore there are at most $\binom{n+m}{m}$ unique slack forms. Therefore, if SIMPLEX runs for more than $\binom{n+m}{m}$ iterations, it must cycle (pigeonhole principle). $\blacksquare$ Cycling is theoretically possible, but extremely rare. We can prevent it by choosing the entering and leaving variables somewhat more carefully. One option is to perturb the input slightly so that it is impossible to have two solutions with the same objective value. Another option is to break ties by always choosing the variable with the smallest index, a strategy known as **Bland's rule**. We omit the proof that these strategies avoid cycling. > **Lemma 29.6.** If lines 4 and 9 of SIMPLEX always break ties by choosing the variable with the smallest index, then SIMPLEX must terminate. We conclude this section with the following lemma. > **Lemma 29.7.** Assuming that INITIALIZE-SIMPLEX returns a slack form for which the basic solution is feasible, SIMPLEX either reports that a linear program is unbounded, or it terminates with a feasible solution in at most $\binom{n+m}{m}$ iterations. > **Proof.** Lemmas 29.2 and 29.6 show that if INITIALIZE-SIMPLEX returns a slack form for which the basic solution is feasible, SIMPLEX either reports that a linear program is unbounded, or it terminates with a feasible solution. By the contrapositive of Lemma 29.5, if SIMPLEX terminates with a feasible solution, then it terminates in at most $\binom{n+m}{m}$ iterations. $\blacksquare$ ### Exercises 29.3-1 to 29.3-8 **29.3-1** Complete the proof of Lemma 29.4 by showing that it must be the case that $c = c'$ and $v = v'$. **29.3-2** Show that the call to PIVOT in line 12 of SIMPLEX never decreases the value of $v$. **29.3-3** Prove that the slack form given to the PIVOT procedure and the slack form that the procedure returns are equivalent. **29.3-4** Suppose we convert a linear program $(A, b, c)$ in standard form to slack form. Show that the basic solution is feasible if and only if $b_i \ge 0$ for $i = 1, 2, \ldots, m$. **29.3-5** Solve the following linear program using SIMPLEX:

\begin{aligned} \text{maximize} \quad & 18x_1 + 12.5x_2\ \text{subject to} \quad & x_1 + x_2 \le 20\ & x_1 \le 12\ & x_2 \le 16\ & x_1, x_2 \ge 0 . \end{aligned}

* * 29.3 - 6 * * S o l v e t h e f o l l o w in g l in e a r p r o g r am u s in g S I M P L E X :

\begin{aligned} \text{maximize} \quad & 5x_1 - 3x_2\ \text{subject to} \quad & x_1 - x_2 \le 1\ & 2x_1 + x_2 \le 2\ & x_1, x_2 \ge 0 . \end{aligned}

* * 29.3 - 7 * * S o l v e t h e f o l l o w in g l in e a r p r o g r am u s in g S I M P L E X :

\begin{aligned} \text{minimize} \quad & x_1 + x_2 + x_3\ \text{subject to} \quad & 2x_1 + 7.5x_2 + 3x_3 \ge 10000\ & 20x_1 + 5x_2 + 10x_3 \ge 30000\ & x_1, x_2, x_3 \ge 0 . \end{aligned}

**29.3-8** Show that in a linear program in standard form with $n$ original variables and $m$ constraints, there are at most $\binom{m+n}{m}$ ways to choose a set $B$ of basic variables. Give an example of a linear program in which there are strictly fewer than $\binom{m+n}{m}$ ways to choose the set $B$. ## 29.4 Duality We have proven that, under certain assumptions, SIMPLEX terminates. We have not yet shown that it actually finds an optimal solution to a linear program, however. In order to do so, we introduce a powerful concept called **linear-programming duality**. Duality enables us to prove that a solution is indeed optimal. We saw an example of duality in Chapter 26 with Theorem 26.6, the max-flow min-cut theorem. Suppose that, given an instance of a maximum-flow problem, we find a flow $f$ with value $|f|$. How do we know whether $f$ is a maximum flow? By the max-flow min-cut theorem, if we can find a cut whose value is also $|f|$, then we have verified that $f$ is indeed a maximum flow. This relationship provides an example of duality: given a maximization problem, we define a related minimization problem such that the two problems have the same optimal objective values. *Think about it: a cut is a "certificate" of optimality. If someone claims a flow is the best possible, what small piece of evidence could convince you without redoing the whole search?* Given a linear program in which the objective is to maximize, we shall describe how to formulate a **dual** linear program in which the objective is to minimize and whose optimal value is identical to that of the original linear program. When referring to dual linear programs, we call the original linear program the **primal**. Given a primal linear program in standard form, as in (29.16)–(29.18), we define the dual linear program as

\begin{aligned} \text{minimize} \quad & \sum_{i=1}^{m} b_i y_i && (29.83)\ \text{subject to} \quad & \sum_{i=1}^{m} a_{ij} y_i \ge c_j \quad \text{for } j = 1, 2, \ldots, n && (29.84)\ & y_i \ge 0 \quad \text{for } i = 1, 2, \ldots, m. && (29.85) \end{aligned}

To form the dual, we change the maximization to a minimization, exchange the roles of coefficients on the right-hand sides and the objective function, and replace each less-than-or-equal-to by a greater-than-or-equal-to. Each of the $m$ constraints in the primal has an associated variable $y_i$ in the dual, and each of the $n$ constraints in the dual has an associated variable $x_j$ in the primal. For example, consider the linear program given in (29.53)-(29.57). The dual of this linear program is

\begin{aligned} \text{minimize} \quad & 30y_1 + 24y_2 + 36y_3 && (29.86)\ \text{subject to} \quad & y_1 + 2y_2 + 4y_3 \ge 3 && (29.87)\ & y_1 + 2y_2 + y_3 \ge 1 && (29.88)\ & 3y_1 + 5y_2 + 2y_3 \ge 2 && (29.89)\ & y_1, y_2, y_3 \ge 0. && (29.90) \end{aligned}

We shall show in Theorem 29.10 that the optimal value of the dual linear program is always equal to the optimal value of the primal linear program. Furthermore, the simplex algorithm actually implicitly solves both the primal and the dual linear programs simultaneously, thereby providing a proof of optimality. We begin by demonstrating **weak duality**, which states that any feasible solution to the primal linear program has a value no greater than that of any feasible solution to the dual linear program. > **Lemma 29.8 (Weak linear-programming duality).** Let $\bar x$ be any feasible solution to the primal linear program in (29.16)–(29.18) and let $\bar y$ be any feasible solution to the dual linear program in (29.83)–(29.85). Then we have > $$\sum_{j=1}^{n} c_j \bar x_j \le \sum_{i=1}^{m} b_i \bar y_i .$$ > **Proof.** We have > $$ > \begin{aligned} > \sum_{j=1}^{n} c_j \bar x_j &\le \sum_{j=1}^{n}\left(\sum_{i=1}^{m} a_{ij}\bar y_i\right)\bar x_j && \text{(by inequalities (29.84), since each } \bar x_j \ge 0\text{)}\\ > &= \sum_{i=1}^{m}\left(\sum_{j=1}^{n} a_{ij}\bar x_j\right)\bar y_i && \text{(exchanging the order of summation)}\\ > &\le \sum_{i=1}^{m} b_i \bar y_i && \text{(by inequalities (29.17), since each } \bar y_i \ge 0\text{)}. \qquad \blacksquare > \end{aligned} > $$ > **Corollary 29.9.** Let $\bar x$ be a feasible solution to a primal linear program $(A, b, c)$, and let $\bar y$ be a feasible solution to the corresponding dual linear program. If > $$\sum_{j=1}^{n} c_j \bar x_j = \sum_{i=1}^{m} b_i \bar y_i,$$ > then $\bar x$ and $\bar y$ are optimal solutions to the primal and dual linear programs, respectively. > **Proof.** By Lemma 29.8, the objective value of a feasible solution to the primal cannot exceed that of a feasible solution to the dual. The primal linear program is a maximization problem and the dual is a minimization problem. Thus, if feasible solutions $\bar x$ and $\bar y$ have the same objective value, neither can be improved. $\blacksquare$ *Think about it: weak duality sandwiches the unknown optimum between every feasible primal value (from below) and every feasible dual value (from above). When the two ends touch, the squeeze is complete.* Before proving that there always is a dual solution whose value is equal to that of an optimal primal solution, we describe how to find such a solution. When we ran the simplex algorithm on the linear program in (29.53)–(29.57), the final iteration yielded the slack form (29.72)–(29.75) with objective $z = 28 - \tfrac{x_3}{6} - \tfrac{x_5}{6} - \tfrac{2x_6}{3}$, $B = \{1, 2, 4\}$, and $N = \{3, 5, 6\}$. As we shall show below, the basic solution associated with the final slack form is indeed an optimal solution to the linear program; an optimal solution to linear program (29.53)–(29.57) is therefore $(x_1, x_2, x_3) = (8, 4, 0)$, with objective value $(3 \cdot 8) + (1 \cdot 4) + (2 \cdot 0) = 28$. As we also show below, we can read off an optimal dual solution: the negatives of the coefficients of the primal objective function are the values of the dual variables. More precisely, suppose that the last slack form of the primal is

\begin{aligned} z &= v’ + \sum_{j \in N} c’j x_j\ x_i &= b’i - \sum{j \in N} a’{ij}x_j \quad \text{for } i \in B. \end{aligned}

T h e n, t o p r o d u ce an o pt ima l d u a l so l u t i o n, w ese t

\bar y_i = \begin{cases} -c’_{n+i} & \text{if } (n+i) \in N,\ 0 & \text{otherwise.} \end{cases} \qquad (29.91)

Thus, an optimal solution to the dual linear program defined in (29.86)–(29.90) is $\bar y_1 = 0$ (since $n + 1 = 4 \in B$), $\bar y_2 = -c'_5 = 1/6$, and $\bar y_3 = -c'_6 = 2/3$. Evaluating the dual objective function (29.86), we obtain an objective value of $$(30 \cdot 0) + (24 \cdot (1/6)) + (36 \cdot (2/3)) = 0 + 4 + 24 = 28,$$ which confirms that the objective value of the primal is indeed equal to the objective value of the dual. Combining these calculations with Lemma 29.8 yields a proof that the optimal objective value of the primal linear program is $28$. We now show that this approach applies in general: we can find an optimal solution to the dual and simultaneously prove that a solution to the primal is optimal. > **Theorem 29.10 (Linear-programming duality).** Suppose that SIMPLEX returns values $\bar x = (\bar x_1, \bar x_2, \ldots, \bar x_n)$ for the primal linear program $(A, b, c)$. Let $N$ and $B$ denote the nonbasic and basic variables for the final slack form, let $c'$ denote the coefficients in the final slack form, and let $\bar y = (\bar y_1, \bar y_2, \ldots, \bar y_m)$ be defined by equation (29.91). Then $\bar x$ is an optimal solution to the primal linear program, $\bar y$ is an optimal solution to the dual linear program, and > $$\sum_{j=1}^{n} c_j \bar x_j = \sum_{i=1}^{m} b_i \bar y_i. \qquad (29.92)$$ > **Proof.** By Corollary 29.9, if we can find feasible solutions $\bar x$ and $\bar y$ that satisfy equation (29.92), then $\bar x$ and $\bar y$ must be optimal primal and dual solutions. We shall now show that the solutions $\bar x$ and $\bar y$ described in the statement of the theorem satisfy equation (29.92). > > **Step 1: the final slack form.** Suppose that we run SIMPLEX on a primal linear program, as given in lines (29.16)–(29.18). The algorithm proceeds through a series of slack forms until it terminates with a final slack form with objective function > $$z = v' + \sum_{j \in N} c'_j x_j. \qquad (29.93)$$ > Since SIMPLEX terminated with a solution, by the condition in line 3 we know that > $$c'_j \le 0 \quad \text{for all } j \in N. \qquad (29.94)$$ > If we define > $$c'_j = 0 \quad \text{for all } j \in B, \qquad (29.95)$$ > we can rewrite equation (29.93) as > $$ > \begin{aligned} > z &= v' + \sum_{j \in N} c'_j x_j\\ > &= v' + \sum_{j \in N} c'_j x_j + \sum_{j \in B} c'_j x_j && \text{(because } c'_j = 0 \text{ if } j \in B\text{)}\\ > &= v' + \sum_{j=1}^{n+m} c'_j x_j && \text{(because } N \cup B = \{1, 2, \ldots, n+m\}\text{)}. \qquad (29.96) > \end{aligned} > $$ > > **Step 2: the primal objective value.** For the basic solution $\bar x$ associated with this final slack form, $\bar x_j = 0$ for all $j \in N$, and $z = v'$. Since all slack forms are equivalent, if we evaluate the original objective function on $\bar x$, we must obtain the same objective value: > $$ > \begin{aligned} > \sum_{j=1}^{n} c_j \bar x_j &= v' + \sum_{j=1}^{n+m} c'_j \bar x_j && (29.97)\\ > &= v' + \sum_{j \in N} c'_j \bar x_j + \sum_{j \in B} c'_j \bar x_j\\ > &= v' + \sum_{j \in N} (c'_j \cdot 0) + \sum_{j \in B} (0 \cdot \bar x_j) && (29.98)\\ > &= v'. > \end{aligned} > $$ > > **Step 3: the dual solution.** We shall now show that $\bar y$, defined by equation (29.91), is feasible for the dual linear program and that its objective value $\sum_{i=1}^{m} b_i \bar y_i$ equals $\sum_{j=1}^{n} c_j \bar x_j$. Equation (29.97) says that the first and last slack forms, evaluated at $\bar x$, are equal. More generally, the equivalence of all slack forms implies that for any set of values $x = (x_1, x_2, \ldots, x_n)$, we have > $$\sum_{j=1}^{n} c_j x_j = v' + \sum_{j=1}^{n+m} c'_j x_j,$$ > where, for $i = 1, \ldots, m$, the slack variable $x_{n+i}$ is defined by equation (29.32). Therefore, for any particular set of values $x = (x_1, x_2, \ldots, x_n)$, we have > $$ > \begin{aligned} > \sum_{j=1}^{n} c_j x_j &= v' + \sum_{j=1}^{n+m} c'_j x_j\\ > &= v' + \sum_{j=1}^{n} c'_j x_j + \sum_{j=n+1}^{n+m} c'_j x_j\\ > &= v' + \sum_{j=1}^{n} c'_j x_j + \sum_{i=1}^{m} c'_{n+i} x_{n+i}\\ > &= v' + \sum_{j=1}^{n} c'_j x_j + \sum_{i=1}^{m} (-\bar y_i)\,x_{n+i} && \text{(by equations (29.91) and (29.95))}\\ > &= v' + \sum_{j=1}^{n} c'_j x_j + \sum_{i=1}^{m} (-\bar y_i)\left(b_i - \sum_{j=1}^{n} a_{ij}x_j\right) && \text{(by equation (29.32))}\\ > &= v' + \sum_{j=1}^{n} c'_j x_j - \sum_{i=1}^{m} b_i \bar y_i + \sum_{i=1}^{m}\sum_{j=1}^{n} (a_{ij}x_j)\bar y_i\\ > &= v' + \sum_{j=1}^{n} c'_j x_j - \sum_{i=1}^{m} b_i \bar y_i + \sum_{j=1}^{n}\left(\sum_{i=1}^{m} a_{ij}\bar y_i\right)x_j && \text{(exchanging the order of summation)}\\ > &= \left(v' - \sum_{i=1}^{m} b_i \bar y_i\right) + \sum_{j=1}^{n}\left(c'_j + \sum_{i=1}^{m} a_{ij}\bar y_i\right)x_j, > \end{aligned} > $$ > so that > $$\sum_{j=1}^{n} c_j x_j = \left(v' - \sum_{i=1}^{m} b_i \bar y_i\right) + \sum_{j=1}^{n}\left(c'_j + \sum_{i=1}^{m} a_{ij}\bar y_i\right)x_j. \qquad (29.99)$$ > Equation (29.99) holds for all values of $x_1, \ldots, x_n$, and the left side has no constant term. Applying Lemma 29.3 to equation (29.99), we obtain > $$ > \begin{aligned} > v' - \sum_{i=1}^{m} b_i \bar y_i &= 0, && (29.100)\\ > c'_j + \sum_{i=1}^{m} a_{ij}\bar y_i &= c_j \quad \text{for } j = 1, 2, \ldots, n. && (29.101) > \end{aligned} > $$ > By equation (29.100), we have that $\sum_{i=1}^{m} b_i \bar y_i = v'$, and hence the objective value of the dual $\left(\sum_{i=1}^{m} b_i \bar y_i\right)$ is equal to that of the primal $(v')$. > > **Step 4: feasibility of the dual.** It remains to show that the solution $\bar y$ is feasible for the dual problem. From inequalities (29.94) and equations (29.95), we have that $c'_j \le 0$ for all $j = 1, 2, \ldots, n+m$. Hence, for any $j = 1, 2, \ldots, n$, equations (29.101) imply that > $$ > \begin{aligned} > c_j &= c'_j + \sum_{i=1}^{m} a_{ij}\bar y_i\\ > &\le \sum_{i=1}^{m} a_{ij}\bar y_i, > \end{aligned} > $$ > which satisfies the constraints (29.84) of the dual. Finally, since $c'_j \le 0$ for each $j \in N \cup B$, when we set $\bar y$ according to equation (29.91), we have that each $\bar y_i \ge 0$, and so the nonnegativity constraints are satisfied as well. $\blacksquare$ We have shown that, given a feasible linear program, if INITIALIZE-SIMPLEX returns a feasible solution, and if SIMPLEX terminates without returning "unbounded," then the solution returned is indeed an optimal solution. We have also shown how to construct an optimal solution to the dual linear program. ### Exercises 29.4-1 to 29.4-6 **29.4-1** Formulate the dual of the linear program given in Exercise 29.3-5. **29.4-2** Suppose that we have a linear program that is not in standard form. We could produce the dual by first converting it to standard form, and then taking the dual. It would be more convenient, however, to be able to produce the dual directly. Explain how we can directly take the dual of an arbitrary linear program. **29.4-3** Write down the dual of the maximum-flow linear program, as given in lines (29.47)–(29.50) of Section 29.2. Explain how to interpret this formulation as a minimum-cut problem. **29.4-4** Write down the dual of the minimum-cost-flow linear program, as given in lines (29.51)–(29.52) of Section 29.2. Explain how to interpret this problem in terms of graphs and flows. **29.4-5** Show that the dual of the dual of a linear program is the primal linear program. **29.4-6** Which result from Chapter 26 can be interpreted as weak duality for the maximum-flow problem? ## 29.5 The initial basic feasible solution In this section, we first describe how to test whether a linear program is feasible, and if it is, how to produce a slack form for which the basic solution is feasible. We conclude by proving the **fundamental theorem of linear programming**, which says that the SIMPLEX procedure always produces the correct result. ### Finding an initial solution In Section 29.3, we assumed that we had a procedure INITIALIZE-SIMPLEX that determines whether a linear program has any feasible solutions, and if it does, gives a slack form for which the basic solution is feasible. We describe this procedure here. A linear program can be feasible, yet the initial basic solution might not be feasible. Consider, for example, the following linear program:

\begin{aligned} \text{maximize} \quad & 2x_1 - x_2 && (29.102)\ \text{subject to} \quad & 2x_1 - x_2 \le 2 && (29.103)\ & x_1 - 5x_2 \le -4 && (29.104)\ & x_1, x_2 \ge 0. && (29.105) \end{aligned}

If we were to convert this linear program to slack form, the basic solution would set $x_1 = 0$ and $x_2 = 0$. This solution violates constraint (29.104), and so it is not a feasible solution. Thus, INITIALIZE-SIMPLEX cannot just return the obvious slack form. In order to determine whether a linear program has any feasible solutions, we will formulate an **auxiliary linear program**. For this auxiliary linear program, we can find (with a little work) a slack form for which the basic solution is feasible. Furthermore, the solution of this auxiliary linear program determines whether the initial linear program is feasible and if so, it provides a feasible solution with which we can initialize SIMPLEX. *Think about it: to find a starting point for a hike, you might first ask "how far am I from the trail?" and walk to where that distance becomes zero. The auxiliary program asks exactly that question with the single variable $x_0$.* > **Lemma 29.11.** Let $L$ be a linear program in standard form, given as in (29.16)–(29.18). Let $x_0$ be a new variable, and let $L_{\text{aux}}$ be the following linear program with $n + 1$ variables: > $$ > \begin{aligned} > \text{maximize} \quad & -x_0 && (29.106)\\ > \text{subject to} \quad & \sum_{j=1}^{n} a_{ij}x_j - x_0 \le b_i \quad \text{for } i = 1, 2, \ldots, m, && (29.107)\\ > & x_j \ge 0 \quad \text{for } j = 0, 1, \ldots, n. && (29.108) > \end{aligned} > $$ > Then $L$ is feasible if and only if the optimal objective value of $L_{\text{aux}}$ is $0$. > **Proof.** ($\Rightarrow$) Suppose that $L$ has a feasible solution $\bar x = (\bar x_1, \bar x_2, \ldots, \bar x_n)$. > 1. Setting $\bar x_0 = 0$ and combining it with $\bar x$ satisfies (29.107), because $\sum_j a_{ij}\bar x_j - 0 \le b_i$ by feasibility of $\bar x$ in $L$. > 2. All variables are nonnegative, so (29.108) holds. Hence this is a feasible solution to $L_{\text{aux}}$ with objective value $-0 = 0$. > 3. Since $x_0 \ge 0$ is a constraint of $L_{\text{aux}}$ and the objective function is to maximize $-x_0$, no feasible solution has objective value above $0$. > 4. Therefore this solution is optimal for $L_{\text{aux}}$, and the optimal objective value is $0$. > > ($\Leftarrow$) Conversely, suppose that the optimal objective value of $L_{\text{aux}}$ is $0$. Then $\bar x_0 = 0$ in an optimal solution, and substituting $\bar x_0 = 0$ into (29.107) shows that the remaining solution values $\bar x_1, \ldots, \bar x_n$ satisfy $\sum_j a_{ij}\bar x_j \le b_i$ for all $i$, which are exactly the constraints of $L$. They are also nonnegative, so $L$ is feasible. $\blacksquare$ We now describe our strategy to find an initial basic feasible solution for a linear program $L$ in standard form: ```text INITIALIZE-SIMPLEX(A, b, c) 1 let k be the index of the minimum b_i 2 if b_k ≥ 0 // is the initial basic solution feasible? 3 return ({1, 2, ..., n}, {n + 1, n + 2, ..., n + m}, A, b, c, 0) 4 form L_aux by adding -x_0 to the left-hand side of each constraint and setting the objective function to -x_0 5 let (N, B, A, b, c, v) be the resulting slack form for L_aux 6 l = n + k 7 // L_aux has n + 1 nonbasic variables and m basic variables. 8 (N, B, A, b, c, v) = PIVOT(N, B, A, b, c, v, l, 0) 9 // The basic solution is now feasible for L_aux. 10 iterate the while loop of lines 3-12 of SIMPLEX until an optimal solution to L_aux is found 11 if the optimal solution to L_aux sets x_0 to 0 12 if x_0 is basic 13 perform one (degenerate) pivot to make it nonbasic 14 from the final slack form of L_aux, remove x_0 from the constraints and restore the original objective function of L, but replace each basic variable in this objective function by the right-hand side of its associated constraint 15 return the modified final slack form 16 else return "infeasible" ``` INITIALIZE-SIMPLEX works as follows. In lines 1–3, we implicitly test the basic solution to the initial slack form for $L$ given by $N = \{1, 2, \ldots, n\}$, $B = \{n+1, n+2, \ldots, n+m\}$, $\bar x_i = b_i$ for all $i \in B$, and $\bar x_j = 0$ for all $j \in N$. (Creating the slack form requires no explicit effort, as the values of $A$, $b$, and $c$ are the same in both slack and standard forms.) If line 2 finds this basic solution to be feasible, that is, $\bar x_i \ge 0$ for all $i \in N \cup B$, then line 3 returns the slack form. Otherwise, in line 4, we form the auxiliary linear program $L_{\text{aux}}$ as in Lemma 29.11. Since the initial basic solution to $L$ is not feasible, the initial basic solution to the slack form for $L_{\text{aux}}$ cannot be feasible either. To find a basic feasible solution, we perform a single pivot operation. Line 6 selects $l = n + k$ as the index of the basic variable that will be the leaving variable in the upcoming pivot operation. Since the basic variables are $x_{n+1}, x_{n+2}, \ldots, x_{n+m}$, the leaving variable $x_l$ will be the one with the most negative value. Line 8 performs that call of PIVOT, with $x_0$ entering and $x_l$ leaving. We shall see shortly that the basic solution resulting from this call of PIVOT will be feasible. Now that we have a slack form for which the basic solution is feasible, we can, in line 10, repeatedly call PIVOT to fully solve the auxiliary linear program. As the test in line 11 demonstrates, if we find an optimal solution to $L_{\text{aux}}$ with objective value $0$, then in lines 12–14, we create a slack form for $L$ for which the basic solution is feasible. To do so, we first, in lines 12–13, handle the degenerate case in which $x_0$ may still be basic with value $\bar x_0 = 0$. In this case, we perform a pivot step to remove $x_0$ from the basis, using any $e \in N$ such that $a_{0e} \ne 0$ as the entering variable. The new basic solution remains feasible; the degenerate pivot does not change the value of any variable. Next we delete all $x_0$ terms from the constraints and restore the original objective function for $L$. The original objective function may contain both basic and nonbasic variables. Therefore, in the objective function we replace each basic variable by the right-hand side of its associated constraint. Line 15 then returns this modified slack form. If, on the other hand, line 11 discovers that the original linear program $L$ is infeasible, then line 16 returns this information. ### Example: running INITIALIZE-SIMPLEX We now demonstrate the operation of INITIALIZE-SIMPLEX on the linear program (29.102)–(29.105). This linear program is feasible if we can find nonnegative values for $x_1$ and $x_2$ that satisfy inequalities (29.103) and (29.104). Using Lemma 29.11, we formulate the auxiliary linear program

\begin{aligned} \text{maximize} \quad & -x_0 && (29.109)\ \text{subject to} \quad & 2x_1 - x_2 - x_0 \le 2 && (29.110)\ & x_1 - 5x_2 - x_0 \le -4 && (29.111)\ & x_1, x_2, x_0 \ge 0. \end{aligned}

By Lemma 29.11, if the optimal objective value of this auxiliary linear program is $0$, then the original linear program has a feasible solution. If the optimal objective value of this auxiliary linear program is negative, then the original linear program does not have a feasible solution. We write this linear program in slack form, obtaining

\begin{aligned} z &= -x_0\ x_3 &= 2 - 2x_1 + x_2 + x_0\ x_4 &= -4 - x_1 + 5x_2 + x_0 . \end{aligned}

We are not out of the woods yet, because the basic solution, which would set $x_4 = -4$, is not feasible for this auxiliary linear program. We can, however, with one call to PIVOT, convert this slack form into one in which the basic solution is feasible. As line 8 indicates, we choose $x_0$ to be the entering variable. In line 6, we choose as the leaving variable $x_4$, which is the basic variable whose value in the basic solution is most negative. **Pivot 1** ($x_0$ enters, $x_4$ leaves). 1. Solve the $x_4$ equation for $x_0$: $x_0 = 4 + x_1 - 5x_2 + x_4$. 2. Substitute into the objective: $z = -x_0 = -4 - x_1 + 5x_2 - x_4$. 3. Substitute into the $x_3$ equation: $x_3 = 2 - 2x_1 + x_2 + (4 + x_1 - 5x_2 + x_4) = 6 - x_1 - 4x_2 + x_4$. After pivoting, we have the slack form

\begin{aligned} z &= -4 - x_1 + 5x_2 - x_4\ x_0 &= 4 + x_1 - 5x_2 + x_4\ x_3 &= 6 - x_1 - 4x_2 + x_4 . \end{aligned}

The associated basic solution is $(x_0, x_1, x_2, x_3, x_4) = (4, 0, 0, 6, 0)$, which is feasible. We now repeatedly call PIVOT until we obtain an optimal solution to $L_{\text{aux}}$. In this case, one call to PIVOT with $x_2$ entering and $x_0$ leaving yields: **Pivot 2** ($x_2$ enters, $x_0$ leaves). 1. The entering variable $x_2$ has positive objective coefficient $5$. The $x_0$ equation limits $x_2$ to $4/5$, and the $x_3$ equation limits it to $6/4 = 3/2$. The tighter limit is $4/5$, so $x_0$ leaves. 2. Solve the $x_0$ equation for $x_2$: $x_2 = \tfrac45 + \tfrac{x_1}{5} - \tfrac{x_0}{5} + \tfrac{x_4}{5}$. 3. Substitute into the objective: $z = -4 - x_1 + 4 + x_1 - x_0 + x_4 - x_4 = -x_0$. 4. Substitute into the $x_3$ equation: $x_3 = 6 - x_1 - \tfrac{16}{5} - \tfrac{4x_1}{5} + \tfrac{4x_0}{5} - \tfrac{4x_4}{5} + x_4 = \tfrac{14}{5} - \tfrac{9x_1}{5} + \tfrac{4x_0}{5} + \tfrac{x_4}{5}$.

\begin{aligned} z &= -x_0\ x_2 &= \tfrac{4}{5} + \tfrac{x_1}{5} - \tfrac{x_0}{5} + \tfrac{x_4}{5}\ x_3 &= \tfrac{14}{5} - \tfrac{9x_1}{5} + \tfrac{4x_0}{5} + \tfrac{x_4}{5} . \end{aligned}

This slack form is the final solution to the auxiliary problem. Since this solution has $\bar x_0 = 0$, we know that our initial problem was feasible. Furthermore, since $\bar x_0 = 0$, we can just remove it from the set of constraints. We then restore the original objective function, with appropriate substitutions made to include only nonbasic variables. In our example, we get the objective function $$2x_1 - x_2 = 2x_1 - \left(\tfrac45 + \tfrac{x_1}{5} - \tfrac{x_0}{5} + \tfrac{x_4}{5}\right).$$ Setting $x_0 = 0$ and simplifying, we get the objective function $$-\tfrac{4}{5} + \tfrac{9x_1}{5} - \tfrac{x_4}{5}$$ and the slack form

\begin{aligned} z &= -\tfrac{4}{5} + \tfrac{9x_1}{5} - \tfrac{x_4}{5}\ x_2 &= \tfrac{4}{5} + \tfrac{x_1}{5} + \tfrac{x_4}{5}\ x_3 &= \tfrac{14}{5} - \tfrac{9x_1}{5} + \tfrac{x_4}{5} . \end{aligned}

This slack form has a feasible basic solution, and we can return it to procedure SIMPLEX. We now formally show the correctness of INITIALIZE-SIMPLEX. > **Lemma 29.12.** If a linear program $L$ has no feasible solution, then INITIALIZE-SIMPLEX returns "infeasible." Otherwise, it returns a valid slack form for which the basic solution is feasible. > **Proof.** First suppose that the linear program $L$ has no feasible solution. Then by Lemma 29.11, the optimal objective value of $L_{\text{aux}}$, defined in (29.106)–(29.108), is nonzero, and by the nonnegativity constraint on $x_0$, the optimal objective value must be negative. Furthermore, this objective value must be finite, since setting $x_j = 0$ for $j = 1, 2, \ldots, n$, and $x_0 = \left|\min_{i=1}^{m}\{b_i\}\right|$ is feasible, and this solution has objective value $-\left|\min_{i=1}^{m}\{b_i\}\right|$. Therefore, line 10 of INITIALIZE-SIMPLEX finds a solution with a nonpositive objective value. Let $\bar x$ be the basic solution associated with the final slack form. We cannot have $\bar x_0 = 0$, because then $L_{\text{aux}}$ would have objective value $0$, which contradicts that the objective value is negative. Thus the test in line 11 results in line 16 returning "infeasible." > > Suppose now that the linear program $L$ does have a feasible solution. From Exercise 29.3-4, we know that if $b_i \ge 0$ for $i = 1, 2, \ldots, m$, then the basic solution associated with the initial slack form is feasible. In this case, lines 2–3 return the slack form associated with the input. (Converting the standard form to slack form is easy, since $A$, $b$, and $c$ are the same in both.) > > In the remainder of the proof, we handle the case in which the linear program is feasible but we do not return in line 3. We argue that in this case, lines 4–10 find a feasible solution to $L_{\text{aux}}$ with objective value $0$. First, by lines 1–2, we must have > $$b_k < 0,$$ > and > $$b_k \le b_i \quad \text{for each } i \in B. \qquad (29.112)$$ > In line 8, we perform one pivot operation in which the leaving variable $x_l$ (recall that $l = n + k$, so that $b_l < 0$) is the left-hand side of the equation with minimum $b_i$, and the entering variable is $x_0$, the extra added variable. We now show > that after this pivot, all entries of $b$ are nonnegative, and hence the basic solution to $L_{\text{aux}}$ is feasible. > **Proof (continued).** Let $\bar x$ be the basic solution after the call to PIVOT, and let $\hat b$ and $\hat B$ be the values returned by PIVOT. Lemma 29.1 implies that > $$ > \bar x_i = > \begin{cases} > b_i - a_{ie}\hat b_e & \text{if } i \in \hat B - \{e\},\\[2pt] > b_l / a_{le} & \text{if } i = e. > \end{cases} > \qquad (29.113) > $$ > The call to PIVOT in line 8 has $e = 0$. If we rewrite inequalities (29.107) to include the coefficients $a_{i0}$, > $$\sum_{j=0}^{n} a_{ij}x_j \le b_i \quad \text{for } i = 1, 2, \ldots, m, \qquad (29.114)$$ > then > $$a_{i0} = a_{ie} = -1 \quad \text{for each } i \in B. \qquad (29.115)$$ > (Note that $a_{i0}$ is the coefficient of $x_0$ as it appears in inequalities (29.114), not the negation of the coefficient, because $L_{\text{aux}}$ is in standard rather than slack form.) > > Now check the sign of every basic variable after the pivot. > > 1. Since $l \in B$, we have $a_{le} = -1$ by (29.115). Also $b_l < 0$. Thus $b_l / a_{le} > 0$, and so $\bar x_e = \bar x_0 > 0$. > 2. For the remaining basic variables $i \in B - \{l\}$, we have > $$ > \begin{aligned} > \bar x_i &= b_i - a_{ie}\hat b_e && \text{(by equation (29.113))}\\ > &= b_i - a_{ie}\,(b_l / a_{le}) && \text{(by line 3 of PIVOT)}\\ > &= b_i - b_l && \text{(by equation (29.115), which gives } a_{ie} = a_{le} = -1)\\ > &\ge 0 && \text{(by inequality (29.112), } b_l \le b_i). > \end{aligned} > $$ > 3. The leaving variable $x_l$ is now nonbasic and has value $0$. > > Hence each basic variable is nonnegative, and the basic solution after the call to PIVOT in line 8 is feasible. > > We next execute line 10, which solves $L_{\text{aux}}$. > > 4. Since we have assumed that $L$ has a feasible solution, Lemma 29.11 implies that $L_{\text{aux}}$ has an optimal solution with objective value $0$. > 5. Since all the slack forms are equivalent, the final basic solution to $L_{\text{aux}}$ must have $\bar x_0 = 0$. > 6. After removing $x_0$ from the linear program, we obtain a slack form that is feasible for $L$. > > Line 15 then returns this slack form. $\blacksquare$ ### Fundamental theorem of linear programming We conclude this chapter by showing that the SIMPLEX procedure works. In particular, any linear program either is infeasible, is unbounded, or has an optimal solution with a finite objective value. In each case, SIMPLEX acts appropriately. > **Theorem 29.13 (Fundamental theorem of linear programming).** Any linear program $L$, given in standard form, either > 1. has an optimal solution with a finite objective value, > 2. is infeasible, or > 3. is unbounded. > > If $L$ is infeasible, SIMPLEX returns "infeasible." If $L$ is unbounded, SIMPLEX returns "unbounded." Otherwise, SIMPLEX returns an optimal solution with a finite objective value. > **Proof.** > 1. By Lemma 29.12, if linear program $L$ is infeasible, then INITIALIZE-SIMPLEX returns "infeasible," and so SIMPLEX returns "infeasible." > 2. Now suppose that the linear program $L$ is feasible. By Lemma 29.12, INITIALIZE-SIMPLEX returns a slack form for which the basic solution is feasible. > 3. By Lemma 29.7, therefore, SIMPLEX either returns "unbounded" or terminates with a feasible solution. > 4. If it terminates with a finite solution, then Theorem 29.10 tells us that this solution is optimal. > 5. On the other hand, if SIMPLEX returns "unbounded," Lemma 29.2 tells us the linear program $L$ is indeed unbounded. > 6. Since SIMPLEX always terminates in one of these ways, the proof is complete. $\blacksquare$ *Think about it: the theorem says there are exactly three possible fates for any linear program. Can you sketch one tiny example in two variables for each fate?* ### Exercises 29.5-1 to 29.5-9 **29.5-1** Give detailed pseudocode to implement lines 5 and 14 of INITIALIZE-SIMPLEX. **29.5-2** Show that when the main loop of SIMPLEX is run by INITIALIZE-SIMPLEX, it can never return "unbounded." **29.5-3** Suppose that we are given a linear program $L$ in standard form, and suppose that for both $L$ and the dual of $L$, the basic solutions associated with the initial slack forms are feasible. Show that the optimal objective value of $L$ is $0$. **29.5-4** Suppose that we allow strict inequalities in a linear program. Show that in this case, the fundamental theorem of linear programming does not hold. **29.5-5** Solve the following linear program using SIMPLEX:

\begin{aligned} \text{maximize} \quad & x_1 + 3x_2\ \text{subject to} \quad & x_1 - x_2 \le 8\ & -x_1 - x_2 \le -3\ & -x_1 + 4x_2 \le 2\ & x_1, x_2 \ge 0 \end{aligned}

* * 29.5 - 6 * * S o l v e t h e f o l l o w in g l in e a r p r o g r am u s in g S I M P L E X :

\begin{aligned} \text{maximize} \quad & x_1 - 2x_2\ \text{subject to} \quad & x_1 + 2x_2 \le 4\ & -2x_1 - 6x_2 \le -12\ & x_2 \le 1\ & x_1, x_2 \ge 0 \end{aligned}

* * 29.5 - 7 * * S o l v e t h e f o l l o w in g l in e a r p r o g r am u s in g S I M P L E X :

\begin{aligned} \text{maximize} \quad & x_1 + 3x_2\ \text{subject to} \quad & -x_1 + x_2 \le -1\ & -x_1 - x_2 \le -3\ & -x_1 + 4x_2 \le 2\ & x_1, x_2 \ge 0 \end{aligned}

**29.5-8** Solve the linear program given in (29.6)-(29.10). **29.5-9** Consider the following 1-variable linear program, which we call $P$:

\begin{aligned} \text{maximize} \quad & tx\ \text{subject to} \quad & rx \le s\ & x \ge 0, \end{aligned}

where $r$, $s$, and $t$ are arbitrary real numbers. Let $D$ be the dual of $P$. State for which values of $r$, $s$, and $t$ you can assert that 1. Both $P$ and $D$ have optimal solutions with finite objective values. 2. $P$ is feasible, but $D$ is infeasible. 3. $D$ is feasible, but $P$ is infeasible. 4. Neither $P$ nor $D$ is feasible. ## Problems ### 29-1 Linear-inequality feasibility Given a set of $m$ linear inequalities on $n$ variables $x_1, x_2, \ldots, x_n$, the **linear-inequality feasibility problem** asks whether there is a setting of the variables that simultaneously satisfies each of the inequalities. **a.** Show that if we have an algorithm for linear programming, we can use it to solve a linear-inequality feasibility problem. The number of variables and constraints that you use in the linear-programming problem should be polynomial in $n$ and $m$. **b.** Show that if we have an algorithm for the linear-inequality feasibility problem, we can use it to solve a linear-programming problem. The number of variables and linear inequalities that you use in the linear-inequality feasibility problem should be polynomial in $n$ and $m$, the number of variables and constraints in the linear program. ### 29-2 Complementary slackness **Complementary slackness** describes a relationship between the values of primal variables and dual constraints and between the values of dual variables and primal constraints. Let $\bar x$ be a feasible solution to the primal linear program given in (29.16)–(29.18), and let $\bar y$ be a feasible solution to the dual linear program given in (29.83)–(29.85). Complementary slackness states that the following conditions are necessary and sufficient for $\bar x$ and $\bar y$ to be optimal: $$\sum_{i=1}^{m} a_{ij}\bar y_i = c_j \quad \text{or} \quad \bar x_j = 0 \quad \text{for } j = 1, 2, \ldots, n$$ and $$\sum_{j=1}^{n} a_{ij}\bar x_j = b_i \quad \text{or} \quad \bar y_i = 0 \quad \text{for } i = 1, 2, \ldots, m.$$ **a.** Verify that complementary slackness holds for the linear program in lines (29.53)–(29.57). **b.** Prove that complementary slackness holds for any primal linear program and its corresponding dual. **c.** Prove that a feasible solution $\bar x$ to a primal linear program given in lines (29.16)–(29.18) is optimal if and only if there exist values $\bar y = (\bar y_1, \bar y_2, \ldots, \bar y_m)$ such that 1. $\bar y$ is a feasible solution to the dual linear program given in (29.83)–(29.85), 2. $\sum_{i=1}^{m} a_{ij}\bar y_i = c_j$ for all $j$ such that $\bar x_j > 0$, and 3. $\bar y_i = 0$ for all $i$ such that $\sum_{j=1}^{n} a_{ij}\bar x_j < b_i$. ### 29-3 Integer linear programming An **integer linear-programming problem** is a linear-programming problem with the additional constraint that the variables $x$ must take on integral values. Exercise 34.5-3 shows that just determining whether an integer linear program has a feasible solution is NP-hard, which means that there is no known polynomial-time algorithm for this problem. **a.** Show that weak duality (Lemma 29.8) holds for an integer linear program. **b.** Show that duality (Theorem 29.10) does not always hold for an integer linear program. **c.** Given a primal linear program in standard form, let us define $P$ to be the optimal objective value for the primal linear program, $D$ to be the optimal objective value for its dual, $IP$ to be the optimal objective value for the integer version of the primal (that is, the primal with the added constraint that the variables take on integer values), and $ID$ to be the optimal objective value for the integer version of the dual. Assuming that both the primal integer program and the dual integer program are feasible and bounded, show that $$IP \le P = D \le ID.$$ ### 29-4 Farkas's lemma Let $A$ be an $m \times n$ matrix and $c$ be an $n$-vector. Then **Farkas's lemma** states that exactly one of the systems

\begin{aligned} Ax &\le 0,\ c^{\mathsf T}x &> 0 \end{aligned}

an d

\begin{aligned} A^{\mathsf T}y &= c,\ y &\ge 0 \end{aligned}

(n-m+1),\frac{1-d^{-m}}{1-d^{-1}} \le 2(n-m+1)

over all executions of this loop. (Assume that the naive algorithm stops comparing characters for a given shift once it finds a mismatch or matches the entire pattern.) Thus, for randomly chosen strings, the naive algorithm is quite efficient. **32.1-4** Suppose we allow the pattern $P$ to contain occurrences of a **gap character** $\diamondsuit$ that can match an *arbitrary* string of characters (even one of zero length). For example, the pattern $ab\diamondsuit ba\diamondsuit c$ occurs in the text $cabccbacbacab$ as ```text c ab cc ba cba c ab ab ◇ ba ◇ c ``` and as ```text c ab ccbac ba c ab ab ◇ ba ◇ c ``` Note that the gap character may occur an arbitrary number of times in the pattern but not at all in the text. Give a polynomial-time algorithm to determine whether such a pattern $P$ occurs in a given text $T$, and analyze the running time of your algorithm. ## 32.2 The Rabin-Karp algorithm Rabin and Karp proposed a string-matching algorithm that performs well in practice and that also generalizes to other algorithms for related problems, such as two-dimensional pattern matching. The Rabin-Karp algorithm uses $\Theta(m)$ preprocessing time, and its worst-case running time is $\Theta((n-m+1)m)$. Based on certain assumptions, however, its average-case running time is better. This algorithm makes use of elementary number-theoretic notions such as the equivalence of two numbers modulo a third number. You might want to refer to Section 31.1 for the relevant definitions. For expository purposes, let us assume that $\Sigma = \{0, 1, 2, \ldots, 9\}$, so that each character is a decimal digit. (In the general case, we can assume that each character is a digit in radix-$d$ notation, where $d = |\Sigma|$.) We can then view a string of $k$ consecutive characters as representing a length-$k$ decimal number. The character string $31415$ thus corresponds to the decimal number $31{,}415$. Because we interpret the input characters as both graphical symbols and digits, we find it convenient in this section to denote them as we would digits. Given a pattern $P[1..m]$, let $p$ denote its corresponding decimal value. In a similar manner, given a text $T[1..n]$, let $t_s$ denote the decimal value of the length-$m$ substring $T[s+1..s+m]$, for $s = 0, 1, \ldots, n-m$. Certainly, $t_s = p$ if and only if $T[s+1..s+m] = P[1..m]$; thus, $s$ is a valid shift if and only if $t_s = p$. If we could compute $p$ in time $\Theta(m)$ and all the $t_s$ values in a total of $\Theta(n-m+1)$ time,<sup>1</sup> then we could determine all valid shifts $s$ in time $\Theta(m) + \Theta(n-m+1) = \Theta(n)$ by comparing $p$ with each of the $t_s$ values. (For the moment, let's not worry about the possibility that $p$ and the $t_s$ values might be very large numbers.) We can compute $p$ in time $\Theta(m)$ using Horner's rule (see Section 30.1):

p = P[m] + 10\bigl(P[m-1] + 10(P[m-2] + \cdots + 10(P[2] + 10,P[1]) \cdots)\bigr).

Similarly, we can compute $t_0$ from $T[1..m]$ in time $\Theta(m)$. <sup>1</sup> We write $\Theta(n-m+1)$ instead of $\Theta(n-m)$ because $s$ takes on $n-m+1$ different values. The "+1" is significant in an asymptotic sense because when $m = n$, computing the lone $t_s$ value takes $\Theta(1)$ time, not $\Theta(0)$ time. To compute the remaining values $t_1, t_2, \ldots, t_{n-m}$ in time $\Theta(n-m)$, we observe that we can compute $t_{s+1}$ from $t_s$ in constant time, since

t_{s+1} = 10\bigl(t_s - 10^{m-1},T[s+1]\bigr) + T[s+m+1] . \tag{32.1}

Subtracting $10^{m-1}T[s+1]$ removes the high-order digit from $t_s$, multiplying the result by $10$ shifts the number left by one digit position, and adding $T[s+m+1]$ brings in the appropriate low-order digit. For example, if $m = 5$ and $t_s = 31415$, then we wish to remove the high-order digit $T[s+1] = 3$ and bring in the new low-order digit (suppose it is $T[s+5+1] = 2$) to obtain

\begin{aligned} t_{s+1} &= 10,(31415 - 10000 \cdot 3) + 2 \ &= 10 \cdot 1415 + 2 \ &= 14152 . \end{aligned}

If we precompute the constant $10^{m-1}$ (which we can do in time $O(\lg m)$ using the techniques of Section 31.6, although for this application a straightforward $O(m)$-time method suffices), then each execution of equation (32.1) takes a constant number of arithmetic operations. Thus, we can compute $p$ in time $\Theta(m)$, and we can compute all of $t_0, t_1, \ldots, t_{n-m}$ in time $\Theta(n-m+1)$. Therefore, we can find all occurrences of the pattern $P[1..m]$ in the text $T[1..n]$ with $\Theta(m)$ preprocessing time and $\Theta(n-m+1)$ matching time. > **Think about it.** A rolling window is like a car odometer that you advance by one click: you do not re-read every digit, you only drop the leftmost and add a new rightmost one. What breaks if the window is $1000$ digits long? Until now, we have intentionally overlooked one problem: $p$ and $t_s$ may be too large to work with conveniently. If $P$ contains $m$ characters, then we cannot reasonably assume that each arithmetic operation on $p$ (which is $m$ digits long) takes "constant time." Fortunately, we can solve this problem easily, as Figure 32.5 shows: compute $p$ and the $t_s$ values modulo a suitable modulus $q$. We can compute $p$ modulo $q$ in $\Theta(m)$ time and all the $t_s$ values modulo $q$ in $\Theta(n-m+1)$ time. If we choose the modulus $q$ as a prime such that $10q$ just fits within one computer word, then we can perform all the necessary computations with single-precision arithmetic. In general, with a $d$-ary alphabet $\{0, 1, \ldots, d-1\}$, we choose $q$ so that $dq$ fits within a computer word and adjust the recurrence equation (32.1) to work modulo $q$, so that it becomes

t_{s+1} = \bigl(d,(t_s - T[s+1],h) + T[s+m+1]\bigr) \bmod q , \tag{32.2}

where $h \equiv d^{m-1} \pmod q$ is the value of the digit "1" in the high-order position of an $m$-digit text window. The solution of working modulo $q$ is not perfect, however: $t_s \equiv p \pmod q$ does not imply that $t_s = p$. On the other hand, if $t_s \not\equiv p \pmod q$, then we definitely have that $t_s \ne p$, so that shift $s$ is invalid. We can thus use the test $t_s \equiv p \pmod q$ as a fast heuristic test to rule out invalid shifts $s$. Any shift $s$ for which $t_s \equiv p \pmod q$ must be tested further to see whether $s$ is really valid or we just have a **spurious hit**. This additional test explicitly checks the condition $P[1..m] = T[s+1..s+m]$. If $q$ is large enough, then we hope that spurious hits occur infrequently enough that the cost of the extra checking is low. *Figure 32.5: The Rabin-Karp algorithm. Each character is a decimal digit, and we compute values modulo 13. (a) A text string $2359023141526739921$. A window of length 5 is shown shaded (the window $31415$). The numerical value of the shaded number, computed modulo 13, yields the value 7. (b) The same text string with values computed modulo 13 for each possible position of a length-5 window. Assuming the pattern $P = 31415$, we look for windows whose value modulo 13 is 7, since $31415 \equiv 7 \pmod{13}$. The algorithm finds two such windows, shown shaded in the figure. The first, beginning at text position 7, is indeed an occurrence of the pattern, while the second, beginning at text position 13, is a spurious hit. (c) How to compute the value for a window in constant time, given the value for the previous window. The first window has value $31415$. Dropping the high-order digit $3$, shifting left (multiplying by $10$), and then adding in the low-order digit $2$ gives us the new value $14152$. Because all computations are performed modulo $13$, the value for the first window is $7$, and the value for the new window is $8$.* The window values modulo 13 for positions $1, 2, \ldots, 15$ of the text in part (b) are: | Window start | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | Value mod 13 | 8 | 9 | 3 | 11 | 0 | 1 | **7** | 8 | 4 | 5 | 10 | 11 | **7** | 9 | 11 | Position 7 is a valid match, position 13 is a spurious hit. Part (c) of the figure performs the update with residues:

\begin{aligned} 14152 &\equiv (31415 - 3 \cdot 10000)\cdot 10 + 2 \pmod{13} \ &\equiv (7 - 3 \cdot 3)\cdot 10 + 2 \pmod{13} \ &\equiv -18 + 2 + 0 \cdot 13 + 24 \pmod{13}\quad\text{(since } (7-9)\cdot 10 = -20\text{)} \ &\equiv 8 \pmod{13} . \end{aligned}

Here $h = 10^4 \bmod 13 = 3$ and $(7 - 9)\cdot 10 + 2 = -18 \equiv 8 \pmod{13}$. The following procedure makes these ideas precise. The inputs to the procedure are the text $T$, the pattern $P$, the radix $d$ to use (which is typically taken to be $|\Sigma|$), and the prime $q$ to use. ```text RABIN-KARP-MATCHER(T, P, d, q) 1 n = T.length 2 m = P.length 3 h = d^(m-1) mod q 4 p = 0 5 t_0 = 0 6 for i = 1 to m // preprocessing 7 p = (d*p + P[i]) mod q 8 t_0 = (d*t_0 + T[i]) mod q 9 for s = 0 to n - m // matching 10 if p == t_s 11 if P[1..m] == T[s+1..s+m] 12 print "Pattern occurs with shift" s 13 if s < n - m 14 t_{s+1} = (d*(t_s - T[s+1]*h) + T[s+m+1]) mod q ``` The procedure RABIN-KARP-MATCHER works as follows. All characters are interpreted as radix-$d$ digits. The subscripts on $t$ are provided only for clarity; the program works correctly if all the subscripts are dropped. Line 3 initializes $h$ to the value of the high-order digit position of an $m$-digit window. Lines 4-8 compute $p$ as the value of $P[1..m] \bmod q$ and $t_0$ as the value of $T[1..m] \bmod q$. The **for** loop of lines 9-14 iterates through all possible shifts $s$, maintaining the following invariant: > **Loop invariant.** Whenever line 10 is executed, $t_s = T[s+1..s+m] \bmod q$. If $p = t_s$ in line 10 (a "hit"), then line 11 checks to see whether $P[1..m] = T[s+1..s+m]$ in order to rule out the possibility of a spurious hit. Line 12 prints out any valid shifts that are found. If $s < n-m$ (checked in line 13), then the **for** loop will execute at least one more time, and so line 14 first executes to ensure that the loop invariant holds when we get back to line 10. Line 14 computes the value of $t_{s+1} \bmod q$ from the value of $t_s \bmod q$ in constant time using equation (32.2) directly. RABIN-KARP-MATCHER takes $\Theta(m)$ preprocessing time, and its matching time is $\Theta((n-m+1)m)$ in the worst case, since (like the naive string-matching algorithm) the Rabin-Karp algorithm explicitly verifies every valid shift. If $P = a^m$ and $T = a^n$, then verifying takes time $\Theta((n-m+1)m)$, since each of the $n-m+1$ possible shifts is valid. In many applications, we expect few valid shifts, perhaps some constant $c$ of them. In such applications, the expected matching time of the algorithm is only $O((n-m+1) + cm) = O(n+m)$, plus the time required to process spurious hits. We can base a heuristic analysis on the assumption that reducing values modulo $q$ acts like a random mapping from $\Sigma^*$ to $\mathbb{Z}_q$. (See the discussion on the use of division for hashing in Section 11.3.1. It is difficult to formalize and prove such an assumption, although one viable approach is to assume that $q$ is chosen randomly from integers of the appropriate size. We shall not pursue this formalization here.) We can then expect that the number of spurious hits is $O(n/q)$, since we can estimate the chance that an arbitrary $t_s$ will be equivalent to $p$, modulo $q$, as $1/q$. Since there are $O(n)$ positions at which the test of line 10 fails and we spend $O(m)$ time for each hit, the expected matching time taken by the Rabin-Karp algorithm is

O(n) + O\bigl(m,(v + n/q)\bigr) ,

where $v$ is the number of valid shifts. This running time is $O(n)$ if $v = O(1)$ and we choose $q \ge m$. That is, if the expected number of valid shifts is small ($O(1)$) and we choose the prime $q$ to be larger than the length of the pattern, then we can expect the Rabin-Karp procedure to use only $O(n+m)$ matching time. Since $m \le n$, this expected matching time is $O(n)$. > **Why $q \ge m$ suffices.** With $q \ge m$ and $v = O(1)$: > $$ > \begin{aligned} > O(n) + O\bigl(m(v + n/q)\bigr) &= O(n) + O(mv) + O(mn/q) \\ > &\le O(n) + O(m) + O(n) \quad\text{(since } v = O(1)\text{ and } m/q \le 1\text{)} \\ > &= O(n+m) = O(n) \quad\text{(since } m \le n\text{)} . > \end{aligned} > $$ ### Exercises 32.2-1 to 32.2-4 **32.2-1** Working modulo $q = 11$, how many spurious hits does the Rabin-Karp matcher encounter in the text $T = 3141592653589793$ when looking for the pattern $P = 26$? **32.2-2** How would you extend the Rabin-Karp method to the problem of searching a text string for an occurrence of any one of a given set of $k$ patterns? Start by assuming that all $k$ patterns have the same length. Then generalize your solution to allow the patterns to have different lengths. **32.2-3** Show how to extend the Rabin-Karp method to handle the problem of looking for a given $m \times m$ pattern in an $n \times n$ array of characters. (The pattern may be shifted vertically and horizontally, but it may not be rotated.) **32.2-4** Alice has a copy of a long $n$-bit file $A = \langle a_{n-1}, a_{n-2}, \ldots, a_0 \rangle$, and Bob similarly has an $n$-bit file $B = \langle b_{n-1}, b_{n-2}, \ldots, b_0 \rangle$. Alice and Bob wish to know if their files are identical. To avoid transmitting all of $A$ or $B$, they use the following fast probabilistic check. Together, they select a prime $q > 1000n$ and randomly select an integer $x$ from $\{0, 1, \ldots, q-1\}$. Then, Alice evaluates

A(x) = \Bigl(\sum_{i=0}^{n-1} a_i x^i\Bigr) \bmod q

and Bob similarly evaluates $B(x)$. Prove that if $A \ne B$, there is at most one chance in $1000$ that $A(x) = B(x)$, whereas if the two files are the same, $A(x)$ is necessarily the same as $B(x)$. (*Hint:* See Exercise 31.4-4.) ## 32.3 String matching with finite automata Many string-matching algorithms build a finite automaton, a simple machine for processing information, that scans the text string $T$ for all occurrences of the pattern $P$. This section presents a method for building such an automaton. These string-matching automata are very efficient: they examine each text character exactly once, taking constant time per text character. The matching time used, after preprocessing the pattern to build the automaton, is therefore $\Theta(n)$. The time to build the automaton, however, can be large if $\Sigma$ is large. Section 32.4 describes a clever way around this problem. We begin this section with the definition of a finite automaton. We then examine a special string-matching automaton and show how to use it to find occurrences of a pattern in a text. Finally, we shall show how to construct the string-matching automaton for a given input pattern. ### Finite automata A **finite automaton** $M$, illustrated in Figure 32.6, is a 5-tuple $(Q, q_0, A, \Sigma, \delta)$, where - $Q$ is a finite set of **states**, - $q_0 \in Q$ is the **start state**, - $A \subseteq Q$ is a distinguished set of **accepting states**, - $\Sigma$ is a finite **input alphabet**, - $\delta$ is a function from $Q \times \Sigma$ into $Q$, called the **transition function** of $M$. *Figure 32.6: A simple two-state finite automaton with state set $Q = \{0, 1\}$, start state $q_0 = 0$, and input alphabet $\Sigma = \{a, b\}$. (a) A tabular representation of the transition function $\delta$. (b) An equivalent state-transition diagram. State 1, shown blackened, is the only accepting state. Directed edges represent transitions. For example, the edge from state 1 to state 0 labeled $b$ indicates that $\delta(1, b) = 0$. This automaton accepts those strings that end in an odd number of $a$'s. More precisely, it accepts a string $x$ if and only if $x = yz$, where $y = \varepsilon$ or $y$ ends with a $b$, and $z = a^k$, where $k$ is odd. For example, on input $abaaa$, including the start state, this automaton enters the sequence of states $\langle 0, 1, 0, 1, 0, 1 \rangle$, and so it accepts this input. For input $abbaa$, it enters the sequence of states $\langle 0, 1, 0, 0, 1, 0 \rangle$, and so it rejects this input.* | state | input $a$ | input $b$ | |---|---|---| | 0 | 1 | 0 | | 1 | 0 | 0 | The finite automaton begins in state $q_0$ and reads the characters of its input string one at a time. If the automaton is in state $q$ and reads input character $a$, it moves ("makes a transition") from state $q$ to state $\delta(q, a)$. Whenever its current state $q$ is a member of $A$, the machine $M$ has **accepted** the string read so far. An input that is not accepted is **rejected**. A finite automaton $M$ induces a function $\varphi$, called the **final-state function**, from $\Sigma^*$ to $Q$ such that $\varphi(w)$ is the state $M$ ends up in after scanning the string $w$. Thus, $M$ accepts a string $w$ if and only if $\varphi(w) \in A$. We define the function $\varphi$ recursively, using the transition function:

\begin{aligned} \varphi(\varepsilon) &= q_0 , \ \varphi(wa) &= \delta(\varphi(w), a) \quad \text{for } w \in \Sigma^*,\ a \in \Sigma . \end{aligned}

### String-matching automata For a given pattern $P$, we construct a string-matching automaton in a preprocessing step before using it to search the text string. Figure 32.7 illustrates the automaton for the pattern $P = ababaca$. From now on, we shall assume that $P$ is a given fixed pattern string; for brevity, we shall not indicate the dependence upon $P$ in our notation. In order to specify the string-matching automaton corresponding to a given pattern $P[1..m]$, we first define an auxiliary function $\sigma$, called the **suffix function** corresponding to $P$. The function $\sigma$ maps $\Sigma^*$ to $\{0, 1, \ldots, m\}$ such that $\sigma(x)$ is the length of the longest prefix of $P$ that is also a suffix of $x$:

\sigma(x) = \max{k : P_k \sqsupset x} . \tag{32.3}

*Figure 32.7: (a) A state-transition diagram for the string-matching automaton that accepts all strings ending in the string $ababaca$. State 0 is the start state, and state 7 (shown blackened) is the only accepting state. The transition function $\delta$ is defined by equation (32.4), and a directed edge from state $i$ to state $j$ labeled $a$ represents $\delta(i, a) = j$. The right-going edges forming the "spine" of the automaton, shown heavy in the figure, correspond to successful matches between pattern and input characters. Except for the edges from state 7 to states 1 and 2, the left-going edges correspond to mismatches. Some edges corresponding to mismatches are omitted; by convention, if a state $i$ has no outgoing edge labeled $a$ for some $a \in \Sigma$, then $\delta(i, a) = 0$. (b) The corresponding transition function $\delta$, and the pattern string $P = ababaca$. The entries corresponding to successful matches between pattern and input characters are shown shaded. (c) The operation of the automaton on the text $T = abababacaba$. Under each text character $T[i]$ appears the state $\varphi(T_i)$ that the automaton is in after processing the prefix $T_i$. The automaton finds one occurrence of the pattern, ending in position 9.* The transition function of part (b) (computed directly from equation (32.4)): | state | $a$ | $b$ | $c$ | $P$ | |---|---|---|---|---| | 0 | 1 | 0 | 0 | $a$ | | 1 | 1 | 2 | 0 | $b$ | | 2 | 3 | 0 | 0 | $a$ | | 3 | 1 | 4 | 0 | $b$ | | 4 | 5 | 0 | 0 | $a$ | | 5 | 1 | 4 | 6 | $c$ | | 6 | 7 | 0 | 0 | $a$ | | 7 | 1 | 2 | 0 | | The run on $T = abababacaba$ (part (c)): | $i$ | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | |---|---|---|---|---|---|---|---|---|---|---|---|---| | $T[i]$ | | $a$ | $b$ | $a$ | $b$ | $a$ | $b$ | $a$ | $c$ | $a$ | $b$ | $a$ | | state $\varphi(T_i)$ | 0 | 1 | 2 | 3 | 4 | 5 | 4 | 5 | 6 | 7 | 2 | 3 | The suffix function $\sigma$ is well defined since the empty string $P_0 = \varepsilon$ is a suffix of every string. As examples, for the pattern $P = ab$, we have $\sigma(\varepsilon) = 0$, $\sigma(ccaca) = 1$, and $\sigma(ccab) = 2$. For a pattern $P$ of length $m$, we have $\sigma(x) = m$ if and only if $P \sqsupset x$. From the definition of the suffix function, $x \sqsupset y$ implies $\sigma(x) \le \sigma(y)$. We define the **string-matching automaton** that corresponds to a given pattern $P[1..m]$ as follows: - The state set $Q$ is $\{0, 1, \ldots, m\}$. The start state $q_0$ is state $0$, and state $m$ is the only accepting state. - The transition function $\delta$ is defined by the following equation, for any state $q$ and character $a$:

\delta(q, a) = \sigma(P_q a) . \tag{32.4}

We define $\delta(q, a) = \sigma(P_q a)$ because we want to keep track of the longest prefix of the pattern $P$ that has matched the text string $T$ so far. We consider the most recently read characters of $T$. In order for a substring of $T$ (say, the substring ending at $T[i]$) to match some prefix $P_j$ of $P$, this prefix $P_j$ must be a suffix of $T_i$. Suppose that $q = \varphi(T_i)$, so that after reading $T_i$, the automaton is in state $q$. We design the transition function $\delta$ so that this state number, $q$, tells us the length of the longest prefix of $P$ that matches a suffix of $T_i$. That is, in state $q$, $P_q \sqsupset T_i$ and $q = \sigma(T_i)$. (Whenever $q = m$, all $m$ characters of $P$ match a suffix of $T_i$, and so we have found a match.) Thus, since $\varphi(T_i)$ and $\sigma(T_i)$ both equal $q$, we shall see (in Theorem 32.4, below) that the automaton maintains the following invariant:

\varphi(T_i) = \sigma(T_i) . \tag{32.5}

If the automaton is in state $q$ and reads the next character $T[i+1] = a$, then we want the transition to lead to the state corresponding to the longest prefix of $P$ that is a suffix of $T_i a$, and that state is $\sigma(T_i a)$. Because $P_q$ is the longest prefix of $P$ that is a suffix of $T_i$, the longest prefix of $P$ that is a suffix of $T_i a$ is not only $\sigma(T_i a)$, but also $\sigma(P_q a)$. (Lemma 32.3 proves that $\sigma(T_i a) = \sigma(P_q a)$.) Thus, when the automaton is in state $q$, we want the transition function on character $a$ to take the automaton to state $\sigma(P_q a)$. There are two cases to consider. - **Case 1:** $a = P[q+1]$, so that the character $a$ continues to match the pattern. In this case, because $\delta(q, a) = q + 1$, the transition continues to go along the "spine" of the automaton (the heavy edges in Figure 32.7). - **Case 2:** $a \ne P[q+1]$, so that $a$ does not continue to match the pattern. Here, we must find a smaller prefix of $P$ that is also a suffix of $T_i$. Because the preprocessing step matches the pattern against itself when creating the string-matching automaton, the transition function quickly identifies the longest such smaller prefix of $P$. Let us look at an example. The string-matching automaton of Figure 32.7 has $\delta(5, c) = 6$, illustrating the first case, in which the match continues. To illustrate the second case, observe that the automaton of Figure 32.7 has $\delta(5, b) = 4$. We make this transition because if the automaton reads a $b$ in state $q = 5$, then $P_q b = ababab$, and the longest prefix of $P$ that is also a suffix of $ababab$ is $P_4 = abab$. > **Think about it.** Why can the automaton never need to remember more than a single number $q$? Because $q$ already encodes the whole relevant past: the last $q$ characters read are exactly $P_q$. *Figure 32.8: An illustration for the proof of Lemma 32.2. The figure shows that $r \le \sigma(x) + 1$, where $r = \sigma(xa)$. (It aligns $P_r$ as a suffix of $xa$ and $P_{r-1}$ as a suffix of $x$.)* To clarify the operation of a string-matching automaton, we now give a simple, efficient program for simulating the behavior of such an automaton (represented by its transition function $\delta$) in finding occurrences of a pattern $P$ of length $m$ in an input text $T[1..n]$. As for any string-matching automaton for a pattern of length $m$, the state set $Q$ is $\{0, 1, \ldots, m\}$, the start state is $0$, and the only accepting state is state $m$. ```text FINITE-AUTOMATON-MATCHER(T, δ, m) 1 n = T.length 2 q = 0 3 for i = 1 to n 4 q = δ(q, T[i]) 5 if q == m 6 print "Pattern occurs with shift" i - m ``` From the simple loop structure of FINITE-AUTOMATON-MATCHER, we can easily see that its matching time on a text string of length $n$ is $\Theta(n)$. This matching time, however, does not include the preprocessing time required to compute the transition function $\delta$. We address this problem later, after first proving that the procedure FINITE-AUTOMATON-MATCHER operates correctly. Consider how the automaton operates on an input text $T[1..n]$. We shall prove that the automaton is in state $\sigma(T_i)$ after scanning character $T[i]$. Since $\sigma(T_i) = m$ if and only if $P \sqsupset T_i$, the machine is in the accepting state $m$ if and only if it has just scanned the pattern $P$. To prove this result, we make use of the following two lemmas about the suffix function $\sigma$. **Lemma 32.2 (Suffix-function inequality).** For any string $x$ and character $a$, we have $\sigma(xa) \le \sigma(x) + 1$. > **Proof.** Referring to Figure 32.8, let $r = \sigma(xa)$. > > **Case $r = 0$.** > $$ > \sigma(xa) = r = 0 \le \sigma(x) + 1 , > $$ > which holds trivially because $\sigma(x) \ge 0$ (it is a length). > > **Case $r > 0$.** > $$ > \begin{aligned} > P_r &\sqsupset xa && \text{(by the definition of } \sigma\text{, since } r = \sigma(xa)\text{)} \\ > P_{r-1} &\sqsupset x && \text{(drop the last character } a \text{ from both } P_r \text{ and } xa\text{; the last character of } P_r \text{ is } a\text{)} \\ > r - 1 &\le \sigma(x) && \text{(since } \sigma(x) \text{ is the largest } k \text{ with } P_k \sqsupset x\text{)} \\ > \sigma(xa) = r &\le \sigma(x) + 1 . > \end{aligned} > $$ > $\blacksquare$ *Figure 32.9: An illustration for the proof of Lemma 32.3. The figure shows that $r = \sigma(P_q a)$, where $q = \sigma(x)$ and $r = \sigma(xa)$. (It shows $P_q$ as a suffix of $x$, $P_q a$ as a suffix of $xa$, and $P_r$ as a suffix of $xa$.)* **Lemma 32.3 (Suffix-function recursion lemma).** For any string $x$ and character $a$, if $q = \sigma(x)$, then $\sigma(xa) = \sigma(P_q a)$. > **Proof.** Let $q = \sigma(x)$ and $r = \sigma(xa)$. We show $r \le \sigma(P_q a)$ and $\sigma(P_q a) \le r$. > > **Step 1: $\sigma(xa) \le \sigma(P_q a)$.** > $$ > \begin{aligned} > P_q &\sqsupset x && \text{(by the definition of } \sigma\text{, since } q = \sigma(x)\text{)} \\ > P_q a &\sqsupset xa && \text{(append the same character } a \text{ to both strings, as in Figure 32.9)} \\ > P_r &\sqsupset xa && \text{(by the definition of } \sigma\text{, since } r = \sigma(xa)\text{)} \\ > r &\le q + 1 && \text{(by Lemma 32.2)} \\ > |P_r| = r &\le q + 1 = |P_q a| && \text{(lengths of the two strings)} \\ > P_r &\sqsupset P_q a && \text{(Lemma 32.1: both are suffixes of } xa \text{ and } |P_r| \le |P_q a|\text{)} \\ > r &\le \sigma(P_q a) && \text{(} \sigma(P_q a) \text{ is the largest } k \text{ with } P_k \sqsupset P_q a\text{)} > \end{aligned} > $$ > That is, $\sigma(xa) \le \sigma(P_q a)$. > > **Step 2: $\sigma(P_q a) \le \sigma(xa)$.** Since $P_q a \sqsupset xa$ (shown above) and $\sigma$ is monotone under the suffix relation ($y \sqsupset z$ implies $\sigma(y) \le \sigma(z)$), we have $\sigma(P_q a) \le \sigma(xa)$. > > **Conclusion.** Combining the two inequalities, $\sigma(xa) = \sigma(P_q a)$. $\blacksquare$ We are now ready to prove our main theorem characterizing the behavior of a string-matching automaton on a given input text. As noted above, this theorem shows that the automaton is merely keeping track, at each step, of the longest prefix of the pattern that is a suffix of what has been read so far. In other words, the automaton maintains the invariant (32.5). **Theorem 32.4.** If $\varphi$ is the final-state function of a string-matching automaton for a given pattern $P$ and $T[1..n]$ is an input text for the automaton, then

\varphi(T_i) = \sigma(T_i) \quad \text{for } i = 0, 1, \ldots, n .

> **Proof.** The proof is by induction on $i$. > > **Base case ($i = 0$).** Since $T_0 = \varepsilon$, > $$ > \varphi(T_0) = \varphi(\varepsilon) = 0 = \sigma(\varepsilon) = \sigma(T_0) , > $$ > because the automaton starts in state $0$, and the empty string has $\sigma(\varepsilon) = 0$ (only $P_0 = \varepsilon$ is a suffix of $\varepsilon$). > > **Inductive step.** Assume $\varphi(T_i) = \sigma(T_i)$ and prove $\varphi(T_{i+1}) = \sigma(T_{i+1})$. Let $q$ denote $\varphi(T_i)$, and let $a$ denote $T[i+1]$. Then > $$ > \begin{aligned} > \varphi(T_{i+1}) &= \varphi(T_i a) && \text{(by the definitions of } T_{i+1} \text{ and } a\text{)} \\ > &= \delta(\varphi(T_i), a) && \text{(by the definition of } \varphi\text{)} \\ > &= \delta(q, a) && \text{(by the definition of } q\text{)} \\ > &= \sigma(P_q a) && \text{(by the definition (32.4) of } \delta\text{)} \\ > &= \sigma(T_i a) && \text{(by Lemma 32.3, since } q = \varphi(T_i) = \sigma(T_i) \text{ by the induction hypothesis)} \\ > &= \sigma(T_{i+1}) && \text{(by the definition of } T_{i+1}\text{)} . > \end{aligned} > $$ > $\blacksquare$ By Theorem 32.4, if the machine enters state $q$ on line 4, then $q$ is the largest value such that $P_q \sqsupset T_i$. Thus, we have $q = m$ on line 5 if and only if the machine has just scanned an occurrence of the pattern $P$. We conclude that FINITE-AUTOMATON-MATCHER operates correctly. ### Computing the transition function The following procedure computes the transition function $\delta$ from a given pattern $P[1..m]$. ```text COMPUTE-TRANSITION-FUNCTION(P, Σ) 1 m = P.length 2 for q = 0 to m 3 for each character a ∈ Σ 4 k = min(m + 1, q + 2) 5 repeat 6 k = k - 1 7 until P_k ⊐ P_q a 8 δ(q, a) = k 9 return δ ``` This procedure computes $\delta(q, a)$ in a straightforward manner according to its definition in equation (32.4). The nested loops beginning on lines 2 and 3 consider all states $q$ and all characters $a$, and lines 4-8 set $\delta(q, a)$ to be the largest $k$ such that $P_k \sqsupset P_q a$. The code starts with the largest conceivable value of $k$, which is $\min(m, q+1)$. It then decreases $k$ until $P_k \sqsupset P_q a$, which must eventually occur, since $P_0 = \varepsilon$ is a suffix of every string. > **Running time.** > $$ > \begin{aligned} > \text{outer loops (lines 2-3)} &: (m+1)\,|\Sigma| \text{ iterations} = O(m\,|\Sigma|) , \\ > \text{repeat loop (lines 5-7)} &: \text{at most } m + 1 \text{ iterations per pair } (q, a) , \\ > \text{test } P_k \sqsupset P_q a \text{ (line 7)} &: \text{compares up to } m \text{ characters.} > \end{aligned} > $$ > Multiplying the three factors, the running time of COMPUTE-TRANSITION-FUNCTION is $O(m^3 |\Sigma|)$. Much faster procedures exist; by utilizing some cleverly computed information about the pattern $P$ (see Exercise 32.4-8), we can improve the time required to compute $\delta$ from $P$ to $O(m\,|\Sigma|)$. With this improved procedure for computing $\delta$, we can find all occurrences of a length-$m$ pattern in a length-$n$ text over an alphabet $\Sigma$ with $O(m\,|\Sigma|)$ preprocessing time and $\Theta(n)$ matching time. ### Exercises 32.3-1 to 32.3-5 **32.3-1** Construct the string-matching automaton for the pattern $P = aabab$ and illustrate its operation on the text string $T = aaababaabaababaab$. **32.3-2** Draw a state-transition diagram for a string-matching automaton for the pattern $ababbabbababbababbabb$ over the alphabet $\Sigma = \{a, b\}$. **32.3-3** We call a pattern $P$ **nonoverlappable** if $P_k \sqsupset P_q$ implies $k = 0$ or $k = q$. Describe the state-transition diagram of the string-matching automaton for a nonoverlappable pattern. **32.3-4\*** Given two patterns $P$ and $P'$, describe how to construct a finite automaton that determines all occurrences of either pattern. Try to minimize the number of states in your automaton. **32.3-5** Given a pattern $P$ containing gap characters (see Exercise 32.1-4), show how to build a finite automaton that can find an occurrence of $P$ in a text $T$ in $O(n)$ matching time, where $n = |T|$. ## 32.4 The Knuth-Morris-Pratt algorithm We now present a linear-time string-matching algorithm due to Knuth, Morris, and Pratt. This algorithm avoids computing the transition function $\delta$ altogether, and its matching time is $\Theta(n)$ using just an auxiliary function $\pi$, which we precompute from the pattern in time $\Theta(m)$ and store in an array $\pi[1..m]$. The array $\pi$ allows us to compute the transition function $\delta$ efficiently (in an amortized sense) "on the fly" as needed. Loosely speaking, for any state $q = 0, 1, \ldots, m$ and any character $a \in \Sigma$, the value $\pi[q]$ contains the information we need to compute $\delta(q, a)$ but that does not depend on $a$. Since the array $\pi$ has only $m$ entries, whereas $\delta$ has $\Theta(m\,|\Sigma|)$ entries, we save a factor of $|\Sigma|$ in the preprocessing time by computing $\pi$ rather than $\delta$. ### The prefix function for a pattern The prefix function $\pi$ for a pattern encapsulates knowledge about how the pattern matches against shifts of itself. We can take advantage of this information to avoid testing useless shifts in the naive pattern-matching algorithm and to avoid precomputing the full transition function $\delta$ for a string-matching automaton. Consider the operation of the naive string matcher. Figure 32.10(a) shows a particular shift $s$ of a template containing the pattern $P = ababaca$ against a text $T$. For this example, $q = 5$ of the characters have matched successfully, but the 6th pattern character fails to match the corresponding text character. The information that $q$ characters have matched successfully determines the corresponding text characters. Knowing these $q$ text characters allows us to determine immediately that certain shifts are invalid. In the example of the figure, the shift $s+1$ is necessarily invalid, since the first pattern character ($a$) would be aligned with a text character that we know does not match the first pattern character, but does match the second pattern character ($b$). The shift $s' = s+2$ shown in part (b) of the figure, however, aligns the first three pattern characters with three text characters that must necessarily match. In general, it is useful to know the answer to the following question: > Given that pattern characters $P[1..q]$ match text characters $T[s+1..s+q]$, what is the least shift $s' > s$ such that for some $k < q$, > $$ > P[1..k] = T[s'+1..s'+k] , \tag{32.6} > $$ > where $s' + k = s + q$? In other words, knowing that $P_q \sqsupset T_{s+q}$, we want the longest proper prefix $P_k$ of $P_q$ that is also a suffix of $T_{s+q}$. (Since $s' + k = s + q$, if we are given $s$ and $q$, then finding the smallest shift $s'$ is tantamount to finding the longest prefix length $k$.) We add the difference $q - k$ in the lengths of these prefixes of $P$ to the shift $s$ to arrive at our new shift $s'$, so that $s' = s + (q - k)$. In the best case, $k = 0$, so that $s' = s + q$, and we immediately rule out shifts $s+1, s+2, \ldots, s+q-1$. In any case, at the new shift $s'$ we do not need to compare the first $k$ characters of $P$ with the corresponding characters of $T$, since equation (32.6) guarantees that they match. *Figure 32.10: The prefix function $\pi$. (a) The pattern $P = ababaca$ aligns with a text $T$ so that the first $q = 5$ characters match. Matching characters, shown shaded, are connected by vertical lines. (b) Using only our knowledge of the 5 matched characters, we can deduce that a shift of $s+1$ is invalid, but that a shift of $s' = s+2$ is consistent with everything we know about the text and therefore is potentially valid. (c) We can precompute useful information for such deductions by comparing the pattern with itself. Here, we see that the longest prefix of $P$ that is also a proper suffix of $P_5$ is $P_3$. We represent this precomputed information in the array $\pi$, so that $\pi[5] = 3$. Given that $q$ characters have matched successfully at shift $s$, the next potentially valid shift is at $s' = s + (q - \pi[q])$ as shown in part (b).* We can precompute the necessary information by comparing the pattern against itself, as Figure 32.10(c) demonstrates. Since $T[s'+1..s'+k]$ is part of the known portion of the text, it is a suffix of the string $P_q$. Therefore, we can interpret equation (32.6) as asking for the greatest $k < q$ such that $P_k \sqsupset P_q$. Then, the new shift $s' = s + (q - k)$ is the next potentially valid shift. We will find it convenient to store, for each value of $q$, the number $k$ of matching characters at the new shift $s'$, rather than storing, say, $s' - s$. We formalize the information that we precompute as follows. Given a pattern $P[1..m]$, the **prefix function** for the pattern $P$ is the function $\pi : \{1, 2, \ldots, m\} \to \{0, 1, \ldots, m-1\}$ such that

\pi[q] = \max{k : k < q \text{ and } P_k \sqsupset P_q} .

That is, $\pi[q]$ is the length of the longest prefix of $P$ that is a proper suffix of $P_q$. Figure 32.11(a) gives the complete prefix function $\pi$ for the pattern $ababaca$. *Figure 32.11: An illustration of Lemma 32.5 for the pattern $P = ababaca$ and $q = 5$. (a) The $\pi$ function for the given pattern. Since $\pi[5] = 3$, $\pi[3] = 1$, and $\pi[1] = 0$, by iterating $\pi$ we obtain $\pi^*[5] = \{3, 1, 0\}$. (b) We slide the template containing the pattern $P$ to the right and note when some prefix $P_k$ of $P$ matches up with some proper suffix of $P_5$; we get matches when $k = 3$, $1$, and $0$. In the figure, the first row gives $P$, and the dotted vertical line is drawn just after $P_5$. Successive rows show all the shifts of $P$ that cause some prefix $P_k$ of $P$ to match some suffix of $P_5$. Successfully matched characters are shown shaded. Vertical lines connect aligned matching characters. Thus, $\{k : k < 5 \text{ and } P_k \sqsupset P_5\} = \{3, 1, 0\}$. Lemma 32.5 claims that $\pi^*[q] = \{k : k < q \text{ and } P_k \sqsupset P_q\}$ for all $q$.* | $i$ | 1 | 2 | 3 | 4 | 5 | 6 | 7 | |---|---|---|---|---|---|---|---| | $P[i]$ | $a$ | $b$ | $a$ | $b$ | $a$ | $c$ | $a$ | | $\pi[i]$ | 0 | 0 | 1 | 2 | 3 | 0 | 1 | The pseudocode below gives the Knuth-Morris-Pratt matching algorithm as the procedure KMP-MATCHER. For the most part, the procedure follows from FINITE-AUTOMATON-MATCHER, as we shall see. KMP-MATCHER calls the auxiliary procedure COMPUTE-PREFIX-FUNCTION to compute $\pi$. ```text KMP-MATCHER(T, P) 1 n = T.length 2 m = P.length 3 π = COMPUTE-PREFIX-FUNCTION(P) 4 q = 0 // number of characters matched 5 for i = 1 to n // scan the text from left to right 6 while q > 0 and P[q + 1] ≠ T[i] 7 q = π[q] // next character does not match 8 if P[q + 1] == T[i] 9 q = q + 1 // next character matches 10 if q == m // is all of P matched? 11 print "Pattern occurs with shift" i - m 12 q = π[q] // look for the next match ``` ```text COMPUTE-PREFIX-FUNCTION(P) 1 m = P.length 2 let π[1..m] be a new array 3 π[1] = 0 4 k = 0 5 for q = 2 to m 6 while k > 0 and P[k + 1] ≠ P[q] 7 k = π[k] 8 if P[k + 1] == P[q] 9 k = k + 1 10 π[q] = k 11 return π ``` These two procedures have much in common, because both match a string against the pattern $P$: KMP-MATCHER matches the text $T$ against $P$, and COMPUTE-PREFIX-FUNCTION matches $P$ against itself. We begin with an analysis of the running times of these procedures. Proving these procedures correct will be more complicated. ### Running-time analysis The running time of COMPUTE-PREFIX-FUNCTION is $\Theta(m)$, which we show by using the aggregate method of amortized analysis (see Section 17.1). The only tricky part is showing that the **while** loop of lines 6-7 executes $O(m)$ times altogether. We shall show that it makes at most $m - 1$ iterations. > **Proof.** We make some observations about $k$. > > 1. **Total increase of $k$.** Line 4 starts $k$ at $0$, and the only way that $k$ increases is by the increment operation in line 9, which executes at most once per iteration of the **for** loop of lines 5-10. That loop runs for $q = 2, \ldots, m$, which is $m - 1$ iterations. Thus, the total increase in $k$ is at most $m - 1$. > 2. **Each while iteration strictly decreases $k$.** We have $k < q$ upon entering the **for** loop (initially $k = 0 < 2$), and each iteration of the loop increments $q$ by $1$ while $k$ grows by at most $1$, so we always have $k < q$. Therefore, the assignments in lines 3 and 10 ensure that $\pi[q] < q$ for all $q = 1, 2, \ldots, m$, which means that each iteration of the **while** loop (line 7: $k = \pi[k]$ with $\pi[k] < k$) decreases $k$. > 3. **$k$ never becomes negative.** The **while** test requires $k > 0$, and $\pi[k] \ge 0$. > > Putting these facts together: > $$ > \begin{aligned} > \text{(number of while iterations)} &\le \text{(total decrease of } k \text{ by the while loop)} && \text{(each iteration decreases } k \text{ by at least 1)} \\ > &\le \text{(total increase of } k \text{ by line 9)} && \text{(} k \text{ starts at } 0 \text{ and never goes negative)} \\ > &\le m - 1 . > \end{aligned} > $$ > Thus, the **while** loop iterates at most $m - 1$ times in all. Every other line executes $O(1)$ times per **for** iteration, so COMPUTE-PREFIX-FUNCTION runs in time $\Theta(m)$. $\blacksquare$ Exercise 32.4-4 asks you to show, by a similar aggregate analysis, that the matching time of KMP-MATCHER is $\Theta(n)$. Compared with FINITE-AUTOMATON-MATCHER, by using $\pi$ rather than $\delta$, we have reduced the time for preprocessing the pattern from $O(m\,|\Sigma|)$ to $\Theta(m)$, while keeping the actual matching time bounded by $\Theta(n)$. > **Think about it.** Variable $k$ is like a bank balance: line 9 deposits at most one coin per step, and each trip through the **while** loop spends at least one. You cannot spend more than you ever deposited. ### Correctness of the prefix-function computation We shall see a little later that the prefix function $\pi$ helps us simulate the transition function $\delta$ in a string-matching automaton. But first, we need to prove that the procedure COMPUTE-PREFIX-FUNCTION does indeed compute the prefix function correctly. In order to do so, we will need to find all prefixes $P_k$ that are proper suffixes of a given prefix $P_q$. The value of $\pi[q]$ gives us the longest such prefix, but the following lemma, illustrated in Figure 32.11, shows that by iterating the prefix function $\pi$, we can indeed enumerate all the prefixes $P_k$ that are proper suffixes of $P_q$. Let

\pi^*[q] = {\pi[q],\ \pi^{(2)}[q],\ \pi^{(3)}[q],\ \ldots,\ \pi^{(t)}[q]} ,

where $\pi^{(i)}[q]$ is defined in terms of functional iteration, so that $\pi^{(0)}[q] = q$ and $\pi^{(i)}[q] = \pi[\pi^{(i-1)}[q]]$ for $i \ge 1$, and where the sequence in $\pi^*[q]$ stops upon reaching $\pi^{(t)}[q] = 0$. **Lemma 32.5 (Prefix-function iteration lemma).** Let $P$ be a pattern of length $m$ with prefix function $\pi$. Then, for $q = 1, 2, \ldots, m$, we have

\pi^*[q] = {k : k < q \text{ and } P_k \sqsupset P_q} .

> **Proof.** Write $S = \{k : k < q \text{ and } P_k \sqsupset P_q\}$. We show $\pi^*[q] \subseteq S$ and $S \subseteq \pi^*[q]$. > > **Part 1: $\pi^*[q] \subseteq S$.** It suffices to show > $$ > i \in \pi^*[q] \text{ implies } i < q \text{ and } P_i \sqsupset P_q . \tag{32.7} > $$ > If $i \in \pi^*[q]$, then $i = \pi^{(u)}[q]$ for some $u \ge 1$. We prove (32.7) by induction on $u$. > > - *Base case $u = 1$.* Then $i = \pi[q]$, and the claim holds, since $i < q$ and $P_{\pi[q]} \sqsupset P_q$ by the definition of $\pi$. > - *Inductive step $u > 1$.* Let $j = \pi^{(u-1)}[q]$, so $i = \pi[j]$. By the induction hypothesis, $j < q$ and $P_j \sqsupset P_q$. By the definition of $\pi$ applied to $j$, we have $i < j$ and $P_i \sqsupset P_j$. Then > $$ > \begin{aligned} > i &< j < q && \text{(transitivity of } < \text{)} \\ > P_i &\sqsupset P_j \sqsupset P_q && \text{(so } P_i \sqsupset P_q \text{ by transitivity of } \sqsupset \text{)} . > \end{aligned} > $$ > > Therefore $\pi^*[q] \subseteq S$. > > **Part 2: $S \subseteq \pi^*[q]$, by contradiction.** Suppose to the contrary that $S - \pi^*[q]$ is nonempty, and let $j$ be the largest number in it. > > 1. Because $\pi[q]$ is the largest value in $S$ and $\pi[q] \in \pi^*[q]$, the element $j \in S - \pi^*[q]$ cannot equal $\pi[q]$, so $j < \pi[q]$. > 2. Let $j'$ denote the smallest integer in $\pi^*[q]$ that is greater than $j$. (We can choose $j' = \pi[q]$ if no other number in $\pi^*[q]$ is greater than $j$.) > 3. We have $P_j \sqsupset P_q$ because $j \in S$, and from $j' \in \pi^*[q]$ and equation (32.7), we have $P_{j'} \sqsupset P_q$ and $j' < q$. > 4. Since $j < j'$, both $P_j$ and $P_{j'}$ are suffixes of $P_q$ with $|P_j| < |P_{j'}|$. Thus, $P_j \sqsupset P_{j'}$ by Lemma 32.1. > 5. We claim that $j$ is the largest value less than $j'$ with $P_j \sqsupset P_{j'}$. Suppose some $k$ with $j < k < j'$ had $P_k \sqsupset P_{j'}$. Then $P_k \sqsupset P_q$ by transitivity and $k < q$, so $k \in S$. Since $k > j$ and $j$ is the largest element of $S - \pi^*[q]$, we get $k \in \pi^*[q]$. But then $k$ is an element of $\pi^*[q]$ with $j < k < j'$, contradicting the choice of $j'$ as the smallest such element. > 6. Therefore, by the definition of $\pi$, we must have $\pi[j'] = j$. > 7. Since $j' \in \pi^*[q]$, the next element of the iteration, $\pi[j']$, is also in $\pi^*[q]$ (the sequence $\pi^*[q]$ is closed under applying $\pi$). Hence $j = \pi[j'] \in \pi^*[q]$. > > This contradicts $j \in S - \pi^*[q]$. This contradiction proves the lemma. $\blacksquare$ The algorithm COMPUTE-PREFIX-FUNCTION computes $\pi[q]$, in order, for $q = 1, 2, \ldots, m$. Setting $\pi[1]$ to $0$ in line 3 of COMPUTE-PREFIX-FUNCTION is certainly correct, since $\pi[q] < q$ for all $q$. We shall use the following lemma and its corollary to prove that COMPUTE-PREFIX-FUNCTION computes $\pi[q]$ correctly for $q > 1$. **Lemma 32.6.** Let $P$ be a pattern of length $m$, and let $\pi$ be the prefix function for $P$. For $q = 1, 2, \ldots, m$, if $\pi[q] > 0$, then $\pi[q] - 1 \in \pi^*[q-1]$. > **Proof.** Let $r = \pi[q] > 0$. > $$ > \begin{aligned} > r < q \ \text{and}\ P_r &\sqsupset P_q && \text{(by the definition of } \pi\text{)} \\ > r - 1 < q - 1 \ \text{and}\ P_{r-1} &\sqsupset P_{q-1} && \text{(drop the last character from } P_r \text{ and } P_q\text{, allowed since } r > 0\text{)} \\ > r - 1 &\in \pi^*[q-1] && \text{(by Lemma 32.5)} \\ > \pi[q] - 1 = r - 1 &\in \pi^*[q-1] . > \end{aligned} > $$ > $\blacksquare$ For $q = 2, 3, \ldots, m$, define the subset $E_{q-1} \subseteq \pi^*[q-1]$ by

\begin{aligned} E_{q-1} &= {k \in \pi^*[q-1] : P[k+1] = P[q]} \ &= {k : k < q-1 \text{ and } P_k \sqsupset P_{q-1} \text{ and } P[k+1] = P[q]} \qquad \text{(by Lemma 32.5)} \ &= {k : k < q-1 \text{ and } P_{k+1} \sqsupset P_q} . \end{aligned}

The set $E_{q-1}$ consists of the values $k < q-1$ for which $P_k \sqsupset P_{q-1}$ and for which, because $P[k+1] = P[q]$, we have $P_{k+1} \sqsupset P_q$. Thus, $E_{q-1}$ consists of those values $k \in \pi^*[q-1]$ such that we can extend $P_k$ to $P_{k+1}$ and get a proper suffix of $P_q$. **Corollary 32.7.** Let $P$ be a pattern of length $m$, and let $\pi$ be the prefix function for $P$. For $q = 2, 3, \ldots, m$,

\pi[q] = \begin{cases} 0 & \text{if } E_{q-1} = \emptyset , \ 1 + \max{k \in E_{q-1}} & \text{if } E_{q-1} \neq \emptyset . \end{cases}

\pi’[q] = \begin{cases} 0 & \text{if } \pi[q] = 0 , \ \pi’[\pi[q]] & \text{if } \pi[q] \ne 0 \text{ and } P[\pi[q]+1] = P[q+1] , \ \pi[q] & \text{if } \pi[q] \ne 0 \text{ and } P[\pi[q]+1] \ne P[q+1] . \end{cases}

Explain why the modified algorithm is correct, and explain in what sense this change constitutes an improvement. **32.4-7** Give a linear-time algorithm to determine whether a text $T$ is a cyclic rotation of another string $T'$. For example, $arc$ and $car$ are cyclic rotations of each other. **32.4-8\*** Give an $O(m\,|\Sigma|)$-time algorithm for computing the transition function $\delta$ for the string-matching automaton corresponding to a given pattern $P$. (Hint: Prove that $\delta(q, a) = \delta(\pi[q], a)$ if $q = m$ or $P[q+1] \ne a$.) ## Problems **32-1 String matching based on repetition factors.** Let $y^i$ denote the concatenation of string $y$ with itself $i$ times. For example, $(ab)^3 = ababab$. We say that a string $x \in \Sigma^*$ has **repetition factor** $r$ if $x = y^r$ for some string $y \in \Sigma^*$ and some $r > 0$. Let $\rho(x)$ denote the largest $r$ such that $x$ has repetition factor $r$. **a.** Give an efficient algorithm that takes as input a pattern $P[1..m]$ and computes the value $\rho(P_i)$ for $i = 1, 2, \ldots, m$. What is the running time of your algorithm? **b.** For any pattern $P[1..m]$, let $\rho^*(P)$ be defined as $\max_{1 \le i \le m} \rho(P_i)$. Prove that if the pattern $P$ is chosen randomly from the set of all binary strings of length $m$, then the expected value of $\rho^*(P)$ is $O(1)$. **c.** Argue that the following string-matching algorithm correctly finds all occurrences of pattern $P$ in a text $T[1..n]$ in time $O(\rho^*(P)\, n + m)$: ```text REPETITION-MATCHER(P, T) 1 m = P.length 2 n = T.length 3 k = 1 + ρ*(P) 4 q = 0 5 s = 0 6 while s ≤ n - m 7 if T[s + q + 1] == P[q + 1] 8 q = q + 1 9 if q == m 10 print "Pattern occurs with shift" s 11 if q == m or T[s + q + 1] ≠ P[q + 1] 12 s = s + max(1, ⌈q/k⌉) 13 q = 0 ``` This algorithm is due to Galil and Seiferas. By extending these ideas greatly, they obtained a linear-time string-matching algorithm that uses only $O(1)$ storage beyond what is required for $P$ and $T$. ## Chapter notes The relation of string matching to the theory of finite automata is discussed by Aho, Hopcroft, and Ullman [5]. The Knuth-Morris-Pratt algorithm [214] was invented independently by Knuth and Pratt and by Morris; they published their work jointly. Reingold, Urban, and Gries [294] give an alternative treatment of the Knuth-Morris-Pratt algorithm. The Rabin-Karp algorithm was proposed by Karp and Rabin [201]. Galil and Seiferas [126] give an interesting deterministic linear-time string-matching algorithm that uses only $O(1)$ space beyond that required to store the pattern and text. # 33 Computational Geometry > **Big picture.** Imagine teaching a robot to see the world only through coordinates: points, segments, polygons. How could it decide whether two roads cross, or find the fence of minimal length around a flock of sheep, without ever "looking" at a picture? This chapter shows how a handful of clever, division-free arithmetic tricks turn geometric intuition into fast, reliable algorithms. Computational geometry is the branch of computer science that studies algorithms for solving geometric problems. In modern engineering and mathematics, computational geometry has applications in such diverse fields as computer graphics, robotics, VLSI design, computer-aided design, molecular modeling, metallurgy, manufacturing, textile layout, forestry, and statistics. The input to a computational-geometry problem is typically a description of a set of geometric objects, such as a set of points, a set of line segments, or the vertices of a polygon in counterclockwise order. The output is often a response to a query about the objects, such as whether any of the lines intersect, or perhaps a new geometric object, such as the **convex hull** (smallest enclosing convex polygon) of the set of points. In this chapter, we look at a few computational-geometry algorithms in two dimensions, that is, in the plane. We represent each input object by a set of points $\{p_1, p_2, p_3, \ldots\}$, where each $p_i = (x_i, y_i)$ and $x_i, y_i \in \mathbb{R}$. For example, we represent an $n$-vertex polygon $P$ by a sequence $\langle p_0, p_1, p_2, \ldots, p_{n-1} \rangle$ of its vertices in order of their appearance on the boundary of $P$. Computational geometry can also apply to three dimensions, and even higher-dimensional spaces, but such problems and their solutions can be very difficult to visualize. Even in two dimensions, however, we can see a good sample of computational-geometry techniques. Section 33.1 shows how to answer basic questions about line segments efficiently and accurately: whether one segment is clockwise or counterclockwise from another that shares an endpoint, which way we turn when traversing two adjoining line segments, and whether two line segments intersect. Section 33.2 presents a technique called "sweeping" that we use to develop an $O(n \lg n)$-time algorithm for determining whether a set of $n$ line segments contains any intersections. Section 33.3 gives two "rotational-sweep" algorithms that compute the convex hull of a set of $n$ points: Graham's scan, which runs in time $O(n \lg n)$, and Jarvis's march, which takes $O(nh)$ time, where $h$ is the number of vertices of the convex hull. Finally, Section 33.4 gives an $O(n \lg n)$-time divide-and-conquer algorithm for finding the closest pair of points in a set of $n$ points in the plane. --- ## 33.1 Line-segment properties Several of the computational-geometry algorithms in this chapter require answers to questions about the properties of line segments. A **convex combination** of two distinct points $p_1 = (x_1, y_1)$ and $p_2 = (x_2, y_2)$ is any point $p_3 = (x_3, y_3)$ such that for some $\alpha$ in the range $0 \le \alpha \le 1$, we have

\begin{aligned} x_3 &= \alpha x_1 + (1-\alpha) x_2 , \ y_3 &= \alpha y_1 + (1-\alpha) y_2 . \end{aligned}

We also write that $p_3 = \alpha p_1 + (1-\alpha) p_2$. Intuitively, $p_3$ is any point that is on the line passing through $p_1$ and $p_2$ and is on or between $p_1$ and $p_2$ on the line. Given two distinct points $p_1$ and $p_2$, the **line segment** $\overline{p_1 p_2}$ is the set of convex combinations of $p_1$ and $p_2$. We call $p_1$ and $p_2$ the **endpoints** of segment $\overline{p_1 p_2}$. Sometimes the ordering of $p_1$ and $p_2$ matters, and we speak of the **directed segment** $\overrightarrow{p_1 p_2}$. If $p_1$ is the origin $(0,0)$, then we can treat the directed segment $\overrightarrow{p_1 p_2}$ as the vector $p_2$. In this section, we shall explore the following questions: 1. Given two directed segments $\overrightarrow{p_0 p_1}$ and $\overrightarrow{p_0 p_2}$, is $\overrightarrow{p_0 p_1}$ clockwise from $\overrightarrow{p_0 p_2}$ with respect to their common endpoint $p_0$? 2. Given two line segments $\overline{p_0 p_1}$ and $\overline{p_1 p_2}$, if we traverse $\overline{p_0 p_1}$ and then $\overline{p_1 p_2}$, do we make a left turn at point $p_1$? 3. Do line segments $\overline{p_1 p_2}$ and $\overline{p_3 p_4}$ intersect? There are no restrictions on the given points. We can answer each question in $O(1)$ time, which should come as no surprise since the input size of each question is $O(1)$. Moreover, our methods use only additions, subtractions, multiplications, and comparisons. We need neither division nor trigonometric functions, both of which can be computationally expensive and prone to problems with round-off error. For example, the "straightforward" method of determining whether two segments intersect (compute the line equation of the form $y = mx + b$ for each segment, where $m$ is the slope and $b$ is the $y$-intercept, find the point of intersection of the lines, and check whether this point is on both segments) uses division to find the point of intersection. When the segments are nearly parallel, this method is very sensitive to the precision of the division operation on real computers. The method in this section, which avoids division, is much more accurate. > **Think about it.** Why might dividing by a tiny number be dangerous? What happens to $1/\varepsilon$ when $\varepsilon$ is almost, but not quite, zero? ### Cross products Computing cross products lies at the heart of our line-segment methods. Consider vectors $p_1$ and $p_2$, shown in Figure 33.1(a). We can interpret the cross product $p_1 \times p_2$ as the signed area of the parallelogram formed by the points $(0,0)$, $p_1$, $p_2$, and $p_1 + p_2 = (x_1 + x_2, y_1 + y_2)$. An equivalent, but more useful, definition gives the cross product as the determinant of a matrix:[^1]

\begin{aligned} p_1 \times p_2 &= \det \begin{pmatrix} x_1 & x_2 \ y_1 & y_2 \end{pmatrix} \ &= x_1 y_2 - x_2 y_1 \ &= - p_2 \times p_1 . \end{aligned}

*Figure 33.1: (a) The cross product of vectors $p_1$ and $p_2$ is the signed area of the parallelogram with vertices $(0,0)$, $p_1$, $p_2$, and $p_1 + p_2$. (b) The lightly shaded region contains vectors that are clockwise from $p$. The heavily shaded region contains vectors that are counterclockwise from $p$.* If $p_1 \times p_2$ is positive, then $p_1$ is clockwise from $p_2$ with respect to the origin $(0,0)$; if this cross product is negative, then $p_1$ is counterclockwise from $p_2$. (See Exercise 33.1-1.) Figure 33.1(b) shows the clockwise and counterclockwise regions relative to a vector $p$. A boundary condition arises if the cross product is $0$; in this case, the vectors are **colinear**, pointing in either the same or opposite directions. To determine whether a directed segment $\overrightarrow{p_0 p_1}$ is closer to a directed segment $\overrightarrow{p_0 p_2}$ in a clockwise direction or in a counterclockwise direction with respect to their common endpoint $p_0$, we simply translate to use $p_0$ as the origin. That is, we let $p_1 - p_0$ denote the vector $p_1' = (x_1', y_1')$, where $x_1' = x_1 - x_0$ and $y_1' = y_1 - y_0$, and we define $p_2 - p_0$ similarly. We then compute the cross product

(p_1 - p_0) \times (p_2 - p_0) = (x_1 - x_0)(y_2 - y_0) - (x_2 - x_0)(y_1 - y_0) .

\begin{aligned} T’(n) &= T(n) + O(n \lg n) \end{aligned}

an d

T(n) = \begin{cases} 2T(n/2) + O(n) & \text{if } n > 3, \ O(1) & \text{if } n \le 3. \end{cases}

\max\left(\frac{C}{C^}, \frac{C^}{C}\right) \le \rho(n) . \tag{35.1}

c(A) = \sum_{(u,v) \in A} c(u,v) .

In many practical situations, the least costly way to go from a place $u$ to a place $w$ is to go directly, with no intermediate steps. Put another way, cutting out an intermediate stop never increases the cost. We formalize this notion by saying that the cost function $c$ satisfies the **triangle inequality** if, for all vertices $u, v, w \in V$,

c(u,w) \le c(u,v) + c(v,w) .

X = \bigcup_{S \in \mathcal{F}} S .

We say that a subset $S \in \mathcal{F}$ **covers** its elements. The problem is to find a minimum-size subset $\mathcal{C} \subseteq \mathcal{F}$ whose members cover all of $X$:

X = \bigcup_{S \in \mathcal{C}} S . \tag{35.8}

We say that any $\mathcal{C}$ satisfying equation (35.8) **covers** $X$. Figure 35.3 illustrates the set-covering problem. The size of $\mathcal{C}$ is the number of sets it contains, rather than the number of individual elements in these sets, since every subset $\mathcal{C}$ that covers $X$ must contain all $|X|$ individual elements. In Figure 35.3, the minimum set cover has size 3. The set-covering problem abstracts many commonly arising combinatorial problems. As a simple example, suppose that $X$ represents a set of skills that are needed to solve a problem and that we have a given set of people available to work on the problem. We wish to form a committee, containing as few people as possible, such that for every requisite skill in $X$, at least one member of the committee has that skill. In the decision version of the set-covering problem, we ask whether a covering exists with size at most $k$, where $k$ is an additional parameter specified in the problem instance. The decision version of the problem is NP-complete, as Exercise 35.3-2 asks you to show. ### A greedy approximation algorithm The greedy method works by picking, at each stage, the set $S$ that covers the greatest number of remaining elements that are uncovered. ```text GREEDY-SET-COVER(X, F) 1 U = X 2 C = emptyset 3 while U != emptyset 4 select an S in F that maximizes |S ∩ U| 5 U = U - S 6 C = C ∪ {S} 7 return C ``` In the example of Figure 35.3, GREEDY-SET-COVER adds to $\mathcal{C}$, in order, the sets $S_1$, $S_4$, and $S_5$, followed by either $S_3$ or $S_6$. The algorithm works as follows. The set $U$ contains, at each stage, the set of remaining uncovered elements. The set $\mathcal{C}$ contains the cover being constructed. Line 4 is the greedy decision-making step, choosing a subset $S$ that covers as many uncovered elements as possible (breaking ties arbitrarily). After $S$ is selected, line 5 removes its elements from $U$, and line 6 places $S$ into $\mathcal{C}$. When the algorithm terminates, the set $\mathcal{C}$ contains a subfamily of $\mathcal{F}$ that covers $X$. We can easily implement GREEDY-SET-COVER to run in time polynomial in $|X|$ and $|\mathcal{F}|$. Since the number of iterations of the loop on lines 3-6 is bounded from above by $\min(|X|, |\mathcal{F}|)$, and we can implement the loop body to run in time $O(|X||\mathcal{F}|)$, a simple implementation runs in time $O(|X||\mathcal{F}|\min(|X|, |\mathcal{F}|))$. Exercise 35.3-3 asks for a linear-time algorithm. ### Analysis We now show that the greedy algorithm returns a set cover that is not too much larger than an optimal set cover. For convenience, in this chapter we denote the $d$th harmonic number $H_d = \sum_{i=1}^{d} 1/i$ (see Section A.1) by $H(d)$. As a boundary condition, we define $H(0) = 0$. > **Theorem 35.4.** GREEDY-SET-COVER is a polynomial-time $\rho(n)$-approximation algorithm, where > > $$ > \rho(n) = H(\max\{|S| : S \in \mathcal{F}\}) . > $$ > > **Proof.** We have already shown that GREEDY-SET-COVER runs in polynomial time. > > *Step 1: charging scheme.* To show that GREEDY-SET-COVER is a $\rho(n)$-approximation algorithm, we assign a cost of 1 to each set selected by the algorithm, distribute this cost over the elements covered for the first time, and then use these costs to derive the desired relationship between the size of an optimal set cover $\mathcal{C}^*$ and the size of the set cover $\mathcal{C}$ returned by the algorithm. Let $S_i$ denote the $i$th subset selected by GREEDY-SET-COVER; the algorithm incurs a cost of 1 when it adds $S_i$ to $\mathcal{C}$. We spread this cost of selecting $S_i$ evenly among the elements covered for the first time by $S_i$. Let $c_x$ denote the cost allocated to element $x$, for each $x \in X$. Each element is assigned a cost only once, when it is covered for the first time. If $x$ is covered for the first time by $S_i$, then > > $$ > c_x = \frac{1}{|S_i - (S_1 \cup S_2 \cup \cdots \cup S_{i-1})|} . > $$ > > *Step 2: total cost equals the cover size.* Each step of the algorithm assigns 1 unit of cost, and so > > $$ > |\mathcal{C}| = \sum_{x \in X} c_x . > \tag{35.9} > $$ > > *Step 3: compare with the optimal cover.* Each element $x \in X$ is in at least one set in the optimal cover $\mathcal{C}^*$, and so we have > > $$ > \sum_{S \in \mathcal{C}^*} \sum_{x \in S} c_x \ge \sum_{x \in X} c_x . > \tag{35.10} > $$ > > (Every $x$ is counted at least once on the left, and all $c_x$ are positive.) Combining equation (35.9) and inequality (35.10), we have that > > $$ > |\mathcal{C}| \le \sum_{S \in \mathcal{C}^*} \sum_{x \in S} c_x . > \tag{35.11} > $$ > > *Step 4: the key inequality.* The remainder of the proof rests on the following key inequality, which we shall prove shortly. For any set $S$ belonging to the family $\mathcal{F}$, > > $$ > \sum_{x \in S} c_x \le H(|S|) . > \tag{35.12} > $$ > > From inequalities (35.11) and (35.12), it follows that > > $$ > \begin{aligned} > |\mathcal{C}| &\le \sum_{S \in \mathcal{C}^*} H(|S|) && \text{(by (35.11) and (35.12))} \\ > &\le |\mathcal{C}^*| \cdot H(\max\{|S| : S \in \mathcal{F}\}) && \text{(each } |S| \le \max \text{ and } H \text{ is nondecreasing)} , > \end{aligned} > $$ > > thus proving the theorem, provided (35.12) holds. > > *Step 5: proof of inequality (35.12).* Consider any set $S \in \mathcal{F}$ and any $i = 1, 2, \ldots, |\mathcal{C}|$, and let > > $$ > u_i = |S - (S_1 \cup S_2 \cup \cdots \cup S_i)| > $$ > > be the number of elements in $S$ that remain uncovered after the algorithm has selected sets $S_1, S_2, \ldots, S_i$. We define $u_0 = |S|$ to be the number of elements of $S$, which are all initially uncovered. Let $k$ be the least index such that $u_k = 0$, so that every element in $S$ is covered by at least one of the sets $S_1, S_2, \ldots, S_k$ and some element in $S$ is uncovered by $S_1 \cup S_2 \cup \cdots \cup S_{k-1}$. Then $u_{i-1} \ge u_i$, and $u_{i-1} - u_i$ elements of $S$ are covered for the first time by $S_i$, for $i = 1, 2, \ldots, k$. Thus, > > $$ > \sum_{x \in S} c_x = \sum_{i=1}^{k} (u_{i-1} - u_i) \cdot \frac{1}{|S_i - (S_1 \cup S_2 \cup \cdots \cup S_{i-1})|} . > $$ > > Observe that > > $$ > \begin{aligned} > |S_i - (S_1 \cup \cdots \cup S_{i-1})| &\ge |S - (S_1 \cup \cdots \cup S_{i-1})| \\ > &= u_{i-1} , > \end{aligned} > $$ > > because the greedy choice of $S_i$ guarantees that $S$ cannot cover more new elements than $S_i$ does (otherwise, the algorithm would have chosen $S$ instead of $S_i$). Consequently, we obtain > > $$ > \sum_{x \in S} c_x \le \sum_{i=1}^{k} (u_{i-1} - u_i) \cdot \frac{1}{u_{i-1}} . > $$ > > We now bound this quantity as follows: > > $$ > \begin{aligned} > \sum_{x \in S} c_x &\le \sum_{i=1}^{k} (u_{i-1} - u_i) \cdot \frac{1}{u_{i-1}} \\ > &= \sum_{i=1}^{k} \sum_{j = u_i + 1}^{u_{i-1}} \frac{1}{u_{i-1}} && \text{(the inner sum has } u_{i-1} - u_i \text{ equal terms)} \\ > &\le \sum_{i=1}^{k} \sum_{j = u_i + 1}^{u_{i-1}} \frac{1}{j} && \text{(because } j \le u_{i-1}\text{, so } 1/j \ge 1/u_{i-1}) \\ > &= \sum_{i=1}^{k} \left( \sum_{j=1}^{u_{i-1}} \frac{1}{j} - \sum_{j=1}^{u_i} \frac{1}{j} \right) \\ > &= \sum_{i=1}^{k} \bigl( H(u_{i-1}) - H(u_i) \bigr) \\ > &= H(u_0) - H(u_k) && \text{(because the sum telescopes)} \\ > &= H(u_0) - H(0) \\ > &= H(u_0) && \text{(because } H(0) = 0) \\ > &= H(|S|) , > \end{aligned} > $$ > > which completes the proof of inequality (35.12) and hence of the theorem. $\blacksquare$ > **Corollary 35.5.** GREEDY-SET-COVER is a polynomial-time $(\ln|X| + 1)$-approximation algorithm. > > **Proof.** Every set in $\mathcal{F}$ has at most $|X|$ elements, so $H(\max\{|S| : S \in \mathcal{F}\}) \le H(|X|)$. By inequality (A.14), $H(|X|) \le \ln|X| + 1$. Use this together with Theorem 35.4. $\blacksquare$ In some applications, $\max\{|S| : S \in \mathcal{F}\}$ is a small constant, and so the solution returned by GREEDY-SET-COVER is at most a small constant times larger than optimal. One such application occurs when this heuristic finds an approximate vertex cover for a graph whose vertices have degree at most 3. In this case, the solution found by GREEDY-SET-COVER is not more than $H(3) = 1 + \tfrac12 + \tfrac13 = 11/6$ times as large as an optimal solution, a performance guarantee that is slightly better than that of APPROX-VERTEX-COVER. > **Think about it.** The charging trick turns one global quantity (cover size) into per-element prices. Notice how the harmonic number appears: the last few uncovered elements of a set can each be "expensive", but only in a $1, \tfrac12, \tfrac13, \ldots$ pattern. ### Exercises 35.3-1 to 35.3-5 **35.3-1** Consider each of the following words as a set of letters: $\{\text{arid}, \text{dash}, \text{drain}, \text{heard}, \text{lost}, \text{nose}, \text{shun}, \text{slate}, \text{snare}, \text{thread}\}$. Show which set cover GREEDY-SET-COVER produces when we break ties in favor of the word that appears first in the dictionary. **35.3-2** Show that the decision version of the set-covering problem is NP-complete by reducing it from the vertex-cover problem. **35.3-3** Show how to implement GREEDY-SET-COVER in such a way that it runs in time $O\!\left(\sum_{S \in \mathcal{F}} |S|\right)$. **35.3-4** Show that the following weaker form of Theorem 35.4 is trivially true:

|\mathcal{C}| \le |\mathcal{C}^*| \max{|S| : S \in \mathcal{F}} .

**35.3-5** GREEDY-SET-COVER can return a number of different solutions, depending on how we break ties in line 4. Give a procedure BAD-SET-COVER-INSTANCE($n$) that returns an $n$-element instance of the set-covering problem for which, depending on how we break ties in line 4, GREEDY-SET-COVER can return a number of different solutions that is exponential in $n$. ## 35.4 Randomization and linear programming In this section, we study two useful techniques for designing approximation algorithms: randomization and linear programming. We shall give a simple randomized algorithm for an optimization version of 3-CNF satisfiability, and then we shall use linear programming to help design an approximation algorithm for a weighted version of the vertex-cover problem. This section only scratches the surface of these two powerful techniques. The chapter notes give references for further study of these areas. ### A randomized approximation algorithm for MAX-3-CNF satisfiability Just as some randomized algorithms compute exact solutions, some randomized algorithms compute approximate solutions. We say that a randomized algorithm for a problem has an **approximation ratio** of $\rho(n)$ if, for any input of size $n$, the *expected* cost $C$ of the solution produced by the randomized algorithm is within a factor of $\rho(n)$ of the cost $C^*$ of an optimal solution:

\max!\left(\frac{C}{C^}, \frac{C^}{C}\right) \le \rho(n) . \tag{35.13}

We call a randomized algorithm that achieves an approximation ratio of $\rho(n)$ a **randomized $\rho(n)$-approximation algorithm**. In other words, a randomized approximation algorithm is like a deterministic approximation algorithm, except that the approximation ratio is for an expected cost. A particular instance of 3-CNF satisfiability, as defined in Section 34.4, may or may not be satisfiable. In order to be satisfiable, there must exist an assignment of the variables so that every clause evaluates to 1. If an instance is not satisfiable, we may want to compute how "close" to satisfiable it is, that is, we may wish to find an assignment of the variables that satisfies as many clauses as possible. We call the resulting maximization problem **MAX-3-CNF satisfiability**. The input to MAX-3-CNF satisfiability is the same as for 3-CNF satisfiability, and the goal is to return an assignment of the variables that maximizes the number of clauses evaluating to 1. We now show that randomly setting each variable to 1 with probability $1/2$ and to 0 with probability $1/2$ yields a randomized $8/7$-approximation algorithm. According to the definition of 3-CNF satisfiability from Section 34.4, we require each clause to consist of exactly three distinct literals. We further assume that no clause contains both a variable and its negation. (Exercise 35.4-1 asks you to remove this last assumption.) > **Theorem 35.6.** Given an instance of MAX-3-CNF satisfiability with $n$ variables $x_1, x_2, \ldots, x_n$ and $m$ clauses, the randomized algorithm that independently sets each variable to 1 with probability $1/2$ and to 0 with probability $1/2$ is a randomized $8/7$-approximation algorithm. > > **Proof.** Suppose that we have independently set each variable to 1 with probability $1/2$ and to 0 with probability $1/2$. For $i = 1, 2, \ldots, m$, we define the indicator random variable > > $$ > Y_i = I\{\text{clause } i \text{ is satisfied}\} , > $$ > > so that $Y_i = 1$ as long as we have set at least one of the literals in the $i$th clause to 1. > > *Step 1: probability that one clause is satisfied.* Since no literal appears more than once in the same clause, and since we have assumed that no variable and its negation appear in the same clause, the settings of the three literals in each clause are independent. A clause is not satisfied only if all three of its literals are set to 0, and so > > $$ > \Pr\{\text{clause } i \text{ is not satisfied}\} = \left(\frac12\right)^3 = \frac18 . > $$ > > Thus, we have > > $$ > \Pr\{\text{clause } i \text{ is satisfied}\} = 1 - \frac18 = \frac78 , > $$ > > and by Lemma 5.1, we have $\mathrm{E}[Y_i] = 7/8$. > > *Step 2: expected number of satisfied clauses.* Let $Y$ be the number of satisfied clauses overall, so that $Y = Y_1 + Y_2 + \cdots + Y_m$. Then, we have > > $$ > \begin{aligned} > \mathrm{E}[Y] &= \mathrm{E}\!\left[\sum_{i=1}^{m} Y_i\right] \\ > &= \sum_{i=1}^{m} \mathrm{E}[Y_i] && \text{(by linearity of expectation)} \\ > &= \sum_{i=1}^{m} \frac78 \\ > &= \frac{7m}{8} . > \end{aligned} > $$ > > *Step 3: the ratio.* Clearly, $m$ is an upper bound on the number of satisfied clauses, and hence the approximation ratio is at most > > $$ > \frac{m}{7m/8} = \frac87 . \qquad \blacksquare > $$ ### Approximating weighted vertex cover using linear programming In the **minimum-weight vertex-cover problem**, we are given an undirected graph $G = (V, E)$ in which each vertex $v \in V$ has an associated positive weight $w(v)$. For any vertex cover $V' \subseteq V$, we define the weight of the vertex cover $w(V') = \sum_{v \in V'} w(v)$. The goal is to find a vertex cover of minimum weight. We cannot apply the algorithm used for unweighted vertex cover, nor can we use a random solution; both methods may return solutions that are far from optimal. We shall, however, compute a lower bound on the weight of the minimum-weight vertex cover, by using a linear program. We shall then "round" this solution and use it to obtain a vertex cover. Suppose that we associate a variable $x(v)$ with each vertex $v \in V$, and let us require that $x(v)$ equals either 0 or 1 for each $v \in V$. We put $v$ into the vertex cover if and only if $x(v) = 1$. Then, we can write the constraint that for any edge $(u,v)$, at least one of $u$ and $v$ must be in the vertex cover as $x(u) + x(v) \ge 1$. This view gives rise to the following 0-1 integer program for finding a minimum-weight vertex cover:

\text{minimize} \quad \sum_{v \in V} w(v), x(v) \tag{35.14}

s u bj ec tt o

\begin{aligned} x(u) + x(v) &\ge 1 && \text{for each } (u,v) \in E \ \end{aligned} \tag{35.15}

x(v) \in {0, 1} \quad \text{for each } v \in V . \tag{35.16}

In the special case in which all the weights $w(v)$ are equal to 1, this formulation is the optimization version of the NP-hard vertex-cover problem. Suppose, however, that we remove the constraint that $x(v) \in \{0,1\}$ and replace it by $0 \le x(v) \le 1$. We then obtain the following linear program, which is known as the **linear-programming relaxation**:

\text{minimize} \quad \sum_{v \in V} w(v), x(v) \tag{35.17}

s u bj ec tt o

x(u) + x(v) \ge 1 \quad \text{for each } (u,v) \in E \tag{35.18}

x(v) \le 1 \quad \text{for each } v \in V \tag{35.19}

x(v) \ge 0 \quad \text{for each } v \in V . \tag{35.20}

Any feasible solution to the 0-1 integer program in lines (35.14)-(35.16) is also a feasible solution to the linear program in lines (35.17)-(35.20). Therefore, the value of an optimal solution to the linear program gives a lower bound on the value of an optimal solution to the 0-1 integer program, and hence a lower bound on the optimal weight in the minimum-weight vertex-cover problem. The following procedure uses the solution to the linear-programming relaxation to construct an approximate solution to the minimum-weight vertex-cover problem: ```text APPROX-MIN-WEIGHT-VC(G, w) 1 C = emptyset 2 compute x-bar, an optimal solution to the linear program in lines (35.17)-(35.20) 3 for each v in V 4 if x-bar(v) >= 1/2 5 C = C ∪ {v} 6 return C ``` The APPROX-MIN-WEIGHT-VC procedure works as follows. Line 1 initializes the vertex cover to be empty. Line 2 formulates the linear program in lines (35.17)-(35.20) and then solves this linear program. An optimal solution gives each vertex $v$ an associated value $\bar{x}(v)$, where $0 \le \bar{x}(v) \le 1$. We use this value to guide the choice of which vertices to add to the vertex cover $C$ in lines 3-5. If $\bar{x}(v) \ge 1/2$, we add $v$ to $C$; otherwise we do not. In effect, we are "rounding" each fractional variable in the solution to the linear program to 0 or 1 in order to obtain a solution to the 0-1 integer program in lines (35.14)-(35.16). Finally, line 6 returns the vertex cover $C$. > **Theorem 35.7.** Algorithm APPROX-MIN-WEIGHT-VC is a polynomial-time 2-approximation algorithm for the minimum-weight vertex-cover problem. > > **Proof.** Because there is a polynomial-time algorithm to solve the linear program in line 2, and because the **for** loop of lines 3-5 runs in polynomial time, APPROX-MIN-WEIGHT-VC is a polynomial-time algorithm. > > Now we show that APPROX-MIN-WEIGHT-VC is a 2-approximation algorithm. Let $C^*$ be an optimal solution to the minimum-weight vertex-cover problem, and let $z^*$ be the value of an optimal solution to the linear program in lines (35.17)-(35.20). > > *Step 1: lower bound.* Since an optimal vertex cover is a feasible solution to the linear program, $z^*$ must be a lower bound on $w(C^*)$, that is, > > $$ > z^* \le w(C^*) . > \tag{35.21} > $$ > > *Step 2: $C$ is a vertex cover.* Next, we claim that by rounding the fractional values of the variables $\bar{x}(v)$, we produce a set $C$ that is a vertex cover and satisfies $w(C) \le 2z^*$. To see that $C$ is a vertex cover, consider any edge $(u,v) \in E$. By constraint (35.18), we know that $\bar{x}(u) + \bar{x}(v) \ge 1$, which implies that at least one of $\bar{x}(u)$ and $\bar{x}(v)$ is at least $1/2$. Therefore, at least one of $u$ and $v$ is included in the vertex cover, and so every edge is covered. > > *Step 3: weight of $C$.* Now, we consider the weight of the cover. We have > > $$ > \begin{aligned} > z^* &= \sum_{v \in V} w(v)\, \bar{x}(v) && \text{(optimal value of the LP)} \\ > &\ge \sum_{v \in V : \bar{x}(v) \ge 1/2} w(v)\, \bar{x}(v) && \text{(dropping nonnegative terms)} \\ > &\ge \sum_{v \in V : \bar{x}(v) \ge 1/2} w(v) \cdot \frac12 && \text{(since } \bar{x}(v) \ge 1/2 \text{ in each term)} \\ > &= \sum_{v \in C} w(v) \cdot \frac12 && \text{(by the definition of } C) \\ > &= \frac12 \sum_{v \in C} w(v) \\ > &= \frac12\, w(C) . > \end{aligned} > \tag{35.22} > $$ > > *Step 4: combine.* Combining inequalities (35.21) and (35.22) gives > > $$ > w(C) \le 2z^* \le 2w(C^*) , > $$ > > and hence APPROX-MIN-WEIGHT-VC is a 2-approximation algorithm. $\blacksquare$ ### Exercises 35.4-1 to 35.4-4 **35.4-1** Show that even if we allow a clause to contain both a variable and its negation, randomly setting each variable to 1 with probability $1/2$ and to 0 with probability $1/2$ still yields a randomized $8/7$-approximation algorithm. **35.4-2** The *MAX-CNF satisfiability problem* is like the MAX-3-CNF satisfiability problem, except that it does not restrict each clause to have exactly 3 literals. Give a randomized 2-approximation algorithm for the MAX-CNF satisfiability problem. **35.4-3** In the *MAX-CUT problem*, we are given an unweighted undirected graph $G = (V, E)$. We define a cut $(S, V - S)$ as in Chapter 23 and the *weight* of a cut as the number of edges crossing the cut. The goal is to find a cut of maximum weight. Suppose that for each vertex $v$, we randomly and independently place $v$ in $S$ with probability $1/2$ and in $V - S$ with probability $1/2$. Show that this algorithm is a randomized 2-approximation algorithm. **35.4-4** Show that the constraints in line (35.19) are redundant in the sense that if we remove them from the linear program in lines (35.17)-(35.20), any optimal solution to the resulting linear program must satisfy $x(v) \le 1$ for each $v \in V$. ## 35.5 The subset-sum problem An instance of the subset-sum problem is a pair $(S, t)$, where $S$ is a set $\{x_1, x_2, \ldots, x_n\}$ of positive integers and $t$ is a positive integer. This decision problem asks whether there exists a subset of $S$ that adds up exactly to the target value $t$. It is NP-complete (see Section 34.5.5). The optimization problem associated with this decision problem arises in practical applications. In the optimization problem, we wish to find a subset of $\{x_1, x_2, \ldots, x_n\}$ whose sum is as large as possible but not larger than $t$. For example, we may have a truck that can carry no more than $t$ pounds, and $n$ different boxes to ship, the $i$th of which weighs $x_i$ pounds. We wish to fill the truck with as heavy a load as possible without exceeding the given weight limit. In this section, we present an exponential-time algorithm for this optimization problem and then show how to modify the algorithm so that it becomes a fully polynomial-time approximation scheme. (Recall that a fully polynomial-time approximation scheme has a running time that is polynomial in $1/\varepsilon$ as well as in the size of the input.) ### An exponential-time exact algorithm Suppose that we computed, for each subset $S'$ of $S$, the sum of the elements in $S'$, and then selected, among the subsets whose sum does not exceed $t$, the one whose sum was closest to $t$. Clearly, this algorithm would return the optimal solution, but it could take exponential time. To implement this algorithm, we could use an iterative procedure that, in iteration $i$, computes the sums of all subsets of $\{x_1, x_2, \ldots, x_i\}$, using as a starting point the sums of all subsets of $\{x_1, x_2, \ldots, x_{i-1}\}$. In doing so, we would realize that once a particular subset $S'$ had a sum exceeding $t$, there would be no reason to maintain it, since no superset of $S'$ could be the optimal solution. We now give an implementation of this strategy. The procedure EXACT-SUBSET-SUM takes an input set $S = \{x_1, x_2, \ldots, x_n\}$ and a target value $t$; we'll see its pseudocode in a moment. This procedure iteratively computes $L_i$, the list of sums of all subsets of $\{x_1, \ldots, x_i\}$ that do not exceed $t$, and then it returns the maximum value in the list $L_n$. If $L$ is a list of positive integers and $x$ is another positive integer, then we let $L + x$ denote the list of integers derived from $L$ by increasing each element of $L$ by $x$. For example, if $L = \langle 1, 2, 3, 5, 9 \rangle$, then $L + 2 = \langle 3, 4, 5, 7, 11 \rangle$. We also use this notation for sets, so that

S + x = {s + x : s \in S} .

We use an auxiliary procedure MERGE-LISTS$(L, L')$, which returns the sorted list that is the merge of its two sorted input lists $L$ and $L'$, with duplicate values removed. Like the MERGE procedure we used in merge sort in Section 2.3.1, MERGE-LISTS runs in time $O(|L| + |L'|)$. (We omit the pseudocode for MERGE-LISTS.) ```text EXACT-SUBSET-SUM(S, t) 1 n = |S| 2 L_0 = <0> 3 for i = 1 to n 4 L_i = MERGE-LISTS(L_{i-1}, L_{i-1} + x_i) 5 remove from L_i every element that is greater than t 6 return the largest element in L_n ``` To see how EXACT-SUBSET-SUM works, let $P_i$ denote the set of all values obtained by selecting a (possibly empty) subset of $\{x_1, x_2, \ldots, x_i\}$ and summing its members. For example, if $S = \{1, 4, 5\}$, then

\begin{aligned} P_1 &= {0, 1} , \ P_2 &= {0, 1, 4, 5} , \ P_3 &= {0, 1, 4, 5, 6, 9, 10} . \end{aligned}

G i v e n t h e i d e n t i t y

P_i = P_{i-1} \cup (P_{i-1} + x_i) , \tag{35.23}

we can prove by induction on $i$ (see Exercise 35.5-1) that the list $L_i$ is a sorted list containing every element of $P_i$ whose value is not more than $t$. Since the length of $L_i$ can be as much as $2^i$, EXACT-SUBSET-SUM is an exponential-time algorithm in general, although it is a polynomial-time algorithm in the special cases in which $t$ is polynomial in $|S|$ or all the numbers in $S$ are bounded by a polynomial in $|S|$. > **Think about it.** Identity (35.23) says: a subset of the first $i$ items either skips $x_i$ or uses it. Why does merging with duplicates removed not save us in the worst case? (Hint: take $x_i = 2^{i}$, so all sums are distinct.) ### A fully polynomial-time approximation scheme We can derive a fully polynomial-time approximation scheme for the subset-sum problem by "trimming" each list $L_i$ after it is created. The idea behind trimming is that if two values in $L$ are close to each other, then since we want just an approximate solution, we do not need to maintain both of them explicitly. More precisely, we use a trimming parameter $\delta$ such that $0 < \delta < 1$. When we trim a list $L$ by $\delta$, we remove as many elements from $L$ as possible, in such a way that if $L'$ is the result of trimming $L$, then for every element $y$ that was removed from $L$, there is an element $z$ still in $L'$ that approximates $y$, that is,

\frac{y}{1 + \delta} \le z \le y . \tag{35.24}

We can think of such a $z$ as "representing" $y$ in the new list $L'$. Each removed element $y$ is represented by a remaining element $z$ satisfying inequality (35.24). For example, if $\delta = 0.1$ and

L = \langle 10, 11, 12, 15, 20, 21, 22, 23, 24, 29 \rangle ,

then we can trim $L$ to obtain

L’ = \langle 10, 12, 15, 20, 23, 29 \rangle ,

where the deleted value 11 is represented by 10, the deleted values 21 and 22 are represented by 20, and the deleted value 24 is represented by 23. Because every element of the trimmed version of the list is also an element of the original version of the list, trimming can dramatically decrease the number of elements kept while keeping a close (and slightly smaller) representative value in the list for each deleted element. The following procedure trims list $L = \langle y_1, y_2, \ldots, y_m \rangle$ in time $\Theta(m)$, given $L$ and $\delta$, and assuming that $L$ is sorted into monotonically increasing order. The output of the procedure is a trimmed, sorted list. ```text TRIM(L, delta) 1 let m be the length of L 2 L' = <y_1> 3 last = y_1 4 for i = 2 to m 5 if y_i > last * (1 + delta) // y_i >= last because L is sorted 6 append y_i onto the end of L' 7 last = y_i 8 return L' ``` The procedure scans the elements of $L$ in monotonically increasing order. A number is appended onto the returned list $L'$ only if it is the first element of $L$ or if it cannot be represented by the most recent number placed into $L'$. Given the procedure TRIM, we can construct our approximation scheme as follows. This procedure takes as input a set $S = \{x_1, x_2, \ldots, x_n\}$ of $n$ integers (in arbitrary order), a target integer $t$, and an "approximation parameter" $\epsilon$, where

0 < \epsilon < 1 . \tag{35.25}

It returns a value $z^*$ whose value is within a $1 + \epsilon$ factor of the optimal solution. ```text APPROX-SUBSET-SUM(S, t, epsilon) 1 n = |S| 2 L_0 = <0> 3 for i = 1 to n 4 L_i = MERGE-LISTS(L_{i-1}, L_{i-1} + x_i) 5 L_i = TRIM(L_i, epsilon/(2n)) 6 remove from L_i every element that is greater than t 7 let z* be the largest value in L_n 8 return z* ``` Line 2 initializes the list $L_0$ to be the list containing just the element 0. The **for** loop in lines 3-6 computes $L_i$ as a sorted list containing a suitably trimmed version of the set $P_i$, with all elements larger than $t$ removed. Since we create $L_i$ from $L_{i-1}$, we must ensure that the repeated trimming does not introduce too much compounded inaccuracy. In a moment, we shall see that APPROX-SUBSET-SUM returns a correct approximation if one exists. As an example, suppose we have the instance

S = \langle 104, 102, 201, 101 \rangle

with $t = 308$ and $\epsilon = 0.40$. The trimming parameter $\delta$ is $\epsilon/(2n) = 0.40/8 = 0.05$. APPROX-SUBSET-SUM computes the following values on the indicated lines: ```text line 2: L_0 = <0>, line 4: L_1 = <0, 104>, line 5: L_1 = <0, 104>, line 6: L_1 = <0, 104>, line 4: L_2 = <0, 102, 104, 206>, line 5: L_2 = <0, 102, 206>, line 6: L_2 = <0, 102, 206>, line 4: L_3 = <0, 102, 201, 206, 303, 407>, line 5: L_3 = <0, 102, 201, 303, 407>, line 6: L_3 = <0, 102, 201, 303>, line 4: L_4 = <0, 101, 102, 201, 203, 302, 303, 404>, line 5: L_4 = <0, 101, 201, 302, 404>, line 6: L_4 = <0, 101, 201, 302>. ``` The algorithm returns $z^* = 302$ as its answer, which is well within $\epsilon = 40\%$ of the optimal answer $307 = 104 + 102 + 101$; in fact, it is within 2%. > **Theorem 35.8.** APPROX-SUBSET-SUM is a fully polynomial-time approximation scheme for the subset-sum problem. > > **Proof.** The operations of trimming $L_i$ in line 5 and removing from $L_i$ every element that is greater than $t$ maintain the property that every element of $L_i$ is also a member of $P_i$. Therefore, the value $z^*$ returned in line 8 is indeed the sum of some subset of $S$. Let $y^* \in P_n$ denote an optimal solution to the subset-sum problem. Then, from line 6, we know that $z^* \le y^*$. By inequality (35.1), we need to show that $y^*/z^* \le 1 + \epsilon$. We must also show that the running time of this algorithm is polynomial in both $1/\epsilon$ and the size of the input. > > *Step 1: invariant on the lists.* As Exercise 35.5-2 asks you to show, for every element $y$ in $P_i$ that is at most $t$, there exists an element $z \in L_i$ such that > > $$ > \frac{y}{(1 + \epsilon/2n)^i} \le z \le y . > \tag{35.26} > $$ > > *Step 2: apply it to the optimum.* Inequality (35.26) must hold for $y^* \in P_n$, and therefore there exists an element $z \in L_n$ such that > > $$ > \frac{y^*}{(1 + \epsilon/2n)^n} \le z \le y^* , > $$ > > and thus > > $$ > \frac{y^*}{z} \le \left(1 + \frac{\epsilon}{2n}\right)^n . > \tag{35.27} > $$ > > Since there exists an element $z \in L_n$ fulfilling inequality (35.27), the inequality must hold for $z^*$, which is the largest value in $L_n$ (so $z^* \ge z$); that is, > > $$ > \frac{y^*}{z^*} \le \left(1 + \frac{\epsilon}{2n}\right)^n . > \tag{35.28} > $$ > > *Step 3: bound the factor by $1 + \epsilon$.* Now, we show that $y^*/z^* \le 1 + \epsilon$. We do so by showing that $(1 + \epsilon/2n)^n \le 1 + \epsilon$. By equation (3.14), we have $\lim_{n \to \infty} (1 + \epsilon/2n)^n = e^{\epsilon/2}$. Exercise 35.5-3 asks you to show that > > $$ > \frac{d}{dn} \left(1 + \frac{\epsilon}{2n}\right)^n > 0 . > \tag{35.29} > $$ > > Therefore, the function $(1 + \epsilon/2n)^n$ increases with $n$ as it approaches its limit of $e^{\epsilon/2}$, and we have > > $$ > \begin{aligned} > \left(1 + \frac{\epsilon}{2n}\right)^n &\le e^{\epsilon/2} && \text{(increasing sequence bounded by its limit)} \\ > &\le 1 + \frac{\epsilon}{2} + \left(\frac{\epsilon}{2}\right)^2 && \text{(by inequality (3.13), valid since } \epsilon/2 \le 1) \\ > &\le 1 + \epsilon && \text{(by inequality (35.25): } (\epsilon/2)^2 \le \epsilon/2) . > \end{aligned} > \tag{35.30} > $$ > > Combining inequalities (35.28) and (35.30) completes the analysis of the approximation ratio. > > *Step 4: length of the lists.* To show that APPROX-SUBSET-SUM is a fully polynomial-time approximation scheme, we derive a bound on the length of $L_i$. After trimming, successive elements $z$ and $z'$ of $L_i$ must have the relationship $z'/z > 1 + \epsilon/2n$. That is, they must differ by a factor of at least $1 + \epsilon/2n$. Each list, therefore, contains the value 0, possibly the value 1, and up to $\lfloor \log_{1+\epsilon/2n} t \rfloor$ additional values. The number of elements in each list $L_i$ is at most > > $$ > \begin{aligned} > \log_{1+\epsilon/2n} t + 2 &= \frac{\ln t}{\ln(1 + \epsilon/2n)} + 2 && \text{(change of base)} \\ > &\le \frac{2n(1 + \epsilon/2n) \ln t}{\epsilon} + 2 && \text{(by inequality (3.17): } \ln(1+x) \ge \tfrac{x}{1+x}) \\ > &< \frac{3n \ln t}{\epsilon} + 2 && \text{(by inequality (35.25): } 2n + \epsilon < 3n) . > \end{aligned} > $$ > > This bound is polynomial in the size of the input, which is the number of bits $\lg t$ needed to represent $t$ plus the number of bits needed to represent the set $S$, which is in turn polynomial in $n$, and in $1/\epsilon$. Since the running time of APPROX-SUBSET-SUM is polynomial in the lengths of the $L_i$, we conclude that APPROX-SUBSET-SUM is a fully polynomial-time approximation scheme. $\blacksquare$ ### Exercises 35.5-1 to 35.5-5 **35.5-1** Prove equation (35.23). Then show that after executing line 5 of EXACT-SUBSET-SUM, $L_i$ is a sorted list containing every element of $P_i$ whose value is not more than $t$. **35.5-2** Using induction on $i$, prove inequality (35.26). **35.5-3** Prove inequality (35.29). **35.5-4** How would you modify the approximation scheme presented in this section to find a good approximation to the smallest value not less than $t$ that is a sum of some subset of the given input list? **35.5-5** Modify the APPROX-SUBSET-SUM procedure to also return the subset of $S$ that sums to the value $z^*$. ## Problems **35-1 Bin packing** Suppose that we are given a set of $n$ objects, where the size $s_i$ of the $i$th object satisfies $0 < s_i < 1$. We wish to pack all the objects into the minimum number of unit-size bins. Each bin can hold any subset of the objects whose total size does not exceed 1. **a.** Prove that the problem of determining the minimum number of bins required is NP-hard. (*Hint:* Reduce from the subset-sum problem.) The *first-fit* heuristic takes each object in turn and places it into the first bin that can accommodate it. Let $S = \sum_{i=1}^{n} s_i$. **b.** Argue that the optimal number of bins required is at least $\lceil S \rceil$. **c.** Argue that the first-fit heuristic leaves at most one bin less than half full. **d.** Prove that the number of bins used by the first-fit heuristic is never more than $\lceil 2S \rceil$. **e.** Prove an approximation ratio of 2 for the first-fit heuristic. **f.** Give an efficient implementation of the first-fit heuristic, and analyze its running time. **35-2 Approximating the size of a maximum clique** Let $G = (V, E)$ be an undirected graph. For any $k \ge 1$, define $G^{(k)}$ to be the undirected graph $(V^{(k)}, E^{(k)})$, where $V^{(k)}$ is the set of all ordered $k$-tuples of vertices from $V$ and $E^{(k)}$ is defined so that $(v_1, v_2, \ldots, v_k)$ is adjacent to $(w_1, w_2, \ldots, w_k)$ if and only if for $i = 1, 2, \ldots, k$, either vertex $v_i$ is adjacent to $w_i$ in $G$, or else $v_i = w_i$. **a.** Prove that the size of the maximum clique in $G^{(k)}$ is equal to the $k$th power of the size of the maximum clique in $G$. **b.** Argue that if there is an approximation algorithm that has a constant approximation ratio for finding a maximum-size clique, then there is a polynomial-time approximation scheme for the problem. **35-3 Weighted set-covering problem** Suppose that we generalize the set-covering problem so that each set $S_i$ in the family $\mathcal{F}$ has an associated weight $w_i$ and the weight of a cover $\mathcal{C}$ is $\sum_{S_i \in \mathcal{C}} w_i$. We wish to determine a minimum-weight cover. (Section 35.3 handles the case in which $w_i = 1$ for all $i$.) Show how to generalize the greedy set-covering heuristic in a natural manner to provide an approximate solution for any instance of the weighted set-covering problem. Show that your heuristic has an approximation ratio of $H(d)$, where $d$ is the maximum size of any set $S_i$. **35-4 Maximum matching** Recall that for an undirected graph $G$, a matching is a set of edges such that no two edges in the set are incident on the same vertex. In Section 26.3, we saw how to find a maximum matching in a bipartite graph. In this problem, we will look at matchings in undirected graphs in general (i.e., the graphs are not required to be bipartite). **a.** A *maximal matching* is a matching that is not a proper subset of any other matching. Show that a maximal matching need not be a maximum matching by exhibiting an undirected graph $G$ and a maximal matching $M$ in $G$ that is not a maximum matching. (*Hint:* You can find such a graph with only four vertices.) **b.** Consider an undirected graph $G = (V, E)$. Give an $O(E)$-time greedy algorithm to find a maximal matching in $G$. In this problem, we shall concentrate on a polynomial-time approximation algorithm for maximum matching. Whereas the fastest known algorithm for maximum matching takes superlinear (but polynomial) time, the approximation algorithm here will run in linear time. You will show that the linear-time greedy algorithm for maximal matching in part (b) is a 2-approximation algorithm for maximum matching. **c.** Show that the size of a maximum matching in $G$ is a lower bound on the size of any vertex cover for $G$. **d.** Consider a maximal matching $M$ in $G = (V, E)$. Let

T = {v \in V : \text{some edge in } M \text{ is incident on } v} .

What can you say about the subgraph of $G$ induced by the vertices of $G$ that are not in $T$? **e.** Conclude from part (d) that $2|M|$ is the size of a vertex cover for $G$. **f.** Using parts (c) and (e), prove that the greedy algorithm in part (b) is a 2-approximation algorithm for maximum matching. **35-5 Parallel machine scheduling** In the *parallel-machine-scheduling problem*, we are given $n$ jobs, $J_1, J_2, \ldots, J_n$, where each job $J_k$ has an associated nonnegative processing time of $p_k$. We are also given $m$ identical machines, $M_1, M_2, \ldots, M_m$. Any job can run on any machine. A *schedule* specifies, for each job $J_k$, the machine on which it runs and the time period during which it runs. Each job $J_k$ must run on some machine $M_i$ for $p_k$ consecutive time units, and during that time period no other job may run on $M_i$. Let $C_k$ denote the *completion time* of job $J_k$, that is, the time at which job $J_k$ completes processing. Given a schedule, we define $C_{\max} = \max_{1 \le j \le n} C_j$ to be the *makespan* of the schedule. The goal is to find a schedule whose makespan is minimum. For example, suppose that we have two machines $M_1$ and $M_2$ and that we have four jobs $J_1, J_2, J_3, J_4$, with $p_1 = 2$, $p_2 = 12$, $p_3 = 4$, and $p_4 = 5$. Then one possible schedule runs, on machine $M_1$, job $J_1$ followed by job $J_2$, and on machine $M_2$, it runs job $J_4$ followed by job $J_3$. For this schedule, $C_1 = 2$, $C_2 = 14$, $C_3 = 9$, $C_4 = 5$, and $C_{\max} = 14$. An optimal schedule runs $J_2$ on machine $M_1$, and it runs jobs $J_1$, $J_3$, and $J_4$ on machine $M_2$. For this schedule, $C_1 = 2$, $C_2 = 12$, $C_3 = 6$, $C_4 = 11$, and $C_{\max} = 12$. Given a parallel-machine-scheduling problem, we let $C^*_{\max}$ denote the makespan of an optimal schedule. **a.** Show that the optimal makespan is at least as large as the greatest processing time, that is,

C^*{\max} \ge \max{1 \le k \le n} p_k .

* * b . * * S h o w t ha tt h eo pt ima l mak es p ani s a tl e a s t a s l a r g e a s t h e a v er a g e ma c hin e l o a d, t ha t i s,

C^*{\max} \ge \frac{1}{m} \sum{1 \le k \le n} p_k .

S u pp ose t ha tw e u se t h e f o l l o w in g g r ee d y a l g or i t hm f or p a r a l l e l ma c hin esc h e d u l in g : w h e n e v er ama c hin e i s i d l e, sc h e d u l e an y j o b t ha t ha s n o t y e t b ee n sc h e d u l e d . * * c . * * W r i t e p se u d oco d e t o im pl e m e n tt hi s g r ee d y a l g or i t hm . W ha t i s t h er u nnin g t im eo f y o u r a l g or i t hm ? * * d . * * F or t h esc h e d u l er e t u r n e d b y t h e g r ee d y a l g or i t hm, s h o w t ha t

C_{\max} \le \frac{1}{m} \sum_{1 \le k \le n} p_k + \max_{1 \le k \le n} p_k .

A = \begin{pmatrix} a_{11} & a_{12} & a_{13} \ a_{21} & a_{22} & a_{23} \end{pmatrix} = \begin{pmatrix} 1 & 2 & 3 \ 4 & 5 & 6 \end{pmatrix} \tag{D.1}

is a $2 \times 3$ matrix $A = (a_{ij})$, where for $i = 1, 2$ and $j = 1, 2, 3$, we denote the element of the matrix in row $i$ and column $j$ by $a_{ij}$. We use uppercase letters to denote matrices and corresponding subscripted lowercase letters to denote their elements. We denote the set of all $m \times n$ matrices with real-valued entries by $\mathbb{R}^{m \times n}$ and, in general, the set of $m \times n$ matrices with entries drawn from a set $S$ by $S^{m \times n}$. The **transpose** of a matrix $A$ is the matrix $A^{\mathrm T}$ obtained by exchanging the rows and columns of $A$. For the matrix $A$ of equation (D.1),

A^{\mathrm T} = \begin{pmatrix} 1 & 4 \ 2 & 5 \ 3 & 6 \end{pmatrix}.

A * * v ec t or * * i s a o n e - d im e n s i o na l a r r a y o f n u mb er s . F or e x am pl e,

x = \begin{pmatrix} 2 \ 3 \ 5 \end{pmatrix}

is a vector of size 3. We sometimes call a vector of length $n$ an **$n$-vector**. We use lowercase letters to denote vectors, and we denote the $i$th element of a size-$n$ vector $x$ by $x_i$, for $i = 1, 2, \ldots, n$. We take the standard form of a vector to be as a **column vector** equivalent to an $n \times 1$ matrix; the corresponding **row vector** is obtained by taking the transpose:

x^{\mathrm T} = \begin{pmatrix} 2 & 3 & 5 \end{pmatrix}.

The **unit vector** $e_i$ is the vector whose $i$th element is 1 and all of whose other elements are 0. Usually, the size of a unit vector is clear from the context. A **zero matrix** is a matrix all of whose entries are 0. Such a matrix is often denoted $0$, since the ambiguity between the number 0 and a matrix of 0s is usually easily resolved from context. If a matrix of 0s is intended, then the size of the matrix also needs to be derived from the context. ### Square matrices Square $n \times n$ matrices arise frequently. Several special cases of square matrices are of particular interest: 1. A **diagonal matrix** has $a_{ij} = 0$ whenever $i \neq j$. Because all of the off-diagonal elements are zero, we can specify the matrix by listing the elements along the diagonal:

\operatorname{diag}(a_{11}, a_{22}, \ldots, a_{nn}) = \begin{pmatrix} a_{11} & 0 & \cdots & 0 \ 0 & a_{22} & \cdots & 0 \ \vdots & \vdots & \ddots & \vdots \ 0 & 0 & \cdots & a_{nn} \end{pmatrix}.

2. The $n \times n$ **identity matrix** $I_n$ is a diagonal matrix with 1s along the diagonal:

I_n = \operatorname{diag}(1, 1, \ldots, 1) = \begin{pmatrix} 1 & 0 & \cdots & 0 \ 0 & 1 & \cdots & 0 \ \vdots & \vdots & \ddots & \vdots \ 0 & 0 & \cdots & 1 \end{pmatrix}.

When $I$ appears without a subscript, we derive its size from the context. The $i$th column of an identity matrix is the unit vector $e_i$. 3. A **tridiagonal matrix** $T$ is one for which $t_{ij} = 0$ if $|i - j| > 1$. Nonzero entries appear only on the main diagonal, immediately above the main diagonal ($t_{i,i+1}$ for $i = 1, 2, \ldots, n-1$), or immediately below the main diagonal ($t_{i+1,i}$ for $i = 1, 2, \ldots, n-1$):

T = \begin{pmatrix} t_{11} & t_{12} & 0 & 0 & \cdots & 0 & 0 & 0 \ t_{21} & t_{22} & t_{23} & 0 & \cdots & 0 & 0 & 0 \ 0 & t_{32} & t_{33} & t_{34} & \cdots & 0 & 0 & 0 \ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots \ 0 & 0 & 0 & 0 & \cdots & t_{n-2,n-2} & t_{n-2,n-1} & 0 \ 0 & 0 & 0 & 0 & \cdots & t_{n-1,n-2} & t_{n-1,n-1} & t_{n-1,n} \ 0 & 0 & 0 & 0 & \cdots & 0 & t_{n,n-1} & t_{nn} \end{pmatrix}.

4. An **upper-triangular matrix** $U$ is one for which $u_{ij} = 0$ if $i > j$. All entries below the diagonal are zero:

U = \begin{pmatrix} u_{11} & u_{12} & \cdots & u_{1n} \ 0 & u_{22} & \cdots & u_{2n} \ \vdots & \vdots & \ddots & \vdots \ 0 & 0 & \cdots & u_{nn} \end{pmatrix}.

An upper-triangular matrix is **unit upper-triangular** if it has all 1s along the diagonal. 5. A **lower-triangular matrix** $L$ is one for which $l_{ij} = 0$ if $i < j$. All entries above the diagonal are zero:

L = \begin{pmatrix} l_{11} & 0 & \cdots & 0 \ l_{21} & l_{22} & \cdots & 0 \ \vdots & \vdots & \ddots & \vdots \ l_{n1} & l_{n2} & \cdots & l_{nn} \end{pmatrix}.

A lower-triangular matrix is **unit lower-triangular** if it has all 1s along the diagonal. 6. A **permutation matrix** $P$ has exactly one 1 in each row and column, and 0s elsewhere. An example of a permutation matrix is

P = \begin{pmatrix} 0 & 1 & 0 & 0 & 0 \ 0 & 0 & 0 & 1 & 0 \ 1 & 0 & 0 & 0 & 0 \ 0 & 0 & 0 & 0 & 1 \ 0 & 0 & 1 & 0 & 0 \end{pmatrix}.

Such a matrix is called a permutation matrix because multiplying a vector $x$ by a permutation matrix has the effect of permuting (rearranging) the elements of $x$. Exercise D.1-4 explores additional properties of permutation matrices. 7. A **symmetric matrix** $A$ satisfies the condition $A = A^{\mathrm T}$. For example,

\begin{pmatrix} 1 & 2 & 3 \ 2 & 6 & 4 \ 3 & 4 & 5 \end{pmatrix}

is a symmetric matrix. ### Basic matrix operations The elements of a matrix or vector are numbers from a number system, such as the real numbers, the complex numbers, or integers modulo a prime. The number system defines how to add and multiply numbers. We can extend these definitions to encompass addition and multiplication of matrices. We define **matrix addition** as follows. If $A = (a_{ij})$ and $B = (b_{ij})$ are $m \times n$ matrices, then their matrix sum $C = (c_{ij}) = A + B$ is the $m \times n$ matrix defined by

c_{ij} = a_{ij} + b_{ij}

for $i = 1, 2, \ldots, m$ and $j = 1, 2, \ldots, n$. That is, matrix addition is performed componentwise. A zero matrix is the identity for matrix addition:

A + 0 = A = 0 + A.

If $\lambda$ is a number and $A = (a_{ij})$ is a matrix, then $\lambda A = (\lambda a_{ij})$ is the **scalar multiple** of $A$ obtained by multiplying each of its elements by $\lambda$. As a special case, we define the **negative** of a matrix $A = (a_{ij})$ to be $-1 \cdot A = -A$, so that the $ij$th entry of $-A$ is $-a_{ij}$. Thus,

A + (-A) = 0 = (-A) + A.

We use the negative of a matrix to define **matrix subtraction**: $A - B = A + (-B)$. We define **matrix multiplication** as follows. We start with two matrices $A$ and $B$ that are **compatible** in the sense that the number of columns of $A$ equals the number of rows of $B$. (In general, an expression containing a matrix product $AB$ is always assumed to imply that matrices $A$ and $B$ are compatible.) If $A = (a_{ik})$ is an $m \times n$ matrix and $B = (b_{kj})$ is an $n \times p$ matrix, then their matrix product $C = AB$ is the $m \times p$ matrix $C = (c_{ij})$, where

c_{ij} = \sum_{k=1}^{n} a_{ik} b_{kj} \tag{D.2}

for $i = 1, 2, \ldots, m$ and $j = 1, 2, \ldots, p$. The procedure SQUARE-MATRIX-MULTIPLY in Section 4.2 implements matrix multiplication in the straightforward manner based on equation (D.2), assuming that the matrices are square: $m = n = p$. To multiply $n \times n$ matrices, SQUARE-MATRIX-MULTIPLY performs $n^3$ multiplications and $n^2(n-1)$ additions, and so its running time is $\Theta(n^3)$. Matrices have many (but not all) of the algebraic properties typical of numbers. Identity matrices are identities for matrix multiplication:

I_m A = A I_n = A

for any $m \times n$ matrix $A$. Multiplying by a zero matrix gives a zero matrix:

A 0 = 0.

M a t r i x m u l t i pl i c a t i o ni s a ssoc ia t i v e :

A(BC) = (AB)C

for compatible matrices $A$, $B$, and $C$. Matrix multiplication distributes over addition:

\begin{aligned} A(B + C) &= AB + AC, \ (B + C)D &= BD + CD. \end{aligned}

For $n > 1$, multiplication of $n \times n$ matrices is not commutative. For example, if $A = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}$ and $B = \begin{pmatrix} 0 & 0 \\ 1 & 0 \end{pmatrix}$, then

AB = \begin{pmatrix} 1 & 0 \ 0 & 0 \end{pmatrix} \quad\text{and}\quad BA = \begin{pmatrix} 0 & 0 \ 0 & 1 \end{pmatrix}.

> **Think about it.** Rotating a book then flipping it is not the same as flipping then rotating. Matrices model such transformations, so why would we expect $AB = BA$? We define matrix-vector products or vector-vector products as if the vector were the equivalent $n \times 1$ matrix (or a $1 \times n$ matrix, in the case of a row vector). Thus, if $A$ is an $m \times n$ matrix and $x$ is an $n$-vector, then $Ax$ is an $m$-vector. If $x$ and $y$ are $n$-vectors, then

x^{\mathrm T} y = \sum_{i=1}^{n} x_i y_i

is a number (actually a $1 \times 1$ matrix) called the **inner product** of $x$ and $y$. The matrix $x y^{\mathrm T}$ is an $n \times n$ matrix $Z$ called the **outer product** of $x$ and $y$, with $z_{ij} = x_i y_j$. The (euclidean) **norm** $\lVert x \rVert$ of an $n$-vector $x$ is defined by

\begin{aligned} \lVert x \rVert &= \left(x_1^2 + x_2^2 + \cdots + x_n^2\right)^{1/2} \ &= \left(x^{\mathrm T} x\right)^{1/2}. \end{aligned}

Thus, the norm of $x$ is its length in $n$-dimensional euclidean space. ### Exercises D.1-1 to D.1-4 **D.1-1** Show that if $A$ and $B$ are symmetric $n \times n$ matrices, then so are $A + B$ and $A - B$. **D.1-2** Prove that $(AB)^{\mathrm T} = B^{\mathrm T} A^{\mathrm T}$ and that $A^{\mathrm T} A$ is always a symmetric matrix. **D.1-3** Prove that the product of two lower-triangular matrices is lower-triangular. **D.1-4** Prove that if $P$ is an $n \times n$ permutation matrix and $A$ is an $n \times n$ matrix, then the matrix product $PA$ is $A$ with its rows permuted, and the matrix product $AP$ is $A$ with its columns permuted. Prove that the product of two permutation matrices is a permutation matrix. ## D.2 Basic matrix properties In this section, we define some basic properties pertaining to matrices: inverses, linear dependence and independence, rank, and determinants. We also define the class of positive-definite matrices. ### Matrix inverses, ranks, and determinants We define the **inverse** of an $n \times n$ matrix $A$ to be the $n \times n$ matrix, denoted $A^{-1}$ (if it exists), such that $A A^{-1} = I_n = A^{-1} A$. For example,

\begin{pmatrix} 1 & 1 \ 1 & 0 \end{pmatrix}^{-1} = \begin{pmatrix} 0 & 1 \ 1 & -1 \end{pmatrix}.

Many nonzero $n \times n$ matrices do not have inverses. A matrix without an inverse is called **noninvertible**, or **singular**. An example of a nonzero singular matrix is

\begin{pmatrix} 1 & 0 \ 1 & 0 \end{pmatrix}.

If a matrix has an inverse, it is called **invertible**, or **nonsingular**. Matrix inverses, when they exist, are unique. (See Exercise D.2-1.) If $A$ and $B$ are nonsingular $n \times n$ matrices, then

(BA)^{-1} = A^{-1} B

T h e in v er seo p er a t i o n co mm u t es w i t h t h e t r an s p oseo p er a t i o n :

\left(A^{-1}\right)^{\mathrm T} = \left(A^{\mathrm T}\right)

The vectors $x_1, x_2, \ldots, x_n$ are **linearly dependent** if there exist coefficients $c_1, c_2, \ldots, c_n$, not all of which are zero, such that $c_1 x_1 + c_2 x_2 + \cdots + c_n x_n = 0$. The row vectors $x_1 = (1\ 2\ 3)$, $x_2 = (2\ 6\ 4)$, and $x_3 = (4\ 11\ 9)$ are linearly dependent, for example, since $2x_1 + 3x_2 - 2x_3 = 0$. Check it entry by entry:

\begin{aligned} 2x_1 + 3x_2 - 2x_3 &= (2, 4, 6) + (6, 18, 12) - (8, 22, 18) \ &= (2 + 6 - 8,; 4 + 18 - 22,; 6 + 12 - 18) \ &= (0, 0, 0). \end{aligned}

If vectors are not linearly dependent, they are **linearly independent**. For example, the columns of an identity matrix are linearly independent. The **column rank** of a nonzero $m \times n$ matrix $A$ is the size of the largest set of linearly independent columns of $A$. Similarly, the **row rank** of $A$ is the size of the largest set of linearly independent rows of $A$. A fundamental property of any matrix $A$ is that its row rank always equals its column rank, so that we can simply refer to the **rank** of $A$. The rank of an $m \times n$ matrix is an integer between 0 and $\min(m, n)$, inclusive. (The rank of a zero matrix is 0, and the rank of an $n \times n$ identity matrix is $n$.) An alternate, but equivalent and often more useful, definition is that the rank of a nonzero $m \times n$ matrix $A$ is the smallest number $r$ such that there exist matrices $B$ and $C$ of respective sizes $m \times r$ and $r \times n$ such that

A = BC.

A square $n \times n$ matrix has **full rank** if its rank is $n$. An $m \times n$ matrix has **full column rank** if its rank is $n$. The following theorem gives a fundamental property of ranks. > **Theorem D.1** > A square matrix has full rank if and only if it is nonsingular. A **null vector** for a matrix $A$ is a nonzero vector $x$ such that $Ax = 0$. The following theorem (whose proof is left as Exercise D.2-7) and its corollary relate the notions of column rank and singularity to null vectors. > **Theorem D.2** > A matrix $A$ has full column rank if and only if it does not have a null vector. > **Corollary D.3** > A square matrix $A$ is singular if and only if it has a null vector. The $ij$th **minor** of an $n \times n$ matrix $A$, for $n > 1$, is the $(n-1) \times (n-1)$ matrix $A_{[ij]}$ obtained by deleting the $i$th row and $j$th column of $A$. We define the **determinant** of an $n \times n$ matrix $A$ recursively in terms of its minors by

\det(A) = \begin{cases} a_{11} & \text{if } n = 1, \ \displaystyle\sum_{j=1}^{n} (-1)^{1+j}, a_{1j} \det\left(A_{[1j]}\right) & \text{if } n > 1. \end{cases}

The term $(-1)^{i+j} \det\left(A_{[ij]}\right)$ is known as the **cofactor** of the element $a_{ij}$. The following theorems, whose proofs are omitted here, express fundamental properties of the determinant. > **Theorem D.4 (Determinant properties)** > The determinant of a square matrix $A$ has the following properties: > > - If any row or any column of $A$ is zero, then $\det(A) = 0$. > - The determinant of $A$ is multiplied by $\lambda$ if the entries of any one row (or any one column) of $A$ are all multiplied by $\lambda$. > - The determinant of $A$ is unchanged if the entries in one row (respectively, column) are added to those in another row (respectively, column). > - The determinant of $A$ equals the determinant of $A^{\mathrm T}$. > - The determinant of $A$ is multiplied by $-1$ if any two rows (or any two columns) are exchanged. > > Also, for any square matrices $A$ and $B$, we have $\det(AB) = \det(A)\det(B)$. > **Theorem D.5** > An $n \times n$ matrix $A$ is singular if and only if $\det(A) = 0$. ### Positive-definite matrices Positive-definite matrices play an important role in many applications. An $n \times n$ matrix $A$ is **positive-definite** if $x^{\mathrm T} A x > 0$ for all $n$-vectors $x \neq 0$. For example, the identity matrix is positive-definite, since for any nonzero vector $x = (x_1\ x_2\ \cdots\ x_n)^{\mathrm T}$,

\begin{aligned} x^{\mathrm T} I x &= x^{\mathrm T} x \ &= \sum_{i=1}^{n} x_i^2 \ &> 0. \end{aligned}

The last step holds because $x \neq 0$ means some $x_i \neq 0$, so at least one term $x_i^2$ is strictly positive and none is negative. Matrices that arise in applications are often positive-definite due to the following theorem. > **Theorem D.6** > For any matrix $A$ with full column rank, the matrix $A^{\mathrm T} A$ is positive-definite. > > **Proof.** We must show that $x^{\mathrm T}(A^{\mathrm T} A)x > 0$ for any nonzero vector $x$. For any vector $x$, > > $$ > \begin{aligned} > x^{\mathrm T}\left(A^{\mathrm T} A\right)x &= (Ax)^{\mathrm T}(Ax) \quad \text{(by Exercise D.1-2: } (Ax)^{\mathrm T} = x^{\mathrm T}A^{\mathrm T}\text{)} \\ > &= \lVert Ax \rVert^2 . > \end{aligned} > $$ > > Note that $\lVert Ax \rVert^2$ is just the sum of the squares of the elements of the vector $Ax$. Therefore, $\lVert Ax \rVert^2 \ge 0$. If $\lVert Ax \rVert^2 = 0$, every element of $Ax$ is 0, which is to say $Ax = 0$. Since $A$ has full column rank, $Ax = 0$ implies $x = 0$, by Theorem D.2. Hence, for $x \neq 0$ we cannot have $Ax = 0$, so $\lVert Ax \rVert^2 > 0$, and $A^{\mathrm T} A$ is positive-definite. $\blacksquare$ Section 28.3 explores other properties of positive-definite matrices. ### Exercises D.2-1 to D.2-8 **D.2-1** Prove that matrix inverses are unique, that is, if $B$ and $C$ are inverses of $A$, then $B = C$. **D.2-2** Prove that the determinant of a lower-triangular or upper-triangular matrix is equal to the product of its diagonal elements. Prove that the inverse of a lower-triangular matrix, if it exists, is lower-triangular. **D.2-3** Prove that if $P$ is a permutation matrix, then $P$ is invertible, its inverse is $P^{\mathrm T}$, and $P^{\mathrm T}$ is a permutation matrix. **D.2-4** Let $A$ and $B$ be $n \times n$ matrices such that $AB = I$. Prove that if $A'$ is obtained from $A$ by adding row $j$ into row $i$, then subtracting column $i$ from column $j$ of $B$ yields the inverse $B'$ of $A'$. **D.2-5** Let $A$ be a nonsingular $n \times n$ matrix with complex entries. Show that every entry of $A^{-1}$ is real if and only if every entry of $A$ is real. **D.2-6** Show that if $A$ is a nonsingular, symmetric, $n \times n$ matrix, then $A^{-1}$ is symmetric. Show that if $B$ is an arbitrary $m \times n$ matrix, then the $m \times m$ matrix given by the product $B A B^{\mathrm T}$ is symmetric. **D.2-7** Prove Theorem D.2. That is, show that a matrix $A$ has full column rank if and only if $Ax = 0$ implies $x = 0$. (*Hint:* Express the linear dependence of one column on the others as a matrix-vector equation.) **D.2-8** Prove that for any two compatible matrices $A$ and $B$,

\operatorname{rank}(AB) \le \min(\operatorname{rank}(A), \operatorname{rank}(B)),

where equality holds if either $A$ or $B$ is a nonsingular square matrix. (*Hint:* Use the alternate definition of the rank of a matrix.) ## Problems ### D-1 Vandermonde matrix Given numbers $x_0, x_1, \ldots, x_{n-1}$, prove that the determinant of the **Vandermonde matrix**

V(x_0, x_1, \ldots, x_{n-1}) = \begin{pmatrix} 1 & x_0 & x_0^2 & \cdots & x_0^{n-1} \ 1 & x_1 & x_1^2 & \cdots & x_1^{n-1} \ \vdots & \vdots & \vdots & \ddots & \vdots \ 1 & x_{n-1} & x_{n-1}^2 & \cdots & x_{n-1}^{n-1} \end{pmatrix}

i s

\det\left(V(x_0, x_1, \ldots, x_{n-1})\right) = \prod_{0 \le j < k \le n-1} (x_k - x_j).

(*Hint:* Multiply column $i$ by $-x_0$ and add it to column $i + 1$ for $i = n-2, n-3, \ldots, 1$, and then use induction.) ### D-2 Permutations defined by matrix-vector multiplication over GF(2) One class of permutations of the integers in the set $S_n = \{0, 1, 2, \ldots, 2^n - 1\}$ is defined by matrix multiplication over GF(2). For each integer $x$ in $S_n$, we view its binary representation as an $n$-bit vector

\begin{pmatrix} x_0 \ x_1 \ x_2 \ \vdots \ x_{n-1} \end{pmatrix},

where $x = \sum_{i=0}^{n-1} x_i 2^i$. If $A$ is an $n \times n$ matrix in which each entry is either 0 or 1, then we can define a permutation mapping each value $x$ in $S_n$ to the number whose binary representation is the matrix-vector product $Ax$. Here, we perform all arithmetic over GF(2): all values are either 0 or 1, and with one exception the usual rules of addition and multiplication apply. The exception is that $1 + 1 = 0$. You can think of arithmetic over GF(2) as being just like regular integer arithmetic, except that you use only the least significant bit. As an example, for $S_2 = \{0, 1, 2, 3\}$, the matrix

A = \begin{pmatrix} 1 & 0 \ 1 & 1 \end{pmatrix}

defines the following permutation $\pi_A$: $\pi_A(0) = 0$, $\pi_A(1) = 3$, $\pi_A(2) = 2$, $\pi_A(3) = 1$. To see why $\pi_A(3) = 1$, observe that, working in GF(2),

\pi_A(3) = \begin{pmatrix} 1 & 0 \ 1 & 1 \end{pmatrix}\begin{pmatrix} 1 \ 1 \end{pmatrix} = \begin{pmatrix} 1 \cdot 1 + 0 \cdot 1 \ 1 \cdot 1 + 1 \cdot 1 \end{pmatrix} = \begin{pmatrix} 1 \ 0 \end{pmatrix},

which is the binary representation of 1. For the remainder of this problem, we work over GF(2), and all matrix and vector entries are 0 or 1. We define the **rank** of a 0-1 matrix (a matrix for which each entry is either 0 or 1) over GF(2) the same as for a regular matrix, but with all arithmetic that determines linear independence performed over GF(2). We define the **range** of an $n \times n$ 0-1 matrix $A$ by

R(A) = {, y : y = Ax \text{ for some } x \in S_n ,},

so that $R(A)$ is the set of numbers in $S_n$ that we can produce by multiplying each value $x$ in $S_n$ by $A$. **a.** If $r$ is the rank of matrix $A$, prove that $|R(A)| = 2^r$. Conclude that $A$ defines a permutation on $S_n$ only if $A$ has full rank. For a given $n \times n$ matrix $A$ and a given value $y \in R(A)$, we define the **preimage** of $y$ by

P(A, y) = {, x : Ax = y ,},

so that $P(A, y)$ is the set of values in $S_n$ that map to $y$ when multiplied by $A$. **b.** If $r$ is the rank of $n \times n$ matrix $A$ and $y \in R(A)$, prove that $|P(A, y)| = 2^{n-r}$. Let $0 \le m \le n$, and suppose we partition the set $S_n$ into blocks of consecutive numbers, where the $i$th block consists of the $2^m$ numbers $i 2^m, i 2^m + 1, i 2^m + 2, \ldots, (i+1)2^m - 1$. For any subset $S \subseteq S_n$, define $B(S, m)$ to be the set of size-$2^m$ blocks of $S_n$ containing some element of $S$. As an example, when $n = 3$, $m = 1$, and $S = \{1, 4, 5\}$, then $B(S, m)$ consists of blocks 0 (since 1 is in the 0th block) and 2 (since both 4 and 5 are in block 2). **c.** Let $r$ be the rank of the lower left $(n-m) \times m$ submatrix of $A$, that is, the matrix formed by taking the intersection of the bottom $n - m$ rows and the leftmost $m$ columns of $A$. Let $S$ be any size-$2^m$ block of $S_n$, and let $S' = \{\, y : y = Ax \text{ for some } x \in S \,\}$. Prove that $|B(S', m)| = 2^r$ and that for each block in $B(S', m)$, exactly $2^{m-r}$ numbers in $S$ map to that block. Because multiplying the zero vector by any matrix yields a zero vector, the set of permutations of $S_n$ defined by multiplying by $n \times n$ 0-1 matrices with full rank over GF(2) cannot include all permutations of $S_n$. Let us extend the class of permutations defined by matrix-vector multiplication to include an additive term, so that $x \in S_n$ maps to $Ax + c$, where $c$ is an $n$-bit vector and addition is performed over GF(2). For example, when

A = \begin{pmatrix} 1 & 0 \ 1 & 1 \end{pmatrix}

an d

c = \begin{pmatrix} 0 \ 1 \end{pmatrix},

We investigated several drawing programs that run under Mac OS X, but all had significant shortcomings compared with MacDraw Pro. We briefly attempted to produce the illustrations for this book with a different, well known drawing program. We found that it took at least five times as long to produce each illustration as it took with MacDraw Pro, and the resulting illustrations did not look as good. Hence the decision to revert to MacDraw Pro running on older Macintoshes. ↩

input	after sort on digit 1	after sort on digit 2	after sort on digit 3
329	720	720	329
457	355	329	355
657	436	436	436
839	457	839	457
436	657	355	657
720	329	457	720
355	839	657	839

input	after sort on digit 1	after sort on digit 2	after sort on digit 3
329	720	720	329
457	355	329	355
657	436	436	436
839	457	839	457
436	657	355	657
720	329	457	720
355	839	657	839

Vault

Explorer

Cormen Introduction to Algorithms

Introduction to Algorithms

Contents

Part I: Foundations (Introduction, p. 3)

Part II: Sorting and Order Statistics (Introduction, p. 147)

Part III: Data Structures (Introduction, p. 229)

Part IV: Advanced Design and Analysis Techniques (Introduction, p. 357)

Part V: Advanced Data Structures (Introduction, p. 481)

Part VI: Graph Algorithms (Introduction, p. 587)

Part VII: Selected Topics (Introduction, p. 769)

Part VIII: Appendix: Mathematical Background (Introduction, p. 1143)

Preface

To the teacher

To the student

To the professional

To our colleagues

Changes for the third edition

Website

How we produced this book

Acknowledgments for the third edition

Part I: Foundations

Introduction

1 The Role of Algorithms in Computing

1.1 Algorithms

What kinds of problems are solved by algorithms?

Data structures

Technique

Hard problems

Parallelism

Exercises 1.1-1 to 1.1-5

1.2 Algorithms as a technology

Efficiency

Algorithms and other technologies

Exercises 1.2-1 to 1.2-3

Problems

1-1 Comparison of running times

Chapter notes

2 Getting Started

2.1 Insertion sort

Loop invariants and the correctness of insertion sort

Pseudocode conventions

Exercises 2.1-1 to 2.1-4

2.2 Analyzing algorithms

Analysis of insertion sort

Worst-case and average-case analysis

Order of growth

Exercises 2.2-1 to 2.2-4

2.3 Designing algorithms

2.3.1 The divide-and-conquer approach

2.3.2 Analyzing divide-and-conquer algorithms

Analysis of merge sort

Exercises 2.3-1 to 2.3-7

Problems

Chapter notes

3 Growth of Functions

3.1 Asymptotic notation

Asymptotic notation, functions, and running times

Θ-notation

O-notation

Ω-notation

Asymptotic notation in equations and inequalities

o-notation

ω-notation

Comparing functions

Exercises 3.1-1 to 3.1-8

3.2 Standard notations and common functions

Monotonicity

Floors and ceilings

Modular arithmetic

Polynomials

Exponentials

Logarithms

Factorials

Functional iteration

The iterated logarithm function

Fibonacci numbers

Exercises 3.2-1 to 3.2-8

Problems

$Θ$ -notation

$O$ -notation

$Ω$ -notation

$o$ -notation

$ω$ -notation

Example: $T (n) = 3 T (⌊ n /4 ⌋) + Θ (n^{2})$

Example: $T (n) = T (n /3) + T (2 n /3) + O (n)$

5.4 Probabilistic analysis and further uses of indicator random variables $⋆$

input	after sort on digit 1	after sort on digit 2	after sort on digit 3
329	720	720	329
457	355	329	355
657	436	436	436
839	457	839	457
436	657	355	657
720	329	457	720
355	839	657	839