penis

3 Discrete Random Variables

Consider the experiment of tossing a coin three times where the probability of a head on an individual toss is $p$. Suppose that for each toss that comes up heads we win $1, but for each toss that comes up tails we lose $1. Clearly, a quantity of interest in this situation is our total winnings. Let $X$ denote this quantity. It is clear that $X$ can only be one of the values $3, $1, -\1$,and$-$3$.Wecannotwithcertaintysaywhichofthesevalues$X$willbe,sincethatvaluedependsontheoutcomeofourrandomexperiment.IfforexampletheoutcomeisHHH,then$X will be \3; while for the outcome HTH, $X$ will be $1. In the following table we list the values of $X$ (in dollars) corresponding to each of the eight possible outcomes.

ω	X(ω)	P{ω}
HHH	3	p³
HHT	1	p²(1-p)
HTH	1	p²(1-p)
THH	1	p²(1-p)
HTT	-1	p(1-p)²
THT	-1	p(1-p)²
TTH	-1	p(1-p)²
TTT	-3	(1-p)³

We can think of $X$ as a real-valued function on the probability space corresponding to the experiment. For each $\omega \in \Omega$, $X(\omega )$ is then one of the values, 3, 1, -1, -3. Consider, for example, the event $\{\omega :X(\omega )=1\}$. This set contains the three points ${\omega }_2,{\omega }_3$, and ${\omega }_4$ corresponding to the outcomes HHT, HTH, and THH, respectively. The last column in the table gives the probabilities associated with the eight possible outcomes of our experiment. From that table we see that the event $\{\omega :X(\omega )=1\}$ has probability $3p^2(1-p)$. We usually abbreviate this by saying that $\{X=1\}$ has probability $3p^2(1-p)$. Similar considerations, of course, apply to the other values that $X$ assumes. We see, therefore, that for each possible value of $X$, there is a precisely defined probability for $X$ assuming that value. As we shall see in the next section, the quantity $X$ is an example of what is called a discrete random variable.

3.1. Definitions

Let $(\Omega ,\mathcal{A},P)$ be an arbitrary probability space, and let $X$ be a real-valued function on $\Omega$ taking only a finite or countably infinite number of values $x_1,x_2,\dots$. As in the example just given, we would certainly like to be able to talk about the probability that $X$ assumes the value $x_i$, for each $i$. For this to be the case we need to know that for each $i$, $\{\omega \in \Omega :X(\omega )=x_i\}$ is an event, i.e., is a member of $\mathcal{A}$. If, as in the previous example, $\mathcal{A}$ is the $\sigma$-field of all subsets of $\Omega$ then this is certainly the case. For in that case, no matter what $x_i$ might be, $\{\omega :X(\omega )=x_i\}$ is a subset of $\Omega$ and hence a member of $\mathcal{A}$, since $\mathcal{A}$ contains every possible subset of $\Omega$. However, as was indicated in Section 1.2, in general $\mathcal{A}$ does not consist of all subsets of $\Omega$, so we have no guarantee that $\{\omega \in \Omega :X(\omega )=x_i\}$ is in $\mathcal{A}$. The only reasonable way out is to explicitly assume that $X$ is a function on $\Omega$ such that this desired property holds. This leads us to the following.

Definition 1 A discrete real-valued random variable $X$ on a probability space $(\Omega ,\mathcal{A},P)$ is a function $X$ with domain $\Omega$ and range a finite or countably infinite subset $\{x_1,x_2,\dots \}$ of the real numbers $\mathbb{R}$ such that $\{\omega :X(\omega )=x_i\}$ is an event for all $i$.

By definition, then, $\{\omega :X(\omega )=x_i\}$ is an event so we can talk about its probability. For brevity we usually write the event $\{\omega :X(\omega )=x_i\}$ as $\{X=x_i\}$ and denote the probability of this event as $P(X=x_i)$ rather than $P(\{\omega :X(\omega )=x_i\})$.

Let $X$ be a discrete real-valued random variable. Then for any real number $x$, $\{\omega :X(\omega )=x\}$ is an event. Indeed, if $x_1,x_2,\dots$ are the values that $X$ can assume, then $\{\omega :X(\omega )=x_i\}$ is an event by the definition of a discrete real-valued random variable. If $x$ is not one of these numbers, then $\{\omega :X(\omega )=x\}=\varnothing$, which is also an event.

If the possible values of a discrete random variable $X$ consist only of integers or of nonnegative integers, we say that $X$ is respectively an integer-valued random variable or a nonnegative integer-valued random variable. Most of the discrete random variables that arise in applications are nonnegative integer-valued.

Definition 2 The real-valued function $f$ defined on $\mathbb{R}$ by $f(x)=P(X=x)$ is called the discrete density function of $X$. A number $x$ is called a possible value of $X$ if $f(x)>0$.

Whenever necessary, we will denote the density function of $X$ by $f_X$ to show that it is the density function for the random variable $X$.

Example 1. Let $X$ be the random variable introduced at the beginning of this chapter in our discussion of tossing a coin three times with, say, $p=.4$. Then $X$ has the discrete density $f$ given by

$$f(-3)=.216,f(-1)=.432,f(1)=.288,f(3)=.064,$$

and $f(x)=0$ if $x\ne -3,-1,1,3$. This density can be represented in terms of a diagram as illustrated in Figure 1.

Figure 1

Example 2. Binomial distribution. Consider $n$ independent repetitions of the simple success-failure experiment discussed in Section 1.5. Let $S_n$ denote the number of successes in the $n$ trials. Then $S_n$ is a random variable that can only assume the values 0, 1, 2, …, $n$. In Chapter 1 we showed that for the integer $k$, $0\le k\le n$,

$$P(S_n=k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k};$$

hence the density $f$ of $S_n$ is given by

$$f(k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k},k=0,1,\dots ,n,$$

and $f(x)=0$ elsewhere. This density, which is among the most important densities that occur in probability theory, is called the binomial density with parameters $n$ and $p$. The density from Example 1 is a binomial density with parameters $n=3$ and $p=.4$.

One often refers to a random variable $X$ having a binomial density by saying that $X$ has a binomial distribution (with parameters $n$ and $p$ if one wants to be more precise). Similar phraseology is also used for other random variables having a named density.

As explained in Chapter 2, the binomial distribution arises in random sampling with replacement. For random sampling without replacement we have the following.

Example 3. Hypergeometric distribution. Consider a population of $r$ objects, of which $r_1$ are of one type and $r_2=r-r_1$ are of a second type. Suppose a random sample of size $n\le r$ is drawn from the population. Let $X$ be the number of objects of the first type in the sample. Then $X$ is a random variable whose possible values are 0, 1, 2, …, $n$. From the results in Section 2.4 we know that

$$P(X=x)=\frac{\left(\genfrac{}{}{0ex}{}{r_1}{x}\right)\left(\genfrac{}{}{0ex}{}{r-r_1}{n-x}\right)}{\left(\genfrac{}{}{0ex}{}{r}{n}\right)},x=0,1,2,\dots ,n.$$

Now we can write

$$\frac{\left(\genfrac{}{}{0ex}{}{r_1}{x}\right)\left(\genfrac{}{}{0ex}{}{r-r_1}{n-x}\right)}{\left(\genfrac{}{}{0ex}{}{r}{n}\right)}=\frac{(r_1{)}_x(r-r_1{)}_{n-x}}{x!(n-x)!}\cdot \frac{n!}{(r{)}_n}=\left(\genfrac{}{}{0ex}{}{n}{x}\right)\frac{(r_1{)}_x(r-r_1{)}_{n-x}}{(r{)}_n}.$$

Thus the density $f$ of $X$ can be written in the two forms

$$f(x)=\frac{\left(\genfrac{}{}{0ex}{}{r_1}{x}\right)\left(\genfrac{}{}{0ex}{}{r-r_1}{n-x}\right)}{\left(\genfrac{}{}{0ex}{}{r}{n}\right)},x=0,1,\dots ,n,$$

or

$$f(x)=\left(\genfrac{}{}{0ex}{}{n}{x}\right)\frac{(r_1{)}_x(r-r_1{)}_{n-x}}{(r{)}_n},x=0,1,\dots ,n.$$

This density is called the hypergeometric density.

Here are a few more examples of random variables.

Example 4. Constant random variable. Let $c$ be a real number. Then the function $X$ defined by $X(\omega )=c$ for all $\omega$ is a discrete random variable, since the set $\{\omega :X(\omega )=c\}$ is the entire set $\Omega$ and $\Omega$ is an event. Clearly, $P(X=c)=1$, so the density $f$ of $X$ is simply $f(c)=1$ and $f(x)=0$, $x\ne c$. Such a random variable is called a constant random variable. It is from this point of view that a numerical constant is considered a random variable.

Example 5. Indicator random variable. Let $A$ be an event. Set $X(\omega )=1$ if $\omega \in A$ and $X(\omega )=0$ if $\omega \notin A$. Then the event $A$ occurs if and only if $X=1$. This random variable is called the indicator random variable of $A$ because the value of $X$ tells whether or not the event $A$ occurs.

Conversely, if $X$ is a random variable on a probability space $(\Omega ,\mathcal{A},P)$ taking the values 1 or 0, then $X$ is the indicator random variable of the event

$$A=\{\omega :X(\omega )=1\}.$$

Let $p=P(X=1)$. The density $f$ of $X$ is then given by

$$f(0)=1-p,f(1)=p,\text{and}f(x)=0,x\ne 0\text{ or }1.$$

Example 6. Consider the following game of chance. A circular target of radius 1 is zoned into $n$ concentric disks of radius $1/n,2/n,\dots ,n/n=1$, as illustrated in Figure 2 for the case $n=5$. A dart is tossed at random onto the circle, and if it lands in the annular zone between the circles with radii $i/n$ and $(i+1)/n$, $n-i$ dollars are won, $i=0,1,2,\dots ,n-1$. Let $X$ denote the amount of money won. Find the density of $X$.

Figure 2

The probability space for this experiment will be chosen to be the uniform probability space on the disk of radius 1. Clearly $X$ is a discrete random variable on this space with the possible values 1, 2, …, $n$. The event $A=\{X=n-i\}$ occurs if and only if the dart lands in the region bounded by the circles of radii $i/n$ and $(i+1)/n$. According to our discussion in Section 1.2 the probability of $A$ is the area of $A$ divided by the area of the unit disk. Thus for $i=0,1,2,\dots ,n-1$

$$P(X=n-i)=P(A)=\frac{\pi \left[{\left(\frac{i+1}{n}\right)}^2-{\left(\frac{i}{n}\right)}^2\right]}{\pi }=\frac{2i+1}{n^2}.$$

Setting $n-i=x$ we see that the density of $X$ is

$$f(x)=\frac{2(n-x)+1}{n^2}=\frac{2n-2x+1}{n^2},x=1,2,\dots ,n.$$

The density $f$ of a discrete random variable $X$ has the following three important properties:

(i) $f(x)\ge 0,x\in \mathbb{R}$. (ii) $\{x:f(x)\ne 0\}$ is a finite or countably infinite subset of $\mathbb{R}$. Let $\{x_1,x_2,\dots \}$ denote this set. Then (iii) ${\sum }_{i}f(x_i)=1$.

Properties (i) and (ii) are immediate from the definition of the discrete density function of $X$. To see that (iii) holds, observe that the events $\{\omega :X(\omega )=x_i\}$ are mutually disjoint and their union is $\Omega$. Thus

$$\sum _{i}f(x_i)=\sum _{i}P(X=x_i)=P\left(\bigcup _i\{X=x_i\}\right)=P(\Omega )=1.$$

Definition 3 A real-valued function $f$ defined on $\mathbb{R}$ is called a discrete density function provided that it satisfies properties (i), (ii), and (iii) stated above.

It is easy to see that any discrete density function $f$ is the density function of some random variable $X$. In other words, given $f$ we can construct a probability space $(\Omega ,\mathcal{A},P)$ and a random variable $X$ defined on $\Omega$ whose discrete density is $f$. Indeed, let $f$ be given and suppose $\{x_1,x_2,\dots \}$ is the set of values where $f(x)\ne 0$. Take $\Omega =\{x_1,x_2,\dots \}$, $\mathcal{A}$ all subsets of $\Omega$, and $P$ the probability measure defined on $\mathcal{A}$ by $P(\{\omega \})=f(x_i)$ if $\omega =x_i$. The random variable $X$ defined by $X(\omega )=x_i$ if $\omega =x_i$ is then such a random variable. To see this note that $\{\omega :X(\omega )=x_i\}=\{x_i\}$ and thus

$$P(X=x_i)=P(\{x_i\})=f(x_i).$$

The above result assures us that statements like “Let $X$ be a random variable with discrete density $f$” always make sense, even if we do not specify directly a probability space upon which $X$ is defined. To save writing we will henceforth use the term density instead of discrete density throughout the remainder of this chapter.

The notion of a discrete random variable forms a convenient way of describing a random experiment that has a finite or countably infinite number of possible outcomes. We need not bother to set up a probability space for the experiment. Instead we can simply introduce a random variable $X$ taking values $x_1,x_2,\dots$ such that $X=x_i$ if and only if the experiment results in the $i$th outcome. Thus, for example, in drawing a card at random from a deck of $n$ cards, we can let $X=i$ if the $i$th card was drawn. Then $P(X=i)=n^{-1}$, so we could describe the experiment by saying we observe a random variable $X$ taking integer values 1, 2, …, $n$ and having $f(x)=n^{-1}$ for $x=1,2,\dots ,n$, and $f(x)=0$ elsewhere for its density function.

In general, conducting an experiment which has a finite or countably infinite number of possible outcomes can be described as observing the value of a discrete random variable $X$. Many times, in fact, this is how the experiment already appears to us, and often it is easier to think of the experiment in these terms rather than in terms of a probability space.

As an illustration of this idea consider the experiment of picking a point at random from the finite subset $S$ of $\mathbb{R}$ consisting of the distinct points $x_1,x_2,\dots ,x_s$. Then the function $f$ defined by

$$f(x)=\{\begin{array}{ll}\frac{1}{s},& x=x_1,x_2,\dots ,x_s,\\ 0,& \text{elsewhere},\end{array}$$

is clearly a discrete density function. A random variable $X$ having this density is said to be uniformly distributed on $S$. Observing a value of $X$ corresponds to our intuitive notion of choosing a point at random from $S$.

We will now introduce two more discrete densities that are very useful for solving certain classes of problems whose importance will become apparent later.

Example 7. Geometric densities. Let $0<p<1$. Then the real-valued function $f$ defined on $\mathbb{R}$ by

$$f(x)=\{\begin{array}{ll}p(1-p{)}^x,& x=0,1,2,\dots ,\\ 0,& \text{elsewhere},\end{array}$$

is a discrete density function called the geometric density with parameter $p$.

To see that $f$ is a density, all that needs to be checked is that condition (iii) holds, for here conditions (i) and (ii) are obviously satisfied. But (iii) follows from the familiar fact that the sum of the geometric series ${\sum }_{x=0}^{\infty }(1-p{)}^x$ is just $p^{-1}$.

Example 8. Negative binomial densities. Let $\alpha$ be any positive real number and let $0<p<1$. A density closely related to the geometric is the negative binomial density with parameters $\alpha$ and $p$ defined by

$$f(x)=\{\begin{array}{ll}{p}^{\alpha }\left(\genfrac{}{}{0ex}{}{-\alpha }{x}\right)(-1{)}^x(1-p{)}^x,& x=0,1,2,\dots ,\\ 0,& \text{elsewhere}.\end{array}$$

To show that this is a density we must verify that properties (i)-(iii) hold. Here property (ii) is obviously true. That (i) holds may be seen as follows. For $x$ a nonnegative integer,

$$\left(\genfrac{}{}{0ex}{}{-\alpha }{x}\right)=\frac{(-\alpha {)}_x}{x!}=\frac{(-\alpha )(-\alpha -1)\cdots (-\alpha -x+1)}{x!}$$ $$=\frac{(-1{)}^x(\alpha )(\alpha +1)\cdots (\alpha +x-1)}{x!}=(-1{)}^x\frac{(\alpha +x-1{)}_x}{x!}=(-1{)}^x\left(\genfrac{}{}{0ex}{}{\alpha +x-1}{x}\right).$$

Thus

$$p^{\alpha }\left(\genfrac{}{}{0ex}{}{-\alpha }{x}\right)(-1{)}^x(1-p{)}^x=p^{\alpha }\left(\genfrac{}{}{0ex}{}{\alpha +x-1}{x}\right)(1-p{)}^x.\text{\hspace{1em}(2)}$$

Since the right-hand side of (2) is clearly nonnegative we see that (i) holds. To verify (iii), recall that the Taylor series of $(1-t{)}^{-\alpha }$ for $-1<t<1$ is

$$(1-t{)}^{-\alpha }=\sum _{x=0}^{\infty }(\genfrac{}{}{0ex}{}{-\alpha }{x})(-t{)}^x.\text{\hspace{1em}(3)}$$

From (3) with $t=1-p$, we see that

$$p^{-\alpha }=\sum _{x=0}^{\infty }\left(\genfrac{}{}{0ex}{}{-\alpha }{x}\right)(-1{)}^x(1-p{)}^x$$

and hence that ${\sum }_{x}f(x)=1$.

From (2) we see that we can write the negative binomial density in the alternate form

$$f(x)=\{\begin{array}{ll}{p}^{\alpha }\left(\genfrac{}{}{0ex}{}{\alpha +x-1}{x}\right)(1-p{)}^x,& x=0,1,2,\dots ,\\ 0,& \text{elsewhere}.\end{array}\text{\hspace{1em}(4)}$$

For some purposes this form is more useful than that given in (1). Observe that the geometric density with parameter $p$ is a negative binomial density with parameters $\alpha =1$ and $p$.

Example 9. Poisson densities. Let $\lambda$ be a positive number. The Poisson density with parameter $\lambda$ is defined as

$$f(x)=\{\begin{array}{ll}\frac{{\lambda }^x{e}^{-\lambda }}{x!},& x=0,1,2,\dots ,\\ 0,& \text{elsewhere}.\end{array}$$

It is obvious that this function satisfies properties (i) and (ii) in the definition of a discrete density function. Property (iii) follows immediately from the Taylor series expansion of the exponential function, namely,

$$e^{\lambda }=\sum _{x=0}^{\infty }\frac{{\lambda }^x}{x!}.$$

Many counting type random phenomena are known from experience to be approximately Poisson distributed. Some examples of such phenomena are the number of atoms of a radioactive substance that disintegrate in a unit time interval, the number of calls that come into a telephone exchange in a unit time interval, the number of misprints on a page of a book, and the number of bacterial colonies that grow on a petri dish that has been smeared with a bacterial suspension. A full treatment of these models requires the notion of a Poisson process, which will be discussed in Chapter 9.

3.2. Computations with densities

So far we have restricted our attention to computing $P(X=x)$. Often we are interested in computing the probability of $\{\omega :X(\omega )\in A\}$ where $A$ is some subset of $\mathbb{R}$ other than a one-point set.

Let $A$ be any subset of $\mathbb{R}$ and let $X$ be a discrete random variable having distinct possible values $x_1,x_2,\dots$. Then $\{\omega :X(\omega )\in A\}$ is an event. To see this, observe that

$$\{\omega \mid X(\omega )\in A\}=\bigcup _{x_i\in A}\{\omega \mid X(\omega )=x_i\},\text{\hspace{1em}(5)}$$

where by ${\bigcup }_{x_i\in A}$ we mean the union over all $i$ such that $x_i\in A$. Usually the event $\{\omega :X(\omega )\in A\}$ is abbreviated to $\{X\in A\}$, and its probability is denoted by $P(X\in A)$. If $-\infty \le a<b\le \infty$ and $A$ is an interval with end points $a$ and $b$ say $A=(a,b]$ then we usually write $P(a<X\le b)$ instead of $P(X\in (a,b])$. Similar notation is used for the other intervals with these endpoints.

An abbreviated notation is also used for conditional probabilities. Thus, for example, if $A$ and $B$ are two subsets of $\mathbb{R}$ we write $P(X\in A\mid X\in B)$ for the conditional probability of the event $\{X\in A\}$ given the event $\{X\in B\}$.

Let $f$ be the density of $X$. We can compute $P(X\in A)$ directly from the density $f$ by means of the formula

$$P(X\in A)=\sum _{x_i\in A}f(x_i),\text{\hspace{1em}(6)}$$

where by ${\sum }_{x_i\in A}$ we mean the sum over all $i$ such that $x_i\in A$. This formula follows immediately from (5) since the events $\{\omega \mid X(\omega )=x_i\}$, $i=1,2,\dots$, are disjoint. The right side of (6) is usually abbreviated as ${\sum }_{x\in A}f(x)$. In terms of this notation (6) becomes

$$P(X\in A)=\sum _{x\in A}f(x).\text{\hspace{1em}(7)}$$

The function $F(t)$, $-\infty <t<\infty$, defined by

$$F(t)=P(X\le t)=\sum _{x\le t}f(x),-\infty <t<\infty ,$$

is called the distribution function of the random variable $X$ or of the density $f$. It follows immediately from the definition of the distribution function that

$$P(a<X\le b)=P(X\le b)-P(X\le a)=F(b)-F(a).$$

If $X$ is an integer-valued random variable, then

$$F(t)=\sum _{x=-\infty }^{[t]}f(x),$$

where $[t]$ denotes the greatest integer less than or equal to $t$ (e.g., $[2.6]=[2]=2$). We see that $F$ is a nondecreasing function and that, for any integer $x$, $F$ has a jump of magnitude $f(x)$ at $x$ and $F$ is constant on the interval $[x,x+1)$. Further properties of distribution functions will be obtained, from a more general viewpoint, in Chapter 5.

Example 10. Set $S=\{1,2,\dots ,10\}$ and let $X$ be uniformly distributed on $S$. Then $f(x)=1/10$ for $x=1,2,\dots ,10$ and $f(x)=0$, elsewhere. The distribution function of $X$ is given by $F(t)=0$ for $t<1$, $F(t)=1$ for $t>10$ and

$$F(t)=\sum _{x=1}^{[t]}f(x)=\frac{[t]}{10},1\le x\le 10.$$

A graph of this distribution function is given in Figure 3. The probability $P(3<X\le 5)$ can be calculated either as

$$P(3<X\le 5)=f(4)+f(5)=2/10$$

or as

$$P(3<X\le 5)=F(5)-F(3)=5/10-3/10=2/10.$$

Similarly $P(3\le X\le 5)$ is obtained as

$$P(3\le X\le 5)=f(3)+f(4)+f(5)=3/10$$

or as

$$P(3\le X\le 5)=P(2<X\le 5)=F(5)-F(2)=5/10-2/10=3/10.$$

Example 11. Let $X$ have a geometric distribution with parameter $p$. Find the distribution function of $X$ and find $P(X\ge x)$ for the nonnegative integer $x$.

The density of $X$, according to Example 7, is

$$f(x)=p(1-p{)}^x,x=0,1,2,\dots$$

Thus $F(t)=0$ for $t<0$ and

$$F(t)=\sum _{x=0}^{[t]}p(1-p{)}^x,t\ge 0.$$

Using the formula for the sum of a finite geometric progression we conclude that

$$F(t)=1-(1-p{)}^{[t]+1},t\ge 0.$$

In particular, for $x$ a nonnegative integer, $F(x-1)=1-(1-p{)}^x$ and hence

$$P(X\ge x)=1-P(X<x)=1-P(X\le x-1)=1-F(x-1)=(1-p{)}^x.$$

Geometrically distributed random variables arise naturally in applications. Suppose we have a piece of equipment, such as an electrical fuse, that neither deteriorates nor improves in the course of time but can fail due to sporadic chance happenings that occur homogeneously in time. Let the object be observed at fixed time periods such as hours or days, and let $X$ be the number of time units up to and including the first failure, assuming that the object is new at time 0. Clearly $X$ is a discrete random variable whose possible values are found among the integers 1, 2, 3, … The event $\{X=n\}$ occurs if and only if the object first fails at the $n$th time period. Our intuitive notion that the object neither deteriorates nor improves with time can be precisely formulated as follows. If we know that the object has not failed by time $n$, i.e., the first failure is after time $n$ so $X>n$, then the probability that it does not fail until after time $n+m$, i.e., $P(X>n+m\mid X>n)$, should be the same as the probability of starting with an object which is new at time $n$ and having it not fail until after time $n+m$. The fact that the failure causes occur homogeneously in time can be taken to mean that this probability depends only on the number of time periods that elapse between $n$ and $n+m$, namely $m$, but not on $n$. Thus $P(X>n)$ should satisfy the equation

$$P(X>n+m\mid X>n)=P(X>m).\text{\hspace{1em}(8)}$$

Since

$$P(X>n+m\mid X>n)=\frac{P(X>n+m)}{P(X>n)},$$

we see that $P(X>n)$ satisfies the functional equation

$$\frac{P(X>n+m)}{P(X>n)}=P(X>m).$$

The only solution to this equation with $P(X>0)=1$ is $P(X>n)=c^n$ for some constant $c$. Indeed, let $c=P(X>1)$. Then by induction, $P(X>n)=c^n$. Since $P(X>n)$ is a decreasing function of $n$, $0<c<1$. Setting $p=1-c$ we see that $P(X>n)=(1-p{)}^n$. Thus

$$P(X=n)=P(X>n-1)-P(X>n)=(1-p{)}^{n-1}-(1-p{)}^n=p(1-p{)}^{n-1},$$

for $n=1,2,\dots$. If we let $Y=X-1$, then $Y$ has the geometric distribution with parameter $p$ as given in Example 7.

3.3. Discrete random vectors

Suppose we are interested in two random variables $X$ and $Y$ defined on the same probability space $(\Omega ,\mathcal{A},P)$. For each $\omega \in \Omega$, the two values $X(\omega )$ and $Y(\omega )$ define a point $(X(\omega ),Y(\omega ))$ in the plane ${\mathbb{R}}^2$. This defines a function $\mathbf{X}:\Omega \to {\mathbb{R}}^2$, which we usually write as $\mathbf{X}=(X,Y)$. In general, if $X_1,X_2,\dots ,X_r$ are $r$ random variables defined on the same probability space, then for each $\omega \in \Omega$, $X_i(\omega )$ is one of the finite or countably infinite number of values that the random variable $X_i$ can assume.

Let ${\mathbb{R}}^r$ denote the collection of all $r$-tuples of real numbers. A point $\mathbf{x}=(x_1,\dots ,x_r)$ of ${\mathbb{R}}^r$ is usually called an $r$-dimensional vector. Thus for each $\omega \in \Omega$, the $r$ values $X_1(\omega ),\dots ,X_r(\omega )$ define a point

$$\mathbf{X}(\omega )=(X_1(\omega ),X_2(\omega ),\dots ,X_r(\omega ))$$

of ${\mathbb{R}}^r$. This defines an $r$-dimensional vector-valued function on $\Omega$, $\mathbf{X}:\Omega \to {\mathbb{R}}^r$, which is usually written $\mathbf{X}=(X_1,X_2,\dots ,X_r)$. The function $\mathbf{X}$ is called a discrete $r$-dimensional random vector.

We have just defined an $r$-dimensional random vector in terms of $r$ real-valued random variables. Alternatively, an $r$-dimensional random vector can be defined directly as a function $\mathbf{X}:\Omega \to {\mathbb{R}}^r$ by extending the definition of a real-valued random variable almost verbatim.

Definition 4 A discrete $r$-dimensional random vector $\mathbf{X}$ is a function $\mathbf{X}$ from $\Omega$ to ${\mathbb{R}}^r$ taking on a finite or countably infinite number of values ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots$ such that

$$\{\omega :\mathbf{X}(\omega )={\mathbf{x}}_i\}$$

is an event for all $i$.

The discrete density function $f$ for the random vector $\mathbf{X}$ is defined by

$$f(x_1,\dots ,x_r)=P(X_1=x_1,\dots ,X_r=x_r)$$

or equivalently

$$f(\mathbf{x})=P(\mathbf{X}=\mathbf{x}),\mathbf{x}\in {\mathbb{R}}^r.$$

The probability that $\mathbf{X}$ belongs to the subset $A$ of ${\mathbb{R}}^r$ can be found by using the analog of (7), namely,

$$P(\mathbf{X}\in A)=\sum _{\mathbf{x}\in A}f(\mathbf{x}).$$

As in the one-dimensional case, this function $f$ has the following three properties:

(i) $f(\mathbf{x})\ge 0,\mathbf{x}\in {\mathbb{R}}^r$ (ii) $\{\mathbf{x}:f(\mathbf{x})\ne 0\}$ is a finite or countably infinite subset of ${\mathbb{R}}^r$, which will be denoted by $\{{\mathbf{x}}_1,{\mathbf{x}}_2,\dots \}$ (iii) ${\sum }_{i}f({\mathbf{x}}_i)=1$

Any real-valued function $f$ defined on ${\mathbb{R}}^r$ having these three properties will be called a discrete $r$-dimensional density function. The argument given in the one-dimensional case applies verbatim to show that any $r$-dimensional discrete density function is the density function of some $r$-dimensional random vector.

There is a certain amount of traditional terminology that goes along with random vectors and their density functions. Let $\mathbf{X}=(X_1,X_2,\dots ,X_r)$ be an $r$-dimensional random vector with density $f$. Then the function $f$ is usually called the joint density of the random variables $X_1,X_2,\dots ,X_r$. The density function of the random variable $X_i$ is then called the $i$th marginal density of $\mathbf{X}$ or of $f$.

Let $X$ and $Y$ be two discrete random variables. For any real numbers $x$ and $y$ the set $\{\omega \mid X(\omega )=x\text{ and }Y(\omega )=y\}$ is an event that we will usually denote by $\{X=x,Y=y\}$. Suppose that the distinct possible values of $X$ are $x_1,x_2,\dots$, and that the distinct possible values of $Y$ are $y_1,y_2,\dots$. For each $x$, the events $\{X=x,Y=y_j\}$, $j=1,2,\dots$, are disjoint and their union is the event $\{X=x\}$. Thus

$$P(X=x)=P\left(\bigcup _j\{X=x,Y=y_j\}\right)=\sum _{j}P(X=x,Y=y_j)=\sum _{y}P(X=x,Y=y).$$

This last expression results from using the same notational convention that was introduced for random variables in Section 3.2. Similarly,

$$P(Y=y)=P\left(\bigcup _i\{X=x_i,Y=y\}\right)=\sum _{i}P(X=x_i,Y=y)=\sum _{x}P(X=x,Y=y).$$

In other words, if we know the joint density of two discrete random variables $X$ and $Y$ then we can compute the density $f_X$ of $X$ by summing over $y$ and the density $f_Y$ of $Y$ by summing over $x$. Thus, in terms of densities, if $f$ is the joint density of $X$ and $Y$, then

$$f_X(x)=\sum _{y}f(x,y)\text{\hspace{1em}(11)}$$

and

$$f_Y(y)=\sum _{x}f(x,y).\text{\hspace{1em}(12)}$$

Example 12. Suppose two cards are drawn at random without replacement from a deck of 3 cards numbered 1, 2, 3. Let $X$ be the number on the first card and let $Y$ be the number on the second card. Then the joint density $f$ of $X$ and $Y$ is given by $f(1,2)=f(1,3)=f(2,1)=f(2,3)=f(3,1)=f(3,2)=1/6$ and $f(x,y)=0$ elsewhere. The first marginal density, that is, the density of $X$ is given by

$$f_X(1)=f(1,1)+f(1,2)+f(1,3)=0+1/6+1/6=2/6=1/3$$

and similarly for $x=2$ and 3. Therefore $f_X(x)=1/3$, $x=1,2,3$, and $f_X(x)=0$ elsewhere, as it should be.

Example 13. Suppose $X$ and $Y$ are random variables that assume the values $x$ and $y$, where $x=1$ or 2 and $y=1,2,3,4$, with probabilities given by the following table.

X \ Y	1	2	3	4
1	1/4	1/8	1/16	1/16
2	1/16	1/8	1/4	1/16

Then $f_X(1)={\sum }_{y=1}^{4}f(1,y)=1/4+1/8+1/16+1/16=1/2$, and $f_X(2)=1-f_X(1)=1/2$ so $X$ has the uniform distribution on 1,2. Similarly

$$f_Y(1)=1/4+1/16=5/16,f_Y(2)=3/16,f_Y(3)=5/16,f_Y(4)=3/16.$$

3.4. Independent random variables

Consider the experiment of tossing a coin and rolling a die. Intuitively, we believe that whatever the outcome of the coin toss is, it should have no influence on the outcome of the die roll, and vice-versa. We now wish to construct a probability model that reflects these views. Let $X$ be a random variable that is 1 or 0 according as the coin lands heads or tails, i.e., such that the event $\{X=1\}$ represents the outcome that the coin lands heads and the event $\{X=0\}$ represents the outcome that the coin lands tails. In a similar way we represent the outcome of the die roll by a random variable $Y$ that takes the value 1, 2, …, or 6 according as the die roll results in the face number 1, 2, …, or 6. The outcome of the combined experiment can then be given by the random vector $(X,Y)$. Our intuitive notion that the outcome of the coin toss and die roll have no influence on each other can be stated precisely by saying that if $x$ is one of the numbers 1 or 0 and $y$ is one of the numbers 1, 2, …, 6, then the events $\{X=x\}$ and $\{Y=y\}$ should be independent. Thus, the random vector $(X,Y)$ should have the joint density $f(x,y)$ given by

$$f(x,y)=P(X=x,Y=y)=P(X=x)P(Y=y)=f_X(x)f_Y(y).$$

In other words the joint density $f$ of $X$ and $Y$ should be given by

$$f(x,y)=f_X(x)f_Y(y).$$

Definition 5 Let $X_1,X_2,\dots ,X_r$ be $r$ discrete random variables having densities $f_1,f_2,\dots ,f_r$ respectively. These random variables are said to be mutually independent if their joint density function $f$ is given by

$$f(x_1,x_2,\dots ,x_r)=f_1(x_1)f_2(x_2)\cdots f_r(x_r).\text{\hspace{1em}(13)}$$

The random variables are said to be dependent if they are not independent. As in the case of the combined experiment of tossing a coin and rolling a die, the notion of independent random variables forms a convenient way to precisely formulate our intuitive notions that experiments are independent of each other.

Consider two independent discrete random variables having densities $f_X$ and $f_Y$, respectively. Then for any two subsets $A$ and $B$ of $\mathbb{R}$

$$P(X\in A,Y\in B)=P(X\in A)P(Y\in B).\text{\hspace{1em}(14)}$$

To see this, note that

$$P(X\in A,Y\in B)=\sum _{x\in A}\sum _{y\in B}{f}_{X,Y}(x,y)=\sum _{x\in A}\sum _{y\in B}{f}_X(x)f_Y(y)$$ $$=\left[\sum _{x\in A}{f}_X(x)\right]\left[\sum _{y\in B}{f}_Y(y)\right]=P(X\in A)P(Y\in B).$$

Formula (14) above extends easily from 2 to $r$ independent random variables. Thus if $A_1,A_2,\dots ,A_r$ are any $r$ subsets of $\mathbb{R}$ then

$$P(X_1\in A_1,\dots ,X_r\in A_r)=P(X_1\in A_1)\cdots P(X_r\in A_r).\text{\hspace{1em}(15)}$$

Example 14. Let $X$ and $Y$ be independent random variables each geometrically distributed with parameter $p$.

(a) Find the distribution of $\min (X,Y)$ (b) Find $P(\min (X,Y)=X)=P(Y\ge X)$ (c) Find the distribution of $X+Y$ (d) Find $P(Y=y\mid X+Y=z)$ for $y=0,1,\dots ,z$

To solve (a) we observe that for $z$ a nonnegative integer

$$P(\min (X,Y)\ge z)=P(X\ge z,Y\ge z)=P(X\ge z)P(Y\ge z),$$

so by Example 11

$$P(\min (X,Y)\ge z)=(1-p{)}^z(1-p{)}^z=(1-p{)}^{2z}.$$

It follows from Example 11 that $\min (X,Y)$ has a geometric distribution with parameter

$$1-(1-p{)}^2=2p-p^2.$$

To solve (b) we observe that

$$P(Y\ge X)=\sum _{x=0}^{\infty }P(X=x,Y\ge X)=\sum _{x=0}^{\infty }P(X=x,Y\ge x)$$ $$=\sum _{x=0}^{\infty }P(X=x)P(Y\ge x)=\sum _{x=0}^{\infty }p(1-p{)}^x(1-p{)}^x$$ $$=p\sum _{x=0}^{\infty }(1-p{)}^{2x}=\frac{p}{1-(1-p{)}^2}=\frac{p}{2p-p^2}.$$

To solve (c) we let $z$ be a nonnegative integer. Then

$$P(X+Y=z)=\sum _{x=0}^{z}P(X=x,X+Y=z)=\sum _{x=0}^{z}P(X=x,Y=z-x)$$ $$=\sum _{x=0}^{z}P(X=x)P(Y=z-x)=\sum _{x=0}^{z}p(1-p{)}^{x}p(1-p{)}^{z-x}$$ $$=(z+1)p^2(1-p{)}^z.$$

The solution to (d) is given by

$$P(Y=y\mid X+Y=z)=\frac{P(Y=y,X+Y=z)}{P(X+Y=z)}=\frac{P(X=z-y,Y=y)}{P(X+Y=z)}$$ $$=\frac{P(X=z-y)P(Y=y)}{P(X+Y=z)}=\frac{p(1-p{)}^{z-y}p(1-p{)}^y}{(z+1)p^2(1-p{)}^z}=\frac{1}{z+1}.$$

Consider some experiment (such as rolling a die) that has only a finite or countably infinite number of possible outcomes. Then, as already explained, we can think of this experiment as that of observing the value of a discrete random variable $X$. Suppose the experiment is repeated $n$ times. The combined experiment can be described as that of observing the values of the random variables $X_1,X_2,\dots ,X_n$, where $X_i$ is the outcome of the $i$th experiment. If the experiments are repeated under identical conditions, presumably the chance mechanism remains the same, so we should require that these $n$ random variables all have the same density. The intuitive notion that the repeated experiments have no influence on each other can now be formulated by demanding that the random variables $X_1,X_2,\dots ,X_n$ be mutually independent. Thus, in summary, $n$ independent random variables $X_1,\dots ,X_n$ having a common discrete density $f$ can be used to represent an $n$-fold independent repetition of an experiment having a finite or countably infinite number of outcomes.

The simplest random experiments are those that have only two possible outcomes, which we may label as success and failure. In tossing a coin, for example, we may think of getting a head as a success, while in drawing a card from a deck of $r$ cards we may consider getting an ace as a success. Suppose we make $n$ independent repetitions of our simple experiment. We can then describe the situation by letting $X_1,X_2,\dots ,X_n$ be $n$ independent indicator random variables such that $X_i=1$ or 0 according as the $i$th trial of the experiment results in a success or failure. In the literature, trials that can result in either success or failure are called Bernoulli trials, and the above situation is described by saying we perform $n$ Bernoulli trials with common probability $p=P(X_i=1)$ for success. In this context a random variable that takes on the values 1 and 0 with probabilities $p$ and $1-p$ respectively is said to have a Bernoulli density with parameter $p$.

The outcome of performing $n$ Bernoulli trials can be given by the random vector $\mathbf{X}=(X_1,X_2,\dots ,X_n)$. The information conveyed in this vector tells exactly which trials were a success and which were a failure. Often, such precise information is not required, and all we want to know is the number $S_n$ of trials that yielded a success among the $n$ trials. In Example 2 we showed that $S_n$ was binomially distributed with parameters $n$ and $p$. Observe that $S_n=X_1+\cdots +X_n$. Any random variable $Y$ that is binomially distributed with these same parameters can be thought of as the sum of $n$ independent Bernoulli random variables $X_1,\dots ,X_n$ each having parameter $p$.

Let us now consider independent repetitions of an experiment that has a finite number $r\ge 2$ of possible outcomes.

3.4.1. The multinomial distribution

Consider an experiment, such as rolling a die, that can result in only a finite number $r$ of distinct possible outcomes. We may represent this experiment by saying we observe a random variable $Y$ that assumes the values $1,2,\dots ,r$, so that the event $\{Y=i\}$ represents the fact that the experiment yielded the $i$th outcome. Let $p_i=P(Y=i)$. If we perform an $n$-fold independent repetition of the experiment, we can represent the outcome of these $n$ trials as an $n$-dimensional random vector $(Y_1,\dots ,Y_n)$, where the random variable $Y_j$ corresponds to the $j$th trial. Here the random variables $Y_1,\dots ,Y_n$ are mutually independent and $P(Y_j=i)=p_i$.

The random vector $(Y_1,\dots ,Y_n)$ tells us the outcomes of these $n$ trials. As in the case of $r=2$ outcomes we often are not interested in such a detailed account, but only would like to know how many of the $n$ trials resulted in each of the various possible outcomes. Let $X_i,i=1,2,\dots ,r$, denote the number of trials that yield the $i$th outcome. Then $X_i=x_i$ if and only if exactly $x_i$ of the $n$ random variables $Y_1,\dots ,Y_n$ assume the value $i$, i.e., exactly $x_i$ of the $n$ trials yield the $i$th outcome.

For example, for $r=3,n=5$, if

$$Y_1=2,Y_2=3,Y_3=3,Y_4=2,\text{ and }{Y}_5=2,$$

then

$$X_1=0,X_2=3,\text{ and }{X}_3=2.$$

We will now compute the joint density of $X_1,\dots ,X_r$. To this end let $x_1,x_2,\dots ,x_r$ be $r$ nonnegative integers with sum $x_1+\cdots +x_r=n$. A moment’s thought shows that since the random variables $Y_1,Y_2,\dots ,Y_n$ are independent with a common density, every specified choice of $x_1$ of them having value 1, $x_2$ of them having value 2, …, $x_r$ of them having value $r$, has the same probability, namely

$$p_1^{x_1}{p}_2^{x_2}\cdots p_r^{x_r}.$$

Thus letting $C(n;x_1,\dots ,x_r)$ denote the number of possible choices, we see that

$$P(X_1=x_1,\dots ,X_r=x_r)=C(n;x_1,\dots ,x_r)p_1^{x_1}\cdots p_r^{x_r}.$$

The computation of $C(n;x_1,\dots ,x_r)$ is a problem in combinatorial analysis that can easily be solved by the methods of Chapter 2. The simplest way of doing this is to think of the $r$ values 1, 2, …, $r$ as $r$ boxes and the $n$ trials as $n$ balls. Then $C(n;x_1,\dots ,x_r)$ is the number of ways we can place the $n$ balls into the $r$ boxes in such a manner as to have exactly $x_1$ balls in box 1, …, exactly $x_r$ balls in box $r$. If this is so, then box 1 has $x_1$ balls. These $x_1$ balls can be chosen from the $n$ balls in $\left(\genfrac{}{}{0ex}{}{n}{x_1}\right)$ ways. The remaining $n-x_1$ balls must be placed into the $r-1$ boxes 2, …, $r$ in such a manner as to have $x_2$ balls in box 2, …, $x_r$ balls in box $r$. Thus

$$C(n;x_1,\dots ,x_r)=\left(\genfrac{}{}{0ex}{}{n}{x_1}\right)C(n-x_1;x_2,\dots ,x_r).\text{\hspace{1em}(16)}$$

It now follows by induction on $r$ that

$$C(n;x_1,\dots ,x_r)=\frac{n!}{x_1!x_2!\cdots x_r!}.\text{\hspace{1em}(17)}$$

Indeed, for $r=1$ there is nothing to prove. Assume that (17) holds for $r-1$ boxes. Then from (16) we see that

$$C(n;x_1,\dots ,x_r)=\frac{n!}{x_1!(n-x_1)!}\cdot \frac{(n-x_1)!}{x_2!\cdots x_r!}=\frac{n!}{x_1!\cdots x_r!}$$

as desired.

The joint density $f$ of $X_1,\dots ,X_r$ is therefore given by

$$f(x_1,\dots ,x_r)=\{\begin{array}{ll}\frac{n!}{x_1!\cdots x_r!}{p}_1^{x_1}\cdots p_r^{x_r},& x_i\ge 0,{\sum }_{i=1}^r{x}_i=n,\\ 0,& \text{elsewhere}.\end{array}\text{\hspace{1em}(18)}$$

This density is called the multinomial density with parameters $n$ and $p_1,\dots ,p_r$.

We observe at once that the $r$ random variables $X_1,\dots ,X_r$ are not independent. In fact, since $X_1+\cdots +X_r=n$, any $r-1$ of them determine the $r$th. This, plus the fact that $p_1+\cdots +p_r=1$, is sometimes used to express the multinomial distribution in a different form. Let $x_1,x_2,\dots ,x_{r-1}$ be $r-1$ nonnegative integers such that $x_1+\cdots +x_{r-1}\le n$. Then

$$\begin{array}{rl}& P(X_1=x_1,\dots ,X_{r-1}=x_{r-1})\\ & =\frac{n!}{x_1!\cdots x_{r-1}!(n-x_1-\cdots -x_{r-1})!}\\ & \times p_1^{x_1}\cdots p_{r-1}^{x_{r-1}}(1-p_1-\cdots -p_{r-1}{)}^{n-x_1-\cdots -x_{r-1}}.\end{array}\text{\hspace{1em}(19)}$$

This form is convenient when we are interested in the first $r-1$ outcomes and think of the $r$th outcome as the outcome which is “not one of the $r-1$ outcomes.” Thus in rolling a die we might be interested in only knowing if a 2, 4, or 6 appeared. The experiment would then have four possible outcomes “2,” “4,” “6,” and “not (2, 4, 6).”

Let $k$ be a nonnegative integer, $k\le r$. A simple probability argument shows that for $x_1,x_2,\dots ,x_k$ nonnegative integers such that $x_1+\cdots +x_k\le n$,

$$\begin{array}{rl}& P(X_1=x_1,\dots ,X_k=x_k)\\ & =\frac{n!}{x_1!\cdots x_k![n-(x_1+\cdots +x_k)]!}\\ & \times p_1^{x_1}\cdots p_k^{x_k}(1-(p_1+\cdots +p_k){)}^{n-(x_1+\cdots +x_k)}.\end{array}\text{\hspace{1em}(20)}$$

3.4.2. Poisson approximation to the binomial distribution

There is an important connection between the binomial distribution and the Poisson distribution. Suppose, for example, that we perform $n$ Bernoulli trials with success probability $p_n=\lambda /n$ at each trial. Then the probability of having $S_n=k$ successes in the $n$ trials is given by

$$P(S_n=k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)(p_n{)}^k(1-p_n{)}^{n-k}=\frac{{\lambda }^k}{k!}\frac{(n{)}_k}{n^k}{\left(1-\frac{\lambda }{n}\right)}^n{\left(1-\frac{\lambda }{n}\right)}^{-k}.$$

Now as $n\to \infty$, $(n{)}_k/n^k\to 1$, $(1-\lambda /n{)}^n\to e^{-\lambda }$, and $(1-\lambda /n{)}^{-k}\to 1$. Consequently,

$$\underset{n\to \infty }{\lim }\left(\genfrac{}{}{0ex}{}{n}{k}\right)(p_n{)}^k(1-p_n{)}^{n-k}=\frac{{\lambda }^k}{k!}{e}^{-\lambda }.\text{\hspace{1em}(21)}$$

In the derivation of (21) we had $n{p}_n=\lambda$. Actually (21) holds whenever $n{p}_n\to \lambda$ as $n\to \infty$.

Equation (21) is used in applications to approximate the binomial distribution by the Poisson distribution when the success probability $p$ is small and $n$ is large. This is done by approximating the binomial probability $P(S_n=x)$ by means of $f(x)$ where $f$ is the Poisson density with parameter $\lambda =np$. The approximation is quite good if $n{p}^2$ is small. The following example illustrates the use of this technique.

Example 15. A machine produces screws, $1\%$ of which are defective. Find the probability that in a box of 200 screws there are no defectives.

Here we have $n=200$ Bernoulli trials, with success probability $p=.01$. The probability that there are no defective screws is

$$(1-.01{)}^{200}=(.99{)}^{200}=.1340.$$

The Poisson approximation to this is given by

$$e^{-200(0.01)}=e^{-2}=.1353.$$

The fact that the Poisson distribution can arise as a limit of binomial distributions has important theoretical consequences. It is one justification for developing models based on Poisson processes, which will be discussed in Chapter 9. The use of the Poisson approximation as a labor-saving device in computing binomial probabilities is of secondary importance, since the binomial probabilities themselves are readily computed.

3.5. Infinite sequences of Bernoulli trials

Consider repeatedly performing a success-failure experiment having probability $p$ of success until the first success appears. For any prescribed number $n$ of trials, there is the nonzero probability $(1-p{)}^n$ that no success occurs. Thus, in considering the number of trials until the first success, we cannot limit ourselves to any prescribed number of trials, but instead must consider an unending sequence of trials.

A given finite number $n$ of trials constitute $n$ Bernoulli trials, represented by $n$ independent Bernoulli random variables, $X_1,\dots ,X_n$. To represent an infinite sequence of Bernoulli trials we consider an infinite sequence $\{X_n\}$, $n\ge 1$, of independent Bernoulli random variables having the same parameter $p$.

In general, random variables $X_1,X_2,\dots$ are said to be independent if for any positive integer $n$, the random variables $X_1,\dots ,X_n$ are mutually independent. It can be shown that, given any discrete density $f$, there is a probability space $(\Omega ,\mathcal{A},P)$ upon which are defined mutually independent random variables $X_1,X_2,\dots$ each having the density $f$.

As our model for performing an unlimited sequence of Bernoulli trials, we therefore take an infinite sequence $\{X_n\}$, $n\ge 1$, of mutually independent Bernoulli random variables such that $P(X_n=1)=p$, $n\ge 1$. We interpret $X_n=1$ as meaning that the $n$th trial results in success, and $X_n=0$ as meaning that it results in failure.

Consider the number of trials $W_1$ until the first success. The random variable $W_1$ can assume only the integer values $1,2,\dots$. The event $\{W_1=n\}$ occurs if and only if the first $n-1$ trials are failures and the $n$th trial is a success. Therefore

$$\{W_1=n\}=\{X_1=0,\dots ,X_{n-1}=0,X_n=1\}.$$

It follows that

$$P(W_1=n)=P(X_1=0,\dots ,X_{n-1}=0,X_n=1)$$ $$=P(X_1=0)\cdots P(X_{n-1}=0)P(X_n=1)=(1-p{)}^{n-1}p.$$

Consequently

$$P(W_1-1=n)=p(1-p{)}^n.$$

Thus $W_1-1$ is geometrically distributed with parameter $p$.

Let $r\ge 1$ be an integer and let $T_r$ denote the number of trials until the $r$th success (so that the $r$th success occurs at trial $T_r$). Then $T_r$ is a random variable that can assume only the integer values $r,r+1,\dots$. The event $\{T_r=n\}$ occurs if and only if there is a success at the $n$th trial and during the first $n-1$ trials there are exactly $r-1$ successes. Thus

$$\{T_r=n\}=\{X_1+\cdots +X_{n-1}=r-1\}\cap \{X_n=1\}.$$

Since the two events on the right are independent and $X_1+\cdots +X_{n-1}$ has a binomial distribution with parameters $n-1$ and $p$, we see that for $n=r,r+1,\dots$

$$P(T_r=n)=P(X_1+\cdots +X_{n-1}=r-1)P(X_n=1)$$ $$=\left(\genfrac{}{}{0ex}{}{n-1}{r-1}\right)p^{r-1}(1-p{)}^{n-r}p=(\genfrac{}{}{0ex}{}{n-1}{r-1})p^r(1-p{)}^{n-r}.$$

Consequently

$$P(T_r-r=n)=\left(\genfrac{}{}{0ex}{}{r+n-1}{r-1}\right)p^r(1-p{)}^n.$$

We see from Equations (4) and (23) that $T_r-r$ has the negative binomial distribution with parameters $\alpha =r$ and $p$.

Let $T_0=0$ and for any integer $r\ge 1$ let $T_r$ be as above. Define $W_i=T_i-T_{i-1},i=1,2,\dots$. Then $W_i$ is the number of trials after the $(i-1)$st success until the $i$th success. We will now show that for any integer $r\ge 1$ the random variables $W_1-1,W_2-1,\dots ,W_r-1$ are mutually independent and have the same geometric density with parameter $p$.

To see this let $n_1,n_2,\dots ,n_r$ be any $r$ positive integers. Then the event $\{W_1=n_1,\dots ,W_r=n_r\}$ occurs if and only if among the first $n_1+\cdots +n_r$ trials all are failures except for trials

$$n_1,n_1+n_2,\dots ,n_1+\cdots +n_r,$$

which are successes. Since the trials are mutually independent with success probability $p$ we see that

$$P(W_1=n_1,\dots ,W_r=n_r)=(1-p{)}^{n_1-1}p(1-p{)}^{n_2-1}p\cdots (1-p{)}^{n_r-1}p$$ $$=\prod _{i=1}^r[p(1-p{)}^{n_i-1}].$$

Thus the random variables $W_1-1,\dots ,W_r-1$ are independent, geometrically distributed with parameter $p$.

Now clearly $T_r-r=(W_1-1)+\cdots +(W_r-1)$, so we see that $T_r-r$ is the sum of $r$ independent, geometrically distributed random variables. We have previously found that $T_r-r$ is negative binomially distributed with parameters $r$ and $p$. We have thus established the interesting and important fact that the distribution of the sum of $r$ independent, identically distributed geometric random variables with parameter $p$ is negative binomially distributed with parameters $r$ and $p$.

Further properties of infinite sequences of independent Bernoulli trials will be treated in the exercises.

3.6. Sums of independent random variables

In this section we discuss methods for finding the distribution of the sum of a finite number of independent discrete random variables. Let us start by considering two such variables $X$ and $Y$.

We assume, then, that $X$ and $Y$ are independent discrete random variables. Let $x_1,x_2,\dots$ denote the distinct possible values of $X$. For any $z$, the event $\{X+Y=z\}$ is the same as the event

$$\bigcup _i\{X=x_i,Y=z-x_i\}.$$

Since the events $\{X=x_i,Y=z-x_i\}$ are disjoint for distinct values of $i$, it follows that

$$P(X+Y=z)=\sum _{i}P(X=x_i,Y=z-x_i)$$ $$=\sum _{i}P(X=x_i)P(Y=z-x_i)=\sum _i{f}_X(x_i)f_Y(z-x_i).$$

In other words

$$f_{X+Y}(z)=\sum _x{f}_X(x)f_Y(z-x).\text{\hspace{1em}(24)}$$

If $X$ and $Y$ are integer-valued random variables, then $X+Y$ is also integer-valued. In this case we can interpret (24) as being valid when $z$ is an integer and the variable $x$ in the right-hand side of (24) ranges over the integers. One further specialization is useful. Suppose that $X$ and $Y$ assume only nonnegative integer values. Then $X+Y$ also assumes only nonnegative integer values. If $z$ is a nonnegative integer, then $f_X(x)f_Y(z-x)=0$ unless $x$ is one of the values 0, 1, …, $z$. Thus under these assumptions (24) can be written as

$$f_{X+Y}(z)=\sum _{x=0}^z{f}_X(x)f_Y(z-x).\text{\hspace{1em}(25)}$$

Although Equation (25) is useful for computing the density of $X+Y$, it is usually simpler to use probability generating functions. We will next describe such functions and then give several important applications of their use in computing the density of the sum of independent random variables.

Definition 6 Let $X$ be a nonnegative integer-valued random variable. The probability generating function ${\Phi }_X$ of $X$ is defined as

$${\Phi }_X(t)=\sum _{x=0}^{\infty }P(X=x)t^x=\sum _{x=0}^{\infty }{f}_X(x)t^x,-1\le t\le 1.$$

We will now calculate ${\Phi }_X(t)$ in three specific cases.

Example 16. Binomial distribution. Let $X$ have a binomial distribution with parameters $n$ and $p$. Then

$$P(X=x)=\left(\genfrac{}{}{0ex}{}{n}{x}\right)p^x(1-p{)}^{n-x}$$

and hence

$${\Phi }_X(t)=\sum _{x=0}^{n}P(X=x)t^x=\sum _{x=0}^n\left(\genfrac{}{}{0ex}{}{n}{x}\right)(pt{)}^x(1-p{)}^{n-x}.$$

From the binomial expansion formula

$$(a+b{)}^n=\sum _{x=0}^n(\genfrac{}{}{0ex}{}{n}{x})a^x{b}^{n-x},$$

we conclude that

$${\Phi }_X(t)=(pt+1-p{)}^n.\text{\hspace{1em}(26)}$$

Example 17. Negative binomial distribution. Let $X$ have a negative binomial distribution with parameters $\alpha$ and $p$. Then

$$P(X=x)=p^{\alpha }\left(\genfrac{}{}{0ex}{}{-\alpha }{x}\right)(-1{)}^x(1-p{)}^x$$

and hence

$${\Phi }_X(t)=\sum _{x=0}^{\infty }{p}^{\alpha }\left(\genfrac{}{}{0ex}{}{-\alpha }{x}\right)(-1{)}^x(1-p{)}^x{t}^x=p^{\alpha }\sum _{x=0}^{\infty }\left(\genfrac{}{}{0ex}{}{-\alpha }{x}\right)(-t(1-p){)}^x.$$

From the Taylor series expansion

$$(1+s{)}^{-\alpha }=\sum _{x=0}^{\infty }(\genfrac{}{}{0ex}{}{-\alpha }{x})s^x,$$

with $s=-t(1-p)$, it follows that

$${\Phi }_X(t)={\left(\frac{p}{1-t(1-p)}\right)}^{\alpha }.\text{\hspace{1em}(27)}$$

Example 18. Poisson distribution. Let $X$ have a Poisson distribution with parameter $\lambda$. Then

$$P(X=x)=\frac{{\lambda }^x{e}^{-\lambda }}{x!}$$

and hence

$${\Phi }_X(t)=e^{-\lambda }\sum _{x=0}^{\infty }\frac{(\lambda t{)}^x}{x!}.$$

By setting $s=\lambda t$ in the Taylor series expansion

$$e^s=\sum _{x=0}^{\infty }\frac{s^x}{x!},$$

we see that

$${\Phi }_X(t)=e^{\lambda t}{e}^{-\lambda }=e^{\lambda (t-1)}.\text{\hspace{1em}(28)}$$

Let $X$ and $Y$ be independent, nonnegative integer-valued random variables. Then

$${\Phi }_{X+Y}(t)={\Phi }_X(t){\Phi }_Y(t).\text{\hspace{1em}(29)}$$

To see this, note that by (25)

$${\Phi }_{X+Y}(t)=\sum _{z=0}^{\infty }{f}_{X+Y}(z)t^z=\sum _{z=0}^{\infty }{t}^z\sum _{x=0}^z{f}_X(x)f_Y(z-x)$$ $$=\sum _{x=0}^{\infty }{f}_X(x)t^x\sum _{z=x}^{\infty }{f}_Y(z-x)t^{z-x}=\sum _{x=0}^{\infty }{f}_X(x)t^x\sum _{y=0}^{\infty }{f}_Y(y)t^y$$ $$={\Phi }_X(t){\Phi }_Y(t),$$

which is the desired result.

It follows easily from (29) by induction that if $X_1,\dots ,X_r$ are independent, nonnegative integer-valued random variables, then

$${\Phi }_{X_1+\cdots +X_r}(t)={\Phi }_{X_1}(t)\cdots {\Phi }_{X_r}(t).\text{\hspace{1em}(30)}$$

The conclusions of the next theorem can be proven most easily by the “generating function technique,” which is based upon the fact that if

$$\sum _{x=0}^{\infty }{a}_x{t}^x=\sum _{x=0}^{\infty }{b}_x{t}^x,-1<t<1,$$

then we may equate the coefficients of $t^x$ in the two power series and conclude that $a_x=b_x,x=0,1,2,\dots$. This shows that if two nonnegative integer-valued random variables have the same probability generating function, they must have the same distribution. In other words, the probability generating function of a nonnegative integer-valued random variable uniquely determines the distribution of that random variable.

Theorem 1 Let $X_1,\dots ,X_r$ be independent random variables.

(i) If $X_i$ has the binomial distribution with parameters $n_i$ and $p$ then $X_1+\cdots +X_r$ has the binomial distribution with parameters $n_1+\cdots +n_r$ and $p$.

(ii) If $X_i$ has the negative binomial distribution with parameters ${\alpha }_i$ and $p$, then $X_1+\cdots +X_r$ has the negative binomial distribution with parameters ${\alpha }_1+\cdots +{\alpha }_r$ and $p$.

(iii) If $X_i$ has the Poisson distribution with parameter ${\lambda }_i$, then $X_1+\cdots +X_r$ has the Poisson distribution with parameter ${\lambda }_1+\cdots +{\lambda }_r$.

Proof of (i). If the $X_i$‘s are as in (i), then by Example 16

$${\Phi }_{X_1+\cdots +X_r}(t)={\Phi }_{X_1}(t)\cdots {\Phi }_{X_r}(t)$$ $$=(pt+1-p{)}^{n_1}\cdots (pt+1-p{)}^{n_r}=(pt+1-p{)}^{n_1+\cdots +n_r}.$$

Thus the probability generating function of $X_1+\cdots +X_r$ is the same as that of a random variable having a binomial distribution with parameters $n_1+\cdots +n_r$ and $p$. This implies that $X_1+\cdots +X_r$ must have that binomial distribution. For let

$$a_x=\left(\genfrac{}{}{0ex}{}{n_1+\cdots +n_r}{x}\right)p^x(1-p{)}^{n_1+\cdots +n_r-x}$$

denote the corresponding binomial probabilities. Then

$$\sum _{x=0}^{\infty }P(X_1+\cdots +X_r=x)t^x={\Phi }_{X_1+\cdots +X_r}(t)$$ $$=(pt+1-p{)}^{n_1+\cdots +n_r}=\sum _{x=0}^{\infty }{a}_x{t}^x.$$

Thus by equating coefficients we see that

$$P(X_1+\cdots +X_r=x)=a_x$$

and hence that $X_1+\cdots +X_r$ is binomially distributed as stated in (i).

Proof of (ii). If the $X_i$‘s are as in (ii), then by Example 17

$${\Phi }_{X_1+\cdots +X_r}(t)={\Phi }_{X_1}(t)\cdots {\Phi }_{X_r}(t)$$ $$={\left(\frac{p}{1-t(1-p)}\right)}^{{\alpha }_1}\cdots {\left(\frac{p}{1-t(1-p)}\right)}^{{\alpha }_r}={\left(\frac{p}{1-t(1-p)}\right)}^{{\alpha }_1+\cdots +{\alpha }_r}.$$

Thus the probability generating function of $X_1+\cdots +X_r$ is the same as that of a random variable having a negative binomial distribution with parameters ${\alpha }_1+\cdots +{\alpha }_r$ and $p$. It now follows by the same argument used in proving (i) that $X_1+\cdots +X_r$ has that negative binomial distribution.

The proof of (iii) is similar to that of (i) and (ii) and is left as an exercise for the reader.

Suppose ${\alpha }_1=\cdots ={\alpha }_r=1$ in statement (ii) of Theorem 1. Then $X_1,\dots ,X_r$ are each geometrically distributed with parameter $p$, and (ii) states that $X_1+\cdots +X_r$ has a negative binomial distribution with parameters $r$ and $p$. This provides an alternative proof to the result obtained in Section 3.5.

The next example illustrates the use of conditional probabilities.

Example 19 Let $X_1,X_2,\dots$ be independent nonnegative integer-valued random variables having a common density. Set $S_0=0$ and $S_n=X_1+\cdots +X_n,n\ge 1$. Let $N$ be a nonnegative integer-valued random variable and suppose that $N,X_1,X_2,\dots$ are independent. Then $S_N=X_1+\cdots +X_N$ is the sum of a random number of random variables. For an interpretation of $S_N$ suppose that at time 0 a random number $N$ of bacteria enters a system and that by time 1 the colony started by the $i$th bacterium contains $X_i$ members. Then $S_N$ is the total number of bacteria present at time 1. Show that the probability generating function of $S_N$ is given by

$${\Phi }_{S_N}(t)={\Phi }_N({\Phi }_{X_1}(t)),-1\le t\le 1.\text{\hspace{1em}(31)}$$

To verify (31) we observe first that

$$P(S_N=x)=\sum _{n=0}^{\infty }P(S_N=x,N=n)=\sum _{n=0}^{\infty }P(S_n=x,N=n)$$ $$=\sum _{n=0}^{\infty }P(N=n)P(S_n=x\mid N=n).$$

Since $N$ is independent of $X_1,X_2,\dots ,X_n$, it is independent of $S_n$, and hence $P(S_n=x\mid N=n)=P(S_n=x)$. Thus

$$P(S_N=x)=\sum _{n=0}^{\infty }P(N=n)P(S_n=x).\text{\hspace{1em}(32)}$$

Consequently for $-1\le t\le 1$

$${\Phi }_{S_N}(t)=\sum _{x=0}^{\infty }{t}^{x}P(S_N=x)=\sum _{x=0}^{\infty }{t}^x\sum _{n=0}^{\infty }P(N=n)P(S_n=x)$$ $$=\sum _{n=0}^{\infty }P(N=n)\sum _{x=0}^{\infty }{t}^{x}P(S_n=x)=\sum _{n=0}^{\infty }P(N=n){\Phi }_{S_n}(t)$$ $$=\sum _{n=0}^{\infty }P(N=n)({\Phi }_{X_1}(t){)}^n={\Phi }_N({\Phi }_{X_1}(t)).$$

4 Expectation of Discrete Random Variables

4.1 Definition of expectation

The expected value $EX$ is also called the mean of $X$ (or of the density $f$ of $X$) and is customarily denoted by $\mu$. The mean is one way of trying to summarize a probability distribution by a single number that is supposed to represent a “typical value” of $X$. How good this is depends on how closely the values of $X$ are clustered about the value $\mu$. We will examine this question in more detail when we discuss the variance of $X$ in Section 4.3.

Example 1. Binomial distribution

Let $X$ have the binomial distribution with parameters $n$ and $p$. Find $EX$.

For $n=1$, $X$ assumes the two values 0 and 1 with probabilities $(1-p)$ and $p$ respectively. Hence

$$EX=0\cdot P(X=0)+1\cdot P(X=1)=p.$$

Since a random variable having a binomial density with parameters 1 and $p$ is just an indicator random variable, we see that we can find the probability of the event $A$ that $X=1$ by computing the expectation of its indicator.

We now compute $EX$ for any $n\ge 1$. In this case $X$ assumes the values $0,1,2,\dots ,n$, and

$$EX=\sum _{j=0}^{n}j\left(\genfrac{}{}{0ex}{}{n}{j}\right)p^j(1-p{)}^{n-j}.$$

To calculate this quantity we observe that

$$j\left(\genfrac{}{}{0ex}{}{n}{j}\right)=\frac{jn!}{j!(n-j)!}=\frac{n(n-1)!}{(j-1)![(n-1)-(j-1)]!}=n\left(\genfrac{}{}{0ex}{}{n-1}{j-1}\right).$$

Thus

$$EX=n\sum _{j=1}^n\left(\genfrac{}{}{0ex}{}{n-1}{j-1}\right)p^j(1-p{)}^{n-j}.$$

Making the change of variable $i=j-1$ we see that

$$EX=np\sum _{i=0}^{n-1}\left(\genfrac{}{}{0ex}{}{n-1}{i}\right)p^i(1-p{)}^{n-i-1}.$$

By the binomial theorem

$$\sum _{i=0}^{n-1}\left(\genfrac{}{}{0ex}{}{n-1}{i}\right)p^i(1-p{)}^{n-i-1}=[p+(1-p){]}^{n-1}=1,$$

so we see that

$$EX=np.$$

---

Suppose now that $X$ is any discrete random variable having possible values $x_1,x_2,\dots$. We would like to define the expectation of $X$ as

$$EX=\sum _{j=1}^{\infty }{x}_{j}f(x_j).\text{\hspace{1em}(2)}$$

If $X$ has only a finite number of possible values $x_1,\dots ,x_r$, then (2) is just our previous definition. In the general discrete case, this definition is valid provided that the sum ${\sum }_j{x}_{j}f(x_j)$ is well defined. For this to be the case we require that ${\sum }_j|x_j|f(x_j)<\infty$. This leads to the following.

Definition 1 Let $X$ be a discrete random variable having density $f$. If ${\sum }_j|x_j|f(x_j)<\infty$ then we say that $X$ has finite expectation and we define its expectation by (2). On the other hand if ${\sum }_{j=1}^{\infty }|x_j|f(x_j)=\infty$ then we say $X$ does not have finite expectation and $EX$ is undefined.

If $X$ is a nonnegative random variable, the fact that $X$ has finite expectation is usually denoted by $EX<\infty$.

Example 2. Poisson distribution

Let $X$ be Poisson distributed with parameter $\lambda$. Then

$$EX=\sum _{j=1}^{\infty }j\frac{{\lambda }^j}{j!}{e}^{-\lambda }=\sum _{j=1}^{\infty }\frac{{\lambda }^j}{(j-1)!}{e}^{-\lambda }=\lambda e^{-\lambda }\sum _{j=0}^{\infty }\frac{{\lambda }^j}{j!}=\lambda e^{-\lambda }{e}^{\lambda }=\lambda .$$

Example 3. Geometric distribution

Let $X$ have a geometric distribution with parameter $p$. Find $EX$.

Now

$$EX=\sum _{j=0}^{\infty }jp(1-p{)}^j=p(1-p)\sum _{j=0}^{\infty }j(1-p{)}^{j-1}=-p(1-p)\sum _{j=0}^{\infty }\frac{d}{dp}(1-p{)}^j.$$

Since a power series can be differentiated term by term, it follows that

$$EX=-p(1-p)\frac{d}{dp}\sum _{j=0}^{\infty }(1-p{)}^j.$$

Using the formula for the sum of a geometric progression, we see that

$$EX=-p(1-p)\frac{d}{dp}\left(\frac{1}{p}\right)=-p(1-p)\left(-\frac{1}{p^2}\right)=\frac{1-p}{p}.$$

Example 4. A density without finite mean

Let $f$ be the function defined on $\mathbb{R}$ by

$$f(x)=\frac{1}{x(x+1)},x=1,2,\dots$$

and $f(x)=0$ otherwise. The function $f$ obviously satisfies properties (i) and (ii) in the definition of density functions. To see that $f$ satisfies property (iii) we note that

$$\frac{1}{x(x+1)}=\frac{1}{x}-\frac{1}{x+1}$$

and hence

$$\sum _{x=1}^{\infty }f(x)=\sum _{x=1}^{\infty }\left[\frac{1}{x}-\frac{1}{x+1}\right]=(1-1/2)+(1/2-1/3)+\cdots =1.$$

Thus $f$ is a density. Now $f$ does not have finite mean because

$$\sum _{x=1}^{\infty }|x|f(x)=\sum _{x=1}^{\infty }\frac{1}{x+1}$$

and it is well known that the harmonic series ${\sum }_{x=1}^{\infty }{x}^{-1}$ does not converge.

---

4.2 Properties of expectation

Often we want to compute the expectation of a random variable such as $Z=X_1+X_2$ or $Z=X^2$ that is itself a function $\varphi (\mathbf{X})$ of the random vector $\mathbf{X}$. Of course, if we know the density $f_Z$ of $Z$, this can be done by using (2). Quite frequently, however, the density of $Z$ may not be known, or the computation of $EZ$ from a known density of $Z$ may be quite difficult. Our next result will give us a way of deciding if $Z$ has finite expectation and, if it does, of computing $EZ$ directly in terms of the density $f_{\mathbf{X}}$ and the function $\varphi$.

Before stating this result we introduce a notational convention. Let $\mathbf{X}$ be a discrete $r$-dimensional random vector having possible values ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots$ and density $f$, and let $\varphi$ be any real-valued function defined on ${\mathbb{R}}^r$. Then ${\sum }_{\mathbf{x}}\varphi (\mathbf{x})f(\mathbf{x})$ is defined as

$$\sum _{\mathbf{x}}\varphi (\mathbf{x})f(\mathbf{x})=\sum _j\varphi ({\mathbf{x}}_j)f({\mathbf{x}}_j).$$

Theorem 1 Let $\mathbf{X}$ be a discrete random vector having density $f$ and let $\varphi$ be a real-valued function defined on ${\mathbb{R}}^r$. Then the random variable $Z=\varphi (\mathbf{X})$ has finite expectation if and only if

$$\sum _{\mathbf{x}}|\varphi (\mathbf{x})|f(\mathbf{x})<\infty \text{\hspace{1em}(4)}$$

and, when (4) holds,

$$EZ=\sum _{\mathbf{x}}\varphi (\mathbf{x})f(\mathbf{x}).\text{\hspace{1em}(5)}$$

Proof. Let $z_1,z_2,\dots$ denote the distinct possible values of $Z$ and let ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots$ denote the distinct possible values of $\mathbf{X}$. For any $z_j$ there is at least one ${\mathbf{x}}_i$ such that $z_j=\varphi ({\mathbf{x}}_i)$, but there may be more than one such ${\mathbf{x}}_i$. Let $A_j$ denote the collection of such ${\mathbf{x}}_i$‘s, that is,

$$A_j=\{{\mathbf{x}}_i\mid \varphi ({\mathbf{x}}_i)=z_j\}.$$

Then $\{\mathbf{X}\in A_j\}$ and $\{Z=z_j\}$ denote exactly the same events. Thus

$$P(Z=z_j)=P(\mathbf{X}\in A_j)=\sum _{\mathbf{x}\in A_j}{f}_{\mathbf{X}}(\mathbf{x}).$$

Consequently,

$$\sum _j|z_j|f_Z(z_j)=\sum _j|z_j|P(Z=z_j)=\sum _j|z_j|\sum _{\mathbf{x}\in A_j}{f}_{\mathbf{X}}(\mathbf{x})=\sum _j\sum _{\mathbf{x}\in A_j}|z_j|f_{\mathbf{X}}(\mathbf{x}).$$

Since $\varphi (\mathbf{x})=z_j$ for $\mathbf{x}$ in $A_j$, it follows that

$$\sum _j|z_j|f_Z(z_j)=\sum _j\sum _{\mathbf{x}\in A_j}|\varphi (\mathbf{x})|f_{\mathbf{X}}(\mathbf{x}).$$

By their definition, the sets $A_j$ are disjoint for distinct values of $j$, and their union is the set of all possible values of $\mathbf{X}$. Therefore

$$\sum _j|z_j|f_Z(z_j)=\sum _{\mathbf{x}}|\varphi (\mathbf{x})|f_{\mathbf{X}}(\mathbf{x}).$$

This shows that $Z$ has finite expectation if and only if (4) holds. If $Z$ does have finite expectation, then by repeating the above argument with the absolute signs eliminated, we conclude that (5) holds. ∎

Let $X$ be a random variable having density $f$ and let $\varphi (x)=|x|$. Then by Theorem 1, $|X|$ has finite expectation if and only if ${\sum }_x|x|f(x)<\infty$. But, according to our definition of expectation, $X$ has finite expectation if and only if the same series converges. We see therefore that $X$ has finite expectation if and only if $E|X|<\infty$.

We shall now use Theorem 1 to establish the following important properties of expectation.

Theorem 2 Let $X$ and $Y$ be two random variables having finite expectation.

(i) If $c$ is a constant and $P(X=c)=1$, then $EX=c$.

(ii) If $c$ is a constant, then $cX$ has finite expectation and $E(cX)=cEX$.

(iii) $X+Y$ has finite expectation and

$$E(X+Y)=EX+EY.$$

(iv) Suppose $P(X\ge Y)=1$. Then $EX\ge EY$; moreover, $EX=EY$ if and only if $P(X=Y)=1$.

(v) $|EX|\le E|X|$.

Proof. The proof of (i) is quite simple. If $P(X=c)=1$, then $X$ has density $f_X(x)=0$ for $x\ne c$ and $f_X(c)=1$. Thus by (2)

$$EX=\sum _{x}x{f}_X(x)=c{f}_X(c)=c.$$

To prove (ii) let $\varphi (x)=cx$ and observe that

$$\sum _x|cx|f_X(x)=|c|\sum _x|x|f_X(x)<\infty ,$$

so $cX$ has finite expectation. Thus by (5)

$$E(cX)=\sum _x(cx)f_X(x)=c\sum _{x}x{f}_X(x)=cEX.$$

To establish (iii) set $\varphi (x,y)=x+y$ and let $f$ denote the joint density of $X$ and $Y$. Then

$$\sum _{x,y}|x+y|f(x,y)\le \sum _{x,y}|x|f(x,y)+\sum _{x,y}|y|f(x,y)=\sum _x|x|\sum _{y}f(x,y)+\sum _y|y|\sum _{x}f(x,y)=\sum _x|x|f_X(x)+\sum _y|y|f_Y(y)<\infty$$

and hence $X+Y$ has finite expectation. Applying (5) we see that

$$E(X+Y)=\sum _{x,y}(x+y)f(x,y)=\sum _{x,y}xf(x,y)+\sum _{x,y}yf(x,y)=EX+EY.$$

To prove (iv) observe that $Z=X-Y=X+(-Y)$, and by (ii) and (iii) we see that

$$EX-EY=E(X-Y)=EZ=\sum _{z}z{f}_Z(z).$$

Since $P(Z\ge 0)=P(X\ge Y)=1$, the values $z_i$ that $Z=X-Y$ assumes must all be nonnegative. Thus ${\sum }_{z}z{f}_Z(z)\ge 0$ and hence $EX-EY\ge 0$. This yields the first part of (iv). If $EX=EY$, then $EZ=0$. But then

$$0=EZ=\sum _i{z}_i{f}_Z(z_i).$$

Now the sum of nonnegative terms can equal zero only if all the individual terms equal zero. Since $f_Z(z_i)>0$ it must be that $z_i=0$. Thus the only possible value of $Z$ is 0 and consequently $P(Z=0)=1$.

Finally, (v) follows from (iv) and (ii) because $-|X|\le X\le |X|$ and hence $-E|X|\le EX\le E|X|$. This completes the proof of the theorem. ∎

It easily follows from (ii) and (iii) that if $X_1,\dots ,X_n$ are any $n$ random variables having finite expectation, and $c_1,\dots ,c_n$ are any $n$ constants, then

$$E(c_1{X}_1+\cdots +c_n{X}_n)=c_{1}E{X}_1+\cdots +c_{n}E{X}_n.\text{\hspace{1em}(6)}$$

It is useful to know that a bounded random variable always has finite expectation. More precisely,

Theorem 3 Let $X$ be a random variable such that for some constant $M$, $P(|X|\le M)=1$. Then $X$ has finite expectation and $|EX|\le M$.

Proof. Let $x_1,x_2,\dots$ be the possible values of $X$. Then $|x_i|\le M$ for all $i$. Indeed, if $|x_i|>M$ for some possible value $x_i$, then

$$P(|X|>M)\ge P(|X|=|x_i|)>0,$$

which contradicts the fact that $P(|X|\le M)=1$. Consequently

$$\sum _i|x_i|f(x_i)\le M\sum _{i}f(x_i)\le M,$$

so $X$ has finite expectation. Moreover by (v) of Theorem 2,

$$|EX|\le E|X|=\sum _i|x_i|f(x_i)\le M.$$

∎

It follows easily from Theorem 3 and (iii) of Theorem 2 that if $X$ and $Y$ are two random variables such that $Y$ has finite expectation and for some constant $M$, $P(|X-Y|\le M)=1$, then $X$ also has finite expectation and $|EX-EY|\le M$. We leave the proof of this fact for the reader.

Since the expectation of the sum of two random variables is the sum of their expectations, one might suppose that the expectation of a product is the product of the expectations. That this is not true in general can be seen by considering the random variable $X$ taking values 1 and $-1$ each with probability $1/2$ and setting $Y=X$. Then $EX=EY=0$ but $EXY=E{X}^2=1$.

There is an important case when this product rule is valid, namely, when $X$ and $Y$ are independent random variables. We state this more formally as follows.

Theorem 4 Let $X$ and $Y$ be two independent random variables having finite expectations. Then $XY$ has finite expectation and

$$E(XY)=(EX)(EY).\text{\hspace{1em}(7)}$$

Proof. Observe that since $X$ and $Y$ are independent, the joint density of $X$ and $Y$ is $f_X(x)f_Y(y)$. Thus

$$\sum _{x,y}|xy|f(x,y)=\sum _{x,y}|x||y|f_X(x)f_Y(y)=\left(\sum _x|x|f_X(x)\right)\left(\sum _y|y|f_Y(y)\right)<\infty ,$$

so $XY$ has finite expectation. Using Theorem 1, we conclude that

$$E(XY)=\sum _{x,y}(xy)f_X(x)f_Y(y)=\left[\sum _{x}x{f}_X(x)\right]\left[\sum _{y}y{f}_Y(y)\right]=(EX)(EY).$$

∎

The converse of this property does not hold; two random variables $X$ and $Y$ may be such that $E(XY)=(EX)(EY)$ even though $X$ and $Y$ are not independent.

Example 5

Let $(X,Y)$ assume the values $(1,0)$, $(0,1)$, $(-1,0)$, and $(0,-1)$ with equal probabilities. Then $EX=EY=0$. Since $XY=0$ it follows that $E(XY)=0$ and hence $E(XY)=(EX)(EY)$. To see that $X$ and $Y$ are not independent observe, for example, that $P(X=0)=P(Y=0)=1/2$ whereas $P(X=0,Y=0)=0$. Thus

$$P(X=0,Y=0)\ne P(X=0)P(Y=0).$$

It is often easier to compute expectations by using the properties given in Theorem 2 than by using the definition directly. We now illustrate this technique with several examples.

Example 6. Binomial distribution

We already know from Example 1 that the mean of the binomial distribution with parameters $n$ and $p$ is $np$. We can also derive this fact in a very simple manner by using the property that the expectation of a sum is the sum of the expectations (iii) of Theorem 2. To this end, let $X_1,\dots ,X_n$ be $n$ independent Bernoulli random variables having parameter $p$ and set $S_n=X_1+\cdots +X_n$. Then $S_n$ has the binomial distribution with parameters $n$ and $p$. By the first part of Example 1, $E{X}_i=p$ for $1\le i\le n$, and hence

$$E{S}_n=E(X_1+\cdots +X_n)=\sum _{i=1}^{n}E{X}_i=np.$$

Example 7. Hypergeometric distribution

Suppose we have a population of $r$ objects, $r_1$ of which are of type one and $r-r_1$ of type two. A sample of size $n$ is drawn without replacement from this population. Let $S_n$ denote the number of objects of type one that are obtained. Compute $E{S}_n$.

We know that $S_n$ has the hypergeometric distribution, so we could compute $E{S}_n$ by using (2). It is far simpler, however, to proceed by introducing indicator random variables $X_1,\dots ,X_n$ as follows. The random variable $X_i=1$ if and only if the $i$th element in the sample is of type one. Then

$$E{X}_i=P(X_i=1)=\frac{r_1}{r}.$$

But $S_n=X_1+\cdots +X_n$, so using (iii) of Theorem 2 we see that

$$E{S}_n=\sum _{i=1}^{n}E{X}_i=n\frac{r_1}{r}.$$

Note that the random variables $X_i$ are not independent.

Example 8

Suppose we have a population of $r$ distinct objects labeled $1,2,\dots ,r$. Objects are drawn with replacement until exactly $k\le r$ distinct objects have been obtained. Let $S_k$ denote the size of the sample required. Compute $E{S}_k$.

It is clear that $S_1=1$ and hence $E{S}_1=1$. Assume $k\ge 2$ and let $X_i=S_{i+1}-S_i$, $i=1,2,\dots ,k-1$. Then clearly $S_k=1+X_1+\cdots +X_{k-1}$. Now $X_i$ is the number of objects that must be drawn after the $i$th new object enters the sample and until the $(i+1)$st new object enters the sample. A moment’s thought shows that the event $\{X_i=n\}$ occurs if and only if the first $n-1$ objects drawn after the $i$th new object enters the sample duplicate one of the previous $i$ objects, and the $n$th object drawn after the $i$th new object enters is different from the previous $i$ objects. Thus, as the trials are independent,

$$P(X_i=n)={\left(\frac{i}{r}\right)}^{n-1}\left(1-\frac{i}{r}\right),n=1,2,\dots$$

This shows that the random variable $X_i-1$ is geometric with parameter $p_i=1-(i/r)$. Hence by Example 3, $E(X_i-1)=p_i^{-1}(1-p_i)$, and

$$E{X}_i=p_i^{-1}(1-p_i)+1=p_i^{-1}=(1-i/r{)}^{-1}=\frac{r}{r-i}.$$

Consequently,

$$E{S}_k=1+\sum _{i=1}^{k-1}\frac{r}{r-i}=\sum _{i=0}^{k-1}\frac{r}{r-i}=r\left(\frac{1}{r}+\frac{1}{r-1}+\cdots +\frac{1}{r-k+1}\right).$$

We point out for later use that it is clear from the construction of the $X_i$ that they are mutually independent random variables.

In the previous chapter we have seen that nonnegative integer-valued random variables $X$ play a prominent role. For these random variables the following theorem can frequently be applied both to decide if $X$ has finite expectation and to compute the expectation of $X$.

Theorem 5 Let $X$ be a nonnegative integer-valued random variable. Then $X$ has finite expectation if and only if the series ${\sum }_{x=1}^{\infty }P(X\ge x)$ converges. If the series does converge, then

$$EX=\sum _{x=1}^{\infty }P(X\ge x).\text{\hspace{1em}(9)}$$

Proof. We will show that

$$\sum _{x=1}^{\infty }xP(X=x)=\sum _{x=1}^{\infty }P(X\ge x),\text{\hspace{1em}(10)}$$

from which the theorem follows immediately. To this end we first write the left side of (10) as

$$\sum _{x=1}^{\infty }P(X=x)\sum _{y=1}^{x}1.$$

It is permissible to interchange the order of summation and rewrite this expression as

$$\sum _{y=1}^{\infty }\sum _{x=y}^{\infty }P(X=x)=\sum _{y=1}^{\infty }P(X\ge y).$$

Replacing the dummy variable $y$ by the dummy variable $x$ in the right side of this equality, we obtain the right side of (10). This shows that (10) holds, as desired. ∎

For an elementary application of this theorem, suppose that $X$ is a geometrically distributed random variable having parameter $p$. Then $P(X\ge x)=(1-p{)}^x$ and thus by the theorem

$$EX=\sum _{x=1}^{\infty }(1-p{)}^x=(1-p)+(1-p{)}^2+\cdots =\frac{1-p}{p}.$$

This agrees with the result found in Example 3.

---

4.3 Moments

Let $X$ be a discrete random variable, and let $r\ge 0$ be an integer. We say that $X$ has a moment of order $r$ if $X^r$ has finite expectation. In that case we define the $r$th moment of $X$ as $E{X}^r$. If $X$ has a moment of order $r$ then the $r$th moment of $X-\mu$, where $\mu$ is the mean of $X$, is called the $r$th central moment (or the $r$th moment about the mean) of $X$. By Theorem 1 we can compute the $r$th moment and the $r$th central moment of $X$ directly from the density $f$ by the formulas

$$E{X}^r=\sum _x{x}^{r}f(x)$$

and

$$E(X-\mu {)}^r=\sum _x(x-\mu {)}^{r}f(x).$$

In view of (11) and (12), the $r$th moment and $r$th central moment are determined by the density $f$, and it therefore makes perfectly good sense to speak of them as the $r$th moment and $r$th central moment of this density.

Suppose $X$ has a moment of order $r$; then $X$ has a moment of order $k$ for all $k\le r$. To see this, observe that if $|x|\le 1$, then

$$|x^k|=|x{|}^k\le 1$$

while for $|x|>1$

$$|x{|}^k\le |x{|}^r.$$

Thus in either case it is always true that

$$|x{|}^k\le |x{|}^r+1.$$

Thus, by the comparison theorem for the convergence of series, we see that

$$\sum _x|x{|}^{k}f(x)\le \sum _x[|x{|}^r+1]f(x)=E(|X{|}^r)+1<\infty ,$$

so $X^k$ has finite expectation.

On the other hand, as was shown in Example 4, a random variable $X$ may not have even a first moment. A simple modification of this example shows that a random variable may have a moment of order $r$ but possess no higher order moment. (See Exercise 9.)

The first moment $(r=1)$ is just the mean of $X$. In general, the more moments of $X$ that we know the more information we have gained about the distribution of $X$; however, in applications it is the first two moments that are usually of major interest.

By property (iii) of Theorem 2 we know that if $X$ and $Y$ both have a finite first moment, then so does $X+Y$. We will now show that this desirable property holds true for moments of order $r$ as well.

Theorem 6 If the random variables $X$ and $Y$ have moments of order $r$, then $X+Y$ also has a moment of order $r$.

Proof. This theorem rests on the following simple inequality. For any nonnegative integer $j\le r$,

$$|x{|}^j|y{|}^{r-j}\le |x{|}^r+|y{|}^r,x,y\in \mathbb{R}.\text{\hspace{1em}(13)}$$

To see this, observe that if $|x|\le |y|$, then $|x{|}^j|y{|}^{r-j}\le |y{|}^j|y{|}^{r-j}=|y{|}^r\le |x{|}^r+|y{|}^r$; while if $|x|\ge |y|$, then $|x{|}^j|y{|}^{r-j}\le |x{|}^r\le |x{|}^r+|y{|}^r$. Thus (13) holds. Using (13) and the binomial theorem we now see that

$$|x+y{|}^r\le (|x|+|y|{)}^r=\sum _{j=0}^r\left(\genfrac{}{}{0ex}{}{r}{j}\right)|x{|}^j|y{|}^{r-j}\le \sum _{j=0}^r\left(\genfrac{}{}{0ex}{}{r}{j}\right)(|x{|}^r+|y{|}^r).$$

But

$$\sum _{j=0}^r\left(\genfrac{}{}{0ex}{}{r}{j}\right)=2^r$$

because

$$2^r=(1+1{)}^r=\sum _{j=0}^r(\genfrac{}{}{0ex}{}{r}{j})1^j{1}^{r-j}=\sum _{j=0}^r(\genfrac{}{}{0ex}{}{r}{j}).$$

Consequently

$$|x+y{|}^r\le 2^r(|x{|}^r+|y{|}^r).$$

Let $f$ be the joint density of $X$ and $Y$. Then

$$\sum _{x,y}|x+y{|}^{r}f(x,y)\le 2^r\sum _{x,y}(|x{|}^r+|y{|}^r)f(x,y)=2^r(E|X{|}^r+E|Y{|}^r)<\infty .$$

Hence by Theorem 1, $(X+Y{)}^r$ has finite expectation. ∎

It follows easily by induction that if $X_1,X_2,\dots ,X_n$ all have a moment of order $r$, then so does $X_1+\cdots +X_n$.

Let $X$ be a random variable having a finite second moment. Then the variance of $X$, denoted by $\mathrm{Var}X$ or $V(X)$, is defined by

$$\mathrm{Var}X=E[(X-EX{)}^2].$$

By expanding the right-hand side we see that

$$\mathrm{Var}X=E[X^2-2X(EX)+(EX{)}^2]=E{X}^2-(EX{)}^2.$$

One often denotes $EX$ by $\mu$ and $\mathrm{Var}X$ by ${\sigma }^2$. The nonnegative number $\sigma =\sqrt{\mathrm{Var}X}$ is called the standard deviation of $X$ or of $f_X$.

According to our previous discussion, the mean $\mu$ is the average value of the random variable $X$. One use of the variance is as a measure of the spread of the distribution of $X$ about the mean. The more $X$ tends to deviate from its mean value $\mu$, the larger $(X-\mu {)}^2$ tends to be, and hence the larger the variance becomes.

On the other hand, $\mathrm{Var}X=0$ if and only if $X$ is a constant. To see this, observe that if $P(X=c)=1$ for some constant $c$, then $EX=c$ and $\mathrm{Var}X=0$. Conversely, if $\mathrm{Var}X=0$, then $E[(X-EX{)}^2]=0$ and hence $P((X-EX{)}^2=0)=1$. Consequently $P(X=EX)=1$.

An alternative use of the mean and variance is provided by the following problem, which is of interest in statistics. Let $X$ be a random variable having a finite second moment, and suppose we want to choose the value of $a$ that minimizes $E(X-a{)}^2$. Such a value would provide the best fit to $X$ by a constant if the error were measured by the mean square deviation.

One way of solving this problem is to use calculus. Note that

$$E(X-a{)}^2=E{X}^2-2aEX+a^2.$$

If we differentiate with respect to $a$ and set the derivative equal to zero, we see that $a=EX$. Since the second derivative is positive (in fact, it equals 2), the point corresponds to a minimum, and the minimum value is $\mathrm{Var}X$.

There is a second way of solving this problem that is also important to understand. Note that

$$(X-a{)}^2=[(X-\mu )+(\mu -a){]}^2=(X-\mu {)}^2+2(X-\mu )(\mu -a)+(\mu -a{)}^2.$$

Since $E(X-\mu )=0$, it follows that the cross-product term has zero expectation and hence

$$E(X-a{)}^2=E(X-\mu {)}^2+(\mu -a{)}^2=\mathrm{Var}X+(\mu -a{)}^2.$$

It is now clear from (15) that $E(X-a{)}^2$ is at a minimum when $\mu =a$ and this minimum value is $\mathrm{Var}X$.

We can often find the moments of a nonnegative integer-valued random variable $X$ most simply by differentiating its probability generating function ${\Phi }_X$. Suppose for simplicity that

$$\sum _{x=0}^{\infty }{f}_X(x)t_0^x<\infty$$

for some $t_0>1$. Then we can regard ${\Phi }_X$ as being defined on $-t_0<t<t_0$ by

$${\Phi }_X(t)=\sum _{x=0}^{\infty }{f}_X(x)t^x,-t_0<t<t_0.$$

We can differentiate ${\Phi }_X(t)$ any number of times by differentiating the corresponding power series term by term. In particular

$${\Phi }_X^{\prime }(t)=\sum _{x=1}^{\infty }x{f}_X(x)t^{x-1},-t_0<t<t_0,$$

and

$${\Phi }_X^{\prime \prime }(t)=\sum _{x=2}^{\infty }x(x-1)f_X(x)t^{x-2},-t_0<t<t_0.$$

By our assumptions on $t_0$, we can let $t=1$ in these formulas, obtaining

$${\Phi }_X^{\prime }(1)=\sum _{x=1}^{\infty }x{f}_X(x)=EX$$

and

$${\Phi }_X^{\prime \prime }(1)=\sum _{x=2}^{\infty }x(x-1)f_X(x)=EX(X-1).$$

Thus the mean and variance of $X$ can be obtained from ${\Phi }_X$ by means of the formulas

$$EX={\Phi }_X^{\prime }(1)$$

and

$$\mathrm{Var}X=E{X}^2-(EX{)}^2={\Phi }_X^{\prime \prime }(1)+{\Phi }_X^{\prime }(1)-({\Phi }_X^{\prime }(1){)}^2.$$

Similar formulas, in terms of the higher derivatives of ${\Phi }_X(t)$ at $t=1$ can be developed for the other moments of $X$.

We now illustrate the use of these formulas with the following examples.

Example 9. Negative binomial distribution

Let $X$ be a random variable having a negative binomial distribution with parameters $\alpha$ and $p$. Find the mean and variance of $X$.

From Example 17 of Chapter 3, we know that the generating function of $X$ is given by ${\Phi }_X(t)=p^{\alpha }[1-t(1-p){]}^{-\alpha }$. Consequently,

$${\Phi }_X^{\prime }(t)=\alpha p^{\alpha }[1-t(1-p){]}^{-(\alpha +1)}(1-p)$$

and

$${\Phi }_X^{\prime \prime }(t)=(\alpha +1)\alpha p^{\alpha }[1-t(1-p){]}^{-(\alpha +2)}(1-p{)}^2.$$

Thus

$${\Phi }_X^{\prime }(1)=\alpha \frac{1-p}{p}$$

and

$${\Phi }_X^{\prime \prime }(1)=(\alpha +1)\alpha {\left(\frac{1-p}{p}\right)}^2.$$

Hence,

$$EX=\alpha \frac{1-p}{p}$$

and

$$\mathrm{Var}X=(\alpha +1)\alpha {\left(\frac{1-p}{p}\right)}^2+\alpha \frac{1-p}{p}-{\alpha }^2{\left(\frac{1-p}{p}\right)}^2=\alpha \frac{1-p}{p^2}.$$

In particular, if $X$ has a geometric distribution with parameter $p$, then $EX=(1-p)/p$ (as we have already seen) and $\mathrm{Var}X=(1-p)/p^2$.

Example 10. Poisson distribution

Let $X$ have a Poisson distribution with parameter $\lambda$. Find the mean and variance of $X$.

In Example 18 of Chapter 3 we found that ${\Phi }_X(t)=e^{\lambda (t-1)}$. Thus

$${\Phi }_X^{\prime }(t)=\lambda e^{\lambda (t-1)}$$

and

$${\Phi }_X^{\prime \prime }(t)={\lambda }^2{e}^{\lambda (t-1)}.$$

Consequently ${\Phi }_X^{\prime }(1)=\lambda$ and ${\Phi }_X^{\prime \prime }(1)={\lambda }^2$. It follows immediately that

$$EX=\lambda ,$$

which agrees with the answer found in Example 2, and

$$\mathrm{Var}X={\lambda }^2+\lambda -{\lambda }^2=\lambda .$$

This shows that if $X$ has a Poisson distribution with parameter $\lambda$, then the mean and variance of $X$ both equal $\lambda$.

---

4.4 Variance of a sum

Let $X$ and $Y$ be two random variables each having finite second moment. Then $X+Y$ has finite second moment and hence finite variance. Now

$$\mathrm{Var}(X+Y)=E[(X+Y)-E(X+Y){]}^2=E[(X-EX)+(Y-EY){]}^2=E(X-EX{)}^2+E(Y-EY{)}^2+2E[(X-EX)(Y-EY)]=\mathrm{Var}X+\mathrm{Var}Y+2E[(X-EX)(Y-EY)].$$

Thus, unlike the mean, the variance of a sum of two random variables is, in general, not the sum of the variances. The quantity

$$E[(X-EX)(Y-EY)]$$

is called the covariance of $X$ and $Y$ and written $\mathrm{Cov}(X,Y)$. Thus we have the important formula

$$\mathrm{Var}(X+Y)=\mathrm{Var}X+\mathrm{Var}Y+2\mathrm{Cov}(X,Y).\text{\hspace{1em}(16)}$$

Now

$$(X-EX)(Y-EY)=XY-(Y)(EX)-X(EY)+(EX)(EY),$$

and hence taking expectations we see that

$$\mathrm{Cov}(X,Y)=E[(X-EX)(Y-EY)]=E(XY)-(EX)(EY).\text{\hspace{1em}(17)}$$

From this form, it is clear that $\mathrm{Cov}(X,Y)=0$ whenever $X$ and $Y$ are independent. (Example 5 shows that the converse is false.) We see from (16) that if $X$ and $Y$ are independent random variables having finite second moments, then $\mathrm{Var}(X+Y)=\mathrm{Var}X+\mathrm{Var}Y$.

In particular if $P(Y=c)=1$ for some constant $c$, then $X$ and $Y$ are independent and the variance of $Y$ equals zero; consequently

$$\mathrm{Var}(X+c)=\mathrm{Var}X+\mathrm{Var}(c)=\mathrm{Var}X.\text{\hspace{1em}(18)}$$

More generally, if $X_1,X_2,\dots ,X_n$ are $n$ random variables each having a finite second moment, then

$$\mathrm{Var}\left(\sum _{i=1}^n{X}_i\right)=\sum _{i=1}^n\mathrm{Var}{X}_i+2\sum _{i=1}^{n-1}\sum _{j=i+1}^n\mathrm{Cov}(X_i,X_j),\text{\hspace{1em}(19)}$$

and, in particular, if $X_1,\dots ,X_n$ are mutually independent, then

$$\mathrm{Var}\left(\sum _{i=1}^n{X}_i\right)=\sum _{i=1}^n\mathrm{Var}{X}_i.\text{\hspace{1em}(20)}$$

These formulas can be derived by a direct computation similar to (but more complicated than) that used for the case $n=2$, or they can be established from the case $n=2$ by induction on $n$.

In particular, if $X_1,\dots ,X_n$ are independent random variables having a common variance ${\sigma }^2$ (for example, if they each had the same density), then

$$\mathrm{Var}(X_1+\cdots +X_n)=n\mathrm{Var}{X}_1=n{\sigma }^2.\text{\hspace{1em}(21)}$$

Another elementary but quite useful fact is that $\mathrm{Var}(aX)=a^2\mathrm{Var}X$. We leave the verification of this fact to the reader.

Example 11. Binomial distribution

Let $X_1,\dots ,X_n$ be $n$ independent Bernoulli random variables each having the same probability $p$ of assuming the value 1. Then (see Example 6) the sum $S_n=X_1+\cdots +X_n$ is binomially distributed with parameters $n$ and $p$. We have previously shown that $E{S}_n=np$. Using (21) we find at once that

$$\mathrm{Var}{S}_n=n\mathrm{Var}{X}_1.$$

Now $X_1^2=X_1$ because $X_1$ is either 0 or 1. Thus $E{X}_1^2=E{X}_1=p$ and hence

$$\mathrm{Var}{X}_1=E{X}_1^2-(E{X}_1{)}^2=p-p^2=p(1-p).$$

Consequently $\mathrm{Var}{S}_n=np(1-p)$.

In summary then, the mean of a binomially distributed variable is $np$ and its variance is $np(1-p)$.

Example 12. Hypergeometric distribution

Consider the situation in Example 7. We now want to compute $\mathrm{Var}{S}_n$, so as to obtain the variance of a hypergeometric distribution. To this end we will use Equation (19).

For the dependent indicators $X_1,\dots ,X_n$, we previously found that

$$P(X_i=1)=E{X}_i=\frac{r_1}{r}.$$

Since $X_i^2=X_i$ we see that

$$\mathrm{Var}{X}_i=E{X}_i^2-(E{X}_i{)}^2=\frac{r_1}{r}-{\left(\frac{r_1}{r}\right)}^2=\frac{r_1}{r}\left(1-\frac{r_1}{r}\right).$$

Next we must compute the covariances. Assume that $1\le i<j\le n$. Now $X_i{X}_j=0$ unless both $X_i$ and $X_j$ are 1, so

$$E{X}_i{X}_j=P(X_i=1,X_j=1)=\left(\frac{r_1}{r}\right)\left(\frac{r_1-1}{r-1}\right).$$

Thus

$$\mathrm{Cov}(X_i,X_j)=E(X_i{X}_j)-(E{X}_i)(E{X}_j)=\frac{r_1(r_1-1)}{r(r-1)}-{\left(\frac{r_1}{r}\right)}^2=\frac{r_1}{r}\left(\frac{r_1-1}{r-1}-\frac{r_1}{r}\right)=\frac{r_1}{r}\cdot \frac{r_1-r}{r(r-1)}.$$

Hence

$$\sum _{i=1}^{n-1}\sum _{j=i+1}^n\mathrm{Cov}(X_i,X_j)=\frac{n(n-1)}{2}\cdot \frac{r_1}{r}\cdot \frac{r_1-r}{r(r-1)}.$$

It now follows from (19) that

$$\mathrm{Var}{S}_n=n\frac{r_1(r-r_1)}{r^2}-n(n-1)\frac{r_1(r-r_1)}{r^2(r-1)}=n\frac{r_1}{r}\left(1-\frac{r_1}{r}\right)\left(1-\frac{n-1}{r-1}\right).$$

It is interesting to compare the mean and variance for the hypergeometric distribution with those of the binomial distribution having the same success probability $p=r_1/r$. Suppose we have a population of $r$ objects of which $r_1$ are of type one and $r-r_1$ are of type two. A random sample of size $n$ is drawn from the population. Let $Y$ denote the number of objects of type one in the sample.

If the sampling is done with replacement then $Y$ is binomially distributed with parameters $n$ and $p=r_1/r$, and hence

$$EY=n\frac{r_1}{r},\mathrm{Var}Y=n\frac{r_1}{r}\left(1-\frac{r_1}{r}\right).$$

On the other hand, if the sampling is done without replacement, then $Y$ has a hypergeometric distribution,

$$EY=n\frac{r_1}{r},\mathrm{Var}Y=n\frac{r_1}{r}\left(1-\frac{r_1}{r}\right)\left(1-\frac{n-1}{r-1}\right).$$

The mean is the same in the two cases, but in sampling without replacement the variance is smaller. Intuitively, the closer $n$ is to $r$ the more deterministic $Y$ becomes when we sample without replacement. Indeed, if $n=r$ then the variance is zero and $P(Y=r_1)=1$. But if $r$ is large compared to $n$, so that $n/r$ is close to zero, the ratio of the variances obtained in sampling with and without replacement is close to one. This is as it should be, since for fixed $n$ and large $r$ there is little difference between sampling with replacement and sampling without replacement.

---

4.5 Correlation coefficient

Let $X$ and $Y$ be two random variables having finite nonzero variances. One measure of the degree of dependence between the two random variables is the correlation coefficient $\rho (X,Y)$ defined by

$$\rho =\rho (X,Y)=\frac{\mathrm{Cov}(X,Y)}{\sqrt{(\mathrm{Var}X)(\mathrm{Var}Y)}}.$$

These random variables are said to be uncorrelated if $\rho =0$. Since $\mathrm{Cov}(X,Y)=0$ if $X$ and $Y$ are independent, we see at once that independent random variables are uncorrelated. It is also possible for dependent random variables to be uncorrelated, as can be seen from Example 5.

It is important for applications in statistics to know that the correlation coefficient $\rho$ is always between $-1$ and $1$, and that $|\rho |=1$ if and only if $P(X=aY)=1$ for some constant $a$. These facts are easy consequences of the following basic inequality called the Schwarz inequality.

Theorem 7 (Schwarz Inequality) Let $X$ and $Y$ have finite second moments. Then

$$[E(XY){]}^2\le (E{X}^2)(E{Y}^2).\text{\hspace{1em}(23)}$$

Furthermore, equality holds in (23) if and only if either $P(Y=0)=1$ or $P(X=aY)=1$ for some constant $a$.

Proof. If $P(Y=0)=1$, then $P(XY=0)=1$, $EXY=0$, and $E{Y}^2=0$; thus in this case (23) holds with equality. Also, if $P(X=aY)=1$, then a simple computation will show that both sides of (23) are equal to $(a^{2}E{Y}^2{)}^2$.

We now show that (23) always holds. From the above discussion we can assume that $P(Y=0)<1$ and hence $E{Y}^2>0$. The proof is based on a simple but clever trick. Observe that for any real number $\lambda$

$$0\le E(X-\lambda Y{)}^2={\lambda }^{2}E{Y}^2-2\lambda EXY+E{X}^2.$$

This is a quadratic function of $\lambda$. Since the coefficient $E{Y}^2$ of ${\lambda }^2$ is positive, the minimum is achieved for some value of $\lambda$, say $\lambda =a$, that can be found by the usual calculus method of setting the derivative equal to zero and solving. The answer is $a=[E(XY)][E{Y}^2{]}^{-1}$. Since the corresponding function value is

$$0\le E(X-aY{)}^2=E{X}^2-\frac{[E(XY){]}^2}{E{Y}^2},\text{\hspace{1em}(24)}$$

it follows that (23) holds. If equality holds in the Schwarz inequality (23), then from (24) we see that $E(X-aY{)}^2=0$, so that

$$P[(X-aY)=0]=1.$$

∎

Applying the Schwarz inequality to the random variables $(X-EX)$ and $(Y-EY)$ we see that

$$[E[(X-EX)(Y-EY)]{]}^2\le [E(X-EX{)}^2][E(Y-EY{)}^2];$$

that is,

$$[\mathrm{Cov}(X,Y){]}^2\le (\mathrm{Var}X)(\mathrm{Var}Y).$$

Thus by the definition of $\rho$

$$|\rho (X,Y)|\le 1.$$

We also see from Theorem 7 that $|\rho |=1$ if and only if $P(X=aY)=1$ for some constant $a$.

The correlation coefficient is of limited use in probability theory. It arises mainly in statistics and further discussion of it will be postponed to Volume II.

---

4.6 Chebyshev’s inequality

Let $X$ be a nonnegative random variable having finite expectation, and let $t$ be a positive real number. Define the random variable $Y$ by setting $Y=0$ if $X<t$ and $Y=t$ if $X\ge t$. Then $Y$ is a discrete random variable having the two possible values 0 and $t$ which it assumes with probabilities $P(Y=0)=P(X<t)$ and $P(Y=t)=P(X\ge t)$ respectively. Thus

$$EY=tP(Y=t)+0\cdot P(Y=0)=tP(Y=t)=tP(X\ge t).$$

Now clearly $X\ge Y$ and hence $EX\ge EY$. Thus

$$EX\ge EY=tP(X\ge t)$$

or

$$P(X\ge t)\le \frac{EX}{t}.\text{\hspace{1em}(25)}$$

Quite a variety of useful inequalities can be deduced from (25). The most important of these is the Chebyshev inequality.

Chebyshev’s Inequality. Let $X$ be a random variable with mean $\mu$ and finite variance ${\sigma }^2$. Then for any real number $t>0$

$$P(|X-\mu |\ge t)\le \frac{{\sigma }^2}{t^2}.\text{\hspace{1em}(26)}$$

To prove (26), we apply (25) to the nonnegative random variable $(X-\mu {)}^2$ and the number $t^2$. We conclude that

$$P((X-\mu {)}^2\ge t^2)\le \frac{E(X-\mu {)}^2}{t^2}=\frac{{\sigma }^2}{t^2}.$$

Since $(X-\mu {)}^2\ge t^2$ if and only if $|X-\mu |\ge t$ we see that (26) holds.

Chebyshev’s inequality gives an upper bound in terms of $\mathrm{Var}X$ and $t$ for the probability that $X$ deviates from its mean by more than $t$ units. Its virtue lies in its great generality. No assumption on the distribution of $X$ is made other than that it has finite variance. This inequality is the starting point for several theoretical developments. For most distributions that arise in practice, there are far sharper bounds for $P(|X-\mu |\ge t)$ than that given by Chebyshev’s inequality; however, examples show that in general the bound given by Chebyshev’s inequality cannot be improved upon (see Exercise 26).

Let $X_1,\dots ,X_n$ be $n$ independent random variables having the same distribution. These random variables may be thought of as $n$ independent measurements of some quantity that is distributed according to their common distribution. In this sense we sometimes speak of the random variables $X_1,\dots ,X_n$ as constituting a random sample of size $n$ from this distribution.

Suppose that the common distribution of these random variables has finite mean $\mu$. Then for $n$ sufficiently large we would expect that their arithmetic mean $S_n/n=(X_1+\cdots +X_n)/n$ should be close to $\mu$. If the $X_i$ also have finite variance, then

$$\mathrm{Var}\left(\frac{S_n}{n}\right)=\frac{n{\sigma }^2}{n^2}=\frac{{\sigma }^2}{n}$$

and thus $\mathrm{Var}(S_n/n)\to 0$ as $n\to \infty$. As discussed in Section 4.3, this implies that as $n$ gets large the distribution of $S_n/n$ becomes more concentrated about its mean $\mu$. More precisely, by applying Chebyshev’s inequality to $S_n/n$ we obtain the inequality

$$P\left(\left|\frac{S_n}{n}-\mu \right|\ge \delta \right)\le \frac{\mathrm{Var}(S_n/n)}{{\delta }^2}=\frac{{\sigma }^2}{n{\delta }^2}.\text{\hspace{1em}(27)}$$

In particular, it follows from (27) that for any $\delta >0$

$$\underset{n\to \infty }{\lim }P\left(\left|\frac{S_n}{n}-\mu \right|\ge \delta \right)=0.\text{\hspace{1em}(28)}$$

We may interpret (28) in the following way. The number $\delta$ can be thought of as the desired accuracy in the approximation of $\mu$ by $S_n/n$. Equation (28) assures us that no matter how small $\delta$ may be chosen the probability that $S_n/n$ approximates $\mu$ to within this accuracy, that is, $P(|S_n/n-\mu |<\delta )$, converges to 1 as the number of observations gets large. This fact is called the Weak Law of Large Numbers. We have proven the weak law only under the assumption that the common variance of the $X_i$ is finite. Actually this is not necessary; all that is needed is that the $X_i$ have a finite mean. We state this more general result in the following theorem. The proof will be given in Chapter 8.

Theorem 8 (Weak Law of Large Numbers) Let $X_1,X_2,\dots ,X_n$ be independent random variables having a common distribution with finite mean $\mu$ and set $S_n=X_1+\cdots +X_n$. Then for any $\delta >0$

$$\underset{n\to \infty }{\lim }P\left(\left|\frac{S_n}{n}-\mu \right|\ge \delta \right)=0.$$

Whenever the $X_i$ have finite mean, the weak law holds. However, when they also have finite variance, then (27) holds. This is a more precise statement since it gives us an upper bound for $P(|S_n/n-\mu |\ge \delta )$ in terms of $n$. We now illustrate the use of (27) by applying it to binomially distributed random variables.

Let $X_1,X_2,\dots ,X_n$ be $n$ independent Bernoulli random variables assuming the value 1 with common probability $p$. Then $\mu =p$ and ${\sigma }^2=p(1-p)$. Thus (27) shows that

$$P\left(\left|\frac{S_n}{n}-p\right|\ge \delta \right)\le \frac{p(1-p)}{n{\delta }^2}.\text{\hspace{1em}(29)}$$

Since $p(1-p)\le 1/4$ if $0<p<1$ (because by the usual calculus methods it can easily be shown that $p(1-p)$ has its maximum value at $p=1/2$), it follows that regardless of what $p$ may be,

$$P\left(\left|\frac{S_n}{n}-p\right|\ge \delta \right)\le \frac{1}{4n{\delta }^2}.\text{\hspace{1em}(30)}$$

Equation (29) is useful when we know the value of $p$, while (30) gives us a bound on $P(|S_n/n-p|\ge \delta )$ that is valid for any value of $p$. If $p$ is near $1/2$, (29) and (30) do not differ by much, but if $p$ is far from $1/2$ the estimate given by (29) may be much better. (Actually even the bounds given by (29) are quite poor. We shall discuss another method in Chapter 7 that yields much better estimates.)

Suppose $\delta$ and $ϵ>0$ are given. We may use (29) or (30) to find a lower bound on the number of trials needed to assure us that

$$P\left(\left|\frac{S_n}{n}-p\right|\ge \delta \right)\le ϵ.$$

Indeed, from (29) we see that this will be the case if $p(1-p)/n{\delta }^2\le ϵ$. Solving for $n$ we find that $n\ge p(1-p)/(ϵ{\delta }^2)$. If we use (30), then $n\ge (4ϵ{\delta }^2{)}^{-1}$ trials will do. We state again that these bounds on $n$ given by Chebyshev’s inequality are poor and that in fact a much smaller number of trials may be sufficient.

As an illustration of the difference between these two estimates for $n$, choose $\delta =.1$ and $ϵ=.01$. Then ${\delta }^2ϵ=10^{-4}$ and from (30) we see that to guarantee that

$$P\left(\left|\frac{S_n}{n}-p\right|\ge .1\right)\le .01$$

we would need $n=10^4/4=2500$ observations. Suppose, however, we knew that $p=.1$. Then since $p(1-p)=.09$ we see from (29) that $n\ge .09\times 10^4=900$ observations will suffice. For $p=1/2$, (29) gives the same estimate as (30), namely 2500.

To illustrate that the Chebyshev bounds are really poor in the case of the binomial distribution, suppose $n=100$ and $p=1/2$. From (29) we then obtain

$$P\left(\left|\frac{S_n}{n}-.5\right|\ge .1\right)\le .25.$$

This should be compared with the exact value for this probability which is .038.

4 Expectation of Discrete Random Variables

4.1 Definition of Expectation

The expected value $EX$ is also called the mean of $X$ (or of the density $f$ of $X$) and is customarily denoted by $\mu$. The mean is one way of trying to summarize a probability distribution by a single number that is supposed to represent a “typical value” of $X$. How good this is depends on how closely the values of $X$ are clustered about the value $\mu$. We will examine this question in more detail when we discuss the variance of $X$ in Section 4.3.

Example 1: Binomial Distribution

Problem: Let $X$ have the binomial distribution with parameters $n$ and $p$. Find $EX$.

Solution:

Case 1: $n=1$ For $n=1$, $X$ assumes the two values $0$ and $1$ with probabilities $(1-p)$ and $p$ respectively.

$$\begin{array}{rl}EX& =\sum _{x}xP(X=x)\\ & =0\cdot P(X=0)+1\cdot P(X=1)\\ & =0\cdot (1-p)+1\cdot p\\ & =p\end{array}$$

Since a random variable having a binomial density with parameters $1$ and $p$ is just an indicator random variable, we see that we can find the probability of the event $A$ that $X=1$ by computing the expectation of its indicator.

Case 2: General $n\ge 1$ In this case $X$ assumes the values $0,1,2,\dots ,n$.

$$\begin{array}{rl}EX& =\sum _{j=0}^{n}j\left(\genfrac{}{}{0ex}{}{n}{j}\right)p^j(1-p{)}^{n-j}\\ & =\sum _{j=1}^{n}j\left(\genfrac{}{}{0ex}{}{n}{j}\right)p^j(1-p{)}^{n-j}(\text{Term for }j=0\text{ is }0)\\ & =\sum _{j=1}^{n}j\frac{n!}{j!(n-j)!}{p}^j(1-p{)}^{n-j}(\text{Definition of binomial coefficient})\\ & =\sum _{j=1}^n\frac{n!}{(j-1)!(n-j)!}{p}^j(1-p{)}^{n-j}(\text{Cancel }j\text{ in numerator and denominator})\\ & =\sum _{j=1}^{n}n\frac{(n-1)!}{(j-1)![(n-1)-(j-1)]!}{p}^j(1-p{)}^{n-j}(\text{Factor out }n)\\ & =n\sum _{j=1}^n\left(\genfrac{}{}{0ex}{}{n-1}{j-1}\right)p^j(1-p{)}^{n-j}(\text{Recognize binomial coefficient})\\ & =np\sum _{j=1}^n\left(\genfrac{}{}{0ex}{}{n-1}{j-1}\right)p^{j-1}(1-p{)}^{n-j}(\text{Factor out }p)\\ & =np\sum _{i=0}^{n-1}\left(\genfrac{}{}{0ex}{}{n-1}{i}\right)p^i(1-p{)}^{n-1-i}(\text{Change variable }i=j-1)\\ & =np{\left[p+(1-p)\right]}^{n-1}(\text{Binomial Theorem})\\ & =np\cdot (1{)}^{n-1}\\ & =np\end{array}$$

---

Suppose now that $X$ is any discrete random variable having possible values $x_1,x_2,\dots$. We would like to define the expectation of $X$ as:

$$EX=\sum _{j=1}^{\infty }{x}_{j}f(x_j)\text{\hspace{1em}(2)}$$

If $X$ has only a finite number of possible values $x_1,\dots ,x_r$, then (2) is just our previous definition. In the general discrete case, this definition is valid provided that the sum ${\sum }_j{x}_{j}f(x_j)$ is well defined. For this to be the case we require that ${\sum }_j|x_j|f(x_j)<\infty$. This leads to the following.

Definition 1: Let $X$ be a discrete random variable having density $f$.

1. If ${\sum }_j|x_j|f(x_j)<\infty$, then we say that $X$ has finite expectation and we define its expectation by (2).
2. On the other hand, if ${\sum }_{j=1}^{\infty }|x_j|f(x_j)=\infty$, then we say $X$ does not have finite expectation and $EX$ is undefined.

If $X$ is a nonnegative random variable, the fact that $X$ has finite expectation is usually denoted by $EX<\infty$.

Example 2: Poisson Distribution

Problem: Let $X$ be Poisson distributed with parameter $\lambda$. Find $EX$.

Solution:

$$\begin{array}{rl}EX& =\sum _{j=0}^{\infty }j\frac{{\lambda }^j}{j!}{e}^{-\lambda }\\ & =\sum _{j=1}^{\infty }j\frac{{\lambda }^j}{j!}{e}^{-\lambda }(\text{Term for }j=0\text{ is }0)\\ & =\sum _{j=1}^{\infty }\frac{{\lambda }^j}{(j-1)!}{e}^{-\lambda }(\text{Cancel }j\text{ with }j!)\\ & =\lambda e^{-\lambda }\sum _{j=1}^{\infty }\frac{{\lambda }^{j-1}}{(j-1)!}(\text{Factor out }\lambda )\\ & =\lambda e^{-\lambda }\sum _{i=0}^{\infty }\frac{{\lambda }^i}{i!}(\text{Change variable }i=j-1)\\ & =\lambda e^{-\lambda }\cdot e^{\lambda }(\text{Taylor series for }{e}^{\lambda })\\ & =\lambda \end{array}$$

Example 3: Geometric Distribution

Problem: Let $X$ have a geometric distribution with parameter $p$. Find $EX$.

Solution:

$$\begin{array}{rl}EX& =\sum _{j=0}^{\infty }jp(1-p{)}^j\\ & =p\sum _{j=1}^{\infty }j(1-p{)}^j(\text{Term for }j=0\text{ is }0)\\ & =p(1-p)\sum _{j=1}^{\infty }j(1-p{)}^{j-1}(\text{Factor out }(1-p))\\ & =-p(1-p)\sum _{j=1}^{\infty }\frac{d}{dp}(1-p{)}^j(\text{Since }\frac{d}{dp}(1-p{)}^j=-j(1-p{)}^{j-1})\\ & =-p(1-p)\frac{d}{dp}\sum _{j=1}^{\infty }(1-p{)}^j(\text{Differentiate term by term})\\ & =-p(1-p)\frac{d}{dp}\left(\frac{1-p}{1-(1-p)}\right)(\text{Sum of geometric series }\sum _{k=1}^{\infty }{r}^k=\frac{r}{1-r})\\ & =-p(1-p)\frac{d}{dp}\left(\frac{1-p}{p}\right)\\ & =-p(1-p)\frac{d}{dp}\left(\frac{1}{p}-1\right)\\ & =-p(1-p)\left(-\frac{1}{p^2}\right)(\text{Derivative of constant is 0})\\ & =\frac{1-p}{p}\end{array}$$

(Note: The original text uses a slightly different indexing for the geometric series sum $\frac{1}{p}$. If the sum starts at $j=0$, $\sum (1-p{)}^j=1/p$. If the derivation follows the text’s specific steps:)

$$\begin{array}{rl}EX& =-p(1-p)\frac{d}{dp}\sum _{j=0}^{\infty }(1-p{)}^j\\ & =-p(1-p)\frac{d}{dp}\left(\frac{1}{p}\right)\\ & =-p(1-p)\left(-\frac{1}{p^2}\right)\\ & =\frac{1-p}{p}\end{array}$$

Example 4: A Density Without Finite Mean

Problem: Let $f$ be the function defined on $\mathbb{R}$ by $f(x)=\frac{1}{x(x+1)}$ for $x=1,2,\dots$ and $f(x)=0$ otherwise. Show $f$ is a density but does not have a finite mean.

Solution: Step 1: Verify $f$ is a density. We check if the sum of probabilities equals 1 using partial fraction decomposition.

$$\begin{array}{rl}\sum _{x=1}^{\infty }f(x)& =\sum _{x=1}^{\infty }\frac{1}{x(x+1)}\\ & =\sum _{x=1}^{\infty }\left[\frac{1}{x}-\frac{1}{x+1}\right](\text{Partial fractions})\\ & =\left(1-\frac{1}{2}\right)+\left(\frac{1}{2}-\frac{1}{3}\right)+\left(\frac{1}{3}-\frac{1}{4}\right)+\dots (\text{Telescoping sum})\\ & =1\end{array}$$

Thus $f$ is a valid density.

Step 2: Check for finite mean. We examine the sum $\sum |x|f(x)$.

$$\begin{array}{rl}\sum _{x=1}^{\infty }|x|f(x)& =\sum _{x=1}^{\infty }x\cdot \frac{1}{x(x+1)}\\ & =\sum _{x=1}^{\infty }\frac{1}{x+1}\\ & =\frac{1}{2}+\frac{1}{3}+\frac{1}{4}+\dots \end{array}$$

This is the harmonic series (minus the first term), which is well known to diverge (${\sum }_{x=1}^{\infty }{x}^{-1}=\infty$). Therefore, $X$ does not have finite expectation.

---

4.2 Properties of Expectation

Often we want to compute the expectation of a random variable such as $Z=X_1+X_2$ or $Z=X^2$ that is itself a function $\varphi (\mathbf{X})$ of the random vector $\mathbf{X}$. Of course, if we know the density $f_Z$ of $Z$, this can be done by using (2). Quite frequently, however, the density of $Z$ may not be known, or the computation of $EZ$ from a known density of $Z$ may be quite difficult. Our next result will give us a way of deciding if $Z$ has finite expectation and, if it does, of computing $EZ$ directly in terms of the density $f_{\mathbf{X}}$ and the function $\varphi$.

Before stating this result we introduce a notational convention. Let $\mathbf{X}$ be a discrete $r$-dimensional random vector having possible values ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots$ and density $f$, and let $\varphi$ be any real-valued function defined on ${\mathbb{R}}^r$. Then ${\sum }_{\mathbf{x}}\varphi (\mathbf{x})f(\mathbf{x})$ is defined as:

$$\sum _{\mathbf{x}}\varphi (\mathbf{x})f(\mathbf{x})=\sum _j\varphi ({\mathbf{x}}_j)f({\mathbf{x}}_j).$$

Theorem 1: Let $\mathbf{X}$ be a discrete random vector having density $f$ and let $\varphi$ be a real-valued function defined on ${\mathbb{R}}^r$. Then the random variable $Z=\varphi (\mathbf{X})$ has finite expectation if and only if:

$$\sum _{\mathbf{x}}|\varphi (\mathbf{x})|f(\mathbf{x})<\infty \text{\hspace{1em}(4)}$$

and, when (4) holds:

$$EZ=\sum _{\mathbf{x}}\varphi (\mathbf{x})f(\mathbf{x}).\text{\hspace{1em}(5)}$$

Proof: Let $z_1,z_2,\dots$ denote the distinct possible values of $Z$ and let ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots$ denote the distinct possible values of $\mathbf{X}$. For any $z_j$ there is at least one ${\mathbf{x}}_i$ such that $z_j=\varphi ({\mathbf{x}}_i)$, but there may be more than one such ${\mathbf{x}}_i$. Let $A_j$ denote the collection of such ${\mathbf{x}}_i$‘s, that is:

$$A_j=\{{\mathbf{x}}_i\mid \varphi ({\mathbf{x}}_i)=z_j\}.$$

Then $\{\mathbf{X}\in A_j\}$ and $\{Z=z_j\}$ denote exactly the same events. Thus:

$$\begin{array}{rl}P(Z=z_j)& =P(\mathbf{X}\in A_j)\\ & =\sum _{\mathbf{x}\in A_j}{f}_{\mathbf{X}}(\mathbf{x}).\end{array}$$

Consequently, considering the absolute sum for expectation:

$$\begin{array}{rl}\sum _j|z_j|f_Z(z_j)& =\sum _j|z_j|P(Z=z_j)\\ & =\sum _j|z_j|\sum _{\mathbf{x}\in A_j}{f}_{\mathbf{X}}(\mathbf{x})\\ & =\sum _j\sum _{\mathbf{x}\in A_j}|z_j|f_{\mathbf{X}}(\mathbf{x}).\end{array}$$

Since $\varphi (\mathbf{x})=z_j$ for $\mathbf{x}$ in $A_j$, it follows that:

$$\sum _j|z_j|f_Z(z_j)=\sum _j\sum _{\mathbf{x}\in A_j}|\varphi (\mathbf{x})|f_{\mathbf{X}}(\mathbf{x}).$$

By their definition, the sets $A_j$ are disjoint for distinct values of $j$, and their union is the set of all possible values of $\mathbf{X}$. Therefore:

$$\sum _j|z_j|f_Z(z_j)=\sum _{\mathbf{x}}|\varphi (\mathbf{x})|f_{\mathbf{X}}(\mathbf{x}).$$

This shows that $Z$ has finite expectation if and only if (4) holds. If $Z$ does have finite expectation, then by repeating the above argument with the absolute signs eliminated, we conclude that (5) holds. $\square$

Let $X$ be a random variable having density $f$ and let $\varphi (x)=|x|$. Then by Theorem 1, $|X|$ has finite expectation if and only if ${\sum }_x|x|f(x)<\infty$. But, according to our definition of expectation, $X$ has finite expectation if and only if the same series converges. We see therefore that $X$ has finite expectation if and only if $E|X|<\infty$.

We shall now use Theorem 1 to establish the following important properties of expectation.

Theorem 2: Let $X$ and $Y$ be two random variables having finite expectation.

1. If $c$ is a constant and $P(X=c)=1$, then $EX=c$.
2. If $c$ is a constant, then $cX$ has finite expectation and $E(cX)=cEX$.
3. $X+Y$ has finite expectation and $E(X+Y)=EX+EY$.
4. Suppose $P(X\ge Y)=1$. Then $EX\ge EY$; moreover, $EX=EY$ if and only if $P(X=Y)=1$.
5. $|EX|\le E|X|$.

Proof: (i) If $P(X=c)=1$, then $X$ has density $f_X(x)=0$ for $x\ne c$ and $f_X(c)=1$. Thus by (2):

$$\begin{array}{rl}EX& =\sum _{x}x{f}_X(x)\\ & =c{f}_X(c)\\ & =c\cdot 1\\ & =c\end{array}$$

(ii) Let $\varphi (x)=cx$. Observe that:

$$\sum _x|cx|f_X(x)=|c|\sum _x|x|f_X(x)<\infty ,$$

so $cX$ has finite expectation. Thus by (5):

$$\begin{array}{rl}E(cX)& =\sum _x(cx)f_X(x)\\ & =c\sum _{x}x{f}_X(x)\\ & =cEX\end{array}$$

(iii) Set $\varphi (x,y)=x+y$ and let $f$ denote the joint density of $X$ and $Y$. Then:

$$\begin{array}{rl}\sum _{x,y}|x+y|f(x,y)& \le \sum _{x,y}(|x|+|y|)f(x,y)(\text{Triangle Inequality})\\ & =\sum _{x,y}|x|f(x,y)+\sum _{x,y}|y|f(x,y)\\ & =\sum _x|x|\sum _{y}f(x,y)+\sum _y|y|\sum _{x}f(x,y)\\ & =\sum _x|x|f_X(x)+\sum _y|y|f_Y(y)\\ & <\infty (\text{Since }EX,EY\text{ are finite})\end{array}$$

Hence $X+Y$ has finite expectation. Applying (5) we see that:

$$\begin{array}{rl}E(X+Y)& =\sum _{x,y}(x+y)f(x,y)\\ & =\sum _{x,y}xf(x,y)+\sum _{x,y}yf(x,y)\\ & =EX+EY\end{array}$$

(iv) Observe that $Z=X-Y=X+(-Y)$. By (ii) and (iii) we see that:

$$EX-EY=E(X-Y)=EZ=\sum _{z}z{f}_Z(z).$$

Since $P(Z\ge 0)=P(X\ge Y)=1$, the values $z_i$ that $Z=X-Y$ assumes must all be nonnegative. Thus ${\sum }_{z}z{f}_Z(z)\ge 0$ and hence $EX-EY\ge 0$. If $EX=EY$, then $EZ=0$. But then:

$$0=EZ=\sum _i{z}_i{f}_Z(z_i).$$

Now the sum of nonnegative terms can equal zero only if all the individual terms equal zero. Since $f_Z(z_i)>0$ it must be that $z_i=0$. Thus the only possible value of $Z$ is 0 and consequently $P(Z=0)=1$.

(v) This follows from (iv) and (ii) because $-|X|\le X\le |X|$ and hence $-E|X|\le EX\le E|X|$. $\square$

It easily follows from (ii) and (iii) that if $X_1,\dots ,X_n$ are any $n$ random variables having finite expectation, and $c_1,\dots ,c_n$ are any $n$ constants, then:

$$E(c_1{X}_1+\cdots +c_n{X}_n)=c_{1}E{X}_1+\cdots +c_{n}E{X}_n.\text{\hspace{1em}(6)}$$

It is useful to know that a bounded random variable always has finite expectation. More precisely:

Theorem 3: Let $X$ be a random variable such that for some constant $M$, $P(|X|\le M)=1$. Then $X$ has finite expectation and $|EX|\le M$.

Proof: Let $x_1,x_2,\dots$ be the possible values of $X$. Then $|x_i|\le M$ for all $i$. Indeed, if $|x_i|>M$ for some possible value $x_i$, then:

$$P(|X|>M)\ge P(|X|=|x_i|)>0,$$

which contradicts the fact that $P(|X|\le M)=1$. Consequently:

$$\begin{array}{rl}\sum _i|x_i|f(x_i)& \le M\sum _{i}f(x_i)\\ & \le M\cdot 1\\ & =M\end{array}$$

So $X$ has finite expectation. Moreover by (v) of Theorem 2:

$$|EX|\le E|X|=\sum _i|x_i|f(x_i)\le M.\square$$

It follows easily from Theorem 3 and (iii) of Theorem 2 that if $X$ and $Y$ are two random variables such that $Y$ has finite expectation and for some constant $M$, $P(|X-Y|\le M)=1$, then $X$ also has finite expectation and $|EX-EY|\le M$.

Since the expectation of the sum of two random variables is the sum of their expectations, one might suppose that the expectation of a product is the product of the expectations. That this is not true in general can be seen by considering the random variable $X$ taking values $1$ and $-1$ each with probability $1/2$ and setting $Y=X$. Then $EX=EY=0$ but $EXY=E{X}^2=1$.

There is an important case when this product rule is valid, namely, when $X$ and $Y$ are independent random variables. We state this more formally as follows.

Theorem 4: Let $X$ and $Y$ be two independent random variables having finite expectations. Then $XY$ has finite expectation and:

$$E(XY)=(EX)(EY).\text{\hspace{1em}(7)}$$

Proof: Observe that since $X$ and $Y$ are independent, the joint density of $X$ and $Y$ is $f_X(x)f_Y(y)$. Thus:

$$\begin{array}{rl}\sum _{x,y}|xy|f(x,y)& =\sum _{x,y}|x||y|f_X(x)f_Y(y)\\ & =\left(\sum _x|x|f_X(x)\right)\left(\sum _y|y|f_Y(y)\right)\\ & <\infty (\text{Since }EX,EY\text{ are finite})\end{array}$$

So $XY$ has finite expectation. Using Theorem 1, we conclude that:

$$\begin{array}{rl}E(XY)& =\sum _{x,y}(xy)f_X(x)f_Y(y)\\ & =\left[\sum _{x}x{f}_X(x)\right]\left[\sum _{y}y{f}_Y(y)\right]\\ & =(EX)(EY).\square \end{array}$$

The converse of this property does not hold; two random variables $X$ and $Y$ may be such that $E(XY)=(EX)(EY)$ even though $X$ and $Y$ are not independent.

Example 5

Problem: Let $(X,Y)$ assume the values $(1,0)$, $(0,1)$, $(-1,0)$, and $(0,-1)$ with equal probabilities. Show $E(XY)=(EX)(EY)$ but $X,Y$ are not independent.

Solution: Step 1: Compute Expectations. Each outcome has probability $1/4$.

$$\begin{array}{rl}EX& =1(1/4)+0(1/4)+(-1)(1/4)+0(1/4)=0\\ EY& =0(1/4)+1(1/4)+0(1/4)+(-1)(1/4)=0\\ XY& =0\text{ in all cases}⟹E(XY)=0\end{array}$$

Thus $E(XY)=0=(EX)(EY)$.

Step 2: Check Independence.

$$\begin{array}{rl}P(X=0)& =P(0,1)+P(0,-1)=1/4+1/4=1/2\\ P(Y=0)& =P(1,0)+P(-1,0)=1/4+1/4=1/2\\ P(X=0,Y=0)& =0(\text{This outcome is not in the support})\end{array}$$

Since $P(X=0,Y=0)\ne P(X=0)P(Y=0)$ (as $0\ne 1/4$), $X$ and $Y$ are not independent.

It is often easier to compute expectations by using the properties given in Theorem 2 than by using the definition directly. We now illustrate this technique with several examples.

Example 6: Binomial Distribution (Revisited)

Problem: Derive the mean of the binomial distribution using linearity of expectation.

Solution: Let $X_1,\dots ,X_n$ be $n$ independent Bernoulli random variables having parameter $p$. Set $S_n=X_1+\cdots +X_n$. Then $S_n$ has the binomial distribution with parameters $n$ and $p$. By Example 1 (Case 1), $E{X}_i=p$ for $1\le i\le n$.

$$\begin{array}{rl}E{S}_n& =E(X_1+\cdots +X_n)\\ & =\sum _{i=1}^{n}E{X}_i(\text{Linearity of Expectation})\\ & =\sum _{i=1}^{n}p\\ & =np\end{array}$$

Example 7: Hypergeometric Distribution

Problem: Suppose we have a population of $r$ objects, $r_1$ of which are of type one and $r-r_1$ of type two. A sample of size $n$ is drawn without replacement. Let $S_n$ denote the number of objects of type one obtained. Compute $E{S}_n$.

Solution: Introduce indicator random variables $X_1,\dots ,X_n$. The random variable $X_i=1$ if and only if the $i$th element in the sample is of type one.

$$\begin{array}{rl}E{X}_i& =P(X_i=1)\\ & =\frac{r_1}{r}(\text{By symmetry, any draw has prob }{r}_1/r)\end{array}$$

But $S_n=X_1+\cdots +X_n$. Using linearity (Theorem 2 iii):

$$\begin{array}{rl}E{S}_n& =\sum _{i=1}^{n}E{X}_i\\ & =\sum _{i=1}^n\frac{r_1}{r}\\ & =n\frac{r_1}{r}\end{array}$$

(Note: The random variables $X_i$ are not independent, but linearity holds regardless.)

Example 8: Coupon Collector Problem Variant

Problem: Suppose we have a population of $r$ distinct objects labeled $1,2,\dots ,r$. Objects are drawn with replacement until exactly $k\le r$ distinct objects have been obtained. Let $S_k$ denote the size of the sample required. Compute $E{S}_k$.

Solution: It is clear that $S_1=1$ and hence $E{S}_1=1$. Assume $k\ge 2$ and let $X_i=S_{i+1}-S_i$ for $i=1,2,\dots ,k-1$. Then clearly $S_k=1+X_1+\cdots +X_{k-1}$. Now $X_i$ is the number of objects that must be drawn after the $i$th new object enters the sample and until the $(i+1)$st new object enters the sample. The event $\{X_i=n\}$ occurs if and only if the first $n-1$ objects drawn after the $i$th new object enters the sample duplicate one of the previous $i$ objects, and the $n$th object drawn is different from the previous $i$ objects. Thus, as the trials are independent:

$$P(X_i=n)={\left(\frac{i}{r}\right)}^{n-1}\left(1-\frac{i}{r}\right),n=1,2,\dots$$

This shows that the random variable $X_i$ is geometric with parameter $p_i=1-(i/r)$. (Note: The text says $X_i-1$ is geometric, but based on the PMF $P(X=n)=(1-p{)}^{n-1}p$, $X_i$ itself is geometric starting at 1). Using the mean of a geometric distribution ($1/p$):

$$\begin{array}{rl}E{X}_i& =\frac{1}{p_i}\\ & =\frac{1}{1-i/r}\\ & =\frac{r}{r-i}\end{array}$$

Consequently:

$$\begin{array}{rl}E{S}_k& =1+\sum _{i=1}^{k-1}E{X}_i\\ & =1+\sum _{i=1}^{k-1}\frac{r}{r-i}\\ & =\frac{r}{r}+\sum _{i=1}^{k-1}\frac{r}{r-i}(\text{Since }1=r/r)\\ & =\sum _{i=0}^{k-1}\frac{r}{r-i}\\ & =r\left(\frac{1}{r}+\frac{1}{r-1}+\cdots +\frac{1}{r-k+1}\right)\end{array}$$

In the previous chapter we have seen that nonnegative integer-valued random variables $X$ play a prominent role. For these random variables the following theorem can frequently be applied both to decide if $X$ has finite expectation and to compute the expectation of $X$.

Theorem 5: Let $X$ be a nonnegative integer-valued random variable. Then $X$ has finite expectation if and only if the series ${\sum }_{x=1}^{\infty }P(X\ge x)$ converges. If the series does converge, then:

$$EX=\sum _{x=1}^{\infty }P(X\ge x).\text{\hspace{1em}(9)}$$

Proof: We will show that:

$$\sum _{x=1}^{\infty }xP(X=x)=\sum _{x=1}^{\infty }P(X\ge x).\text{\hspace{1em}(10)}$$

To this end we first write the left side of (10) as:

$$\sum _{x=1}^{\infty }P(X=x)\sum _{y=1}^{x}1.$$

It is permissible to interchange the order of summation and rewrite this expression as:

$$\begin{array}{rl}\sum _{y=1}^{\infty }\sum _{x=y}^{\infty }P(X=x)& =\sum _{y=1}^{\infty }P(X\ge y).\end{array}$$

Replacing the dummy variable $y$ by the dummy variable $x$ in the right side of this equality, we obtain the right side of (10). $\square$

Application: Suppose that $X$ is a geometrically distributed random variable having parameter $p$.

$$\begin{array}{rl}P(X\ge x)& =(1-p{)}^x\\ EX& =\sum _{x=1}^{\infty }(1-p{)}^x\\ & =(1-p)+(1-p{)}^2+\dots \\ & =\frac{1-p}{1-(1-p)}\\ & =\frac{1-p}{p}\end{array}$$

This agrees with the result found in Example 3.

---

4.3 Moments

Let $X$ be a discrete random variable, and let $r\ge 0$ be an integer. We say that $X$ has a moment of order $r$ if $X^r$ has finite expectation. In that case we define the $r$th moment of $X$ as $E{X}^r$. If $X$ has a moment of order $r$ then the $r$th moment of $X-\mu$, where $\mu$ is the mean of $X$, is called the $r$th central moment (or the $r$th moment about the mean) of $X$. By Theorem 1 we can compute the $r$th moment and the $r$th central moment of $X$ directly from the density $f$ by the formulas:

$$E{X}^r=\sum _x{x}^{r}f(x)$$

and

$$E(X-\mu {)}^r=\sum _x(x-\mu {)}^{r}f(x).$$

Suppose $X$ has a moment of order $r$; then $X$ has a moment of order $k$ for all $k\le r$. To see this, observe that if $|x|\le 1$, then $|x^k|=|x{|}^k\le 1$, while for $|x|>1$, $|x{|}^k\le |x{|}^r$. Thus in either case it is always true that $|x{|}^k\le |x{|}^r+1$. Thus, by the comparison theorem for the convergence of series:

$$\sum _x|x{|}^{k}f(x)\le \sum _x[|x{|}^r+1]f(x)=E(|X{|}^r)+1<\infty ,$$

so $X^k$ has finite expectation.

Theorem 6: If the random variables $X$ and $Y$ have moments of order $r$, then $X+Y$ also has a moment of order $r$.

Proof: This theorem rests on the following simple inequality. For any nonnegative integer $j\le r$:

$$|x{|}^j|y{|}^{r-j}\le |x{|}^r+|y{|}^r,x,y\in \mathbb{R}.\text{\hspace{1em}(13)}$$

To see this, observe that if $|x|\le |y|$, then $|x{|}^j|y{|}^{r-j}\le |y{|}^j|y{|}^{r-j}=|y{|}^r\le |x{|}^r+|y{|}^r$; while if $|x|\ge |y|$, then $|x{|}^j|y{|}^{r-j}\le |x{|}^r\le |x{|}^r+|y{|}^r$. Thus (13) holds. Using (13) and the binomial theorem we now see that:

$$\begin{array}{rl}|x+y{|}^r& \le (|x|+|y|{)}^r\\ & =\sum _{j=0}^r\left(\genfrac{}{}{0ex}{}{r}{j}\right)|x{|}^j|y{|}^{r-j}\\ & \le \sum _{j=0}^r\left(\genfrac{}{}{0ex}{}{r}{j}\right)(|x{|}^r+|y{|}^r)\\ & =(|x{|}^r+|y{|}^r)\sum _{j=0}^r\left(\genfrac{}{}{0ex}{}{r}{j}\right)\\ & =(|x{|}^r+|y{|}^r)2^r\end{array}$$

Let $f$ be the joint density of $X$ and $Y$. Then:

$$\begin{array}{rl}\sum _{x,y}|x+y{|}^{r}f(x,y)& \le 2^r\sum _{x,y}(|x{|}^r+|y{|}^r)f(x,y)\\ & =2^r(E|X{|}^r+E|Y{|}^r)\\ & <\infty \end{array}$$

Hence by Theorem 1, $(X+Y{)}^r$ has finite expectation. $\square$

Let $X$ be a random variable having a finite second moment. Then the variance of $X$, denoted by $\mathrm{Var}X$ or $V(X)$, is defined by:

$$\mathrm{Var}X=E[(X-EX{)}^2].$$

By expanding the right-hand side we see that:

$$\begin{array}{rl}\mathrm{Var}X& =E[X^2-2X(EX)+(EX{)}^2]\\ & =E{X}^2-2(EX)(EX)+(EX{)}^2\\ & =E{X}^2-(EX{)}^2.\end{array}$$

One often denotes $EX$ by $\mu$ and $\mathrm{Var}X$ by ${\sigma }^2$. The nonnegative number $\sigma =\sqrt{\mathrm{Var}X}$ is called the standard deviation of $X$.

Optimization Property: Let $X$ be a random variable having a finite second moment. We want to choose the value of $a$ that minimizes $E(X-a{)}^2$.

$$\begin{array}{rl}E(X-a{)}^2& =E[(X-\mu )+(\mu -a){]}^2\\ & =E[(X-\mu {)}^2+2(X-\mu )(\mu -a)+(\mu -a{)}^2]\\ & =E(X-\mu {)}^2+2(\mu -a)E(X-\mu )+(\mu -a{)}^2\\ & =\mathrm{Var}X+0+(\mu -a{)}^2(\text{Since }E(X-\mu )=0)\\ & =\mathrm{Var}X+(\mu -a{)}^2\end{array}$$

It is now clear that $E(X-a{)}^2$ is at a minimum when $\mu =a$ and this minimum value is $\mathrm{Var}X$.

We can often find the moments of a nonnegative integer-valued random variable $X$ most simply by differentiating its probability generating function ${\Phi }_X$.

$${\Phi }_X(t)=\sum _{x=0}^{\infty }{f}_X(x)t^x.$$

Differentiating term by term:

$${\Phi }_X^{\prime }(t)=\sum _{x=1}^{\infty }x{f}_X(x)t^{x-1}⟹{\Phi }_X^{\prime }(1)=EX.$$ $${\Phi }_X^{\prime \prime }(t)=\sum _{x=2}^{\infty }x(x-1)f_X(x)t^{x-2}⟹{\Phi }_X^{\prime \prime }(1)=E[X(X-1)].$$

Thus:

$$\mathrm{Var}X=E{X}^2-(EX{)}^2={\Phi }_X^{\prime \prime }(1)+{\Phi }_X^{\prime }(1)-({\Phi }_X^{\prime }(1){)}^2.$$

Example 9: Negative Binomial Distribution

Problem: Let $X$ be a random variable having a negative binomial distribution with parameters $\alpha$ and $p$. Find the mean and variance of $X$.

Solution: The generating function is ${\Phi }_X(t)=p^{\alpha }[1-t(1-p){]}^{-\alpha }$. Step 1: First Derivative.

$$\begin{array}{rl}{\Phi }_X^{\prime }(t)& =p^{\alpha }(-\alpha )[1-t(1-p){]}^{-\alpha -1}\cdot (-(1-p))\\ & =\alpha p^{\alpha }(1-p)[1-t(1-p){]}^{-(\alpha +1)}\end{array}$$

Evaluate at $t=1$:

$$\begin{array}{rl}{\Phi }_X^{\prime }(1)& =\alpha p^{\alpha }(1-p)[1-(1-p){]}^{-(\alpha +1)}\\ & =\alpha p^{\alpha }(1-p)[p{]}^{-(\alpha +1)}\\ & =\alpha p^{\alpha }(1-p)p^{-\alpha -1}\\ & =\alpha \frac{1-p}{p}\end{array}$$

So $EX=\alpha \frac{1-p}{p}$.

Step 2: Second Derivative.

$$\begin{array}{rl}{\Phi }_X^{\prime \prime }(t)& =\alpha p^{\alpha }(1-p)\cdot -(\alpha +1)[1-t(1-p){]}^{-\alpha -2}\cdot (-(1-p))\\ & =(\alpha +1)\alpha p^{\alpha }(1-p{)}^2[1-t(1-p){]}^{-(\alpha +2)}\end{array}$$

Evaluate at $t=1$:

$$\begin{array}{rl}{\Phi }_X^{\prime \prime }(1)& =(\alpha +1)\alpha p^{\alpha }(1-p{)}^2[p{]}^{-\alpha -2}\\ & =(\alpha +1)\alpha {\left(\frac{1-p}{p}\right)}^2\end{array}$$

Step 3: Variance.

$$\begin{array}{rl}\mathrm{Var}X& ={\Phi }_X^{\prime \prime }(1)+{\Phi }_X^{\prime }(1)-({\Phi }_X^{\prime }(1){)}^2\\ & =(\alpha +1)\alpha {\left(\frac{1-p}{p}\right)}^2+\alpha \frac{1-p}{p}-{\alpha }^2{\left(\frac{1-p}{p}\right)}^2\\ & =\alpha {\left(\frac{1-p}{p}\right)}^2[\alpha +1-\alpha ]+\alpha \frac{1-p}{p}\\ & =\alpha {\left(\frac{1-p}{p}\right)}^2+\alpha \frac{1-p}{p}\\ & =\alpha \frac{1-p}{p}\left(\frac{1-p}{p}+1\right)\\ & =\alpha \frac{1-p}{p}\left(\frac{1-p+p}{p}\right)\\ & =\alpha \frac{1-p}{p^2}\end{array}$$

Example 10: Poisson Distribution

Problem: Let $X$ have a Poisson distribution with parameter $\lambda$. Find the mean and variance of $X$.

Solution: The generating function is ${\Phi }_X(t)=e^{\lambda (t-1)}$. Step 1: Derivatives.

$$\begin{array}{rl}{\Phi }_X^{\prime }(t)& =\lambda e^{\lambda (t-1)}\\ {\Phi }_X^{\prime \prime }(t)& ={\lambda }^2{e}^{\lambda (t-1)}\end{array}$$

Step 2: Evaluate at $t=1$.

$$\begin{array}{rl}{\Phi }_X^{\prime }(1)& =\lambda e^0=\lambda \\ {\Phi }_X^{\prime \prime }(1)& ={\lambda }^2{e}^0={\lambda }^2\end{array}$$

Step 3: Mean and Variance.

$$\begin{array}{rl}EX& =\lambda \\ \mathrm{Var}X& ={\lambda }^2+\lambda -{\lambda }^2=\lambda \end{array}$$

---

4.4 Variance of a Sum

Let $X$ and $Y$ be two random variables each having finite second moment. Then $X+Y$ has finite second moment and hence finite variance.

$$\begin{array}{rl}\mathrm{Var}(X+Y)& =E[(X+Y)-E(X+Y){]}^2\\ & =E[(X-EX)+(Y-EY){]}^2\\ & =E[(X-EX{)}^2+(Y-EY{)}^2+2(X-EX)(Y-EY)]\\ & =E(X-EX{)}^2+E(Y-EY{)}^2+2E[(X-EX)(Y-EY)]\\ & =\mathrm{Var}X+\mathrm{Var}Y+2E[(X-EX)(Y-EY)]\end{array}$$

The quantity $E[(X-EX)(Y-EY)]$ is called the covariance of $X$ and $Y$ and written $\mathrm{Cov}(X,Y)$. Thus we have the important formula:

$$\mathrm{Var}(X+Y)=\mathrm{Var}X+\mathrm{Var}Y+2\mathrm{Cov}(X,Y).\text{\hspace{1em}(16)}$$

Now:

$$\begin{array}{rl}(X-EX)(Y-EY)& =XY-X(EY)-(EX)Y+(EX)(EY)\end{array}$$

Taking expectations:

$$\begin{array}{rl}\mathrm{Cov}(X,Y)& =E(XY)-(EX)(EY)-(EX)(EY)+(EX)(EY)\\ & =E(XY)-(EX)(EY).\end{array}\text{\hspace{1em}(17)}$$

From this form, it is clear that $\mathrm{Cov}(X,Y)=0$ whenever $X$ and $Y$ are independent. We see from (16) that if $X$ and $Y$ are independent random variables having finite second moments, then $\mathrm{Var}(X+Y)=\mathrm{Var}X+\mathrm{Var}Y$.

In particular if $P(Y=c)=1$ for some constant $c$, then $X$ and $Y$ are independent and the variance of $Y$ equals zero; consequently:

$$\mathrm{Var}(X+c)=\mathrm{Var}X+\mathrm{Var}(c)=\mathrm{Var}X.\text{\hspace{1em}(18)}$$

More generally, if $X_1,X_2,\dots ,X_n$ are $n$ random variables each having a finite second moment, then:

$$\mathrm{Var}\left(\sum _{i=1}^n{X}_i\right)=\sum _{i=1}^n\mathrm{Var}{X}_i+2\sum _{i=1}^{n-1}\sum _{j=i+1}^n\mathrm{Cov}(X_i,X_j).\text{\hspace{1em}(19)}$$

And, in particular, if $X_1,\dots ,X_n$ are mutually independent, then:

$$\mathrm{Var}\left(\sum _{i=1}^n{X}_i\right)=\sum _{i=1}^n\mathrm{Var}{X}_i.\text{\hspace{1em}(20)}$$

In particular, if $X_1,\dots ,X_n$ are independent random variables having a common variance ${\sigma }^2$, then:

$$\mathrm{Var}(X_1+\cdots +X_n)=n\mathrm{Var}{X}_1=n{\sigma }^2.\text{\hspace{1em}(21)}$$

Example 11: Binomial Distribution Variance

Problem: Let $X_1,\dots ,X_n$ be $n$ independent Bernoulli random variables each having probability $p$. Let $S_n=\sum X_i$. Find $\mathrm{Var}{S}_n$.

Solution: Using (21):

$$\mathrm{Var}{S}_n=n\mathrm{Var}{X}_1.$$

Now $X_1^2=X_1$ because $X_1$ is either 0 or 1.

$$\begin{array}{rl}E{X}_1^2& =E{X}_1=p\\ \mathrm{Var}{X}_1& =E{X}_1^2-(E{X}_1{)}^2\\ & =p-p^2\\ & =p(1-p)\end{array}$$

Consequently $\mathrm{Var}{S}_n=np(1-p)$.

Example 12: Hypergeometric Distribution Variance

Problem: Compute $\mathrm{Var}{S}_n$ for the hypergeometric distribution (sampling without replacement from Example 7).

Solution: We use Equation (19). Step 1: Variance of Indicators. For the dependent indicators $X_1,\dots ,X_n$:

$$E{X}_i=\frac{r_1}{r}.$$

Since $X_i^2=X_i$:

$$\begin{array}{rl}\mathrm{Var}{X}_i& =\frac{r_1}{r}-{\left(\frac{r_1}{r}\right)}^2\\ & =\frac{r_1}{r}\left(1-\frac{r_1}{r}\right).\end{array}$$

Step 2: Covariance. Assume $1\le i<j\le n$.

$$\begin{array}{rl}E{X}_i{X}_j& =P(X_i=1,X_j=1)\\ & =\frac{r_1}{r}\cdot \frac{r_1-1}{r-1}\end{array}$$

Thus:

$$\begin{array}{rl}\mathrm{Cov}(X_i,X_j)& =E(X_i{X}_j)-(E{X}_i)(E{X}_j)\\ & =\frac{r_1(r_1-1)}{r(r-1)}-{\left(\frac{r_1}{r}\right)}^2\\ & =\frac{r_1}{r}\left(\frac{r_1-1}{r-1}-\frac{r_1}{r}\right)\\ & =\frac{r_1}{r}\left(\frac{r(r_1-1)-r_1(r-1)}{r(r-1)}\right)\\ & =\frac{r_1}{r}\left(\frac{r{r}_1-r-r_{1}r+r_1}{r(r-1)}\right)\\ & =\frac{r_1}{r}\cdot \frac{r_1-r}{r(r-1)}\\ & =-\frac{r_1}{r}\cdot \frac{r-r_1}{r(r-1)}\end{array}$$

Step 3: Sum of Covariances. There are $\left(\genfrac{}{}{0ex}{}{n}{2}\right)=\frac{n(n-1)}{2}$ pairs.

$$\sum _{i<j}\mathrm{Cov}(X_i,X_j)=\frac{n(n-1)}{2}\left(-\frac{r_1(r-r_1)}{r^2(r-1)}\right).$$

Step 4: Total Variance.

$$\begin{array}{rl}\mathrm{Var}{S}_n& =\sum \mathrm{Var}{X}_i+2\sum \mathrm{Cov}(X_i,X_j)\\ & =n\frac{r_1(r-r_1)}{r^2}-n(n-1)\frac{r_1(r-r_1)}{r^2(r-1)}\\ & =n\frac{r_1}{r}\left(1-\frac{r_1}{r}\right)\left[1-\frac{n-1}{r-1}\right]\end{array}$$

---

4.5 Correlation Coefficient

Let $X$ and $Y$ be two random variables having finite nonzero variances. One measure of the degree of dependence between the two random variables is the correlation coefficient $\rho (X,Y)$ defined by:

$$\rho =\rho (X,Y)=\frac{\mathrm{Cov}(X,Y)}{\sqrt{(\mathrm{Var}X)(\mathrm{Var}Y)}}.$$

These random variables are said to be uncorrelated if $\rho =0$.

Theorem 7 (Schwarz Inequality): Let $X$ and $Y$ have finite second moments. Then:

$$[E(XY){]}^2\le (E{X}^2)(E{Y}^2).\text{\hspace{1em}(23)}$$

Furthermore, equality holds in (23) if and only if either $P(Y=0)=1$ or $P(X=aY)=1$ for some constant $a$.

Proof: Observe that for any real number $\lambda$:

$$\begin{array}{rl}0& \le E(X-\lambda Y{)}^2\\ & =E(X^2-2\lambda XY+{\lambda }^2{Y}^2)\\ & ={\lambda }^{2}E{Y}^2-2\lambda EXY+E{X}^2.\end{array}$$

This is a quadratic function of $\lambda$. Since the coefficient $E{Y}^2$ of ${\lambda }^2$ is positive, the minimum is achieved for $\lambda =a=[E(XY)][E{Y}^2{]}^{-1}$. Since the corresponding function value is non-negative:

$$\begin{array}{rl}0& \le E(X-aY{)}^2\\ & =E{X}^2-\frac{[E(XY){]}^2}{E{Y}^2}\end{array}$$

It follows that $[E(XY){]}^2\le (E{X}^2)(E{Y}^2)$. $\square$

Applying the Schwarz inequality to the random variables $(X-EX)$ and $(Y-EY)$:

$$\begin{array}{rl}[\mathrm{Cov}(X,Y){]}^2& \le (\mathrm{Var}X)(\mathrm{Var}Y)\\ ⟹|\rho (X,Y)|& \le 1.\end{array}$$

---

4.6 Chebyshev’s Inequality

Let $X$ be a nonnegative random variable having finite expectation, and let $t$ be a positive real number. Define $Y$ by setting $Y=0$ if $X<t$ and $Y=t$ if $X\ge t$.

$$\begin{array}{rl}EY& =tP(Y=t)+0\cdot P(Y=0)\\ & =tP(X\ge t).\end{array}$$

Since $X\ge Y$, $EX\ge EY$. Thus:

$$P(X\ge t)\le \frac{EX}{t}.\text{\hspace{1em}(25)}$$

Chebyshev’s Inequality: Let $X$ be a random variable with mean $\mu$ and finite variance ${\sigma }^2$. Then for any real number $t>0$:

$$P(|X-\mu |\ge t)\le \frac{{\sigma }^2}{t^2}.\text{\hspace{1em}(26)}$$

Proof: Apply (25) to the nonnegative random variable $(X-\mu {)}^2$ and the number $t^2$.

$$\begin{array}{rl}P((X-\mu {)}^2\ge t^2)& \le \frac{E(X-\mu {)}^2}{t^2}\\ & =\frac{{\sigma }^2}{t^2}.\end{array}$$

Since $(X-\mu {)}^2\ge t^2⟺|X-\mu |\ge t$, (26) holds. $\square$

Weak Law of Large Numbers: Let $X_1,\dots ,X_n$ be independent random variables having a common distribution with finite mean $\mu$ and set $S_n=X_1+\cdots +X_n$. Then for any $\delta >0$:

$$\underset{n\to \infty }{\lim }P\left(\left|\frac{S_n}{n}-\mu \right|\ge \delta \right)=0.$$

If the $X_i$ also have finite variance ${\sigma }^2$:

$$\mathrm{Var}\left(\frac{S_n}{n}\right)=\frac{{\sigma }^2}{n}.$$

Applying Chebyshev’s inequality to $S_n/n$:

$$P\left(\left|\frac{S_n}{n}-\mu \right|\ge \delta \right)\le \frac{{\sigma }^2}{n{\delta }^2}.\text{\hspace{1em}(27)}$$

Application to Binomial: Let $X_i$ be Bernoulli($p$). Then $\mu =p$ and ${\sigma }^2=p(1-p)$.

$$P\left(\left|\frac{S_n}{n}-p\right|\ge \delta \right)\le \frac{p(1-p)}{n{\delta }^2}.\text{\hspace{1em}(29)}$$

Since $p(1-p)\le 1/4$:

$$P\left(\left|\frac{S_n}{n}-p\right|\ge \delta \right)\le \frac{1}{4n{\delta }^2}.\text{\hspace{1em}(30)}$$