stat6

Chapter 6: Distributions of Sampling Statistics

6.1 Introduction

The science of statistics deals with drawing conclusions from observed data. For instance, a typical situation in a technological study arises when one is confronted with a large collection, or population, of items that have measurable values associated with them. By suitably sampling from this collection, and then analyzing the sampled items, one hopes to be able to draw some conclusions about the collection as a whole.

To use sample data to make inferences about an entire population, it is necessary to make some assumptions about the relationship between the two. One such assumption, which is often quite reasonable, is that there is an underlying (population) probability distribution such that the measurable values of the items in the population can be thought of as being independent random variables having this distribution. If the sample data are then chosen in a random fashion, then it is reasonable to suppose that they too are independent values from the distribution.

Definition

If $X_1,\dots ,X_n$ are independent random variables having a common distribution $F$, then we say that they constitute a sample (sometimes called a random sample) from the distribution $F$.

In most applications, the population distribution $F$ will not be completely specified and one will attempt to use the data to make inferences about $F$. Sometimes it will be supposed that $F$ is specified up to some unknown parameters (for instance, one might suppose that $F$ was a normal distribution function having an unknown mean and variance, or that it is a Poisson distribution function whose mean is not given), and at other times it might be assumed that almost nothing is known about $F$ (except maybe for assuming that it is a continuous, or a discrete, distribution). Problems in which the form of the underlying distribution is specified up to a set of unknown parameters are called parametric inference problems, whereas those in which nothing is assumed about the form of $F$ are called nonparametric inference problems.

EXAMPLE 6.1a
Suppose that a new process has just been installed to produce computer chips, and suppose that the successive chips produced by this new process will have useful lifetimes that are independent with a common unknown distribution $F$. Physical reasons sometimes suggest the parametric form of the distribution $F$; for instance, it may lead us to believe that $F$ is a normal distribution, or that $F$ is an exponential distribution. In such cases, we are confronted with a parametric statistical problem in which we would want to use the observed data to estimate the parameters of $F$. For instance, if $F$ were assumed to be a normal distribution, then we would want to estimate its mean and variance; if $F$ were assumed to be exponential, we would want to estimate its mean. In other situations, there might not be any physical justification for supposing that $F$ has any particular form; in this case the problem of making inferences about $F$ would constitute a nonparametric inference problem.

In this chapter, we will be concerned with the probability distributions of certain statistics that arise from a sample, where a statistic is a random variable whose value is determined by the sample data. Two important statistics that we will discuss are the sample mean and the sample variance. In Section 6.2, we consider the sample mean and derive its expectation and variance. We note that when the sample size is at least moderately large, the distribution of the sample mean is approximately normal. This follows from the central limit theorem, one of the most important theoretical results in probability, which is discussed in Section 6.3. In Section 6.4, we introduce the sample variance and determine its expected value. In Section 6.5, we suppose that the population distribution is normal and present the joint distribution of the sample mean and the sample variance. In Section 6.6, we suppose that we are sampling from a finite population of elements and explain what it means for the sample to be a “random sample.” When the population size is large in relation to the sample size, we often treat it as if it were of infinite size; this is illustrated and its consequences are discussed.

6.2 The Sample Mean

Consider a population of elements, each of which has a numerical value attached to it. For instance, the population might consist of the adults of a specified community and the value attached to each adult might be his or her annual income, or height, or age, and so on. We often suppose that the value associated with any member of the population can be regarded as being the value of a random variable having expectation $\mu$ and variance ${\sigma }^2$. The quantities $\mu$ and ${\sigma }^2$ are called the population mean and the population variance, respectively. Let $X_1,X_2,\dots ,X_n$ be a sample of values from this population. The sample mean is defined by

$$\overline{X}=\frac{X_1+\cdots +X_n}{n}$$

Since the value of the sample mean $\overline{X}$ is determined by the values of the random variables in the sample, it follows that $\overline{X}$ is also a random variable. Its expected value and variance are obtained as follows:

$$E[\overline{X}]=E\left[\frac{X_1+\cdots +X_n}{n}\right]=\frac{1}{n}(E[X_1]+\cdots +E[X_n])=\mu$$

and

$$\mathrm{Var}(\overline{X})=\mathrm{Var}\left(\frac{X_1+\cdots +X_n}{n}\right)=\frac{1}{n^2}[\mathrm{Var}(X_1)+\cdots +\mathrm{Var}(X_n)]\text{(by independence)}=\frac{n{\sigma }^2}{n^2}=\frac{{\sigma }^2}{n}$$

where $\mu$ and ${\sigma }^2$ are the population mean and variance, respectively. Hence, the expected value of the sample mean is the population mean $\mu$ whereas its variance is $1/n$ times the population variance. As a result, we can conclude that $\overline{X}$ is also centered about the population mean $\mu$, but its spread becomes more and more reduced as the sample size increases. Figure 6.1 plots the probability density function of the sample mean from a standard normal population for a variety of sample sizes.

Figure 6.1: Densities of sample means from a standard normal population. (Figure not reproduced)

6.3 The Central Limit Theorem

In this section, we will consider one of the most remarkable results in probability — namely, the central limit theorem. Loosely speaking, this theorem asserts that the sum of a large number of independent random variables has a distribution that is approximately normal. Hence, it not only provides a simple method for computing approximate probabilities for sums of independent random variables, but it also helps explain the remarkable fact that the empirical frequencies of so many natural populations exhibit a bell-shaped (that is, a normal) curve.

In its simplest form, the central limit theorem is as follows:

Theorem 6.3.1 (The Central Limit Theorem)

Let $X_1,X_2,\dots ,X_n$ be a sequence of independent and identically distributed random variables each having mean $\mu$ and variance ${\sigma }^2$. Then for $n$ large, the distribution of

$$X_1+\cdots +X_n$$

is approximately normal with mean $n\mu$ and variance $n{\sigma }^2$.

It follows from the central limit theorem that

$$\frac{X_1+\cdots +X_n-n\mu }{\sigma \sqrt{n}}$$

is approximately a standard normal random variable; thus, for $n$ large,

$$P\left\{\frac{X_1+\cdots +X_n-n\mu }{\sigma \sqrt{n}}<x\right\}\approx P[Z<x]$$

where $Z$ is a standard normal random variable.

EXAMPLE 6.3a
An insurance company has 25,000 automobile policy holders. If the yearly claim of a policy holder is a random variable with mean 320 and standard deviation 540, approximate the probability that the total yearly claim exceeds 8.3 million.

Solution
Let $X$ denote the total yearly claim. Number the policy holders, and let $X_i$ denote the yearly claim of policy holder $i$. With $n=25,000$, we have from the central limit theorem that $X={\sum }_{i=1}^n{X}_i$ will have approximately a normal distribution with mean $320\times 25,000=8\times 10^6$ and standard deviation $540\sqrt{25,000}=8.5381\times 10^4$. Therefore,

$$P\{X>8.3\times 10^6\}\approx P\left\{Z>\frac{8.3\times 10^6-8\times 10^6}{8.5381\times 10^4}\right\}=P\{Z>3.51\}\approx .00023$$

Thus, there are only 2.3 chances out of 10,000 that the total yearly claim will exceed 8.3 million.

EXAMPLE 6.3b
Civil engineers believe that $W$, the amount of weight (in units of 1,000 pounds) that a certain span of a bridge can withstand without structural damage resulting, is normally distributed with mean 400 and standard deviation 40. Suppose that the weight (again, in units of 1,000 pounds) of a car is a random variable with mean 3 and standard deviation 0.3. How many cars would have to be on the bridge span for the probability of structural damage to exceed 0.1?

Solution
Let $P_n$ denote the probability of structural damage when there are $n$ cars on the bridge. That is,

$$P_n=P\{X_1+\cdots +X_n\ge W\}=P\{X_1+\cdots +X_n-W\ge 0\}$$

where $X_i$ is the weight of the $i$th car. Now it follows from the central limit theorem that ${\sum }_{i=1}^n{X}_i$ is approximately normal with mean $3n$ and variance $0.09n$. Hence, since $W$ is independent of the $X_i$ and is also normal, it follows that ${\sum }_{i=1}^n{X}_i-W$ is approximately normal, with mean and variance given by

$$E\left[\sum _1^n{X}_i-W\right]=3n-400,\mathrm{Var}\left(\sum _1^n{X}_i-W\right)=0.09n+1600.$$

Thus,

$$P_n\approx P\left\{Z\ge \frac{400-3n}{\sqrt{0.09n+1600}}\right\}$$

where $Z$ is standard normal. Now $P\{Z\ge 1.28\}\approx 0.1$, and so if

$$\frac{400-3n}{\sqrt{0.09n+1600}}\le 1.28,$$

then there is at least 1 chance in 10 that structural damage will occur. Solving gives $n\ge 117$.

The central limit theorem is illustrated by Program 6.1 on the text disk. This program plots the probability mass function of the sum of $n$ independent and identically distributed random variables that each take on one of the values 0, 1, 2, 3, 4. Figures 6.2(a)–(f) give the resulting plot for a specified set of probabilities when $n=1,3,5,10,25,100$.

Figure 6.2: (a) $n=1$, (b) $n=3$, (c) $n=5$, (d) $n=10$, (e) $n=25$, (f) $n=100$. (Figures not reproduced)

One of the most important applications of the central limit theorem is in regard to binomial random variables. Since such a random variable $X$ having parameters $(n,p)$ represents the number of successes in $n$ independent trials when each trial is a success with probability $p$, we can express it as

$$X=X_1+\cdots +X_n$$

where $X_i$ are independent Bernoulli($p$) random variables. Because $E[X_i]=p$ and $\mathrm{Var}(X_i)=p(1-p)$, it follows from the central limit theorem that for $n$ large

$$\frac{X-np}{\sqrt{np(1-p)}}$$

will approximately be a standard normal random variable. (See Figure 6.3, which graphically illustrates how the probability mass function of a binomial $(n,p)$ random variable becomes more and more “normal” as $n$ becomes larger and larger.)

Figure 6.3: Binomial probability mass functions converging to the normal density. (Figures not reproduced)

EXAMPLE 6.3c
The ideal size of a first-year class at a particular college is 150 students. The college, knowing from past experience that, on the average, only 30 percent of those accepted for admission will actually attend, uses a policy of approving the applications of 450 students. Compute the probability that more than 150 first-year students attend this college.

Solution
Let $X$ denote the number of students that attend; then assuming that each accepted applicant will independently attend, $X$ is binomial with parameters $n=450$ and $p=0.3$. Using the continuity correction,

$$P\{X>150.5\}=P\left\{\frac{X-450(0.3)}{\sqrt{450(0.3)(0.7)}}\ge \frac{150.5-135}{\sqrt{94.5}}\right\}\approx P\{Z>1.59\}=\mathrm{0.06.}$$

Hence, only 6 percent of the time do more than 150 of the first 450 accepted actually attend.

6.3.1 Approximate Distribution of the Sample Mean

Let $X_1,\dots ,X_n$ be a sample from a population having mean $\mu$ and variance ${\sigma }^2$. The central limit theorem can be used to approximate the distribution of the sample mean $\overline{X}={\sum }_{i=1}^n{X}_i/n$. Since a constant multiple of a normal random variable is also normal, $\overline{X}$ will be approximately normal when $n$ is large. Since the sample mean has expected value $\mu$ and standard deviation $\sigma /\sqrt{n}$, it follows that

$$\frac{\overline{X}-\mu }{\sigma /\sqrt{n}}$$

has approximately a standard normal distribution.

EXAMPLE 6.3d
The weights of a population of workers have mean 167 and standard deviation 27.
(a) If a sample of 36 workers is chosen, approximate the probability that the sample mean lies between 163 and 170.
(b) Repeat part (a) when the sample size is 144.

Solution
(a) $\overline{X}$ is approximately normal with mean 167 and standard deviation $27/\sqrt{36}=4.5$. Then

$$P\{163<\overline{X}<170\}\approx P\{-0.8889<Z<0.8889\}=2P\{Z<0.8889\}-1\approx \mathrm{0.6259.}$$

(b) For $n=144$, standard deviation is $27/\sqrt{144}=2.25$. Then

$$P\{163<\overline{X}<170\}\approx P\{-1.7778<Z<1.7778\}\approx \mathrm{0.9246.}$$

EXAMPLE 6.3e
An astronomer wants to measure the distance from her observatory to a distant star. Due to atmospheric disturbances, any measurement will not yield the exact distance $d$. She will make a series of measurements and use their average as an estimate. If the measurements are independent with mean $d$ and standard deviation 2 light years, how many measurements are needed to be at least 95% certain that her estimate is accurate to within $\pm 0.5$ light years?

Solution
With $n$ measurements, $\overline{X}$ is approximately normal with mean $d$ and standard deviation $2/\sqrt{n}$. We require

$$P\{-0.5<\overline{X}-d<0.5\}\approx 2P\left\{Z<\frac{0.5}{2/\sqrt{n}}\right\}-1=2P\{Z<\sqrt{n}/4\}-1\ge \mathrm{0.95.}$$

Thus $P\{Z<\sqrt{n}/4\}\ge 0.975$, so $\sqrt{n}/4\ge 1.96$, i.e., $n\ge 61.47$. Hence at least 62 observations are necessary.

6.3.2 How Large a Sample Is Needed?

The central limit theorem leaves open the question of how large $n$ needs to be for the normal approximation to be valid. A general rule of thumb is that one can be confident of the normal approximation whenever $n\ge 30$. In many cases, smaller samples (e.g., $n=5$) suffice. Figure 6.4 presents the distribution of sample means from an exponential population for $n=1,5,10$.

Figure 6.4: Densities of the average of $n$ exponential random variables having mean 1. (Figure not reproduced)

6.4 The Sample Variance

Let $X_1,\dots ,X_n$ be a random sample from a distribution with mean $\mu$ and variance ${\sigma }^2$. Let $\overline{X}$ be the sample mean.

Definition

The statistic $S^2$, defined by

$$S^2=\frac{{\sum }_{i=1}^n(X_i-\overline{X}{)}^2}{n-1}$$

is called the sample variance. $S=\sqrt{S^2}$ is called the sample standard deviation.

Using the identity ${\sum }_{i=1}^n(x_i-\overline{x}{)}^2={\sum }_{i=1}^n{x}_i^2-n{\overline{x}}^2$, we obtain

$$(n-1)S^2=\sum _{i=1}^n{X}_i^2-n{\overline{X}}^2.$$

Taking expectations and using $E[W^2]=\mathrm{Var}(W)+(E[W]{)}^2$:

$$(n-1)E[S^2]=nE[X_1^2]-nE[{\overline{X}}^2]=n({\sigma }^2+{\mu }^2)-n\left(\frac{{\sigma }^2}{n}+{\mu }^2\right)=(n-1){\sigma }^2.$$

Hence $E[S^2]={\sigma }^2$. That is, the expected value of the sample variance equals the population variance.

6.5 Sampling Distributions from a Normal Population

Let $X_1,X_2,\dots ,X_n$ be a sample from a normal population having mean $\mu$ and variance ${\sigma }^2$; i.e., they are independent and $X_i\sim \mathcal{N}(\mu ,{\sigma }^2)$. Let $\overline{X}$ and $S^2$ denote the sample mean and sample variance.

6.5.1 Distribution of the Sample Mean

Since the sum of independent normals is normal, $\overline{X}$ is normal with mean $\mu$ and variance ${\sigma }^2/n$. Thus

$$\frac{\overline{X}-\mu }{\sigma /\sqrt{n}}$$

is a standard normal random variable.

6.5.2 Joint Distribution of $\overline{X}$ and $S^2$

We have the fundamental result:

Theorem 6.5.1

If $X_1,\dots ,X_n$ is a sample from a normal population with mean $\mu$ and variance ${\sigma }^2$, then $\overline{X}$ and $S^2$ are independent random variables, with $\overline{X}$ being normal with mean $\mu$ and variance ${\sigma }^2/n$, and $(n-1)S^2/{\sigma }^2$ being chi-square with $n-1$ degrees of freedom.

EXAMPLE 6.5a
The time it takes a CPU to process a certain job is normally distributed with mean 20 seconds and standard deviation 3 seconds. If a sample of 15 jobs is observed, what is the probability that the sample variance exceeds 12?

Solution
Here $n=15$, ${\sigma }^2=9$. Then

$$P\{S^2>12\}=P\left\{\frac{14S^2}{9}>\frac{14}{9}\cdot 12\right\}=P\{{\chi }_{14}^2>18.67\}=1-0.8221=\mathrm{0.1779.}$$

Corollary 6.5.2

Let $X_1,\dots ,X_n$ be a sample from a normal population with mean $\mu$. Then

$$\sqrt{n}\frac{\overline{X}-\mu }{S}\sim t_{n-1}$$

(i.e., a $t$-distribution with $n-1$ degrees of freedom).

Proof
A $t$-random variable with $n$ degrees of freedom is defined as $Z/\sqrt{{\chi }_n^2/n}$ where $Z$ is standard normal independent of ${\chi }_n^2$. From Theorem 6.5.1, $\sqrt{n}(\overline{X}-\mu )/\sigma$ is standard normal and independent of $(n-1)S^2/{\sigma }^2\sim {\chi }_{n-1}^2$. Hence

$$\frac{\sqrt{n}(\overline{X}-\mu )/\sigma }{\sqrt{S^2/{\sigma }^2}}=\sqrt{n}\frac{\overline{X}-\mu }{S}$$

follows a $t$-distribution with $n-1$ degrees of freedom.

6.6 Sampling from a Finite Population

Consider a population of $N$ elements, and suppose that $p$ is the proportion of the population that has a certain characteristic (i.e., $Np$ elements have the characteristic). A random sample of size $n$ is chosen such that each of the $\left(\genfrac{}{}{0ex}{}{N}{n}\right)$ subsets is equally likely.

Define for $i=1,\dots ,n$:

$$X_i=\{\begin{array}{ll}1& \text{if the ith member of the sample has the characteristic}\\ 0& \text{otherwise}\end{array}$$

Then $X={\sum }_{i=1}^n{X}_i$ is the number of sample members with the characteristic, and $\overline{X}=X/n$ is the sample proportion.

Because each member of the population is equally likely to be the $i$th sample member, $P\{X_i=1\}=p$. However, the $X_i$ are not independent (sampling without replacement). But when the population size $N$ is large relative to $n$, the dependence is weak, and the $X_i$ are approximately independent. In that case, $X$ is approximately binomial with parameters $n$ and $p$. (In exact terms, $X$ is hypergeometric.)

For large $N$ relative to $n$, we use the binomial approximation:

$$E[X]=np,\mathrm{SD}(X)=\sqrt{np(1-p)},E[\overline{X}]=p,\mathrm{Var}(\overline{X})=\frac{p(1-p)}{n}.$$

EXAMPLE 6.6a
Suppose that 45% of the population favors a certain candidate. If a random sample of size 200 is chosen, find
(a) the expected value and standard deviation of the number who favor the candidate;
(b) the probability that more than half the sample favors the candidate.

Solution
(a) $E[X]=200(0.45)=90$, $\mathrm{SD}(X)=\sqrt{200(0.45)(0.55)}\approx 7.0356$.
(b) Using the normal approximation with continuity correction:

$$P\{X\ge 101\}=P\{X\ge 100.5\}\approx P\left\{Z\ge \frac{100.5-90}{7.0356}\right\}=P\{Z\ge 1.4924\}\approx \mathrm{0.0678.}$$

EXAMPLE 6.6b
According to the U.S. Department of Agriculture, the per capita consumption of pork in Denmark in 2013 had a mean of 147 pounds and standard deviation 62 pounds. If a random sample of 25 Danes is chosen, approximate the probability that the average amount consumed exceeded 150 pounds.

Solution
By the central limit theorem, $\overline{X}$ is approximately normal with mean 147 and standard deviation $62/\sqrt{25}=12.4$. Then

$$P\{\overline{X}>150\}\approx P\left\{Z>\frac{150-147}{12.4}\right\}=P\{Z>0.2419\}\approx \mathrm{0.404.}$$

Welcome, curious minds! Today, we’re embarking on a fascinating journey into the heart of statistics, exploring how we can peek into the secrets of a vast population by carefully examining a small, thoughtfully chosen sample. Imagine trying to understand the full grandeur of a forest by studying just a handful of its leaves—it seems daunting, doesn’t it? Yet, with the right tools and understanding, we can make incredibly accurate inferences!

Our adventure begins by defining what a “sample” truly means in the world of statistics and how it connects to the hidden patterns of a population. We’ll uncover the power of the sample mean and sample variance, like discovering the average height of trees or how much their heights vary, just from our sample of leaves. And then, we’ll encounter a real gem: the Central Limit Theorem, a probabilistic superpower that makes many complex situations beautifully simple.

Are you ready to transform abstract data into tangible insights? Let’s dive in!

---

Chapter 6: Distributions of Sampling Statistics

6.1 Introduction

The science of statistics is fundamentally about drawing meaningful conclusions from observed data. Picture a vast collection, or population, of items, each with a measurable value attached—perhaps the lifespan of computer chips, the income of individuals, or the yield of different crop varieties. Our goal is to understand this entire collection by judiciously selecting and analyzing a smaller subset, a sample.

To bridge the gap between our sample and the vast population, we often make a crucial assumption: that there’s an underlying population probability distribution (let’s call it $F$) from which all the measurable values in the population are drawn independently. If our sample is chosen in a truly random fashion, then the values we observe in our sample can also be considered independent observations from this same distribution $F$.

Definition: Sample

If $X_1,\dots ,X_n$ are independent random variables having a common distribution $F$, we say that they constitute a sample (sometimes called a random sample) from the distribution $F$.

In most real-world scenarios, this population distribution $F$ isn’t completely known. Our task then becomes using the sample data to make educated guesses or inferences about $F$. Sometimes, we might have a hunch about the form of $F$ but not its specific details—for instance, we might suspect $F$ is a normal distribution but don’t know its mean or variance. These are called parametric inference problems, as we’re trying to estimate unknown parameters of a known distribution type. Other times, we might know almost nothing about $F$, perhaps only that it’s continuous or discrete. These more open-ended investigations are called nonparametric inference problems.

EXAMPLE 6.1a Imagine a brand-new process designed to produce computer chips. The useful lifetimes of these chips, we assume, are independent and follow some unknown distribution $F$. Based on our understanding of physics and engineering, we might have reasons to believe that $F$ is, for example, a normal distribution or an exponential distribution. If we assume it’s a normal distribution, our goal becomes estimating its mean and variance from the observed chip lifetimes. If we assume it’s exponential, we’d aim to estimate its mean. This is the essence of a parametric statistical problem. However, if there’s no strong physical basis to assume a particular form for $F$, then making inferences about it becomes a nonparametric challenge, requiring different statistical approaches.

In this chapter, we’ll focus on the probability distributions of specific statistics derived from a sample. A statistic, in simple terms, is a random variable whose value is entirely determined by the sample data. We’ll pay special attention to two fundamental statistics: the sample mean and the sample variance. We’ll explore their expected values and variances, discover how the sample mean behaves for large samples (thanks to the Central Limit Theorem!), and delve into their joint distribution when sampling from a normal population. Finally, we’ll touch upon the nuances of sampling from finite populations.

6.2 The Sample Mean

Let’s consider a population where each element has a numerical value—think of the annual income of adults in a city, their height, or their age. We often model these values as observations from a random variable with a true, but often unknown, population mean $\mu$ and population variance ${\sigma }^2$.

Now, suppose we draw a sample of $n$ values from this population: $X_1,X_2,\dots ,X_n$. The sample mean, denoted $\overline{X}$, is simply the average of these observed values:

$$\overline{X}=\frac{X_1+X_2+\cdots +X_n}{n}$$

Since each $X_i$ is a random variable, their average, $\overline{X}$, is also a random variable. What can we say about its own expected value and variance? Let’s calculate:

The expected value of the sample mean is:

$$\begin{array}{rlrl}E[\overline{X}]& =E\left[\frac{X_1+\cdots +X_n}{n}\right]& & \\ & =\frac{1}{n}E\left[X_1+\cdots +X_n\right]& & \text{(By linearity of expectation)}\\ & =\frac{1}{n}\left(E[X_1]+\cdots +E[X_n]\right)& & \text{(By linearity of expectation)}\\ & =\frac{1}{n}(\mu +\cdots +\mu )& & \text{(Since all }{X}_i\text{ have population mean }\mu \text{)}\\ & =\frac{1}{n}(n\mu )& & \\ & =\mu & & \end{array}$$

This tells us something profound: the expected value of our sample mean is exactly the population mean! In other words, on average, our sample mean will hit the target $\mu$.

Now, let’s look at its variance:

$$\begin{array}{rlrl}\mathrm{Var}(\overline{X})& =\mathrm{Var}\left(\frac{X_1+\cdots +X_n}{n}\right)& & \\ & ={\left(\frac{1}{n}\right)}^2\mathrm{Var}(X_1+\cdots +X_n)& & \text{(Using }\mathrm{Var}(cX)=c^2\mathrm{Var}(X)\text{)}\\ & =\frac{1}{n^2}(\mathrm{Var}(X_1)+\cdots +\mathrm{Var}(X_n))& & \text{(By independence of }{X}_i\text{)}\\ & =\frac{1}{n^2}({\sigma }^2+\cdots +{\sigma }^2)& & \text{(Since all }{X}_i\text{ have population variance }{\sigma }^2\text{)}\\ & =\frac{1}{n^2}(n{\sigma }^2)& & \\ & =\frac{{\sigma }^2}{n}& & \end{array}$$

So, the variance of the sample mean is the population variance divided by the sample size $n$. This is a critical insight! It means that as we increase our sample size $n$, the variance of $\overline{X}$ shrinks. This implies that the distribution of $\overline{X}$ becomes more concentrated around the true population mean $\mu$. Our estimate gets “tighter” and more precise as we gather more data. Figure 6.1 (not reproduced here) would graphically illustrate this, showing how the probability density function of the sample mean becomes taller and narrower as $n$ increases, even if the underlying population distribution is broad.

6.3 The Central Limit Theorem

Now, let’s delve into one of the most astonishing and powerful results in all of probability and statistics: the Central Limit Theorem (CLT). Imagine you’re taking many small, independent steps in random directions. The CLT says that even if each step is chaotic, the overall path you’ve taken, the sum of all those steps, will tend to follow a remarkably predictable, bell-shaped pattern—the normal distribution.

This theorem is a cornerstone for two main reasons:

1. It offers a straightforward way to approximate probabilities for sums of many independent random variables, even if their individual distributions are complex or unknown.
2. It elegantly explains why so many natural phenomena, from human heights to measurement errors, exhibit a normal distribution: they are often the result of many small, independent influences summing up.

Theorem 6.3.1 (The Central Limit Theorem)

Let $X_1,X_2,\dots ,X_n$ be a sequence of independent and identically distributed (i.i.d.) random variables, each with a mean $\mu$ and a finite variance ${\sigma }^2$. Then, for a sufficiently large $n$, the distribution of the sum $S_n=X_1+\cdots +X_n$ is approximately normal with mean $n\mu$ and variance $n{\sigma }^2$.

A more practical way to state this, which allows us to use standard normal tables, is that the standardized sum:

$$\frac{X_1+\cdots +X_n-n\mu }{\sigma \sqrt{n}}$$

is approximately a standard normal random variable. Therefore, for large $n$:

$$P\left\{\frac{X_1+\cdots +X_n-n\mu }{\sigma \sqrt{n}}<x\right\}\approx P[Z<x]$$

where $Z$ is a standard normal random variable (mean 0, variance 1).

EXAMPLE 6.3a An insurance company has 25,000 automobile policy holders. If the yearly claim of a policy holder is a random variable with a mean of 320 (dollars) and a standard deviation of 540 (dollars), approximate the probability that the total yearly claim exceeds 8.3 million dollars.

Solution Let $X_i$ denote the yearly claim of policy holder $i$. We are interested in the total yearly claim, $X={\sum }_{i=1}^n{X}_i$, where $n=25,000$. We are given:

• Population mean for each $X_i$: $\mu =320$
• Population standard deviation for each $X_i$: $\sigma =540$
• Sample size: $n=25,000$

1. Calculate the mean and variance of the total claim $X$: From the Central Limit Theorem, for large $n$, $X$ is approximately normal.

   $$\begin{array}{rl}E[X]& =n\mu =25,000\times 320=8,000,000\\ \mathrm{Var}(X)& =n{\sigma }^2=25,000\times (540{)}^2=25,000\times 291,600=7,290,000,000\\ \mathrm{SD}(X)& =\sqrt{n{\sigma }^2}=\sqrt{7,290,000,000}\approx 85,381.56\end{array}$$

   So, $X$ is approximately normal with mean $8\times 10^6$ and standard deviation $8.5381\times 10^4$.

2. Formulate the probability statement: We want to find $P\{X>8.3\times 10^6\}$.

3. Standardize $X$ to a standard normal variable $Z$:

   $$\begin{array}{rl}P\{X>8.3\times 10^6\}& \approx P\left\{Z>\frac{8.3\times 10^6-E[X]}{\mathrm{SD}(X)}\right\}\\ & =P\left\{Z>\frac{8.3\times 10^6-8\times 10^6}{8.5381\times 10^4}\right\}\\ & =P\left\{Z>\frac{3\times 10^5}{8.5381\times 10^4}\right\}\\ & =P\{Z>3.5136\}\end{array}$$

4. Look up the probability in a standard normal table:

   $$\begin{array}{rlrl}P\{Z>3.5136\}& =1-P\{Z\le 3.5136\}& & \\ & \approx 1-0.99977& & \text{(Using standard normal CDF tables)}\\ & \approx 0.00023& & \end{array}$$

   Thus, there are only about 2.3 chances out of 10,000 that the total yearly claim will exceed 8.3 million dollars.

EXAMPLE 6.3b Civil engineers estimate that $W$, the amount of weight (in units of 1,000 pounds) that a certain span of a bridge can withstand without structural damage, is normally distributed with a mean of 400 and a standard deviation of 40. Suppose the weight of a car (in units of 1,000 pounds) is a random variable with a mean of 3 and a standard deviation of 0.3. How many cars would need to be on the bridge span for the probability of structural damage to exceed 0.1?

Solution Let $X_i$ be the weight of the $i$-th car. We are given:

• For each car $X_i$: $E[X_i]=3$, $\mathrm{SD}(X_i)=0.3$, so $\mathrm{Var}(X_i)=(0.3{)}^2=0.09$.
• For the bridge’s capacity $W$: $W\sim \mathcal{N}(400,40^2)$, so $E[W]=400$, $\mathrm{Var}(W)=40^2=1600$.

Let $S_n={\sum }_{i=1}^n{X}_i$ be the total weight of $n$ cars. We want to find $n$ such that $P_n=P\{S_n\ge W\}>0.1$. This is equivalent to $P\{S_n-W\ge 0\}>0.1$.

1. Determine the distribution of $S_n$: By the Central Limit Theorem, for large $n$, $S_n$ is approximately normal.

   $$\begin{array}{rl}E[S_n]& =nE[X_i]=3n\\ \mathrm{Var}(S_n)& =n\mathrm{Var}(X_i)=0.09n\end{array}$$

2. Determine the distribution of $S_n-W$: Since $S_n$ is approximately normal and $W$ is normally distributed, and they are independent, their difference $S_n-W$ is also approximately normal.

   $$\begin{array}{rlrl}E[S_n-W]& =E[S_n]-E[W]=3n-400& & \\ \mathrm{Var}(S_n-W)& =\mathrm{Var}(S_n)+\mathrm{Var}(W)& & \text{(By independence)}\\ & =0.09n+1600& & \end{array}$$

3. Formulate the probability statement using standardization: Let $Y=S_n-W$. We want $P\{Y\ge 0\}>0.1$.

   $$\begin{array}{rl}P\{Y\ge 0\}& \approx P\left\{Z\ge \frac{0-E[Y]}{\mathrm{SD}(Y)}\right\}\\ & =P\left\{Z\ge \frac{0-(3n-400)}{\sqrt{0.09n+1600}}\right\}\\ & =P\left\{Z\ge \frac{400-3n}{\sqrt{0.09n+1600}}\right\}\end{array}$$

4. Find the critical Z-value: We need $P\{Z\ge \text{some value}\}>0.1$. Looking at a standard normal table, $P\{Z\ge 1.28\}\approx 0.1003$. So, for the probability to exceed 0.1, the Z-score must be less than or equal to 1.28.

   $$\frac{400-3n}{\sqrt{0.09n+1600}}\le 1.28$$

5. Solve for $n$: We need to solve the inequality. Let’s try to approximate or test values for $n$. The provided solution states $n\ge 117$. Let’s verify for $n=117$:

   $$\begin{array}{rl}\frac{400-3(117)}{\sqrt{0.09(117)+1600}}& =\frac{400-351}{\sqrt{10.53+1600}}\\ & =\frac{49}{\sqrt{1610.53}}\\ & =\frac{49}{40.1314}\approx 1.221\end{array}$$

   Since $1.221\le 1.28$, the probability $P\{Z\ge 1.221\}$ is approximately $1-0.8889=0.1111$, which is greater than $0.1$. For $n=116$:

   $$\begin{array}{rl}\frac{400-3(116)}{\sqrt{0.09(116)+1600}}& =\frac{400-348}{\sqrt{10.44+1600}}\\ & =\frac{52}{\sqrt{1610.44}}\\ & =\frac{52}{40.1303}\approx 1.2955\end{array}$$

   Since $1.2955>1.28$, the probability $P\{Z\ge 1.2955\}$ is approximately $1-0.9024=0.0976$, which is not greater than $0.1$. Thus, $n$ must be at least 117.

The Central Limit Theorem is beautifully illustrated by how distributions evolve. Program 6.1 (from the text disk) would plot the probability mass function of the sum of $n$ independent and identically distributed random variables. Figures 6.2(a)–(f) (not reproduced) would show this for various $n$ (e.g., $n=1,3,5,10,25,100$), visually demonstrating the sum’s distribution gradually taking on the characteristic bell shape of the normal distribution as $n$ increases.

One of the most important practical applications of the Central Limit Theorem is its relationship to binomial random variables. Recall that a binomial random variable $X$ with parameters $(n,p)$ represents the number of successes in $n$ independent Bernoulli trials, where each trial has a probability of success $p$. We can express $X$ as a sum:

$$X=X_1+\cdots +X_n$$

where each $X_i$ is an independent Bernoulli($p$) random variable. For a Bernoulli variable, $E[X_i]=p$ and $\mathrm{Var}(X_i)=p(1-p)$. Therefore, by the Central Limit Theorem, for large $n$, the standardized binomial random variable:

$$\frac{X-np}{\sqrt{np(1-p)}}$$

will be approximately a standard normal random variable. This approximation becomes more accurate as $n$ increases, especially when $p$ is not too close to 0 or 1. Figure 6.3 (not reproduced) would vividly show how the discrete probability mass function of a binomial variable smooths out and converges to the continuous normal density as $n$ grows.

EXAMPLE 6.3c The ideal size for a first-year class at a particular college is 150 students. Past experience shows that, on average, only 30 percent of those accepted for admission actually attend. The college, using this information, approves the applications of 450 students. Compute the probability that more than 150 first-year students attend this college.

Solution Let $X$ denote the number of students who attend. Assuming each accepted applicant makes an independent decision, $X$ follows a binomial distribution with parameters $n=450$ and $p=0.3$. We need to find $P\{X>150\}$. Since $X$ is a discrete variable and we’re using a continuous normal approximation, we apply the continuity correction. “More than 150” for a discrete variable means $X\ge 151$. With continuity correction, this translates to $X\ge 150.5$ for the continuous approximation.

1. Calculate the mean and standard deviation for $X$: For a binomial distribution, $E[X]=np$ and $\mathrm{Var}(X)=np(1-p)$.

   $$\begin{array}{rl}E[X]& =450\times 0.3=135\\ \mathrm{Var}(X)& =450\times 0.3\times (1-0.3)=450\times 0.3\times 0.7=94.5\\ \mathrm{SD}(X)& =\sqrt{94.5}\approx 9.7211\end{array}$$

2. Formulate the probability statement with continuity correction: We want $P\{X>150\}$. Using continuity correction, this is $P\{X\ge 150.5\}$.

3. Standardize $X$ to a standard normal variable $Z$:

   $$\begin{array}{rl}P\{X\ge 150.5\}& \approx P\left\{Z\ge \frac{150.5-E[X]}{\mathrm{SD}(X)}\right\}\\ & =P\left\{Z\ge \frac{150.5-135}{9.7211}\right\}\\ & =P\left\{Z\ge \frac{15.5}{9.7211}\right\}\\ & =P\{Z\ge 1.5944\}\end{array}$$

4. Look up the probability in a standard normal table:

   $$\begin{array}{rlrl}P\{Z\ge 1.5944\}& =1-P\{Z<1.5944\}& & \\ & \approx 1-0.9446& & \text{(Using standard normal CDF tables, for }Z=1.59\text{)}\\ & \approx 0.0554& & \end{array}$$

   The provided solution approximation $0.06$ is a slightly rounded value. So, only about 5.54% of the time (or approximately 6%) do more than 150 of the 450 accepted students actually attend. This highlights the risk of over-enrollment!

6.3.1 Approximate Distribution of the Sample Mean

The Central Limit Theorem isn’t just for sums; it’s equally powerful for the sample mean $\overline{X}$. Since $\overline{X}={\sum }_{i=1}^n{X}_i/n$, it’s simply a constant multiple of the sum. If the sum of many i.i.d. variables is approximately normal, then the average of those variables will also be approximately normal.

As we established in Section 6.2, $E[\overline{X}]=\mu$ and $\mathrm{Var}(\overline{X})={\sigma }^2/n$. Therefore, its standard deviation is $\mathrm{SD}(\overline{X})=\sigma /\sqrt{n}$. It follows that for large $n$, the standardized sample mean:

$$\frac{\overline{X}-\mu }{\sigma /\sqrt{n}}$$

has approximately a standard normal distribution. This allows us to make probability statements about the sample mean itself.

EXAMPLE 6.3d The weights of a population of workers have a mean of 167 pounds and a standard deviation of 27 pounds. (a) If a sample of 36 workers is chosen, approximate the probability that the sample mean lies between 163 and 170 pounds. (b) Repeat part (a) when the sample size is 144.

Solution We are given:

• Population mean: $\mu =167$
• Population standard deviation: $\sigma =27$

(a) For $n=36$:

1. Calculate the mean and standard deviation of the sample mean $\overline{X}$:

   $$\begin{array}{rl}E[\overline{X}]& =\mu =167\\ \mathrm{SD}(\overline{X})& =\frac{\sigma }{\sqrt{n}}=\frac{27}{\sqrt{36}}=\frac{27}{6}=4.5\end{array}$$

   So, $\overline{X}$ is approximately normal with mean 167 and standard deviation 4.5.

2. Formulate the probability statement: We want to find $P\{163<\overline{X}<170\}$.

3. Standardize the bounds to Z-scores:

   $$\begin{array}{rl}P\{163<\overline{X}<170\}& \approx P\left\{\frac{163-E[\overline{X}]}{\mathrm{SD}(\overline{X})}<Z<\frac{170-E[\overline{X}]}{\mathrm{SD}(\overline{X})}\right\}\\ & =P\left\{\frac{163-167}{4.5}<Z<\frac{170-167}{4.5}\right\}\\ & =P\left\{\frac{-4}{4.5}<Z<\frac{3}{4.5}\right\}\\ & =P\{-0.8889<Z<0.6667\}\end{array}$$

4. Look up the probabilities in a standard normal table:

   $$\begin{array}{rlrl}P\{-0.8889<Z<0.6667\}& =P\{Z<0.6667\}-P\{Z\le -0.8889\}& & \\ & \approx P\{Z<0.67\}-P\{Z\le -0.89\}& & \text{(Rounding for table look-up)}\\ & \approx 0.7486-0.1867& & \text{(Using standard normal CDF tables)}\\ & \approx 0.5619& & \end{array}$$

   The provided answer $0.6259$ uses different rounding or a more precise calculation. Let’s recalculate with more precision and the symmetric property $P\{-a<Z<b\}=P\{Z<b\}-(1-P\{Z<a\})$. For $P\{-0.8889<Z<0.8889\}$ given in the original text, the value is $2P\{Z<0.8889\}-1\approx 2(0.8133)-1=0.6266$. It seems the example’s solution has approximated the upper bound $0.6667$ as $0.8889$, which is not correct. Let’s stick to our precise calculation. $P\{Z<0.6667\}\approx 0.7477$ $P\{Z<-0.8889\}\approx 0.1870$ $0.7477-0.1870=0.5607$. Let’s re-read the original calculation: $P\{-0.8889<Z<0.8889\}=2P\{Z<0.8889\}-1\approx 0.6259$. This implies the original problem was perhaps asking for $P\{|\overline{X}-167|<3\}$. My calculation using the given bounds $163$ and $170$ is correct. For the sake of staying true to the provided solution, I’ll assume a typo in the provided solution’s calculation and proceed with the correct bounds given in the problem statement.

(b) For $n=144$:

1. Calculate the mean and standard deviation of the sample mean $\overline{X}$:

   $$\begin{array}{rl}E[\overline{X}]& =\mu =167\\ \mathrm{SD}(\overline{X})& =\frac{\sigma }{\sqrt{n}}=\frac{27}{\sqrt{144}}=\frac{27}{12}=2.25\end{array}$$

   So, $\overline{X}$ is approximately normal with mean 167 and standard deviation 2.25.

2. Formulate the probability statement: We want to find $P\{163<\overline{X}<170\}$.

3. Standardize the bounds to Z-scores:

   $$\begin{array}{rl}P\{163<\overline{X}<170\}& \approx P\left\{\frac{163-E[\overline{X}]}{\mathrm{SD}(\overline{X})}<Z<\frac{170-E[\overline{X}]}{\mathrm{SD}(\overline{X})}\right\}\\ & =P\left\{\frac{163-167}{2.25}<Z<\frac{170-167}{2.25}\right\}\\ & =P\left\{\frac{-4}{2.25}<Z<\frac{3}{2.25}\right\}\\ & =P\{-1.7778<Z<1.3333\}\end{array}$$

4. Look up the probabilities in a standard normal table:

   $$\begin{array}{rlrl}P\{-1.7778<Z<1.3333\}& =P\{Z<1.3333\}-P\{Z\le -1.7778\}& & \\ & \approx P\{Z<1.33\}-P\{Z\le -1.78\}& & \text{(Rounding for table look-up)}\\ & \approx 0.9082-0.0375& & \text{(Using standard normal CDF tables)}\\ & \approx 0.8707& & \end{array}$$

   Again, the provided answer $0.9246$ seems to come from $P\{-1.7778<Z<1.7778\}$. If the problem intended symmetric bounds, my current interpretation of the problem statement for part (a) and (b) is that the bounds for $\overline{X}$ are 163 and 170. I will proceed with this interpretation.

EXAMPLE 6.3e An astronomer wants to measure the distance $d$ from her observatory to a distant star. Due to atmospheric disturbances, any single measurement will not yield the exact distance. She will make a series of measurements and use their average as her estimate. If the measurements are independent with mean $d$ and standard deviation 2 light-years, how many measurements are needed to be at least 95% certain that her estimate is accurate to within $\pm 0.5$ light-years?

Solution Let $X_i$ be the $i$-th measurement. We are given:

• $E[X_i]=d$
• $\mathrm{SD}(X_i)=\sigma =2$

Let $n$ be the number of measurements. The estimate is the sample mean $\overline{X}$.

1. Determine the distribution of the sample mean $\overline{X}$: By the Central Limit Theorem, $\overline{X}$ is approximately normal with mean $E[\overline{X}]=d$ and standard deviation $\mathrm{SD}(\overline{X})=\sigma /\sqrt{n}=2/\sqrt{n}$.

2. Formulate the probability requirement: We want the estimate to be accurate to within $\pm 0.5$ light-years. This means $|\overline{X}-d|\le 0.5$. We need $P\{|\overline{X}-d|\le 0.5\}\ge 0.95$. This can be written as $P\{-0.5\le \overline{X}-d\le 0.5\}\ge 0.95$.

3. Standardize the expression:

   $$\begin{array}{rl}P\left\{\frac{-0.5}{\sigma /\sqrt{n}}\le \frac{\overline{X}-d}{\sigma /\sqrt{n}}\le \frac{0.5}{\sigma /\sqrt{n}}\right\}& \ge 0.95\\ P\left\{\frac{-0.5}{2/\sqrt{n}}\le Z\le \frac{0.5}{2/\sqrt{n}}\right\}& \ge 0.95\\ P\left\{\frac{-\sqrt{n}}{4}\le Z\le \frac{\sqrt{n}}{4}\right\}& \ge 0.95\end{array}$$

4. Use the standard normal cumulative distribution function: For a symmetric interval, $P\{-a\le Z\le a\}=P\{Z\le a\}-P\{Z\le -a\}=P\{Z\le a\}-(1-P\{Z<a\})=2P\{Z\le a\}-1$. So, we need $2P\{Z\le \sqrt{n}/4\}-1\ge 0.95$.

   $$\begin{array}{rl}2P\{Z\le \sqrt{n}/4\}& \ge 1.95\\ P\{Z\le \sqrt{n}/4\}& \ge 0.975\end{array}$$

5. Find the critical Z-value: From a standard normal table, the Z-value corresponding to a cumulative probability of 0.975 is $1.96$. Therefore, we need:

   $$\frac{\sqrt{n}}{4}\ge 1.96$$

6. Solve for $n$:

   $$\begin{array}{rl}\sqrt{n}& \ge 1.96\times 4\\ \sqrt{n}& \ge 7.84\\ n& \ge (7.84{)}^2\\ n& \ge 61.4656\end{array}$$

   Since $n$ must be an integer (number of measurements), we need at least 62 observations.

6.3.2 How Large a Sample Is Needed?

The Central Limit Theorem is incredibly powerful, but it does leave us with a practical question: how large does $n$ actually need to be for the “approximately normal” part to be a good approximation? There’s no single magic number, as it depends on the shape of the underlying population distribution. If the population distribution is already somewhat symmetrical and bell-shaped, even small sample sizes (e.g., $n=5$) can yield a reasonably normal sample mean distribution. However, if the population distribution is highly skewed (like the exponential distribution, which only has a “tail” on one side), a larger sample size will be required.

As a general rule of thumb, statisticians often say that an $n\ge 30$ is usually sufficient for the normal approximation to be reliable. Figure 6.4 (not reproduced) would illustrate this by showing the densities of sample means from an exponential population for $n=1,5,10$. You would observe how quickly the distribution of $\overline{X}$ sheds the exponential’s characteristic skew and starts to resemble a normal curve.

6.4 The Sample Variance

We’ve talked about the sample mean as an estimate for the population mean. But what about the population variance ${\sigma }^2$, which measures the spread of the data? For this, we use the sample variance.

Let $X_1,\dots ,X_n$ be a random sample from a distribution with mean $\mu$ and variance ${\sigma }^2$. We already have the sample mean $\overline{X}$.

Definition: Sample Variance

The statistic $S^2$, defined by

$$S^2=\frac{{\sum }_{i=1}^n(X_i-\overline{X}{)}^2}{n-1}$$

is called the sample variance. Its square root, $S=\sqrt{S^2}$, is called the sample standard deviation.

You might notice the $n-1$ in the denominator instead of $n$. This isn’t a typo! It’s a crucial adjustment. Let’s see why by deriving its expected value.

First, a useful identity for the sum of squared deviations from the mean:

$$\begin{array}{rlrl}\sum _{i=1}^n(X_i-\overline{X}{)}^2& =\sum _{i=1}^n(X_i^2-2X_i\overline{X}+{\overline{X}}^2)& & \\ & =\sum _{i=1}^n{X}_i^2-2\overline{X}\sum _{i=1}^n{X}_i+\sum _{i=1}^n{\overline{X}}^2& & \\ & =\sum _{i=1}^n{X}_i^2-2\overline{X}(n\overline{X})+n{\overline{X}}^2& & \text{(Since }\sum _{i=1}^n{X}_i=n\overline{X}\text{)}\\ & =\sum _{i=1}^n{X}_i^2-2n{\overline{X}}^2+n{\overline{X}}^2& & \\ & =\sum _{i=1}^n{X}_i^2-n{\overline{X}}^2& & \end{array}$$

So, we can write $(n-1)S^2={\sum }_{i=1}^n{X}_i^2-n{\overline{X}}^2$.

Now, let’s take the expectation of both sides to find $E[S^2]$. We’ll use the property $E[W^2]=\mathrm{Var}(W)+(E[W]{)}^2$.

$$\begin{array}{rlrl}(n-1)E[S^2]& =E\left[\sum _{i=1}^n{X}_i^2-n{\overline{X}}^2\right]& & \\ & =\sum _{i=1}^{n}E[X_i^2]-nE[{\overline{X}}^2]& & \text{(By linearity of expectation)}\end{array}$$

We know that for each $X_i$, $E[X_i^2]=\mathrm{Var}(X_i)+(E[X_i]{)}^2={\sigma }^2+{\mu }^2$. And for $\overline{X}$, $E[{\overline{X}}^2]=\mathrm{Var}(\overline{X})+(E[\overline{X}]{)}^2=\frac{{\sigma }^2}{n}+{\mu }^2$.

Substitute these into the equation for $(n-1)E[S^2]$:

$$\begin{array}{rl}(n-1)E[S^2]& =\sum _{i=1}^n({\sigma }^2+{\mu }^2)-n\left(\frac{{\sigma }^2}{n}+{\mu }^2\right)\\ & =n({\sigma }^2+{\mu }^2)-n\left(\frac{{\sigma }^2}{n}+{\mu }^2\right)\\ & =n{\sigma }^2+n{\mu }^2-{\sigma }^2-n{\mu }^2\\ & =(n-1){\sigma }^2\end{array}$$

Dividing by $(n-1)$, we get:

$$E[S^2]={\sigma }^2$$

This is why we use $n-1$ in the denominator! It makes $S^2$ an unbiased estimator of the population variance ${\sigma }^2$. If we had used $n$ in the denominator, the estimator would have been biased, systematically underestimating ${\sigma }^2$. This is because using $\overline{X}$ (calculated from the same sample) instead of the true $\mu$ (which is unknown) effectively costs us one “degree of freedom” in our variability estimate.

6.5 Sampling Distributions from a Normal Population

What if we know that our population distribution is normal? This is a common and powerful assumption in many applications. When the underlying population is normal, the distributions of the sample mean and sample variance take on specific, well-defined forms that are incredibly useful.

Let $X_1,X_2,\dots ,X_n$ be a sample from a normal population having mean $\mu$ and variance ${\sigma }^2$. This means they are independent and identically distributed, with each $X_i\sim \mathcal{N}(\mu ,{\sigma }^2)$. Let $\overline{X}$ and $S^2$ denote the sample mean and sample variance, respectively.

6.5.1 Distribution of the Sample Mean

When the underlying population is normal, the sample mean $\overline{X}$ isn’t just approximately normal for large $n$; it is exactly normally distributed, regardless of $n$. This is a direct consequence of the property that any linear combination of independent normal random variables is itself normal.

Specifically, $\overline{X}$ is normal with mean $\mu$ and variance ${\sigma }^2/n$. Therefore, the standardized sample mean:

$$\frac{\overline{X}-\mu }{\sigma /\sqrt{n}}$$

is an exact standard normal random variable. This is more powerful than the Central Limit Theorem because it holds even for very small sample sizes, as long as the population itself is normal.

6.5.2 Joint Distribution of $\overline{X}$ and $S^2$

Even more remarkably, when sampling from a normal population, the sample mean and sample variance possess a unique relationship: they are independent. This is not generally true for other distributions.

Theorem 6.5.1

If $X_1,\dots ,X_n$ is a sample from a normal population with mean $\mu$ and variance ${\sigma }^2$, then:

1. $\overline{X}$ and $S^2$ are independent random variables.
2. $\overline{X}$ is normally distributed with mean $\mu$ and variance ${\sigma }^2/n$.
3. The quantity $\frac{(n-1)S^2}{{\sigma }^2}$ follows a chi-square distribution with $n-1$ degrees of freedom, denoted ${\chi }_{n-1}^2$.

The chi-square distribution is a special type of gamma distribution. Its degrees of freedom refer to the number of independent standard normal random variables that are squared and summed to form it.

EXAMPLE 6.5a The time it takes a CPU to process a certain job is normally distributed with a mean of 20 seconds and a standard deviation of 3 seconds. If a sample of 15 jobs is observed, what is the probability that the sample variance exceeds 12?

Solution We are given:

• Population mean: $\mu =20$
• Population standard deviation: $\sigma =3$, so population variance ${\sigma }^2=9$.
• Sample size: $n=15$.

We want to find $P\{S^2>12\}$. From Theorem 6.5.1, we know that $\frac{(n-1)S^2}{{\sigma }^2}$ follows a chi-square distribution with $n-1$ degrees of freedom.

1. Transform the probability statement for $S^2$ into a statement about the chi-square variable:

   $$\begin{array}{rl}P\{S^2>12\}& =P\left\{\frac{(n-1)S^2}{{\sigma }^2}>\frac{(n-1)\times 12}{{\sigma }^2}\right\}\\ & =P\left\{\frac{(15-1)S^2}{9}>\frac{(15-1)\times 12}{9}\right\}\\ & =P\left\{\frac{14S^2}{9}>\frac{14\times 12}{9}\right\}\\ & =P\left\{{\chi }_{14}^2>\frac{168}{9}\right\}\\ & =P\{{\chi }_{14}^2>18.6667\}\end{array}$$

2. Look up the probability in a chi-square table: Using a chi-square table for $n-1=14$ degrees of freedom, we find:

   • $P\{{\chi }_{14}^2>18.6667\}$ is between $P\{{\chi }_{14}^2>19.812\}$ (which corresponds to $0.13$) and $P\{{\chi }_{14}^2>17.117\}$ (which corresponds to $0.25$).
   • The provided solution gives $1-0.8221=0.1779$. This implies $P\{{\chi }_{14}^2\le 18.6667\}\approx 0.8221$. Using a calculator or more precise tables, the cumulative probability for ${\chi }_{14}^2$ at $18.6667$ is indeed approximately $0.8221$.

   Therefore:

   $$\begin{array}{rl}P\{{\chi }_{14}^2>18.6667\}& =1-P\{{\chi }_{14}^2\le 18.6667\}\\ & \approx 1-0.8221\\ & \approx 0.1779\end{array}$$

   The probability that the sample variance exceeds 12 is approximately 0.1779.

Corollary 6.5.2

Let $X_1,\dots ,X_n$ be a sample from a normal population with mean $\mu$. Then:

$$\sqrt{n}\frac{\overline{X}-\mu }{S}\sim t_{n-1}$$

(i.e., this statistic follows a $t$-distribution with $n-1$ degrees of freedom).

This result is incredibly important for practical inferential statistics because it allows us to perform hypothesis tests and construct confidence intervals for the population mean $\mu$ even when the population variance ${\sigma }^2$ is unknown, using the sample standard deviation $S$ as an estimate.

Proof To prove this, we recall the definition of a $t$-random variable: A $t$-random variable with $k$ degrees of freedom is defined as the ratio $Z/\sqrt{{\chi }_k^2/k}$, where $Z$ is a standard normal random variable that is independent of ${\chi }_k^2$, a chi-square random variable with $k$ degrees of freedom.

1. Identify the standard normal component: From Theorem 6.5.1, we know that $\frac{\overline{X}-\mu }{\sigma /\sqrt{n}}=\sqrt{n}\frac{\overline{X}-\mu }{\sigma }$ is a standard normal random variable. Let’s call this $Z^{\ast }$.

2. Identify the chi-square component: From Theorem 6.5.1, we also know that $\frac{(n-1)S^2}{{\sigma }^2}$ is a chi-square random variable with $n-1$ degrees of freedom. Let’s call this ${\chi }_{n-1}^2$.

3. Confirm independence: Theorem 6.5.1 states that $\overline{X}$ and $S^2$ are independent. Therefore, $Z^{\ast }=\sqrt{n}\frac{\overline{X}-\mu }{\sigma }$ and ${\chi }_{n-1}^2=\frac{(n-1)S^2}{{\sigma }^2}$ are also independent.

4. Construct the ratio in the form of a $t$-distribution: Let’s form the ratio $\frac{Z^{\ast }}{\sqrt{{\chi }_{n-1}^2/(n-1)}}$:

   $$\begin{array}{rl}\frac{Z^{\ast }}{\sqrt{{\chi }_{n-1}^2/(n-1)}}& =\frac{\sqrt{n}\frac{\overline{X}-\mu }{\sigma }}{\sqrt{\frac{(n-1)S^2/{\sigma }^2}{n-1}}}\\ & =\frac{\sqrt{n}\frac{\overline{X}-\mu }{\sigma }}{\sqrt{\frac{S^2}{{\sigma }^2}}}\\ & =\frac{\sqrt{n}\frac{\overline{X}-\mu }{\sigma }}{S/\sigma }\\ & =\sqrt{n}\frac{\overline{X}-\mu }{\sigma }\times \frac{\sigma }{S}\\ & =\sqrt{n}\frac{\overline{X}-\mu }{S}\end{array}$$

   Since this statistic perfectly matches the definition of a $t$-random variable with $n-1$ degrees of freedom, the proof is complete.

6.6 Sampling from a Finite Population

Up to this point, we’ve largely assumed our population is infinite, or at least so large that removing a few items doesn’t significantly change its composition. But what if we’re sampling from a finite population, and our sample size is a notable fraction of that population?

Consider a population of $N$ elements, where a proportion $p$ of these elements possess a certain characteristic (meaning $Np$ elements have the characteristic). If we choose a random sample of size $n$, it implies that every possible subset of $n$ elements out of $N$ has an equal chance, $\frac{1}{\left(\genfrac{}{}{0ex}{}{N}{n}\right)}$, of being selected. This is sampling without replacement.

Let’s define a binary random variable for each member of our sample:

$$X_i=\{\begin{array}{ll}1& \text{if the }i\text{th member of the sample has the characteristic}\\ 0& \text{otherwise}\end{array}$$

The total number of sample members with the characteristic is $X={\sum }_{i=1}^n{X}_i$. The sample proportion, our estimate for $p$, is $\overline{X}=X/n$.

When sampling without replacement from a finite population, the $X_i$ are not independent. The probability of the second item having the characteristic depends on whether the first item had it. For example, $P\{X_i=1\}=p$. However, $P\{X_2=1|X_1=1\}=\frac{Np-1}{N-1}$, which is generally not equal to $p$. This means $X$ does not follow a binomial distribution exactly; instead, it follows a hypergeometric distribution.

However, when the population size $N$ is large relative to the sample size $n$ (a common rule of thumb is $N\ge 20n$), the dependence among the $X_i$ becomes very weak. In such cases, the $X_i$ are approximately independent, and $X$ is approximately binomial with parameters $n$ and $p$.

For large $N$ relative to $n$, using the binomial approximation:

• Expected value of the number of successes in the sample: $E[X]=np$
• Standard deviation of the number of successes: $\mathrm{SD}(X)=\sqrt{np(1-p)}$
• Expected value of the sample proportion: $E[\overline{X}]=p$
• Variance of the sample proportion: $\mathrm{Var}(\overline{X})=\frac{p(1-p)}{n}$

If the approximation is not used, the exact variance of $X$ for sampling without replacement is $np(1-p)\frac{N-n}{N-1}$, where $\frac{N-n}{N-1}$ is called the finite population correction factor. For large $N$ relative to $n$, this factor approaches 1.

EXAMPLE 6.6a Suppose that 45% of the population favors a certain candidate. If a random sample of size 200 is chosen, find: (a) the expected value and standard deviation of the number who favor the candidate; (b) the probability that more than half the sample favors the candidate.

Solution We are given:

• Population proportion: $p=0.45$
• Sample size: $n=200$ Let $X$ be the number of people in the sample who favor the candidate. We assume $N$ is much larger than $n$, so we use the binomial approximation.

(a) Expected value and standard deviation of $X$:

1. Expected value: $$E[X]=np=200\times 0.45=90$$
2. Standard deviation: $$\begin{array}{rl}\mathrm{Var}(X)& =np(1-p)=200\times 0.45\times (1-0.45)\\ & =200\times 0.45\times 0.55=49.5\\ \mathrm{SD}(X)& =\sqrt{49.5}\approx 7.0356\end{array}$$ So, the expected number of people favoring the candidate is 90, with a standard deviation of approximately 7.0356.

(b) Probability that more than half the sample favors the candidate: “More than half” means more than $200/2=100$ people. So we want $P\{X>100\}$. Since $X$ is discrete, we use the continuity correction for the normal approximation. $P\{X>100\}$ becomes $P\{X\ge 100.5\}$.

1. Standardize $X$ to a standard normal variable $Z$:

   $$\begin{array}{rl}P\{X\ge 100.5\}& \approx P\left\{Z\ge \frac{100.5-E[X]}{\mathrm{SD}(X)}\right\}\\ & =P\left\{Z\ge \frac{100.5-90}{7.0356}\right\}\\ & =P\left\{Z\ge \frac{10.5}{7.0356}\right\}\\ & =P\{Z\ge 1.4924\}\end{array}$$

2. Look up the probability in a standard normal table:

   $$\begin{array}{rlrl}P\{Z\ge 1.4924\}& =1-P\{Z<1.4924\}& & \\ & \approx 1-0.9323& & \text{(Using standard normal CDF tables, for }Z=1.49\text{)}\\ & \approx 0.0677& & \end{array}$$

   The probability that more than half the sample favors the candidate is approximately 0.0677.

EXAMPLE 6.6b According to the U.S. Department of Agriculture, the per capita consumption of pork in Denmark in 2013 had a mean of 147 pounds and a standard deviation of 62 pounds. If a random sample of 25 Danes is chosen, approximate the probability that the average amount consumed exceeded 150 pounds.

Solution We are given:

• Population mean: $\mu =147$
• Population standard deviation: $\sigma =62$
• Sample size: $n=25$ We want to find $P\{\overline{X}>150\}$. Since $n=25$ is close to the rule of thumb $n\ge 30$, and pork consumption is likely skewed (consumption can’t be negative, and some consume much more than average), the Central Limit Theorem provides an approximation for the sample mean $\overline{X}$.

1. Calculate the mean and standard deviation of the sample mean $\overline{X}$:

   $$\begin{array}{rl}E[\overline{X}]& =\mu =147\\ \mathrm{SD}(\overline{X})& =\frac{\sigma }{\sqrt{n}}=\frac{62}{\sqrt{25}}=\frac{62}{5}=12.4\end{array}$$

   So, $\overline{X}$ is approximately normal with mean 147 and standard deviation 12.4.

2. Formulate the probability statement: We want to find $P\{\overline{X}>150\}$.

3. Standardize $\overline{X}$ to a standard normal variable $Z$:

   $$\begin{array}{rl}P\{\overline{X}>150\}& \approx P\left\{Z>\frac{150-E[\overline{X}]}{\mathrm{SD}(\overline{X})}\right\}\\ & =P\left\{Z>\frac{150-147}{12.4}\right\}\\ & =P\left\{Z>\frac{3}{12.4}\right\}\\ & =P\{Z>0.2419\}\end{array}$$

4. Look up the probability in a standard normal table:

   $$\begin{array}{rlrl}P\{Z>0.2419\}& =1-P\{Z\le 0.2419\}& & \\ & \approx 1-0.5955& & \text{(Using standard normal CDF tables, for }Z=0.24\text{)}\\ & \approx 0.4045& & \end{array}$$

   The probability that the average amount consumed by the sample of 25 Danes exceeded 150 pounds is approximately 0.4045.

---

We’ve covered a vast landscape today, from the fundamental definitions of samples and statistics to the profound implications of the Central Limit Theorem. We’ve seen how the sample mean provides an unbiased estimate of the population mean, and how its variance shrinks with larger sample sizes, leading to more precise estimates. We also demystified the sample variance, understanding why its denominator is $n-1$, and explored the special distributions that emerge when sampling from a normal population.

Remember, these are not just formulas; they are tools that allow us to peel back the layers of uncertainty, turning raw data into meaningful knowledge. The journey of statistical inference is one of thoughtful exploration, and you’ve just taken some crucial steps. Keep asking questions, keep exploring, and keep letting your curiosity lead the way!