Introduction to Probability and Statistics

9.1 Some preliminary set theory

Probability is built on sets, so before any probability appears we need to be fluent with the language of sets. Most of this will be revision, but the notation matters enormously later — almost every probability rule in this chapter is really a set identity in disguise.

Example. The set $A=\{1,2,3\}$ has elements $1$, $2$ and $3$; so $1\in A$ but $4\notin A$.

Example. The set $A=\{1,2,3\}$ has eight subsets; for instance $\{2,3\}\subseteq A$. In general a set with $n$ elements has $2^n$ subsets, since each element is independently either in or out — this doubling is worth remembering, as it reappears when we count outcomes.

In any given problem there is a fixed set containing everything under discussion, called the universal set $S$. For three-dimensional vector geometry it is usually $S={\mathbb{R}}^3$; in probability it will be the set of all possible outcomes.

Example. Let $S$ be all students enrolled in MATH1231, let $A$ be those aged 20 or over, and let $B$ be those who own a bicycle. Then $A\cap B$ is the students aged 20 or over who own a bicycle; $A\cup B$ is those who are aged 20 or over, own a bicycle, or both; $A^c$ is those under 20; and $A-B$ is those aged 20 or over who do not own a bicycle. Note that "or" in mathematics is always inclusive — $A\cup B$ includes the students who satisfy both conditions. English "or" is frequently exclusive, and that mismatch causes real errors when translating word problems.

Example. The sets $A_1=\{1,3\}$, $A_2=\{2\}$ and $A_3=\{4,5\}$ partition $B=\{1,2,3,4,5\}$.

Basically, a partition chops a set into non-overlapping pieces that between them use up everything. This is the single most important idea in the chapter: every major probability rule — the addition rule, the law of total probability, Bayes' theorem — works by partitioning a set and adding up the pieces, which is only legitimate because the pieces do not overlap.

Two identities get used constantly and are worth proving once.

Proof. We prove the first; the second is identical with the roles of $\cup$ and $\cap$ swapped. For any $x\in S$,

$$\begin{array}{rl}x\in (A\cup B{)}^c& ⟺x\notin A\cup B\\ & ⟺\text{it is not the case that }(x\in A\text{ or }x\in B)\\ & ⟺x\notin A\text{ and }x\notin B\\ & ⟺x\in A^c\text{ and }x\in B^c\\ & ⟺x\in A^c\cap B^c.\end{array}$$

Since the two sets have exactly the same elements, they are equal. $■$

Basically, De Morgan says that negating an "or" gives an "and", and vice versa. In probability language: "neither $A$ nor $B$ happened" is the same event as "not $A$ and not $B$", and "not both" is the same as "at least one failed". Getting this backwards is a very common source of wrong answers in "at least one" problems.

Proof. Adding $|A|$ and $|B|$ counts every element of $A\cap B$ exactly twice — once as a member of $A$ and once as a member of $B$ — while every other element of $A\cup B$ is counted once. Subtracting $|A\cap B|$ removes the double count. $■$

Example. Of $20$ music students, $7$ play guitar, $8$ play piano, and $3$ play both. How many play at least one of the two, and how many play neither?
Let $G$ and $P$ be the sets of guitarists and pianists, so $|G|=7$, $|P|=8$ and $|G\cap P|=3$. Then

$$\begin{array}{rl}|G\cup P|& =|G|+|P|-|G\cap P|\\ & =7+8-3\\ & =12,\end{array}$$

and by De Morgan the students who play neither form $(G\cup P{)}^c$, of size $20-12=8$. Therefore $12$ students play at least one instrument and $8$ play neither. Notice that $7+8=15\ne 12$; forgetting to subtract the overlap is the classic error, and it is exactly the error the addition rule for probabilities is designed to prevent.

9.2 Probability

9.2.1 Sample space and probability axioms

An experiment is any procedure with an uncertain outcome.

Example. Tossing a coin is an experiment with sample space $S=\{H,T\}$.

Example. Tossing a coin three times and recording the ordered sequence has sample space

$$S=\{HHH,HHT,HTH,HTT,THH,THT,TTH,TTT\},$$

with $|S|=2^3=8$. The event $A$ that exactly two heads are tossed is the subset

$$A=\{HHT,HTH,THH\}.$$

The choice of sample space is part of modelling, not something handed to you. If we only cared about the number of heads we could have taken $S=\{0,1,2,3\}$ — but then the outcomes would not be equally likely, which usually makes life harder. Choose a sample space in which the outcomes are equally likely whenever you can.

Basically, probability is a way of measuring how big an event is compared with the whole sample space: never negative, the whole space has measure $1$, and non-overlapping pieces add. Condition (d) requires disjointness and is worthless without it — this is where almost every incorrect probability calculation goes wrong.

Proof. Statement 1 follows for finite $S$ by induction on $|A|$ using axiom (d), since the singletons $\{a\}$ for $a\in A$ are pairwise disjoint and partition $A$. Statement 3 is statement 1 applied to $A=S$, together with $P(S)=1$. For statement 2, suppose $P(\{a\})=p$ for every $a\in S$. By statement 3,

$$1=\sum _{a\in S}P(\{a\})=\sum _{a\in S}p=|S|p,$$

so $p=\frac{1}{|S|}$. Then by statement 1,

$$P(A)=\sum _{a\in A}P(\{a\})=\sum _{a\in A}\frac{1}{|S|}=\frac{|A|}{|S|}.■$$

Statement 2 is the formula you have used since school — probability equals favourable outcomes over total outcomes. But it is only valid when the outcomes are equally likely, and it is a theorem, not a definition. Applying it to a sample space with unequally likely outcomes is a serious error.

Example. Pick a ball at random from a bag containing $3$ red and $7$ blue balls. Taking $S$ to be the ten individual balls, each equally likely, the event $R$ that the ball is red has $|R|=3$, so

$$P(R)=\frac{|R|}{|S|}=\frac{3}{10}.$$

Therefore the probability of drawing red is $0.3$. Notice that had we taken $S=\{\text{red},\text{blue}\}$ we could not have used the formula, since those two outcomes are not equally likely.

9.2.2 Rules for probabilities

Proof.

1. The sets $A-B$, $A\cap B$ and $B-A$ partition $A\cup B$, so by axiom (d),

$$P(A\cup B)=P(A-B)+P(A\cap B)+P(B-A).$$

Also $A-B$ and $A\cap B$ partition $A$, giving $P(A-B)=P(A)-P(A\cap B)$, and similarly $P(B-A)=P(B)-P(A\cap B)$. Substituting,

$$P(A\cup B)=(P(A)-P(A\cap B))+P(A\cap B)+(P(B)-P(A\cap B))=P(A)+P(B)-P(A\cap B).$$

2. The sets $A$ and $A^c$ partition $S$, so $P(A)+P(A^c)=P(S)=1$.
3. If $A\subseteq B$ then $A$ and $B-A$ partition $B$, so $P(B)=P(A)+P(B-A)\ge P(A)$ by axiom (a). $■$

Notice that this is exactly the inclusion–exclusion principle with $|\cdot |$ replaced by $P$, and for the same reason: the overlap gets counted twice and must be removed once.

The complement rule is the most useful computational tool in the whole chapter. Whenever a question asks for the probability of "at least one" of something, computing $P(\text{none})$ and subtracting from $1$ is almost always dramatically easier, because "none" is a single intersection while "at least one" is a messy union.

Example. What is the probability that at least two of $n$ people share a birthday?
Ignoring leap years, let $Y$ be the $365$ days of the year. The sample space of all possible birthday lists is

$$S_n=\{(b_1,\dots ,b_n):b_1,\dots ,b_n\in Y\},|S_n|=365^n,$$

with all outcomes equally likely. Let $A$ be the event that at least two people share a birthday. Computing $P(A)$ directly would require a horrible union over all pairs, so use the complement: $A^c$ is the event that all $n$ birthdays are distinct, and such a list is an ordered selection of $n$ different days,

$$|A^c|=365\times 364\times \cdots \times (365-n+1).$$

Therefore

$$P(A)=1-P(A^c)=1-\frac{365\times 364\times \cdots \times (365-n+1)}{365^n}.$$

Evaluating this gives famously counter-intuitive numbers:

So with just $23$ people the odds are already better than even. Therefore the answer is $1-\frac{365!}{(365-n)!365^n}$. The intuition that misleads people here is comparing themselves to everyone else ($n-1$ comparisons) rather than counting all $\left(\genfrac{}{}{0ex}{}{n}{2}\right)$ pairs, which for $n=23$ is $253$ pairs — plenty of chances for a match.

Example. In a town, $80\%$ of the population has comprehensive car insurance, $60\%$ has house insurance, and $50\%$ has both. What proportion has at least one, and what proportion has neither?
Let $C$ and $H$ be the events of having car and house cover, so $P(C)=0.8$, $P(H)=0.6$ and $P(C\cap H)=0.5$. By the addition rule,

$$\begin{array}{rl}P(C\cup H)& =P(C)+P(H)-P(C\cap H)\\ & =0.8+0.6-0.5\\ & =0.9,\end{array}$$

and by De Morgan together with the complement rule,

$$P(C^c\cap H^c)=P((C\cup H{)}^c)=1-0.9=\mathrm{0.1.}$$

Therefore $90\%$ have at least one policy and $10\%$ have neither.

9.2.3 Conditional probabilities

Often we learn something partway through, and want to update a probability in the light of it. If we know that $B$ has occurred, then $B$ effectively becomes the new sample space, and we should measure $A$ only by the part of it lying inside $B$.

Basically, we restrict attention to $B$ and rescale so that $B$ has total probability $1$; dividing by $P(B)$ is exactly that rescaling. The condition $P(B)>0$ is not a technicality — conditioning on an impossible event is meaningless.

Rearranging gives the form used most often in practice:

$$\boxed{P(A\cap B)=P(A\mid B)P(B)=P(B\mid A)P(A).}$$

Example. We roll a die. Let $A$ be the event of rolling a six and $B$ the event of rolling an even number. Find $P(A\mid B)$ and $P(B\mid A)$.
Here $A=\{6\}$ and $B=\{2,4,6\}$, so $A\cap B=\{6\}$, and

$$\begin{array}{rl}P(A\mid B)& =\frac{P(A\cap B)}{P(B)}=\frac{1/6}{3/6}=\frac{1}{3},\\ P(B\mid A)& =\frac{P(A\cap B)}{P(A)}=\frac{1/6}{1/6}=1.\end{array}$$

Therefore $P(A\mid B)=\frac{1}{3}$ and $P(B\mid A)=1$. Note how different the two are: $P(A\mid B)\ne P(B\mid A)$ in general, and confusing the two is so common in law and medicine that it has a name — the prosecutor's fallacy. Knowing the roll is even makes a six more likely than before; knowing it is a six makes it certainly even.

Example. A bag contains $3$ red and $3$ blue balls. Two balls are drawn without replacement. Find the probability that both are red.
Let $R_1$ and $R_2$ be the events that the first and second balls are red. Before any draw, $P(R_1)=\frac{3}{6}=\frac{1}{2}$. Given the first ball was red, only $5$ balls remain of which $2$ are red, so $P(R_2\mid R_1)=\frac{2}{5}$. Therefore

$$P(R_1\cap R_2)=P(R_2\mid R_1)P(R_1)=\frac{2}{5}\times \frac{1}{2}=\frac{1}{5}.$$

Drawing without replacement is what makes the draws dependent; with replacement, $P(R_2\mid R_1)$ would still be $\frac{1}{2}$ and the answer would be $\frac{1}{4}$. Always check which of the two a question intends.

Conditioning becomes really powerful when combined with a partition.

Proof. Since the $B_i$ partition $S$, the sets $A\cap B_1,\dots ,A\cap B_n$ are pairwise disjoint and their union is $A$. By axiom (d) applied repeatedly and then the definition of conditional probability,

$$P(A)=\sum _{i=1}^{n}P(A\cap B_i)=\sum _{i=1}^{n}P(A\mid B_i)P(B_i).■$$

Basically, to find the probability of $A$, split the world into cases, work out the chance of $A$ in each case, and average those weighted by how likely each case is. This is exactly what a tree diagram does: each path through the tree is one term of the sum, and the probability of a path is the product of the probabilities along its edges.

Proof. By the definition of conditional probability twice over,

$$P(B_j\mid A)=\frac{P(A\cap B_j)}{P(A)}=\frac{P(A\mid B_j)P(B_j)}{P(A)},$$

and expanding the denominator by the law of total probability gives the result. $■$

Basically, Bayes' theorem reverses a conditional probability: it converts "the chance of the evidence given the cause" into "the chance of the cause given the evidence", which is almost always the direction you actually want. The following example is the standard illustration of how badly intuition fails here, and it is worth studying carefully.

Example. A diagnostic test for disease $X$ correctly indicates the disease $99\%$ of the time in people who have it, and correctly returns negative $98\%$ of the time in people who do not. Suppose $2\%$ of the population has the disease. Find the probability that a person who tests positive does not have the disease.
Let $D$ be the event of having the disease and $T_p$ the event of testing positive, with $T_n=T_p^c$. We are given

$$P(D)=0.02,P(T_p\mid D)=0.99,P(T_n\mid D^c)=0.98,\text{so }P(T_p\mid D^c)=\mathrm{0.02.}$$

The events $D$ and $D^c$ partition $S$, so by Bayes' theorem,

$$\begin{array}{rl}P(D^c\mid T_p)& =\frac{P(T_p\mid D^c)P(D^c)}{P(T_p\mid D^c)P(D^c)+P(T_p\mid D)P(D)}\\ & =\frac{0.02\times 0.98}{0.02\times 0.98+0.99\times 0.02}\\ & =\frac{0.0196}{0.0196+0.0198}\\ & =\frac{0.0196}{0.0394}\\ & \approx \mathrm{0.497.}\end{array}$$

Therefore almost half of all positive results are false, despite the test being $98$–$99\%$ accurate. The reason is that the disease is rare: out of $10000$ people, about $200$ have it and $198$ of those test positive, while of the $9800$ healthy people $2\%$ — that is, $196$ people — also test positive. The two groups are almost the same size, so a positive result is nearly a coin flip.

By contrast, a negative result is extremely informative:

$$\begin{array}{rl}P(D\mid T_n)& =\frac{P(T_n\mid D)P(D)}{P(T_n\mid D)P(D)+P(T_n\mid D^c)P(D^c)}\\ & =\frac{0.01\times 0.02}{0.01\times 0.02+0.98\times 0.98}\\ & =\frac{0.0002}{0.9606}\\ & \approx \mathrm{0.000208.}\end{array}$$

So someone testing negative almost certainly does not have the disease. The lesson is that the accuracy of a test tells you very little on its own; the base rate $P(D)$ matters just as much, and ignoring it is called the base rate fallacy. In practice this is why a cheap screening test is followed up by a more expensive confirmatory one.

Example. Modify the previous example by supposing a fraction $x$ of the population has disease $X$, and that $5\%$ of a large random sample tests positive. What percentage of the population has the disease?
By the law of total probability,

$$\begin{array}{rl}0.05=P(T_p)& =P(T_p\mid D)P(D)+P(T_p\mid D^c)P(D^c)\\ & =0.99x+0.02(1-x)\\ & =0.97x+\mathrm{0.02.}\end{array}$$

Solving, $0.97x=0.03$, so

$$x=\frac{0.03}{0.97}\approx \mathrm{0.031.}$$

Therefore about $3.1\%$ of the population has the disease. Notice that the observed positive rate of $5\%$ substantially overstates the true prevalence of $3.1\%$, again because of false positives among the large healthy majority.

9.2.4 Statistical independence

Sometimes learning that $B$ occurred tells us nothing at all about $A$. That deserves a definition.

When $P(B)>0$ this is equivalent to $P(A\mid B)=P(A)$, which is the more intuitive form: conditioning on $B$ leaves the probability of $A$ unchanged. The product form is preferred as the definition because it is symmetric in $A$ and $B$ and does not require $P(B)>0$.

Independent and mutually exclusive are completely different things, and confusing them is the most common conceptual error in this chapter. In fact they are almost opposites: if $A$ and $B$ are mutually exclusive with $P(A),P(B)>0$, then $P(A\cap B)=P(\varnothing )=0$ but $P(A)P(B)>0$, so they are necessarily dependent. Intuitively, if $A$ and $B$ cannot both happen, then learning $A$ occurred tells you a great deal about $B$ — namely that it did not occur.

Example. Roll a die; let $A$ be rolling a six and $B$ rolling an even number. Are $A$ and $B$ independent?
Here $P(A)=\frac{1}{6}$, $P(B)=\frac{1}{2}$ and $P(A\cap B)=P(\{6\})=\frac{1}{6}$. But

$$P(A)P(B)=\frac{1}{6}\times \frac{1}{2}=\frac{1}{12}\ne \frac{1}{6}=P(A\cap B),$$

so they are not independent. This matches the earlier calculation that $P(A\mid B)=\frac{1}{3}\ne \frac{1}{6}=P(A)$.

Example. Roll a die twice. For each $i$, let $X_i$ be the event that the first roll gives $i$ and $Y_i$ that the second roll gives $i$. Then $P(X_i)=P(Y_j)=\frac{1}{6}$, and since each of the $36$ ordered pairs is equally likely,

$$P(X_i\cap Y_j)=\frac{1}{36}=\frac{1}{6}\times \frac{1}{6}=P(X_i)P(Y_j),$$

so $X_i$ and $Y_j$ are independent for all $i$ and $j$ — as they should be, since the die has no memory.

For more than two events, independence has to be required for every sub-collection, not just for pairs.

So $A$, $B$, $C$ are mutually independent exactly when all four of these hold:

$$\begin{array}{rl}P(A\cap B)& =P(A)P(B),\\ P(A\cap C)& =P(A)P(C),\\ P(B\cap C)& =P(B)P(C),\\ P(A\cap B\cap C)& =P(A)P(B)P(C).\end{array}$$

Pairwise independence does not imply mutual independence. The standard counterexample: draw a ball at random from a bag of four balls marked $0,1,2,3$, and let $A$, $B$, $C$ be the events that the ball is $1$ or $3$, that it is $2$ or $3$, and that it is $0$ or $3$. Each has probability $\frac{1}{2}$, and each pairwise intersection is $\{3\}$ with probability $\frac{1}{4}=\frac{1}{2}\times \frac{1}{2}$, so the events are pairwise independent. But $A\cap B\cap C=\{3\}$ also has probability $\frac{1}{4}$, whereas $P(A)P(B)P(C)=\frac{1}{8}$. So the three are pairwise but not mutually independent.

Basically, independence survives complementing — if $A$ tells you nothing about $B$, then $A$ tells you nothing about "not $B$" either. This is what lets us compute "at least one" probabilities by complementing each event separately.

Example. (Reliability.) A system has three components that fail independently, with failure probabilities $0.1$, $0.2$ and $0.3$. Find the probability the system works, if it needs (a) all three components working, and (b) at least one component working.
Let $F_1,F_2,F_3$ be the failure events, so the working events $F_i^c$ have probabilities $0.9$, $0.8$ and $0.7$ and are mutually independent by the theorem above.

(a) All three working:

$$P(F_1^c\cap F_2^c\cap F_3^c)=0.9\times 0.8\times 0.7=\mathrm{0.504.}$$

(b) At least one working is the complement of all three failing, so

$$\begin{array}{rl}P(\text{at least one works})& =1-P(F_1\cap F_2\cap F_3)\\ & =1-0.1\times 0.2\times 0.3\\ & =1-0.006\\ & =\mathrm{0.994.}\end{array}$$

Therefore the series system works with probability $0.504$ and the parallel system with probability $0.994$. Notice how much redundancy buys you — and notice that (b) was computed via the complement, because "at least one" as a union would have needed the full three-set inclusion–exclusion formula.

9.3 Random variables

Describing events as subsets of $S$ gets clumsy fast, and it gives us nothing to do arithmetic with. The fix is to attach a number to each outcome.

A random variable is neither random nor a variable; it is a function. The name is entirely historical and actively misleading, so it is worth fixing the correct picture early: $X$ takes an outcome $s\in S$ and returns a number $X(s)$. The randomness lives in which $s$ occurs, not in $X$.

Example. Toss a coin, with $S=\{H,T\}$. Define $X(H)=1$ and $X(T)=0$. Then $X$ counts the heads.

Example. Toss a coin three times and let $X$ count the number of heads. Then $X(HHT)=2$, $X(TTT)=0$, and so on. The event "exactly two heads" is now written compactly as $\{X=2\}$, meaning $\{s\in S:X(s)=2\}=\{HHT,HTH,THH\}$, and

$$P(X=2)=\frac{3}{8}.$$

This notation is the whole point: $\{X=2\}$, $\{X\le 2\}$ and $\{X>0\}$ are all just events, described far more conveniently than by listing outcomes.

9.3.1 Discrete random variables

By the summation theorem, the $p_k$ are non-negative and sum to $1$; these two conditions characterise a valid probability distribution, and checking that your probabilities sum to $1$ is the cheapest error check available.

Example. Toss a coin three times and let $X$ count the heads. Since all eight outcomes are equally likely and the numbers of outcomes giving $0,1,2,3$ heads are $1,3,3,1$:

and indeed $\frac{1}{8}+\frac{3}{8}+\frac{3}{8}+\frac{1}{8}=1$.

Example. Roll a die twice and let $X$ be the sum of the two rolls. The sample space has $36$ equally likely ordered pairs, and counting those summing to each value:

The numerators sum to $36$, as they must. Therefore $7$ is the most likely total, with probability $\frac{1}{6}$.

9.3.2 The mean and variance of a discrete random variable

Random variables let us do arithmetic on outcomes. The two numbers we care about most are the weighted average of the values, and a measure of how spread out they are.

Basically, $E(X)$ is the long-run average value of $X$ if the experiment were repeated many times, with each possible value weighted by how often it occurs. The expected value need not be a value $X$ can actually take — the expected number of heads in three tosses is $\frac{3}{2}$, which is impossible on any single trial.

Example. Toss a coin three times and let $X$ count the heads. Using the table above,

$$\begin{array}{rl}E(X)& =0\times \frac{1}{8}+1\times \frac{3}{8}+2\times \frac{3}{8}+3\times \frac{1}{8}\\ & =\frac{0+3+6+3}{8}=\frac{12}{8}=\frac{3}{2}.\end{array}$$

Therefore $E(X)=\frac{3}{2}$, which is exactly what symmetry suggests: half of three tosses.

This is more useful than it looks: it says you can compute $E(g(X))$ without first finding the distribution of $g(X)$, which would usually be a nuisance. In particular $E(X^2)={\sum }_k{x}_k^2{p}_k$.

A warning that is worth a lot of marks: in general $E(g(X))\ne g(E(X))$. For instance with the coin example, $E(X^2)=\frac{0+3+12+9}{8}=3$ while $(E(X){)}^2=\frac{9}{4}$. The two are only equal when $g$ is linear.

Basically, the variance is the average squared distance from the mean; squaring stops positive and negative deviations from cancelling. The standard deviation then undoes the squaring so that the answer is back in the original units, which is why it is the more interpretable of the two.

Proof. Write $\mu =E(X)$. Expanding the square,

$$\begin{array}{rl}\mathrm{Var}(X)& =\sum _k(x_k-\mu {)}^2{p}_k\\ & =\sum _k(x_k^2-2x_k\mu +{\mu }^2)p_k\\ & =\sum _k{x}_k^2{p}_k-2\mu \sum _k{x}_k{p}_k+{\mu }^2\sum _k{p}_k\\ & =E(X^2)-2\mu \cdot \mu +{\mu }^2\cdot 1\\ & =E(X^2)-(E(X){)}^2.■\end{array}$$

This is the formula to use for hand calculation, since it needs only two sums rather than a subtraction inside every term. As a free sanity check, $\mathrm{Var}(X)\ge 0$ always; if your computation gives a negative variance you have made an arithmetic error, most often by squaring the mean before rather than after averaging.

Example. Toss a coin three times and let $X$ count the heads. We found $E(X)=\frac{3}{2}$ and $E(X^2)=3$, so

$$\mathrm{Var}(X)=3-{\left(\frac{3}{2}\right)}^2=3-\frac{9}{4}=\frac{3}{4}.$$

Therefore $\mathrm{Var}(X)=\frac{3}{4}$ and $\mathrm{SD}(X)=\frac{\sqrt{3}}{2}\approx 0.866$.

Example. A random variable $X$ has the distribution

Find its mean, variance and standard deviation.
First check the probabilities sum to $0.2+0.06+0.2+0.04+0.5=1$. Then

$$\begin{array}{rl}E(X)& =0(0.2)+1(0.06)+2(0.2)+4(0.04)+7(0.5)\\ & =0+0.06+0.4+0.16+3.5\\ & =4.12,\\ E(X^2)& =0^2(0.2)+1^2(0.06)+2^2(0.2)+4^2(0.04)+7^2(0.5)\\ & =0+0.06+0.8+0.64+24.5\\ & =26.0,\end{array}$$

so

$$\mathrm{Var}(X)=26.0-(4.12{)}^2=26.0-16.9744=9.0256,$$

and $\mathrm{SD}(X)=\sqrt{9.0256}\approx 3.004$. Therefore the values of $X$ sit on average about $3$ away from the mean of $4.12$. (The printed course notes give the variance here as $9.256$; that is a typo — $26-4.12^2=9.0256$, and their own conclusion that the standard deviation is roughly $3$ matches the corrected figure.)

Proof. For the mean,

$$E(aX+b)=\sum _k(a{x}_k+b)p_k=a\sum _k{x}_k{p}_k+b\sum _k{p}_k=aE(X)+b.$$

For the variance, using the mean result,

$$\begin{array}{rl}\mathrm{Var}(aX+b)& =E\left(\right(aX+b-E(aX+b){)}^2)\\ & =E\left(\right(aX+b-aE(X)-b{)}^2)\\ & =E\left(a^2\right(X-E(X){)}^2)\\ & =a^2\mathrm{Var}(X),\end{array}$$

and the third statement follows by taking square roots. $■$

Notice that $b$ disappears from the variance entirely. Shifting every value by a constant moves the mean but does not change the spread, which is exactly what a measure of dispersion ought to do. The $a^2$ rather than $a$ is because variance is measured in squared units.

9.4 Special distributions

A handful of distributions turn up so often that they are worth studying once and for all. Both in this section arise from the same setup.

All three conditions matter, and questions are frequently designed so that one of them fails. Drawing balls without replacement, for instance, is not a Bernoulli process, because $p$ changes from trial to trial and the draws are not independent.

9.4.1 The binomial distribution

Proof. Any particular sequence of $n$ trials containing exactly $k$ successes and $n-k$ failures has probability $p^k{q}^{n-k}$, since the trials are independent and so the probabilities multiply. The number of such sequences is the number of ways to choose which $k$ of the $n$ trials are the successes, namely $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$. Summing over these disjoint outcomes gives the result. $■$

Basically, the $p^k{q}^{n-k}$ is the probability of one particular pattern and the $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ counts how many patterns there are. The binomial coefficient is exactly what students leave out — if the question asks for a specific order ("heads then tails then heads") there is no coefficient, but if it asks for a count ("two heads in three tosses") there is.

As a check that this is a distribution at all, the binomial theorem gives

$$\sum _{k=0}^n\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k{q}^{n-k}=(p+q{)}^n=1^n=1,$$

which is also where the name comes from.

Example. Roll a die $12$ times and let $X$ be the number of sixes. The rolls are identical and independent with $p=\frac{1}{6}$, so $X\sim B(12,\frac{1}{6})$ and

$$P(X=2)=\left(\genfrac{}{}{0ex}{}{12}{2}\right){\left(\frac{1}{6}\right)}^2{\left(\frac{5}{6}\right)}^{10}=66\times \frac{1}{36}\times {\left(\frac{5}{6}\right)}^{10}\approx \mathrm{0.296.}$$

Therefore about a $29.6\%$ chance of exactly two sixes.

Example. Ask $n=40$ people whether today is their birthday, and let $X$ count the yes answers. Ignoring leap years and twins, $X\sim B(40,\frac{1}{365})$, and the probability that nobody says yes is

$$P(X=0)=\left(\genfrac{}{}{0ex}{}{40}{0}\right){\left(\frac{1}{365}\right)}^0{\left(\frac{364}{365}\right)}^{40}={\left(\frac{364}{365}\right)}^{40}\approx \mathrm{0.896.}$$

Therefore there is about a $10.4\%$ chance that at least one person has a birthday today. Compare this with the birthday problem of Section 9.2.2, which asked a completely different question — there we compared people with each other (many pairs), here we compare each person against one fixed day. The two are constantly confused.

Basically, each trial contributes on average $p$ successes and variance $pq$, and with $n$ independent trials these simply add. The mean $np$ is exactly the intuitive answer: rolling a die $12$ times, you expect $12\times \frac{1}{6}=2$ sixes.

Example. Toss a coin $n=3$ times and let $X$ count the heads. Then $X\sim B(3,\frac{1}{2})$, so

$$E(X)=3\times \frac{1}{2}=\frac{3}{2},\mathrm{Var}(X)=3\times \frac{1}{2}\times \frac{1}{2}=\frac{3}{4},$$

which agrees exactly with the values we computed by hand from the distribution table earlier — a satisfying check on both.

Example. Roll a die $12$ times. Then $E(X)=12\times \frac{1}{6}=2$ sixes, with $\mathrm{Var}(X)=12\times \frac{1}{6}\times \frac{5}{6}=\frac{5}{3}$ and $\mathrm{SD}(X)\approx 1.29$. So two sixes is typical, and anything from about $1$ to $3$ would be unremarkable.

9.4.2 Geometric distribution

Instead of fixing the number of trials and counting successes, we can fix the number of successes at one and count the trials.

Proof. The event $\{X=k\}$ consists of exactly one outcome: the first $k-1$ trials all fail and the $k$​th succeeds. By independence its probability is $q^{k-1}p$. $■$

Note that in principle a success might never occur, but that event has probability ${\lim }_{k\to \infty }{q}^k=0$ when $p>0$, so we discard it. As a check the probabilities sum correctly, using the geometric series from Section 4.4:

$$\sum _{k=1}^{\infty }{q}^{k-1}p=p\sum _{j=0}^{\infty }{q}^j=\frac{p}{1-q}=\frac{p}{p}=1.$$

Be careful about the convention. Here $X$ counts all trials including the successful one, so $X\ge 1$. Some texts instead count only the failures before the success, giving $P(X=k)=q^{k}p$ for $k\ge 0$ and a mean one smaller. Always check which convention a question uses; this off-by-one is the standard trap.

Example. Toss a coin until a head appears and let $X$ count the tosses. Then $X\sim G(\frac{1}{2})$, and the probability of needing exactly seven tosses is

$$P(X=7)={\left(\frac{1}{2}\right)}^6\times \frac{1}{2}=\frac{1}{2^7}=\frac{1}{128}\approx 0.8\%.$$

Proof. The event $\{X>n\}$ says the first $n$ trials all failed, which by independence has probability $q^n$. The rest follows by complementing. $■$

That is a genuinely useful shortcut — the tail probability requires no summation at all, so questions about "how many trials might it take" should always be attacked through $P(X>n)$ rather than by summing the distribution.

Example. Roll a die until a six appears, and let $X$ count the rolls. Then $X\sim G(\frac{1}{6})$, and

$$\begin{array}{rl}P(X\le 4)& =1-{\left(\frac{5}{6}\right)}^4=1-0.4823\approx 52\%,\\ P(X\le 6)& =1-{\left(\frac{5}{6}\right)}^6=1-0.3349\approx 67\%,\\ P(X>7)& ={\left(\frac{5}{6}\right)}^7\approx 28\%.\end{array}$$

Therefore you have roughly an even chance of rolling a six within four attempts, and a better-than-a-quarter chance of still waiting after seven.

Proof (of the mean). Using term-by-term differentiation of power series,

$$\sum _{k=1}^{\infty }k{q}^{k-1}=\frac{d}{dq}\sum _{k=0}^{\infty }{q}^k=\frac{d}{dq}\left(\frac{1}{1-q}\right)=\frac{1}{(1-q{)}^2},$$

valid for $|q|<1$. Hence

$$E(X)=\sum _{k=1}^{\infty }k{q}^{k-1}p=\frac{p}{(1-q{)}^2}=\frac{p}{p^2}=\frac{1}{p}.■$$

This is a nice illustration of Chapter 4 doing real work in a statistics problem.

The mean is extremely intuitive: if a success happens one time in $p$, you expect to wait $\frac{1}{p}$ trials for it.

Example. Toss a coin until a head appears: $p=\frac{1}{2}$, so $E(X)=2$ tosses on average. Roll a die until a six appears: $p=\frac{1}{6}$, so $E(X)=6$ rolls on average, with $\mathrm{Var}(X)=\frac{5/6}{1/36}=30$ and $\mathrm{SD}(X)\approx 5.5$. Note how enormous that standard deviation is relative to the mean — waiting times are highly variable, which is why "it took me 20 rolls" is not really surprising.

9.4.3 Sign tests

We finish the discrete material with a first taste of statistical inference. Suppose we have independent observations of some quantity and want to know whether they differ systematically from a fixed reference value. A sign test answers this using nothing but the binomial distribution:

• Count the observations strictly greater than the target value ("+").
• Count the total that are strictly greater or strictly smaller (discarding exact ties).
• Compute the probability of seeing at least as many "+" as observed, assuming "+" and "−" were equally likely. This is the tail probability.

The logic is a proof by contradiction in probabilistic clothing. We assume there is no real effect (so $p=\frac{1}{2}$), compute how surprising our data would be under that assumption, and reject the assumption if the data are too surprising. In this course we call a tail probability below $5\%$ significant.

Example. A new variety of corn yields, in bushels per acre, on $15$ plots:

The variety currently in use yields $110$ bushels per acre. Is the new variety better?
Counting, the yield exceeds $110$ on $12$ plots and falls below on $3$; there are no ties, so all $15$ observations count. Under the assumption that the new variety is no better, a plot is equally likely to exceed or fall short of $110$, so the number $X$ of "+" plots satisfies $X\sim B(15,\frac{1}{2})$. The tail probability is

$$\begin{array}{rl}P(X\ge 12)& =\sum _{k=12}^{15}\left(\genfrac{}{}{0ex}{}{15}{k}\right){\left(\frac{1}{2}\right)}^k{\left(\frac{1}{2}\right)}^{15-k}\\ & =\frac{1}{2^{15}}\left[\left(\genfrac{}{}{0ex}{}{15}{12}\right)+\left(\genfrac{}{}{0ex}{}{15}{13}\right)+\left(\genfrac{}{}{0ex}{}{15}{14}\right)+\left(\genfrac{}{}{0ex}{}{15}{15}\right)\right]\\ & =\frac{455+105+15+1}{32768}\\ & =\frac{576}{32768}\\ & \approx 1.76\%.\end{array}$$

Since $1.76\%<5\%$, it is quite unlikely we would have seen $12$ of $15$ plots above the old yield if the new variety were no better. Therefore we conclude the new variety has improved yield.

Two warnings about what this does and does not show. First, the test uses only the signs, throwing away the magnitudes — which makes it robust but not very powerful. Second, a tail probability of $1.76\%$ is not "the probability that the new variety is no better"; it is the probability of data this extreme given that it is no better. Reversing those two is exactly the conditional-probability error from Section 9.2.3.

9.5 Continuous random variables

So far our random variables have taken countably many values. But many quantities — a time, a height, a wage — vary continuously, and for these the whole framework needs adjusting.

The essential difficulty: if $X$ is the exact time of day, uniformly distributed over $24$ hours, what is $P(X=\text{3pm exactly})$? There are uncountably many instants, all equally likely, so any positive probability would sum to infinity. The only consistent answer is $0$.

The density $f(x)$ is NOT a probability, and it may exceed $1$. Probability is area under the density, not the height of it. This is the single biggest conceptual jump in the chapter, and the two defining conditions are the exact analogues of "probabilities are non-negative" and "probabilities sum to $1$".

An immediate consequence is that

$$P(X=a)={\int }_a^{a}f(x)dx=0$$

for every single value $a$. So for a continuous random variable, $P(X\le a)=P(X<a)$, and endpoints never matter — which is a relief, because it means you never have to fret about $<$ versus $\le$. For a discrete variable that distinction is critical, so keep track of which kind you are dealing with.

Proof. The first is the fundamental theorem of calculus applied to $F(x)={\int }_{-\infty }^{x}f(t)dt$; the second is additivity of the integral. $■$

Example. At a random point during the day we note the time, ignoring the date. Model this by $X$ uniformly distributed on $[0,24)$, so the density is constant: $f(x)=c$ on $[0,24)$ and $0$ elsewhere. Find $c$, the cdf, and the probability the time is between 9am and 5pm.
The density must integrate to $1$:

$${\int }_0^{24}cdx=24c=1⟹c=\frac{1}{24}.$$

The cdf is $F(x)={\int }_0^x\frac{1}{24}dt=\frac{x}{24}$ for $0\le x<24$ (and $0$ below, $1$ above). Hence

$$P(9\le X\le 17)=F(17)-F(9)=\frac{17}{24}-\frac{9}{24}=\frac{8}{24}=\frac{1}{3}.$$

Therefore the probability is $\frac{1}{3}$ — as it must be, since that window is $8$ of the $24$ hours.

Example. A continuous random variable has density $f(x)=c{x}^2$ on $[0,3]$ and $0$ elsewhere. Find $c$ and $P(1\le X\le 2)$.
Normalising,

$${\int }_0^{3}c{x}^{2}dx=c{\left[\frac{x^3}{3}\right]}_0^3=9c=1⟹c=\frac{1}{9}.$$

Then

$$P(1\le X\le 2)={\int }_1^2\frac{x^2}{9}dx=\frac{1}{9}{\left[\frac{x^3}{3}\right]}_1^2=\frac{1}{27}(8-1)=\frac{7}{27}.$$

Therefore $c=\frac{1}{9}$ and the probability is $\frac{7}{27}\approx 0.259$. Finding the normalising constant first is always step one; a density that does not integrate to $1$ is not a density.

9.5.1 The mean and variance of a continuous random variable

Every formula from the discrete case carries over with sums replaced by integrals and $p_k$ replaced by $f(x)dx$.