To Notes

IT 403 -- Oct 12, 2016

Review Exercises

1. What is the regression fallacy?
   Ans: In a pretest/posttest situation, the regression fallacy is the mistaken notion that if someone does well on the pretest, he or she should do equally well on the posttest; if that person does poorly on the pretest, he or she should do equally poorly on the posttest. In fact, unless the pretest and posttest scores are perfectly correlated (r = 1), someone that obtains a pretest score k SD_xs above x, will, on the average, obtain a posttest score r × k SD_ys above the average. In other words, the posttest score will be lower than the pretest score, on the average. The situation is the opposite if the person obtains a below average score on the pretest. If the pretest score is k SD_xs below x, then, on the average, the posttest score will be r × k SD_ys below the average. In other words, the posttest score will be higher than the pretest score, on the average.
2. What is the RMSE for a regression model?
   Ans: The root mean squared error (RMSE is the standard deviation of the residuals. In particular, RMSE is the SD of the residuals in a thin rectangle that contains a specific x value. RMSE is defined as √(1 - r²). If r < 1, r < SD_y
3. Find the errors with this probability distribution function:
   
   

   Outcome	Probability
   3	60%
   5	30%
   6	0%
   7	-10%
   9	110%

   
   Ans: Probabilities cannot be negative; probabilties cannot be more than 100%; the probabilities in the table must sum to 100%.

Learning Outcomes for Today

• Know the definitions of these terms:
  Sample Space, A Priori Probability, Empirical, Subjective Probability, Probability Distribution, Random Variable, Expected Value, Risk.
• Be able to:
  
  1. Define a probability distribution in three ways: a priori, empirical and subjective.
     
  2. Define a probability distribution, given a sample space.
  3. Compute the expected value of a random variable.
  4. Understand the difference between independent and mutually exclusive events.
  5. Use the multiplication rule for independent events.
  6. Use the addition rule for mutually exclusive events.
  7. Find the expected value of a constant multiple of random variables.
  8. Find the expected value of a sum of random variables.
  9. Use n factorial (n!) to find the number or ways to arrange n objects.

Probability

• The sample space is the set of all possible outcomes in a probability situation. We use the greek letter Ω (big omega) to represent the sample space.
• Example 1:   Flip a coin once: Ω = {head, tail} = {H, T}.
• Example 2:   Roll a die once: Ω = {1, 2, 3, 4, 5, 6}.
• Example 3:   An urn contains 3 red balls and 4 green balls. Choose a random ball from the urn: Ω = {red₁, red₂, red₃, green₁, green₂, green₃, green₄}. The subscripts are used to distinguish between the balls of the same color.
• Example 4:   Choose a random part on an assembly line and test it: Ω = {defective, nondefective}.
• An event is a subset of the sample space. The event A occurs if the outcome of a probability experiment is a member of the event subspace A.
  Example 5: Ω = {HHH, THH, HTH, HHT, TTH, THT, HTT, TTT}. Let A be the event that 2 heads are obtained out of 3 coin tosses. In other words, A = {THH, HTH, HHT}.
• The probability that an event occurs is a value between 0 and 1 that indicates how likely that event is to occur; 0 means that the event is impossible, 1 means that the event is certain to occur.
• Three common ways of determining probabilities:
  
  1. Theoretical or A Priori Probability
     Assume that all outcomes are equally likely. Suppose you are interested in the probability that the event A occurs.
     P(A) = # ways A can occur / Total # of events
     Example 6:   What is the probability of getting one head out of two coin flips?
     Sample space Ω = {HH, HT, TH, TT}
     A = Event of getting exactly one head = {HT, TH}
     P(A) = # of events in A / # events in S = 2 / 4 = 0.5
  2. Empirical Probability
     Perform an experiment, repeated a large number of times. Count the number of times s the event A occurs and the total number of times n the experiment is repeated.
     P(A) = s / n
     Example 7: Flip a coin 1 million times. 499,523 heads were obtained. The probability of obtaining a head is 499,523 / 1,000,000 = 0.499523.
  3. Subjective Probability
     Each person uses personal intuition or judgement.
     What is the probability that I will need an umbrella today?
     Las Vegas oddsmakers combine subjective probabilities for many sources when determining betting odds.
     Example 8:   What is the probability that the Cubs will win the World Series in 2017?
     A subjective probability can be defined as a fair bet, that is a bet for which you would be willing to take either side.
• No matter how probabilities are chosen, a probability distribution is a table that assigns a probability to each outcome in the sample space:
  

  Probability Distribution
  Outcome	Probability
  HH	0.25
  TH	0.25
  HT	0.25
  TT	0.25

  
  When flipping 2 coins, what is the probability of obtaining 1 head?
  Ans: There are two ways in the sample space of obtaining one head: HT, TH. Therefore the probability of obtaining one head is 0.25 + 0.25 = 0.50.
• Rules for Probabilities:
  
  1. Each probability is between 0 and 1, inclusive:
          If s ∈ Ω, then 0 ≤ P(s) ≤ 1.
  2. The probability that some event in the sample space occurs is 1:
          P(Ω) = 1
  3. If p is the probability that the event occurs, then 1 - p is the probability that the event does not occur.
          P(A^c) = 1 - P(A)

Practice Problems

1. What is wrong with this argument? Either the Cubs will win the World Series or they won't. Therefore the probability that the Cubs will win the World Series is 50%.
   Ans: Just because there are two outcomes doesn't mean they are equally likely. Some outcomes are very likely; they have probabilities close to 1. Some outcomes are unlikely; they have outcomes close to 0. Other outcomes have probabilities close to 0.5.
2. What is wrong with this strategy? Double down after after each loss. Eventually you win and recoup your losses. For example:
   -1 - 2 - 4 - 8 - 16 + 32 = 1.
   Now start over with 1 and repeat the double down strategy.
   Ans: The problem with this strategy is that, eventually, you will either reach the casino betting limit or you will run out of money.
3. A bookmaker offers 7 to 4 odds that the Cubs will win the World Series. If this is a fair bet, what is the probability p that the Cubs will win the World Series?
   Ans: The expected amount that you win is 7p + (-4)(1 - p); this expression is 0 because we are assuming that is a fair bet. Now solve for p:
        7p + (-4)(1 - p) = 0
        7p - 4 + 4p = 0
        11p = 4
        p = 4 / 11 = 0.364 = 36%.

Random Variables

• A random variable is a function from the sample space to the set of real numbers.
  
  Example 9:
  

  Outcome	Real Number	Probability
  HH	2	0.25
  TH	1	0.25
  HT	1	0.25
  TT	0	0.25

  
  This table defines a function that assigns HH → 2, TH → 1, HT → 2, TT → 0.
  
  A random variable can also be described directly with a probability distribution:
  

  Outcome	Probability
  x1	P(x1)
  x2	P(x2)
  ...	...
  xk	P(xk)

  
  In the case of a random variable that counts the number of heads obtained by flipping two coins:
  

  x	Probability
  0	0.25
  1	0.50
  2	0.25

  
  
• Example 10:   The random variable x is the number of heads obtained out of three coin flips.
  

  x	Probability
  0	0.125
  1	0.375
  2	0.375
  3	0.125

  
  
• Example 11:   Pick a person at random. That person's height is a random variable.
• Example 12:   In the case of a continuous random variable, where P(z) = 0, for every specific z value, we must use a probability density instead of a probability distribution. Here is the familiar probability density for a standard normal random variable.
  
  
  If z is a normal random variable with mean 0 and variance 1 (z ∼ N(0, 1)), what is the probability that z ∈ [-0.7, 1.3]? Ans: 0.661 = 66%.
• A random variable that has only two outcomes 0 and 1 is called a Bernoulli random variable; 1 denotes success and 0 denotes failure. Here is the probability table for a Bernoulli random variable:
  

  x	Probability
  0	1 - p
  1	p

  
  
• Some examples of Bernoulli random variables: flipping a coin (T = 0, H = 1), rolling an ace with a single die (non-ace = 0, ace), shooting a basketball free throw (failure = 0, success = 1), choosing a part from an assembly line to inspect (non-defective = 0, defective = 1).

Expected Value

• The expected value of a random variable is its theoretical average, based on the probabilities of its outcomes.
• The expected value of a random variable x can be calculated using the probability table.
  

  Outcome x	Probability
  x1	p1
  x2	p2
  ...	...
  xk	pk

  
  The expected value is:
       E(x) = x₁ p₁ + x₂ p₂ + ... + x_k p_k
• The expected value is the weighted average of the outcomes of the random variable. You don't need to divide by the sum of the weights because the sum of the probabilities is 1.
• The expected value is also called the risk.
  Example 13:   Suppose the probability that the Bears will the Super Bowl next year is 1 / 20 = 0.05. You make a bet with your friend. If the Bears will you get $100. If the bears do not will, you pay him $10. What is the expected value of this bet?
  

  Payoff	Probability
  -10	0.95
  100	0.05

  Note: the payoff of 30 in the table is negative because you are losing $10.

  Expected value = (-10) × 0.95 + 100 × 0.05 = -4.5

  The conclusion is that you will lose $4.50 on the average every time you make this bet.

  Example 14:   Flip a coin 3 times. You win $10 every time 3 heads come up, you lose $1 every time 1 head comes up and you lose $5 every time 0 heads come up. Do do not lose or win anything if 2 heads come up. Compute the expected value.
  Ans: Here is the payoff table:
  

  Payoff	Probablity
  10	1/8
  0	3/8
  -1	3/8
  -5	1/8

  
  E(x) = 10(1/8) + 0(1/8) + (-1)(3/8) + (-5)(1/8) = 2/8 = $0.25

  Example 15:   A tropical island has the same probability distribution for the amount of rain every day:
  

  Rainfall	Probability
  0	0.3
  1	0.4
  2	0.2
  3	0.1

  
  What is the expected value for the amount of rain in one day?
  Ans: E(x) = x₁ p₁ + x₂ p₂ + x₃ p₃ + x₄ p₄

                = 0 × 0.3 + 1 × 0.4 + 2 × 0.2 + 3 × 0.1 = 1.1
  Example 16:   You pay $75 per year on a $500,000 house for insurance. The probability that your house is destroyed in a given year is 0.0001.
  

  1. What is the expected value for the insurance company per year?
     Ans: Here is the payout table:
     

     Payout	Probability
     75	0.9999
     -500,000 + 75	0.0001

     
     The expected value is 75 × 0.9999 + (-500,000 + 75) × 0.0001 = 75 × (0.9999 + 0.0001) + (-500,000) × 0.0001 = 75 + (-50) = $25.
  2. What is your risk per year?

     Ans: You pay $75 to the insurance company whether or not your house is destroyed, so the expected value is $-75.

  3. What would your risk be if you did not have insurance?
     Ans: The payout table is
     

     Payout	Probability
     0	0.9999
     -500,000	0.0001

     
     The expected value is 0 × 0.9999 + (-500,000) × 0.0001 = -$50.
  4. What would the "pure premium" to the insurance company be? The pure premium is the premium you would have to pay if the expected value of the insurance company were $0.
     Ans: Solve for p in the following: p (0.9999) + (-500,000 + p) 0.0001 = 0. This gives p = 50.

The Multiplication Rule for Independent Events

• Two events A and B are independent if the occurence of A does not affect the probability that B occurs.
• Example 17:   Flip a coin twice. Define the event A to be getting a head on the first toss. Define the event B to be getting a head on the second toss. The events A and B are independent.
• The Multiplication Rule:   If A and B are independent, then
       P(A and B) = P(A) × P(B)
• Example 18:   Flip a coin 10 times. What is the probability of getting all heads?

  Ans: P(H and H and ... and H (10 heads)) = P(H) × P(H) × ... × P(H) = 0.5¹⁰ = 0.0009766.

• Example 19:   The probability that I will die in a single plane ride is 0.0001. What is my probability of dying in 500 independent plane rides?
  Ans: Let A_i be the event of not dying in the ith plane flight. Then P(A₁ and ... and A₅₀₀) = P(A₁) × ... × P(A₅₀₀) = 0.9999⁵⁰⁰ = 0.951227. Therefore the probability of dying in one of the 500 flights is 1 - 0.9512 = 0.04877 = 4.9%.
  Working from the formula gives us
  1 - (1 - p)ⁿ = 1 - (1 - 0.0001)⁵⁰⁰ = 0.04877 = 4.9%
• Example 20:   The probability of contracting AIDS from a single sexual encounter is about 1 out of 1000. What is the probability of contracting AIDS from 20 independent encounters?
  Ans: 1 - (1 - p)ⁿ = 1 - (1 - 1 / 1000)²⁰ = 0.01981 = 1.98%
• In the real world, determining if two events A and B are independent is difficult. There are many lurking variables that can cause A and B to occur or not to occur.

The Addition Rule for Mutually Exclusive Events

• Two events A and B are mutually exclusive if A and B cannot both occur at the same time.
• Example 21:   "Obtaining a 1 with one roll of a die" and "obtaining a 2 with one roll of a die" are mutually excusive.
• Example 22:   In a baseball game, "getting a hit in one at bat" and "getting a base on balls" in one at bat are mutually exclusive.
• The Addition Rule:   If A and B are mutually exclusive events, then
       P(A or B) = P(A) + P(B)
• Example 23:   If event A is "obtaining a 1 with one die" and B is "obtaining a 2 with one die", since A and B are mutually exclusive, P(A or B) = P(A) + P(B) = 1/6 + 1/6 = 1/3.

The Standard Deviation of a Random Variable

• We already know the definition of the expected value:
  E(x) = x₁ p₁ + ... + x_n p_n
• The textbook uses the symbol μ_x instead of E(x) for the expected value of the random variable x.
• The variance of a random variable is denoted by Var(x) = σ_x²
  σ_x² = E[(x - E(x))²] = (x₁ - E(x))² p₁ + ... + (x_n - E(x))² p_n
• The textbook uses the symbol σ_x² instead of Var(x).
• The standard deviation of a random variable is denoted as σ_x. It is the square root of the variance:

        σ_x = √Var(x) = √(σ_x)

Practice Problems

4. Recall that the expected value for the Rainfall on a Tropical Island example is 1.1 inches. Here is the probability distribution:
   
   

   Rainfall	Probability
   0	0.3
   1	0.4
   2	0.2
   3	0.1

   
   
   Compute the variance and standard deviation of this random variable.
   Ans: (0 - 1.1)² 0.3 + (1 - 1.1)² 0.4 + (2 - 1.1)² 0.2 + (3 - 1.1)² 0.1
   = 1.21 × 0.3 + 0.01 × 0.4 + 0.81 × 0.2 + 3.61 × 0.1 = 0.89
   The standard deviation is sqrt(0.89) = 0.943
5. Compute the variance and standard deviation of a Bernoulli random variable.
   Ans: The expected value of a Bernoulli random variable is 0(1 - p) + 1p = p.

   Variance = (0 - p)² (1-p) + (1 - p)² p = p²(1-p) + (1 - 2p + p²)p
   = p² - p³ + p - 2p² + p³ = p - p² = p(1 - p)
   The standard deviation is the square root of the variance = √p(1 - p)

6. Use your result from Problem 2 to obtain the mean and standard deviation of the number of heads obtained in a single coin flip.
   Ans: E(x) = p = 0.5; σ_x = √0.5(1-0.5) = √0.25 = 0.5

Properties of Random Variables

1. E(cx) = c E(x)
2. E(x + y) = E(x) + E(y)
3. E(x₁ + ... + x_n) = E(x₁) + ... + E(x_n)
4. Var(x) = E(x²) - E(x)²
5. Definition:   x and y are independent if E(xy) = E(x)E(y).
   If x and y are independent, Var(x + y) = Var(x) + Var(y)

Here are the derivations.

Practice Problem

7. Compute the variance and SD of a Bernoulli random variable with the formula Var(x) = E(x²) - E(x)².
        Var(x) = E(x²) - E(x)² = (0² (1 - p) + ... + 1² p) - p²
                 = p - p² = p(1 - p)
   This is the same result that we obtained earlier.

Sums of Random Variables

• Define S = x₁ + ... + x_n, where each x_i has the same expected value and variance, and all of the x_i are independent.
• The expected value of the sum of n outcomes of the random variable x:
  E(S) = E(x₁ + ... + x_n) = E(x₁) + ... + E(x_n) = nE(x)
  

  E(S) = nE(x)

• The variance of the sum of n independent outcomes of the random variable x:
  Var(S) = Var(x₁ + ... + x_n) = Var(x₁) + ... + Var(x_n) = nVar(x)
  

  Var(S) = nVar(x)

• The standard deviation of the sum of n independent outcomes of a random variable:
  σ_S = σ_{x1 + ... + xn} = √[Var(x₁ + ... + x_n)]
      = √[Var(x₁) + ... + Var(x_n) = √nVar(x) = σ_x√n
  

  σS = σx√n

Practice Problems

7. Compute the expected value and standard deviation of the random variable in Practice Problem 1 for 365 days.

   Ans: Recall that E(x) = 1.1 and σ_x = 0.943; E(S) = nE(x) = 365 × 1.1 = 401.5
   σ_S = σ_x√n = 0.943√365 = 18.0

8. Compute the expected value and standard deviation of a Bernoulli random variable (Practice Problem 2) for the sum of n trials.
   Ans: E(S) = nE(x) = np
   σ_S = σ_x √n = √p(1-p) √n = √np(1-p)
9. Suppose that my true percentage of making free throws is 70%. Out of 100 attempts, what is the expected value and standard deviation of the number of free throws made? Find a 95% confidence interval for the number of free throws made.
   Ans: E(S) = np = 100 × 0.7 = 70.
   Var(S) = np(1 - p) = 100 × 0.7 × (1 - 0.7) = 21.
   σ_S = √21 = 4.58.
   The confidence interval is
   [E(S) - 1.96 × σ_S, E(S) + 1.96 × σ_S]
   [70 - 1.96 × 4.58, 70 + 1.96 × 4.58] = [61.02, 78.98].
   

Averages of Random Variables

This section will be discussed on Oct 24.

• The expected value of the average of n outcomes of a random variable:
  E(x) = E[(x₁ + ... + x_n) / n] = E[(x₁ / n + ... + x_n) / n]
      = E(x₁ / n) + ... + E(x_n / n) = E(x₁) / n + ... + E(x_n) / n
      = nE(x) / n = E(x)
  

  E(x) = E(x)

• The variance of the sum of n independent outcomes of a random variable:
  Var(x) = Var[(x₁ + ... + x_n) / n] = Var[x₁ / n + ... + x_n / n]
      = Var(x₁ / n) + ... + Var(x_n / n) = Var(x₁) / n² + ... + Var(x_n) / n²
      = = nVar(x) / n² = nVar(x)
  

  Var(x) = nVar(x)

• The standard deviation of the sum of n independent outcomes of a random variable:
  σ_x = √[Var(x)] = √[Var(x) / n] = σ_x / √n
  

  σx = σx / √n

• Recall that E(x) = E(x). How does σ_x compare to σ_x?

The Law of Averages

• The official name of the Law of Averages is the Law of Large Numbers (LLN). It was first proved by Jakob Bernoulli from Basil, Switzerland in 1713.
• LLN states that if x₁, ... , x_n are independent outcomes of the same random variable x and n is large enough, then the average x is close to the expected value E(x) = E(x) with high probability.
• LLN works by swamping, not by compensation. What does this mean?
  Ans: It means that if the average of outcomes of a random variable is larger than the expected value of the average, there is no reason to expect that if you continue to collect outcomes from the random variable that those new outcomes will be smaller than the expected value of the average to compensate.  Instead it means that by swamping the average with many more outcomes, the new outcomes will be close to the average with high probability, so the overall average will also tend to be closer to the expected value.  Note that this is true with high probability; there will always be a small probability that the overall average is not close to the expected value of the average.
• Experiment: Use SPSS to simulate 100, 1000, and 10,000 tosses of a fair coin. Are the averages obtained consistant with LLN?
  Ans: Here are the results:
  
  

  n	S	x
  1,000	469	0.4690
  10,000	4,925	0.4925
  100,000	49,781	0.4978
  1,000,000	500,107	0.5001

The Central Limit Theorem

• Even if a random variable x is not normally distributed, both the sum S and average x of n independent outcomes of a random variable are approximately normally distributed if n is large. Usually we say that n is large if n > 30.
• Example 24:   Flip a coin 1,000 times and obtain 514 heads. Find a 95% confidence interval for the true value p of obtaining a head. This will give us an idea if the coin is fair.
  Ans: Flipping a coin once is a Bernoulli random variable with p = 1/2, so E(x) = 1/2 and σ_x = 1/2. The sum of n independent Bernoulli random variables S is the number of heads; E(S) = np = 1,000(1/2) = 500 and σ_S = √np = √1,000(1/2) = 22.36. Now a 95% confidence interval for z is -2 ≤ x ≤ 2 (more precisely, -1.96 ≤ z ≤ 1.96. We can solve for p:
       -2 ≤ x ≤ 2
       -2 ≤ (S - np) / σ_S ≤ 2
       -2 ≤ (514 - 1,000p) / 22.36 ≤ 2
       -44.72 ≤ 514 - 1,000p ≤ 44.72
       -44.72 - 514 ≤ - 1,000p ≤ 44.72 - 514
       -558.72 ≤ - 1,000p ≤ -469.72
  0.55872 ≥ p ≥ 0.46972
  This means that the 95% confidence interval is [0.46972, 0.55872] or [47%, 56%].

Factorials and Counting Combinations

• Recall that n! (read n factorial, n shriek, n bang) is defined as
       n! = n × (n-1) × (n-2) × ... × 3 × 2 × 1
• n! is the number of ways of arranging n distict objects.
• 3! = 6. For example, the ways of arranging A, B, and C is
       (ABC)   (ACB)   (BAC)   (BCA)   (CAB)   (CBA)
• 2! = 2. The ways of arranging A and B is
       (AB)   (BA)
• 1! = 1. The ways of arranging the single object A is
       (A)
• What is 0!?

  Ans: There is exactly one way of arranging zero objects: ( ). Thus 0! = 1.