To Documents

Introduction to Bootstrapping

Bootstrap Methods

• Bootstrap methods are a set of related techniques that involve resampling the original sample of data.
• The first paper on bootstrap methods was published in 1979 by Bradley Efron of Stanford University.
• A simple application of the bootstrap is to estimate the accuracy of a sample statistic, such as the sample mean.
• We know of two methods for estimating the accuracy of the sample mean:
  
  1. Repeat the experiment multiple times and compute SD+ for the the sample means of all the experiments.
  2. Only conduct the experiment once, but use
           SE_ave = SD+ / √x
     as an estimate of the accuracy of the mean.
• A third choice is to use a bootstrap method to estimate to obtain an 80% confidence interval of the sample mean.
• Example 1: Suppose that our sample is
        x₁=3   x₂=8   x₃=2   x₄=7   x₅=4
  We estimate the 80% confidence interval by resampling our original sample.
• Draw 14 random samples of size 5 with replacement. A low-tech method for resampling is
  rolling a die to determine which of the five x_is to select. Ignore roll outcomes of 6. For example, if the outcomes of the five die rolls are 4, 5, 3, 1, 1 (ignore outcomes of 6), the resample is x₄=7, x₅=4, x₃=2, x₁=3, x₁=3. Here are the results:
  

  Resample  1:  7 4 2 3 3  xbar: 3.8
  Resample  2:  3 7 2 4 4  xbar: 4.0
  Resample  3:  2 3 3 7 3  xbar: 3.6
  Resample  4:  4 8 2 3 3  xbar: 4.0
  Resample  5:  2 3 7 8 4  xbar: 4.8
  Resample  6:  2 4 2 8 3  xbar: 3.8
  Resample  7:  7 2 2 3 8  xbar: 4.4
  Resample  8:  7 2 2 7 3  xbar: 4.2
  Resample  9:  3 2 4 4 3  xbar: 3.2
  Resample 10:  3 8 2 7 2  xbar: 4.4
  Resample 11:  8 4 7 4 7  xbar: 6.0
  Resample 12:  4 8 4 4 8  xbar: 5.6
  Resample 13:  8 8 3 2 4  xbar: 5.0
  Resample 14:  8 7 4 7 2  xbar: 5.6
  

• Now find the 10th and 90th percentiles of the sorted xbars:
  
  

  i	1	2	3	4	5	6	7
  100*i/(n+1)	6.67	13.33	20.00	26.67	33.33	40.00	46.67
  xbari	3.2	3.6	3.8	4.0	4.0	4.0	4.2
  i	8	9	10	11	12	13	14
  100*i/(n+1)	53.33	60.00	66.67	73.33	80.00	86.67	93.33
  xbari	4.4	4.4	4.8	5.0	5.6	5.6	6.0

• According to this table, the 6.67 percentile is 3.2 and the 12.33 percentile is 3.6. Since 10.00 is halfway between 6.67 and 12.33, using linear interpolation, the 10th percentile is halfway between 3.2 and 3.6: 3.4. Also, 90.00 is halfway between 86.67 and 93.33, so the 90th percentile is halfway between 5.6 and 6.0: 5.8. Therefore an 80% confidence interval for x is [3.4, 5.8]
• Compare this confidence interval with 80% confidence interval (0.1 upper tail probability) obtained using the t-table with 5 degrees of freedom:
        x = 4.8   SD+ = 2.59 
        SE_ave = SD+ / √n = 2.59 / √5 = 1.16
        -1.533 ≤ t ≤ 1.533
        -1.533 ≤ (4.8 - x) / 1.16 ≤ 1.533
        3.02 ≤ x ≤ 6.58
  Thus the 80% confidence interval for x is [3.02, 6.58] .
• Now lets try this with SPSS, using 1,000 resamplings instead of only 14.

Variance vs. Bias

• Another name for a sample statistic is an estimator.
• The sample mean x is an estimator of the population mean.
• These diagrams show visually of the difference between variance and bias.
• Variance is variation from observation to observation. Estimators with low variance are better than estimators with large variance.
• Bias is the systematic difference between the estimator and population parameter which is being estimated.
• If T is the estimator and θ is the population parameter being estimated, T is unbiased for θ if E(T) = θ
• For example, E(x) = μ, so the sample mean x is unbiased for the population mean μ
• E(SD+²) = σ², so the sample variance SD+ is unbiased for the population variance σ².
• If y = ax + b is the true regression equation and a^{^} and b^{^} are the estimated parameters, then E(a^{^}) = a and E(b^{^}) = b: a^{^} and b^{^} are unbiased for a and b, respectively.
• One of the assumptions for the Ideal Measurement Model and the Simple Linear Regression Model is that the residuals are unbiased.
• However, in practice, bias may exist. Bootstrap methods can give us an estimate of the bias.
• Example: 2.Time to Start a Business Example
  Find a bootstrapped 95% confidence interval, for the sample mean, the 5% trimmed mean, and the median for the time to start a business using the data.
  Use the dataset time-to-start-bus.xslx.
  
        Use SPSS to construct a histogram and a normal plot of the Time To Start a Business Example.
  
        Select Analyze >> Descriptive Statistics >> Explore.
        In the Explore Dialog, drag Time into the Dependent List Box.
        Click the Bootstrap Button; In the Bootstrap Dialog, check the Perform Bootstrap checkbox.
        Click OK.

Robustness

• An estimator is robust if it gives an accurate value even if the model assumptions are violated.
• Another name for a robust estimator is a sturdy estimator.
• For example, if a random sample comes from an N(μ, σ²) population, x is the most efficient estimator of μ.
• However, even if there are as little as 5% outliers, x will not be as efficient as the median or the 5% trimmed-mean.
• (The 5% trimmed-mean is actually the most efficient estimator if there are 5% outliers.)
• Recall that the standard one sample t-test assumes that the population is normally distributed.
• The one sample t-test is fairly robust to departures from the normality assumption.
• However, for highly skewed distributions or distributions with very thick tails, the t-test is not very accurate.
• The bootstrap version of the one-sample test does better with these sorts of distributions.
• Example: 3. Comparing GPA Scores by Sex
  
        Use SPSS to construct a histogram and a normal plot of the GPA scores separately by sex.
  
        Perform a bootstrapped independent 2-sample t-test using the data in gpa.xslx
        Analyze >> Compare Means >> Independent Samples T-Test.
        Drag GPA into the Test Variable(s) Box;
              drag sex into the Grouping Variables Box;
              define the groups as Group 1 as 1 and Group 2 as 2.
        Click the Bootstrap Button; check the Perform Bootstrap Button.
        Click OK.