Nonresponse Bias Analysis in a Survey of Banks

1. Nonresponse Bias Analysis in a Survey of Banks Carl Ramirez U.S. Government Accountability Office ramirezc@gao.gov DRAFT - not for publication

2. Overview • Describe NR bias analysis techniques and results in one survey of small, minority-owned financial institutions. • Make observations about NR bias analyses in small population surveys • Describe emerging NR bias analysis practices at one US Federal research agency DRAFT - not for publication

3. Survey of 196 Minority-Owned Banks • Census of all institutions defined by at least one regulator as minority-owned as of March 2006. • Web survey, mail/fax options upon request • President/CEO was targeted respondent • 76% unit response rate (AAPOR RR2). • Key estimates: Awareness of and attendance at regulator programs, rating of regulator efforts to preserve minority ownership, financial outlook DRAFT - not for publication

4. Survey Fieldwork • Field period: March 14 to April 28 – 6 weeks & 3 days • Contacts: • Precontact call in Feb. to determine eligibility and get email • Prenotification email in early March • Survey cover email March 14 • 2 email NR followups • National Bankers Association endorsement contacts • Multiple phone NR followups by program analysts • Final reminder and closeout emails in late April DRAFT - not for publication

5. NR Bias Analysis Typology(Groves & Brick) • Compare survey estimates to other benchmarks • Compare R’s to NR’s on auxiliary variables (frame, external data, fieldwork observations, etc.) • Examine variation within respondents (subgroups, converted nonrespondents, early/late responders) • Alter weighting adjustment DRAFT - not for publication

6. Unique Aspects of Establishment Survey NR Bias Analyses • More longitudinal sample designs = previous response benchmark opportunities • Richer administrative data = more benchmarks and survey variable correlates • Smaller populations and subgroups = potential for examining individual nonrespondents DRAFT - not for publication

7. NR Bias Evaluation Methods Applied to Survey of Banks • Compare respondents to sample on key variables • Compare response rates of subgroups defined by frame information, related to survey variables of interest • Level of Effort (Time of Return) • Subsample of nonrespondents converted with high-effort followup DRAFT - not for publication

8. Respondents vs. Entire Sample on key auxiliary variable: Regulator DRAFT - not for publication

9. Respondents vs. Sample: Minority Type DRAFT - not for publication

10. Respondents vs. Sample: Size DRAFT - not for publication

11. Differences in Response Rate by Regulator of Bank DRAFT - not for publication

12. Differences in Response Rate by Minority Type DRAFT - not for publication

13. Differences in Response Rate by Profitability DRAFT - not for publication

14. Level of Effort Analysis(Time of Return) DRAFT - not for publication

15. Analysis of Respondents with other NR Characteristics • Comparison of low-salience (low response propensity) respondents to all other respondents: 12 banks not considered by primary regulator as “minority owned” but so designated by one of the other 3 regulators. • Removing these respondents changed key estimates by 1-2% , none by more than 4-5% DRAFT - not for publication

16. Micro-Level Examination of Nonrespondents • Small sample allowed in-depth study of individual banks not responding • 4 explicit refusals – known reasons appeared largely unrelated to key measures • Personal contacts for NR followup yields known reason for delay DRAFT - not for publication

17. Emerging GAO practice on NR Bias Analysis? • Groves & Brick typology • Suggests hierarchy of methods on 2 dimensions: • Prefer use of data related to survey variable over comparison of subgroups or time of return analysis • Prefer methods using data available for entire population over portion of population or sample • If conduct analyses, describe in report DRAFT - not for publication