Public Works and Government Services Canada
www.pwgsc.gc.ca
Common menu bar links
Improving Respondent Cooperation for Telephone Surveys
Introduction
In recent years, the Public Opinion Research Directorate (PORD) of Public Works and Government Services Canada (PWGSC) and other departments of the Government of Canada have raised concerns about survey quality issues. In particular, declining response rates in telephone surveys have been a key concern and discussion item at meetings of the Government of Canada's Community of Practice. Industry associations–especially the Marketing Research and Intelligence Association (MRIA)–and the Office of the Auditor General of Canada (November 2005) have expressed similar concerns. This focus on response rates is not unfounded, given the general consensus among survey research practitioners that response rates have been declining over the past few decades (de Leeuw and de Heer, 2001; Groves and Couper, 1998). People are becoming harder to reach and less willing to participate in survey research.
This decline has cast doubt on the validity of data resulting from surveys with lower response rates and has increased the cost of conducting research to reach target response rates. Improving response rates requires a multi-dimensional response that addresses the issue of non-response at different stages of the research process. This set of best practices has been developed to help Government of Canada departments and agencies design and conduct public opinion research (POR) that strives to achieve the highest response rates possible.1
Purpose and Scope
This set of best practices was developed to help maximize response rates in telephone surveys. The focus is on strategies to help ensure that each Government of Canada telephone survey achieves the best response rate possible within the parameters of the study. These best practices incorporate guidelines and procedures that should be considered throughout the research study, from the design phase of the project through the reporting phase. The document is intended for use by departments and agencies for their own review and for discussions with research suppliers. Since the research process in the federal government context is typically a collaboration between PORD, the departmental POR coordinator, the end client and the research supplier, readers will find that some of the areas covered by the best practices may apply only indirectly to their role in a particular study. Research suppliers consider some of these best practices to be standard quality control practices. Others will involve consideration and reflection on the part of the department or agency commissioning the survey.
The Process of Developing the Best Practices
Developing this set of best practices involved undertaking a comprehensive review of relevant literature (see bibliography); contacting industry associations and research institutes; and carrying out a series of consultative interviews and correspondence with POR buyers in the Government of Canada, top field suppliers to the federal government, and key academics in Canada and the United States with expertise related to survey response rates. In total, 26 stakeholders and organizations were consulted to validate and strengthen the best practices. Often, multiple individuals within an organization offered feedback. A detailed discussion of the methodology can be found in Checklist of Best Practices and Assessment of Relative Impact of Best Practices on Response Rates.
Defining Response and Non-Response Rates
Response rate refers to the proportion of people who participated in a survey compared to the actual number of people sampled from the target population. In general terms, it is calculated by dividing the number of people who completed the survey by the number of people selected to participate. Non-response occurs when a unit of the sample does not complete a survey. Typically, non-respondents fall into one of two groups: people who refuse to participate in the survey (refusals) and those who cannot be reached during data collection (non-contacts).
Response Rate Calculations
MRIA is the leading national association for POR professionals in Canada, and its definitions and methods are the ones most widely applied by private industry in Canada. Outside of Quebec, surveys conducted by private industry for the Government of Canada generally use the MRIA method to calculate response rates. MRIA recently adopted a new standard response rate calculation, which has been endorsed by Statistics Canada and l'Association de l'Industrie de la Recherche Marketing et Sociale in Quebec. It comprises two rates: a primary (Empirical Method) response rate and a secondary (Estimation Method) response rate. The Empirical Method should be used to measure data collection efforts, and the Estimation Method should be used during the analysis as a secondary measure to assess the quality of the survey data. Described below is the MRIA Empirical Method, the main response rate calculation.
Using the Empirical Method, the response rate is calculated by dividing the number of responding units by the sum of all in-scope and unresolved units. To understand this calculation, a few terms require definition.
- Unresolved units (U): These are all non-responding units for which eligibility cannot be determined–for example, telephone numbers that result in calls where the phone rings but no one answers. All other telephone numbers are classified as resolved.
- Resolved units: These are divided into in-scope and out-of-scope units.
- Out-of-scope units: These are telephone numbers that are invalid–that is, they are not in service, or they are business numbers in a household survey or vice versa. These numbers are not calculated as part of the response rate.
- In-scope units: These are classified as either responding or non-responding units. Non-responding in-scope units (IS) include people who refuse to participate, who are not available to talk or who face a language barrier in responding.
- Responding units (R): These are all in-scope units (respondents) who provided usable information. They include disqualified or ineligible respondents–that is, those screened out at the beginning of the interview.
MRIA Empirical Method
Response Rate = R/(U+IS+R)
Source: Vue Magazine, June 2006.
To calculate the response rate for a telephone survey, divide the number of respondents by the sum of all units of the sample: unresolved units, in-scope non-responding units and responding units.
Why Are Response Rates Important?
Response rates are an important measurement in survey research because they reflect the level of effort undertaken during data collection and help describe the reliability of the resulting data. Survey non-response can bias samples (and therefore survey data) by making the sample composition substantively different from the target population. Bias, in this instance, refers to the difference between the sampled units and the target population. Just as a randomly selected sample represents the target population, so too must the actual survey respondents. The biasing effect of non-response can be greater as the response rate drops; therefore, survey organizations seek higher response rates to decrease the likelihood of non-response bias. Survey error resulting from non-response, however, will only occur when there are significant differences between respondents and non-respondents. Why? Error resulting from non-response is a function of both the response rate and the extent of differences between respondents and non-respondents. This fact means that low response rates do not necessarily result in low data quality.
As a result of the general decline in response rates, the research community has begun focusing on the validity of data related to low response rates. Numerous studies have been undertaken during the last decade to advance knowledge in this area. Overall, the findings of these studies question the methodological tenet that low response rates necessarily compromise data validity (Visser et al., 1996; Keeter et al., 2000; Curtin et al., 2000; Merkle and Edelman, 2001; Halpenny and Ambrose, 2006). These studies suggest that higher response rates do not necessarily produce more accurate data, and that surveys with low response rates can still provide useful and valid data, other things being equal–for example, provided sample selection and weighting are undertaken carefully. The studies reinforce the premise that survey error resulting from non-response will only occur when respondents differ from non-respondents. The problem for survey researchers is understanding when non-response will not cause survey error and when it will introduce bias that will affect data reliability–that is, under which conditions are respondents and non-respondents most likely to differ?
In the absence of being able to predict when non-response will bias a sample, obtaining the highest response rate possible within the constraints of a particular study is beneficial to all those involved in survey research. Though high response rates are increasingly difficult to achieve, efforts should always be made to maximize response rates. However, efforts to increase response rates should be considered within the context of total survey error; sampling, coverage and measurement errors may all decrease data quality. Any effort to maximize response rates beyond a certain point can be counterproductive if the measures divert resources from these other important sources of potential error. As well, increasing response rates often costs money. Therefore, measures to maximize response rates need to be considered in light of the study budget, timeframe, the way in which the results will be used and the level of accuracy needed. Response rates, in short, should be one consideration among many when undertaking research design.
As illustrated in the diagram below, response rates are only one of numerous areas where error may affect the quality of the survey data.
Total Survey Error
Text description of Total Survey Error is available on a separate page.
Source: Dr. Robert Groves, Practical Tools for Non-response Bias Studies seminar (March, 2006).
What Are Realistic Response Rates?
In Canada, there are no standards for minimum acceptable response rates. In addition, industry does not have a no-response rate threshold that can be used to determine when survey results might be subject to non-response bias (Groves, forthcoming). Realistic response rates will vary depending on the data collection method used–for instance, telephone, online or mail–and the specific parameters of the survey, including budget, time, target population, survey length and sample frame. Typical response rates for most commercial telephone surveys now tend to range from 10% to 20%, although some surveys–such as omnibus studies and political polls–can yield response rates in the single digits (Halpenny and Ambrose, 2006).
The MRIA Response Rate Committee analyzed telephone survey refusal rates and response rates in 1995, 1999 and 2002. These studies looked at response rates for one-time telephone surveys–that is, surveys that were not tracking or omnibus surveys. These surveys had incidence rates of 50% or more, used random samples and had no identifiable sponsor. The MRIA analyses showed that response rates for these surveys declined to 12% in 2002 from 16% to 17% in 1995–1999.
Some organizations, including Statistics Canada and other statistical agencies, continue to achieve response rates of 70% or more. These organizations benefit from a unique set of circumstances: mandated compliance, sponsorship advantages, long field times and, often, much larger budgets than those available to other organizations. Further, unlike POR researchers, who measure attitudes, knowledge and opinions, these organizations tend to collect factual information, through surveys such as the Census.
The Council for Marketing and Opinion Research (CMOR) in the United States also tracks response, cooperation and refusal rates for studies. Recent averages for all telephone surveys and for random digit dialling (RDD) surveys are presented in the following table.
| Average response rate | Number of surveys | Rate |
|---|---|---|
| Telephone overall | 1,364 | 17.0% |
| Telephone RDD | 761 | 9.17% |
Source: CMOR, September 2004.
While these data are not directly comparable to the MRIA data due to differences in response rate calculations, they do suggest a similar direction in response rates. The 2004 average response rate in the U.S. (for all types of telephone surveys) is 17% based on 1,364 industry surveys, and for RDD surveys it is 9.17% based on 761 industry surveys. These averages are somewhat lower than those in 2001, when the average overall response rate was reported to be 23.8% for telephone surveys and 12.2% for RDD telephone surveys.
Average response rates for customer satisfaction, list-based sample and business-to-business telephone surveys, as tracked by CMOR, are shown in the following table.
| Average response rate | Number of surveys | Rate |
|---|---|---|
| Customer satisfaction | 69 | 32.96% |
| List | 414 | 30.93% |
| Business-to-business | 120 | 17.15% |
Source: CMOR, September 2004.
As would be expected, these response rates are significantly higher than those reported for RDD telephone surveys. In all cases, the data collection efforts could draw on lists, while in some instances–particularly the customer satisfaction surveys–the respondents had an interest in responding to the survey.
Organization and Use of the Best Practices
This document includes 50 best practices designed to help ensure that organizations achieve the highest response rate possible for a particular study. These best practices were developed after a comprehensive review of the most current literature related to telephone survey response rates. They are organized according to the four main stages of a research study: design, data collection, analysis and reporting. Each best practice description includes the following elements:
- identification of the best practice or research standard;
- the rationale behind the best practice and issues of importance to consider; and
- information on implementing related guidelines–that is, mini best practices.
This document is designed to be a basic reference for people conducting Government of Canada telephone surveys, augmented by a bibliography providing more detailed information. For the convenience of readers, cross-referencing is used where applicable throughout the document. In addition, a Best Practices Checklist can be found on page 11. It can be used on its own or in conjunction with this fuller reference document. The best practices are labelled in a corresponding manner so readers can move easily between the Checklist and the full reference document.
Finally, while response rate issues can be addressed throughout a project, emphasis should always be placed on design and data collection features to maximize response rates. In addition, not all of the best practices outlined in this document will be appropriate or feasible for all POR studies. With this in mind, some of the best practices are more effective at maximizing response rates than others. All factors being equal, one of the most important things an organization can do to help maximize response rates is to allocate adequate time to collect the data. The organization should also focus on the survey questionnaire to make sure that it is free of bias, measures what it is intended to measure, and is as short, clear and simple as possible.
1 The Office of Management and Budget (OMB) in the United States released a list of standards for statistical surveys in September 2006. (Back to 1)