Response rates are best addressed during the design and data collection phases of research. During analysis, however, organizations can follow a number of best practices to help account for non-response and guard against non-response bias. These efforts will not improve the response rate per se, but they will help compensate for non-response and increase confidence in data quality.
Compute and compare response rates across key sub-groups of the target population. These subgroups might include age, gender and region, for example, for a survey of the general public. This method does not help determine the extent of non-response bias, but it can indicate whether there might be non-response bias. If the response rates are quite similar across sub-groups, non-response bias–should it exist–will likely have a limited impact on the data. The problem with this approach is that there are other causes of non-response aside from such sub-group variables. In other words, it is unlikely that response propensity is affected only by the sub-group variables (Groves, forthcoming). Comparing these response rates is a good place start, but doing so should not take the place of other methods of addressing survey non-response.
Statistical adjustment, or weighting by observable variables, is one of the most common approaches used to address survey non-response (Groves et al., 2001). Weighting the data can help the researcher ensure that the results presented to the reader are representative of the target population, where it can be assumed that no or little non-response bias exists. With this method, the provincial, demographic, socio-economic and other descriptive parameters of the survey sample are weighted to account for non-respondents. If, in a survey of the general public, 30% of the responses came from Ontario, weights could be applied so that these cases had the appropriate relative importance–that is, so that they represent 38% of the sample, which is Ontario's actual share of Canada's population. A description of the weighting procedures and source of the weights should be included in the methodological section of the report to help ensure that the survey can be replicated in the future.
Weighting is less effective in addressing bias that can result from non-response. Bias occurs when survey non-response results in differences between respondents and non-respondents. To correct for non-response bias, researchers need to understand what factors affect the likelihood of an individual agreeing to an interview.18 Such understanding is important because systemic differences between respondents and non-respondents may extend beyond observable variables –such as gender, income and region–which are adjusted by weighting based on available demographic data.
Statistics Canada demographic data on factors such as region, education, income, gender, age and language are often used for weighting purposes. Statistics Canada demographic data can become dated, so it is a good idea to find out whether other sources of more current demographic data are available for weighting purposes. Regardless of the source, ensure the use of valid external weights. If the weights are derived from an unreliable source, weighting the data will not improve survey data quality.
Non-response bias is specific to each survey. It is important, then, to try to assess non-response bias on key survey variables, not just demographics available through Census data. As discussed in section 3.0.2, even if the sample composition closely matches the demographic distribution of Canadians, there may be bias on other substantive, topic-related survey variables.
Where possible, conduct non-response analyses after data collection, to compare the characteristics of respondents and non-respondents. Such a comparison will enable organizations to assess how respondents differ from non-respondents, based on demographic or other known variables. Any difference between the two is an indication of non-response bias. Practical ways to compare the characteristics of respondents and non-respondents include the following.
Depending on the amount and nature of the information known about units in the sample frame, these approaches may be very useful in assessing the reliability of survey data.19 If there are few differences observed between respondents and non-respondents, the likelihood of biased data resulting from non-response is correspondingly low.
Another way to deal with non-response is to compare responses of "early" and "late" respondents. Late respondents are people who did not quickly complete an interview, thus requiring more effort from the interviewer–for example, people who required incentives, multiple callbacks or refusal conversion. They are not people who completed an interview toward the end of the data collection period, per se. The rationale for comparing early and late respondents is the assumption that late respondents might approximate non-respondents to some degree, because if the interviewer had not made extra efforts to reach these people, they would have been non-respondents, too. If a comparison of these two groups reveals no statistical differences across core measures, then the survey data can be generalized with more confidence. The weakness of this method is that it assumes non-respondents are similar to those respondents who are difficult to reach. Since this assumption may not apply universally, this technique offers imprecise information about the existence of non-response bias.20
A key to dealing with non-response is knowing the extent to which it might affect the data by introducing bias. One method used to quantify the difference between respondents and non-respondents is to survey a sub-sample of non-respondents when the fieldwork is complete. The responses from this sub-group can be compared to those of the respondent group. Typically, the full questionnaire is not used as the research instrument. Instead, a shorter version of the original questionnaire is administered, one that includes critical variables of interest to the department or agency. If no statistical differences are observed between the respondents and the sub-sample of non-respondents across these key survey measures, the overall results can more confidently generalized to the target population.
The drawback of a non-respondent follow-up is that it can be costly and it requires time. The researcher, or the department or agency commissioning the telephone survey, might not have the time or the financial resources to do this type of follow-up. It stands to reason that if non-respondents were difficult to reach or reluctant during the fieldwork, they will be equally difficult to reach or reluctant during a follow-up exercise. Such follow-ups involve callbacks and/or a change of data collection mode, and they may require the use of incentives or elite interviewers to persuade non-respondents to complete an interview. Organizations need to decide whether the level of survey non-response is likely to lead to non-response bias. If there is a reasonable concern that non-respondents may be systematically different from respondents, surveying a sub-sample of non-respondents may be necessary or valuable.
18 See Wang (2006) and Peress (2006) for promising research related to response propensity and non-response bias that warrants further investigation. (Back to 18)
19 See Groves (forthcoming) for a good discussion of the various methods researchers can use to test for non-response bias. (Back to 19)
20 Teitler, Reichman and Sprachman (2003) found that increasing efforts to reach difficult-to-contact respondents had little effect on reducing potential non-response error because these respondents were poor proxies for non-respondents. As well, other studies have found that late respondents do not different enough from early respondents to affect survey results (Gentry, 2006; Curtin et al., 2000). This is not to discount this technique altogether, but rather to suggest that it does not appear to be a strong means of determining whether non-response bias exists in survey data. (Back to 20)