Stage 3: Analysis

3.0 Address survey non-response

Rationale

Response rates are best addressed during the design and data collection phases of research. During analysis, however, organizations can follow a number of best practices to help account for non-response and guard against non-response bias. These efforts will not improve the response rate per se, but they will help compensate for non-response and increase confidence in data quality.

Best Practices

3.0.1 Compare response rates across sub-groups.

Compute and compare response rates across key sub-groups of the target population. These sub-groups might include age, gender and region, for example, for a survey of the general public. This method does not help determine the extent of non-response bias, but it can indicate whether there might be non-response bias. If the response rates are quite similar across sub-groups, non-response bias–should it exist–will likely have a limited impact on the data. The problem with this approach is that there are other causes of non-response aside from such sub-group variables. In other words, it is unlikely that response propensity is affected only by the sub-group variables (Groves, forthcoming). Comparing these response rates is a good place start, but doing so should not take the place of other methods of addressing survey non-response.

3.0.2 Weight survey data, where possible.

Statistical adjustment, or weighting by observable variables, is one of the most common approaches used to address survey non-response (Groves et al., 2001). Weighting the data can help the researcher ensure that the results presented to the reader are representative of the target population, where it can be assumed that no or little non-response bias exists. With this method, the provincial, demographic, socio-economic and other descriptive parameters of the survey sample are weighted to account for non-respondents. If, in a survey of the general public, 30% of the responses came from Ontario, weights could be applied so that these cases had the appropriate relative importance–that is, so that they represent 38% of the sample, which is Ontario's actual share of Canada's population. A description of the weighting procedures and source of the weights should be included in the methodological section of the report to help ensure that the survey can be replicated in the future.

Weighting is less effective in addressing bias that can result from non-response. Bias occurs when survey non-response results in differences between respondents and non-respondents. To correct for non-response bias, researchers need to understand what factors affect the likelihood of an individual agreeing to an interview.Footnote 18 Such understanding is important because systemic differences between respondents and non-respondents may extend beyond observable variables –such as gender, income and region–which are adjusted by weighting based on available demographic data.

Statistics Canada demographic data on factors such as region, education, income, gender, age and language are often used for weighting purposes. Statistics Canada demographic data can become dated, so it is a good idea to find out whether other sources of more current demographic data are available for weighting purposes. Regardless of the source, ensure the use of valid external weights. If the weights are derived from an unreliable source, weighting the data will not improve survey data quality.

3.0.3 Compare respondents and non-respondents.

Non-response bias is specific to each survey. It is important, then, to try to assess non-response bias on key survey variables, not just demographics available through Census data. As discussed in section 3.0.2, even if the sample composition closely matches the demographic distribution of Canadians, there may be bias on other substantive, topic-related survey variables.

Where possible, conduct non-response analyses after data collection, to compare the characteristics of respondents and non-respondents. Such a comparison will enable organizations to assess how respondents differ from non-respondents, based on demographic or other known variables. Any difference between the two is an indication of non-response bias. Practical ways to compare the characteristics of respondents and non-respondents include the following.

  • Use external sources: External data sources may contain variables similar to key survey variables. The external data source is the benchmark against which the survey results can be compared for variations. When selecting a comparable survey, look for one with very high response rates to decrease the likelihood that the comparison source will contain non-response error. For example, if the survey measures Internet use, Statistics Canada's Household Internet Use Survey might be a useful comparison source. Any difference related to frequency of Internet use between the survey and the other data source provides an estimate of overall bias.
  • Use available variables: Rather than looking for an external source to compare survey variables to, consider comparing the final sample composition to key characteristics in the sample frame or in a comparable source of administrative data. The difference between the two provides an estimate of non-response bias. For example, departments or agencies undertaking program evaluations or client satisfaction surveys might have administrative data they can use to compare survey respondents and non-respondents directly. If there is no information that can be used to compare respondents and non-respondents, which is quite likely to be the case with a RDD survey of the general public, consider linking the telephone exchanges to postal codes. From the postal codes, socio-demographic information about households in the area can be obtained from sources such as Canada Post's GeoPost and then used to compare respondents and non-respondents.
  • Observations during data collection: Consider asking interviewers to make observations about respondents and non-respondents. When these observations correlate with key survey variables, they may reveal non-response bias in survey variables. For instance, when determining why a program contributor might not have applied for program benefits, an organization could ask interviewers to assess the language and comprehension skills of all potential respondents contacted, using a short assessment tool created for that purpose. The organization could look at responses from people with limited abilities to converse in English or French, and compare the proportion of respondents in that group to the proportion of non-respondents in that group. This exercise could help the organization determine whether the survey findings understate the impact of language barriers–in other words, whether people with language barriers are under-represented in the final sample pool.
  • Seed the sample: Find a source with highly reliable characteristics related to key survey variables, and include some units with and without this characteristic in the sample. At the end of data collection, compute the response rates by characteristic– the seeded cases versus the main sample–to estimate response propensity. If the rates are similar, estimates correlated with the characteristic have small biases.

Depending on the amount and nature of the information known about units in the sample frame, these approaches may be very useful in assessing the reliability of survey data.Footnote 19 If there are few differences observed between respondents and non-respondents, the likelihood of biased data resulting from non-response is correspondingly low.

3.0.4 Compare "early" to "later" respondents.

Another way to deal with non-response is to compare responses of "early" and "late" respondents. Late respondents are people who did not quickly complete an interview, thus requiring more effort from the interviewer–for example, people who required incentives, multiple callbacks or refusal conversion. They are not people who completed an interview toward the end of the data collection period, per se. The rationale for comparing early and late respondents is the assumption that late respondents might approximate non-respondents to some degree, because if the interviewer had not made extra efforts to reach these people, they would have been non-respondents, too. If a comparison of these two groups reveals no statistical differences across core measures, then the survey data can be generalized with more confidence. The weakness of this method is that it assumes non-respondents are similar to those respondents who are difficult to reach. Since this assumption may not apply universally, this technique offers imprecise information about the existence of non-response bias.Footnote 20

3.0.5 Conduct non-respondent follow-ups.

A key to dealing with non-response is knowing the extent to which it might affect the data by introducing bias. One method used to quantify the difference between respondents and non-respondents is to survey a sub-sample of non-respondents when the fieldwork is complete. The responses from this sub-group can be compared to those of the respondent group. Typically, the full questionnaire is not used as the research instrument. Instead, a shorter version of the original questionnaire is administered, one that includes critical variables of interest to the department or agency. If no statistical differences are observed between the respondents and the sub-sample of non-respondents across these key survey measures, the overall results can more confidently generalized to the target population.

The drawback of a non-respondent follow-up is that it can be costly and it requires time. The researcher, or the department or agency commissioning the telephone survey, might not have the time or the financial resources to do this type of follow-up. It stands to reason that if non-respondents were difficult to reach or reluctant during the fieldwork, they will be equally difficult to reach or reluctant during a follow-up exercise. Such follow-ups involve callbacks and/or a change of data collection mode, and they may require the use of incentives or elite interviewers to persuade non-respondents to complete an interview. Organizations need to decide whether the level of survey non-response is likely to lead to non-response bias. If there is a reasonable concern that non-respondents may be systematically different from respondents, surveying a sub-sample of non-respondents may be necessary or valuable.

Footnotes

Footnote 18

See Wang (2006) and Peress (2006) for promising research related to response propensity and non-response bias that warrants further investigation.

Return to footnote 18

Footnote 19

See Groves (forthcoming) for a good discussion of the various methods researchers can use to test for non-response bias.

Return to footnote 19

Footnote 20

Teitler, Reichman and Sprachman (2003) found that increasing efforts to reach difficult-to-contact respondents had little effect on reducing potential non-response error because these respondents were poor proxies for non-respondents. As well, other studies have found that late respondents do not different enough from early respondents to affect survey results (Gentry, 2006; Curtin et al., 2000). This is not to discount this technique altogether, but rather to suggest that it does not appear to be a strong means of determining whether non-response bias exists in survey data.

Return to footnote 20

Document "Improving Respondent Cooperation for Telephone Surveys" Navigation