Stage 2: Data Collection

2.0 Ensure effective sample management

Rationale

According to most definitions, the response rate is normally the proportion of completed interviews to the total number of sample records drawn. Effective management of the sample is required in order to achieve a higher response rate. Attempting to complete the survey based on the sample drawn initially is a major element of good survey practice, without "burning through the sample" (abandoning attempts to contact the respondent after the first call and moving on to contact other respondents in the sample) and resorting to sample replacement. Adding sample to the initial sample drawn will normally result in a reduced response rate.

Best Practices

2.0.1 Hire a data collection firm that submits to recognized field audits.

Most reputable data collection firms are Corporate Gold Seal members of the Marketing Research and Intelligence Association (MRIA) or, in Quebec, members of l'Association de l'industrie de la recherche marketing et sociale. Speak with the research supplier to determine whether the firm, or the data collection company conducting fieldwork on behalf of the firm, is a member of either association. This will not guarantee quality fieldwork, but it does offer greater insurance that the firm consistently meets professional and ethical standards of research practice. The MRIA, for example, conducts regular independent audits of its Gold Seal members.

2.0.2 Ration sample resources.

Any new sample added to the data collection process must not invalidate the sample frame. Sample replicates, or small random samples drawn from the larger sample, can guard against invalidating the frame. Since each replicate is a miniature sample in itself, replicates can be added to the active cases as needed, without concern that they will skew the sample.

Discuss the appropriate rationing of sample with research suppliers. It is tempting to add new sample so that interviewers can quickly achieve the target number of completed surveys. Efforts to contact hard-to-reach people are sometimes not as stringent as they could be, as interviewers may focus instead on completing interviews with easier-to-reach people. In practical terms, this tendency means that fewer callbacks are attempted per sample record and that many of the people invited to take part in the survey are simply those available at certain times of the day when interviewers make calls.

Sometimes it is not possible to "work the sample" thoroughly because the field period is not long enough. The trade-off, however, is lower response rates. The more telephone numbers that interviewers attempt to obtain the required number of interviews, the lower the response rate.

Time in field is a critical factor when it comes to achieving high response rates (see BP 1.0.5). In addition to decreasing response rates, "burning" through the sample may compromise the representativeness of the respondents. If interviewers do not reach some segments of the target population, the risk that non-respondents may differ systematically from respondents increases. The result is a low response rate and the potential for non-response bias.

2.0.3 Accurately track the disposition of calls.

A record of all call attempts to reach a potential respondent should be kept for all telephone surveys. This record, often referred to as the call history or call disposition, is needed to accurately calculate the response rate at the end of a project. Interviewers assign a code to the outcome of each call, including busy signals, not-in-service numbers and wrong numbers. The MRIA uses the following standard record of contact for telephone surveys.

MRIA Standard Record of Contact Format for Telephone Surveys
Disposition of Last Attempt Explanatory Notes
1. Not in Service Includes list errors
2. Fax/Modem Line Includes list errors
3. Business/Residential Line Includes list errors
4. Busy Includes outstanding callbacks and appointments
5. Answering Machine Includes outstanding callbacks and appointments
6. Call Blocking Device Includes outstanding callbacks and appointments
7. No Answer Includes outstanding callbacks and appointments
8. Language Includes outstanding callbacks and appointments
9. Illness, Incapable Includes outstanding callbacks and appointments
10. Selected Respondent Unavailable Includes outstanding callbacks and appointments
11. Household Refusal Before respondent selected
12. Respondent Refusal Before answering all qualifying questions
13. Qualified Respondent Break-Off Any termination after qualifying
14. Disqualified Any disqualification of household or respondent during screening process. Includes quota filled, if identified during screening. Does not include quota filled based on sample data.
15. Completed Interview Includes any qualified completes that are rejected in post-edit. Where needed, add sub-categories for rejects and in-tab sample.

Tracking the dispositions of calls will not directly maximize response rates, but it will clearly identify the reasons for non-response–that is, non-contacts versus refusals. The research supplier can use this information to target non-respondents and to take steps during data collection to try to contact hard-to-reach respondents.

2.1 Make efforts to maximize contact rates

Rationale

Just as the design of questionnaires should incorporate features to encourage participation, data should be collected in a way that increases the likelihood of reaching potential respondents. Contacting people is the first step to completing an interview.

Best Practices

2.1.1 Vary the call scheduling.

Varying the call scheduling reduces call attempts and increases the likelihood of reaching a household (Cunningham et al., 2003). Potential respondents are more likely to agree to be interviewed if it is convenient for them to do so. Calling during the dinner hour may increase the likelihood of reaching people at home, but it may decrease the likelihood of participation because some people find these calls intrusive. If the ideal time to reach people at home is also the same time of day they are the least likely to agree to participate, then consider alternative call times. This strategy could be as simple as beginning calls earlier in the afternoon, perhaps at 3:00 p.m. rather than 4:00 p.m. A good contact rate is insufficient on its own. People must also agree to participate in an interview.

Certain audiences are more likely to be reached at different times of the day. For instance, during the day, it might be easiest to reach seniors at home and members of most occupational groups at work. The general public is generally best reached during the evening and on weekends. Unemployed individuals might best be reached using a combination of daytime and evening calls. Maximizing response rates requires calling at times that are most suitable for the survey's target audience while still ensuring coverage across a range of times in order to maintain the representativeness of the sample.

2.1.2 Offer flexible callbacks and appointments.

Potential respondents are busy and their participation in survey research is essential. When possible, therefore, offer potential respondents flexible callbacks and interview appointments for all telephone surveys. Discuss this aspect of the data collection with the research supplier. Making it as convenient as possible for an individual to take part in a survey can improve the response rate. Someone might be willing to take part in the survey but might not have the time to consider doing so when he or she receives the telephone call. Scheduling a callback at a suitable time might result in a completed interview–a step towards a higher response rate.

Much like offering flexible callbacks, allowing potential respondents to schedule a time to conduct the telephone interview can improve response rates. Appointments differ from flexible callbacks in that the intent of the first contact or initial telephone call is to schedule, not conduct, the interview.

Most senior-level executives are very busy people who may not handle their own telephone calls or time scheduling. Building the use of appointments into a survey of executives, for example, will help make the survey as convenient as possible for these individuals. If interviewers are able to reach these executives on the telephone, the likelihood of them being able to conduct the interview at that time is low. Moreover, if interviewers ask such people to participate in a survey at an inconvenient time, executives are more likely to refuse completely than to schedule a general or specific callback. Telling these individuals that the purpose of the telephone call is to inform them of the survey and to try to schedule an interview appointment is a much more effective approach. Not only does it address the fact that these are busy people; it also conveys to them the importance of their participation by demonstrating that the interviewer is willing to work around their schedule.

When using specific callbacks and appointments, interviewers must abide by the callback commitments, calling back at the designated date and time. Otherwise, they might decrease the response rate. A missed callback or appointment can reduce the credibility of the survey and the research firm, and can jeopardize the respondent's willingness to participate.

2.1.3 Ensure an adequate number of callbacks.

The number of contacts attempted per potential respondent improves response rates. The rationale is simple: the greater the number of calls, the greater the likelihood of contact. The actual number of callbacks will vary depending on the research design, budget and length of field period available, but numerous attempts to contact an individual or household will reduce non-response.

Using callbacks well is one of the most common and effective ways to increase response rates. Under the current Standing Offer of the Government of Canada, eight callbacks are required before a sample record is retired.Footnote 15 This number includes callbacks made to establish initial contact with the potential respondent (for instance, when there was no answer or the line was busy on previous calls) and callbacks made after contact has been established (for instance, when the potential respondent asked for a callback at a more convenient time, or when the selected respondent was not home). Increasing the number of callbacks up to a certain point–depending on the audience, time of year and other factors–can improve response rates. In addition, varying the times of day and days of the week when the callbacks happen can make them more effective, because it increases the chances of making contact with people when they are at home or in their office.

2.1.4. Schedule extra callbacks to households with an initial language barrier.

Scheduling extra callbacks to a household with an initial language barrier may yield a completed interview. Depending on the target population and the survey topic, this practice may help maximize response rates. In the call history, the interviewer enters a language barrier code to a record when the person who answers the telephone does not understand the interviewer in either official language. If the target population is 18- to 30-year-olds, for example, it might be a good idea to call back records with a language barrier code. A young person who does not face a language barrier may live in the household. By calling back at a different time of day or day of the week, the interviewer may reach the eligible respondent and thus increase response rates.

2.1.5 Leave messages, for some studies.

Interviewers can leave messages for potential respondents, either to inform them of the survey or to invite them to call back for an interview. After a specific number of callback attempts that result in non-contact (that is, there is no answer, the line is busy or an answering machine responds), it may be useful to leave a message informing potential respondents that a Government of Canada telephone survey is being conducted and that an interviewer will call back at a later time or date. If using this tactic, draft a standard message for interviewers to help ensure they convey the appropriate information to each potential respondent.

As a general rule, it is not effective to leave voice messages for telephone surveys of the general public, inviting potential respondents to call back to be interviewed (Sangster, 2003). Not surprisingly, few call back. However, this approach can work for some studies of stakeholder groups, where the potential respondents are motivated to participate in the survey.

2.1.6 Provide a toll-free number for studies with hard-to-reach respondents.

The number of callback attempts can be expected to improve the response rate for a study (see BP 2.1.3). However, for some difficult-to-reach audiences (such as transportation workers, who are rarely in their office), simply increasing the number of contact attempts likely will not help improve the response rate significantly. In such cases, setting up a toll-free telephone during the study will allow potential respondents to call the data collection firm at a time that is convenient for them. The toll-free line could be linked directly to the call centre supervisor, who could the route the call to an interviewer working on the study. This technique is useful when potential respondents have a vested interest in participating in the survey, such as a monetary or other incentive, or an interest in the topic.

2.2 Take steps to minimize refusals and terminations

Rationale

A well-designed questionnaire with an effective introduction is only part of the equation. Effective interviewers are a critical component of a high-quality telephone survey. No matter how good the research instruments are, they must be in the hands of a capable, well-trained interviewer with a professional telephone manner, who is skilled at encouraging participation and preventing early terminations.

Best Practices

2.2.1 Ensure use of well-trained, effective interviewers.

Telephone interviewers need to be well trained in order to encourage people to participate in surveys and to collect data accurately. They should be professional and friendly, and should sound mature, when they speak on the telephone. This is critical to the success of any of POR telephone survey. To ensure that the data collection firm has well-trained staff, look for field suppliers that offer interviewers an acceptable industry training program. The program should train interviewers to do the following:

  • introduce themselves to potential respondents;
  • encourage participation appropriately, a task that includes dealing with reluctant respondents;
  • use voice intonation to sound more professional, confident and assertive;
  • avoid refusals by using techniques, strategies and phrases to foster dialogue with respondents, in order to understand why they are "not interested" and explore ways to gain their cooperation;
  • read questions as worded and record responses accurately;
  • record open-ended responses verbatim;
  • probe for details and clarification effectively;
  • avoid leading respondents to answers;
  • accurately code the results of calls; and
  • explain how the research firm obtained a respondent's name and telephone number, and why he or she is being included in the study.

Interviewers should be briefed for all telephone surveys, through both verbal briefings and, ideally, briefing notes. These briefings are project specific and differ from the general training interviewers receive. The briefings should include background information on the topic or program that is the subject of the survey; any special challenges the researchers anticipate; information on the target audience, if relevant; and related issues. If the questionnaire is complex or requires specialized knowledge, interviewers should receive additional instructions and information to help them administer it. Not only will such briefings help ensure data accuracy; they can also help maximize response rates. If the interviewer is skilled at moving through the questionnaire, respondent terminations will be minimized, since the burden on the respondent will be lessened.

For smaller samples of special-audience research, it is usually advisable to use a research supplier's best interviewers. Mature, experienced interviewers are better able to handle studies where the target population is elite or difficult to reach–for instance, physicians or executives– and where the survey topic is sensitive. Having fewer, but more experienced, interviewers can also help maximize the response rate, because the interviewers will become very familiar with the study–something that is particularly important for senior or elite audiences–and will be focused on completing interviews.

2.2.2 Request monitoring of data collection at all times.

Look for a field supplier that monitors at least a portion of the interviews throughout the data collection. Monitoring will help ensure the quality of the interviews throughout the field period by keeping interviewers striving for top performance. In addition, interviewers' interaction with potential respondents can affect the response rate. For example, not tailoring the initial script based on cues from the potential respondent might result in a refusal rather than a completed interview. Monitoring data collection can help determine under which circumstances interviewer behaviour influences non-response and may affect non-response error (Lavrakas, 1993). Field suppliers that are corporate members of MRIA are required under the MRIA code of conduct to validate at least 10% of interviews for each project, via onsite or remote call monitoring, unless the client specifies otherwise.Footnote 16

2.2.3 Monitor reasons for non-response during data collection.

It is useful to monitor the progress of the fieldwork. Not only will this approach help keep the study on target and on budget; reviewing the reasons for non-response can also help the research supplier make any adjustments necessary to maximize response rates. If the non-response rate is high due to non-contacts, the research supplier could ask interviewers to conduct callbacks at different times and on different days of the week, in an attempt to reach more potential respondents. Conversely, if the call disposition reveals that a high non-response rate is due to refusals, the research supplier could try to ensure that only interviewers with the best interview completion rates are working on the survey.

2.2.4 Monitor non-response levels among different segments of the target population.

In addition to monitoring overall response rates during the fieldwork, it can be important to review the response rates for specific segments of the target population. High response rates do not necessarily reduce or eliminate non-response error. While a study may achieve a high response rate, the results may not be confidently generalized to the full target population if non respondents differ substantively from respondents. Consider the example of a survey of self-employed individuals. To complete the interviews quickly, regional, age or gender distribution might be sacrificed. More careful, slower fieldwork would strive to conduct interviews in the appropriate proportions for key characteristics.

If working the survey sample only results in more responses from one segment of the target population and not others, a high response rate will be achieved at the possible expense of quality data. Low response rates among sub-groups of the target population might introduce non-response bias. If non-response is expected to be higher among certain sub-groups, monitor these elements of the sample closely, in addition to overall non-response call dispositions (Statistics Canada, 2003).

2.2.5 Attempt refusal conversions.

Refusal conversions
  • Senior interviewers
  • Specialized training
  • Secondary intro

Refusal conversions are an essential aspect of data collection for survey organizations. They involve attempting to convert someone who has already said that he or she does not want to take part in a survey, or who has terminated an interview, into a respondent. Senior, experienced interviewers call back people who initially refused to be interviewed to try to persuade them to participate. Typically, the interviewer uses a modified introduction that acknowledges the potential respondent's earlier reluctance to take part in an interview. The time and day of refusal conversion callbacks vary. In some cases, it can be useful to send a letter to non-respondents, asking them to reconsider. This approach can be less intrusive than a telephone call, but it still requires the careful use of language. It might not be possible to send letters, depending on the length of the data collection period and the availability of mailing addresses.

If handled with care, refusal conversions do increase telephone survey response rates.Footnote 17 Not only can this technique turn a refusal into a completed interview, but it may also yield new potential respondents: another call to a household after an initial refusal might reach someone other than the person who refused and result in a completed interview with that person. Depending on the department and agency sponsoring the POR survey, and the study topic, there may be heightened sensitivity around the practice of refusal conversions.

Footnotes

Footnote 15

Cunningham et al. (2003) found that the majority of completed interviews, refusals and ineligible cases are established by the sixth or seventh call attempt.

Return to footnote 15

Footnote 16

Burks et al. (2006) provides a good discussion of the range of practices survey organizations use to monitor interviewing.

Return to footnote 16

Footnote 17

See, for example, Triplett, T., J. Scheib and J. Blair (2001). Moreover, Curtin et al. (2002) found that the proportion of refusal conversions in the final sample of a national telephone survey increased from 7.8% in 1979 to 15.1% in 2002. Together with other efforts, such as callbacks, refusal conversions were required to maintain consistently high response rates over time.

Return to footnote 17

Document "Improving Respondent Cooperation for Telephone Surveys" Navigation