Stage 4: Reporting

4.0 Document the response rate

Rationale

To address the issue of response rates in a comprehensive way, measure and record them consistently over time. While response rates are best addressed during the design and data collection phases of the study, properly documenting them will help organizations track them over time.

Best Practices

4.0.1 Ensure the research supplier provides the record of calls.

After completing the fieldwork for a telephone survey, the research supplier should provide the survey's sponsor with the record of calls. This record of calls, known as the call disposition, allows organizations to calculate the refusal rate, the response rate and the incidence rate for the survey (see BP 2.0.3). Beyond providing input data for these calculations, the record of calls offers valuable insight into the amount of effort required to reach respondents, as well as the quality of the sample frame.

4.0.2 Calculate the response rate using an approved method.

As a best practice, organizations should collect and report response rates for all Government of Canada telephone surveys. Ask the research supplier to use the standard MRIA approach to calculating response rates, endorsed by Statistics Canada (see Introduction). While reporting response rates will not affect the current study, doing so will ensure that response rates can be monitored over time and across different surveys. Monitoring response rates, including refusal and contact rates, will help survey sponsors and research suppliers develop effective strategies to maximize respondent cooperation.

4.0.3 Ensure the response rate is recorded in the final report.

As a best practice, the methodology section of final reports for telephone surveys should include the response rate and describe how it was calculated. The only way to effectively track response rates over time is to ensure they are reported consistently. This information is now routinely included in reports for telephone surveys commissioned by the Government of Canada. Dealing with response rates, and their decline, requires trackable information that can be periodically assessed to provide direction for government POR.

In addition, it may be desirable to note in the report the response rate for each sub-group and callback attempt. The former provides some indication of data reliability, and the latter offers insight into the performance of the sample frame, which can be valuable, especially for tracking studies.

Conclusion

This set of best practices was developed as a reference for Government of Canada POR telephone surveys. To this end, we hope that the document provides useful strategies to help ensure that each survey achieves the maximum response rate possible. For more detailed information on the topics covered in the best practices, we invite readers to consult the bibliography on the following pages.

Keep the following in mind when using the best practices.

  • Not all best practices will apply to all telephone surveys.
  • Emphasize design and data collection features to maximize response rates.
  • Some best practices will have a greater impact on response rates than others. Length of time in field is particularly crucial.
  • Adopting as many best practices as possible when doing a study can be expected to improve response rates.
  • Not incorporating the best practices appropriate to a study may decrease response rates.

Bibliography

  • AMERICAN ASSOCIATION FOR PUBLIC OPINION RESEARCH (AAPOR). (2006). Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. Lenexa, Kansas: AAPOR.
  • _____. (2005). Best Practices for Survey and Public Opinion Research (www). Retrieved December 1, 2005.
  • BEEBE, T. (2006). Demographic, Experiential, and Attitudinal Correlates of Self-Reported Future Survey Participation: Results of a statewide survey. Paper presented at the 61st Annual Conference of the American Association for Public Opinion Research, Montréal, Quebec, May 18-21, 2006.
  • BERRENS, R.P., A.K. BOHARA, H. JENKINS-SMITH, C. SILVA and D.L. WEIMER. (2003). "The Advent of Internet Surveys for Political Research: A Comparison of Telephone and Internet Samples." Political Analysis, 11(1): 1-22.
  • BLAIR, J., G. MENON and B. BICKART. (1991). "Measurement effects in self vs. proxy responses to survey questions: an information-processing perspective." In P.P. Biemer, R.M. Groves, L.E. Lyberg, N.A. Mathiowetz and S. Sudman (eds.), Measurement Errors in Surveys. New York: Wiley.
  • BOGEN, K. (1996). The Effect of Questionnaire Length on Response Rates: A Review of the Literature (PDF 48.78 KB) (Help on File Formats). Retrieved December 1, 2005.
  • BRICK, M.J., D. MARTIN, P. WARREN and J. WIVAGG. (2003). Increased Efforts in RDD Surveys (PDF 85.51 KB) (Help on File Formats). Paper presented at the Annual Meeting of the American Statistical Association, San Francisco, California, August 3-7, 2003. Retrieved July 15, 2006.
  • BURKS, A.T., P.J. LAVARAKAS, K. STEVE, K. BROWN, B. HOOVER, J. SHERMAN and R. WANG. (2006). How Organizations Monitor the Quality of Work Performed by Their Telephone Interviewers. Paper presented at the 61st Annual Conference of the American Association for Public Opinion Research, Montréal, Quebec, May 18-21, 2006.
  • CHAKRAPANI, C. and K. DEAL. (2005). Modern Marketing Research: Step by Step. Toronto: Pearson Education Canada, Inc.
  • CHURCH, A.H. (1993). "Estimating the effect of incentives on mail survey response rates: A meta-analysis." Public Opinion Quarterly, 57: 62-79.
  • COUNCIL OF AMERICAN SURVEY RESEARCH ORGANIZATIONS (CASRO). (1998). CASRO Guidelines for Survey Research Quality. New York: CASRO Publications.
  • COUNCIL FOR MARKETING AND OPINION RESEARCH (CMOR). (2004). CMOR Tracking System–Cooperation, Refusal and Response Rates. Retrieved August 23, 2004.
  • CUNNINGHAM, P., D. MARTIN and J.M. BRICK. (2003). Experiment in Call Scheduling (PDF 110.49 KB) (Help on File Formats). Paper presented at the Annual Meeting of the American Statistical Association, San Francisco, California, August 3-7, 2003. Retrieved August 8, 2006.
  • CURTIN, R., E. SINGER and S. PRESSER. (1999). "The effect of response rate changes on the index of consumer sentiment." Public Opinion Quarterly, 64: 413-428.
  • DE LEEUW, E., J. HOX, E. KORENDIJK and G. LENSVELT-MULDERS. (2006). The influence of advance letters on response in telephone surveys: A meta-analysis. Paper presented at the Second International Conference on Telephone Survey Methodology, Miami, Florida, January 11-15, 2006.
  • DE LEEUW, E. and W. DE HEER. (2001). "Trends in Household Survey Non-response: A Longitudinal and International Comparison." In R. Groves, D. Dillman, J. Eltinge and R. Little (eds.), Survey Non-Response. New York: John Wiley and Sons, pp. 41-54.
  • DILLMAN, D.A. (2000). Mail and Internet surveys: The tailored design method. New York: John Wiley and Sons.
  • _____. (1978). Mail and Telephone Surveys: The Total Design Method. New York: John Wiley and Sons.
  • DILLMAN, D.A., M.D. SINCLAIR and J.R. CLARK. (1993). "Effects of Questionnaire Length, Respondent-Friendly Design, and a Difficult Question on Response Rates for Occupant-Addressed Census Mail Surveys." Public Opinion Quarterly, 57(3): 289-304.
  • FAHIMI, M., R. WHITMORE, J. CHROMY, P. SIEGEL, M. CAHALAN and L. ZIMBLER. (2006). Efficacy of Incentives in Increasing Response Rates. Paper presented at the Second International Conference on Telephone Survey Methodology, Miami, Florida, January 11-15, 2006.
  • FISHER, S., J. BOSLEY, K. GOLDENBERG, W. MOCKOVAK and C. TUCKER. (2003). A Qualitative Study of Non-response Factors Affecting BLS Establishment Surveys: Results (PDF 82.32 KB) (Help on File Formats). Retrieved July 15, 2006.
  • FOWLER, F.J. (2002). Survey Research Methods, 3rd Edition. California: Sage Publications.
  • GALLAGHER, P., A. ROMAN, and K. LARSEN. (2006). Response Rates to Telephone Surveys Over Time: A comparison of cost and efforts in repeated RDD surveys. Paper presented at the Second International Conference on Telephone Survey Methodology, Miami, Florida, January 11-15, 2006.
  • GENTRY, R. (2006). Do Late Responders Change Survey Results? Paper presented at the 61st Annual Conference of the American Association for Public Opinion Research, Montréal, Quebec, May 18-21, 2006.
  • GOLDSTEIN, K.M. and M.K. JENNINGS. (2002). "The effect of advance letters on cooperation in a list sample telephone survey." Public Opinion Quarterly, 66: 608-617.
  • GROVES, R.M. (forthcoming). "Research Synthesis: Non-response Rates and Non-response Bias In Household Surveys." Public Opinion Quarterly.
  • _____. (1990). "Theories and Methods of Telephone Surveys." Annual Review of Sociology, 16: 221-240.
  • _____. (1989). Survey Errors and Survey Costs. New York: John Wiley and Sons, Inc.
  • _____. (1987). "Research on survey data quality." Public Opinion Quarterly, 51: 156-172.
  • GROVES, R.M., F.J. FOWLER, M.P. COUPER, J.M. LEPKOWSKI, E. SINGER and R. TOURANGEAU. (2004). Survey Methodology. New York: Wiley.
  • GROVES, R.M., S. PRESSER and S. DIPKO. (2004). "The role of topic interest in survey participation decisions." Public Opinion Quarterly, 68: 2-31.
  • GROVES, R.M., D.A. DILLMAN, J. ELTINGE and R. LITTLE (eds.) (2001). Survey Non-response. New York: John Wiley and Sons.
  • GROVES, R.M., E. SINGER, A.D. CORNING and A. BOWERS. (1999). "A Laboratory Approach to Measuring the Effects on Survey Participation of Interview Length, Incentives, Differential Incentives and Refusal Conversion." Journal of Official Statistics, 15: 251-268.
  • GROVES, R.M. and M.P. COUPER. (1998). Non-response in Household Interview Surveys. New York: John Wiley and Sons.
  • GUHA, A. (2006). "Trends, Challenges and Opportunities in Online Research." Vue Magazine, 7: 27.
  • HALPENNY, G. (2006). "The Approved Way to Measure Field Effort in Telephone Surveys." Vue Magazine, 6: 16-18.
  • HALPENNY, G. and D. AMBROSE. (2006). "Whither Survey Response Rates: Do They Still Matter?" Proceedings from the Marketing Research and Intelligence Association's ASCENT 2006 Conference, Calgary, Alberta, May 31-June 2, 2006.
  • HARRIS-KOJETIN, B. and C. TUCKER. (1999). "Exploring the Relation of Economic and Political Conditions with Refusal Rates to a Government Survey." Journal of Official Statistics, 15(2): 167-184.
  • HEBERLEIN, T.A. and R. BAUMGARTNER. (1978) "Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature." American Sociological Review, 43: 447-62.
  • KEETER, S., C. MILLER, A. KOHUT, R.M. GROVES and S. PRESSER. (2000). "Consequences of Reducing Non-response in a National Telephone Survey." Public Opinion Quarterly, 64(2): 125-148.
  • LAVRAKAS, P.J. (1993). Telephone Survey Methods: Sampling, selection, and supervision. Newbury Park, California: Sage Publications.
  • LIN, I. and N.C. SCHAEFFER. (1995). "Using survey participants to estimate the impact of nonparticipation." Public Opinion Quarterly, 59: 236-258.
  • LINK, M.W. and A. MOKDAD. (2005). "Advance Letters as a Means of Improving Respondent Cooperation in Random Digit Dial Studies." Public Opinion Quarterly, 69(4): 572-587.
  • MANN, C. (2005). "Do Advance Letters Improve Prelection Forecast Accuracy?" Public Opinion Quarterly, 69(4): 561-571.
  • MAYER, T. and E. O'BRIEN. (2001). Interviewer Refusal Aversion Training to Increase Survey Participation. Washington, D.C.: United States Census Bureau.
  • MCCARTY, M., M. HOUSE, J. HARMAN and S. RICHARDS. (2006). "Effort in Telephone Survey Response Rates: The Effects of Vendor and Client-Controlled Factors." Field Methods, 18(2): 172-188.
  • MERKLE, D. and M. EDELMAN. (2001). "Non-response in Exit Polls: A Comprehensive Analysis." In R. Groves, D. Dillman, J. Eltinge, and R. Little (eds.), Survey Non-response. New York: John Wiley and Sons, pp. 243-257.
  • NATIONAL CENTRE FOR EDUCATION STATISTICS. (2003). NCES Statistical Standards. Washington, D.C.: National Centre for Education Statistics, U.S. Department of Education Services.
  • OFFICE OF MANAGEMENT AND BUDGET (OMB). (2006). Standards and Guidelines for Statistical Surveys (PDF 317.74 KB) (Help on File Formats). Retrieved September 25, 2006.
  • _____. (2006). Questions and Answers When Designing Surveys for Information Collections (PDF 813.78 KB) (Help on File Formats). Retrieved February 7, 2006.
  • PERESS, M. (2006). Correcting for Survey Non-response Using Variable Response Propensity. Paper presented at the 61st Annual Conference of the American Association for Public Opinion Research, Montréal, Quebec, May 18-21, 2006.
  • (THE) PEW RESEARCH CENTRE FOR THE PEOPLE AND THE PRESS. (2006). Survey Experiment Shows: Polls Face Growing Resistance, But Still Representative (www). Retrieved July 15, 2006.
  • PURCELL, K., J. BEST, L. RAINIE, J. HORRIGAN and J. SU. (2006). A Cell Telephone Solution: Supplementing RDD Samples with Cell Telephone Interviews. Draft paper presented at the 61st Annual Conference of the American Association for Public Opinion Research, Montréal, Quebec, May 18-21, 2006.
  • SANGSTER, R.L. (2003). Do Current Methods Used to Improve Response to Telephone Surveys Reduce Non-Response Bias? (PDF 178.97 KB) (Help on File Formats).
  • SHREFFLER, K.M., J. MCQUILLAN, D.R. JOHNSON and K. JOHNSON. (2006). The Effect of Screener Wording on Response Rates. Paper presented at the American Association for Public Opinion Research Conference, Montréal, Quebec, May 18-21, 2006.
  • SINGER, E., J. VAN HOEWYK and M.P. MAHER. (2000). "Experiments with incentives on telephone surveys." Public Opinion Quarterly, 64: 171-188.
  • SINGER, E., R.M. GROVES and A.D. CORNING. (1999). "Differential incentives: Beliefs about practices, perceptions of equity, and effects on survey participation." Public Opinion Quarterly, 63: 251-260.
  • SINGER, E., J. VAN HOEWYK, N. GEBLER, T. RAGHUNATHAN and K. MCGONAGLE. (1999). "The effect of incentives on response rates in interviewer-mediated surveys." Journal of Official Statistics, 15: 217-230.
  • SINGER, E., D.R. VON THURN and E.R. MILLER. (1995). "Confidentiality assurances and response: A quantitative review of the experimental literature." Public Opinion Quarterly, 59: 66-77.
  • SMITH, T. (2003). "A Review of Methods to Estimate the Status of Cases With Unknown Eligibility." Report prepared for the American Association for Public Opinion Research Standard Definitions Committee.
  • _____. (2001). "Developing non-response standards." In R. Groves, D. Dillman, J. Eltinge and R. Little (eds.), Survey Non-response. New York: John Wiley and Sons, pp. 27-40.
  • _____. (1995). "Trends in non-response rates." International Journal of Public Opinion Research, 7(2): 157-171.
  • STATISTICS CANADA. (2003). Statistics Canada Quality Guidelines, 4th Edition. Ottawa: Statistics Canada. Catalogue no.: 12-539-XIE.
  • _____. (2002). Statistics Canada Quality Assurance Framework. Ottawa: Statistics Canada. Catalogue no.: 12-586-XIE.
  • _____. (2001). Standards and Guidelines for Reporting of Non-response Rates: Definitions, Framework and Detailed Guidelines. Ottawa: Statistics Canada.
  • STEEH, C., N. KIRGIS, B. CANNON and J. DEWITT. (2001). "Are they really as bad as they seem? Non-response rates at the end of the twentieth century." Journal of Official Statistics, 17: 227-247.
  • TAVASSOLI, N.T. and G.J. FITZSIMONS. (2006). "Spoken and Typed Expressions of Repeated Attitudes: Matching Response Modes Leads to Attitude Retrieval versus Construction." Journal of Consumer Research, 33: 179-187. Retrieved July 24, 2006.
  • TEITLER, J., N. REICHMAN and S. SPRACHMAN. (2003). "Costs and benefits of improving response rates for a hard to reach population." Public Opinion Quarterly, 67: 126-138.
  • TRIPLETT, T., J. SCHEIB and J. BLAIR. (2001). How Long Should You Wait Before Attempting to Convert a Telephone Refusal? (PDF 61.93 KB) (Help on File Formats). Paper presented at the Annual Meeting of the American Statistical Association, Atlanta, Georgia, August 5-9, 2001. Retrieved July 15, 2006.
  • TUCKEL, P., S. DANIELS and G. FEINBERG. (2006). Ownership and Usage Patterns of Cell Telephones: 2000-2006. Paper presented at the American Association for Public Opinion Research Conference, Montréal, Quebec, May 18-21, 2006.
  • TUCKEL, P. and H. O'NEILL. (2001). The vanishing respondent in telephone surveys (www). Paper presented at the Annual Meeting of the American Statistical Association, Atlanta, Georgia, August 5-9, 2001. Retrieved July 15, 2004.
  • VISSER, P.S., J.A. KROSNICK, J. MARQUETTE and M. CURTIN. (1996). "Mail surveys for election forecasting? An evaluation of the Columbus Dispatch Poll." Public Opinion Quarterly, 39: 557-562.
  • WANG, Rui. (2006). Wave Non-response in a National Telephone Survey. Paper presented at the 61st Annual Conference of the American Association for Public Opinion Research, Montréal, Quebec, May 18-21, 2006.
  • WEEKS, M.F., R.A. KULKA and S.A. PIERSON. (1987). "Optimal Call Scheduling for a Telephone Survey," Public Opinion Quarterly, 51(4): 540-54.
  • ZUWALLACK, R. (2006). RDD Selection Method to Increase Response Rates. Paper presented at the 61st Annual Conference of the American Association for Public Opinion Research, Montréal, Quebec, May 18-21, 2006.

Document "Improving Respondent Cooperation for Telephone Surveys" Navigation