Executive Summary

Introduction

The Department of Public Works and Government Services Canada (PWGSC) required a panel composed of practitioners, methodologists and scholars with a high degree of expertise in the field from the private sector, the academic sector, Statistics Canada and Government of Canada departments and agencies that conduct public opinion research to advise on standards and guidelines for public opinion surveys conducted online.

PWGSC's specific objectives are:

  • To use this knowledge to establish requirements for Government of Canada public opinion online survey data quality under the next wave of contracting tools (Standing Offer) planned for 2008
  • To provide Government of Canada departments and agencies commissioning online survey research with standard contract requirements that each could choose to incorporate into contracts with public opinion research suppliers
  • To provide Government of Canada departments and agencies undertaking internal online survey research with specific benchmark levels of quality indicators

Per the Statement of Work, the scope of the Panel's enquiry was the following:

  • Limited to quantitative surveys, both probability-based and non-probability based; qualitative research is specifically excluded
  • Online surveys of the public, business and other populations
  • Online surveys conducted by the Government of Canada
  • Issues related to the distinction between probability and non-probability-based surveys and the acceptability of non-probability based surveys; this specifically includes issues related to statistical inference
  • Issues related to the reporting of results
  • Issues related to the assessment of data quality and the measures of success and response to online surveys
  • Guidelines related to accessibility and literacy-related issues
  • Specific standards and guidelines for statements of work, proposal documentation, questionnaire design and pre-testing as they relate to online research

The work of the Advisory Panel on Online Public Opinion Survey Quality builds on the work of the 2006-2007 Advisory Panel on Telephone Public Opinion Survey Quality. The Online Advisory Panel focused on areas where standards and guidelines specific to online surveys are required; for areas where standards and guidelines apply equally to telephone and online surveys, those recommended by the Telephone Advisory Panel are restated in this report.

The role of the Panel was also to reach consensus where possible, although this was not an essential outcome of the work of the Panel.

Top of Page

Method

The Panel consisted of nine members representing the Government of Canada, the market research industry and the fields of social science and business research in the academic community, and was chaired by a representative of PWGSC.

Market Research Industry

Doug Church
Marketing Research and Intelligence Association (MRIA)
Phase 5 Consulting Group

Cam Davis
Marketing Research and Intelligence Association (MRIA)
Social Data Research

Academic Community

Scott Bennett
Faculty of Public Affairs
Carleton University

Sylvain Sénécal
Department of Marketing
HEC Montréal

Government of Canada

Cathy Ladds
Senior Communications Strategist
Research and Analysis
Treasury Board of Canada Secretariat

Normand Laframboise
Senior Communications Advisor
Research and Advertising
Industry Canada

Jacqueline (Jackey) Mayda
Director, Special Surveys Division
Statistics Canada

Line Patry
Manager, Public Opinion Research
Industry Canada

Darryl Somers
Manager, Channel Intelligence and Innovation
Web Channel Office
Service Canada

Sage Research acted as facilitator and prepared the report on the Panel's deliberations. The Panel met twice (one conference call and one in-person meeting) and participated in a series of three online discussion boards. Other methods including telephone calls and emails were used to consult with members of the Panel, as appropriate.

The Panel's work took place between December 2007 and March 2008.

Top of Page

Report Overview

The report summarizes the recommendations of the Panel, expressed as standards or guidelines:

Standards

Practices that should be requirements for all online studies conducted by the Government of Canada.

Guidelines

Practices that are recommended, but would not be requirements; that is, known good practices or criteria that serve as a checklist to ensure quality research but are not necessarily applied to every study.

While it was not the mandate of the Panel to reach consensus, the Panel did do so on most aspects of data quality standards and guidelines.

The standards and guidelines are organized under six main sections:

  • Pre-field Planning, Preparation and Documentation
  • Sampling
  • Data Collection
  • Success Rate
  • Data Management and Processing
  • Data Analysis/Reporting and Survey Documentation

Document "The Advisory Panel on Online Public Opinion Survey Quality - Final Report June 4, 2008" Navigation