Standards and Guidelines for: Pre-field Planning, Preparation and Documentation

Statement of Work

Overview

There was agreement on the principle stated by ESOMAR on the role of the Statement of Work (SOW) in the research process:

The more relevant the background information the client can give the researcher, the greater the chances are that the project will be carried out effectively and efficiently.

The SOW is also an important document as an internal Government of Canada tool:

  • A guide to the overall research process for the department
  • A central document of the statement of the needs of the department

There was agreement that:

  • There are some types of information that must be included in all SOWs, i.e., requirements
  • Guidelines may also be useful in certain areas to help coordinators/clients (in departments where there is no coordinator) clearly specify the project requirements, albeit without:
    1. Duplicating the research proposal
    2. Making the process of preparing SOWs too cumbersome

There was also the suggestion that an "exhaustive" SOW checklist should be developed for all types of studies, not just telephone surveys:

  • For use as a reminder to coordinators/clients, including the cross-referencing of items on the checklist to sections of policies
  • As a way of assessing the supplier proposal in general and as it relates to data quality issues

Outcome of Panel's Deliberations

There was consensus among the Panel on the following standards and guidelines for the Statement of Work (SOW).

STANDARDS

A Statement of Work must be a written plan that provides the research supplier with the following information:

Table Summary

The table under 'Standards' has two columns and eight rows. The first column under Standards lists the categories for the statement of work document. The second column lists the information to be included in each Standard category of work that is to be provided to the research supplier:

Standards Information to be Included:
Background
  • To provide context for the research, describe events/decisions that led to why research is required/being considered.
  • Include information/available resources to help the contractor better understand the subject matter of the survey (e.g., past research, web sites).
Purpose, how the research will be used
  • Provide information on the types of decisions or actions that are to be based on the findings, i.e., (a) what activities it will support; (b) how; (c) who will use the information.
  • Include any internal or external commitments regarding scheduling/timelines that may rely on the research findings (e.g., reporting requirements, events).
Objectives, research questions
  • Include, in the information requirements, the broad research questions that the study needs to answer. This will help in the development of the survey questionnaire, the data analysis and the report outline.
  • If relevant, prioritize the information required to ensure data quality in the event of budgetary or scheduling constraints.
Target population
  • Wherever necessary and possible, indicate:
    • The demographic, behavioural and/or attitudinal characteristics of the target population for the survey
    • Information or estimates available on the size/incidence of these groups
Data collection method
  • If relevant, ask for input on other data collection approaches.
Deliverables
  • It should list major project milestones with anticipated timelines.
  • At minimum, details of reporting should reference all requirements identified by PORD.
Sample size assumptions
  • To help the supplier generate a reasonable sample size assumption for costing purposes, at least one of the following indicators must be included:
    • Sample size required
    • Level of precision required
    • Study budget

GUIDELINES

Other useful information that may be included in a Statement of Work include the following:

Table Summary

The table under 'Guidelines' has two columns and three rows. The first column under Guidelines lists these categories for the statement of work document. The second column lists the information that may be included for each guideline category of the statement of work that is to be provided to the research supplier:

Guidelines Information to be Included:
Sample considerations
  • Provide any relevant information on the sampling frame, e.g., the availability of lists.
  • Indicate any requirements to be taken into consideration in finalizing the total sample size and structure/composition of the sample, e.g., regional requirements, demographic groups, population segments (those aware vs. those not aware; users vs. non-users, etc.).
Data analysis
  • Identify any need for special analyses, e.g., segmentation.

Proposal Documentation

Overview

There was general agreement among the Panel members that the elements currently listed by PWGSC as requirements for proposal documentation in its RFSO should be expanded, particularly the section on Methodology.

While several organizations (ESOMAR, MRIA, ISO) have described specific elements for what should be included in proposal documentation, there was no one organization Panel members agreed best met the desired level of detail for Government of Canada proposal documentation. Accordingly, the elements suggested by the Panel for proposal documentation represent a hybrid from these various organizations.

With regard to the statement of requirements for proposals, it should be kept in mind that some contracts for telephone surveys are issued to firms on the Standing Offer, while others may be awarded through competition on MERX or as sole source contracts (e.g., syndicated studies, omnibus surveys). Firms on the Standing Offer will have already committed to certain practices, which may be required elements in a proposal. For example, it was suggested that various quality control procedures be spelled out in proposal documentation. Firms on the Standing Offer may have already committed to some of these as their standard practices. In this case, we suggest they not be required to describe these again in each proposal they submit.

In the proposed standards on the next page, an asterisk has been placed next to items that might already have been addressed by firms in their Standing Offer submissions. Firms awarded telephone survey contracts who are not on the Standing Offer would be required to address all required elements in their proposals. Firms on the Standing Offer would be required to address only the non-asterisked items.

Several themes emerged in the Panel discussion:

  1. A clear delineation between the SOW and the Research Proposal:
    • The SOW is what the GOC needs to know, from whom and when it needs this information
    • The Research Proposal is what the research firm will do to meet the needs of the GOC and how this will be done

    Therefore, there is much more detail required from research firms in the Proposal Documentation than is required from the GOC in the SOW.

  2. There is a need to find a balance between all information required by GOC as responses to a SOW, and ensuring all data quality issues are also covered, but without overburdening either the research supplier or the GOC. It is also for this reason that we made the point above regarding differences between those firms on the Standing Offer and those not on the Standing Offer.
  3. Especially among the GOC Panel members, there was a perceived need for consistency in proposal documentation to make it easier to assess/confirm that the research firm has provided all the categories of information and the detail required in each proposal.

In light of this, the Panel as a whole recommended standards rather than guidelines for proposal documentation.

There was no consensus on whether or not different requirements should be established for Proposal Documentation based on survey characteristics such as type of study, expenditure level, etc.:

  • In principle, there were suggestions that studies that need to be more rigorous (e.g., estimate of size populations vs. comparing options) or that deal with complex issues generally require more detailed documentation.
  • In practice, the level of detail provided in any research proposal will necessarily vary depending on the above characteristics, i.e., these types of projects require more in-depth explanations of the various aspects of the survey design.

Outcome of Panel's Deliberations

There was consensus among the Panel on the following standards for proposal documentation.

Standards

The research proposal must be a written document that uses the following headings and provides the following information, at a minimum. Note that an asterisk identifies the areas that apply only to proposals from firms not awarded PWGSC's Quantitative Standing Offer.

A: Introduction
Table Summary

The table under 'Standards' has two columns and ninerows. The first column under Standards lists the categories of the statement of work document. The second column lists the information that must be included:

Standards Information to be Included:
Purpose
  • Describe the firm's understanding of the problem/issues to be investigated and how the GOC will use this information.
Research Objectives
  • Detail the information needs/research questions the research will address.
B: Technical Specifications of the Research
Table Summary

Technical Specifications of the Research: The table two columns and nine rows of specifications for information that must be included by the research supplier.

Standards Information to be Included:
Overview
  • Provide a brief statement summarizing:
    • data collection method, including rationale for proposed methodology
    • total sample size
    • target population
Sampling/Sampling Details
  • Provide details related to target population:
    • the definition of the target population in terms of its specific characteristics and geographic scope, including the assumed incidence of the population and any key sub-groups
    • the total sample size and the sample sizes of any key sub-groups
  • Describe the sample frame, including:
    • the sample source
    • sampling procedures
    • any known sampling limitations and how these might affect the findings
  • Explain respondent selection procedures.
  • Indicate the number of call-backs and explain call-back procedures.
  • Define respondent eligibility/screening criteria, including any quota controls.
Response Rate/Error Rate
  • State the expected response rate for the total sample and for key sub-groups, if relevant.
  • State the level of precision, including the margin of error and confidence interval for the total sample and any key sub-groups.
  • Indicate any other potential source of error based on the study design that might affect the accuracy of the data.
Description of Data Collection
  • State the method of data collection (in this case telephone interviewing).
  • Provide details on any incentives/honoraria, including rationale.
  • Describe how language requirements will be addressed.
  • *Describe quality control procedures related to data collection, including at minimum:
    • Recruitment, training and management of field staff
    • Fieldwork validation methods and procedures
  • *Describe how:
    • The rights of respondents will be respected, including if relevant the rights of children, youth and vulnerable respondents
    • Respondent anonymity and confidentiality will be protected
Questionnaire Design
  • Provide either an outline of the survey questionnaire or list the topics that will be covered in the questionnaire, including specifying the number of open-ends.
  • Provide an estimate of the length of the questionnaire.
  • Describe how the questionnaire will be pre-tested, including:
    • The objectives of the pre-test
    • The method for the pre-test, including the description of the types of interviewers/researchers that will conduct the pre-test interviews
    • The total number of interviews to be completed in total and by key sub-groups (e.g., language, age, gender)
    • How the results of the pre-test will be communicated to the GOC
    Note: If no pre-test is to be conducted, a rationale must be provided.
Description of Data Processing/Data Management
  • Describe any weighting required.
  • *Describe quality control procedures related to data processing/data management, including at minimum:
    • Data entry
    • Coding/coding training
    • Data editing
    • Data tabulation
    • File preparation/electronic data delivery
Data Analysis/ Reporting
  • Describe how the data will be analyzed related to the objectives/research questions, including any special analyses (e.g., segmentation).
  • *Provide an outline of the sections of the report.
Deliverables
  • List all deliverables including their coverage, scope, format, means of delivery and number of copies, including at minimum:
    • Questionnaire(s), including pre-test, if relevant
    • Data tabulation/processing
    • The report format(s), including the number of copies, language of report
    • The nature, location and number of presentations, including the language of presentations
Project Schedule
  • Provide a detailed workplan with dates and identify responsibilities
C: Project Cost
Table Summary

Project Cost is two columns and two rows to indicate that project costs are to be presented in a format designated by PWGSC.

Standards Information to be Included:
Project Cost
  • Cost information must be presented in the format designated by PWGSC.

Questionnaire Design

Overview

There was agreement that for questionnaire design only very broad standards and guidelines are required, and several reasons were given for why this overall approach should be adopted.

  • It is unnecessary to develop standards/guidelines for questionnaire design because this is primarily the responsibility of the research firms.
    1. The industry has good "rules of thumb" to follow relating to every aspect of questionnaire design.
    2. Research firms already need to demonstrate their capabilities in the RFSO or an RFP by the requirement to provide both project credentials and references.
  • It is both unrealistic and undesirable to impose either standards or guidelines on an activity, i.e., questionnaire development, that:
    1. Is as much art as it is science
    2. Needs to be tailored to the specific information needs of each GOC survey and the different departmental requirements in certain areas (e.g., inclusion of references to specific pieces of legislation is required in some departments but not in others)
  • It is best left to the research industry to develop a set of standards and guidelines for questionnaire design, not the POR community in the GOC. Notably, though, the point was also made that the research industry itself does not have enough of a consensus on best practices upon which to base standards.
  • There is a need to avoid creating a process that becomes unduly burdensome for both GOC and suppliers in terms of the level of effort, time and financial resources that are devoted to developing and finalizing survey questionnaires.

Some members of the Panel suggested it would be useful to have a questionnaire design "checklist" for internal purposes, i.e., a tool that would serve as a guide to questionnaire development for those with limited research experience and an aide memoire for experienced POR coordinators. The report prepared by Phoenix Strategic Perspectives, Best Practices: Improving Respondent Cooperation for Telephone Surveys, could serve this role, particularly for those with a limited research background.

One of the issues considered by the Panel was whether there should be standard demographic and socioeconomic questions across GOC telephone surveys. The Panel agreed, in principle that:

  • Whenever possible, standard demographic and socio-economic questions should be used in GOC surveys
  • The use of this common approach across departments would help the GOC compare surveys for non-response bias
  • It will be useful for the Government of Canada to have a database of standard questions accessible to GOC departments and research suppliers

However, there was no agreement on whether or not these types of questions should be requirements in practice, at least for the time being:

  • The GOC Community of Practice will need to do preliminary work:
    1. To inform and educate both internal clients and research suppliers on what would be standard demographic and socio-economic questions
    2. To provide a glossary of standard definitions and classifications
  • There are two dimensions to "standard demographic and socio-economic questions" — the questions themselves and the response categories. There was no agreement on standardization in practice on either of these dimensions.

    At one level, it was suggested there could be standards on both dimensions, but only for studies with a general population target group.

    At another level, (a) it was suggested that the questions could be standard but the response categories would need to be flexible, or that (b) both the questions and response categories would need to be flexible, i.e., guidelines only to meet the needs of the range of surveys undertaken by the GOC.

Outcome of Panel's Deliberations

There was general agreement by the Panel on the following standards and guidelines for questionnaire design.

STANDARDS

  • Survey questionnaires must be designed:
    1. To collect only the information essential to the objectives of the study, and
    2. To minimize the burden placed on respondents while maximizing data quality
  • The following are required elements of all Government of Canada telephone survey questionnaires:
    1. Inform respondents of
      1. the subject and purpose of the study and
      2. the expected length of the interview
    2. Identify the research firm and either the Government of Canada or the department/agency sponsoring the survey
    3. Inform respondents that their participation in the study is voluntary and the information provided will be administered according to the requirements of the Privacy Act
    4. Inform respondents briefly of their rights under the Access to Information Act, most importantly the right to access a copy of the report and their responses
  • Firms are required to translate the questionnaire into the other official language (unless interviewing is to be unilingual), and where required into other languages. All translations must be in written form.

GUIDELINES

The following strategies may be used to achieve the standards.

  1. The questionnaire is a reasonable length, e.g., surveys 20 minutes or longer can often result in lower cooperation rates.
  2. The introduction to the survey and the respondent screening section are welldesigned and as short as possible in order to maximize the likelihood people will agree to an interview.
  3. Questions are clearly written and use language appropriate to the target group.
  4. Methods to reduce item non-response are adopted (e.g., answer options match question wording; "other," "don't know" and "refused" categories are included, as appropriate).
  5. The questionnaire is designed for clear and smooth transition from question to question and from topic to topic.

Pretesting

There was consensus that in-field pre-testing should be required for all surveys using new or revised questionnaires. This recommendation is reflected in the standards below.

The Panel had differing views, though, on cognitive pre-testing and the circumstances under which these pre-tests should occur. Some felt that cognitive pre-testing should be required for all new and revised questionnaires unless it can be demonstrated that these have been previously tested using cognitive methods or supported by other means (e.g., past experience, academic literature). Others said that this should be assessed on a case-by-case basis depending on the survey, its purpose and budget. Given the differing views of the Panel, there was agreement to include guidelines for when this type of pre-testing should be considered along with more in-depth testing of the data collection and data capture processes.

The Panel distinguished between syndicated studies and omnibus surveys with regard to pre-testing requirements. For syndicated studies, it was felt that firms offering these studies should be required to demonstrate and document that the survey questionnaire has been pre-tested in some form. For omnibus surveys, the majority of Panel members suggested stating guidelines rather than standards given these factors.

  • Pre-testing of questions is rarely an element in the standard costing formula for omnibus questions and thus would likely increase cost. Further, given the schedules for most omnibus studies, this would also likely increase the required time to finalize survey questions.
  • Often these vehicles are used when:

    1. Information is required on an urgent basis, or
    2. Information needs can be addressed by a few question, or
    3. Budget is limited

STANDARDS

  • In-field pre-testing of all components of a new or revised telephone survey questionnaire that may influence data quality, respondent behaviour and interviewer performance is required.
  • A periodic review of questionnaires used in ongoing or longitudinal surveys is required.
  • A minimum of 30 pre-test interviews are to be completed in total, 15 in English and 15 in French. Provision is to be made for the project authority to monitor pre-tests should they want to.
  • The result(s) of the pre-test(s) must be documented, i.e., at minimum:
    • A description of the pre-test approach and number of interviews completed
    • A summary of the results including a record of the decisions/changes made as a result of the pre-test findings
  • For syndicated telephone studies, research firms are required (a) to demonstrate that the survey questionnaire has been pre-tested and (b) to provide details on the pre-test approach and number of interviews completed.

GUIDELINES

For complex studies, highly influential surveys or surveys that are planned to be ongoing or longitudinal, a more complete in-field test of other components of a survey, not just the survey questionnaire, may be desirable. This may be a pilot test that, on a small scale, duplicates the final survey design including such elements as data capture, analysis of results, etc.

If there is a need to pre-test the questionnaire on criteria other than language, at least 4 should be completed with each sub-group.

Pre-test interviews should not be included in the final dataset unless

  1. there were no changes to the questionnaire and
  2. the interview was implemented in the exact same manner as in the final survey design.

Cognitive pre-testing (using qualitative methods) should be considered prior to field testing for new survey questionnaires or where there are revisions to wording or content of existing questionnaires, and particularly for complex surveys, highly influential surveys or surveys that are planned as ongoing or longitudinal. The main uses of cognitive pre-testing are:

  • To provide insight into how respondents react to a questionnaire:
    • Their understanding of the wording of questions and the flow of the questionnaire
    • Their ability to respond to questions accurately
    • Their thought processes as they answer the questions
  • To identify the impact of changes to an existing questionnaire (e.g., a tracking survey)

Whenever possible, schedule and budget permitting, omnibus survey questions should at least be pre-tested in-field. Whenever a pre-test had been conducted, the details of the pre-test should be documented, including the number of pre-test interviews completed.

Document "Advisory Panel On Telephone Public Opinion Survey Quality - Final Report February 1, 2007" Navigation