Standards for the Conduct of Government of Canada Public Opinion Research—Telephone surveys

About the standards

These standards apply to custom telephone public opinion research surveys conducted for the Government of Canada by contracted third parties. Custom studies are those in which ownership of the data rests with Canada.

This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Public Services and Procurement Canada for all other uses.

For more information on this publication, please contact Public Services and Procurement Canada at dgsiopinionpublique.isbpublicopinion@tpsgc-pwgsc.gc.ca.

Catalogue number: P103-8/2-2020E-PDF
International standard book number (ISBN): 978-0-660-32912-3

© Her Majesty the Queen in Right of Canada, as represented by the Minister of Public Services and Procurement, 2020

To download or print a PDF version of the standards [1.2MB]

Help with alternative formats and plug-ins

On this page

Important notes

The following standards are for custom telephone public opinion research surveys conducted for the Government of Canada Footnote 1, where respondent input is provided primarily by voice. The standards were developed to help ensure that the level of quality of these surveys is consistently high, is transparent to research users, and aligns with the intended use of the research. If you have any questions related to this document, please contact the Public Opinion Research Directorate at Public Services and Procurement Canada (PSPC) by email at dgsiopinionpublique.isbpublicopinion@tpsgc-pwgsc.gc.ca.

A. Applicability of these standards

These standards apply to custom telephone public opinion research surveys conducted for the Government of Canada by contracted third parties, including questions added to omnibus/syndicated surveys. Custom studies are those in which ownership of the data rests with Canada. The standards do not apply to syndicated studies (those in which ownership of the data rests with the researcher), unless the Government of Canada contracts to place custom questions in addition to those already included in those studies. They also do not apply to internal surveys, that is, those undertaken directly by Government of Canada departments and agencies without contracting to a third party. Nonetheless, PSPC invites all individuals to view these standards as a resource in support of the planning and execution of their internal telephone survey research projects.

To ensure that these standards are a primary consideration for researchers as they develop survey proposals, we recommend the following statement be included in all public-opinion-survey-related Statements of Work and Requests for Proposals issued by the Government of Canada:

“The Standards for the Conduct of Government of Canada Public Opinion Research must be respected and applied in whole to all aspects of the conduct of the research. The project authority must be informed should the firm determine that certain elements of the standards cannot be accommodated in the course of the contract. Justification must be provided and permission received from the project authority in writing in these instances.”

These standards should be understood to be in addition to relevant laws, regulations, and policies, for example, the Personal Information Protection and Electronic Documents Act, the Privacy Act, the Official Languages Act, the Accessible Canada Act, the Financial Administration Act, the Public Opinion Research Contract Regulations, and the Policy on Communications and Federal Identity.

B. Applicability of other standards

Where no relevant Government of Canada standards exist, researchers must meet or exceed industry standards, including those identified by the project authority (client).

C. Exceptions

These standards were developed with “typical” studies in mind. It is recognized, however, that public opinion research surveys have individual characteristics, which can sometimes clash with a “one size fits all” approach. Therefore, while departments and agencies are generally encouraged to apply the standards, they can also choose to waive a standard when they believe the characteristics of a study warrant it, or when a standard is not implementable. For example, some Government of Canada (GC) standards may not be implementable for custom questions added to omnibus/syndicated studies.

This document notes some of the more common instances in which it would be appropriate to waive or modify standards.

As departments and agencies are responsible for the quality of the research they undertake, no related procedure is required to waive or modify a standard. In keeping with due diligence, however, they must discuss the intention to waive or modify standards with the Public Opinion Research Directorate. They should also consider specifying which standard is being waived/modified and the related rationale in Statements of Work and other documents, such as researcher proposals and final reports.

D. Five-year review

The following standards were first developed in 2009. Changes in technologies and other factors necessitate their periodic revision, which is carried out approximately every five years. These standards were most recently updated in 2020.

1. Proposal documentation

In this section

Public opinion research contracts are based on the accepted final proposal from a research firm. As such, researchers must ensure that the proposal provides sufficient detail to clearly document all requirements, including those specified in the project authority’s (client’s) Statement of Work. Proposals must include all relevant information described below.

1.1. Introduction

1.1.1. Research purpose

Broadly describe the topic or issue to be examined and the overall intent of the research, including how the Government of Canada intends to use the results.

1.1.2. Research objectives

Detail the specific research objectives that the research project will address (or the key questions that will be answered) in order to achieve the broader purpose.

1.2. Technical specifications of the research

1.2.1. Overview
  1. Provide a brief statement summarizing:
    1. Data collection mode(s), method(s) used to select a sample (probability, non-probability or census), and rationale for proposed methodology.
    2. Expected final sample size (that is, number of completed interviews).
    3. Target population.
  2. State that the recommended research design is cost-effective, will produce quality results, and is appropriate to meet the objectives of the research, along with a rationale for the recommendation.
  3. Describe how accessibility for persons with disabilities will be taken into consideration during research design and implementation.
  4. Describe how respondents will be able to verify the legitimacy of the survey as a research initiative sponsored by the Government of Canada, for example by providing a contact at the department or research firm, or by using an industry-based research registration system.
  5. Identify the individual researcher(s) who will have supervising authority and accountability for the project, and state that they will be replaced only in consultation with the client and following written agreement from PSPC.
  6. Identify all subcontractors that will be used for any aspect of the research (for example, for fieldwork/data collection).
  7. State that all field staff directly involved in data collection (that is, interviewers/ recruiters) will be located in Canada, and that survey data will be stored on servers and back-up servers located solely in Canada (as specified in detail in section 13.3.).
  8. State that a signed statement of political neutrality will be submitted as part of the final research report.
  9. Agree that the executive summary and report can be posted on the Library and Archives Canada (LAC) website or on any other Government of Canada website.
  10. State that respondents will be informed of their rights under the Privacy Act, the Personal Information Protection and Electronic Documents Act and the Access to Information Act, and that respondents’ rights will be protected throughout the research process.
1.2.2. Sample / sampling details
  1. Provide details related to the target population and sample size:
    1. The definition of the target population in terms of its specific characteristics and geographic scope, including the assumed incidence of the population and any key sub-groups, and how the incidence was determined (for example, supplied by the client, previous research, or a credible external source).
    2. The total sample size and the sample composition, including the targeted sample sizes of any key sub-groups.
  2. Describe the sampling method, including:
    1. Whether a sample survey or a census will be conducted. To be described as a census, an attempt must be made to collect data from every member of the target population.
    2. If a sample survey is used, describe whether probability or non-probability sampling will be applied. To be described as a probability sample, the sampling method must meet both of the following conditions: (1) respondents are randomly selected from the survey’s target population, and (2) all units in the target population have a known, non-zero probability of being included in the sample.
    3. A rationale for the chosen sampling method (whether census, probability or non-probability) and how it supports the achievement of the research objectives. In the case of non-probability samples, the rationale must take into account the limitations in generalizing the results to the target population and acknowledge that inferential statistics must not be applied when reporting on the collected data.
    4. If non-probability sampling will be used, it must not be described as a representative sampling of the target population.
  3. Describe the sampling frame for the census or sample survey, including:
    1. The type of sampling frame(s) (for example, client list, telephone directory) and the extent to which the sampling frame covers the target population, including, if applicable:
      1. Identifying under-coverage by describing any segments of the target population that are not covered by the frame (for example, excluded provinces/territories or rural areas, households without landlines, or excluded non-panel members in the case of panel surveys). National probability samples must not exclude any area codes, including the territories.
      2. The estimated size of non-covered segments or an explanation why a size estimate cannot be provided.
      3. A description of any over-coverage and/or overlap in sampling frames (for example, when a dual sampling frame is used), in the same manner as for under-coverage.
    2. If applicable, the reason why no sampling frame or list will be used.
  4. Describe the origin of the sample for the census or sample survey, including:
    1. The sample source(s) (for example, Government of Canada department or program, online telephone directory, panel). If it will be provided by a third party, provide the name of the sample supplier.
    2. If websites or social media platforms will be used as sources for a sample survey (for example, to collect phone numbers of potential respondents), the measures that the researcher will implement to confirm respondent eligibility and authenticity.
  5. When more than one sample source or panel will be used to create the sampling frame, also provide the following:
    1. The rationale for sourcing samples from more than one source or panel.
    2. Steps that the researcher will take to ensure that individuals included by more than one of the sources or panels are not contacted to respond to the survey more than once.
    3. For sample surveys intended to produce generalizable results about the general population or most of its subgroups, the ratio of cellphone sample to landline sample and the rationale for the ratio.
  6. Describe the sample design and selection procedures for sample surveys, including:
    1. Sample stratification variables (if any).
    2. Sample selection procedures when selecting from a list (if relevant).
    3. Any multi-stage sampling steps taken (for example, enumeration areas, followed by households, followed by respondents.)
    4. Respondent eligibility/screening criteria and all variables used, and targets set, for strata sample sizes or for quota controls (for example, gender, age group), including the rationale for each target or quota control (for example, underlying benchmarks or parameters from Census data or another high-quality data source).
    5. In the case of panel samples, that respondents who are panel members may not have participated in any Government of Canada survey in the previous 30 days as a member of that panel or in a survey on similar subject matter by any mode, except when a study’s research design requires a follow-up survey with respondents who have agreed during the initial survey to be re-contacted for that purpose.
  7. When the research design or selection procedures will not allow the calculation of a response rate or participation rate, describe the design/selection procedure and any potential sampling bias that may result.
  8. Provide an assessment of how likely the targeted final sample size can be achieved by the census or sample survey, taking into account such factors as the expected response rate, and in the case of panels, the active panel size and exclusion rules involving past survey participation.
  9. When the sample will be drawn from one or more panels, describe the following for each panel:
    1. The panel identity and source.
    2. Active panel size in relation to the survey’s target population, that is, the number of panel members that correspond to the target population and that are available to be surveyed or screened.
    3. How the panel is recruited and maintained, including the recruitment sources used to populate it on an ongoing basis, and the proportion of panelists per source, if possible.
    4. Any ongoing rewards or incentives for panel members, that are not associated with taking part in a specific research project/survey.
    5. How a sample of the survey’s target population will be drawn from the panel, including whether existing panelist profile data will be used to identify eligible respondents and whether eligibility criteria will be re-validated during survey administration and, if not, how profile data is kept accurate and up to date.
    6. The incentive/honoraria respondents will be offered for taking part in the survey.
  10. Describe any additional sampling limitations or potential for sampling bias in sample surveys.
1.2.3. Response rate / participation rate and sampling error
  1. State expected response rates/participation rates for each survey mode or research stage that is proposed (that is, separate rates for telephone and online components, when both are used, and separate rates for cellphones and landlines, if applicable). The expected response rate/participation rate is an estimation based on various factors such as previous response rates/participation rates or established trends. The proposal should include a brief discussion of the factors that might cause the actual response/participation rate to fall short of the stated target. For surveys where it is not possible to calculate a response rate or participation rate, as described in section 1.2.2. (7), disclose this in the proposal.
  2. For a census or attempted census, do not state a margin of sampling error, as no sample will be drawn.
  3. For probability samples, state the level of precision, including the margin of sampling error and confidence interval for the sample(s). (See section 14.2.5. for detailed requirements related to margin of sampling error.)
  4. Describe specific procedures that will be applied to prevent or minimize the impact of potential sampling errors or non-sampling errors (for example, coverage, non-response), on the research findings.
  5. For non-probability samples, research proposals must not include any of the following:
    1. Margins of sampling error.
    2. Alternative measures of precision.
    3. Statements indicating that the sample has a level of sampling error equivalent to a probability sample of similar size.
1.2.4. Data collection
  1. State the mode of data collection (that is, telephone) and whether landline and/or cellphone will be used.
  2. For multi-mode surveys, that is, where some respondents will complete the survey via one mode (for example, telephone) and some respondents will complete the survey via another mode (for example, online), provide the following:
    1. The rationale for using a multi-mode rather than single-mode method.
    2. The rationale for the specific modes that will be used.
    3. A description of the steps that will be taken to reduce the likelihood of mode biases, and to detect any mode biases.
  3. For multi-stage surveys, where respondents complete the questionnaire in distinct stages, which may also involve switching modes between stages (for example the first part of the questionnaire is completed by phone and then the second part is completed online), provide a rationale for using a multi-stage rather than single-stage method, and a rationale for all modes that will be used.
  4. Provide details on respondents’ incentives/honoraria where applicable, including a rationale for the following (see section 6.4 for considerations regarding incentives):
    1. The type of incentive/honorarium (for example, monetary, non-monetary).
    2. The nature of the incentive (for example, cash, prizes, reward points, charitable donations).
    3. The estimated dollar value.
  5. Describe how language requirements will be met during data collection.
  6. Indicate the number of call attempts and explain call-back procedures.
  7. Describe the planned fieldwork validation methods and procedures.
  8. Describe how the rights of respondents/participants will be respected, including those of children, youth and vulnerable respondents, if applicable (see section 5.)
  9. Describe efforts to make the telephone survey accessible for all eligible respondents and to facilitate their participation, for example using assistive technologies or alternative modes of data collection (for example, online data collection).
1.2.5. Questionnaire design
  1. Provide the approximate average duration of the questionnaire in minutes.
  2. Specify the maximum number of open-ends.
  3. Describe how the questionnaire will be pre-tested, including:
    1. The objectives of the pre-test.
    2. The method for the pre-test.
    3. The number of pre-test interviews to be completed, and if relevant, the number of pre-tests for specific subgroups of the target population (for example, when questionnaire effectiveness may vary by language, age, level of education or other characteristic).
    4. How the client can monitor the pre-test if the client so chooses, and how the results of the pre-test will be documented and provided to the client.
Note

If no pre-test is to be conducted, a rationale must be provided.

1.2.6. Data processing / data management
  1. Describe safeguards to ensure respondent confidentiality and protection of personally identifiable information throughout data processing and data management.
  2. Where known in advance, describe any proposed weighting.
1.2.7. Data analysis
  1. Briefly describe how the data will be analyzed in order to address the objectives/research questions, including any special analyses (for example, segmentation, conjoint analysis).
  2. For multi-mode surveys, state whether results will be reported separately by mode and/or by combining the data across modes.
1.2.8. Deliverables

List all deliverables, including their language, format, means of delivery, and number of copies, including at a minimum:

  1. Questionnaire(s) and pre-test documentation, if relevant.
  2. Tabulated data.
  3. Report(s) and executive summary(ies).
  4. Presentation(s), including the specific locations, if relevant.
1.2.9. Project work plan

Provide a detailed work plan with scheduled dates, and identify responsibilities.

1.2.10. Project cost

Present cost information in the format designated by PSPC.

2. Questionnaire design

In this section

2.1. General

Survey questionnaires must be designed:

  1. To collect only the information relevant to the stated research objectives of the study.
  2. To minimize respondent burden while maximizing data value and utility.
  3. To be completed in an average duration of 15 minutes when administered by a live interviewer, or an average duration of five minutes for Interactive Voice Response (IVR). If supported by a strong rationale, exceptions to these duration limits could include projects with specialized audiences and those with pre-arranged interviews when the respondent is aware the survey will take longer than 15 minutes.

2.2. Questionnaire introductions

  1. The following are required elements of all Government of Canada telephone survey questionnaire introductions:
    1. In accordance with the Official Languages Act and policies, introductions must include an active offer of both official languages (English and French). The language that is offered first will depend on the province in which the respondent resides (in Quebec “Bonjour/Hello”; in the rest of Canada “Hello/Bonjour”.) When relevant to a specific survey, non-official languages may also be offered.
    2. Identify the Government of Canada or the department/agency sponsoring the survey. When there is a strong rationale, the sponsor may be identified at the end of the survey instead of in the introduction (for example, for advertising post-tests).
    3. Inform respondents of the general subject and purpose of the study. The subject and purpose of the study may be expressed in general terms, as long as the wording used cannot be construed as an attempt to misinform respondents. When the researcher makes a convincing case that stating the subject and purpose of the survey at the outset is likely to bias the sample that chooses to participate and/or to bias participants’ responses, these statements may be deferred to the conclusion of the survey.
    4. Offer alternative modes of data collection and/or assistive technologies that are available to make the survey accessible to persons with disabilities.
    5. Inform respondents of the expected length of the interview.
    6. Identify the researcher (research firm) and interviewer (a pseudonym may be used for the interviewer as long as the individual remains identifiable to the research firm’s internal management, for quality control purposes).
    7. Inform respondents that their participation in the study is voluntary and confidential.
    8. Inform respondents that their responses will remain anonymous. In exceptional cases where the research objectives require that respondents’ identity be revealed, the informed consent of the respondent must be obtained.
    9. Confirm that respondents are in a location where it is safe and convenient to be interviewed. If the interviewer perceives that the respondents’ setting may endanger them (for example, driving a vehicle), hinder their participation (for example, background distractions), or hamper their ability to respond openly (for example, lack of privacy), the interviewer should take proactive steps, such as rescheduling the interview.
    10. Inform respondents how they can verify the legitimacy of the survey as a research initiative sponsored by the Government of Canada.

The following introduction provides a mandatory first paragraph that offers respondents both official languages, as well as additional sample wording that could be used for telephone questionnaires:

“Hello / Bonjour (pause), the Government of Canada is conducting a research survey on subject. Would you prefer that I continue in English or French? Préférez-vous continuer en français ou en anglais?”

Note

If at this point the respondent prefers to respond in French, then the interviewer must be able to either proceed with the interview in French or read the following statement: « Je vous remercie. Quelqu’un vous rappellera bientôt pour mener le sondage en français. »

“My name is first name of name of firm, the company hired to do the survey. The survey takes about ## minutes to complete. Should you have any questions about the survey, I can give you a contact person within the Government of Canada / Department name. Your participation is voluntary and confidential. Your answers will remain anonymous, and the information you provide will be administered according to the requirements of the Privacy Act, the Access to Information Act, and any other pertinent legislation. Is this a safe and convenient time for you? May I continue?”

  1. Interviewers must be prepared to repeat all or parts of the survey introduction or instructions, if requested by the respondent. In addition, interviewers must have information readily available for answering respondents’ questions regarding how respondents were chosen, how privacy is protected, how they can verify that the survey is legitimately sponsored by the Government of Canada, and where the survey results can be obtained. (See section 6.1 for more information on project briefing of fieldworkers).
  2. Interviewers must be prepared to address individual respondents’ accessibility issues by offering and facilitating their participation through alternative modes of data collection and/or assistive technologies, as feasible and available.

2.3. Demographic questions

  1. For surveys of the general population, the data from the demographic questions (along with the recording of geographic location) should allow comparison with official Statistics Canada population data for the purpose of non-response analysis, and facilitate the comparison of results among Government of Canada public opinion research studies. (See section 14.2.5 for further detail on non-response bias.)
  2. The standard set of demographic questions below must be included in all telephone surveys of the general population, unless a convincing argument is made that the research objectives are better served by excluding one or more of them. This does not preclude the use of additional demographic questions.
  3. When surveying business or organizational populations, these standard demographic questions should be included whenever relevant, such as when the unit of analysis is the individual rather than the business or organization.
  4. The wording used for each demographic question and its response categories must be as provided below, unless a convincing argument is made that particular research objectives require alternative wording. Note that for certain questions (for example, age, income), the response categories may be further broken down if desired, as long as they can be reconstituted to yield the mandatory categories.

    The exception is that for demographic questions that have relatively long lists of response options that proceed linearly (for example age, household income), researchers have the option of modifying the wording of the question to allow the interviewer to instruct a respondent to stop at the category that applies to them. For example, the required wording of a question could be modified by prefacing it with an introduction like: Please stop me at the category that best describes your…(Insert demographic dimension).

  5. For screening purposes, one or more of the demographic questions may be administered near the start of the questionnaire, as required. Prior to administering demographic questions at the end of the questionnaire, respondents should be informed of the purpose of the questions, and be reassured that their answers will remain anonymous and confidential.
  6. Although the ordering of questions and the preamble wording may be modified, the following demographic questions must be used for telephone questionnaires:

[Preamble] These last few questions will allow us to compare the survey results among different groups of respondents. Your answers will remain anonymous and confidential.

Gender

What is your gender?

  • Female
  • Male
  • Other Footnote 2
  • [Do not read] Prefer not to answer
Age

In what year were you born? [Record year – YYYY]

[If the respondent prefers not to provide a precise birth year, ask:]

Would you be willing to tell me in which of the following age categories you belong?

  • 18 to 24
  • 25 to 34
  • 35 to 44
  • 45 to 54
  • 55 to 64
  • or 65 or older?
  • [Do not read] Prefer not to answer
Education

What is the highest level of formal education that you have completed? [Read list]

  • Less than a High School diploma or equivalent
  • High School diploma or equivalent
  • Registered Apprenticeship or other trades certificate or diploma
  • College, CEGEP or other non-university certificate or diploma
  • University certificate or diploma below bachelor's level
  • Bachelor's degree
  • Post graduate degree above bachelor's level
  • [Do not read] Prefer not to answer
Language Spoken at Home

What language do you speak most often at home? [Accept all that apply]

  • English
  • French
  • Other [Specify or do not specify depending on the needs of the study]
  • [Do not read] Prefer not to answer
Household Income

Which of the following best describes your total household income last year, before taxes, from all sources for all household members? [Read list - accept one answer only]

  • Under $20,000
  • $20,000 to just under $40,000
  • $40,000 to just under $60,000
  • $60,000 to just under $80,000
  • $80,000 to just under $100,000
  • $100,000 to just under $150,000
  • $150,000 and above
  • [Do not read] Prefer not to answer
Household Telephone Status

Does your household subscribe to a home phone service, also known as a landline?

  • Yes
  • No
  • Don’t know
  • [Do not read] Prefer not to answer

How many working cellphones does your household have?

  • 0
  • 1
  • 2
  • 3
  • 4 or more
  • Don’t know
  • [Do not read] Prefer not to answer
Geographic Location

May I have the first three digits of your postal code?

  • First three digits of your postal code
  • [Do not read] Prefer not to answer
  1. Additional demographic questions may be included when required to satisfy the survey objectives. When a demographic question from the list below is included in a survey, the researcher is strongly encouraged to use the exact wording provided for both the question and its response categories, in order to facilitate the cross-comparability of surveys commissioned by the Government of Canada.
Disability

Do you identify as a person with a disability? A person with a disability is a person who has a long-term or recurring impairment (such as vision, hearing, mobility, flexibility, dexterity, pain, learning, developmental, memory or mental health-related) which limits their daily activities inside or outside the home (such as at school, work, or in the community in general).

  • Yes
  • No
  • [Do not read] Don’t know
  • [Do not read] Prefer not to answer
Sexual Orientation

What is your sexual orientation?

  • Heterosexual (straight)
  • Homosexual (lesbian or gay)
  • Bisexual
  • Other [Specify or do not specify depending on the needs of the study]
  • [Do not read] Prefer not to answer
Mother Tongue

What is the language you first learned at home as a child and still understand? [Read list - accept all that apply]

  • English
  • French
  • Other [Specify or do not specify depending on the needs of the study]
  • [Do not read] Don’t know
  • [Do not read] Prefer not to answer
Employment Status

Which of the following categories best describes your current employment status? Are you… [Read list - accept one answer only]

  • Working full-time, that is, 35 or more hours per week
  • Working part-time, that is, less than 35 hours per week
  • Self-employed
  • Unemployed, but looking for work
  • A student attending school full-time
  • Retired
  • Not in the workforce [Full-time homemaker, unemployed, not looking for work]
  • [Do not read] Other—[Do not specify]
  • [Do not read] Prefer not to answer

2.4. Multi-mode questionnaires

When designing questionnaires for multi-mode surveys, researchers should:

  1. Ensure as much comparability as possible in the wording, ordering and presentation of questions and response options across the different survey modes, to facilitate combining and comparing the collected data.
  2. Include benchmarking questions for which there are high-quality population data available to facilitate the detection of any mode biases, for example, demographic, behavioural or attitudinal questions that may correlate with key measures in the survey.

3. Pre-testing

3.1. General

  1. Pre-test all components of a new or revised survey questionnaire that may influence data quality, respondent behaviour, or interviewer performance.
  2. Include probing that invites pre-test participants to provide input about their comprehension of, and reaction to, the questions. If requested by the client, a full cognitive pre-test should be conducted.
  3. If the client requires it, target key socio-demographic characteristics when recruiting pre-test participants, to ensure questionnaire effectiveness for key subgroups (by age, level of education, etc.)
  4. Provide a means for the client to monitor pre-tests if the client so chooses (for example, by telephone, in-person, or by audio-recording).
  5. Conduct a minimum of 10 pre-test interviews in each language in which the final survey will be fielded. An exception could be projects with small survey populations, in which case the researcher must seek the client’s approval to conduct a smaller number of pre-tests to ensure questionnaire quality; alternatively a cognitive pre-test may be warranted.
  6. For multi-mode surveys, if one mode is interviewer-administered and another mode is self-administered by the respondent, conduct a minimum of 10 pre-test interviews in each language per mode.
  7. Pre-test completions must not be included in the final dataset. Exceptions could be:
    1. Projects with hard-to-reach target groups.
    2. When no changes are made to the questionnaire following the pre-test.
  8. Results of the pre-test(s) must be shared with the client before the questionnaire is finalized and approved. If shared verbally, written documentation of the pre-test results must follow as soon as possible afterwards. The documentation must include (at a minimum):
    1. A description of the pre-test approach and number of interviews completed in each language.
    2. Findings and any resulting modifications.
    3. Average survey completion time.
    4. A statement of whether or not pre-test cases will be retained in the final data set.

4. Retaining public confidence

In this section

4.1. General

  1. Researchers must make every reasonable effort to gain, retain and increase public confidence in their own organizations, and in public opinion research conducted for the Government of Canada.
  2. Researchers must act in a manner that respects survey respondents and ensures their rights are protected as they relate to participation in Government of Canada survey research. Interviews must be conducted free of embarrassment and in an environment of trust and goodwill.
  3. The researcher must take all reasonable steps to ensure that respondents are not in any way harmed or embarrassed by any interview, and that they are not in any way adversely affected as a result of it. Researchers must address sensitive subject matter in a way that will minimize discomfort and apprehension for both respondents and interviewers.

4.2. Respondent rights and protections

4.2.1. Informed consent and use of information
  1. In obtaining the necessary agreement from respondents to participate, the researcher must inform them of the study sponsor and of the general subject and purpose of the survey (subject to exceptions described in section 2.2. paragraph 1), and that their participation is voluntary and confidential.
  2. Survey questions must be limited to gathering information relevant to the stated research objectives. The researcher must ensure that the data collected will not be used for any other purpose, unless the respondent provides explicit informed consent for that additional use (for example, to help resolve a customer complaint).
  3. The survey data and personal information must not, under any circumstances, be used for direct marketing or other sales approaches to the respondent or the respondent’s household.
  4. Researchers must answer any questions that the respondent has about the research in a clear, honest, and non-deceptive manner.
4.2.2. Right to refuse

Researchers must respect the right of a respondent to refuse to participate in a survey or to respond to particular questions, and to terminate the interview at any point.

4.2.3. Language of participation

Interviewers must include an active offer of both official languages (English and French) in questionnaire introductions, in accordance with the Official Languages Act and policies.

4.2.4. Accessibility

Researchers are required to make telephone surveys accessible for eligible respondents when standard data collection practices impede their participation, by accommodating their needs, for example, through assistive technologies, message or video relay services, or alternative modes of data collection, as feasible. Researchers should inform respondents of the available accommodations or alternatives.

4.2.5. Identity of interviewer and research firm

Interviewers must clearly state their name (first name or unique identifying nickname is sufficient) and the name of the research firm. If a respondent requests it, interviewers must provide sufficient contact information to allow them to re-contact the researcher without difficulty.

4.2.6. Protection of anonymity and confidentiality
  1. The anonymity of respondents must be preserved, unless they have given their informed and explicit consent to the contrary. The researcher must ensure that the information will be used for research purposes only, or, if requested by the respondent, to resolve a customer complaint. The same holds true when respondents’ answers are, with their informed consent, linked or merged with pre-existing data that allows their identification (for example, with administrative data from a Government of Canada program). Information must not be used for any non-research purpose, such as direct marketing, list-building, credit rating, fund-raising, or any marketing activities directed at those individual respondents.
  2. Completed questionnaires or other material (for example, digital media) containing respondent identity, or information that might allow respondents to be identified, must not be released by researchers to clients or other third parties.
  3. When verbatim comments are collected, they must not be attributed to the respondent, directly or indirectly, without the respondent’s explicit informed consent. Care must be taken when reporting verbatim comments to ensure that nothing in the comment, including content, vocabulary and/or style of speech, could serve to identify the individual respondent.
  4. When recording or observation techniques will be used, researchers must advise the respondent at the beginning of the interview.
4.2.7. Use of lists
  1. Where lists are used for sample selection, the source of the list must be disclosed to potential respondents, if respondents ask. When the sample comprises individuals who are not employees of the Government of Canada, researchers must ensure that lists are permission-based for research purposes, including, when appropriate, obtaining written assurances from the list provider (who may be the client) that this is the case.
  2. When the sample comprises employees of the Government of Canada, employee lists may be used even if no explicit permission to use that information for survey purposes has been obtained, as long as the survey topic is relevant to the respondents’ employment (for example, an employee satisfaction survey).
  3. Client-supplied lists provided for specific projects must not be used for any other project or for adding names to the researcher’s databases or panels. If they have been updated by the researcher during the project (for example, to correct errors), those updated lists must be returned to the client upon completion of the project. All client-supplied lists still in the researcher’s possession at project completion must be destroyed after any updates have been forwarded to the client.

4.3. Avoiding nuisance or harassment

  1. Using the respondent’s local time, telephone interviewing can be conducted only from 9:00 to 21:00 Monday to Friday, 10:00 to 21:00 Saturday, and 12:00 to 21:00 Sunday and statutory holidays, except when a pre-arranged interview is requested by the respondent outside these hours.
  2. There must be no or minimal (one-second) pause before the interviewer acknowledges that a potential respondent has answered the telephone.

4.4. Use of interactive voice response

The Government of Canada usually wishes to engage Canadians in a personal manner when conducting telephone interviewing, rather than using an automated approach.

However:

  1. Interactive Voice Response (IVR) surveys can be a suitable alternative data collection method when the specific information to be collected is essential for making important decisions and cannot be obtained effectively through other means. For example,
    1. When the opinions of a hard-to-reach (low-incidence) group are critical to the research objectives, and the very high call volume made economical by IVR is likely to markedly increase participation from members of that group in the survey.
    2. For time-sensitive surveys where the field duration is very short.
    3. As part of a mixed-mode design, for example, when respondents identified through another survey mode are directed to an IVR, or a design in which there is switching between a live interviewer and IVR.
    4. When there is significant risk of social desirability bias, and for recruitment of participants for other phases of research (for example, to complete a longer online survey).
    5. When respondents have agreed beforehand to this method.
  2. When IVR is used, the survey introduction must include the same information required for interviews conducted by live interviewers (active offer in English and French, sponsor, purpose, researcher, participation is voluntary, assurance of confidentiality, etc.)
  3. As required by the Unsolicited Telecommunications Rules, Part IV, section 4(d), of the Canadian Radio-television and Telecommunications Commission (CRTC), the introduction must provide an email address or postal mailing address and a local or toll-free number at which a representative can be reached, in addition to identifying the survey sponsor and purpose. If the IVR message exceeds sixty seconds, the same information must be repeated at the end of the call.
  4. Early in the introduction, respondents must be provided with an easy method to opt out of the survey (for example, by pressing a specific key) so that the call is terminated gracefully and no more calls are made to that number.
  5. The same requirements for the time-of-day of calls and delay in acknowledging an answered call (see section 4.3.) apply to the use of IVR.
  6. IVR surveys must follow the relevant rules in the Unsolicited Telecommunications Rules of the CRTC.

4.5. Photographs and recordings

Photographs, video and audio recordings collected from survey respondents must be collected, processed and stored as personal data. They can be shared with a client only after respondents give informed consent, with knowledge of the specific purpose for which it will be used and once the data has been stripped of all personally identifiable information (such as through pixelization or voice modification). The client must agree to make no attempt to identify individual identities.

If respondents are asked to make recordings as part of their survey input, they should be given specific limited tasks (for example taking images of objects) that do not have the potential to capture the personal data of others.

5. Data collection from children, young people or vulnerable respondents

In this section

5.1. General

  1. Researchers must take special care when interviewing children and young people and respondents who are otherwise vulnerable. (A "child" is defined as "under the age of 13" and a "young person" as "aged 13-15".) The welfare of children, young people and vulnerable persons must be the researcher’s primary consideration. These respondents must not be disturbed or harmed by the interview experience. Sensitive topics or issues that could upset children and young people (for example, relationships with other children or with parents, sexual activities, use of drugs or alcohol) or vulnerable persons (for example, social embarrassment related to the source of their vulnerability) must be dealt with using special care, to minimize any discomfort or apprehension.
  2. All explanations related to data protection, privacy policy, adult consent and other notices must be capable of being understood by children.
  3. In addition to these standards, researchers must observe all relevant laws specifically related to children and young people.

5.2. Consent of responsible adult

  1. When interviewing a child or young person under 16, interviewers must obtain the consent of a responsible adult (parent or guardian) before the child/young person is approached for an interview. Sufficient information must be provided to this responsible adult to enable an informed decision about giving such consent. In particular, the researcher must specify the nature of any potentially sensitive questions when seeking parental/guardian consent for the child’s/young person’s participation in the research. The name of the responsible adult giving the consent must be recorded, as well as that person’s relationship to the child/young person.
  2. The responsible adult’s consent allows the interviewer only to invite the child/young person to participate. The child/young person must be given an opportunity to agree or refuse to take part in the survey. To that end, the researcher must inform the child/young person of the purpose of the research in terms that the person is likely to readily understand.
  3. When conducting telephone interviews, it might be difficult to determine with certainty the age of the respondent and whether or not the consent of a responsible adult is required. The researcher must make every reasonable effort to comply with the requirements set out in these standards.
  4. When it is not possible to obtain parental/guardian consent (for example, street kids) and the client deems that the benefits of the research will outweigh the possible harm to respondents (for example, youth engaged in risky behaviour), the client may choose to waive the researcher’s responsibility to follow this standard.

5.3. Adult accompaniment

For studies with sensitive subject matter, it is recommended that interviewers suggest that a responsible adult (aside from the interviewer) remains close – not necessarily in the same room – while the interview is carried out, in the event that the child or young person reacts negatively during the interview and requires attention.

5.4. Collection of personal information

Personal information relating to other people (for example, parents) must not be collected from children. Consistent with the Privacy Act and the Personal Information Protection and Electronic Documents Act, “personal information” means information about an identifiable individual. The researcher may collect the contact information of a parent or guardian for purposes of obtaining parental consent.

6. Data collection

In this section

6.1. Project briefing

  1. Fieldworkers must be given a briefing and/or instructions for each project.
  2. The briefing and/or instructions will be the responsibility of a staff member with a full understanding of the project requirements.
  3. Briefings and/or instructions should be delivered face-to-face whenever possible; however, other means are acceptable (for example, phone or recordings). Records must be kept to show the content of the briefing/instructions and that all fieldworkers allocated to the project were briefed/instructed. Briefings and/or instructions in writing only should be avoided.
  4. The briefing must be of sufficient detail as to ensure that each interviewer is familiar with the questionnaire, the sample and respondent selection procedures, and potential problem areas in the administration of the survey. Interviewers must be familiar with question wording before commencing the interviews. Normal standards of practice include rehearsal interviews and pre-testing.
  5. The briefing and/or instructions should also include, as appropriate, the following information:
    1. Purpose of the project.
    2. Sponsor of the research.
    3. How the information will be used.
    4. Means for respondents to verify survey legitimacy (for example, client contact information).
    5. Fieldwork dates.
    6. Sampling procedure.
    7. Quotas to be attained.
    8. Methodology.
    9. Alternative modes of data collection and/or assistive technologies that are available to address respondents’ potential accessibility issues.
    10. Other special requirements of the project.
    11. Instructions/conditions for administration of the questionnaire.

6.2. Call attempts

  1. For landline telephone numbers and cellphone numbers, researchers must make a minimum of five call attempts before retiring a number or substituting it with another number. The call attempts must be made at varying days and times over a minimum seven-day period, except when the field period is too short to allow this.
  2. Call attempts must be recorded and retained as part of the disposition of the contact sample, so that outcomes rates may be calculated, as per section 7.
  3. If an appointment is arranged, every effort must be made to call back the respondent on the agreed date and time.
  4. Researchers must make no attempt to call back refusals, except when the prospective respondent indicates that a call back may be convenient at another time.

6.3. Fieldwork validation

6.3.1 General
  1. Researchers will carry out validation to establish that data collection by fieldworkers has been carried out according to project instructions, including administering questionnaires in the manner that they were designed to be administered.
  2. Validation must be carried out as soon as practical after the fieldwork, and before the resulting data are processed and/or reported to clients.
  3. Clients must be given an opportunity to check the quality/validity of fieldwork, subject to any applicable legal constraints.
6.3.2. Validation methods
  1. Depending on the type of data collection, validation must be undertaken as specified below, by means of ongoing monitoring, checking the data records produced (for example, questionnaires, Computer-Assisted Telephone Interview (CATI) data files), and/or re-contacting respondents.
  2. Ongoing monitoring must be carried out throughout the field period. Monitoring will involve listening to interviews as they are being carried out (using appropriate equipment) or listening to recordings of the interviews. When monitoring an interview recording, this must be carried out within 24 hours after the interview was conducted.
  3. Checking of data records may involve, as appropriate, validating the completeness of the data records, that strata sample sizes or quotas are respected, that responses are consistent, and comparing responses against normal data or between fieldworkers. Checks may be made manually or by computer.
  4. Respondent re-contact must include confirming that the interview took place and that instructions were followed, the length of the interview, and responses to key questions, including demographics and other qualifying questions related to strata/quotas, etc. Re-contact may be by any medium involving direct communication with the respondent (for example, face-to-face, by phone, or by email).
  5. Both interviewer and respondent must be audible. Interviewers should be aware that any interview may be monitored, but not know whether a specific interview is being monitored. Where interviewing is multilingual, staff fluent in the relevant languages must carry out monitoring.
6.3.3. Validation levels
  1. The researcher must validate the data collection, by either monitoring or re-contacting respondents to the levels shown below, whether or not data records are also checked. Validation levels must be calculated on the basis of achieved interviews and achieved validations.
    1. The minimum monitoring validation level must be 10%of the interviews/cases, with at least 75% of the whole interview monitored/listened to.
    2. The minimum re-contact validation level must be 10% of the interviews/cases.
  2. Every fieldworker working on a project must be validated or monitored.
  3. If re-contact validation identifies frequent or recurring problems with an individual fieldworker’s work, that fieldworker’s interviews (on the project) must be 100% validated, and any non-validated interviews must be rejected.
  4. In exceptional cases, it can be organizationally impossible to carry out re-contact or monitoring to the required level, or at all, or it may be considered contrary to respondents’ interests. In such cases, project records must explain why this is the case and what other steps (for example, checking data records) have been taken to validate data collection.

6.4. Incentives / honoraria

6.4.1. General
  1. When including incentives/honoraria in a project, researchers must ensure that it is legal and ethical to do so. Survey incentives must not be coercive or unduly affect a respondent’s choice to voluntarily participate.
  2. Researchers should consider whether incentives/honoraria are necessary for achieving the research objectives within the project timeline, and whether alternative means could be effective for motivating participation.
  3. When selecting the type and value of incentives/honoraria, researchers should minimize their potential to introduce bias or error in the final sample, for example, by affecting the response rates of various segments of the target population differently.
6.4.2. Children, young people or vulnerable individuals

When survey respondents are children, young people or vulnerable individuals, the researcher must:

  1. Take ethical considerations into account before including an incentive or determining its type and value (for example, ensure payment is not coercive, and does not expose children, young people or vulnerable individuals to a risk that they would otherwise have avoided).
  2. Decide in advance who will receive an incentive, that is, the responsible adult (parent or guardian), the respondent, or both.
  3. Obtain the agreement of the parent or guardian before offering any incentive to the child, young person, or vulnerable individual.
6.4.3. Incentives for Government of Canada employees
  1. Incentives should not be provided to employees of the Government of Canada for research conducted while on duty (that is, during working hours).
  2. Incentives may be offered to employees of the Government of Canada for research studies conducted while off duty (that is, outside of working hours), as long as:
    1. Employees are made aware that their acceptance cannot be misconstrued as an official duty and therefore subject to overtime.
    2. Employees are made aware that they must consult the Treasury Board Directive on Conflict of Interest or their organizational policy on conflict of interest, where applicable, and consider whether their acceptance of the incentive is likely to give rise to a conflict of interest with their official duties, to constrain their capacity to perform their official duties, or to undermine the neutrality of the public service.
    3. If they have doubts as to whether accepting an incentive could breach the Directive, they should discuss the situation with their manager and, if necessary, report and seek written direction from their deputy head or their delegated authority.

7. Outcome rates

In this section

7.1. General

  1. Researchers should apply the definitions below when assigning cases to each case category and when calculating outcome rates.
  2. Researchers must calculate the response rate/participation rate for telephone surveys in the manner described below, and include it in the final report that the Government of Canada’s client will submit to Library and Archives Canada, as per the Public Opinion Research Contract Regulations.
  3. Monitoring of call dispositions/reasons for non-response must be carried out on an ongoing basis, throughout the entire field period, and this information must be provided to the client upon request.

7.2. Case category definitions

Telephone survey cases should be assigned to four broad categories:

Invalid numbers
These can include only clearly invalid telephone numbers when surveying households, such as numbers not in service, fax/modem, and business/non-residential.
Unresolved (U)
These include all the cases where it cannot be established whether the call was made to an eligible or an ineligible respondent or unit (for example, household). Examples include when a line is always busy, there is no response, or an answering machine or voice mail is reached but it cannot be established that this is not a household.
In-scope non-responding (IS)
These include all refusals, either implicit or explicit, whether at the household or respondent level; early break-offs of known eligible respondents; and other eligible non-respondents (for example, illness/incapacity, language barrier).
Responding units (R)

These include cases (for example, households) who would have participated but who were disqualified by the interviewer (for example, not of the language group targeted; no one 18 years old or older living at this address). It also includes all completed interviews or partially completed interviews that meet the criteria set by the researcher to be included in the analysis of the data.

Unresolved (U), in-scope (IS), and responding units (R) are all included in the broad category of “potentially eligible” cases. However, invalid numbers/cases are not included in the calculation of outcome rates.

7.3. Outcome rates definitions

Researchers must calculate and report the response rate/participation rate as defined below. However, the other rates (cooperation, refusal and contact) are optional; researchers are encouraged to include them in reports when the rates help to understand the causes of low response rates or provide context and insight regarding data quality.

Response Rate/Participation Rate
Total number of responding units divided by the total number of potentially eligible cases (including those whose eligibility could not be determined).
Response Rate/Participation Rate = R/(U + IS + R)
The term “response rate” pertains only to probability sample surveys and censuses, while the term “participation rate” pertains only to non-probability sample surveys.
Cooperation Rate
Total number of responding units divided by the total number of known eligible cases (excluding those whose eligibility could not be determined).
Refusal Rate
Total number of refusals (including break-offs) divided by the total number of potentially eligible cases (including those whose eligibility could not be determined).
Contact Rate
Total number of contacts that were made successfully (regardless of whether or not a contact was made with the targeted respondent, for example, when a family member who does not qualify intercepts the call) divided by the total number of potentially eligible cases (including those whose eligibility could not be determined).

7.4. Outcome rates for multi-mode surveys

When calculating outcome rates for mixed-mode surveys, researchers should calculate the outcome rate for each mode or stage of sampling, and if feasible and appropriate, calculate a single aggregated outcome rate.

8. Data entry

  1. The researcher must ensure that data entry specifications for CATI accurately reflect the client-approved questionnaire.
  2. In rare cases where data are captured on paper questionnaires and entered into a database or other software:
    1. The researcher must systematically verify data entry at a minimum level of 10% of completed surveys.
    2. The work of each employee performing data entry must be verified, and verification must be undertaken by a second person.
    3. If an employee’s work contains frequent errors, that employee’s work (on the project) must be 100% verified and all errors corrected. If necessary, appropriate retraining must be given to that employee until error rates are acceptable.

9. Coding of open-ended responses

In this section

9.1. Developing code frames

  1. The initial code list/frame must be developed based on a systematic review of a minimum of 10% of open-ended responses and 50% of partial open-ended responses, where a frame does not already exist. The initial code list/frame must be provided to the client for approval.
  2. The researcher must ensure that coders working on the project are fluent in the language of the collected data that they are coding, and provided with instructions and training, which must include, as a minimum:
    1. An overview of the project.
    2. Identification of questions or variables to be coded.
    3. The minimum proportion or number of completed questionnaires used to produce code frames.
    4. Where necessary or appropriate, specific segments of the sample to consider when developing code frames (for example, regions, user or non-user).
    5. Guidelines for the inclusion of codes in the code frame (for example, decisions or rules regarding what must be included or excluded from a given code).
    6. Any use to be made of code frames from a previous project or stage.
    7. Any other requirements or special instructions specific to the project.

9.2. Coding procedures

  1. Where “don’t know” and “no response” options have been used during data collection, these must remain distinguishable from each other during coding.
  2. “Other” or catch-all categories must not exceed 10% of responses to any question.
  3. Researchers must ensure that there is a systematic method of verifying the accuracy of coding for a minimum of 10% of questionnaires per project, and that the verification must be undertaken by a second person.

10. Data editing / imputation

10.1. General

  1. Researchers must keep an accurate record of any changes made to the original data set for a minimum of three years, to ensure that any changes undertaken can be replicated at a later date. All imputation processes used, and their logic, must be documented and provided to the client. All edit specifications must be documented.
  2. Data editing/imputation must be used cautiously. The degree and impact of imputation must be considered when analyzing the data, as the imputation methods used may have a significant impact on distributions of data and the variance of estimates.
  3. In those rare cases where data are captured in paper questionnaires and editing is required prior to data entry:
    1. It must be possible to distinguish the original answers of the respondent or interviewer from the codes or answers allocated by the person(s) carrying out the editing.
    2. The logic and rules applied must be documented, and employees performing the editing must be briefed about the types of verification and corrections they should carry out.

11. Data analysis

In this section

11.1. Inferences and comparisons

Researchers must base statements of comparisons and other statistical conclusions derived from survey data on generally accepted statistical practice.

11.2. Analysis records

Researchers must keep accurate and descriptive records of the analysis process for a minimum of three years, to ensure that any analysis undertaken can be replicated at a later date.

11.3. Data analysis verification

Researchers must establish and follow procedures to verify the tabulations and other outputs to ensure the following:

  1. Their completeness, that is, that all tables are present as specified, including the results of all reported significance tests.
  2. That abbreviations for headings or open-ended responses accurately reflect the full content.
  3. That the base for each table is correct against other tables or frequency counts.
  4. That the standard breaks/banner points are checked against source questions.
  5. That all derived data items are checked against their source.
  6. That the figures for subgroups and nets are correct.
  7. That weighting is applied appropriately (for example, by test tables).
  8. That frequency counts are run prior to running tables, in order to ensure the accuracy of data and to determine base sizes for subgroups.
  9. Spelling and legibility.
  10. That any statistical analysis used is appropriate and correct, in both its descriptive and inferential aspects.

12. Delivery of data tables

In this section

12.1. Producing standalone data tables

When data are provided to the client, the following must be included, as appropriate:

  1. The source question to which the data pertain.
  2. A description of any weighting method applied to the data.
  3. Clear identification of any subgroups used.
  4. The sample bases for each question (that is, the number of respondents who have actually answered the question).
  5. The number or proportion of respondents who replied “don’t know” or gave no answer.
  6. Both weighted and unweighted bases.
  7. A definition and explanation of all variables used in the analysis of the data, such as any significance testing, indexing, scoring, scaling and calculations of means, medians, modes and standard deviations.
  8. The types of statistical tests being used and their level of precision.
  9. Information on cell suppression and other measures to assure confidentiality.
  10. Warnings on results that may be unreliable because of very small sample sizes.

12.2. Electronic data delivery

Researchers must provide the client with a data file containing individual records in the original language of each interview, which has been checked for the following:

  1. Completeness (that is, all records are in the file).
  2. Inclusion of all appropriate documentation to allow for replication of the data analysis and additional analyses, including, where applicable:
    1. Labelling of the contents of the file, that is, fully labelled variables and value labels.
    2. Identification and description of any computed or recoded variables, and instructions on limitations of use.
    3. Labelled weighting variables and a description of how these were applied.
    4. All personal identifiers per the Personal Information Protection and Electronic Documents Act have been removed from the files (that is, any information that could identify specific individuals).

12.3. Respondent confidentiality

  1. To ensure respondent confidentiality, tabulated data must not show demographic or respondent characteristic categories with a column or row total of fewer than 10 respondents.
  2. Particular care must be taken when respondent verbatim statements are included in data files.

13. Data security and storage

In this section

13.1. Retention of technical data

Researchers must, for a period of three years, maintain the technical data that are not already included in the final report and provide them to the client, if requested. Technical data to be maintained include:

  1. The information pertaining to data processing and analysis, such as:
    1. Raw data files.
    2. Code frames.
    3. Project files, including project management information and survey programming files.
    4. Relevant correspondence.
  2. Documentation of all changes must be kept when the format, content or layout of the raw data has been changed in any way, for example through editing, cleaning, or recoding the data.

13.2. Retention of personally identifiable information

After their contract ends, researchers should retain personally identifiable information or data (whether provided by the client for sampling or data collection, or collected during the research) for no longer than one year, without a strong rationale agreed to by the client.

13.3. Protection of data / servers

  1. Researchers must use up-to-date technologies to protect survey data collected or stored on servers or on portable devices (for example, laptops, USB keys) against illegal or unsanctioned access by third parties (that is “hacking”). The researcher must also control access to all databases on which any data relating to the survey are stored, so that only individuals with the appropriate security clearance are able to access the database, either by using a password or other form of access control (such as biometric controls).
  2. Because some jurisdictions allow their authorities, under certain circumstances, to access all data stored on servers located in that jurisdiction (for example, in the United States under provisions of the Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism Act, known as the USA Patriot Act: see the Treasury Board Secretariat’s overview), the researcher must protect data against legally sanctioned access by ensuring that all call centres and interviewers employed in survey data collection be located in Canada. The researcher must also ensure that all databases containing any information related to the survey be stored on servers and back-up servers located solely in Canada.
    1. If the client has first consented in writing, these call centres and/or servers may be located in another country where:
      1. Equivalent protections are given to personal information as in Canada under legislation such as the Privacy Act, R.S. 1985, c. P-21, and the Personal Information Protection and Electronic Documents Act, S.C. 2000, c. 5, and under any applicable policies of the Government of Canada.
      2. The laws do not allow the government of that country or any other entity or person to seek or obtain the right to view or copy any information relating to the survey without first obtaining the client’s written consent.
    2. In connection with giving consent to locating a call centre and/or server in another country, the client may, at the client’s option, require the researcher to provide a legal opinion (from a lawyer qualified in the foreign country) that the laws in that country meet the requirements of subsection 2) a) above, or may require the researcher to pay for the Government of Canada to obtain such a legal opinion. The Government of Canada has the right to reject any request to store survey data in a country other than Canada if there is any reason to be concerned about the security, privacy, or integrity of the data. The Government of Canada may also require that any data sent or processed outside of Canada be encrypted with Government of Canada-approved cryptography, and that the private key required to decrypt the data be kept in Canada, in accordance with key management and storage processes approved by the Government of Canada.
    3. The researcher must ensure that all servers, including back-up servers, on which any data relating to the survey are stored, be physically and logically independent (meaning there is no direct or indirect connection of any kind) from all other databases, unless those databases are located in Canada (or in another country approved by the client under subsection 2) a)) and otherwise meet the requirements of this section.
    4. The researcher must ensure that all data relating to the survey be processed only in Canada or in another country approved by the client under subsection 2) a).
    5. The researcher must ensure that all domestic network traffic (meaning traffic or transmissions initiated in one part of Canada to a destination or individual located in another part of Canada) be routed exclusively through Canada, unless the client has first consented in writing to an alternative route. The client will consider only requests to route domestic traffic through another country that meets the requirements of subsection 2) a).
  3. The researcher must not subcontract (including to an affiliate) any function that involves providing a subcontractor with access to any data relating to the survey unless the client first consents in writing.
  4. Protection of data from international surveys: When the target population for a survey comprises individuals residing in a jurisdiction outside Canada, researchers must ensure that data be stored in a manner consistent with the relevant laws of that jurisdiction.
  5. Protection against physical damage to servers: Researchers must also put in place measures to ensure the physical security of data and servers.

13.4. Temporary storage of data on servers

If the temporary storage of data collected takes place on a server that is operated by another provider, the researcher must place the provider under the obligation to take the necessary steps to ensure that the requirements described in section 13.3 be met. Temporary storage of the collected data on the server must be terminated at the earliest possible time.

13.5. Transmission of data internationally

Before data are sent over the Internet to another country, researchers must check with competent authorities that the data transfer is permissible. The recipient may need to provide safeguards necessary for the protection of the data and ensure compliance with the requirements described in section 13.3.

13.6. In the event of any data breach

In the event of any data breach, the client must be informed immediately, and provided with details about both the nature and the extent of the data breach.

14. Survey report requirements

In this section

This section describes the survey report requirements that are mandatory under the Public Opinion Research Contract Regulations for all public opinion research conducted by an external research firm for the Government of Canada, as well as other legal, policy-based and administrative reporting requirements. Contracted research firms are responsible for complying with the Public Opinion Research Contract Regulations. Federal institutions must ensure that when contracting for public opinion research, the responsibilities for meeting all other mandatory reporting requirements are clearly assigned among the contractor, the federal institution (client), or other parties.

14.1. Regulatory and administrative mandatory requirements

It is a legal requirement for federal institutions to send written reports to Library and Archives Canada (LAC) within six months of the completion of the data collection for all contracted public opinion research.

Pursuant to the collective requirements of the Public Opinion Research Contract Regulations, the Directive on the Management of Communications (Appendix C: Mandatory Procedures for Public Opinion Research), LAC deposit instructions, and PSPC instructions, final reports must be submitted separately in English and French and include the following, at a minimum:

  1. On the covering page:
    1. The title of the project.
    2. The name of the research firm that entered into the contract.
    3. The contract number, the contract value, and the award date.
    4. The POR Registration Number.
    5. The delivery date (the date the final version of the report was received by the project authority).
    6. The name of the client department or agency sponsoring the research.
    7. The departmental contact information in the form of a generic email address.
    8. The departmental signature and the Canada wordmark.
    9. For the English report, the statement « Ce rapport est aussi disponible en français. »
    10. For the French report, the statement “This report is also available in English.”
  2. Copyright information, including the Government of Canada catalogue number, the International Standard Book Number (ISBN), copyright notice and year of publication, and departmental notice on the rights to reproduce the report.
  3. A narrative executive summary, submitted separately in English and French and consisting of, at a minimum:
    1. A statement of the research purpose and objectives.
    2. A summary of key findings, except where the person who entered into the contract is not responsible for the design, development of the methodology and analysis of the research.
    3. A brief description of the methodology used.
    4. A statement as to the extent to which the findings can be extrapolated to a broader audience.
    5. An outline of how the results were used, if possible, and if not, how the information is expected to be used.
    6. A political neutrality certification signed by the research firm.
    7. The contract value of the POR study.
  4. Appendices containing:
    1. A full set of tabulated data submitted separately in English and French, which, to ensure respondent confidentiality, must in no case show demographic or respondent characteristic categories with a column or row total of fewer than 10 respondents.
    2. Sample size, sampling procedures and dates of research fieldwork.
    3. A discussion of the potential for bias in the research findings due to sampling errors or non-sampling errors (for example, coverage, non-response).
    4. If applicable, weighting procedures, the confidence interval, and the margin of error.
    5. If applicable, the response rate/participation rate and method of calculation.
    6. The research instruments in all languages and, if applicable, the test material in all languages in which they were used and tested.
    7. All other information about the execution of the fieldwork that would be needed to replicate the research initiative.

14.2. Additional reporting requirements

In addition to the reporting requirements described in 14.1, the following minimum details must be documented in the project report. These allow the reader to understand how the research project was conducted and the implications of its results.

14.2.1. Background
  1. Detailed description of the project background including, at a minimum:
    1. Research purpose.
    2. How the research will be used.
  2. Specific research objectives, research questions.
14.2.2. Sample
  1. Whether a sample survey or a census was conducted.
  2. Detailed description of the following:
    1. The target population for the research project and its geographic location (where applicable), as well as any key sub-groups.
    2. The sample source(s), and if the sampling frame and/or the sample itself was provided by a third party, the name of the sample supplier.
    3. The sampling frame(s) (for example, lists, telephone records) and the extent to which the sampling frame covers the target population, identifying under-coverage by describing any segments of the target population that are not covered by the frame (for example, excluded provinces/territories or rural areas, households without landlines, or excluded non-panel members in the case of panel surveys), and the estimated size of non-covered segments or an explanation why a size estimate cannot be provided.
    4. If applicable, over-coverage and/or overlapping sampling frames (for example, when a dual sampling frame is used, for example landline and cellphone frames) must be described in the same manner as under-coverage. If no sampling frame or list was utilized, this must be indicated.
    5. The sampling method, including probability or non-probability, and the sampling design (for example, stratification, clustering) or procedure for selecting respondents.
    6. The rationale for the chosen sampling method, including how it supported the achievement of the research objectives. In the case of non-probability samples, the rationale must take into account the limitations in generalizing the results to the target population and acknowledge that inferential statistics must not be applied when reporting on the collected data.
    7. Screening procedures, including eligibility/screening criteria and any variables used, and targets set for strata sample sizes or for quota controls (for example, gender), including the rationale for each target or quota control (for example, underlying benchmarks or parameters from Census data or another high-quality data source).
    8. If the sample was drawn from a pre-recruited panel or pool, the methods used to recruit the panelists or participants and the procedures for managing the membership, participation, maintenance and refreshment of the panel or pool.
    9. The achieved sample size compared with the projected sample size, both for the overall sample and for any key sub-groups, as well as reasons for not obtaining the projected sample size, strata size targets, or quota controls, if applicable.
    10. The weighting procedures, if applicable, including any underlying parameters or benchmarks used to establish the weights, and their source.
14.2.3. Data collection

Detailed description of the methodology, including:

  1. The data collection method(s), mode(s), and their rationale.
  2. Accessibility provisions for respondents using adaptive technologies, if applicable.
  3. The dates of fieldwork (for each phase/wave, if applicable).
  4. The average interview duration and the range.
  5. The type and amount of incentives, if applicable.
  6. Any relevant stimuli (for example, radio advertisement).
  7. Details of any strategies used to help gain cooperation (for example, advance contact, incentives/honoraria), whether for participation in a panel or for participation in a particular survey.
14.2.4. Quality controls
  1. A description of questionnaire pre-testing and any resulting modifications to improve data quality or respondent experience, if applicable.
  2. The estimating and imputation procedures, and any editing of the data, if applicable.
  3. A brief summary of other quality controls and procedures used. Where applicable, this includes re-contacts to confirm that the interview occurred and/or to verify the respondent’s identity, measures taken to prevent respondents from completing the same survey more than once, and other quality control procedures (for example, logic checks and tests for speeding and patterning). If no such efforts were undertaken, this will be disclosed.
  4. For multi-mode surveys, detailed description of any data quality issues arising from combining data collected via different modes/instruments. The description must also include:
    1. Rationale for decisions about combining or not combining data from different modes in reporting the results.
    2. Description of any steps taken to detect and mitigate mode effects and mode biases in the survey results.
14.2.5. Results

The detailed results in reports must include:

  1. An executive summary of key results and conclusions, linked to the survey objectives and research questions.
  2. Specifications for any indices constructed or statistical modeling in sufficient detail to allow replication.
  3. For probability samples, state the level of precision, including the margin of sampling error and confidence interval for an observed percentage of 50% for yes-no (two option) type questions asked of the total sample and any key subgroups.

    Clearly indicate that the results for subgroups have a larger margin of sampling error than for the overall sample because of their smaller sample sizes. Also, the margin of sampling error is highest for questions where 50% of the respondents gave one answer and the other 50% gave another answer. The margin of sampling error decreases for questions where the observed percentage for a particular response approaches 0% or 100%.

    Results for subgroups must not be reported as significant differences in instances where differences between the groups are within the margin of sampling error for the survey as a whole.

  4. When subsets of the sample (for example, regions, age groups, language groups) have been over- or under-sampled, adjust for the design effect Footnote 3 due to weighting when calculating the margin of sampling error.
  5. For non-probability samples, the report and executive summary must contain a prominently displayed statement in the methodology description regarding why no margin of sampling error is reported, based on the following template: “The results of this survey are not statistically projectable to the target population because the sampling method used does not ensure that the sample represents the target population with a known margin of sampling error. Reported percentages are not generalizable to any group other than the sample studied, and therefore no formal statistical inferences can be drawn between the sample results and the broader target population it may be intended to reflect. [If weighting was done, include a sentence to describe the weighting, for example: “The sample data have been weighted to reflect the demographic composition of (target population).”]
  6. For non-probability surveys, no alternative measures of precision may be cited, and no statements may be made to indicate that the sample has a level of sampling error equivalent to a probability sample of similar size.
  7. For census surveys, the report must contain a statement on why no margin of sampling error is reported, based on the following template: “Because the entire population of [target population] was invited to participate in this study, there is no margin of sampling error to be estimated or reported. The potential impact of non-sampling error due to non-response is discussed in the results section of the report. [If weighting was done, include the following statement: “The data have been weighted to reflect the composition of [the target population (if known) or the sampling frame (for example, client-supplied list)] in relation to its key known characteristics."]
  8. For multi-mode surveys, if the data collected via different modes are combined for analysis and reporting purposes, the report must discuss whether this leads to any data quality issues. This could include, for example, discussion of possible impacts of mode on key survey variables, the impact of any differences in response rate/participation rate by mode, and non-response bias analyses by mode.
  9. The report must provide the final dispositions of cases in sufficient detail to allow secondary analysis of all reported outcome rates. The record of final case dispositions must identify the number of invalid numbers, unresolved cases (U), in-scope non-responding units (IS), and responding units (R).
  10. For all types of surveys (probability sample, non-probability sample, census) the report must state the response rate/participation rate achieved, which must be calculated according to the formula:

    Response Rate/Participation Rate = R/(U + IS + R)

  11. All survey reports must discuss the potential for non-response bias for the survey as a whole and for key survey variables. The report must describe, to the extent possible, the degree to which characteristics of the respondents match or depart from known characteristics (for example, demographics) of the target population or sampling frame, and discuss the possible influence of any resulting non-response bias on the interpretation of survey results.
    1. When authoritative sources of comparison exist, as is the case when conducting a survey of the general population, the analysis of non-response bias will consist of a comparison of at least three variables in the survey sample with the equivalent parameters of the population, normally available from Statistics Canada. No extra surveys or interviews are required.
    2. If no authoritative sources of comparison exist, key variables in the sample could alternatively be compared to variables in the sampling frame. If no such variables exist, only then should the non-response bias discussion be based on comparison of early versus late responders or on observations made during data collection about characteristics of non-responders.
  12. The results that are based on subgroups and the number of cases used in subgroup analysis (that is, unweighted sample sizes for reported subgroup estimates.)
14.2.6. Detailed research instruments and test materials

When appending the research instruments and test materials to the final report, as required by section 14.1 d. vi, the following details must be included:

  1. Study materials, including, when relevant, the recruiting instruments, questionnaires, descriptions, transcriptions or representations of any visual or auditory aids, and other data collection documents, in all languages in which the research was conducted.
  2. A version of the questionnaires displaying the following:
    1. The exact wording and presentation of questions and response options whose results are reported. This includes preceding respondent instructions and any preceding questions that might reasonably be expected to influence responses to the reported results.
    2. Any other instructions (skip, terminate, etc.) needed to understand the logic and flow of the questionnaire.
Date modified: