2006-606 Evaluation Framework for the Policy on Green Procurement, Final Report

2007-09-28

Table of Contents

• Executive Summary
• 1. Introduction
  • 1.1 Objective of the Evaluation Framework
  • 1.2 Scope of the Evaluation Framework
  • 1.3 Methodology Used to Develop the Evaluation Framework
    • 1.3.1 Collaboration with Environment Canada and Natural Resources Canada
• 2. Profile
  • 2.1 Background and Rationale
  • 2.2 Policy Objective
  • 2.3 Expected Results
  • 2.4 Beneficiaries of the Policy
  • 2.5 Policy Implementation
  • 2.6 Stakeholder Roles and Responsibilities
  • 2.7 Progress Report
• 3. The Logic Model for the Policy on Green Procurement
  • 3.1 Interrelation Between Activities, Outputs and Outcomes
• 4. Evaluation Strategy
  • 4.1 Evaluation Issues and Questions
    • 4.1.1 Relevance
    • 4.1.2 Success
    • 4.1.3 Cost-effectiveness
  • 4.2 Evaluation Methods
  • 4.3 Measuring Success
• 5. Recommendation
• Annex A: Definitions
• Annex B: List of Documents Consulted
• Annex C: List of Persons Consulted

Please note that under the Access to Information Act a limited amount of text within these documents may not be disclosed, and will be shown as [ * ].

Executive Summary

This document presents the Evaluation Framework Report for the Policy on Green Procurement.

This evaluation framework will ensure that appropriate evaluation issues/questions and indicators are identified and will support the identification of appropriate data and strategies for gathering data to conduct the evaluation of the Policy on Green Procurement in accordance with the evaluation standards for the Government of Canada and for the Audit and Evaluation Branch of Public Works and Government Services Canada (PWGSC).

The federal government is one of the largest purchasers of goods and services in Canada. PWGSC purchases on average $12 billion annually, which represents over 85% of the total value of government procurement⁽¹⁾. The government is in a position to deliver substantial benefits to the environment by reducing the negative impacts of its procurements. To this end, the Policy on Green Procurement was launched on April 1, 2006. The objective of the policy is to advance the protection of the environment and support sustainable development by integrating environmental performance considerations into the procurement decision-making process, including the planning, acquisition, use and disposal phases. The Government of Canada expects that the application of this policy will benefit the environment by reducing the government's environmental impact, such as: reducing greenhouse gas emissions and air contaminants; improving energy and water efficiency; reducing ozone depleting substances; reducing waste and supporting reuse and recycling; reducing hazardous waste; and reducing toxic and hazardous chemicals and substances.

The evaluation questions to be examined during the evaluation will focus on three issues: relevance, success and cost-effectiveness.

Relevance

• At the time of the evaluation, which is scheduled for 2010-11, does the Policy on Green Procurement continue to be consistent with PWGSC and Government of Canada priorities and to realistically address an actual need?

Success

• At the time of the evaluation, are the Office of Greening Government Operations (OGGO), the policy partners and the Acquisitions Branch making available to government departments appropriate information and tools necessary to support the policy (e.g., guidance, performance measures, training, procurement instruments, as well as directives and standards if applicable)?
• Are program managers, certain designated staff, materiel managers and the procurement community sufficiently knowledgeable about the green procurement policy and its underlying principles and reporting requirements?
• Has the policy resulted in (a) increased environmentally responsible planning, acquisition, use and disposal/closure and (b) consistent monitoring and reporting in each department/Government of Canada?
• Has the policy contributed to protecting the environment and supporting sustainable development by reducing the Government of Canada's environmental footprint and to creating a healthier environment?
• Has the Government of Canada been able to leverage its purchasing power to achieve economies of scale in acquiring environmentally preferable goods and services, and to increase the supply of such goods and services?

Cost-Effectiveness

• Depending on the level of success of the policy, would an alternative approach for green procurement (e.g., a financial incentive, targeting specific environmental goods or sectors) be more cost-effective in ensuring that environmental performance considerations are integrated into departments' procurement decision-making processes?

In measuring success, the evaluation team must consider external factors (e.g., political or economical) that might contribute to or hinder the achievement of success. The general measures of immediate (1-3 years) and intermediate (5-8 years) success are as follows:

Immediate success would involve:

• Selected government employees are sufficiently knowledgeable about the policy, its underlying principles and the reporting requirements.
• The policy partners have developed appropriate and useful tools necessary to support the policy and provided them to all government departments.
• Evidence of a change in behaviour in the way employees integrate environmental performance in their procurement decisions at each phase of procurement (i.e., planning, acquisition, use and disposal).
• Departments are achieving their green procurement targets as established in accordance with the guidelines issued by the policy partners and their mandate, buying patters and environmental risk profile.

Intermediate success would involve:

• Employees are contributing towards protecting the environment by reducing their department's environmental impact.
• Government achieves economies of scale using its leveraged purchasing power.
• Suppliers are making more environmentally preferable goods and services available.

It is recommended that the Assistant Deputy Minister, Corporate Services, Policy and Communications Branch, and the Assistant Deputy Minister, Acquisitions Branch, accept this evaluation framework as the basis for undertaking the evaluation of the Policy on Green Procurement.

1. Introduction

This document presents the Evaluation Framework Report for the Policy on Green Procurement. This section provides a description of the objective and scope of the evaluation framework and of the methodology used to develop it. Throughout this document, words marked with a delta (~) are defined in Annex A: Definitions.

1.1 Objective of the Evaluation Framework

This evaluation framework will ensure that appropriate evaluation issues~/questions~ and indicators~ are identified and will support the identification of appropriate data and strategies for gathering data to conduct the evaluation of the Policy on Green Procurement in accordance with the evaluation standards for the Government of Canada and for the Audit and Evaluation Branch of Public Works and Government Services Canada (PWGSC).

The Policy on Green Procurement requires that the evaluation take place in 2010-11. The evaluation is to assess the impact and net effect of the policy (i.e., relevance, cost-effectiveness and extent to which it will have been successful in achieving its planned outcomes).

1.2 Scope of the Evaluation Framework

As the policy is a Government of Canada policy, the evaluation framework touches all departments. It focuses on the policy partners as well as on the government departments. The policy partners' role is to support all government departments in their efforts to implement the policy and achieve its objective (i.e., to advance the protection of the environment and support sustainable development by integrating environmental performance considerations into the procurement decision-making process).

1.3 Methodology Used to Develop the Evaluation Framework

The following approach was used to develop this evaluation framework:

• Review of key documents related to green procurement, including but not limited to: the Policy on Green Procurement; [ * ]; and the draft Memorandum of Understanding between PWGSC, Environment Canada, Natural Resources Canada and the Treasury Board of Canada Secretariat. See Annex B: List of Documents Consulted for complete list;
• Meetings with key stakeholders to develop a good understanding of the Policy and what it is trying to achieve, as well as the different stakeholders and their respective roles. A list of persons consulted is attached as Annex C: List of Persons Consulted;
• Periodic meetings with representatives from the Office of Greening Government Operations (OGGO) and Government Consulting Services to discuss the development of the logic model and evaluation matrix;
• Meetings with the staff from the Acquisitions Branch of PWGSC, Environment Canada, Natural Resources Canada, Treasury Board of Canada Secretariat and selected departments to validate the logic model and gain consensus;
• Development of the following based on the meetings and document review:
  • a logic model that graphically depicts the Policy on Green Procurement;
  • an evaluation matrix~ containing significant evaluation issues and associated questions; and
  • an evaluation strategy.
• Preparation and submission of a draft Evaluation Framework Report for review and comment by OGGO, Environment Canada and Natural Resources Canada;
• Submission of a revised draft for acceptance by the Director General, Office of Greening Government Operations, and the Assistant Deputy Minister, Acquisitions Branch; and
• Issuance of a final Evaluation Framework Report.

1.3.1 Collaboration with Environment Canada and Natural Resources Canada

Based on an agreement with the evaluation directorates at Environment Canada and Natural Resources Canada, PWGSC's Evaluation Directorate was responsible to develop this evaluation framework for the policy and to consult with both departments at key times during the process (e.g., to validate the logic model and to review the draft evaluation framework). It is anticipated that when the evaluation study will be carried out in 2010-11, the evaluation directorates of Environment Canada and Natural Resources Canada will participate in an advisory role. However, at this time, it is difficult to speculate as to their exact involvement.

2. Profile

This section provides a description of the Policy on Green Procurement in terms of its objective, expected results, beneficiaries, as well as its main stakeholders and their roles and responsibilities.

2.1 Background and Rationale

The federal government is one of the largest purchasers of goods and services in Canada. PWGSC purchases on average $12 billion annually, which represents over 85% of the total value of government procurement ⁽²⁾. In this context and considering its concern with sustainable development, the government is in a position to deliver substantial benefits to the environment by reducing the negative impacts of its procurements.

In the 2004 Speech from the Throne, the Government committed to develop and implement a government-wide green procurement policy. In April 2005, PWGSC launched OGGO to help fulfill the department's mandate to facilitate the implementation of a government-wide approach to the greening of government activities. OGGO also supports and monitors the implementation of the Policy on Green Procurement, which came into effect on April 1, 2006, in cooperation with Environment Canada and Natural Resources Canada. In August 2007, subsequent to the acceptance of the draft Evaluation Framework Report by the Director General, OGGO, and the Assistant Deputy Minister, Acquisitions Branch, OGGO became part of the Corporate Services, Policy and Communications Branch of PWGSC.

2.2 Policy Objective

The objective of the Policy on Green Procurement is to advance the protection of the environment and support sustainable development by integrating environmental performance considerations into the procurement decision-making process, including the planning~, acquisition~, use~ and disposal~ phases. Reducing greenhouse gas emissions and air contaminants, using renewable resources and reducing hazardous waste are just a few examples of environmental performance considerations. It is important to keep in mind that green procurement (i.e., the purchase of environmentally preferable goods and services~) is set within the context of achieving value for money~. This means that—in addition to the cost, performance, availability and quality of a good or service—environmental performance is to be considered when making a procurement decision.

The main intent of the Policy on Green Procurement is to change the behaviour of public servants so their decisions take into account environmental performance considerations (i.e., public servants are expected to consider environmental performance factors in their day-to-day procurement-related activities). Achieving this change will increase green procurement. To this end, departments must reshape their management control frameworks, which guide employees in conducting their work, to reflect the requirements of the policy.

2.3 Expected Results

The Government of Canada expects that the application of this policy will benefit the environment by reducing the government's environmental impact, such as: reducing greenhouse gas emissions and air contaminants; improving energy and water efficiency; reducing ozone depleting substances; reducing waste and supporting reuse and recycling; reducing hazardous waste; and reducing toxic and hazardous chemicals and substances. This will result in a healthier environment for government employees and, more generally, for all Canadians.

In addition, the policy will allow the federal government to lever its purchasing power to achieve economies of scale in the procurement of environmentally preferable goods and services and to increase the supply of such goods and services available to the government.

2.4 Beneficiaries of the Policy

The key beneficiaries of this policy are federal employees and, ultimately, Canadians. By contributing to the protection of the environment, the policy will improve the environment in which Canadians live and, in particular, provide a healthier working environment for government employees. Also, greener markets and industries will also benefit from this policy, which gives increased focus to environmentally preferable goods and services. This will ultimately result in an increased supply of such goods and services throughout Canada and in increased innovation and accelerated commercialization by suppliers of such goods and services.

2.5 Policy Implementation

The implementation focuses on departmental management control frameworks. As previously mentioned, environmental performance considerations have to be reflected in these frameworks to cause a change in behaviour amongst employees in the way they carry out their duties. The frameworks cover multiple aspects, including establishing processes for staff and setting green procurement targets for the department, and applies to all phases of procurement (planning, acquisition, use and disposal).

A tool to facilitate the policy implementation is the commodity management process~. This process can be defined as a way to plan, organize and control all activities through the entire life cycle of a distinct group of goods and services. It has several advantages, one of them being to provide the government with the most advantageous total cost of ownership~ and disposal of its goods and services. The commodity management process provides the mandatory instruments that departments are to use to purchase selected standardized goods and services.

Even though the commodity management process is integral to the policy, the purchase of many goods and services falls outside the scope of this process. The policy also applies to such procurements (which are usually referred to as non-standard, customized or “program-unique” procurements) for which environmental performance considerations have to be incorporated in the decision-making process.

2.6 Stakeholder Roles and Responsibilities

The roles, responsibilities and accountabilities of the key stakeholders involved in the delivery of the project are set out in the policy itself and in the draft Memorandum of Understanding between PWGSC, Environment Canada, Natural Resources Canada and the Treasury Board of Canada Secretariat. They are briefly reviewed below:

Departments and agencies have primary responsibility to implement the policy and achieve the policy objective. Deputy heads are accountable to ensure their management control frameworks incorporate environmental performance considerations and to establish and implement green procurement targets. As such, they must ensure that direction and processes to support green procurement are in place, and that necessary training is provided to staff. In turn, departmental staff will be able to plan its procurements and then acquire, use, maintain and dispose of goods and services in an environmentally-friendly manner, as well as comply to prescribed annual reporting requirements on green procurement performance. Departments may also be asked to play a key role in periodically providing data on green procurement to assist in evaluating the effectiveness of the policy.

The policy partners, comprising OGGO, Environment Canada, Natural Resources Canada and the Treasury Board of Canada Secretariat, are responsible to support all government departments in achieving the policy objective.

• OGGO is responsible for leading green procurement efforts by developing directives~, standards~, guidance and tools as necessary to help departments in implementing the policy. It plays a critical role in developing the green procurement training, and some performance measures that will be used to report on the policy.
• Environment Canada and Natural Resources Canada are responsible for providing expertise and advice on the policy, environmental aspects and impacts, as well as environmental standards and sustainability considerations related to the use and development of natural resources. Both departments support OGGO and the Acquisitions Branch by participating in the commodity management process based on their respective areas of expertise. Finally, they assist PWGSC in evaluating the effectiveness of the policy.
• The Treasury Board of Canada Secretariat provides guidelines to departments on how to report on green procurement in their annual Reports on Plans and Priorities (RPPs) and Departmental Performance Reports (DPRs). It also approves directives and standards developed by the other policy partners, and issues guidelines as required. It is also expected that the Treasury Board of Canada Secretariat will require environmental performance considerations be addressed in approval documents, including TB submissions.

Along with the policy partners, the Acquisitions Branch (PWGSC) is responsible for supporting the achievement of the policy objective. To this end, Branch staff must integrate environmental performance considerations into the policies, procedures and business tools (including procurement instruments such as standing offers~) that support departments in the procurement of goods and services. As the contract managers of over 85% of the government's procurement of goods and services, it is expected that the Branch will work with OGGO, the Treasury Board of Canada Secretariat and selected departments to develop appropriate measures and mechanisms for collecting and reporting on the extent to which clients departments are integrating environmental performance considerations in their procurement decision-making.

2.7 Progress Report

Since the policy came into force on April 1, 2006, the implementation has already started. Activities underway at the time of writing this report are briefly summarized below:

• OGGO is developing a performance measurement framework in consultation with Environment Canada and Natural Resources Canada. The framework will identify some of the elements that departments and the policy partners are expected to track and report on to measure what the policy is achieving. Some of these measures may also be useful in evaluating the effectiveness of the policy.
• The Acquisitions Branch is continuing its work on commodity management. Ten (10) commodity categories (e.g., IT hardware, fuels and lubricants) have been identified for which environmental performance considerations are being integrated in the procurement instruments. These categories make up approximately $7.1 billion of the total government spend of $12 billion and include hundreds of subcategories. Thirty-four (34) subcategories have been identified for the first round, which make up approximately $4.4 billion (of the $7.1 billion). The Acquisitions Branch is presently working on the second round targeting approximately 10 to 15 additional subcategories, within the 10 main commodity categories, as well as others. Those 10 to 15 subcategories are expected to account for an extra $2 billion.

3. The Logic Model for the Policy on Green Procurement

The logic model~ (Figure 1) illustrates how the activities associated with the policy contribute to the achievement of the stated objective and desired outcomes.

3.1 Interrelation Between Activities, Outputs and Outcomes

As discussed in the roles and responsibilities section2.6 Stakeholder Roles and Responsibilities, the policy partners are responsible to support the implementation of the policy. For this purpose, their activities revolve around developing tools necessary to support the Policy to help all government departments implement the policy, including directives and/or guidelines as appropriate to report on green procurement (i.e., TBS directives to report in Report on Plans and Priorities and Departmental Performance Report).

As for the Acquisitions Branch, their activities focus on integrating environmental performance considerations into the procurement policies, procedures and business tools (including procurement instruments, such as standing offers) used by departments to carry out their procurements.

Once the policy partners have developed their information, tools and reporting requirements, these are available to help departments achieve the policy objective. Individual departments may develop supplementary information and tools as may be required to support the incorporation of environmental performance considerations in their management control frameworks in support of procurement. Nevertheless, the onus is on departments to implement the policy by making sure their management control frameworks take into account environmental considerations. This includes, without being limited to, arranging training on green procurement for staff, setting green procurement targets and establishing processes to support the policy.

The activities and outputs described above should lead to an increase in knowledge about the policy and its underlying principles by departmental staff (especially procurement and materiel management personnel, and managers) and to consistent reporting on green procurement performance across departments. Departments can then put this knowledge into action, and be in a better position to make decisions based on environmental considerations. It is expected that this will result in a change in behaviour characterized by increased integration of environmental performance considerations in procurement decisions with respect to:

• procurement planning, including requirement definition;
• acquisition (including requisitions, requests for quotation (RFQs), statements of work (SOWs), contracts and call-ups against standing offers);
• use of goods and services; and
• disposal of goods and closure activities of acquired services.

These outcomes are expected to produce several environmental, economic and industrial outcomes both at the intermediate and ultimate level:

• Increase the protection of the environment by reducing the government's environmental impact (e.g., reduced greenhouse gas emissions);
• Increase the leverage of the government's purchasing power to achieve economies of scale in the acquisition of environmentally preferable goods and services;
• Increase the supply of environmentally preferable goods and services for the government and ultimately throughout Canada;
• Strengthen green markets, industries and suppliers; and
• Foster a healthier environment for employees and, more generally, for Canadians by contributing to the protection of the environment and to sustainable development

Figure 1: Logic Model

Detailed image of Figure 1

4. Evaluation Strategy

4.1 Evaluation Issues and Questions

Figure 2 presents the evaluation matrix for the Policy on Green Procurement. It is organized around the evaluation issues of relevance~, success~ and cost-effectiveness~. The essential evaluation questions that are to be addressed in the evaluation are presented along with evaluation indicators for each. These indicators were derived from consultations with OGGO, Environment Canada, Natural Resources Canada, Treasury Board of Canada Secretariat, Government Consulting Services and the Acquisitions Branch. Associated data sources and methods intended to guide the collection of data are also presented.

4.1.1 Relevance

• At the time of the evaluation, which is scheduled for 2010-11, does the Policy on Green Procurement continue to be consistent with PWGSC and Government of Canada priorities and to realistically address an actual need?

4.1.2 Success

• At the time of the evaluation, are OGGO, the policy partners and the Acquisitions Branch making available to government departments appropriate information and tools necessary to support the policy (e.g., guidance, performance measures~, training, procurement instruments, as well as directives and standards if applicable)?
• Are program managers, certain designated staff, materiel managers and the procurement community sufficiently knowledgeable about the green procurement policy and its underlying principles and reporting requirements?
• Has the policy resulted in (a) increased environmentally responsible planning, acquisition, use and disposal/closure and (b) consistent monitoring and reporting in each department/Government of Canada?
• Has the policy contributed to protecting the environment and supporting sustainable development by reducing the Government of Canada's environmental footprint and to creating a healthier environment?
• Has the Government of Canada been able to leverage its purchasing power to achieve economies of scale in acquiring environmentally preferable goods and services, and to increase the supply of such goods and services?

4.1.3 Cost-effectiveness

• Depending on the level of success of the policy, would an alternative approach for green procurement (e.g., a financial incentive, targeting specific environmental goods or sectors) be more cost-effective in ensuring that environmental performance considerations are integrated into departments' procurement decision-making processes?

Figure 2: Evaluation Matrix

Relevance

Evaluation Issue/Questions

1. At the time of the evaluation, which is scheduled for 2010-11, does the Policy on Green Procurement continue to be consistent with PWGSC and Government of Canada priorities and to realistically address an actual need?

Indicators

• Greening government operations, particularly the purchase of environmentally preferable goods and services in accordance with key GC policy/priority/reporting documents, including the use of mandatory standing offers
• Extent to which environmentally preferable goods and services are available and affordable, especially from suppliers of managed commodities
• Extent to which Deputy Heads' management control frameworks in support of procurement incorporate environment performance considerations with respect to planning, acquisition, use and disposal/closure
• Extent to which departmental staff comply with their respective management control frameworks
• TBS monitoring or tracking of departmental reporting on the PGP in RPP and DPR

Methodology/Data Source

Document review

• Speech from the Throne
• Federal Budget
• Departmental reports, e.g., Departmental Performance Report, Report on Plan and Priorities, Sustainable Development Strategy, procurement statistics
• Managed commodity templates
• Other relevant documents

Literature review

Green procurement in other jurisdictions

Key stakeholder interviews

• OGGO
• Acquisitions Branch
• Treasury Board of Canada Secretariat, Natural Resources Canada, Environment Canada
• Privy Council Office
• Senior management
• Representatives from selected government departments
• Representatives from suppliers

Survey

• Suppliers with respect to availability and affordability of environmentally preferable goods and services.

Success

Evaluation Issue/Questions

2. At the time of the evaluation, are OGGO, the policy partners and the Acquisitions Branch making available to government departments appropriate information and tools necessary to support the policy (e.g., guidance, performance measures, training, procurement instruments, as well as directives and standards if applicable)?

Indicators

• Extent to which government departments adopt or uptake the different information and tools provided by the policy partners and the Acquisitions Branch, including the latter's mandatory managed commodity procurement instruments, as opposed to developing their own information and tools
• Extent to which these information and tools meet the needs of the government departments
• Level of satisfaction with these information and tools
• Responsiveness of policy partners to feedback from departments on information and tools
• Number of hits on policy partners and the Acquisitions Branch Web sites and downloads.
• Treasury Board of Canada Secretariat guidelines to departments for policy reporting in departmental performance reports (DPRs) and reports on plans and priorities (RPPs) which can include, for example:
  • Number and percentage of materiel managers and procurement community receiving appropriate training
  • Adoption or uptake of policy partners and the Acquisitions Branch information and tools
  • Number and percentage of acquisitions incorporating environmental performance considerations (including decisions not to acquire because of environmental performance concerns)
  • Number and percentage of acquisitions of environmentally preferable goods and services
  • Dollar value and percentage of such procurements
  • Evidence-based cases of departments using and/or disposing/terminating goods and services in an environmentally-friendly manner
  • N.B. The scope of these indicators may depend on the performance measurement strategy that is ultimately supported for regular monitoring and reporting
  • Existence of interdepartmental tools (e.g., networks, committees, newsletters) in which policy success stories and best practices can be shared

Methodology/Data Source

Document review

• Departmental reports, e.g., Departmental Performance Report, Report on Plan and Priorities, Sustainable Development Strategy
• Course evaluations
• Directives and standards
• Other relevant documents
  

Data and statistics review

• Data on the extent to which client departments are integrating environmental performance considerations in their procurement decision-making (OGGO, Acquisitions Branch, the Treasury Board of Canada Secretariat and selected departments will work together to develop appropriate measures and mechanisms for collecting and reporting on these data)

Key stakeholder interviews

• OGGO
• Acquisitions Branch
• Treasury Board of Canada Secretariat, Natural Resources Canada, Environment Canada
• Representatives from selected government departments

Survey

• Managers and functional heads of procurement and materiel in government departments

Expert Panel

• Comparative analysis to private sector best practices and to those of municipal, provincial and other federal jurisdictions

Evaluation Issue/Questions

3. Are program managers, certain designated staff, materiel managers and the procurement community sufficiently knowledgeable about the green procurement policy and its underlying principles and reporting requirements?

Indicators

• Number/percentage of designated program managers and staff, materiel managers and procurement community who have received appropriate training for their positions and responsibilities
• Results from course evaluations
• Extent to which departmental management control framework policies, procedures and practices incorporating environment performance considerations have been disseminated to designated program managers and staff, materiel managers and the procurement community
• Number/percentage of managers and functional heads of procurement and materiel whose performance evaluations reflect support for the policy objectives
• Departmental green procurement targets have been set and disseminated to designated program managers and staff, materiel managers and the procurement community
• Mechanisms and capacity in place to monitor/track achievement of targets and to report on green procurement (performance reports, reports on plans and priorities and sustainable development strategies) in accordance with policy partners' reporting requirements

Methodology/Data Source

Document review

• Departmental Report on Plans and Priorities, Departmental Performance Report and Sustainable Development Strategy
• Departmental procurement/ contracting guidelines, policy, procedures (referring to the policy)
• Management control framework documents
• Reporting directives
• Course evaluations

Data and statistics review

• Data on the extent to which client departments are integrating environmental performance considerations in their procurement decision-making (OGGO, Acquisitions Branch, the Treasury Board of Canada Secretariat and selected departments will work together to develop appropriate measures and mechanisms for collecting and reporting on these data)
• Departmental policy training statistics
• Course evaluations

Key stakeholder interviews

• OGGO
• Treasury Board of Canada Secretariat, Natural Resources Canada, Environment Canada
• Sustainable development officials and procurement and materiel managers from selected government departments and from Acquisitions Branch

Survey

Materiel managers, procurement community or other relevant stakeholders with respect to changing procurement decision-making behaviour (e.g., of last 5 procurement decisions, number that incorporated EPCs)

Evaluation Issue/Questions

4. Has the policy resulted in (a) increased environmentally responsible planning, acquisition, use and disposal/closure and (b) consistent monitoring and reporting in each department/Government of Canada?

Indicators

Procurement planning and requirements definition integrating environmental performance considerations

• Representative data to support the estimated total spend on selected goods and services (for both managed commodities and program-unique requirements) by commodity and type
• Representative data to support the estimated total spend on selected goods and services for which environmental performance is considered in the planning and requirement definition (i.e., environmental performance incorporated in selection criteria), in use, and in closure and disposal.
• Evidence-based cases of departments deciding not to acquire because of environmental performance concerns

Acquisition of goods and services integrating environmental performance considerations

• Spend on environmentally preferable goods and services procurements (e.g., contracts, call-ups), and percent of total procurements
• Extent to which departmental green procurement targets for acquisition have been achieved

Use of goods/services in an environmentally-friendly manner

• Evidence of environmentally preferable use, re-use and maintenance of goods
• Change in behaviour of public servants in how they use goods, e.g., using less, duplex printing

Closure of acquired services and re-cycling and disposal of goods in an environmentally-friendly manner

• Evidence of environmentally friendly re-cycling and disposal and termination of services
• Change in behaviour of public servants in how they re-cycle and dispose of goods

Monitoring and reporting on green procurement performance

• Reporting is consistent across departments and in accordance with reporting requirements as specified by the policy partners

Methodology/Data Source

Document review

• Departmental Report on Plans and Priorities, Departmental Performance Report and Sustainable Development Strategy
• Departmental procurement/ contracting guidelines, policy, procedures (referring to the policy)
• Management control framework documents
• Performance measurement framework (PMF)
• Directives and standards

Data and statistics review

• Data on the extent to which client departments are integrating environmental performance considerations in their procurement decision-making (OGGO, Acquisitions Branch, the Treasury Board of Canada Secretariat and selected departments will work together to develop appropriate measures and mechanisms for collecting and reporting on these data)

Key stakeholder interviews

• OGGO
• Treasury Board of Canada Secretariat, Natural Resources Canada, Environment Canada
• Procurement and materiel managers from selected government departments and from the Acquisitions Branch

Survey

• Materiel managers, procurement community or other relevant stakeholders with respect to changing procurement decision-making behaviour (e.g., of last 5 procurement decisions, number that incorporated environmental performance considerations)

Evaluation Issue/Questions

5. Has the policy contributed to protecting the environment and supporting sustainable development by reducing Government of Canada's environmental footprint and to creating a healthier environment?

Indicators

Extent to which the following are attributable to the policy for selected commodities and types of procurements:

• Reduction in greenhouse gas emissions and air contaminants
• Increase in energy and water efficiency
• Reduction in ozone depleting substances
• Reduction in waste and increased support for reuse and recycling
• Reduction of hazardous waste
• Reduction in toxic and hazardous chemicals and substances

Methodology/Data Source

Document review

• Departmental Report on Plans and Priorities, Departmental Performance Report and Sustainable Development Strategy

Model development and application

• Use of Environment Canada/Natural Resources Canada models for estimating change in government's environmental footprint

Data and statistics review

• Data on the extent to which client departments are integrating environmental performance considerations in their procurement decision-making (OGGO, Acquisitions Branch, the Treasury Board of Canada Secretariat and selected departments will work together to develop appropriate measures and mechanisms for collecting and reporting on these data)

Evaluation Issue/Questions

6. Has the Government of Canada been able to leverage its purchasing power to achieve economies of scale in acquiring environmentally preferable goods and services, and to increase the supply of such goods and services?

Indicators

• Evidence of reduced cost to the Government of Canada for environmentally preferable goods and services
• Evidence that suppliers have increased their supply or availability of affordable environmentally preferable goods and services to the Government of Canada and to all consumers
• Evidence of increased innovation by suppliers in the development and commercialization of environmentally preferable goods and services
• Evidence that the policy has supported the development of emerging environmental technologies

Methodology/Data Source

Document review

• Departmental Report on Plans and Priorities, Departmental Performance Report and Sustainable Development Strategy, procurement statistics
• Other relevant documents

Literature review

• Reports describing increased commitment by suppliers to supply environmentally preferred goods and services
• Green procurement in other jurisdictions

Data and statistics review

• Data on the extent to which client departments are integrating environmental performance considerations in their procurement decision-making (OGGO, Acquisitions Branch, the Treasury Board of Canada Secretariat and selected departments will work together to develop appropriate measures and mechanisms for collecting and reporting on these data)
• Industry statistics on availability of “green” goods and services

Key stakeholder interviews

• Procurement and materiel managers from selected government departments and from the Acquisitions Branch
• Representatives from selected government departments
• Representatives from suppliers

Cost-Effectiveness/Alternatives to the Policy on Green Procurement

Evaluation Issue/Questions

7. Depending on the level of success of the policy, would an alternative approach for green procurement (e.g., a financial incentive, targeting specific environmental goods or sectors) be more cost-effective in ensuring that environmental performance considerations are integrated into departments' procurement decision-making processes?

Indicators

• Evidence of commitment of managers and functional heads of procurement and materiel to integrating environmental performance considerations would suggest that the current approach is appropriate
• Extent of success (e.g., training, commitment, use of knowledge in procurement process) of the policy will indicate if an alternative approach should be considered
• Evidence that increased investment (e.g., paying a premium for specific goods) would produce more cost-effective green procurement results

Methodology/Data Source

Key stakeholder interviews

• OGGO
• Treasury Board of Canada Secretariat, Natural Resources Canada, Environment Canada
• Procurement and materiel managers from selected government departments and from the Acquisitions Branch
  

Literature review

• Green procurement in other jurisdictions

4.2 Evaluation Methods

To assess the evaluation issues and questions described in the previous section, we plan on using several methods. These include:

• Literature review: A review of green procurement programs and results in place in other jurisdictions (Canadian provincial and municipal governments, as well foreign organizations and governments) will be examined to understand the different approaches used. In addition, reports and articles dealing with how industry is “greening” the goods and services it supplies.
• Document review: Documents to be reviewed will include strategic documents such as departmental Reports on Plans and Priorities, Departmental Performance Reports, Sustainable Development Strategies, Speeches from the Throne and the Federal Budget Speeches. Many other documents may be reviewed, all of them with varying purposes and levels of specificity. Some examples include the performance measurement framework for the policy; documents related to departmental management control frameworks; all types of directives, standards and guidelines issued by the policy partners; managed commodity templates; procurement-related documents such as requisitions and contracts, and course evaluations from training providers.
• Data and statistics review: Data and statistics to be reviewed will focus on the extent to which client departments are integrating environmental performance considerations in their procurement decision-making, e.g., the percentage of procurement decisions in which environmental performance considerations were incorporated. OGGO, Acquisitions Branch, the Treasury Board of Canada Secretariat and selected departments will work together to develop appropriate measures and mechanisms for collecting and reporting on these data. Data collected during the first five years of the policy will be useful to the evaluation team in addressing the effectiveness of the policy. At a minimum, it would be necessary to examine procurement data for at least two periods, namely, in year 2 or 3 of the policy, to establish a baseline, and in year 5. As well, industry statistics outlining the changes in the availability of “green” goods and services would be reviewed.
• Interviews with key stakeholders: Stakeholders to be interviewed will include members from OGGO, the Acquisitions Branch, the Treasury Board of Canada Secretariat, Natural Resources Canada, Environment Canada and the Privy Council Office, as well as representatives from selected government departments and suppliers. Other stakeholders may include officials in key management, procurement and oversight positions; managers and functional heads of procurement and materiel; sustainable development officials; and procurement and materiel management staff from selected government departments and the Acquisitions Branch. Stakeholders will be asked questions based on their roles and responsibilities and their familiarity with the evaluation question being assessed.
• Surveys: Conducting surveys can be an economical method for obtaining useful feedback from specific groups. Suppliers may be surveyed on the availability of environmentally preferable goods and services. Key stakeholders identified above may also be surveyed on individual aspects of the policy based on their area of expertise and on the implementation and application of the policy (e.g., how the policy has changed their behaviour with respect to procurement decision-making).
• Model development and application: Based on their technical expertise, the two policy partners Environment Canada and Natural Resources Canada will contribute their expertise to the development of models for converting the procurement of environmentally preferable goods and services into estimates for measuring the reduction in the government's environmental impact.
• Expert panel: An expert panel may be brought together to conduct a comparative analysis of the policy implementation and principles with private sector best practices.

4.3 Measuring Success

In measuring success, the evaluation team must consider external factors (e.g., political or economical) that might contribute to or hinder the achievement of success. The general measures of immediate (1-3 years) and intermediate (5-8 years) success are as follows:

Immediate success would involve:

• Selected government employees are sufficiently knowledgeable about the policy, its underlying principles and the reporting requirements.
• The policy partners have developed appropriate and useful tools necessary to support the policy and provided them to all government departments.
• Evidence of a change in behaviour in the way employees integrate environmental performance in their procurement decisions at each phase of procurement (i.e., planning, acquisition, use and disposal).
• Departments are achieving their green procurement targets as established in accordance with the guidelines issued by the policy partners and their mandate, buying patters and environmental risk profile.

Intermediate success would involve:

• Employees are contributing towards protecting the environment by reducing their department's environmental impact.
• Government achieves economies of scale using its leveraged purchasing power.
• Suppliers are making more environmentally preferable goods and services available.

5. Recommendation

It is recommended that the Assistant Deputy Minister, Corporate Services, Policy and Communications Branch, and the Assistant Deputy Minister, Acquisitions Branch, accept this evaluation framework as the basis for undertaking the evaluation of the Policy on Green Procurement.

Annex A: Definitions

The following definitions are provided for clarification and apply solely to this document.

Commodity management process: Process used to plan, organize and control all activities through the entire life cycle of a distinct group of goods and services.

Directives: Contain directions and instructions that oblige departments to undertake or avoid specific actions.

Environmentally preferable goods and services: The Policy on Green Procurement defines them as those that have a lesser or reduced impact on the environment over the life cycle of the good or service when compared with competing products and services serving the same purpose.

Evaluation issue: Broad area which needs to be explored during an evaluation. Usually, there are 3 kinds of issues:

• Relevance: Does the policy continue to be relevant to organizational or governmental priorities and respond to an actual need?
• Success: Does the policy meet its objectives, within budget and without causing unwanted outcomes?
• Cost-effectiveness: Is there a more appropriate and efficient way to achieve outcomes (i.e., change how the policy is designed or delivered)?

Evaluation matrix: Table outlining the evaluation issues and questions to be examined during the evaluation, along with the indicators that will help determine if expected results are being achieved, and data sources to collect the required information.

Evaluation question: Specific questions that need to be answered in order to conclude on each evaluation issue.

Life-cycle analysis: This type of analysis is used to examine the environmental impacts and financial costs of a good or service, from its design through to production and final disposal. Much of the government's focus on life cycle analysis considers the impacts and costs associated with its procurement process, namely the four phases of procurement: planning, acquisition, use/maintenance and disposal of goods/termination of services.

Life-cycle costing: Method used to evaluate the costs of goods and services, based on each phase of their life cycle. From the government's perspective, these costs include those associated with the four phases of procurement (i.e., planning costs, acquisition costs, use/maintenance costs and disposal costs).

Logic model: An illustration of the relationships between activities, outputs and the outcomes of a given policy or program.

• Activity: Work conducted by an organization to produce a specific output (e.g. construction)
• Output: Direct product or service resulting from the activities of an organization or the activities associated with a policy or program. Outputs are usually within the control of the organization itself.
• Outcome: An external consequence attributed, in part, to an organization, policy or program. Outcomes are not within the control of a single organization, policy or program. Instead, the organization, policy or program can influence the outcomes. There are three types of outcomes: immediate, intermediate and ultimate.

Performance indicator: see performance measure

Performance measure: A means of measuring an output or outcome in order to assess the performance of an organization, program or policy.

Policy: Government legislation, regulation, official guidelines or operating principles that influence behaviour towards an outcome.

Procurement process: The procurement process is divided into 4 phases: planning, acquisition, use and disposal.

• Planning phase: At this stage, departments plan their purchase, i.e., they identify their need for a good or service and they define their requirements, including whether to include optional or mandatory selection criteria based on environmental performance. This stage offers the greatest opportunity to consider environmental performance. It is at this stage that departments can decide whether the good or service is required at all and plan whether and how environmental performance is to be integrated into the acquisitions, use and disposal/termination stages.
• Acquisition phase: Once a department identifies a specific need, the acquisition phase begins. This phase draws upon work completed during the planning phase. During this phase, solicitation documents are prepared, bids are evaluated and the contract is awarded and then managed.
• Use/Maintenance phase: Departments properly use and maintain goods to ensure environmental benefits and extend the life of the goods.
• Disposal phase: Departments dispose of goods or terminate services in the most environmentally responsible manner. They must consider some alternatives (e.g., return-to-vendor, reuse, recycle).

Program: Activities designed and managed to meet a specific public need.

Standards: Mandatory operational or technical measures, procedures or practices that are approved for government-wide use.

Standing Offer: Offer from a potential supplier to provide goods or services at pre-arranged prices, under set terms and conditions, when and if required.

Total cost of ownership: All costs associated with a good until its disposal.

Value for money: In the context of procurements, value for money means acquiring a good or service in the most economical manner possible. When deciding to purchase a good or service, departments must consider cost, performance, availability, quality and environmental attributes. Achieving value for money means achieving the optimal balance between the overall benefits and the costs.

Annex B: List of Documents Consulted

1. A Performance Management Framework for Greening Government Operations, TBS, Draft, October 2005
2. Implementation Status of Audit Management Action Plans (in response to CESD Chapter 6 recommendations), OGGO, November 23, 2006
3. Greening Government Operations Interdepartmental Governance Structure, OGGO, October 2006
4. Environmental Awareness Tool Kit, OGGO, June 12, 2006
5. Guidance for Organizations Developing Sustainable Development Strategies (2007-2009), OGGO Web Site
6. Guideline for the Integration of Environmental Performance Considerations in Federal Government Procurement, OGGO (hard copy only)
7. Policy on Green Procurement, OGGO, April 2006
8. Green Procurement Training Course, OGGO (hard copy only)
9. Commodity Management and Green Procurement Workshop, PWGSC, February 17, 2006
10. Greening Government Workshop, Principles of Life Cycle Analysis: In the Context of Total Cost of Ownership, A.T. Kearney, February 17, 2006
11. Commodity Management and Green Procurement, Environmental Issues, Environment Canada, February 17, 2006
12. Promoting Sustainability of Natural Resources, NRCan, February 17, 2006
13. Report on Commodity Based and Green Procurement Workshops, OGGO, March 31, 2006 (Report from presentations 9 through 12)
14. [ * ]
15. Memorandum of Understanding between PWGSC, EC, NRCan and TBS—Original version and update
16. Greening Government Operations, Green Procurement, "Getting Started", September 25, 2006
17. Greening Government Operations, Green Procurement Tools, September 25, 2006
18. Report on the Interdepartmental Workshop: Implementing the Policy on Green Procurement, November 2006 (Report from presentations 16 and 17)
19. Cost Analysis and Risk Assessment on the Implementation of the Policy on Green Procurement, OGGO
20. Commodity templates, OGGO
21. A Quick Guide to the Completion of the Green Procurement Templates, OGGO
22. An Exploration of Green Procurement Performance Measures for the Government of Canada, Environment Canada, June 2007
23. Results-based Management and Accountability Framework for the Policy on Green Procurement: Performance Measurement Options Paper, Government Consulting Services  (prepared for OGGO), draft version 1 (June 16, 2007)

Annex C: List of Persons Consulted

OGGO

• Elizabeth Hopkins - Director, OGGO
• Jean Carruthers - Manager, Policy Development
• Dany Carrière - Senior Policy Analyst, Policy Development
• Caroline MacIntosh - Policy Advisor, Policy Development

Acquisitions Branch

• Normand Masse - Director, Commodity Management and Procurement Engineering Directorate
• Marc Beaudoin - Director, Project Management Office
• Claudette Langdon - Manager, Strategic Relations Division
• Daniel Duguay - Acting Director, Strategic and Horizontal Policy Directorate, Small and Medium Enterprises Sector

Natural Resources Canada

• Allison Christie - Assistant Director, Sustainable Development Policy
• Dianne Payne - Senior Policy/Planning Officer, Sustainable Development Policy
• Gerry Godsoe - Director, Evaluation
• Marc Bennett - Manager, Evaluation
• Zeljka Spasojevic - Senior Officer, Evaluation

Environment Canada

• Berny Latreille - Director, Environmental Affairs
• Loretta Legault - Manager, Stewardship and Compliance
• Shelley Borys - Director, Evaluation Division
• Shelley Tice - Senior Evaluator, Evaluation Division
• Phillip Stirling - Material, Policy and Systems

Treasury Board of Canada Secretariat

• Glenn Richardson - Senior Policy Analyst, Procurement and Project Management Policy Directorate
• Zella Osberg - Senior Analyst, Climate Change Review and Sustainable Development Sector

Government Consulting Services

• Doug McConnachie - Director, Financial, Risk Management and Decision Support Services
• Alti Rodal - Principal ConsultantProgram Evaluation and Performance Measurement Services
• Peter Au - Senior Analyst, Governance, Sustainable Development and Environmental Management Services

Other Government Departments

• Gaétan Viel - Manager, Assets and System Management, PWGSC
• Stéphanie Drouin - Director, Special Projects Division, Health Canada
• Denise Haney - Department of National Defense (DND)
• Roland Dimitriu - Policy Analyst, Assets and Procurement Branch, Royal Canadian Mounted Police (RCMP)
• Bill Lumsdon - Special Advisor, Finance and Administration Directorate, Department of Fisheries and Oceans (DFO)
• Seamus Clifford - Assets and Environmental Policy, Systems and Training, Indian and Northern Affairs (INAC)

Footnotes

⁽¹⁾ 2007-2008 PWGSC Report on Plans and Priorities

⁽²⁾ 2007-2008 PWGSC Report on Plans and Priorities