Public Works and Government Services Canada
www.pwgsc.gc.ca
Common menu bar links
Institutional links
-
Office of Audit and Evaluation
- 2011/2012 Reports
- 2010/2011 Reports
- 2009/2010 Reports
- 2008/2009 Reports
- 2007/2008 Reports
- 2006/2007 Reports
OAE Resources
-
Proactive Disclosure
2006-602 Evaluation Framework for the Interim Cost-Share Agreement (ICSA) for the Sydney Tar Ponds and Coke Ovens Sites Remediation Project (STPCORP), Final Report
December 8, 2006
Table of Contents
- Executive Summary
- 1 Introduction
- 2 Component Profile ICSA
- 3 Logic Model
- 4 Evaluation Strategy
- 5 Recommendation
- Annex A - List of Interviewees
- Annex B - Evaluation Matrix
Please note that under the Access to Information Act a limited amount of text within these documents may not be disclosed, and will be shown as [ * ].
Executive Summary
This document presents the preliminary draft of the Evaluation Framework for the Interim Cost-Share Agreement (ICSA) for the Sydney Tar Ponds and Coke Ovens Sites Remediation Project (STPCORP).
The evaluation framework will ensure that appropriate evaluation issues are identified and addressed and that the appropriate data, information sources, and strategies for gathering data will be in place to conduct a subsequent evaluation.
The evaluation of the ICSA for the STPCORP is to be completed prior to the Government of Canada allocating funds for the second phase of the remediation project.
The Tar Ponds and Coke Ovens sites, located in the heart of Sydney, Nova Scotia and within four kilometres of more than 25,000 people, are one of Canada's largest contaminated sites. They contain sediments, from 100 years of private and public sector steel making, contaminated with polycyclic aromatic hydrocarbons (PAHs) and polychlorinated biphenyls (PCBs), both of which are classified as toxic substances under the 1999 Canadian Environmental Protection Act (CEPA). The sites pose risks to the environment and require action according to national contaminated sites guidelines.
The risk associated with public interest and media scrutiny is high. The public and environmental and citizens' groups have been highly sensitive to the activities undertaken and are generally either highly supportive or intensely critical of all levels of government.
On May 12, 2004, the federal government and the province of Nova Scotia signed a Memorandum of Agreement (MOA) for the remediation of the Tar Ponds and Coke Ovens sites in Sydney, Nova Scotia over a period of ten years.
The remediation project is structured in two phases, each under a separate cost-share agreement. The Interim Cost-Share Agreement (ICSA), signed on September 9, 2005, involves:
- Establishing a provincial implementing agency (Sydney Tar Ponds Agency, [STPA]);
- Progressing with design and engineering work;
- Putting in place of governance, management and stewardship structures, including preliminary works such as a project management framework, risk and performance management;
- Conducting a joint environmental assessment review;
- Establishing control and reporting requirements;
- Planning and implementation of preventative works projects designed to curtail the current dispersion of contaminants; and
- Preparing various aspects of the sites for the remediation.
The second phase, known as, the remediation phase, is scheduled to begin in 2007 under a second and separate cost-share agreement.
Public Works and Government Services Canada (PWGSC) serves as the federal lead and the Nova Scotia Department of Transportation and Public Works (NSDTPW) serves as the provincial lead. A Project Management Committee (PMC) of senior federal and provincial representatives is responsible for ensuring project implementation is consistent with the 2004 Memorandum of Agreement (MOA). Implementation of the project is the responsibility of the province of Nova Scotia. The Government of Nova Scotia has created the Sydney Tar Ponds Agency (STPA), a single purpose entity in order to carry out this function.
[ * ]
Because this is a project with a definite end, not a potentially long-term program or a policy, the RMAF, also recommended that questions related to the relevance issue and to Expenditure Review Criteria (ERC) not be considered. Finally, it was suggested that because the governance, performance and risk management, control and reporting instruments, developed during the ICSA are to be used throughout the ten year life of the project, these aspects also be evaluated from a design and implementation perspective. OAE agrees with all of the RMAF recommendations, and these recommendations are reflected in this evaluation framework report.
The evaluation questions to be examined in the Evaluation of the ICSA are categorized within the following areas:
- Design and Implementation;
- Success; and
- Cost Effectiveness.
Design and Implementation
- Is the governance structure effective and appropriate to achieve expectations of the ICSA? Are there changes needed for the next CSA?
- Are the set of management and control instruments, including the Preliminary Works, effective and appropriate to achieve expectations of the ICSA? Are there changes needed for the next CSA?
- To what extent has the health and safety of workers and general public been protected during the ISCA?
- To what extent have the contaminants entering or leaving the site been controlled?
- To what extent has the public been engaged during the term of the ICSA? Formally (Environmental Assessment consultations and CLC)? Informally (e.g., site visits, presentations; educational events, etc.)?
- To what extent have First Nations been engaged during the term of the ICSA?
- Are there any lessons learned, for next CSA? For other remediation projects? For other major cost-share agreements?
Success
- Have the ICSA Preventative Works projects been completed as planned?
- To what extent has the on-site environmental quality improved during the ICSA?
- Were environmental issues and risks mitigated during the ICSA?
- Was the role of PWGSC leading the federal effort contributing to its success?
- What is the extent of citizen support for the cleanup efforts and the results achieved to date?
- Have there been any unintended impacts to date as a result of the ICSA?
Cost-Effectiveness
- To what extent has the ICSA been managed and implemented efficiently? Were resources adequate? Could there have been savings or could costs have been reduced without increasing risks disproportionately?
It is recommended that the DG, Office of Greening Government Operations (OGGO), and the DG Atlantic Region, accept this evaluation framework the basis for undertaking the summative evaluation of the Interim Cost-Share Agreement for the Sydney Tar Ponds and Coke Ovens Remediation Project.
1.0 Introduction
This document presents the preliminary draft of the Evaluation Framework for the Interim Cost-Share Agreement (ICSA) for the Sydney Tar Ponds and Coke Ovens Remediation Project (STPCORP).
1.1 Objective
The evaluation framework will ensure that appropriate evaluation issues are identified and addressed and that the appropriate data, information sources, and strategies for gathering data are in place to conduct a subsequent evaluation in accordance with the Evaluation Policy of the Government of Canada, and meet the evaluation standards of the Office of Audit and Evalution (OAE) of Public Works and Government Services Canada (PWGSC).
The evaluation of the ICSA for the STPCORP is required prior to the Government of Canada allocating funds for the second phase of the remediation project.
1.2 Scope
This evaluation framework focuses on the activities and results within the realm of PWGSC in relation to the ICSA for the STPCORP.
The scope of this PWGSC evaluation framework does not include a review or assessment of any of the activities and functions that are the exclusive realm of the Government of Nova Scotia.
1.3 Methodology
The following approach was used to develop this evaluation framework:
- A review of key documents related to the STPCORP, including but not limited to: [ * ], the Memorandum of Agreement, the Interim Cost-Share Agreement, The Project Management Framework and the Result-based Management Accountability Framework (RMAF) and the Risk-based Audit Framework (RBAF);
- An initial round of meetings with key stakeholders to validate the issues identified in the RMAF and identify any changes in issues since the RMAF was completed, the list of persons interviewed is attached as Annex A - List of Interviewees;
- Results from the analysis of meetings and the document review were used to develop:
- A logic model that graphically depicts the ICSA for the STPCORP;
- Significant Evaluation Issues/Questions; and
- A Methodology.
- Preparation and submission of a preliminary draft evaluation framework report for review and comment;
- Submission of a revised draft for acceptance by the Director General (DG) Office of Greening Government Operations (OGGO) and the DG Atlantic Region; and
- Issuance of a final evaluation framework report.
1.4 Background
The Tar Ponds and Coke Ovens sites, located in the heart of Sydney, Nova Scotia and within four kilometres of more than 25,000 people, are one of Canada's largest contaminated sites. They contain sediments, from 100 years of private and public sector steel making, contaminated with polycyclic aromatic hydrocarbons (PAHs) and polychlorinated biphenyls (PCBs), both of which are classified as toxic substances under the 1999 Canadian Environmental Protection Act (CEPA). The sites pose risks to the environment and require action according to national contaminated sites guidelines.
Following a number of unsuccessful attempts at clean-up, a cost-share agreement between the federal government, the province of Nova Scotia and the Regional Municipality of Cape Breton (CBRM), supported planning and preparatory work required for the subsequent remediation including detailed investigation of the nature and extent of the contaminants, environmental site assessments, health studies and site stabilization. Various potential remediation approaches were evaluated, and in May 2003 recommendations for the permanent remediation were released following a community-based process. The federal and provincial governments reviewed the options, community recommendations and associated costs and developed a preferred option in September, 2003 which involves the destruction of the worst contaminants, the treatment of others and eventual containment and capping of remaining contaminants.
On May 12, 2004, the federal government and the province of Nova Scotia signed a Memorandum of Agreement (MOA) for the remediation of the Tar Ponds and Coke Ovens sites in Sydney, Nova Scotia over a period of ten years. The total estimated federal costs over the ten-year life of the project is $320 million, comprising the federal cost-share of approximately $282M to be paid to the province of Nova Scotia and related federal operating costs of $38M.
The risk associated with public interest and media scrutiny is high. The public and environmental and citizens' groups have been highly sensitive to the activities undertaken and are generally either highly supportive or intensely critical of all levels of government. The interests of Municipal and Aboriginal governments are expected to be significant factors throughout the remediation project.
2 Component Profile ICSA
The profile provides a description of the ICSA for the STPCORP in terms of its, objectives, governance, stakeholders and beneficiaries, and resource requirements.
2.1 Mandate
The remediation project is structured in two phases, each under a separate cost-share agreement. The Interim Cost-Share Agreement (ICSA), signed on September 9, 2005, involves:
- Establishing a provincial implementing agency (Sydney Tar Ponds Agency, (STPA);
- Progressing with design and engineering work;
- Putting in place of governance, management and stewardship structures, including preliminary works such as a project management framework, risk and performance management;
- Conducting a joint environmental assessment review;
- Establishing control and reporting requirements;
- Planning and implementation of preventative works projects designed to curtail the current dispersion of contaminants; and
- Preparing various aspects of the sites for the remediation.
The second phase, known as, the remediation phase, is scheduled to begin in 2007 under a second and separate cost-share agreement.
The governance, performance and risk management, control and reporting instruments, developed during the ICSA phase, are to ensure good management and stewardship of resources and contribute to successful implementation throughout the life of the STPCORP. The development of mechanisms to inform and engage the community will help demonstrate government's commitment to, and build the citizens' support of, the clean-up efforts.
2.2 Governance
The roles, responsibilities and accountabilities of the key parties involved in the delivery of this project are set out in detail in the ICSA and the Project Management Framework.
PWGSC's role is to lead the federal participation in the project. As such, it is responsible for: negotiating agreements and protocols under the MOA:
- Developing the appropriate project and funding oversight mechanisms to control the flow of federal funds for legal and remediation services that flow through it; acting as the Responsible Authority (RA) during the environmental assessment;
- Liaising with First Nations; and
- Managing federal reporting, submission and communication requirements.
The roles and responsibilities of both Environment Canada (EC) and Health Canada (HC) relate to their mandated responsibilities and their extensive history with the sites. EC is responsible to enforce federal environmental regulations, to provide technical advice to those dealing with contaminated sites, and to act as a Responsible Authority during the environmental assessment. In addition, EC provides assistance to PWGSC on technical issues, historic studies and communications. Because health issues remain sensitive, HC provides assistance to PWGSC on technical issues, historic studies, and communications and health issues during the environmental assessment as well as support to DOJ during the litigation process.
Other government departments participate where necessary. Several other committees have been established to provide advice, support or direction including the Technical/Regulatory Advisory Committee, the Operational Advisory Committee, a Community Liaison Committee (CLC), and the PWGSC Senior Project Advisory Committee (SPAC).
A PMC comprising senior representatives from PWGSC, the NSDTPW and with regular attendance of STPA exercises overall responsibility for the Cost-Share Agreements and ensure implementation of the project is consistent with the agreement, develops an RMAF, reviews performance management and evaluation reports and liaises with elected officials and First Nations representatives.
An Independent Engineer (IE) is responsible to ensure the technical merits of the project are addressed, the implementing party and the general contractor have appropriate and effective management controls in place, and reports on costs and contract compliance. The IE is also responsible for gathering and/or reporting certain items in the performance management and evaluation strategies.
Implementation and management of the project is the responsibility of the province of Nova Scotia, through its implementing agency. The STPA, as the implementing agency, is accountable for all aspects of the project including:
- Design documents and specifications;
- Contract tendering;
- Project scheduling;
- Quality control and assurance;
- Project work verification;
- Collecting performance management and evaluation data; and
- Providing quarterly progress and financial reports to the Project Management Committee (PMC).
The STPA is responsible for all procurement and contract management, including contracting for procurement audits and engaging environmental consultants to audit compliance with environmental regulations and standards in accordance to provincial policy, guidelines and standards.
2.3 Stakeholders and Beneficiaries
The key beneficiaries of this project are the citizens of Sydney where the remediation is expected to generally improve local environmental conditions and transform unsightly, unusable locations into sites that are ready for use. Furthermore, in working with the local university and high schools, the STPA hopes to create opportunities to build a local base of knowledge and skills and to share this knowledge with others.
During the remediation project itself, First Nations members in the area will benefit from the development and implementation of an Aboriginal procurement strategy and contractors, local and otherwise, hired to work on the project will profit. Local business may also benefit from increased sales and there will be employment opportunities for residents.
2.4 Resource Requirements
[ * ].
This amount is expected to cover $24,656,713 for the Federal share of the Preliminary and Preventative Works (60 percent of the total estimated cost), to be transferred to the province of Nova Scotia, as well as $17,633,773 for the operating expenses of three federal departments, of which $11,736,000 represents PWGSC's share.
3 Logic Model
Exhibit 1, on the next page, illustrates how the ICSA activities contribute to the achievement of its stated objectives and desired outcomes.
3.1 Outputs to Outcomes
The ICSA, which is in effect until March 31, 2007, provides for undertaking specific preventative and preliminary works over a three-year period.
For the most part, the expected immediate outcomes of the preventative works projects are to reduce, eliminate or control contaminants that are on, or could enter or leave, specific sites through surface and ground water and to reduce potential environmental risks as well as to otherwise prepare for the main cleanup with required works from a sequencing perspective which precede the main remediation.
The preliminary works are to include the establishment of the provincial implementing agency (Sydney Tar Ponds Agency); progressing with design and engineering work; conduct of the Environmental Assessment process and the development of plans and site activities that will enable the full site remediation to proceed. If successful, these activities will have the immediate outcomes of protecting the health and safety of workers and the public as well as mitigating potential environmental issues. Advancement in these activities will demonstrate governments' commitment to citizens and progress in the clean up, as well as improving on-site environmental quality. The intermediate outcome will be to ensure that the Tar Ponds and Coke Ovens sites are ready for the actual remediation and that citizens support the clean-up efforts.
Exhibit 1 (View detailed figure by clicking on link)
4 Evaluation Strategy
[ * ]
4.1 Evaluation Questions
Because this is a project with a definite end, not a potentially long-term program or a policy, PWGSC had recommended that questions related to the relevance issue and to Expenditure Review Criteria (ERC) not be considered. They also recommended that since the governance, performance and risk management, control and reporting instruments, established during the ICSA phase are expected to be used throughout the life of the project, it would be important that these aspects be evaluated from a design and implementation perspective as part of this evaluation. OAE agrees with these recommendations. Consequently, the evaluation questions for this framework are categorized within the following areas:
- Design and Implementation;
- Success; and
- Cost Effectiveness.
4.1.1 Design and Implementation
- Is the governance structure effective and appropriate to achieve expectations of the ICSA? Are there changes needed for the next CSA?
- Are the set of management and control instruments, including the Preliminary Works, effective and appropriate to achieve expectations of the ICSA? Are there changes needed for the next CSA?
- To what extent has the health and safety of workers and general public been protected during the ISCA?
- To what extent have the contaminants entering or leaving the site been controlled? What has been the effect on the environmental quality off-site?
- To what extent has the public been engaged during the term of the ICSA? Formally (Environmental Assessment consultations and CLC)? Informally (e.g., site visits, presentations; educational events, etc.)
- To what extent have First Nations been engaged during the term of the ICSA?
- Are there any lessons learned, for next CSA? For other remediation projects? For other major cost-share agreements?
4.1.2 Success
- Are the ICSA Preventative Works projects completed as planned?
- To what extent has the on-site environmental quality improved during the ICSA?
- Were environmental issues and risks mitigated during the ICSA?
- Was the role of PWGSC leading the federal effort contributing to its success?
- What is the extent of citizen support for the cleanup efforts and the results achieved to date?
- Have there been any unintended impacts to date as a result of the ICSA?
4.1.3 Cost-Effectiveness
- To what extent has the ICSA been managed and implemented efficiently? Were resources adequate? Could there have been savings or could costs have been reduced without increasing risks disproportionately?
4.2 Methodology
In developing the approach for this evaluation, OAE was influenced by a number of factors, the most important being the RMAF that was developed less than a year ago and accepted by all participants, including TBS.
To this end, Annex B - Evaluation Matrix presents the Evaluation Matrix that outlines the evaluation questions to be examined, indicators, and key data sources.
4.2.1 Documentation and File Review
Relevant documents, files and reports will be reviewed in order to assess the indicators elaborated in the matrix. Documentation related to previous audits and performance reports will also be reviewed to identify common themes and issues related to the design and implementation and achievement of immediate and intermediate outcomes.
The evaluation team asked the responsible entities identified in the RMAF whether the documents and information identified for each of the proposed lines of inquiry are available in order to carry out the evaluation this Fall. The evaluation team was given assurance that the data sources are available. A separate web-based electronic archive has been created for the purposes of the evaluation and staff in the Sydney OGGO Project Office have been ensuring that the archive is populated by all electronic copies of documents, reports etc. requested by the evaluation team. This documentation is integrated with the web-based electronic archive, or document record management system, used by the Project Team in management of its responsibilities.
4.2.2 Key Informant Interviews
Interviews with individuals involved with the development and implementation of STPCORP represent a rich source of information regarding PWGSC achievements, impacts and opportunities for enhancements. Key informants include individuals from the following groups:
- OGGO Sydney Office staff;
- PWGSC senior managers at HQ
- Members of the PMC:
- The province of Nova Scotia;
- Management and Staff of the STPA;
- The Independent Engineer and his staff;
- The Community Liaison Committee (CLC);
- Certain Interest Groups not members of the CLC;
- Municipal representatives;
- Representatives of First Nations; and
- Representatives of HC, EC, IAND and Justice Canada.
Key informants will be asked questions based on their roles and responsibilities, their familiarity with the evaluation question being measured and their involvement in the Project.
5 Recommendation
It is recommended that the DG, OGGO, and the DG Atlantic Region, accept this evaluation framework as the basis for undertaking the summative evaluation of the Interim Cost-Share Agreement for the Sydney Tar Ponds and Coke Ovens Remediation Project.
Annex A - List of Interviewees
Project Management Committee (PMC)
- Ken Swain
- Project Director, PWGSC
- Gary Campbell
- Executive Director, NSTPW
PMC Secretariat
- Randy Vallis
- Project Manager, PWGSC
Public Works And Government Services Canada
- Margaret Kenny
- DG OGGO
- Michael Billowits
- Manager Contaminated Sites, OGGO
(replaced by Paul Frame in August 2006) - Alain Trépanier
- A/ADM CSPCB
- Jonathan Higdon
- A/DG Corporate Policy and Planning
- Janet Randall
- Director Cabinet and Treasury Board
Submissions Services - Roch Huppé
- A/DG Financial Operations
- Steve Cole
- Manager TB Submissions and Corporate
Investments Analysis
Sydney Tar Ponds Agency
- Barb Baillie
- Director, Engineering
- Parker Donham
- Director, Communications
- Wilf Kaiser
- A/Director, Environmental
- Frank Potter
- A/CEO
- Corrie Stewart
- Director, Corporate Services
Independent Engineer - Conestoga-Rovers & Associates
- Walter Van Veen
- Project Manager
Nova Scotia Dept Of Environment & Labour
- Terry MacPherson
Community Liaison Committee (CLC)
- Dr. Alastair MacLeod
- Chair, CLC
Cape Breton Regional Municipality (CBRM)
- Jerry Ryan
- Chief Administrative Officer
Annex B
| Evaluation Issue/Questions | Indicators | Data Sources |
|---|---|---|
| Design and Implementation | ||
1. Is the governance structure effective and appropriate to achieve expectations of the ICSA?
|
|
Key Documents MOA (May 2004); ICSA; RBAF; RMAF, Project Management Plan [sources for governance elements; roles & responsibilities; lines of accountability] Progress Report to TBS (period - May/04 to Mar/06); agenda / records of decision, etc. of PMC Independent Engineer Monthly Reports; STPA quarterly & annual reports; OGGO Key Interviews
|
2. Are the set of management and control instruments, including the Preliminary Works, effective and appropriate to achieve expectations of the ICSA?
|
|
Key Documents Agreement / contract with Independent Engineer; Monthly Reports (IE); STPA Quarterly & Annual Reports; Progress Report to TBS (May/04 to Mar/06); PMC documents, agenda; OGGO Progress Reports; Project Management and Program Management Plans MOA (May 2004); ICSA; RBAF; RMAF Key Interviews
|
3. To what extent has the health and safety of workers and general public been protected during the ISCA? |
|
Key Documents
|
| 4. To what extent have the contaminants entering or leaving the site been controlled? |
|
Key Documents
|
5. To what extent has the public been engaged during the term of the ICSA?
|
|
Key Documents
|
| 6. To what extent have first nations been engaged during the term of the ICSA? |
|
Key Documents
|
7. Are there any lessons learned, for next CSA?
|
|
Key Documents
|
| Success | ||
| 8. Are the ICSA Preventative Works projects completed as planned? |
|
Key Documents
Key Interviews
|
| 9. To what extent has the on-site environmental quality improved during the ICSA? |
|
Key Documents
|
| 10. Were environmental issues and risks mitigated during the ICSA? |
|
Key Documents
|
| 11. Was the role of PWGSC leading the federal effort contributing to its success? |
|
Key Documents
|
| 12. What is the extent of citizen support for the cleanup efforts and the results achieved to date? |
|
Key Documents
Key Interviews
|
| 13. Have there been any unintended impacts to date as a result of the ICSA? |
|
Key Documents
|
| Cost-effectiveness | ||
14. To what extent has the ICSA been managed and implemented efficiently?
|
|
Key Documents
|