2006-605 Evaluation of the Interim Cost-Share Agreement for the Sydney Tar Ponds and Coke Ovens Sites Remediation Project, Final Report

2007-03-21

Table of Contents

• Executive Summary
• 1 Introduction
  • 1.1 Mandate
  • 1.2 Objective and Scope
  • 1.3 Context
  • 1.4 Stakeholders and Beneficiaries
• 2 Interim Cost-Share Agreement Component Profile
  • 2.1 Mandate
  • 2.2 Governance
  • 2.3 Resource Requirements
• 3 Evaluation Study
  • 3.1 Evaluation Issues
  • 3.2 Evaluation Methodology
• 4 Logic Model
• 5 Key Findings
  • 5.1 Design and Implementation
  • 5.2 Success
  • 5.3 Cost-Effectiveness
• 6 Conclusions
  • 6.1 Design and Implementation
  • 6.2 Success
  • 6.3 Cost Effectiveness
• 7 Recommendations
• Appendix A - Summary of Findings
• Appendix B - Key Informant Interviews

Executive Summary

Background

Canada and Nova Scotia entered into an Interim Cost-Share Agreement so that the Sydney Tar Ponds and Coke Ovens Sites Remediation Project could commence as soon as possible, in a manner consistent with the May 2004 Memorandum of Agreement. The Interim Cost-Share Agreement, signed on September 9, 2005, governs projects designed to prepare sites for full remediation. It also requires the design and implementation of the governance, performance and risk management, and control and reporting instruments to ensure good management and stewardship of resources and to contribute to successful implementation throughout the life of the project. The projects began in 2004 and extend to the expiration of the Interim Cost-Share Agreement on March 31, 2007. The second phase, known as the remediation phase, is scheduled to begin in 2007 under a second and separate cost-share agreement.

This evaluation is the first of three proposed evaluations for the ten-year Sydney Tar Ponds and Coke Ovens Sites Remediation Project. It will be followed by a mid-term evaluation of the cost-share agreement to be completed in 2010. The mid-term evaluation will report on early results and provide additional assurances that the project is proceeding as planned. A summative evaluation (final) is to correspond with the completion of the Sydney Tar Ponds and Coke Ovens Sites Remediation Project, and will focus on intermediate outcomes and report on evidence of long-term outcomes.

Objective and Scope

The objective of this evaluation was to assess the design and implementation, success, and cost effectiveness of the Sydney Tar Ponds and Coke Ovens Sites Remediation Project under the Interim Cost-Share Agreement.

This evaluation reports on:

• Implementation of the Interim Cost-Share Agreement in terms of the delivery of the preliminary and preventative works projects/activities;
• Its success in preparing the sites for the full remediation; and
• Its success in building public support for the project.

This evaluation examined the extent to which management, accountability, and governance mechanisms have been established and are working effectively. It also sought to identify any improvements needed to support the next phase of the project.

Conclusions

Design and Implementation

The project governance structure comprises a Project Management Committee with senior representatives from the lead federal and provincial departments. This committee oversees an implementing agency, the Sydney Tar Ponds Agency, created by the Province of Nova Scotia specifically for this purpose. This governance structure is sound and is further strengthened by the involvement of an Independent Engineer. The Independent Engineer provides the Project Management Committee with unbiased third-party assessments of design and implementation activities and ensures that the Memorandum of Agreement is being respected.

Overall, the governance framework has been implemented effectively. However, the role of the province's lead department-as Public Works and Government Services Canada's counterpart-needs to be strengthened and distinguished from the role of the implementing agency to ensure that the governance model is respected.

The Sydney Tar Ponds Agency has been established, and most key staff positions have been filled, though staffing was slow to happen. A large $30 million contract has been awarded for detailed engineering design and construction supervision that could have an impact on the role of the Sydney Tar Ponds Agency during the next phase of the project.

Success

Significant milestones have been achieved for the project under the Interim Cost-Share Agreement. To begin with, the federal and provincial partners came together and agreed to participate in a Joint Review Panel for the environmental assessment of the project. This critical step was run very efficiently, illustrated by the fact that the panel report was released a mere 10 months following the naming of the Panel members. Public Works and Government Services Canada played a significant lead role for this environmental assessment process. And although other federal participants viewed the Department to be over-zealous at times in driving the agenda forward to meet the tight timelines, they agreed that significant issues were properly addressed.

Next, the planned projects, designed to prepare the sites for full remediation, have largely been completed, with the exception of cleanup of an on-site cooling pond. The cooling pond project has been deferred until 2007 to enable an adequate response to certain recommendations from the Joint Review Panel. The delay is not expected to affect the remediation phase schedule.

Finally, significant efforts went into garnering citizen support for the project. Awareness of the project has improved; as well there has been some increase in the level of trust and confidence with the cleanup effort.

Cost Effectiveness

The work conducted in this phase of the project under the Interim Cost-Share Agreement has been done in a cost- effective manner. The planned outcomes have overall been achieved, and they have been achieved with spending projected to be $31.2M, with one major project having a cost estimate of $4.4M outstanding. This is well under the $42M allocated for the Interim Cost-Share Agreement.

Public Works and Government Services Canada officials have also been able to function effectively while spending less than the amount approved by Treasury Board for operating costs. To illustrate, projected PWGSC operating costs of $6.1 million represent only 70% of the $8.7M that were approved.

Recommendations

Based on the key findings and conclusions contained in this report, the Audit and Evaluation Branch recommends that the Regional Director General, Atlantic Region:

1. Work with the Department's provincial counterparts on the Project Management Committee to clearly delineate the province's dual roles and garner full participation of the Nova Scotia Department of Transportation and Public Works in providing project oversight.
2. In partnership with the Nova Scotia Department of Transportation and Public Works, review the terms of reference and operation of committees established to ensure they are consistent with the governance framework.
3. In partnership with the Nova Scotia Department of Transportation and Public Works, examine the project payment mechanism to identify adjustments to enable the timely processing of payment applications.

1.0 Introduction

The purpose of this report is to present the findings from the evaluation of the Interim Cost-Share Agreement for the Sydney Tar Ponds and Coke Ovens Sites Remediation Project.

In May 2004, Canada and Nova Scotia entered into a Memorandum of Agreement that committed the governments to cleanup the Sydney Tar Ponds and Coke Ovens sites over a 10-year period. The parties acknowledged that it would take a reasonable period before definitive steps to begin to fully implement the Sydney Tar Ponds and Coke Ovens Sites Remediation Project could occur, but that certain projects should commence as soon as possible. In September 2005, Canada and Nova Scotia signed the Interim Cost-Share Agreement. It authorized commencement of specific projects and represented agreement for the funding of planning, environmental assessment, preparation and preliminary project costs. The Interim Cost-Share Agreement was designed to govern the project until March 31, 2007 or until a replacement was agreed to by both parties. A second cost-share agreement will be developed for the main remediation project that is scheduled to begin in fiscal year 2007-2008.

This evaluation is the first of three proposed evaluations for the project:

1. An evaluation of the Interim Cost-Share Agreement, to be completed by January 2007;
2. A mid-term evaluation of the cost-share agreement to report on early results and provide additional assurances that the project is proceeding as planned, to be completed March 2010; and
3. A summative evaluation, which will focus on intermediate outcomes and report on evidence of long-term outcomes, to be completed March 2014.

This report has eight sections:

• This section (1.0) provides a background to the evaluation, including the mandate, objective and scope, context, and stakeholders and beneficiaries.
• Section 2.0 contains a component profile of the Interim Cost-Share Agreement, including its mandate, governance, and resource requirements.
• Evaluation issues examined and the Project Logic Model are presented in sections 4.0 and 5.0 respectively.
• Key findings from the evaluation are included in section 6.0.
• Conclusions based on the evaluation issues are in section 7.0.
• Finally, recommendations are presented in section 8.0.

1.1 Mandate

Public Works and Government Services Canada (PWGSC) made a commitment to the Treasury Board to undertake an evaluation prior to requesting funding for the next phase of the Sydney Tar Ponds and Coke Ovens Sites Remediation Project.

The Audit and Evaluation Committee mandated this evaluation as part of the PWGSC 2006–2007 Internal Audit and Evaluation Plan. An Evaluation Framework for the Interim Cost-Share Agreement for the Sydney Tar Ponds and Coke Ovens Sites Remediation Project was completed in September 2006 and was tabled at the Audit and Evaluation Committee in December 2006. At the meeting the committee agreed on the proposed scope and methodology for this evaluation.

1.2 Objective and Scope

The objective of this evaluation was to assess the design and implementation, success, and cost effectiveness of the Sydney Tar Ponds and Coke Ovens Sites Remediation Project under the Interim Cost-Share Agreement.

The evaluation reports on: implementation of the Interim Cost-Share Agreement in terms of delivery of the preliminary and preventative works projects/activities; its success in preparing the sites for the full remediation; and its success in building public support for the project. The evaluation examined the extent to which management, accountability, and governance mechanisms have been established and are working effectively. It also sought to identify any improvements needed to support the next phase of the project.

Canada and Nova Scotia entered into the Interim Cost-Share Agreement so that the project could commence as soon as possible, in a manner consistent with the May 2004 Memorandum of Agreement. The Interim Cost-Share Agreement was signed on September 9, 2005, and governs implementation of the works and the project until March 31, 2007, or until it is replaced or extended by agreement of both parties. The works projects under the Interim Cost-Share Agreement commenced soon after signing the Memorandum of Agreement and are scheduled to extend to March 31, 2007.

Because this is a project with a definite end and not a potentially long-term program or a policy, the Results-based Management and Accountability Framework developed for the project recommended that questions related to the relevance issue and to Expenditure Review Criteria not be considered. It was also suggested that because the governance, performance and risk management, control, and reporting instruments developed during the Interim Cost-Share Agreement are to be used throughout the 10-year life of the project, these aspects should also be evaluated from a design and implementation perspective. Audit and Evaluation Branch agreed to these Results-based Management and Accountability Framework recommendations, and they were included in the Audit and Evaluation Committee approved evaluation framework and subsequently this evaluation.

1.3 Context

The Tar Ponds and Coke Ovens sites, located in the heart of Sydney, Nova Scotia, and within 4 kilometers of more than 25,000 people, are one of Canada's largest contaminated sites. They contain sediments from 100 years of private and public sector steel making and are contaminated with polycyclic aromatic hydrocarbons and polychlorinated biphenyls, both of which are classified as toxic substances under the 1999 Canadian Environmental Protection Act. The sites pose risks to the environment and require action according to national contaminated-sites guidelines.

Following a number of unsuccessful attempts at cleanup, a cost-share agreement between the federal government, the Province of Nova Scotia, and the Cape Breton Regional Municipality¹ supported planning and preparatory work required for the subsequent remediation. This agreement included a detailed investigation of the nature and extent of the contaminants, environmental site assessments, health studies, and site stabilization. Various potential remediation approaches were evaluated, and in May 2003, recommendations for the permanent remediation were released following a community-based process that involved community consultation. The federal and provincial governments reviewed the options, community recommendations, and associated costs and developed a preferred option in September 2003. This option involves the destruction of the worst contaminants, the treatment of others, and the eventual containment and capping of remaining contaminants.

On May 12, 2004, the federal government and the province of Nova Scotia signed an Memorandum of Agreement for the remediation of the Tar Ponds and Coke Ovens sites. The project will be run over a 10-year period and cost $400M to complete; Canada agreed to contribute $280M to the cost of the project. In addition, related federal government operating costs outside the Memorandum of Agreement are estimated to be $39M.

1.4 Stakeholders and Beneficiaries

The key beneficiaries of this project are the citizens of Sydney where the remediation is expected to generally improve local environmental conditions and transform unsightly, unusable locations into sites that are ready for use. Furthermore, in working with the local university and high schools, the implementing agency (Sydney Tar Ponds Agency) hopes to create opportunities to build a local base of knowledge and skills and to share this knowledge with others.

During the remediation project itself, First Nations members in the area will benefit from the development and implementation of an Aboriginal procurement strategy, and contractors, local and otherwise, hired to work on the project will profit. Local business may also benefit from increased sales, and there will be employment opportunities for residents.

2.0 Interim Cost-Share Agreement Component Profile

The profile provides a description of the Interim Cost-Share Agreement for the Sydney Tar Ponds and Coke Ovens Sites Remediation Project in terms of its mandate, governance and resource requirements.

2.1 Mandate

The remediation project is structured into two phases, each under a separate cost-share agreement. The Interim Cost-Share Agreement involves:

• Establishing a provincial implementing agency;
• Progressing with design and engineering work;
• Putting in place governance, management, and stewardship structures, including preliminary works such as a project management framework and a performance and risk management framework;
• Conducting a joint environmental assessment review;
• Establishing control and reporting requirements;
• Planning and implementing preventative works projects designed to curtail the current dispersion of contaminants; and
• Preparing various aspects of the sites for the remediation.

The second phase, known as the remediation phase, is scheduled to begin in 2007 under a second and separate cost-share agreement.

The governance, performance and risk management, and control and reporting instruments developed during the Interim Cost-Share Agreement phase are meant to ensure good management and stewardship of resources and to contribute to successful implementation throughout the life of the project. The development of mechanisms to inform and engage the community will help demonstrate the federal and provincial governments' commitment to the cleanup efforts and build citizens' support of the cleanup efforts.

2.2 Governance

The roles, responsibilities, and accountabilities of the key parties involved in the delivery of this project are set out in the Interim Cost-Share Agreement and the Project Management Framework.

PWGSC's role is to lead the federal participation in the project. As such, it is responsible for:

• Negotiating agreements and protocols under the Memorandum of Agreement;
• Developing the appropriate project and funding oversight mechanisms to control the flow of federal funds for legal and remediation services;
• Acting as the Responsible Authority during the environmental assessment;
• Liaising with First Nations; and
• Managing federal reporting, submission, and communication requirements.

The roles and responsibilities of both Environment Canada and Health Canada relate to their mandated responsibilities and their extensive history with the sites. Environment Canada is responsible for enforcing federal environmental regulations, providing technical advice to those dealing with contaminated sites, and acting as a Responsible Authority during the environmental assessment. In addition, Environment Canada assists PWGSC on technical issues, historic studies, and communications. Because health issues remain sensitive, Health Canada assists PWGSC on technical issues, historic studies, and communications and health issues during the environmental assessment as well as support to the Department of Justice during the litigation process.

Other government departments participate where necessary. Several other committees have been established to provide advice, support, or direction, including the Operational Advisory Committee, the Technical Working Group, a Community Liaison Committee, and the PWGSC Senior Project Advisory Committee.

A Project Management Committee, comprising senior representatives from PWGSC, the Nova Scotia Department of Transportation and Public Works, and with regular attendance of Sydney Tar Ponds Agency, exercises overall responsibility for the cost-share agreements. The Project Management Committee ensures that implementation of the project is consistent with the agreement, develops an Results-based Management and Accountability Framework, reviews performance management and evaluation reports, and liaises with elected officials and First Nations representatives.

An Independent Engineer is responsible for ensuring that: the technical merits of the project are addressed; the implementing agency and the general contractor have appropriate and effective management controls in place; and reports on costs and contract compliance are prepared. The Independent Engineer is also responsible for gathering and/or reporting certain items in the performance management and evaluation strategies.

Implementation and management of the project is the responsibility of the Province of Nova Scotia through its implementing agency, the Sydney Tar Ponds Agency. As the implementing agency, Sydney Tar Ponds Agency is accountable for all aspects of the project including:

• Design documents and specifications;
• Contract tendering;
• Project scheduling;
• Quality control and assurance;
• Project work verification;
• Performance management and evaluation data collection; and
• Provision of quarterly progress and financial reports to the Project Management Committee.

The Sydney Tar Ponds Agency is responsible for all procurement and contract management, including contracting for procurement audits and engaging environmental consultants to audit compliance with environmental regulations and standards in accordance with provincial policy, guidelines, and standards.

2.3 Resource Requirements

PWGSC has received Treasury Board approval to spend $42,290,486 in funding for the Interim Cost-Share Agreement phase:

• $24,656,713 for the federal share of the Preliminary and Preventative Works (60 percent of the total estimated cost), to be transferred to the province of Nova Scotia; and
• $17,633,773 for the operating expenses of three federal departments, of which $11,736,000 represents PWGSC's share.

3.0 Evaluation Study

3.1 Evaluation Issues

The main issues addressed by this evaluation are those that have been outlined in the Evaluation Framework for the Interim Cost-Share Agreement for the Sydney Tar Ponds and Coke Ovens Sites Remediation Project. Again, given the timing of this evaluation, that is, nearing the end of the preparatory phase of the project and moving into the full remediation phase, the issues were examined from a perspective of how successful and cost effective they have been and how well they will serve the project throughout its life. The evaluation issues and associated questions are presented below.

Design and Implementation

• 1. Is the governance structure effective and appropriate to achieve expectations of the Interim Cost-Share Agreement? Are there changes needed for the next cost-share agreement?
• 2. Are the set of management and control instruments, including the Preliminary Works, effective and appropriate to achieve expectations of the Interim Cost-Share Agreement? Are there changes needed for the next cost-share agreement?
• 3. To what extent has the health and safety of workers and the public been protected during the Interim Cost-Share Agreement?
• 4. To what extent have the contaminants entering or leaving the site been controlled?
• 5. To what extent has the public been engaged during the term of the Interim Cost-Share Agreement? Formally (e.g., environmental assessment consultations and Community Liaison Committee)? Informally (e.g., site visits, presentations, educational events, etc.)?
• 6. To what extent have First Nations been engaged during the term of the Interim Cost-Share Agreement?
• 7. Are there any lessons learned for the next cost-share agreement?
  • For other remediation projects?
  • For other major cost-share agreements?

Success

• 8. Have the Interim Cost-Share Agreement Preventative Works projects been completed as planned?
• 9. To what extent has the on-site environmental quality improved during the Interim Cost-Share Agreement?
• 10. Were environmental issues and risks mitigated during the Interim Cost-Share Agreement?
• 11. Was the role of PWGSC leading the federal effort contributing to its success?
• 12. What is the extent of citizen support for the cleanup efforts and the results achieved to date?
• 13. Have there been any unintended impacts to date because of the Interim Cost-Share Agreement?

Cost-Effectiveness

• 14. To what extent has the Interim Cost-Share Agreement been managed and implemented efficiently?
  • Were resources adequate?
  • Could there have been savings, or could costs have been reduced without increasing risks disproportionately?

3.2 Evaluation Methodology

The evaluation methodology incorporated the two lines of evidence outlined in the evaluation framework. The lines of evidence included reviewing project documentation and conducting key informant interviews.

3.2.1 Document Review

There was a large amount of documentary evidence produced individually or jointly by PWGSC, the Province of Nova Scotia, Sydney Tar Ponds Agency, and the Independent Engineer, all of which was made available to the evaluation team. A review of project documentation, reports, committee records, and correspondence resulted in an essential understanding of the project. These documents included background and contextual information on the Sydney Tar Ponds and Coke Ovens sites, agreements between Canada and the Province of Nova Scotia, and ongoing project management and reporting data. The documents reviewed were also used as a basis for developing the questionnaires for the key informant interviews.

3.2.2 Key Informant Interviews

Key informant interviews were conducted with 34 individuals from Sydney, Halifax, and Headquarters (Ottawa/Gatineau). These individuals represented:

• PWGSC
• The Nova Scotia Department of Transportation and Public Works
• The Nova Scotia Environment and Labour
• The Sydney Tar Ponds Agency
• The Cape Breton Regional Municipality
• First Nations
• Community groups
• Other federal government departments, and
• The Independent Engineer.

A full list of interviewees is included in Appendix B - Key Informant Interviews.

Interviews were conducted using semi-structured interview guides based on the evaluation issues. The questions varied according to the perspective of the interviewee. Key informants were provided with a copy of the interview guide prior to the interview to review and prepare their responses. Interviews were primarily conducted during September and October 2006.

From PWGSC, the Nova Scotia Department of Transportation and Public Works, and Sydney Tar Ponds Agency, the evaluation team attempted to interview senior project management with overall accountability for the project from their organizations' perspectives and directors with specialized perspectives, such as environment and communications. Cape Breton Regional Municipality and community group representatives were selected from organizations involved in a Community Liaison Committee and from those with special interests in the project. Interviewees from other federal government departments included those with significant involvement in project activities, such as involvement in the environmental assessment process.

4.0 Logic Model

Logic Model (View detailed figure by clicking on link)

5.0 Key Findings

This evaluation addressed issues of design/implementation, success and cost-effectiveness through a series of 14 questions. In synthesizing and analyzing the results of the research, the evaluation team determined that there was a certain degree of overlap between the key findings from each evaluation question. As such, this report presents the key findings as a series of issues, whereby the key findings have been drawn from one or more evaluation question. A more detailed summary of findings by evaluation question has been enclosed in Appendix A - Summary of Findings.

5.1 Design and Implementation

5.1.1 Provincial Roles Not Adequately Separated

In contrast to the single federal role for oversight of the project, with PWGSC as the lead department, the Province of Nova Scotia's dual roles for oversight and for implementation are split between two bodies. The first is the Nova Scotia Department of Transportation and Public Works, which is PWGSC's provincial counterpart in providing project oversight. The second is the Sydney Tar Ponds Agency, which the province has established as a Special Operating Agency responsible for day-to-day management and implementation of the project, as stipulated in the Memorandum of Agreement. The division between provincial responsibilities is not always apparent.

The Sydney Tar Ponds Agency has a history of heading up the province's tar ponds cleanup efforts. It had initially been created in 2001 to manage provincial involvement in the cleanup of the Sydney Tar Ponds and Coke Ovens sites. In its former incarnation, the Sydney Tar Ponds Agency was also responsible for working directly with its federal counterparts in dealing with the coordination and implementation of projects and strategic actions associated with the tar ponds. When the Memorandum of Agreement was developed, the former Sydney Tar Ponds Agency was identified as Nova Scotia's representative in discussions with PWGSC to conclude the implementation agreements. The current Deputy Minister and Executive Director of the Nova Scotia Department of Transportation and Public Works formerly held senior positions at the Sydney Tar Ponds Agency.

The Nova Scotia Department of Transportation and Public Works's involvement in the project can be described as limited. Two departmental officials based in Halifax have been appointed as members on the Project Management Committee, with one of the two playing dual roles as a committee member and as the provincial secretariat to the committee. This individual was the only resource from the Nova Scotia Department of Transportation and Public Works dedicated full time to the project during the Interim Cost-Share Agreement.

In contrast, PWGSC established a local office in Sydney, assigned a senior project manager, and organized an experienced project team to fulfil the department's responsibilities. The PWGSC Project Director reports directly and is accountable to the Regional Director General in Halifax. Both individuals are members of the Project Management Committee with a third PWGSC official, a senior project manager, functioning as the federal secretariat to the committee.

Disagreements have emerged between PWGSC and the Sydney Tar Ponds Agency. For example, PWGSC officials perceive at times that the Sydney Tar Ponds Agency is trying to act as PWGSC's provincial counterpart, due in part to the perceived “vacuum” created by the under-representation from the Nova Scotia Department of Transportation and Public Works. PWGSC views the Sydney Tar Ponds Agency as being akin to a contractor that was hired by the partners to implement the project, which is consistent with the duties and obligations of the implementing agency identified in the Interim Cost-Share Agreement. The Nova Scotia Department of Transportation and Public Works is aware that their level of resource dedication to the project has been raised as an issue. An Nova Scotia Department of Transportation and Public Works representative suggested that they originally thought PWGSC would have less direct involvement. Based on their own experience thus far in the project, they are considering dedicating more resources to the project. The Nova Scotia Department of Transportation and Public Works did not suggest a change in their federal partner's role as they move forward to the remediation phase of the project.

5.1.2 Public Engagement and Communications Strategy Developed and Implemented

A key strategic risk identified in the project Risk-Based Audit Framework was the potential for strong community opposition to the planned remediation project. Management was aware that community opposition could develop into demonstrations and media events that could interfere with work on the project. It was expected that the situation would remain dynamic for at least the first five years of the project, which includes the period covered by the Interim Cost-Share Agreement. To mitigate the identified risk, a strategy to engage and inform the public was developed and implemented.

Community Engagement Appropriate

Over 20 years ago, Canada and Nova Scotia signed an agreement to clean up the Sydney Tar Ponds. In the years that have passed, there has been a lot of talk about how the Tar Ponds and Coke Ovens sites should be cleaned up but very little by way of tangible results. Most recently, the local community had been intensively engaged through the Joint Action Group process that ran from 1996 to 2003. Joint Action Group members put in more than 100,000 volunteer hours in 950 public meetings seeking public agreement on how to clean up the sites. This was followed by workshops involving 1700 residents to consider a short list of cleanup options. By 2004 it was clear that the local community did not want to hear any more talk about the cleanup. In a poll conducted in October of that year, 89% of Cape Breton Regional Municipality residents indicated that they just wanted the government to get on with the cleanup.

Recognizing the mood of the public, project management chose a less intensive plan for community engagement than had been followed in the past. During this preliminary phase of the project, two key methods were used to engage the community: establishment of a Community Liaison Committee; and public consultation during the environmental assessment process. Lines of inquiry for the Community Liaison Committee included: review of the committee terms of reference and meeting minutes; direct observation of a Community Liaison Committee meeting; and interviews with committee members and project management. Environmental assessment public consultation lines of inquiry included a review of relevant correspondence and interviews with project managers.

The Community Liaison Committee was established to ensure a two-way flow of information between the implementing agency and the organizations represented on the Community Liaison Committee. The committee is composed of individuals representing local organizations from the fields of business, health, organized labour, environment, construction, post-secondary education, recreation, religion, community service, and the municipal government. All meeting minutes are made available to the public on the Sydney Tar Ponds Agency Web site. The minutes indicate that committee members have been acting as advocates for their respective constituent groups, and that project management has been providing regular project updates and responses to committee members' questions and concerns. Evaluators attending the September 26, 2006, Community Liaison Committee meeting corroborated this finding. The meeting was well organized, and there was a good exchange of information among committee members and with Sydney Tar Ponds Agency and PWGSC representatives. One-on-one interviews conducted with five committee members revealed that they believe that PWGSC and the Sydney Tar Ponds Agency have overall been responsive to their questions or issues raised.

The public was invited to comment on the Environmental Impact Statement in advance of the Joint Review Panel hearings held in April and May 2006. There were 55 submissions received, including hundreds of questions, from a variety of interested organizations and individuals between January 13 and February 17, 2006. The Sydney Tar Ponds Agency was able to respond to the Joint Review Panel by the March 2 deadline with detailed responses to all questions posed.

Citizens More Aware and Marginally More Confident about Cleanup Plan

Great efforts have gone into public communications. With the benefit of hindsight from past cleanup efforts, a communications plan was developed and approved by the Project Management Committee in September 2005. The plan was designed to take control of how the project was communicated to the public. During previous cleanup efforts, issues were debated in the media. The strategic focus of the new plan was to directly communicate with the interested public.

Direct communication with the public has been accomplished through a variety of methods consistent with the Project Management Committee-approved plan. The document review included examination of newsletters, news releases, a school outreach program, and the project Internet Web site (www.tarpondscleanup.ca). The Sydney Tar Ponds Agency also offers site tours to the public and attends community meetings. Interviews with senior communications officials indicate that most communications activities were conducted as planned.

Results from communications efforts can be observed through polling conducted on behalf of the Sydney Tar Ponds Agency. These polls indicate that citizens' awareness and understanding of the cleanup plan has improved during the Interim Cost-Share Agreement phase of the project. The number of respondents indicating that they had an understanding of the cleanup plan increased by 15 points to 44% of respondents from the second quarter (Q2 (July – September)) 2005 to Q1 (April – June) 2006. In a later poll conducted in October 2006, half of respondents questioned could name a particular project conducted under the Interim Cost-Share Agreement without being prompted.

Positive trends for citizens' trust and confidence in the cleanup plan have also been observed. In Q2 (July – September) 2005 polling, 25% of Cape Breton Regional Municipality respondents had some or a great deal of trust and confidence in the cleanup plan. In polling conducted the following year, 30% agreed that they had trust and confidence in the plan. In addition, those with little or no trust and confidence in the plan had decreased by 26 points to 47% of respondents.

The evaluation team observed that the communication function has been designed and implemented effectively. During interviews conducted, PWGSC and Sydney Tar Ponds Agency communications officials have advised that the communications plan will be updated to reflect changing communications priorities as the Sydney Tar Ponds and Coke Ovens Sites Remediation Project moves from the preliminary phase, under the Interim Cost-Share Agreement, to full remediation, under the next cost-share agreement.

5.1.3 Way Paved for Economic Benefits for First Nations

Potentially strong Aboriginal opposition was identified as another risk to the project. To ensure adequate Aboriginal involvement in the project, the Memorandum of Agreement committed the parties to consult with the First Nations community to enable meaningful participation in economic activity. PWGSC was assigned responsibility for liaison with First Nations. Relevant discussions with local First Nations communities began within three months of signing the Memorandum of Agreement. By late 2006, significant progress had been made to ensure First Nations involvement in the project; however, direct economic benefits have been delayed.

In October 2005, PWGSC, Sydney Tar Ponds Agency, and Mi'kmaq First Nations of Cape Breton entered into a protocol agreement. The parties agreed to engage in regular meetings to discuss opportunities for First Nations communities of Cape Breton to realize economic benefits through meaningful participation in the project. As a first step in implementing the agreement, the contract for remediation of a cooling pond was identified as an Aboriginal set-aside, which limits bidding to firms with majority First Nations' ownership and control. Terms and conditions of the provincial set-aside were patterned after federal requirements. This was an historic event, as it was the first time the government of Nova Scotia had tendered a contract as an Aboriginal set-aside.

The contract for the cooling pond was tendered as planned. However, it was subsequently cancelled due to consideration of certain recommendations from the environmental review panel. The cooling pond contract is scheduled to be re-tendered in 2007, again under the Aboriginal set-aside.

A First Nations representative was interviewed and expressed pleasure with the level of engagement during the Interim Cost-Share Agreement phase of the project. Specifically:

• The protocol agreement signed in October 2005 is viewed positively.
• Membertou, representing First Nations of Cape Breton, has had a good working relationship with PWGSC representatives.
• First Nations have thus far been engaged in the project to the right degree. And PWGSC has remained very accessible to First Nations' stakeholders throughout the process, and departmental representatives have kept them well informed.

5.1.4 Key Management and Control Instruments Implemented
Independent Engineer Providing Assurance to Project Partners

A key control instrument implemented during the Interim Cost-Share Agreement was the appointment of an Independent Engineer to monitor technical and financial aspects of the project and to report to the Project Management Committee on matters such as costs incurred, cost to complete, agreement/contract compliance, and matters related to engineering design and construction oversight. The federal and provincial partners along with the Sydney Tar Ponds Agency signed the Independent Engineer agreement on October 12, 2005. The Independent Engineer has been fulfilling his roles and responsibilities as they are understood by the federal partner and envisioned in the MOU and Interim Cost-Share Agreement. The work of the Independent Engineer has had a direct and positive impact on the quality of Sydney Tar Ponds Agency management (scheduling, financial, and activity reporting) and on achieving the preventative works discussed in the previous section.

Financial Controls Established but Payment Mechanism has not Been Timely

The Interim Cost-Share Agreement authorized the Project Management Committee to amend the budget as required to accomplish the objectives of the Agreement, so long as the total expenditures did not exceed $42M for the three-year period (2004–05 through 2006–07). Estimated project costs were revised by the Project Management Committee in an amendment to the Interim Cost-Share Agreement in September 2005. The Project Management Committee also approved annual budgets in September 2005, which actually covered fiscal years 2004–05 and 2005–06, and in June 2006 for fiscal year 2006–07. The Sydney Tar Ponds Agency reports project costs incurred and costs to complete to the Project Management Committee on a quarterly and annual basis. These reports flow through the Independent Engineer before receiving formal acceptance by the Project Management Committee.

A payment mechanism has been developed by which Sydney Tar Ponds Agency payment requests require Independent Engineer certification prior to being forwarded to the province. This certification is supposed to precede advancement of federal and provincial funding for the next period of work. Significant delays of many months were experienced during the Independent Engineer payment certification process when the Independent Engineer was not satisfied with the information received to substantiate Sydney Tar Ponds Agency payment requests. There have been disagreements between the Independent Engineer and the Sydney Tar Ponds Agency regarding the level of financial materiality of Independent Engineer requests. However, improvements with the timeliness of the payment application process have been observed in the latter part of 2006.

Committees Established; Mandates Redefined to Improve Effectiveness

The Operational Advisory Committee and the Technical Working Group were established by the Project Management Committee to provide it and/or the Sydney Tar Ponds Agency with advice and support. Both committees are currently in transition, one with revised terms of reference, the other being phased out and replaced with one with a stronger focus on environmental aspects of the project.

The Operational Advisory Committee provides functional direction and advice to senior project management and operational management for the project. The committee provides a forum for discussion and information exchange between PWGSC, the Nova Scotia Department of Transportation and Public Works, the Independent Engineer, and Sydney Tar Ponds Agency. Initially the committee's responsibilities were interpreted to include some decision-making powers. However, a revised terms of reference explicitly states that the committee will not function as a decision-making body.

The Technical Working Group is currently the technical/regulatory advisory committee for the project. Its role has been to provide specific technical and regulatory advice and interpretation on project-related issues. The Technical Working Group has not been very active; there are currently plans to disband the Group and replace it with an environmental management committee. The proposed role of the new committee is to ensure that the project respects the governments' responses to the Joint Review Panel's recommendations. The committee is expected to have a focus on successful environmental management of the project. Membership is expected to include key regulators and players. However, in contrast to the Technical Working Group, it is not expected that the implementing agency will be a committee member.

Performance Management Focused

A Results-based Management and Accountability Framework provides managers with a concise statement or road map to plan, monitor, evaluate and report on the results throughout the lifecycle of a program, policy or initiative. An Results-based Management and Accountability Framework for the Sydney Tar Ponds and Coke Ovens Sites Remediation Project was developed in 2005. It has identified five key issues on which performance measurement activities need to be focused:

1. Procurement (regular contracting and Aboriginal set-aside);
2. Risk management;
3. Project delivery;
4. Adherence to environmental protection and standards; and
5. Communications.

The Performance Management Plan component lists potential indicators and related targets, data sources for monitoring, and who is responsible for reporting on performance and when for each of these issues.

Some of the reporting requirements outlined in the Plan have been or are in the process of being met. For example, the Independent Engineer monthly reports, and the initial audit of the Aboriginal Procurement Strategy indicate that performance criteria are being met (the first audit of the Aboriginal Procurement Strategy) or note changes or corrections that are being made to ensure that these criteria are being met (e.g., risk assessment section in Independent Engineer monthly reports). In other areas where performance is of concern (e.g., communications), there is insufficient evidence to show that Sydney Tar Ponds Agency has maintained meaningful administrative records and files. Yet these are essential sources of information for determining whether desired performance levels have been attained during the Interim Cost-Share Agreement phase of the project.

Quality Assurance/Quality Control Draft Plan in Place

Individual preventative works projects have had their own QA/QC protocols. However, the implementing agency has yet to develop an acceptable Master QA/QC Plan, which is usually thought to be absolutely critical in a project of this type, significance, and complexity. The Independent Engineer recently reported, however, that Earth Tech ² has developed a draft Quality Management Plan for the remediation phase of the project.

Risk Assessment Strategy To Be Developed

The Risk-based Audit Framework incorporates the requirement for a risk assessment strategy for projects of this complexity and materiality. The Risk-based Audit Framework provides managers with a systematic approach to risk management that is to be used by project managers (both governance and implementation) on an ongoing basis to identify, prioritize, and mitigate risks. Risk management is fundamentally a management responsibility. The intent is for the Sydney Tar Ponds Agency to develop its own risk assessment strategy that reflects those outlined in the Risk-based Audit Framework.

At this point, the Sydney Tar Ponds Agency does not have a well-developed risk assessment/risk management protocol in place. Rather, the Independent Engineer reports on the status of each risk category as identified in the Risk-based Audit Framework, based on his understanding of what is happening and not on the results of Sydney Tar Ponds Agency's own risk assessment activity. Earth Tech, however, has included risk management as a component of their proposal. They plan to work with the Sydney Tar Ponds Agency to identify all potential risks that may impact design, construction, commissioning, and long-term performance of the intended remediation works. The contractor plans to work with the Sydney Tar Ponds Agency to develop strategies for risk mitigation that will form the core of a Risk Management Plan.

Audit and Evaluation Plans in Place

A list of required audits, evaluations and other reports is found in the Risk-based Audit Framework.

Each type of audit or report focuses on the risk issues appropriate to it; e.g., the Financial Audit of External Suppliers is expected to address management risk, financial management, and project oversight risk. The Independent Engineer has provided an assessment of risk management issues and project delivery in his monthly reports since these were first submitted in the fall of 2005. Sydney Tar Ponds Agency arranged for the first audit of the Aboriginal set-aside procurement process (the audit has been completed), and a procurement audit (recipient) is apparently underway.

This evaluation is the first of three proposed for the project. Refer to section 1.0 of this report for a list of proposed evaluations and their anticipated completion dates.

Work Breakdown Structure yet to be Developed

A work breakdown structure is a project management organizing and management control tool that defines project tasks and activities in hierarchical fashion, beginning with the highest-level activities and working downward to individual tasks and components. It is normally developed as part of project planning and is usually in place before work begins, although it will change as needed, as the project progresses. Even though the project was already underway, the work breakdown structure was not in place until late December 2005 or January 2006, following appointment of a Director of Corporate Services by Sydney Tar Ponds Agency.

The Independent Engineer assesses the status of each item in the work breakdown structure with respect to Memorandum of Agreement requirements on a monthly basis. The Sydney Tar Ponds Agency is expected to update the work breakdown structure as required and report any changes to the Independent Engineer.

Sydney Tar Ponds Agency officials expect that an effective work breakdown structure will be in place before work starts under the next agreement. A detailed project schedule was included in the Earth Tech proposal.

5.2 Success

5.2.1 Environmental Assessment Efficient and Effective

The potential for the project to fail at the environmental assessment phase was identified as a significant risk to the Sydney Tar Ponds and Coke Ovens Sites Remediation Project. There was the potential that the environmental assessment could affect the project's scope or scale; that there could be changes to regulatory standards; or that special interest groups could sway public opinion away from options proposed for the cleanup.

Under the Canadian Environmental Assessment Act, there are three main types of assessment: screening, comprehensive study, and panel review, with the panel review being the most rigorous form of review possible. Environmental groups, including the Sierra Club of Canada, had been vocal in calling for a full panel review of the tar ponds cleanup. However, other local community groups lobbied the federal government to stick with the comprehensive study approach, which was thought to be more efficient. There was concern that the review panel approach would unnecessarily delay the cleanup effort.

In May 2005, on the recommendation from the Minister of Public Works and Government Services Canada, the Environment Minister ordered a full panel review of the cleanup plan. Unusually tight timelines were set: the Minister ordered the panel to report by June 30, 2006. Nova Scotia, after initially preferring the comprehensive study approach, agreed to participate, and in September 2005, the two governments named three people to conduct the Joint Panel Review.

PWGSC led federal government efforts in advance of, during, and following the Joint Review Panel public hearings. An interdepartmental discussion group was created to inform and coordinate the federal players. Interviews conducted with participants from other federal government departments revealed that PWGSC was well organized and prepared. The department was committed to its role and was effective as a leader throughout the environmental assessment. The process on the whole ran very efficiently.

However, these same participants perceived PWGSC to be at times most concerned with moving the process forward. As a result, they expressed concern that there was little space for compromise or flexibility. During interviews conducted for the evaluation, subject matter experts from other government departments indicated that their comments were not always well received when they potentially threatened the schedule. Notwithstanding this criticism, participants acknowledged that timelines were tight and there was high visibility and pressure. In the end, significant issues raised by the other government department participants were addressed. Disagreements that arose were resolved and did not jeopardize the environmental assessment process.

The Joint Panel reviewed the environmental impact statement as well as written comments submitted by government agencies and members of the public. The panel also heard presentations from the project proponent, government departments, and community stakeholders during public hearings held during April and May 2006. A report outlining the panel's findings and recommendations was released on July 13, 2006, a mere 10 months following appointment of the panel members. The federal and provincial governments developed responses to the 55 recommendations and were expected to have had their responses approved by the end of December 2006.

5.2.2 Major Site Preparation Projects Completed On Time, On Budget

Three projects designed to prepare various aspects of the sites for the remediation phase, identified as preventative works in the Interim Cost-Share Agreement, have overall been completed on time and within the cost estimates. A fourth project, the remediation of an engineered cooling pond on the site, has been deferred until 2007. The completed projects include:

• Construction of a barrier at the mouth of the North Pond (Battery Point);
• Realignment of the Coke Ovens Brook;
• Relocation of a water main serving the Whitney Pier area.

Battery Point Barrier

The description in the Interim Cost-Share Agreement of the barrier to be built at the mouth of the North Pond described a relatively impermeable structure, stretching across the mouth of the North Tar Pond from Battery Point on the western side to SYSCO lands on the east. The design was subsequently changed from the initial concept to one that would have a 50-metre gap near the midpoint to accommodate water flowing through the Muggah Creek estuary. The purpose of the barrier is to protect contaminated sediment that will be capped in the remediation phase from ice forces, waves, storm surges, etc. from Sydney Harbour during and after construction.

The actual cost of the project was projected to be $2.1 million, including engineering design and construction. The projected total cost for the project represents only 46% of the cost estimate included in the Interim Cost-Share Agreement amendment approved by the Project Management Committee in September 2005. The cost saving is attributable to the design change and a favourable construction market that resulted in contractor bids being lower than expected. The contractor issued a certificate of substantial completion on August 30, 2006, although the Independent Engineer has outstanding quality assurance/quality control (QA/QC) concerns that need to be addressed prior to certifying completion. The Interim Cost-Share Agreement identifies a completion certificate signed by the Independent Engineer as the indicator of successful completion.

Coke Oven Brook Realignment

Coke Oven Brook was receiving contaminated groundwater from the Sydney Landfill and the Coke Ovens site. This contaminated groundwater flows down Coke Oven Brook into the Tar Ponds and ultimately into Sydney Harbour. The purpose of the realignment was to redirect clean surface water away from the contaminated areas and provide access to these areas for remediation purposes.

The actual cost to complete the Brook realignment, including design and construction, is projected to be $7.8 million. This figure represents 105% of the cost estimate identified in the Interim Cost-Share Agreement amendment. A certificate of substantial completion has not been received from the Sydney Tar Ponds Agency. However, the Independent Engineer has stated in the October report that project completion is expected in December 2006. The minor slippage is not expected to have an impact on the Sydney Tar Ponds and Coke Ovens Sites Remediation Project's critical path.

Victoria Road Water Main

The new Victoria Road water main was constructed to replace the existing water main running through the Coke Ovens site. As the new water main no longer flows through the site, the potential for outflow of contaminants via the water main has been mitigated. The actual cost to complete the water main project, including design and construction, was $622,912, slightly under the cost estimate in the Interim Cost-Share Agreement amendment. A certificate of substantial completion was issued on November 30, 2005. The Project Management Committee agreed to waive the requirement of a completion certificate from the Independent Engineer due to an issue beyond the control of the implementing agency.

Cooling Pond Remediation

The Cooling Pond preventative works project has been postponed. This postponement is not due to lack of budget or inability to deliver on time. Rather, the project is delayed to enable changes based on certain recommendations from the Joint Review Panel. The panel made several recommendations about air monitoring, cap design, and solidification and stabilization, which is the main technology that will be used to clean up the cooling pond. In an August 2006 press release, an Sydney Tar Ponds Agency representative stated that the cooling pond marked the first opportunity to use solidification and stabilization, and in light of the panel's recommendations, Sydney Tar Ponds Agency was going to take its time to make sure it “got it right”. The cost estimate for the project, as per the Interim Cost-Share Agreement amendment, was $4.4 million. The Sydney Tar Ponds Agency estimates that it will have spent $387,000 on the project by March 31, 2007, though the majority of project costs have yet to be incurred. The Independent Engineer believes that the delay in the cooling pond does not pose a risk to the Sydney Tar Ponds and Coke Ovens Sites Remediation Project's critical path.

5.3 Cost-Effectiveness

5.3.1 Projects Implemented Efficiently

The preliminary phase of the Sydney Tar Ponds and Coke Ovens Sites Remediation Project under the Interim Cost-Share Agreement has been managed in a cost-effective manner. As indicated through the design/implementation and success findings, the building blocks are largely in place to enable the project to succeed during the remediation phase. Project management have been able to accomplish their objectives while remaining within set budget limits.

Canada and Nova Scotia committed $42 million over the fiscal years 2004–2005, 2005–2006, and 2006–2007 to complete the works projects of the Interim Cost-Share Agreement. The Agreement also granted the Project Management Committee authority to amend the budget as necessary but not to exceed the total commitment over the three-year period. The Project Management Committee has approved extensions to project time lines for the majority of works projects in Amendment 1 to the Interim Cost-Share Agreement. All projects were scheduled for completion prior to March 31, 2007.

Spending under the Interim Cost-Share Agreement is forecast to be within allowable amounts. The estimated cost to complete preliminary and preventative works projects and other eligible costs is $31.2 million, well below the $42 million permitted. However, the Cooling Pond project with an estimated cost of $4.4 million was not completed as intended. The delay in the Cooling Pond project was not a result of inefficiencies in the management or implementation of the Interim Cost-Share Agreement.

The Battery Point Coffer Dam was completed significantly under budget. Spending on the cofferdam design and construction are projected to be $2.1 million or 46% of the amount allocated in the Interim Cost-Share Agreement amendment cost estimate. These cost savings are attributable to changes from the original design as well as favourable economic conditions when the project was tendered.

PWGSC Operating Costs for project oversight activities are accounted for separately from project costs paid to the Province of Nova Scotia as per the Interim Cost-Share Agreement. These operating costs, which flow through the PWGSC Real Property Services revolving fund are projected to be only 70% of the approved amount.

6.0 Conclusions

6.1 Design and Implementation

The project governance structure comprises a project management committee with senior representatives from the lead federal and provincial departments who oversee an implementing agency created by the Province of Nova Scotia specifically for this purpose. This governance structure is sound. The governance structure is further strengthened by the involvement of an Independent Engineer that provides the Project Management Committee with unbiased third-party assessments of design and implementation activities and ensures the Memorandum of Agreement is being respected.

Overall, the governance framework has been implemented effectively. However, the role of the province's lead department—as PWGSC's counterpart—needs to be strengthened and distinguished from the role of the implementing agency to ensure the governance model is respected.

The Sydney Tar Ponds Agency has been established, and most key staff positions have been filled, though staffing of the Agency was slow to happen. A large $30 million contract has been awarded for detailed engineering design and construction supervision that could have an impact on the role of the Sydney Tar Ponds Agency during the next phase of the project.

Constructive discussions with First Nations communities have resulted in a protocol agreement that is designed to enable Cape Breton First Nations to benefit economically from the remediation efforts. The environmental management and health and safety planning conducted has not only prevented significant incidents from occurring, but they have also laid the foundation for continued work that will be conducted for the remediation phase of the project.

6.2 Success

Significant milestones have been achieved for the project under the Interim Cost-Share Agreement. The federal and provincial partners came together and agreed to participate in a Joint Review Panel environmental assessment for the project. This critical step for the project was run very efficiently: the Panel report was released a mere 10 months after naming the Panel members. PWGSC played a significant lead role for the environmental assessment process. Other federal participants at times viewed PWGSC to be over-zealous in driving the agenda forward to meet the tight timelines but agreed that significant issues were properly addressed.

The planned projects, designed to prepare the sites for full remediation, have largely been completed, with the exception of the cooling pond. The cooling project has been deferred until 2007 to enable an adequate response to certain recommendations from the Joint Review Panel. The delay is not expected to affect the remediation phase schedule.

Significant efforts went into garnering citizen support for the project. Awareness of the project has improved; as well there has been some increase in the level of trust and confidence with the cleanup effort.

PWGSC is not the implementing agency for Sydney Tar Ponds and Coke Ovens Sites Remediation Project, but the department's history as the implementer of complex projects such as this has positioned it well to provide adequate oversight to this project.

6.3 Cost Effectiveness

The work conducted in this phase of the project, under the Interim Cost-Share Agreement, has been done so in a cost effective manner. The planned outcomes have overall been achieved, and they have been achieved with spending projected to be $31.2M. One major project with a cost estimate of $4.4M is outstanding. But this is well below the $42M budgeted for the Interim Cost-Share Agreement.

The preliminary works, for financial reporting purposes, include the Sydney Tar Ponds Agency and the environmental assessment. The total projected costs of these preliminary works are on budget at $9.65M. The total projected costs for the three completed preventative works projects, designed to prepare the site for full remediation, are $10.9M, which is below the cost estimate of $12.7M approved by the Project Management Committee in September 2005.

PWGSC officials have also been able to function effectively while spending less than the amount approved by Treasury Board for operating costs. Projected PWGSC operating costs of $6.1M represent only 70% of the $8.7M that was approved.

7.0 Recommendations

Based on the key findings and conclusions contained in this report, Audit and Evaluation Branch recommends that the Regional Director General, Atlantic Region:

1. Work with PWGSC's provincial counterparts on the Project Management Committee to clearly delineate the province's dual roles and garner full participation of the Nova Scotia Department of Transportation and Public Works in providing project oversight.
2. In partnership with the Nova Scotia Department of Transportation and Public Works, review the terms of reference and operation of committees established to ensure they are consistent with the governance framework.
3. In partnership with the Nova Scotia Department of Transportation and Public Works, examine the project payment mechanism to identify adjustments to enable the timely processing of payment applications.

Appendix A - Summary of Findings

Design and Implementation

1. Is the governance structure effective and appropriate to achieve the expectations of the Interim Cost-Share Agreement? Are there changes needed for the next cost-share agreement?

• Governance structures refer to the processes and structures through which decision-making authority is exercised for the Sydney Tar Ponds and Coke Ovens Sites Remediation Project.
• PWGSC and the Nova Scotia Department of Transportation and Public Works have come together to jointly form a Project Management Committee to oversee the project. The Province of Nova Scotia has created the Sydney Tar Ponds Agency as a Special Operating Agency to implement and manage the project on a day-to-day basis.
• PWGSC, the Nova Scotia Department of Transportation and Public Works, and Sydney Tar Ponds Agency signed an agreement with an Independent Engineer (Independent Engineer) to provide the with an unbiased third-party assessment of design and implementation activities and assurance that the Memorandum of Agreement has been respected.
• PWGSC has established an office and dedicated a team of experienced resources to satisfy the department's oversight responsibilities. In contrast, the Nova Scotia Department of Transportation and Public Works has had only one person assigned full-time to the project.
• The governance structure is appropriate; however, its effectiveness has been hindered due to the imbalance between the federal and provincial partners.

2. Is the set of management and control instruments, including the Preliminary Works, effective and appropriate to achieve the expectations of the Interim Cost-Share Agreement? Are there changes needed for the next cost-share agreement?

• Sydney Tar Ponds Agency
  The province established a single purpose special operating agency (SOA) to implement the project on September 30, 2004, to be called the Sydney Tar Ponds Agency.

  The Sydney Tar Ponds Agency engaged core staff (A/CEO, Director Corporate Services, Director Communications and Community Relations, Director Engineering, Director Environmental Services, and Economic Benefits Coordinator) after considerable delay, in October 2005.

• Environmental Assessment Review
  Environmental assessments of each of the four preventative works projects were completed through a screening process.

  A full review panel environmental assessment was completed for the project. The panel released their report, including 55 recommendations, to the federal and provincial governments on July 13, 2006.

• Independent Engineer
  The Independent Engineer (IE) was appointed, and the co-chairs of the Project Management Committee signed the agreement with the designated firm on October 12, 2005.

  The Independent Engineer has been fulfilling his roles and responsibilities as they are understood by the federal government (PWGSC) and as envisioned in the MOU and Interim Cost-Share Agreement.

  The Independent Engineer has had a direct and positive impact on the quality of Sydney Tar Ponds Agency management (scheduling, financial, and activity reporting) and on the quality and achievement of preventative works, all of which have been completed on time and within budget (with the exception of the Cooling Pond project).

• Project Description
  The Project Description is a prerequisite to commencing an environmental assessment process. It details the areas affected by the project, the location of activities, and environmental factors that are to be addressed.

  The Project Description was completed in December 2004 as per the Interim Cost-Share Agreement Schedule.

• Work Breakdown Structure
  The WBS was not put in place by the Sydney Tar Ponds Agency until late December 2005 or January 2006.

  The Independent Engineer assesses the status of each item in the WBS with respect to Memorandum of Agreement requirements on a monthly basis. The Sydney Tar Ponds Agency is expected to update the WBS as required and report any changes to the Independent Engineer.

  Sydney Tar Ponds Agency officials expect that an effective WBS will be in place before work starts under the next agreement. A detailed project schedule was included in the Earth Tech proposal.

• Cost-to-Complete Mechanism
  The Sydney Tar Ponds Agency has established a cost-to-complete mechanism that is now in place for quarterly and annual reporting to Project Management Committee.

  The Independent Engineer has been unwilling to accept cost-to-complete data from Sydney Tar Ponds Agency from time to time. The reasons the information from the Sydney Tar Ponds Agency has not been satisfactory need to be clarified.

• Risk Assessment Strategy
  A Risk-based Audit Framework (RBAF) was completed for the project in July 2005. The RBAF incorporates the requirement for a risk assessment strategy for projects of this complexity and materiality.

  The RBAF provides managers with a systematic approach to risk management that is expected to be used by project managers (governance and implementation) on an ongoing basis to identify, prioritize, and mitigate risks.

  The RBAF includes a risk assessment and management summary that describes the risk-based methodology to be used; implementation steps to be taken; the parties involved in the risk assessment and management process; the impact and likelihood of potential risks; and a summary of key risks and incremental risk mitigation strategies.

• Development of a Risk-based Management and Accountability Framework (RMAF)
  The RMAF was completed in August 2005. It is a blueprint for managers to plan, measure, evaluate, and report on results throughout the life cycle of the project. It represents a written commitment to results measurement and reporting.

• Performance Management
  The RMAF has identified five key issues on which performance measurement activities need to be focused: procurement (regular contracting and Aboriginal set-aside); risk management; project delivery; adherence to environmental protection and standards; and communications.

  The Performance Management Plan RMAF component lists potential indicators and related targets, data sources for monitoring, and the individual responsible for reporting on performance and when for each of these issues.

  Some of the reporting requirements outlined in the Plan have been or are in the process of being met (e.g., Independent Engineer monthly reports, the initial audit of the Aboriginal Procurement Strategy) and either indicate that performance criteria are being met (the first audit of the Aboriginal Procurement Strategy) or note changes or corrections that are being made to ensure that these criteria are being met (e.g., Risk assessment section in Independent Engineer monthly reports).

  In other areas where performance is of concern (e.g., communications), there is insufficient evidence to show that Sydney Tar Ponds Agency has maintained meaningful administrative records and files. Yet these are essential sources of information for determining whether desired performance levels have been attained during the Interim Cost-Share Agreement phase of the project.

• Quality Assurance / Quality Control
  Sydney Tar Ponds Agency has yet to develop an acceptable Master Quality Assurance/Quality Control Plan – something that is usually thought to be absolutely critical in a project of this type, complexity, and significance.

  The Earth Tech proposal contains a Quality Management Plan, including a QA/QC component. The Plan was based on the International Standard ISO 9001:2000 and has been adapted to suit this specific project.

• Project Management Framework
  The Project Management Framework was based on principles that are applied to the management of Major Crown Projects and is expected to delineate these roles, responsibilities, and accountabilities in a clear and understandable manner.

  The Framework is updated as needed; the most recent amendment is dated June 2006.

  Because of the number and complexity of the actors in the project, there is a danger that roles, responsibilities, and accountabilities might become confused, and that, in turn, could result in duplication or gaps in oversight. The framework is seen as a mitigation strategy to offset this operational (governance) risk.

  The Project Management Framework is not clear about roles, responsibilities, and accountabilities, particularly with respect to the Nova Scotia Department of Transportation and Public Works versus those of the Sydney Tar Ponds Agency.

• Audit and Evaluation
  A list of required audits, evaluations, and other reports is found in the RBAF.

  Each type of audit or report focuses on the risk issues appropriate to it; e.g., the Financial Audit of External Suppliers is expected to address management risk, financial management, and project oversight risk.

  The Independent Engineer has provided an assessment of risk management issues and project delivery in his monthly reports since these were first submitted in the fall of 2005.

  Sydney Tar Ponds Agency arranged for the first audit of the Aboriginal set-aside procurement process (audit completed), and a procurement audit (recipient) is apparently underway.

  There are three proposed evaluations for the project:

  1. This evaluation of Interim Cost-Share Agreement, to be completed by January 2007;
  2. A mid-term evaluation of the cost-share agreement to report on early results and provide additional assurances that the project is proceeding as planned, to be completed March 2010; and
  3. A summative evaluation, which is to focus on intermediate outcomes and report on evidence of long-term outcomes, to be completed March 2014.

3. To what extent has the health and safety of workers and the general public been protected during the Interim Cost-Share Agreement?

• Overall, the health and safety of workers and the general public is being adequately protected.
• A Master Health and Safety Plan (MHASP) has been developed by the Sydney Tar Ponds Agency to ensure adequate precautions are taken for the protection of workers and the general public. The Independent Engineer has reviewed, commented on, and, following modifications by Sydney Tar Ponds Agency, accepted the plan.
• A review of project files indicates that the MHASP has been implemented effectively and thoroughly and is being respected by all involved parties (Sydney Tar Ponds Agency, Independent Engineer, contractors, subcontractors, etc.). Reporting and monitoring activities appear sound.
• To date, health and safety incidents have been infrequent and very minor (no persons have been injured). Nonetheless, prescribed remedial action has followed all reported incidents.
• There was no evidence of new concerns by the public regarding health and safety issues in project files reviewed. Likewise, major concerns by contractors and unions were not identified.
• A new MHASP that will cover all phases and elements of the project is planned for development.

4. To what extent have the contaminants entering or leaving the site been controlled?

• An Environmental Management Plan (EMP) for the project and supplementary EMPs for individual project aspects have been developed. The EMP will not be fully implemented until the remediation phase of the project.
• Some EMP reporting requirements have not been followed and changes have not being documented in revised EMPs. The Independent Engineer recognized this in August 2006 as an operational risk with a high risk factor. The risk was later reduced to a medium level following action taken by the Sydney Tar Ponds Agency.
• With respect to implementing the EMP, there were occasions of differences in interpretation between the Sydney Tar Ponds Agency and the Independent Engineer. The Sydney Tar Ponds Agency felt that the Independent Engineer was asking for too strict an adherence to the EMP, and believed instead that it should be more liberally interpreted to accommodate operational circumstances.
• During construction there has been regular air-quality monitoring. On a few occasions, air emissions standards were exceeded, prompting work stoppage orders and public advisories being issued.

5. To what extent has the public been engaged during the term of the Interim Cost-Share Agreement?

• Formally (environmental assessment consultations and Community Liaison Committee)?
• Informally (e.g., site visits, presentations, educational events, etc.)?

• A communication plan was developed in July 2005 and approved by the Project Management Committee in September 2005. The plan provides details on current and proposed communications activities and key messages to be conveyed to the public.
• Overall, the plan has been implemented as intended. Wide and varied methods have been used for communicating with the public, including, but not limited to, quarterly newsletters, press releases, an Internet Web site, site tours, school outreach, and attendance at community meetings.
• The public was invited to provide comments to the Joint Review Panel. There were 55 public comments received from a variety of organizations and individuals. The Sydney Tar Ponds Agency responded to all public comments via the panel.
• A Community Liaison Committee (Community Liaison Committee) composed of representatives from community organizations and the Cape Breton Regional Municipality was formed to facilitate two-way flow of information between the Sydney Tar Ponds Agency and the constituent organizations. Membertou First Nations has been invited to join the committee, but as yet they have not accepted. The committee holds formal meetings on a monthly basis and produces meeting records that are made public on the Internet Web site.
• In interviews conducted with Community Liaison Committee members and officials from Sydney Tar Ponds Agency, there was agreement that the committee is working as intended and should be maintained through the remediation phase of the cleanup. A senior representative from PWGSC also believes that the committee should be continued, but members should be made more accountable to themselves and the organizations they represent.
• The Joint Review Panel supported the continuation of the Community Liaison Committee; however, it did recommend modifications in the committee's terms of reference. Changes were recommended to ensure that the appointment process is open and transparent and that members effectively relate and report back to the people they represent.

6. To what extent have First Nations been engaged during the term of the Interim Cost-Share Agreement?

• The Memorandum of Agreement and the Interim Cost-Share Agreement committed Nova Scotia and Canada to provide First Nations with the opportunity to benefit economically from meaningful and direct participation in the clean-up project.
• In an interview with a First Nations' representative, there was general pleasure expressed with the level of engagement during the Interim Cost-Share Agreement phase of the project:
  • The protocol agreement, dated October 28, 2005, entered into by PWGSC, Sydney Tar Ponds Agency, and Mi'kmaq First Nations of Cape Breton is a good one;
  • Membertou, representing First Nations of Cape Breton, has had a good working relationship with PWGSC representatives;
  • First Nations have thus far been engaged in the project to the right degree; and
  • PWGSC has remained very accessible to First Nations stakeholders throughout the process and has kept them well informed.
• For the first time, the government of Nova Scotia tendered a project as an Aboriginal set-aside. The provincial Aboriginal set-aside was patterned after federal requirements. It limited bidding on the Cooling Pond preventative works project to firms with First Nations' ownership and control.
• First Nations have not yet begun to benefit economically from direct participation in the cooling pond preventative works project as had been planned. The contract was tendered through the Aboriginal set-aside. However, it was subsequently cancelled due to consideration of certain recommendations from the environmental review panel. The cooling pond contract is scheduled to be re-tendered in 2007, again under the Aboriginal set-aside.

7. Are there any lessons learned for the next cost-share agreement? For other remediation projects? For other cost-share agreements?

• Critical to the success of a project such as this is the dedication and personal commitment of the people involved. There is a recognition that remediation of these sites will finally lift a stigma that has been felt in the local community for many years.
• Establishing a local presence and becoming part of the local community has enabled PWGSC personnel to be more in touch with local sentiment and responsive to project concerns.
• Despite having PWGSC, the Nova Scotia Department of Transportation and Public Works and Sydney Tar Ponds Agency all sign onto the Independent Engineer agreement, there were differences of opinion surrounding the role of the Independent Engineer for the project. The parties recognized that further clarification of the complex role was required and have been working to clarify the Independent Engineer–Sydney Tar Ponds Agency relationship.
• Having a very robust Memorandum of Agreement has made discussions between the federal and provincial governments easier. This up-front preparation should continue to benefit the project into and throughout the remediation phase.

Success

8. Have the Interim Cost-Share Agreement Preventative Works projects been completed as planned?

• The relocation of the Whitney Pier Water Line and construction of the Battery Point Barrier projects are complete and were completed according to schedule.
• The realignment of the Coke Oven Brook is nearly complete, as per its initial project schedule.
• The Cooling Pond project has been re-scheduled for 2007 to allow for more testing and research.
• Overall projects have been completed within cost estimates.

9. To what extent has the on-site environmental quality improved during the Interim Cost-Share Agreement?

• Overall, the Interim Cost-Share Agreement preventative works projects were conducted to prepare various aspects of the sites for the remediation phase rather than achieve immediate an improvement in environmental quality. Therefore, there have been little data collected on changes in on-site contamination as a result of these projects.
• The new channels constructed for the Coke Oven Brook realignment project have been built with impervious engineered liners to prevent contaminated soil from reaching the water. In addition, these new channels have been diverted away from the landfill, which was a source of contamination. These channels will only carry water once the brook realignment project is complete. Until such time, no significant changes in contaminants will be observed.
• Early on in the process there was a design change for Battery Point cofferdam. What was constructed is not an impervious barrier. Rather, it has been designed to protect contaminated sediment that will be stabilized and capped in the remediation phase from ice forces, waves, storm surges, etc., from Sydney Harbour during and after construction.
• The new Victoria Road water main replaced the existing water main running through the Coke Ovens site. As the new water main no longer flows through the site, the potential for outflow of contaminates via the water main has been mitigated.
• The Cooling Pond, contaminated with residual hydrocarbons from years of steel making, poses a risk to migratory waterfowl. The project to control the contaminants in the cooling pond will not be completed until 2007; thus, no results are observable.
• More significant progress with on-site environmental quality can be expected with the remediation phase to be conducted under the next cost-share agreement.

10. Were environmental issues and risks mitigated during the Interim Cost-Share Agreement?

• An independent Joint Review Panel was appointed on September 19, 2005, to conduct an environmental assessment of the proposed project. The Panel held 17 days of public hearings in Sydney during April and May 2006. The Environmental Assessment Report was issued July 13, 2006, and included 55 recommendations to both levels of government.
• The Sydney Tar Ponds Agency provided the Joint Review Panel with timely responses to hundreds of questions included in 55 public comments, as well as 74 additional information requests from the joint review panel, in advance of the public hearings.
• The Review Panel recommendations were submitted to the Minister of the Environment for Canada and the Nova Scotia Minister of Environment and Labour. PWGSC has led the federal effort to develop responses to the recommendations directed at the federal government. The Province of Nova Scotia is doing likewise.

11. Was the role of PWGSC leading the federal effort contributing to its success?

• PWGSC led the federal government efforts in advance of, during, and following the Environmental Assessment (EA) public hearings. An interdepartmental discussion group made up of concerned departments was created to inform and coordinate the federal players.
• Interviews conducted with representatives from other federal government departments revealed that PWGSC was well organized and prepared. The department was committed to its role and was effective as the federal lead in the EA process. The process as a whole ran very efficiently.
• PWGSC was perceived at times to be most concerned with moving the process forward. As a result, there was not a lot of space for compromise or flexibility. Comments from subject matter experts from other federal departments were not always well received when they potentially threatened the schedule.
• Notwithstanding the point above, participants acknowledged that timelines were tight, and there was high visibility and pressure. Significant issues raised by the other government department participants were addressed.  Disagreements that arose were resolved and did not jeopardize the EA process.

12. What is the extent of citizen support for the cleanup efforts and the results achieved to date?

• The Sydney Tar Ponds Agency conducted a number of public opinion polls to obtain information on Cape Breton Regional Municipality residents' perceptions of the cleanup plan. Relevant questions were asked regarding citizens' awareness and understanding of the cleanup plan as well as their trust and confidence in the plan.
• In advance of the announcement of a review panel environmental assessment process in May 2005, there was a shift towards acceptance of a ‘go slow' approach that included more public consultation and an independent panel for the environmental assessment. However, the majority of respondents still held the ‘just get it done' opinion. 
• Overall, there have been positive trends related to citizens' awareness and understanding of the cleanup plan when later polls repeated similar or identical questions from earlier polling.
  • The number of respondents indicating an understanding for the cleanup plan increased by 15 points to 44% of respondents between Q2 (July – September) 2005 and Q1 (April – June) 2006.
  • Half of respondents questioned during Q3 (October – December) 2006 could name a particular project conducted under the Interim Cost-Share Agreement.
• Polling results show a marginally positive trend in citizens' trust and confidence in the cleanup plan.
  • In a poll conducted in Q1 (April – June) 2005, 25% of respondents had some or a great deal of trust and confidence in the cleanup plan. In polling conducted one year later, 30% agreed that they had trust and confidence in the plan.
  • In addition, the number of respondents with little or no trust and confidence in the cleanup plan had decreased by 26 points to 47%.
• By Q3 (October – December) 2006, 58% of respondents thought that some or a lot of the cleanup work had been conducted versus 42% who thought no cleanup work had been done.
• In Q1 (April – June) 2006, prior to the release of the report by the Joint Review Panel, 69% of residents surveyed indicated a hope that the environmental review panel would approve the plan so the Sydney Tar Ponds Agency could get on with the cleanup.

13. Have there been any unintended impacts to date as a result of the Interim Cost-Share Agreement?

• Tendering a contract as an Aboriginal set-aside by the Province of Nova Scotia represented an historic first for the province, potentially paving the way for similar arrangements for future provincial contracts in Nova Scotia or elsewhere in Canada.

Cost Effectiveness

14. To what extent has the Interim Cost-Share Agreement been managed and implemented efficiently?

• Were resources adequate?
• Could there have been savings, or could costs have been reduced without increasing risks disproportionately?

• Canada and Nova Scotia committed a total of $42 million over the fiscal years 2004–2005, 2005–2006, and 2006–2007 to complete the works projects of the Interim Cost-Share Agreement. The Project Management Committee was granted authority to amend the budget as necessary but not to exceed the total commitment over the three-year period.
• Project Management Committee approved extensions to project time lines for the majority of works projects in Amendment 1 to the Interim Cost-Share Agreement. All projects were scheduled for completion prior to March 31, 2007.
• Spending under the Interim Cost-Share Agreement is forecast to be within allowable amounts. The estimated cost to complete preliminary and preventative works projects and other eligible costs is $31.2 million, well below the $42 million permitted. However, the Cooling Pond project with an estimated cost of $4.4 million was not completed as intended. The delay in the Cooling Pond project was not a result of inefficiencies in the management or implementation of the Interim Cost-Share Agreement.
• The Battery Point Coffer Dam was completed significantly under budget. Spending on the cofferdam design and construction are projected to be $2.1 million or 46% of the amount allocated in the Interim Cost-Share Agreement amendment cost estimate. These cost savings are attributable to changes from the original design as well as favourable economic conditions when the project was tendered.
• PWGSC Operating Costs: Payments to RPS revolving fund for project oversight activities are projected to be 70% of the amount approved by Treasury Board.

Appendix B - Key Informant Interviews

PWGSC

• 1. Appleby, John, Environmental Evaluations Manager, PWGSC, Halifax, NS
• 2. Hilchey, Bruce, Lawyer, PWGSC, Halifax, NS
• 3. Jacquin, Hubert, Communications Manager, PWGSC, Halifax, NS
• 4. Kenny, Margaret, Director General, OGGO, PWGSC, Ottawa, ON
• 5. MacKinnon, Joan, Regional Director, PWGSC, Halifax, NS
• 6. McBain, John, Regional Director General, PWGSC, Halifax, NS
• 7. Powell, Brenda, Financial Analyst, PWGSC, Charlottetown, PEI
• 8. Swain, Ken, Regional Director, OGGO and – STP Project Director of PWGSC, Sydney, NS
• 9. Vallis, Randy, Senior Project Manager, PWGSC, Halifax, NS

The Nova Scotia Department of Transportation and Public Works

• 10. Campbell, Gary, Executive Director, NSTPW, Halifax, NS
• 11. Golden, John, Accounting Services Manager, NSTPW, Halifax, NS

Sydney Tar Ponds Agency

• 12. Baillie, Barbara, Director of Engineering Services, STPA, Sydney, NS
• 13. Donham, Parker, Director of Communications, STPA, Sydney, NS
• 14. Kaiser, Wilf, A/Director of Environmental Services, STPA, Sydney, NS
• 15. MacDonald, Roy, Risk & Schedule Manager, STPA, Sydney, NS
• 16. Potter, Frank, A/CEO, STPA, Sydney, NS
• 17. Stewart, Corrie, Director of Corporate Services, STPA, Sydney, NS

Other Government Departments

• 18. Benjamin, Cheryl, Senior Program Officer, CEAA, Halifax, NS
• 19. Campbell, Diane, Director, EC, Gatineau, QC
• 20. Baker, Jill, Environmental Assessment Officer, EC, Gatineau, QC
• 21. Chapman, Steve, Project Manager, CEAA, Ottawa, ON
• 22. Chard, Sharon, Regional Director, HC, Halifax, NS
• 23. Drake, Anne Marie, Environmental Engineer, Dartmouth, NS
• 24. McLean, Mark, Senior Environmental Analyst, DFO, Dartmouth, NS
• 25. McPherson, Terry, Inspector Specialist, Nova Scotia Environment and Labour, Sydney, NS
• 26. Michaud, Livain, Senior Environmental Assessment Officer, Ottawa, ON

First Nations

• 27. Bernard, Trevor, First Nations, Sydney, NS

Independent Engineer

• 28. Van Veen, Walter, Engineer, Conestoga Rovers & Associates, Sydney, NS

Community Interest Groups

• 29. Bailey, Bill, Professor, Cape Breton University, Sydney, NS
• 30. Green-MacIver, Karen, Cape Breton Black Employment Partnership, Sydney, NS
• 31. MacDonald, Keith, Cape Breton Partnership, Sydney, NS
• 32. Marchoccio, Bruno, Sierra Club, Sydney, NS
• 33. MacLeod, Dr. Alastair, Community Liaison Committee Chair, Sydney, NS
• 34. Van Nostrand, Steve, Sydney Chamber of Commerce, Sydney, NS

Footnote

¹Cape Breton Regional Municipality was formed in 1995 through an amalgamation of eight former municipalities: the City of Sydney and the Towns of Dominion, Glace Bay, New Waterford, North Sydney, Sydney Mines, Louisburg, and Cape Breton County.

² Earth Tech and its local partner CBCL Ltd. have been awarded a $30M contract to carry out detailed engineering for the cleanup and to supervise construction during the remediation phase of the project.