Language selection


Fixed-Wing Search and Rescue Aircraft Replacement Project - Independent Review of Evaluation Plans, Methods and Tools

November 21, 2013

The observations included in this report are limited to the scope described in Section 2. This report must be considered in its entirety; selecting specific words or portions of this report without an understanding of the review timeframe and coverage could result in misinterpretation of our observations.

Table of Contents


The following acronyms are used in this report:

Aircraft Performance Assessment Tool
Capability Delivery Risk Evaluation
Chief Financial Officer
Department of National Defence
Fixed Wing Search and Rescue
FWSAR Project
Fixed Wing SAR Aircraft Replacement Project
Industrial and Regional Benefits
In-Service Support
National Research Council
Office of the Auditor General
Public Works and Government Services Canada
Quality Assurance
Request for Proposal
Requirement Specification
Search and Rescue
Statement of Operational Requirements
Statement of Work

1. Background

Search and Rescue (SAR) in Canada is a shared responsibility among federal, provincial/territorial and municipal organizations, which requires ground, air and maritime capabilities, and must be able to accommodate the varying climates across the country. The Department of National Defence (DND) holds primary responsibility for providing aeronautical SAR services (search for downed aircraft), while the Canadian Coast Guard holds primary responsibility for maritime SAR services. DND's aeronautical SAR services are delivered by both rotary and fixed wing aircraft, each with a distinct purpose.

DND's Fixed Wing SAR (FWSAR) operations are presently carried out by aircraft that are nearing the end of their service life. DND has a fleet of 6 CC115 Buffalo that are 45 years old, as well as a fleet of 13 CC130H Hercules that are between 20 and 39 years old. As part of the Canada First Defence Strategy, the replacement of Canada's aging FWSAR aircraft has been identified as a Government priority.

The Fixed Wing SAR aircraft replacement (FWSAR) project (referred to in this report as “the FWSAR Project”) was initiated in 2002 to replace the aging fleet of FWSAR aircraft. In the fall of 2009, following industry consultation undertaken by the FWSAR Project, the Government of Canada engaged the National Research Council (NRC) to conduct an independent review of the FWSAR Project's Statement of Operational Requirements (SOR). In its report, the NRC recommended improvements to the technical requirements, as well as to the assumptions and constraints that were used to derive the requirements for the FWSAR Project.

Following the NRC's recommendations, the FWSAR Project was restarted with the development of a new SOR. With this restart, the Government of Canada aimed to shift greater focus to the required outcomes (capabilities) of the FWSAR aircraft, rather than the required characteristics of the aircraft product (or “platform”). The intent of this shift was to address NRC's recommendations and allow bidders to devise solutions that would best meet the required outcome. The increased focus on capability-based procurement has given rise to bid evaluations plans, methods, processes, decisions and tools that are new for DND aircraft acquisitions.

Delivery of the new SAR aircraft is currently planned for 2018. The Air Force has extended the operational lives of the Hercules and Buffalo aircraft. As highlighted in the Office of the Auditor General's (OAG) 2013 spring report, maintenance costs have risen and the availability of the FWSAR aircraft has been reduced due to maintenance of the aging fleet.

The replacement of FWSAR aircraft represents a multi-billion dollar acquisition for the Government of Canada. We understand that Public Works and Government Services Canada (PWGSC), Industry Canada and DND are working toward releasing the Request for Proposal (RFP) for the FWSAR Project in 2014 with expected delivery of the first new aircraft in 2018 and Full Operational Capability for the new aircraft in 2020. The FWSAR Project is seeking to promote best value for Canadians by considering the capital, operating and maintenance costs over the full life of the aircraft. The FWSAR Project's procurement approach also seeks to achieve industrial benefits.

2. Objective and scope

In July 2013, through a competitive procurement process, PWGSC contracted Orbis Risk Consulting Inc. to conduct an independent review between July 2013 and November 2013. Our review objective and scope are detailed in the following sections:

  1. Review objective;
  2. Review scope; and
  3. Scope exclusions.

2.1 Review objective

The objective was to perform an independent review of FWSAR Project evaluation plans, methods, processes, decisions and tools, which were at varied stages of development, and provide observations in relation to two of the Government of Canada's procurement objectives, that they be:

We considered the broader framework of FWSAR Project bid evaluation plans, methods, processes, decisions, and tools as they were being developed and, for a subset of these which were nearing completion, we examined them in greater depth.Footnote 1 The aspects examined in greater depth are described in section 2.2. Section 2.3 provides greater clarity on aspects specifically excluded from the scope of our review.

A review does not constitute an audit and consequently we did not seek to express an opinion on whether the FWSAR Project evaluation plans, methods, processes, decisions and tools are free of bias and will achieve best value. 

2.2 Review scope

Our review was conducted between 9 July 2013 and 21 November 2013 and considered activities, decisions and documents from the period of 1 April 2012 to 21 November 2013. Considering that bid evaluation plans, methods, processes, and tools were at varied stages of completion at the time of completing our review, the scope of our work was limited to the information as it existed when it was provided to the review team. We were asked to examine in greater depth, a subset of plans and tools that were further along in their development, which are listed below. Our review also considered, but did not examine in depth, the broad framework of plans and tools being developed by the Government of Canada to support bid evaluation for the FWSAR Project.

In recognition that the Government of Canada was evolving and refining various aspects of the bid evaluation plans, methods, processes, decisions and tools at the time of this report, coupled with the need for a timely independent review report, we were limited to reviewing information when it was provided to the review team and were reliant on DND, PWGSC and Industry Canada to provide us with current, relevant, complete and accurate information.

The plans and tools that were examined in greater depth included:

2.3 Scope exclusions

The following elements were not included in the scope of our independent review:

3. Approach

Our independent review consisted primarily of enquiry, analytical procedures, and discussion related to information supplied to us by representatives of DND, PWGSC and Industry Canada, as well as subject matter experts external to the Government of Canada. Over 750 documents were examined and 33 interviews conducted. We had no contact with known potential bidders throughout the independent review.

A review does not constitute an audit and consequently, our work was not intended to be exhaustive and we did not seek to express an opinion on whether the FWSAR Project evaluation plans, methods, processes, decisions and tools are free of bias and will achieve best value. Our independent review relied on the accuracy of information provided by DND, PWGSC and Industry Canada.

Our independent review was conducted in the following two phases:

This report comments on observations resulting from the two-phased review conducted between 9 July 2013 and 21 November 2013.

4. Limitations of this report

This report is provided for the sole and exclusive use and benefit of the Government of Canada to inform decisions pertaining to the design of bid evaluation methods, processes, and tools. No other person is authorized to rely on, use, copy, duplicate, reproduce or disseminate this report, in whole or in part and for any reason whatsoever, without the express prior written consent of Orbis Risk Consulting Inc. This report should be considered in its entirety; selecting specific portions of this report may result in the misinterpretation of the content. Any person using this report, other than the persons to whom it is addressed, does so entirely at its own risk. Orbis Risk Consulting Inc. assumes no responsibility or liability in connection with the reliance made by any party on this report or the observations and comments contained within this report. Orbis Risk Consulting Inc. reserves the right, but will not be obligated, to review or revise this report if deemed necessary by Orbis Risk Consulting Inc.

Our independent review and this resultant report are limited to the scope described in Section 2. This report should not be acted upon without performing further analysis. The Government of Canada is responsible for decisions made in respect of the FWSAR Project and such decisions will influence the realization of intended outcomes. The value ultimately achieved by the FWSAR Project and the extent to which the bid evaluation methods, processes and tools are unbiased will be impacted by future decisions of the Government of Canada.

5. Review observations

This section details our review observations.

Based on the work performed in our review, as set out in the scope described in section 2, nothing came to our attention that would cause us to believe that the FWSAR Project plans, methods, processes, decisions or tools are biased, where one bidder was inadvertently favoured over another. Also, we noted methods that are consistent with the objective of promoting a best value procurement approach. Our observations are presented under the following headings:

  1. Comments on review observations related to the unbiased procurement objective; and
  2. Comments on review observations related to the best value procurement objective.

5.1 Comments on review observations related to the unbiased procurement objective

Our observations related to the unbiased procurement objective are detailed under the following three sections:

  1. Engaging industry;
  2. Developing evaluation techniques; and
  3. Validating decisions.

5.1.1 Engaging Industry

Our review highlighted measures taken by the Government of Canada, which are consistent with the objective of managing risks related to bias, as follows:

As procedures for industry consultations are finalized and feedback is assessed and responded to, it is important that processes employed and decisions made be well preserved to position the Government of Canada to respond promptly to future scrutiny.

5.1.2 Developing Evaluation Techniques

We observed the following tools and techniques developed and applied by the Government of Canada to promote an unbiased procurement approach:

During our review, we observed that a discretionary clause was being developed for inclusion in the RFP to address clerical errors in the proposals of bidders. The discretionary clause is intended to be used as required by the Government of Canada to permit bidders to correct certain clerical errors in their proposals, thereby avoiding bidder disqualification and improving the likelihood of the procurement being successful. We observed that processes and protocols for administering the discretionary clause remained unclear at the time of writing our report. The Government of Canada could mitigate risks of perceived bias in its application of the discretionary clause by clearly documenting processes and protocols by which discretion will be exercised during bid evaluation.

Within the evaluation framework, the Government of Canada plans to employ new procurement methods for evaluating certain rated criteria, such as within the Capability Delivery Risk Evaluation (CDRE) and the Flight and Ground Test Assessment. During our review, we observed that processes and protocols for conducting on-site and field assessments were not yet finalized. The Government of Canada could mitigate the risks associated with bias during on-site and field assessments by establishing clear, documented protocols detailing the methods, tools and processes to be employed.

As the Government of Canada finalizes its bid evaluation plans, methods, processes and tools, continuing to strengthen records management, quality assurance and change management, including for the APAT and its input parameters, would position it to promptly respond to future scrutiny.

5.1.3 Validating Decisions

We observed the following decision validation methods applied by the Government of Canada to promote an unbiased procurement approach:

We found that, for many of the reviews performed by and for the FWSAR Project, no clearly defined terms of reference were readily available. Retaining clear records and evidence of how decisions were validated through third-party reviews could help the Government of Canada manage the risks of knowledge loss and incomplete quality control.

While the FWSAR Project established specific wording to articulate the project objective in key documents, we noted that this wording was not consistently used across documentation and in the draft RFP of 30 August 2013. The Government of Canada could mitigate risks of misinterpretation and potential perceived bias by performing end-to-end quality assurance (QA) for the final documentation prior to its release to industry.

5.2 Comments on review observations related to the best value procurement objective

In framing our observations with respect to the best value procurement objective, we structured our observations in accordance with the four PWGSC "Smart Procurement" principles. These principles reach beyond the objective of achieving best value, also enforcing values of integrity, transparency, and openness. The Smart Procurement principles are as follows:

  1. Early Engagement establishes a two-way conversation between industry and client departments to better understand both needs and available solutions. Early Engagement mitigates risk, identifies innovative solutions and enhances competition. In conducting early engagement, PWGSC connects buyers and suppliers through industry days, one-on-one supplier consultations and online collaboration tools.
  2. Effective Governance sets ground rules for industry discussions and creates an effective framework for managing project decisions, risks and other issues.
  3. Independent Advice sought during the procurement process, allows PWGSC to be informed by an objective third party (typically an industry expert) who assesses and validates decisions.
  4. Smart Procurement Benefits all Canadians by weighing potential risks and opportunities for both Canada and all Canadians throughout the procurement process. In this way, Smart Procurement supports job creation in Canada, innovation, increased opportunities for suppliers (including small- and medium-sized businesses), and reduced barriers for Aboriginal Canadians.

PWGSC's Smart Procurement approach is designed to align to Government of Canada procurement policy, which seeks, among other objectives, to obtain best value for Canadians.Footnote 2

We identified methods that are consistent with the objective of promoting a best value procurement approach and aligned with the Smart Procurement principles. While the Smart Procurement principles did not serve as the basis against which our review was conducted, we have used them as the structure against which we present our observations related to the objective of achieving best value. As explained earlier, the Smart Procurement principles reach beyond best value; our comments are limited to observations related to the objective of achieving best value.

5.2.1 Early Engagement

We observed that several iterations of industry engagement were initiated by the Government of Canada for the FWSAR Project at various points in the procurement process. We noted that this industry engagement was leveraged by the project to better inform estimates, assess technical feasibility, and identify areas requiring greater clarity in the RFP. The FWSAR industry engagement process included industry days, one-on-one supplier consultations and online notifications supporting the exchange of two-way written communication.

To manage the high volumes of industry feedback gathered through the engagement process, we observed heightened risk associated with records management and change management that should be managed closely by the Government of Canada as the procurement continues.

While we observed several practices in place to document industry correspondence, including one-on-one meetings, we did not observe a consistent level of detail in the meeting minutes and records of decision maintained for industry consultations. To promote knowledge transfer and position the Government of Canada in promptly responding to future scrutiny, it is important that communications with industry, as well as the rationale for all decisions, be well preserved. As the procurement process advances and the Government of Canada responds to the industry feedback received, it is important that responses to industry and the rationale for decisions made in reaction to industry feedback (e.g. changes to bid evaluation plans, methods, processes, and tools) be well documented. Specifically, decision rationale should demonstrate how the interdependencies between the various elements of the procurement approach were considered and weighed to achieve best value by the Government of Canada in advance of making changes. From the perspective of knowledge transfer, managing records and changes throughout the industry consultation approach can help inform future similar procurements.

5.2.2 Effective Governance

Governance structures and processes for the FWSAR Project were not within our scope; however, we did observe how governance impacted decision making as it relates to the elements of our scope. In this regard, we observed general consistency in the direction provided to the FWSAR Project in support of the development of evaluation plans, methods, processes and tools as well as the use of industry engagement.

We also observed the following attributes of effective governance during our review:

The wording used to communicate project objectives was refined over the course of the project. While a clear objective for this procurement has been established, it is not yet consistent across all documentation. To avoid potential misunderstandings that could impact decisions related to best value, it is important to enforce consistency in communication of objectives as this procurement advances. This risk could be reduced through quality control and monitoring processes for documents prior to dissemination and concerted internal communication of the procurement objective.

5.2.3 Independent Review

While other reviews were not re-performed or assessed for completeness, accuracy or alignment within the scope of our independent review, we observed and intersected with many other reviews being undertaken by subject matter specialists to advise on other aspects of the procurement. Also worthy of note is the National Research Council's independent review (2010) which served as the catalyst for a course change, redirecting the FWSAR Project toward an increased focus on capability-driven evaluation with a view toward achieving best value for Canadians. Some examples of other reviews being undertaken include:

We also observed a proactive approach by DND to seek independent advice, gather objective input to promote best value and implement recommendations from independent reviews. This observed proactivity helps to enable a more effective independent review process.

In addition to the observations above, we identified an opportunity to improve the clarity of the scope of independent reviews, including how they intersect with, and reinforce, one another. To promote efficiency and consistency of understanding among internal stakeholders, and ultimately maximize value, the Government of Canada should continue to strengthen the protocols for establishing, documenting and managing the scope of reviews, including retaining fulsome descriptions of the coverage achieved by quality reviews.

5.2.4 Benefit to all Canadians

Our review was not intended to include, nor did it include, an assessment of whether the procurement is designed to maximize benefit to all Canadians. Rather, in completing our review of the subset of bid evaluation plans, methods, processes, decisions and tools within scope, we considered the extent to which related decisions promoted the Government of Canada's objective of achieving benefits for Canada.

The FWSAR Project requires that all bids include a Canadian ISS integrator and methods intended to both meet Canada's Industrial and Regional Benefits Policy (IRB Policy)Footnote 3 and maximize the industrial benefits associated with the Canadian ISS provider(s). While assessing IRB Policy compliance and the evaluation of industrial benefits was outside the scope of our review, we observed that Industry Canada was actively involved in advising the FWSAR Project on the achievement of industrial benefits and the establishment of an IRB evaluation plan in support of intended outcomes. Further, we observed consistency between the use of the APAT to factor projected SAR incident response times in the evaluation of bidders and the NRC's recommendations to shift focus toward evaluating solution capability.

While outside of the scope of the FWSAR Project, we observed that limited SAR incident data was available to support procurement decisions. By acquiring and leveraging additional SAR incident data beyond historical SAR incident coordinates, the FWSAR Project could have, to a greater extent, supplemented expert opinion with data to support decision making.

Consistent with observations identified in other sections of our report, we observed instances in which the documentation supporting underlying plans, methods, processes, decisions and tools could be strengthened to enhance knowledge transfer within the FWSAR Project and to those responsible for future defence procurement initiatives.


Footnote 1

Return to footnote referrer 1

The objective and scope of our independent review were agreed upon with PWGSC officials.

Footnote 2

Return to footnote referrer 2

The Government of Canada Contracting Policy establishes that “the objective of government procurement contracting is to acquire goods and services and to carry out construction in a manner that enhances access, competition and fairness and results in best value or, if appropriate, the optimal balance of overall benefits to the Crown and the Canadian people.” (source: Treasury Board of Canada Secretariat website)

Footnote 3

Return to footnote referrer 3

The IRB Policy ensures that Government of Canada defence and security procurements generate high value-added business activity for Canadian industry. The IRB Policy requires companies undertake business activities in Canada valued at 100 percent of the value of the defence or security contract they have been awarded by the Government of Canada. The IRB obligation is a contractual commitment and part of the overall government procurement contract. The Government of Canada IRB Policy is intended to provide Canadian companies with opportunities to develop and competitively sell innovative products and services to companies with IRB obligations, many of which are large multinational aerospace and defence corporations. Canadian firms that become a valued partner or supplier to an IRB contractor can position themselves to pursue additional and future business opportunities that may last well beyond the initial IRB period. (source: Industrial and Regional Benefits website)

Date modified: