2007-606
Targeted Evaluation of the 2005 Alternative Forms of Delivery Initiative, Final Report

2008-05-02

Table of Contents

• Executive Summary

1. Introduction
2. Profile
   • 2.1 Background
   • 2.2 Contracts
   • 2.3 2005 AFD Objectives and Performance
3. Focus of the Engagement
4. Findings
   • 4.1 Substantial Savings Achieved
   • 4.2 2005 AFD is relevant, well planned, well defined, equitable and transparent
5. Conclusions
6. Management Response
7. Recommendation and Management Action Plan
8. About the Evaluation
   • 8.1 Objective
   • 8.2 Evaluation Methodology
   • 8.3 The Planning Phase
   • 8.4 The Assessment Phase
   • 8.5 The Reporting Phase
   • 8.6 Limitations of the Methodology
   • 8.7 Project Team

• Appendices
  • Appendix A: Previous AFD Studies and Findings
  • Appendix B: Documents Consulted

Executive Summary

What we examined

This report presents the findings of the targeted evaluation of the 2005 Alternative Forms of Delivery (AFD) Initiative. While a number of evaluations and studies conducted since 1998 assessed the relevance and success of AFD, none assessed the cost-effectiveness of the 2005 Initiative. This evaluation was approved as part of the Audit and Evaluation Branch 2007-08 to 2009-10 Risk-Based Multi-Year Audit and Evaluation Plan to determine if the 2005 Initiative provided additional savings over the 1998 Initiative. It was conducted in accordance with the evaluation standards for the Government of Canada (GC) and the Audit and Evaluation Branch of Public Works and Government Services Canada (PWGSC).

PWGSC is the common service provider and custodian of the office space requirements to more than 140 GC organizations. In October 1996, PWGSC received Treasury Board approval to contract out non-core operational services for a major part of its property portfolio. This was called the Alternative Forms of Delivery (AFD) Initiative. Its main objectives were cost savings and the positioning of the PWGSC Real Property Branch to a more strategic investment and asset management role.

At the time, PWGSC procured property management and project delivery services for more than 300 buildings (2.5 million square meters) grouped into 13 regional portfolios across the country. PWGSC was the custodian for these buildings. Property management and project delivery services included in the AFD cost about $250M per year.

Since 1996, PWGSC has put in place two consecutive contracts for the AFD Initiative, including the initial 1998 contract and a re-procurement in 2005. Each had a single service provider win the contract for managing all the portfolios.

Why it's important

PWGSC committed to evaluate the 2005 AFD in three phases. This evaluation addresses the last phase, which is the cost-effectiveness of the initiative.

What we found

The objectives, expectations, structure and services to be delivered under the 2005 contractual agreement have not changed substantially from the 1998 contract. The 1998 Initiative was shown to be cost-effective for the delivery of property management and project delivery services, when costs were compared against provision of these services by PWGSC.

Cost savings for the 2005 contract over the 1998 contract were found to be significant, primarily due to the structure of the management fees under the 2005 contract. Savings have risen to $27M for 2006-07, increasing from $23.3M in 2005-06.

Additional cost savings resulted from a decrease in departmental costs to administer the contracts, attributable to improvements in systems and processes, as well as a maturing of the program.

Previous studies demonstrated that both the 1998 and 2005 Initiatives are relevant and consistent with departmental and government-wide priorities. The 2005 AFD Initiative is well planned, comprehensive, well formulated and has mechanisms in place to ensure consistent performance reporting.

Conclusions

The 2005 AFD Initiative is cost-effective and has achieved substantial savings for the GC. The Initiative is relevant, well planned and comprehensive.

Management Response

Real Property Branch has identified 6 separate metrics to measure and compile savings related to the AFD initiative and is in the process of certifying an additional metric to measure energy savings. New metrics will be identified and developed on an on-going basis. The 6 metrics are:

• Property Management Services Fee
• Cost/Fee Structure Adjustments
• Project Delivery Services Fee
• Optional Services
• Reduced indirect costs
• AFD Management Cost Avoidance

Real Property accepts the evaluation findings and intends to act on the recommendation of the evaluation by implementing their Management Action Plan detailed as follows.

Recommendation

Based on key findings and conclusions contained in this report, the Audit and Evaluation Branch recommends that the Assistant Deputy Minister, Real Property Branch, ensure that the 2005 Alternative Forms of Delivery Initiative address the following recommendation.

Recommendation: Continue to develop and collect metrics based on the Real Property Management System to measure the Initiative's performance. This will be used in the subsequent evaluation to be undertaken within the five-year evaluation cycle.

Action Plan: Yearly compilation of savings with existing metrics will continue. RPB, in conjunction with SNC-Lavalin ProFac, will conduct a yearly review to identify possible new savings metrics.

Timeline: Annually by June 30th.

1. Introduction

This report presents the findings of the targeted evaluation of the 2005 Alternative Forms of Delivery (AFD) Initiative. This evaluation was approved as part of the Audit and Evaluation Branch 2007-08 to 2009-10 Risk-Based Multi-Year Audit and Evaluation Plan to determine if the 2005 Initiative provided additional savings over the 1998 Initiative.

This evaluation considers evaluation issues, questions and indicators, supported by the analysis of associated data, which was collected using standard evaluation methodologies. This evaluation was conducted in accordance with the evaluation standards of the Government of Canada (GC) and the Audit and Evaluation Branch of Public Works and Government Services Canada (PWGSC).

2. Profile

2.1 Background

PWGSC is the common service provider and custodian of the office space requirements to more than 140 GC departments, agencies and Crown corporations. In October 1996, PWGSC received Treasury Board approval to contract out the non-core operational services, including operation and maintenance services and other transactional work, such as property administration, for a major part of its property portfolio. This was called the Alternative Forms of Delivery Initiative and its main objectives were cost savings and the positioning of the PWGSC Real Property Branch into a more strategic investment and asset-management role.

At the time, PWGSC procured property management and project delivery services for more than 300 buildings (2.5 million square meters), grouped into 13 regional portfolios across the country. PWGSC was the custodian for these buildings. Property management and project delivery services included in the AFD cost about $250M per year.

2.2 Contracts

In 1997, PWGSC put in place the first contract for the Initiative. A single service provider, Brookfield LePage Johnson Controls, was successful in winning the contracts for managing all the portfolios. The contract commenced April 1, 1998, with an expiry date of March 31, 2005.

In 2004, PWGSC undertook a re-procurement process for a second contract. Based on the experience of the 1998 contracts, several enhancements to the new initiative were implemented. These included:

• Revisions to the governance structure and accountability framework;
• Realignment of the real properties into eight portfolios from thirteen;
• Creation of the Alternative Forms of Delivery Management System; and
• Replacement of the Management Value Incentive Program fees with the Contractor Incentive Program.

In 2005, SNC-Lavalin ProFac was the only successful bidder to manage all the portfolios. The contract commenced on April 1, 2005 for an initial four-year term, with renewal options for up to six additional years.

2.3 2005 AFD Objectives and Performance

The objectives of the 2005 AFD Initiative were to:

• Harvest the full value of private sector expertise and develop world class public – private relationships;
• Permit PWGSC to focus its resources on the core elements of real property management and client relationships;
• Achieve the best value for money in providing real property services;
• Maintain quality of service to PWGSC clients; and
• Ensure the condition and value of federal real property assets.

A database application — Real Property Management System — was set up to track performance data submitted by the contractor and form the basis for the development of key performance indicators. This process is used to determine the performance fee payments to the contractor. The three principal key performance indicators are:

• Asset integrity (maintaining the condition of Crown assets);
• Satisfaction (ensuring the well-being of clients, tenants and PWGSC); and
• Financial performance (managing and remaining within the budget).

3. Focus of the Engagement

An evaluation uses various methods to provide multiple lines of evidence to assure credibility of the findings, minimize threats to validity and enable an attribution of the results. The evaluation methods used in this study are described in Section 7: About the Evaluation.

The 1998 and 2005 AFD Initiatives were evaluated four times prior to this study. A 2003 summative evaluation of the 1998 Initiative concluded that it was more cost-effective than PWGSC in-house service delivery. However, these previous evaluations did not address the cost-effectiveness of the 2005 Initiative. Therefore, this evaluation will focus on determining if the 2005 Initiative provided additional savings over the 1998 Initiative. Costs to the GC for the 2005 contract were compared with the costs of 2004-05, which was the last year of the 1998 AFD Initiative.

4. Findings

4.1 Substantial Savings Achieved

In 2004, revisions to the 1998 contract terms meant that the 2004 winning bids did not include a Project Management Services fee. This was expected to result in substantial savings.

In September 2005, the Service Transformation Office developed a methodology to quantify at the contract level the savings achieved annually. The methodology, based on common business volumes for the new AFD portfolios, compared the contract terms and conditions of the 1998 service provider and PWGSC's in-house property management to the contract terms and conditions of the new service provider.

Table 1 below presents the main differences between the 1998 and 2005 contracts. Since the 2005 agreement was signed, an additional 0.75M square meters of managed space has been added to the original portfolio of 2.4M square meters. This keeps the 2005 contract within the contract expansion limits permitted by the Treasury Board of Canada.

Table 1: Comparison of 1998 and 2005 Contracts

Fees	2005 Contract vs. 1998 Contract	Comments
Property Management Services	2005 fees are 0% 1998 fees are about 5.86%	Exception is the Performance Management fee for the Pacific Portfolio Annual performance fee is $175,934 for a baseline of $12.5M, subject to annual adjustment based on the baseline allocation
Property Management Services: Allowable Pass-Through costs	Pass-through costs redefined more strictly, on a greater detailed basis Excludes portion of administrative and corporate functions costs accepted under the former contracts	Some pass–through costs activities include:  Building Services Delivery through Subcontracts; Management of Building Level Subcontracts; Labour Costs Including Salaries and Benefits for building level staff dedicated to one or more buildings; (i.e. property managers, property officers; building operators, technical and trades staff, administrative and accounting and information systems data entry personnel) Corporate functions fulfilled by these staff are not allowable pass-through costs. A corporate function is driven from internal requirements rather than by external requirements (i.e. Human Resources; IT Structure) Non-allowable costs: furniture, computers, software, firmware, hardware, related peripheral equipment Head Office charges excluded in 2005, includes accommodation, operations and maintenance of corporate staff
Project Delivery Service Fee	Base fee redefined Excludes labour costs of the project manager and support staff and other allowable costs included in the base fee of the earlier contract 1998 cap of $200,000 2005 contractor incentive fee based on performance	Fees calculated as a percentage of design and construction pass-through costs In the 2005 contract, project manager and support staff labour costs are not included in the base
Project Delivery Service – Optional Services(Note 1)	Allowable pass-through costs are the same as for project Delivery Services 2005 contract projects up to $1 million	Note: Cost avoidance is the estimated difference between total project costs if delivered in-house by PWGSC and actual costs of projects delivered by contractor
Lease Administration and Facilities Management	Project from $200K–$1M not in the 1998 contract
Allowable Pass-Through Costs and Fees for Commercial Leasing	Not permitted
Fees for Commercial Leasing	2005 fees based on set amount per square meter $43.05 for 5-year lease term $75.35 for 10-year fixed term 1998 rates based on the leasing activity (new or renewal) Percentage applied to full value lease amount based on lease terms

Note:
1. Optional Services was not awarded in the 1998 contract.

4.1.1 PWGSC Forecasted Savings

The application of the methodology used by the Service Transformation Office (now the Budget Commitment Office) to forecast savings indicates the savings values detailed in Table 2.

Table 2: Forecasted Savings

Category	Amount ($)
Property Management Services	11,540,000
Cost Fee Structure Adjustment	6,000,000
Project Delivery Services	840,000
Optional Delivery Services	Not calculated
Optional Commercial Leasing Services	Not calculated
Total savings	18,380,000

Information at the building level for the new AFD portfolios, obtained from the Alternative Forms of Delivery Management System, was reconciled to the existing payments within the financial system. The internal cost to the management of the AFD contract was obtained from the Project Business Management System and from the Common Departmental Financial System through the Financial Management Reporting System.

The projected savings were certified by PWGSC Finance to be correct.

4.1.2 PricewaterhouseCoopers Verification of Cost Savings

The independent firm PricewaterhouseCoopers was retained to review the reasonableness of the methodology developed by PWGSC. Also, the firm validated the calculations of the 2004-05 and 2005-06 actual Property Management Service and Project Delivery Service fees and the fee savings for 2005-06 and thereafter.

PricewaterhouseCoopers conclusions are¹:

1. AFD Cost Structure: The cost structure reflected within the savings model appears to capture all cost components under the new contract.
2. Cost of Added Buildings: The savings model appears to reflect an appropriate adjustment for the added buildings under the new contract.
3. SNC-Lavalin ProFac Actual Costs: Actual 2005-06 Property Management Services costs were $220M, compared to estimated costs (using the savings model) of $222M. The actual results appear to support the reliability of the estimated savings, as calculated using the model.
4. Non-Allowable Pass–through Costs: Isolation of non-allowable pass–through costs within the savings methodology appears consistent with the terms of the new contract.
5. Head Office Charge Back: AFD savings model appears to properly address the definition and treatment of Head Office charge backs under the new contract.
6. Field Costs: Full-time equivalent rate of $1,800 per year appears conservative and accommodation costs appear to be applied correctly to building portfolios where PWGSC lacks space for contractor's staff. Moreover, the savings model appears to properly address the definition and treatment of field costs under the new contract.
7. Final Savings Calculations: The calculations used in the savings model generate results that are consistent with the methodology and are mathematically accurate.
   • Actual 2004-05 project management and project delivery services total costs and fees ($250.4M);
   • Adjustments for added buildings (project management $34.7M and project delivery $6.4M);
   • Head Office charge back ($4.4M);
   • Field Costs (accommodation and FTE costs) ($1.6M);
   • Estimation of savings in project management services ($17.5M compared to actual savings of $18.7M);
   • Estimation of savings in project delivery services ($0.84M compared to actual savings of $0.37M); and
   • Estimation of total savings ($18.4M compared to actual savings of $19.1M).
8. Breakdown of Estimated Savings:
   • Total estimated savings (project management services savings and project delivery services savings) $18.4M;
   • Of the total savings, the project management services savings account for $17.5M (after adjustments, the 1998 contract would have cost $239.9M compared to the estimated costs of the 2005 contract at $222.4M);
   • The savings in project management services fees arise from the lower level of payments under the new contract by $11.5M and non–allowable pass–through costs of $6.0M;
   • The non–allowable pass–through costs is attributed to a reduction in head office charge back of $4.4M plus a reduction in field costs of $1.6M; and
   • Of the total savings, the project management services savings account for $0.84M because the old contract could charge at 5% of design and construction, plus labour costs. Under the new contract, the contractor can charge 5% of design and construction only.

4.1.3 Audit and Evaluation Branch Verification of Cost Savings

To obtain background information and documentation pertaining to the cost savings calculations, the Audit and Evaluation Branch met with the A/Director General, Professional and Technical Service Management, as well as the Director and staff from AFD Service Management Coordination. The documents provided were reviewed and the cost savings are documented in Table 3 on the following page with the exception of the reduced indirect costs for 2005-2006 in the amount of $2.5M. See Appendix B for a list of documents consulted.

Table 3: Audit and Evaluation Branch Verification of 2005-06 Cost Savings

Item	Savings Forecast ($) 2005-06	Savings Actual ($) 2005-06	Savings Actual ($) 2006-07	Comments
Property Management Services: Fee Savings	11,540,000	12,690,0002	13,997,500	BLJC~ related savings; does not include other government departments savings of $365,700
Cost/Fee Structure Adjustment-Savings	6,000,000	4,950,0003	-1,747,000	BLJC related savings; 1st year of data excluded annual 3rd party audit costs; and internet costs; + SNC~~ was not fully staffed in first year
Project Delivery Services: Fee Savings	840,000	475,2004	447,000 (PWGSC) 167,000 (OGD)	SNC contract: Labour cost excluded from base calculation (5% of construction cost and design cost) BLJC contract: Labour included in base calculation
Optional Services	Not applicable to BLJC	2,701,200	6,991,100	2nd year exercised optional project work over 200K. Savings include other government department portion of  $155,200
Reduced Indirect Costs	N/A	2,500,0005	3,300,0006	Savings on new initiatives: transfer of buildings previously managed in-house and non-PWGSC custodian facilities. Total for 2006-07 includes $2,500,000 FY 2005-06 and $800,000 for 2006-07
AFD Management Cost Avoidance	N/A	0.00	3,852,9007	As Real Property Branch gained experience with AFD, it simplified its processes and operational procedures. The resources required for contract management were reduced, leading to efficiencies on the internal management costs
Total Savings	18,380,000	23,316,400	27,008,500

~ BLJC refers to Brookfield LePage Johnson Controls
~~ SNC refers to SNC-Lavalin ProFac

Notes:

1. All costs have been adjusted for the Consumer Price Index (CPI) of 2.4% for 2005-06.
2. Adjustments have been made for the buildings added to the AFD contract in 2005-06.
3. Cost/Fee Structure Adjustments are being closely monitored and analyzed by the national AFD Directorate and the Real Property Budget Commitment Office.

PWGSC has undergone a learning process and maturation since the 1998 contract. The systems and processes introduced to monitor and manage the contracts have been refined and become more efficient. With the 2005 contract, portfolios were re-aligned from 13 to eight. The business processes and innovations were simplified; leading to internal departmental cost savings. PWGSC costs for managing the inventory are given in the Table 4; they show a declining trend.

Table 4: PWGSC Costs to Manage AFD Contract

Years	2004-058	2005-06	2006-07	2007-08
Business Volume (total payments to SNC)	$266,316,792	$333,938,895	$385,373,847	Not available
PWGSC Costs	$18,027,100 (6.8% of contact value)	$16,508,900 (4.9% of contact value)	$15,280,400 (4.0% of contact value)	Not available
Building Count	259	315	355	349
Rentable Square Meters	2,457,828	2,923,946	3,129,274	3,145,142

4.2 2005 AFD is relevant, well planned, well defined, equitable and transparent

Previous evaluations of the 2005 AFD Initiative⁹ examined the issues of relevance, success, design and implementation. It is of note that the objectives, expectations, structure and management of the 2005 AFD Initiative have not changed substantially from the 1998 Initiative. The conclusions and observations arrived at in the previous studies are briefly reproduced here. For a complete discussion, refer to Appendix A.

The 2005 Initiative is relevant and consistent with departmental and government-wide priorities (2005-611). The evaluation of the 1998 Initiative came to similar conclusions and noted there was substantial internal stakeholder support for the Initiative, along with a strong rationale for its continuation (2002-640).

Program design and delivery of the 2005 Initiative were well planned and comprehensive. Well-formulated and comprehensive oversight mechanisms are in place to manage the Initiative. There is ever greening of documentation, as required. Mechanisms have been designed to ensure consistent performance reporting. The roles and responsibilities for the Initiative were well defined and communicated. A comprehensive risk assessment was conducted (2005-611).

No issues were found that could compromise the success of the 2005 Initiative. The contracting process was found to be equitable and transparent (2005-611).

Overall there was no significant question that had not been answered.

5. Conclusions

The 2005 AFD Initiative is cost-effective and has achieved substantial savings for the GC. The Initiative is relevant, well planned and comprehensive.

6. Management Response

Real Property Branch has identified 6 separate metrics to measure and compile savings related to the AFD initiative and is in the process of certifying an additional metric to measure energy savings. New metrics will be identified and developed on an on-going basis. The 6 metrics are:

• Property Management Services Fee
• Cost/Fee Structure Adjustments
• Project Delivery Services Fee
• Optional Services
• Reduced indirect costs
• AFD Management Cost Avoidance

Real Property accepts the evaluation findings and intends to act on the recommendation of the evaluation by implementing their Management Action Plan detailed as follows.

7. Recommendation and Management Action Plan

Based on key findings and conclusions contained in this report, the Audit and Evaluation Branch recommends that the Assistant Deputy Minister, Real Property Branch, ensure that the 2005 Alternative Forms of Delivery Initiative address the following recommendation.

Recommendation: Continue to develop and collect metrics based on the Real Property Management System to measure the Initiative's performance. This will be used in the subsequent evaluation to be undertaken within the five-year evaluation cycle.

Action Plan: Yearly compilation of savings with existing metrics will continue. RPB, in conjunction with SNC-Lavalin ProFac, will conduct a yearly review to identify possible new savings metrics.

Timeline: Annually by June 30th.

8. About the Evaluation

Evaluation provides an evidence-based assessment of the relevance and effectiveness of policies and programs. It is the primary source of neutral and systematic information on the ongoing relevance, results and value of policies and programs, alternative ways of achieving expected results and program design improvements. It helps the GC design and deliver programs and services that are accountable, focused on results and that meet the needs of Canadian citizens. It is a key function of government that informs management and guides resource allocation within departments and government-wide.

8.1 Objective

This evaluation is limited to a review and analysis of the costs of Real Property Inventory Management in the first full year of the 2005 Initiative against the costs of the previous 1998 Initiative in the final year (2004) of its contract and a review of past evaluation studies.

8.2 Evaluation Methodology

This evaluation ensures that evaluation issues, questions and indicators are identified and are supported by associated data and data collection strategies. This evaluation of the 2005 AFD Initiative was conducted in accordance with the evaluation standards of the GC and the PWGSC Audit and Evaluation Branch. This evaluation was developed between August and December 2007, using the following lines of evidence:

Document Review: The previous frameworks, evaluations, audits, reviews and client satisfaction surveys (listed in Appendix B) were reviewed for this study.

Financial Data Analysis:Several PWGSC reports detailing the cost savings of the 2005 Initiative were analyzed to understand the methodology and the logic of the analysis leading to the cost savings figures.

Interviews: Meetings were held with staff from the AFD Service Management Coordination unit to obtain documents and to review the calculations.

8.3 The Planning Phase

The planning phase of the Evaluation was limited to: reviewing past evaluation frameworks, evaluations, reviews, Auditor General reports and other assessments carried out in-house by PWGSC.  The purpose of this phase was to develop the strategy for the assessment phase.

8.4 The Assessment Phase

The assessment phase comprised a review of documents, interviews with key stakeholders, collection of relevant departmental studies and analysis of the information collected to judge cost-effectiveness of the Initiative.

8.5 The Reporting Phase

We documented our findings and conclusions in a Director's Draft Report, which was internally cleared through Audit and Evaluation's Quality Assurance function. We provided the Program's Director General with the Director's Draft Report with a request to validate facts and comment on the Report. A Chief Audit Executive Draft Report was prepared and provided to the Office of Primary Interest's Assistant Deputy Minister for acceptance. The Office of Primary Interest was requested to respond with a Management Action Plan. The Draft Final Report, including the Management Action Plan, was then presented to PWGSC's Audit and Evaluation Committee for the Deputy Minister's approval. Once finalized, the Final Report will be submitted to the Treasury Board of Canada Secretariat and posted on the PWGSC website.

8.6 Limitations of the Methodology

It should be noted that AFD management provided a majority of the documents and data reviewed. That does not inherently imply that the data were biased, however, it is a limitation of the methodology.

8.7 Project Team

Staff of the Audit and Evaluation Branch developed this evaluation, under the direction of the Director of Evaluation and Chief Audit Executive of the Audit and Evaluation Branch.

Appendices

Appendix A: Previous AFD Studies and Findings

Study Title	Type of Study	Issues covered	Author / Comments
97-613 Evaluation Framework – AFD for Property Management	Framework	Rationale Design and delivery Success	This document provided a framework and evaluation plan
1999-643 Review of Minister's Accountability for ASD Contracting	Audit	Policy issues Service delivery issues Potential areas of vulnerability in the procurement process	PWGSC Findings were positive
1999 September Report of the Auditor General, Chapter 18—Public Works and Government Services Canada—Alternative Forms of Delivery: Contracting for Property Management Services	Audit	Observations and recommendations related to the Contracting Process	Office of the Auditor General Reviewed contracting process and contract administration
AFD: Program and Operational Review May 31, 2001	Review	Objectives Strategy Delivery Instrument Performance assessment, management structure and processes	Corporate Research Group Dixon Associates Management Consultants Focus on Lessons Learned
2002 AFD Summative Evaluation Revised Evaluation Framework (not available for review)	Framework	Revision and updating of the 97-613 Evaluation Framework	Consulting and Audit Canada
2002-640 Evaluation of the Alternative Forms of Delivery Initiative	Summative Evaluation	Relevance Program design and delivery Success Cost-effectiveness Alternatives	Consulting and Audit Canada Response to TB decision Comprehensive study with positive conclusions
2004-610 Evaluation Framework for the AFD Initiative for Real Property Services, May 26, 2005	Framework	Rationale/Relevance Program Design and Delivery Success Cost-Effectiveness and Alternatives	PWGSC Audit and Evaluation Branch For the AFD Initiative commencing April 1, 2005 Recommended an Interim Evaluation and Final Evaluation
2005-611 Final Report, Interim Evaluation of the New AFD Initiative for Real Property Services, Phase 1	Interim Evaluation	Relevance  Program design and delivery Program success	PWGSC Audit and Evaluation Branch
2006-603 Final Report, Interim Evaluation of the 2005 AFD Initiative for Real Property Services, Phase 2	Interim Evaluation	Program design and delivery Success	PWGSC Audit and Evaluation Branch

Other studies, not directly related to the purposes of the present evaluation, hence not reviewed, are listed below: 

• Alternative Forms of Delivery at Real Property Services: A Report by the AFD Transition Task Force, Public Works and Government Services Canada, April 1999.
• Review of the Implementation of the Alternative Forms of Delivery for Property Management Services: Stakeholders' Perspectives, Consulting and Audit Canada, June 2000.
• Industry Trends and Practices in Outsourcing Realty Services, David Patton, February 2001.
• Client Satisfaction Surveys.

Specific Findings and Conclusions by Evaluation Issues

Evaluation issues covered by the above studies are:

• Rationale/Relevance;
• Program Design and Delivery;
• Success;
• Cost-Effectiveness; and
• Alternatives.

The conclusions from these studies are presented below by evaluation issue. The specific findings and conclusions related to the contracting process are not included here, as the general conclusion was that the contracting process was fair and transparent.

Relevance

AFD 2005

The document analysis supports the finding that the 2005 Initiative is relevant and consistent with departmental and government-wide priorities. (2005-611)

AFD 1998

Given the on-going interest of the federal government in the use of alternative organizational forms to provide programs and services more efficiently, and the substantial internal stakeholder support for the AFD, there is a strong rationale for continuation of the Initiative. (2002-640)

Program Design and Delivery

AFD 2005

The overall conclusion is that program design and delivery of the 2005 Initiative was well planned and comprehensive.

A comprehensive set of oversight mechanisms are in place to manage the AFD within the intended strategic context. They are well formulated, and comprehensive. There is ever greening of documentation as required. (2005-611)

Mechanisms have been designed to ensure consistent performance reporting. Because the Alternative Forms of Delivery Management System (AFDMS) is not fully populated as of yet by the service provider, few outputs are available. Workarounds such as hard copy reports and ad hoc reporting were developed that increased the level of communications and meetings in order to achieve the desired outcomes of adequate oversight and stewardship. (2005-611)

The roles and responsibilities of the AFD were well formulated, comprehensive and well communicated. (2005-611)

Risk assessment for the AFD New Procurement was comprehensive. Mechanisms and elements are in place, are well formulated, and have been well communicated. Oversight instruments to mitigate contractor non-conformances, service quality, health and safety, asset integrity and adherence to legislation are thorough. (2005-611)

The AFD quality assurance and quality measurement mechanisms are ample and well documented. The contract with the new service provider requires that their quality management system is successfully ISO registered within two years of the contract start date. Key performance indicators and contractor incentives for performance are clearly defined. Standard operating procedures for Real Property Branch's quality management activities are complete and communicated and have been successfully implemented. Evergreening of the standard operating procedures occurs and best practices are being developed. (2005-611)

The quality assurance and quality measurement systems in place for the new AFD Initiative are consistent with Real Property Branch policies and practices and linked to GC policy objectives and priorities through the standard operating procedures in the Statement of Work. (2005-611)

AFD 1998

Despite a variety of early problems and "growing pains," Real Property Services Branch has been steadily adjusting and refining its approach to managing the AFD contract. The Statement of Work, performance measurement and reporting requirements, and performance monitoring have all been substantially revised and improved as the AFD has matured. (2002-640)

Our findings support the opinion that sufficient controls exist to prevent the Service Provider from escalating project spending unchecked. Property and Facility Managers in non-AFD facilities and Property Managers in AFD facilities, identify the building maintenance and repair needs each year in the Building Management Plan and/or Annual Building Plan process. (2002-640)

The 1999 Office of the Auditor General report concluded that the bidding process was open, transparent and fair, contracts were well designed and the handover to the contractor went smoothly.

Success

AFD 2005

The Fairness Monitoring Report concluded that the contracting process was equitable and transparent. (2005-611)

We concluded that, overall, there appeared to be no issues significant enough to compromise the effectiveness of the 2005 AFD Initiative. We found that the Performance Management Framework had been implemented and was operating as intended. Further, high levels of client satisfaction had been achieved. Finally, Real Property Branch had succeeded in creating an environment where transformation approaches could be tested and refined while ensuring that the beneficiaries (occupants of federally owned facilities) were not negatively affected. (2006-603)

We noted that management information systems such as the AFD Management System and the Quality Monitoring System were consistently implemented across all contracts. Performance indicators provided a clear description of the level of service being achieved, and actions for improvement were being taken as required. The results indicated that assets are being maintained per the Building Management Plans, and the Department's desired service delivery outcomes were being met. (2006-603)

AFD 1998

To the extent that the Service Provider operates the facilities maintenance program, all targets have been met and are supported by key performance indicator scores. (2002-640)

Key performance indicators lagged somewhat below target for the first 18 months of implementation, thereafter, however it rose dramatically. Real Property Services Branch has since institutionalized tenant satisfaction measurement by developing survey techniques and metrics geared specifically to their clientele and operations. They have expanded this capacity to embrace all non-AFD assets as well. National Service Call Centre data also plays a role instrategic decision-making. Altogether,despite a slow beginning, Real Property Services Branch developments in this area have been above target expectations. (2002-640)

Appropriate provisions were made for respecting Canada's official languages, however, there is room for improvement with respect to performance measurement. (2002-640)

Government priorities for Health and Safety, and Environment and Sustainable Development appear to be duly respected under the AFD program. (2002-640)

By all accounts AFD has been a successful initiative. It is a work-in-progress which claims success in already having achieved many of its varied objectives and having established an effective new way of doing business for Real Property Services Branch. (2002-640)

Cost-Effectiveness

AFD 2005

This issue was not covered in the evaluation of the 2005 AFD Initiative.

AFD 1998

Our analysis has determined that, conservatively estimated, the AFD has saved approximately $12 million in 2001-02 (the subject year of analysis). On the basis of the economies in personnel and operating costs alone, it is reasonable to conclude that the savings obtained in 2001-02 were achieved in 1999-00 and 2000-01.

Based on discussions with staff and a document review, the Consulting and Audit Canada team believes that the data required to identify all of the costs associated with the AFD program and Real Property Programs are available. (2002-640)

Alternatives to AFD

AFD 2005

This issue was not covered in the evaluation of the 2005 AFD Initiative.

AFD 1998

Industry development has been reasonably well served through AFD to the extent that AFD, given its "public good" objectives and overarching governmental imperatives, can achieve that. (2002-640)

Given the current success and ongoing improvements being made to the AFD outsourcing model, there is no immediate or compelling rationale for exploring alternative approaches. (2002-640)

In general, it can be said that the AFD Initiative has been found to be relevant, successful, well managed and cost-effective. The profile sections of the reports 2005-611 and 2006-603 suggest the 2005 contract for the AFD property management initiative is an improvement over the first contract.

Appendix B: Documents Consulted

1. PWGSC, Budget Commitment Office – Budget Commitment, Inventory Management- AFD Contract Savings – Certification Methodology & Methodology Process Overview, June 27, 2006.
2. PricewaterhouseCoopers, Memo to Mr. Ron Lapensee, Special Advisor, PWGSC, dated August 14, 2006, PricewaterhouseCoopers proposal to verify cost savings methodology.
3. PWGSC, Review of the Alternative Forms of Delivery (AFD) Cost Savings Model – New Contract with SNC-Lavalin ProFac Inc., November 23, 2006, prepared by PricewaterhouseCoopers.
4. PricewaterhouseCoopers, Review of the Alternative Forms of Delivery (AFD) Cost Savings Model – New Contract with SNC-Lavalin ProFac Inc., November 23, 2006, Draft for Discussion.
5. Budget Commitment Office, Budget Commitment, Inventory Management – AFD Contract Savings – Reports submitted by the National AFD Directorate, July 28, 2006.
6. PWGSC, Real Property Branch, Inventory Management – AFD Contract Savings, Tracking of Cost-Avoidance for 2005/06 due to Cost/Fee Structure Adjustment, November 21, 2006.
7. PWGSC – Real Property Branch, Inventory Management – AFD Contract Savings, Project Delivery Services, Tracking of Fee Savings for 2005/06, December 8, 2006
8. 2006-07 AFD Budget Commitments – Reduced Indirect Costs related to increased AFD Portfolio, August 14, 2007.
9. 2006-07 AFD Budget Commitments – Cost/Fee Structure Adjustment, DRAFT November 14, 2007.
10. 2006-07 AFD Budget Commitments – PDS Optional Services, September 4, 2007.
11. 2006-07 AFD Budget Commitments AFD Management Costs – September 10, 2007.
12. Budget Commitment Office – Budget Commitment, Inventory Management- AFD Contract Savings – Certification Methodology & Methodology Process Overview, June 27  2006.
13. PricewaterhouseCoopers, Memo to Mr. Ron Lapensee, Special Advisor, PWGSC, dated August 14, 2006, PwC proposal to verify cost savings methodology.
14. PWGSC, Review of the Alternative Forms of Delivery (AFD) Cost Savings Model – New Contract with SNC-Lavalin ProFac Inc., November 23, 2006, prepared by PricewaterhouseCoopers.
15. PricewaterhouseCoopers, Review of the Alternative Forms of Delivery (AFD) Cost Savings Model – New Contract with SNC-Lavalin ProFac Inc., November 23, 2006, Draft for Discussion.
16. PWGSC, Evaluation Framework – AFD for Property Management (97-613).
17. PWGSC, Review of Minister's Accountability for ASD Contracting, (1999-643).
18. Office of the Auditor General, 1999 September Report of the Auditor General of Canada, Chapter 18—Public Works and Government Services Canada—Alternative Forms of Delivery: Contracting for Property Management Services, 1999
19. AFD, Program and Operational Review, May 31, 2001.
20. AFD, Summative Evaluation Revised Evaluation Framework, 2002. (unavailable for review)
21. PWGSC, Evaluation of the Alternative Forms of Delivery Initiative, (2002-640).
22. PWGSC, Evaluation Framework for the AFD Initiative for Real Property Services, (2004-610) May 26, 2005.
23. PWGSC, Final Report, Interim Evaluation of the New AFD Initiative for Real Property Services – Phase 1 (2005-611).
24. PWGSC, Final Report, Interim Evaluation of the 2005 AFD Initiative for Real Property Services, Phase 2, (2006-603).
25. David Patton, Study on Exercising the Option to Extend the AFD Contracts, Corporate Research Group, March 9, 2007.

---

¹ PWGSC, Review of the Alternative Forms of Delivery (AFD) Cost Savings Model – New Contract with SNC Lavalin ProFac Inc., November 23, 2006, prepared by PricewaterhouseCoopers. (Return to Note 1)

² Budget Commitment Office, Budget Commitment, Inventory Management, AFD Contract Savings, Reports submitted by the National AFD Directorate, July 28, 2006. (Return to Note 2)

³ PWGSC, Real Property Branch, Inventory Management, AFD Contract Savings, Tracking of Cost-Avoidance for 2005-06 due to Cost/Fee Structure Adjustment, November 21, 2006. (Return to Note 3)

⁴ PWGSC, Real Property Branch, Inventory Management, AFD Contract Savings, Project Delivery Services, Tracking of Fee Savings for 2005-06, December 8, 2006. (Return to Note 4)

⁵ Budget Commitment Office. AFD Service Management Coordination has not provided any documentation substantiating this amount to Audit and Evaluation Branch. (Return to Note 5)

⁶ 2006-07 AFD Budget Commitments, Reduced Indirect Costs related to increased AFD Portfolio, August 14, 2007. (Return to Note 6)

⁷ 2006-07 AFD Budget Commitments AFD Management Costs, September 10, 2007. (Return to Note 7)

⁸ As the Budget Commitment Office, National Accommodation and Program Management does not have on hand the data for 2004-2005, these figures could not be validated. The figures for 2005-2006 and 2006-2007 have been confirmed. (Return to Note 8)

⁹ See Appendix A for a list of previous AFD studies. The numbers in brackets reference the report numbers in the list. (Return to Note 9)