Appendix A – Contract Portfolios
Appendix B – List of Interviewees
This second phase of the Interim Evaluation of the 2005 Alternative Forms of Delivery Initiative for Real Property Services, referred to as Phase 2, was approved by the Audit and Evaluation Committee as part of the Public Works and Government Services Canada (PWGSC) 2006-07 Evaluation Plan.
The objective of this Phase 2 was to identify any problems being encountered that may compromise the Initiative's effectiveness, and to make recommendations for improvements, if required.
The scope of the evaluation covered the specific questions of design and delivery and one indicator of success that were approved by the Audit and Evaluation Committee in the evaluation framework for the Alternative Forms of Delivery Initiative. We also assessed progress on the implementation of the management action plan that resulted from the Phase 1 Interim Evaluation recommendations. In line with the approved evaluation strategy, cost-effectiveness was not assessed at this time.
In 1997, the Department undertook the first Alternative Forms of Delivery (AFD) Initiative. The main objectives of this Initiative were savings and the positioning of the Real Property Branch into a more strategic investment and asset-management role. The Department procured property management and project delivery services for more than 300 buildings (2.5 million square metres) grouped into 13 regional portfolios across the country. These contracts commenced on April 1, 1998 and expired on March 31, 2005.
In 2004, the Department undertook a re-procurement process for a second AFD Initiative. Based on the "lessons learned" experience of the 1998 contracts, several enhancements were implemented for the next initiative. These included revisions to the governance structure and accountability framework; realignment to eight portfolios; creation of the Alternative Forms of Delivery Management System; and the replacement of the Management Value Incentive Program fees with the Contractor Incentive Program.
As in the 1998 AFD, a single service provider, different from the first, was retained for all of the contracts. The contracts commenced on April 1, 2005 for an initial four-year term with renewal options for up to an additional six years. The objectives of the 2005 AFD Initiative were to:
The transition towards a new organizational structure for the Real Property Branch began in response to the findings of a 2002 Task Force on Service Delivery. During this same time, the Department led a strategy to find innovative ways to deliver services smarter, faster, and at a reduced cost, and to improve how the Government of Canada does business. PWGSC's Way Forward was established, and outsourcing became an integral part of business transformation. The Treasury Board Secretariat identified the real property business transformation initiative as one of the five major Government of Canada modernization initiatives.
We concluded that, overall, there appeared to be no issues significant enough to compromise the effectiveness of the 2005 AFD Initiative. We found that the Performance Management Framework had been implemented and was operating as intended. Further, high levels of client satisfaction had been achieved. Finally, Real Property Branch had succeeded in creating an environment where transformation approaches could be tested and refined while ensuring that the beneficiaries (occupants of federally owned facilities) were not negatively affected.
In addition to the competence of this service provider, we believe other factors contributed to these results, most notably, the Department's experience with AFD; the quality of the Statement of Work and Standard Operating Procedures; and the ability of Real Property Branch staff responsible for exercising contractual oversight.
Furthermore, we noted that management information systems such as the AFD Management System and the Quality Monitoring System were consistently implemented across all contracts. Performance indicators provided a clear description of the level of service being achieved, and actions for improvement were being taken as required. The results indicated that assets are being maintained per the Building Management Plans, and the Department's desired service delivery outcomes were being met.
There were five recommendations that stemmed from the phase 1 Interim Evaluation. The recommended action for one of the recommendations has been superseded by a departmental reorganization that led to the dissolution of the Service Integration Branch. We found evidence that Real Property Branch had achieved progress on the implementation of the management action plan for the four remaining recommendations.
We noted one area for immediate improvement. The Real Property Branch has calculated the total costs for the AFD Initiative. However, it is essential that the definitions and basis for all elements of costs be clearly communicated to all persons using this type of data in the Department, to ensure that complete and correct information is used.
To ensure the continued success of the 2005 AFD Initiative and to be in a position to report on its effectiveness, we recommend that the Assistant Deputy Minister Real Property Branch ensure that:
The purpose of this report is to present the findings, conclusions, and recommendations from the Phase 2 Interim Evaluation of the Alternative Forms of Delivery (AFD) Initiative for Real Property Branch.
This report has four sections:
Public Works and Government Services Canada's Real Property Branch is the Government of Canada's real property service delivery manager and strategic advisor. These responsibilities include acquiring, managing, operating, maintaining, building, repairing, and disposing of federal real property. The inventory ranges from large, multi-purpose complexes in major urban centres to small buildings in towns across Canada, industrial buildings, laboratories, and other specialized facilities.
The Public Works and Government Services Canada (PWGSC) response to the 1994 government-wide Program Review was a strategic decision to outsource day-to-day real property services. The objectives of the AFD concept were to achieve savings, and to refocus the Real Property Branch on a more strategic approach to managing its portfolio. The Department obtained approval to implement its proposed approach to outsourcing in 1996.
In 1998, the Department procured property management and project delivery services for more than 300 buildings (2.5 million square metres) grouped into 13 regional portfolios across the country. These were tendered as 13 separate contracts and were awarded to one service provider. These contracts commenced on April 1, 1998 and expired on March 31, 2005.
In 2004, the Department undertook a re-procurement process. Based on the "lessons learned" experience of the 1998 contracts, several enhancements were implemented for the subsequent AFD Initiative. These included revisions to the Governance Structure and Accountability Framework; realignment to eight portfolios; creation of the Alternative Forms of Delivery Management System and replacement of the Management Value Incentive Program fees with the Contractor Incentive Program.
As in the 1998 AFD , a single service provider, different from the first, was retained for all of the contracts. (A list of the contracts with their respective size and value is presented in Appendix A). These contracts commenced on April 1, 2005 for an initial four-year term with renewal options for up to an additional six years. The objectives of this AFD Initiative were to:
The transition towards a new organizational structure for the Real Property Branch began in response to the findings of a 2002 Task Force on Service Delivery. During this same time, the Department led a strategy to find innovative ways to deliver services smarter, faster, and at a reduced cost, and to improve how the Government of Canada does business. PWGSC's Way Forward was established, and outsourcing became an integral part of business transformation. The Treasury Board Secretariat identified the real property business transformation initiative as one of the five major Government of Canada modernization initiatives.
This second phase of the Interim Evaluation of the of the 2005 Alternative Forms of Delivery Initiative for Real Property Services, referred to as Phase 2, was approved by the Audit and Evaluation Committee as part of the Public Works and Government Services Canada (PWGSC) 2006-07 Evaluation Plan.
An Evaluation Framework for the 2005 AFD Initiative was completed in January 2005 and discussed at the Audit and Evaluation Committee in May 2005. The evaluation strategy presented in the framework included an interim evaluation to be completed in the first two years and a final evaluation to be completed in the third year.
The Phase 1 Interim Evaluation was completed in January 2006 and concluded that the transition from the former service provider to the new service provider was successfully accomplished. It also made five recommendations to ensure that the continued success of the Initiative was not compromised. Real Property Branch accepted these recommendations, and developed a management action plan to address them.
The objective of this Phase 2 evaluation was to identify any problems being encountered that maycompromise the Initiative's effectiveness, and to make recommendations for improvements, if required.
The scope of this evaluation covered the specific questions of design and delivery and an early indicator of success that were approved in the Evaluation Framework for the Phase 2 interim evaluation.
The original Framework had identified two separate evaluation questions dealing with performance management and performance reporting. They were:
As we completed our fieldwork, we came to the realization that it would be difficult to present our findings separately because the differences between the two questions were too subtle. Consequently, we have combined the two questions into this single question:
The following evaluation issues and associated questions were examined:
1. Is PWGSC able to accurately report on costs and savings related to the AFD Initiative?
2. Has PWGSC implemented consistent and effective performance management across contracts?
3. Has the Contractor Incentive Program been consistently applied across contracts?
4. Has the AFD Initiative been allocated sufficient qualified resources (human, financial, materiel) for operation?
5. To what extent did the AFD Initiative lead to high levels of client/customer/tenant satisfaction?
We also assessed progress on the implementation of the management action plan that resulted from the Phase 1 interim evaluation recommendations. In line with the approved evaluation strategy, cost-effectiveness was not assessed at this time.
The approaches used to gather data and assess the evaluation issues are in accordance with the standards and practices of the Audit and Evaluation Branch, the Government of Canada Evaluation Policy, and direction provided by the Treasury Board Secretariat.
The following sources of data were used:
Document and File Review: A variety of background documents were reviewed, such as previous evaluations, the AFD Statement of Work, consultant reports, policy documents, the AFD Management Handbook.
Monitoring Instruments: Also reviewed were the documentation and outputs of the Quality Management Framework and the Alternative Forms of Delivery Management System.
Key Stakeholder Interviews: Consultations with key stakeholders took place. Follow-up sessions were also held with specific stakeholders to validate and obtain further clarification on a variety of issues. A list of interviewees is presented in Appendix B.
The key findings are organized according to the issues/questions presented in section 1.3.
The AFD Management System is the Department's repository for all direct contract costs(1) information related to AFD managed buildings. Controls applied to data accuracy include review of monthly invoices (e.g., historical trends), mid-year reviews, year end reconciliation, annual service provider statements audited by a third party and National-level review and reconciliation of service provider and AFD Management System data.
Our review of the system and documentation revealed that significant efforts to ensure data accuracy had been undertaken by the National AFD Directorate. For example, a project coding reference chart was developed to assist the service provider with data entry; data in the system are monitored on an on-going basis; training for departmental Technical Authorities in the use of the AFD Management System data had resumed. The Directorate reported that the AFD Management System was now fully populated and 100% accuracy was expected imminently.
In March 2005, the Real Property Branch developed a Costing and Charging Model that allowed the Branch to allocate branch and departmental overhead costs(2), as well as branch support costs(3) to the real property services offered to other government departments on an optional and cost-recovery basis.
Real Property Branch staff responsible for resource allocation and reporting, provided us with materials that describe how branch and departmental overhead costs and branch support costs were allocated to the AFD Initiative. The Branch support costs include the costs of the National AFD Directorate, the national and regional management costs associated with the technical and quality support at the portfolios level, the costs of the acquisition team, and the Branch information management systems specific to the AFD.
The branch and departmental overheads and the branch support costs added to the direct contract costs make up the total costs of the AFD Initiative. The total costs for the AFD Initiative have been calculated for 2004-05 and 2005-06 using the model mentioned above. Our analysis of the total AFD costs indicate that the direct contract costs, which are audited by a third party, make up about 95% of the total costs of the AFD Initiative.
Although, Real Property Branch has calculated the total costs of the AFD Initiative for 2004-05 and 2005-06, we found that this information was not produced in a timely manner. Current information on total AFD costs was not available. While the value of departmental costs for managing the AFD (5%), are dwarfed by the contract costs (95%), we are concerned that the definitions and basis for all elements of costs have not been clearly communicated to all persons using this type of data in the Department, to ensure that complete and correct information is used. We found that the lack of attention to the branch and departmental costs may have contributed to a risk that incomplete or wrong costs information be used by departmental officials when making analyses and comparisons of AFD related costs with other service delivery models and industry standards.
Real Property Branch also developed a methodology to calculate potential savings to be had from the 2005 AFD Initiative. We were provided with a demonstration of how the methodology has been used. The approach was used to arrive at a projection of savings for the first five years of the Initiative. These projected savings have been included in the Real Property Branch's Annual Inventory Management Stream Report.
The projected savings have been calculated by comparing estimated contract costs to the actual costs for 2004-05, the last year of the 1998 AFD Initiative, for the same portfolio of buildings and the same types of services with an adjustment for the additional buildings and services included in the 2005 contract. It was not clear how any savings obtained through reductions of departmental and branch costs that could result from other transformation activities such as finance, human resources and information technology would be considered in the future. PWGSC's Finance Branch as well as a third party independent firm provided an independent review and confirmation of the methodology used to determine projected savings.
The National AFD Directorate oversees the AFD Initiative and assists in applying contract provisions. The national level interaction assists in ensuring that contract requirements such as reporting are consistently applied across all eight contracts.
There are five primary sources of performance management information for this Initiative. The sources are the:
*
]PWGSC has verified the performance reported in the Quality Management System by conducting building and service inspections in accordance with the department's Quality Monitoring Program. Key interviewees indicated that consistent performance reporting is facilitated by the fact that the Department has to deal with only one service provider using the same Quality Monitoring System for all contracts.
The Real Property Branch has implemented Standard Operating Procedures for various components of the Quality Management Framework. Completed Quality Monitoring Reports document the consistent application of inspection procedures and include detailed quality monitoring checklists.
According to contract terms, each region has been provided the flexibility to identify key performance indicator baselines and benchmarks to suit local requirements and have done so with the oversight and guidance of the National AFD Directorate. The Directorate reported that they have performed sensitivity analysis to provide informed advice concerning the establishment of the baselines to promote continuous improvement in service delivery.
Client satisfaction for the Initiative is tracked by the National Service Call Center by sampling 10% of all services sent to the service provider. The results of the sampling effort provide the input for the performance indicator on service call satisfaction. Findings on the level of client and tenant satisfaction are presented in section 2.2.1.
The National Tenant Satisfaction Survey, conducted by Statistics Canada, is used for all AFD contracts, at the same time and frequency. A survey was underway at the time that we were conducting this evaluation. The survey, which was previously conducted on an annual basis, is being conducted every second year to reduce costs.
Per contract terms, each PWGSC Region has been provided the flexibility to add Key Performance Indicators to suit local requirements. The National AFD Directorate has been involved in those instances where the Region did identify additional KPIs. The Directorate performed sensitivity analysis of the baselines to provide informed advice concerning their establishment and to promote continuous improvement in service delivery. As indicated in section 2.1.1, the AFD Management System functionality is the same for all eight contracts.
Based on our review of the five primary performance management instruments, and the results of interviews, we found Headquarters and the Regions, with the guidance and leadership of the National AFD Directorate had implemented consistent and effective performance reporting. Interviewees commented that the level of performance management and reporting for the AFD was greater than that for in-house service delivery activities.
The Contractor Incentive Program Standard Operating Procedures contained in the AFD Management Handbook provide clear direction on what the program is and how it is to be applied. It states that the objective of the program is to encourage the service provider to proactively assist the Department in achieving outstanding results in the pursuit of Government of Canada targets and priorities. The focus of the Contractor Incentive Program is on results rather than activities and inputs.
The program provides a monetary incentive designed to encourage, recognize, and reward the service provider for extraordinary creativity and innovation and to motivate the service provider to excel in key areas beyond the performance baselines set out in the contract. The Standard Operating Procedures clearly states that the service provider must have earned 100% of the available contract fees during the fiscal year (including the Key Performance Indicator fee portion) to be eligible for a Contractor Incentive Program award. Consequently, application of the Contractor Incentive Program is dependent upon the Key Performance Indicator process. In cases where there are only Regionally developed Contractor Incentive Program targets the Regions can determine the percentage of the fees to be allocated to each target. Key interviewees indicated that assessments of the Contractor Incentive Program scores were coordinated nationally and tabulated and reviewed in a standard and transparent manner.
Regional managers have independently identified and negotiated Contractor Incentive Program target areas to complement those set nationally. The total maximum percentage fees available are established by the contract and are a combination of national and regional target levels.
Two nationally developed Contractor Incentive Program target areas were formally established for all eight contracts for 2005-06. These target areas were defined in "Letters of Emphasis" to the service provider in accordance with the Standard Operating Procedures. For 2006-07, two nationally developed target areas have been drafted, and regions were in the process of developing their local targets.[
*
]there was still value derived from the exercise of determining the targets. Both the Department and service provider interviewees indicated that service provider staff took great personal pride in achieving the key performance indicator levels.
Key interviewees indicated that the numerous management initiatives underway in the Branch and PWGSC have affected progress on the determination of the appropriate level of departmental resources to be allocated to support the AFD Initiative in the longer term. We found that the size and composition of the departmental teams vary between portfolios and continue to be the subject of discussions and reviews on a number of fronts. We did not find evidence that indicates that the uncertainty related to some transformation initiatives has affected AFD operations.
The National AFD Directorate has completed a benchmarking study of the resources needed to manage the AFD contracts. This study recommended that a Control and Risk Assessment be undertaken to address whether the Quality Management Framework represents a satisfactory approach to managing the critical risks associated with the eight contracts. The assessment is underway, led by the Office of the Chief Risk Officer, in conjunction with the Finance Branch and Legal Services. We expect to include the implementation of the study's recommendations, when we assess the overall cost-effectiveness of the AFD initiative in the Final Evaluation.
Measures of client satisfaction included:
Results of the National Service Call Centre surveys for 2005-06 ranged from 78.51% (for timeliness) to 87.5% (for ease of doing business) for all eight contracts. These results are considered very positive by the National Service Call Centre staff.
Interviewees considered the level and types of complaints received by the National Service Call Centre, the service provider, or Technical Authorities and their staff to be within normal ranges for a start-up year.
One of the Contractor Incentive Program target areas for 2005-06, concerned "satisfaction" as it related to a seamless transition. This element was measured in the National Tenant Satisfaction Survey. The results for the tenant portion of the survey ranged among the eight contracts from 90.57% to 96.06%, indicating uninterrupted operations during the transition of service providers, and, according to interviewees, this represented a very high level of tenant satisfaction.[
*
]
The underlying issues related to four of the five recommendations that stemmed from the phase 1 Interim Evaluation were examined as part of the research for this Evaluation. We have found that Real Property Branch had achieved progress in implementing its action plan on these for these four recommendations.
The one recommendation that was not examined in this Evaluation has been superseded by a departmental reorganization. The recommendation sought clarification of roles between the Real Property Branch and the Service Integration Branch that has been dissolved and the service areas related to real property assigned to the Real Property Branch.
The assessment of progress follows each recommendation. It had been recommended that the Assistant Deputy Minister Real Property Branch:
We concluded that, overall, there did not appear to be any issues significant enough to compromise the effectiveness of the 2005 AFD Initiative. We found that the Performance Management Framework had been implemented and was operating as intended. Further, high levels of client satisfaction had been achieved. Finally, Real Property Branch had succeeded in creating an environment where transformation approaches could be tested and refined while ensuring that the beneficiaries (occupants of federally owned facilities) were not negatively affected.
In addition to the competence of this service provider, we believe other factors contributed to these results, most notably, the Department's experience with AFD; the quality of the Statement of Work and Standard Operating Procedures; and the experience of Real Property Branch staff responsible for exercising contractual oversight.
Furthermore, we noted that management information systems such as the AFD Management System and the Quality Monitoring System were consistently implemented across all contracts. Performance indicators provided a clear description of the level of service being achieved, and actions for improvement were being taken as required. The results indicated that assets are being maintained per the Building Management Plans, and the Department service delivery outcomes were being met.
There were five recommendations that stemmed from the phase 1 Interim Evaluation. The recommended action for one of the recommendations has been superseded by a departmental reorganization that led to the dissolution of the Service Integration Branch. We found evidence that Real Property Branch had achieved progress on the implementation of the management action plan for the four remaining recommendations.
We noted one area for immediate improvement. The Real Property Branch has calculated the total costs for the AFD Initiative. However, it is essential that the definitions and basis for all elements of costs be clearly communicated to all persons using this type of data in the Department, to ensure that complete and correct information is used.
To ensure the continued success of the 2005 AFD Initiative and to be in a position to report on its effectiveness, we recommend that the Assistant Deputy Minister Real Property Branch ensure that:
Properties included in this Initiative have been grouped into eight portfolios, giving rise to an equivalent number of contracts. These portfolios are grouped along regional lines and comprise the following approximate 2005-2006 volumes:
| Portfolio/ Contract | Approximate Size (rm2) | Property Mngt. | Projects $5K to $200K | Projects $200K to $1M |
|---|---|---|---|---|
1. Atlantic |
273,013 | 21,581,932 | 8,244,895 | 0 |
2. Quebec |
381,500 | 28,702,356 | 11,937,522 | 1,261,453 |
3. Ontario |
338,422 | 27,734,272 | 8,223,142 | 424,956 |
4. Gatineau |
503,516 | 32,248,474 | 16,326,563 | 5,271,720 |
5. Ottawa Downtown |
435,034 | 40,020,496 | 18,805,065 | 2,540,959 |
6. Ottawa Perimeter |
510,897 | 34,454,704 | 17,420,415 | 3,447,725 |
7. Western |
294,474 | 22,412,130 | 6,595,953 | 0 |
8. Pacific |
194,151 | 13,024,132 | 3,743,191 | 297,100 |
Total |
2,931,007 | 220,178,496 | 91,296,745 | 13,243,913 |
Note: As of January 1, 2007, additional inventory had been added to existing contracts. Total meters squared are now approximately 3,145,256.
Text description for this Contract Portfolios Table is available on a separate page.