2005-715 Audit of Contract Quality Control and Quality Assurance Functions, Acquisitions Business Line, Final Report

2007-09-28

Table of Contents

• Main Points
  • What We Examined
  • Why it is Important
  • What We Found
• Introduction
• Background
• Focus of the Audit
• Observations
  • Section 1: Contract Quality Control
    • 1.1 Lack of formal procedures to ensure high quality and timely QC review.
    • 1.2 Sufficient documentation to substantiate decisions and actions not always evident in file.
    • 1.3 Efficiency of QC process not measurable.
    • 1.4 Efficiency of QC process not measurable.
  • Section 2: Contract Quality Assurance
    • 2.1 Five informal QA groups identified.
    • 2.2 QA localized with limited sharing or legacy of improving procurements.
      • 2.2.1 Networking
      • 2.2.2 Automated Events Tracking and Warning System
      • 2.2.3 Cost Audit Program
• Conclusions
  • Recommendations
  • Management Response
• About the Audit

Main Points

These Main Points are not a stand-alone document and must be read in conjunction with the body of the report.

What We Examined

The focus of our audit was to provide assurance on the existing quality control (QC) and quality assurance (QA) functions within the Acquisitions business line, and to conclude on its strengths, gaps, effectiveness and efficiency in maintaining and improving the quality of procurement. QC was defined as the overall system of activities whose purpose is to provide a quality product or service that meets the needs of users. QA is defined as a system of activities whose purpose was to assure that the overall quality control was being done effectively.

We examined a sample of 515 contracts and concluded on contracting activities from the identification of a requirement by a client department until the contract award stage. Our findings and conclusions relate only to the sample of 515 contracts we audited, and should not be attributed to all contracts. The Audit of Contract Management - Roles and Responsibilities was undertaken concurrently with this audit and examined the same contracts from the award stage until completion of the requirement.

Why it is Important

Throughout the life of a contract, PWGSC and ultimately its Minister, remains the contracting authority of record. A governing principal of contracting is the integrity of the procurement process, for which PWGSC is accountable. Insufficient consideration of contracting QC and QA can negatively impact the acquisition of quality products or services to meet client's requirements.

Given the current rate of retirement and attrition, the risk of this occurring is increased as a result of the loss of experienced procurement officers who perform QC and QA reviews.

What We Found

Although QC reviews were seen by Acquisitions business line as beneficial to the provision of a quality product or service that met the needs of users, the process is labor intensive. Positive actions to enhance the QC process and the efficiency of contracting include:

• current and ongoing departmental actions for plain language templates;
• modifications to contract terms and conditions; and
• ongoing commodity strategies.

We also determined that increased quality, rigour and efficiencies could be gained through the:

• creation of processes, procedures, and standards for when, under what circumstances, and how QC reviews are to be conducted;
• determination of a balance between the number and structure of reviews versus the delivery of timely contracting services;
• mandatory reviews at the solicitation stage rather then contract approval stage; and
• reinstitution of pre and post award QC sampling and review of contracts, using a standard, computerized program that inputs into a QA program.

Within the Acquisitions business line there are a number of groups that provide an informal contract QA. However, there is no existing centralized program. The Acquisitions business line would benefit from the reinstatement of a centralized QA program that identifies trend analysis, lessons learned, and best practices. Such a program would increase performance of individual procurement officers, increase sharing of commodity knowledge, justify broad training requirements, and identify systematic local problems.

The Acquisitions business line could also improve the effectiveness and efficiency of contracting and better protect the financial interests of the Crown through improvements to existing informal QA, such as the Automated Events Tracking and Warning System, and more consistent, effective, and efficient use of the Contract Cost Audit program.

The audit makes two recommendations. The Assistant Deputy Minister, Acquisitions Branch needs to:

1. develop a common and consistent Quality Control review process from solicitation to contract approval in order to identify and document when, under what circumstances, and how QC review is to be conducted; and
2. formalize a Quality Assurance program and incorporate lessons learned into the contracting process. The QA program should include the development of: a mandated post-award sampling review program; a centralized QA program; a systematic means of identifying contracts with an audit clause and selecting contracts for cost audit; and a formal program for sharing lessons learned.

Management Response

The report's findings relate to a sample of 515 contracts awarded up to March 31, 2005, and not all contracts awarded by PWGSC. Management accepts these findings and agrees that a formal common and consistent quality control review and quality assurance program are important.

Over the past two years during which the audit was conducted, the department had been taking actions to address the report's findings and recommendations. More specifically, the Associate Deputy Minister and responsible PWGSC senior management have examined and clarified the department's roles and responsibilities. Further, the Acquisitions Branch is in the process of and will continue to lead a significant change management initiative through procurement transformation which includes actions in six key areas – changes to procurement policy, streamlining military procurement, improvements and efficiencies to contracting processes, modify relationships with stakeholders, increase people capacity and training, and modernize acquisition tools. Finally, in May 2007, the Department launched a six-point action plan to strengthen procurement activities.

In addition to these actions, the Acquisitions Branch understands that the Audit and Evaluation Branch and Office of the Chief Risk Officer will be considering the audit observations and recommendations contained in other various audits and reviews related to procurement activities and will be identifying lessons learned. The Acquisitions Branch looks forward to the results of this exercise.

Introduction

This audit was proposed as an addition to the 2005-2006 Audit and Evaluation Plan. It was undertaken to provide assurance and information on contract quality control (QC) and quality assurance (QA) functions within the Acquisitions business line of PWGSC.

Background

For the purpose of this audit, the definition of contract QC and QA activities were based on the Quality Assurance Policy of the former Supply and Services Canada, Deputy Minister's Directive #661 (1993). QC is defined as: "the overall system of activities whose purpose is to provide a quality product or service that meets the needs of users."  The aim of QC was to provide quality that was satisfactory, adequate, dependable and economic. QA is defined as: "a system of activities whose purpose was to assure that the overall quality control is being done effectively." This was to be achieved through the continuous evaluation of the adequacy and effectiveness of QC with a view to having corrective measures initiated where necessary.  

Throughout the life of the contract, PWGSC and ultimately its Minister, remains the contracting authority of record. The Acquisitions Branch is the office of primary interest for issues related to procurement, and provides operational authority, direction, and notification of changes in policy or practices to all procurement staff in PWGSC. Acquisitions Branch provides direct authority to procurement staff employed in the Sector Headquarters and functional authority for procurement staff employed in the regions or other branches. The term "Acquisitions business line" refers to all procurement staff in the Acquisitions Branch and in regional offices.

In the early 1990s, there was formal QC and QA functions in the Acquisitions business line. A Contract Quality Assurance Manual and a Procurement Quality Assurance System software program were developed. Gradually over time, the QC and QA functions and the related reporting decreased. Although the Manual was dated it was found to contain good ideas and a formula for post-award sampling.

The contract QC functions within the Acquisitions business line continue to be focused on ensuring that contracts comply with government and departmental policies, procedures, and guidelines. During the pre-award contracting process, the series of internal reviews by senior, experienced contracting personnel tend to be compliance oriented and are a primary means of identifying errors in specific contracts. The benefits of consistent delivery of a product or service to a defined standard include:

• lower costs;
• shorter turnaround times;
• reduced challenges;
• greater availability of resources for other productive pursuits;
• identification of potential problems; and
• enhanced learning.

The contract QA functions within the Acquisitions business line are decentralized to each of the sectors and regions, although the Headquarters Contractual Risk and Quality Control Division provides some QA advice in its review of all submissions. Normally, value-added QA requires the continual evaluation of the adequacy and effectiveness of the QC function, through ongoing QC data analysis and feedback, which identifies the causes of errors and seeks to mitigate them. Types of errors that a QA function can discover and mitigate include:

• officer error;
• lack of training and skills;
• insufficient leadership;
• improperly designed or written policies or guidelines;
• inefficient tools and systems; and
• client's perceptions of quality and timely performance.

The 2005 Budget outlined the Government's procurement transformation actions to: improve service quality and efficiency; strengthen governance and accountability; and, provide sound stewardship of public resources. In being tasked with this transformation, as well as the forecasting of savings for procurement activities, PWGSC has a responsibility to both the client departments and the public. Insufficient consideration of contracting QC and QA can negatively impact PWGSC's credibility, client departments programs, as well as the vested public trust.

Focus of the Audit

This audit was part of the Audit and Evaluation Branch's strategy to review selected departmental monitoring activities. Positive assurance on the function of QC and QA would:

• provide greater impact on the organization subject to audit;
• identify gaps that could strengthen future monitoring activities; and
• limit the requirement for large comprehensive internal audits;

The focus of this 2005 audit was to provide assurance on the existing quality control (QC) and quality assurance (QA) functions within the Acquisitions business line and to conclude on their strengths, gaps, effectiveness and efficiency in maintaining and improving the quality of procurement.

We examined a sample of 515 contracts and concluded on contracting activities from the identification of a requirement by a client department until the contract award stage. The Audit of Contract Management - Roles and Responsibilities was undertaken concurrently with this audit, and examined the same 515 contracts from the award stage until the completion of the requirement. More information on the objectives, scope, approach, and criteria can be found in the section: About the Audit.

Observations

Section 1: Contract Quality Control

Interviews with procurement staff identified that a contract QC is undertaken based on three modes of operation. The first is a ‘Directive QC': the PWGSC Supply Manual directs the circumstances under which a series of QC reviews and approval steps are taken prior to contract award. The dollar amount of the requirement and the type of commodity usually determines the number and type of review steps. Steps can include reviews by:

• the Procurement Review Committee;
• senior management prior to approval of a Contract Planning and Advance Approval or a Procurement Plan document;
• Contractual Risk and Quality Control Division (CRQC);
• cost analysts; and
• Legal Services.

The second is a ‘Subjective QC': as the name implies, the circumstances requiring review and the steps involved are at the discretion of the procurement manager. Reasons for conducting a subjective QC include the:

• recognition of the level of expertise and/or training of the responsible procurement officer and/or the client Project Authority;
• management style and availability of the procurement manager or team leader;
• existing practices within the region/sector; and
• counsel of special advisors to their corporate heads.

The third is an ‘Objective QC': this includes informal criteria that signal a high-risk procurement and therefore the need for a QC review. Such criteria include unique, complex, highly technical, and/or sensitive requirements.

1.1 Lack of formal procedures to ensure high quality and timely QC review.

Procedures, processes and standards provide information on when, under what circumstances, and how to conduct QC activities. They also help clarify expectations and timelines for the activities. Clear communication and documentation of QC activities help ensure consistent, high quality, appropriate and timely conduct of the required activities.

We expected that the Acquisitions Branch would have formal documented procedures or standards for the conduct of contract QC across its business line.

We observed that there were limited procedures, processes or standards supporting QC activities. In the absence of formal documents, procurement staff indicated that managers and team leaders based their QC review on personal procurement knowledge or experience, and accepted practices within their area of work. In the sample of contract files we examined, in which there was evidence of a QC review, we observed inconsistent approaches across the Acquisitions business line.

Staff confirmed that the quality, and therefore value added, of QC reviews was often linked directly to the individual conducting the review. Reviews by individuals with limited procurement knowledge and experience did not provide significant QC value, as there was limited consistency of comments and those comments received were mostly editorial rather than content based.

Limited procedures or standards for contract QC directly impacted the value of those reviews. This was intensified in situations where the reviewer lacked significant experience. Given the current rate of retirement and attrition of experienced senior procurement officers, the need for guidelines to ensure consistent, high quality, appropriate, and timely conduct of reviews is critical.

1.2 Sufficient documentation to substantiate decisions and actions not always evident in file.

Documentation serves as a historical record of actions, decisions, and procurement activities that occurred during the life of the contract, and provides evidence in the event of a financial review or subsequent legal action. As well, they facilitate management oversight and the transfer of information to procurement officers who may consult the file or assume responsibility at a later date.

We expected that procurement officers would comply with the PWGSC Supply Manual and Treasury Board Contracting Policy, which require that procurement files provide a complete audit trail of relevant communications and decisions, including sufficient documentation on QC reviews and the resulting consideration given to the reviewer's comments.

We noted that the files varied in consistency and completeness of documents retained and on file. Procurement staff confirmed that QC reviews are tracked through e-mails, notes to file, approval documents, and draft documents containing comments and edits. They indicated the varying levels of documentation in each file was based on the practice or culture of the office/region /sector.

We concluded that in some cases the Acquisitions business line lacked sufficient documentation on file to substantiate decisions and actions taken with regards to QC reviews. As a result of varying styles and practices, there is a risk that the department will not have sufficient, relevant, and accurate records to support and defend itself in response to audit inquiries, complaints to the Canadian International Trade Tribunal, or legal actions.

1.3 Efficiency of QC process not measurable.

A business process requires a balance between costs (time and resources) to undertake a review, and the achievement of a quality final service. Initially, there is a linear relationship between QC and a quality final service. The more reviews, the greater potential of a high quality and professional contract. However, at some point, no matter how many reviews are completed, the quality of the final product is not increased.

We expected that Acquisitions business line would have an efficient QC contracting process that demonstrated efficient use of resources and achieved a high quality final product or service.

Staff indicated that, on average, the number of reviews ranged from none for a single one-time purchase of a good that used a template and checklist, to greater then ten for a complex procurement.

There was a scope limitation in our audit for the measurement of efficiency of the contract QC review. The audit program initially contained a section to record the number of working days required to complete a QC review. However, due to the limitation of documentation and the lack of recorded dates (sent/receipt), we could not quantify the level of effort or time to complete the QC reviews.

We noted that the plain language template or standard templates for a particular commodity or group, was a useful tool that increased the consistency and quality of the contract terms and conditions portion of the document, and reduced QC review time and level of effort. However, these templates do not reduce the need to review the client department's evaluation criteria and statement of work. The QC review for that portion of the document is critical given that a quarter of the complaints brought to the Canadian International Trade Tribunal was due to improper evaluations.

A procurement officer must review, validate, reply, and document consideration of comments by each reviewer, regardless of whether or not the comments are timely, consistent, or relevant. As a result of the increased numbers of reviews over the years there has been a decreased efficiency and increased costs (time and resources) in the QC review process.

Twenty-five years ago, a seasoned procurement officer would have gone through fewer steps to obtain approval for a $2.5 million competitive contract for Architectural Engineering services. A maximum of 12 persons would have provided QC reviews.
Today, a seasoned procurement officer has to go through multiple steps for the same type of contract. A minimum of 30 persons within PWGSC review, analyze and make changes to the submission and covering memorandum.

1.4 Benefit of QC review is influence by its timing.

A well-functioning QC ensures the consistent delivery of a product or service to a defined standard. The benefits include lower costs, shorter turnaround times, reduced challenges, greater availability of resources for other productive pursuits, identification of potential problems, and enhanced learning.

We expected that the Acquisitions Branch would have a process to strengthen QC reviews and actions to resolve any weaknesses.

One of senior management's roles in QC is approval of the contract strategy prior to the preparation of a solicitation document in the Contract Planning and Advance Approval or a Procurement Plan document. Procurement staff felt that the most common strengths of the QC review process were the:

• increased consistency and completeness of the final product;
• incorporation of correct/up-to-date policies, terms and condition clauses, and trade acts;
• assurance that the correct method of solicitation was used; and
• perception by client departments and the public of being more professional when fewer solicitation or contract amendments were required.

Only the Western Region continued to have a formal process for QC review of a sample of pre-award contracts within its site offices. We observed there was evidence of a contract quality assurance checklist and comments by a review officer for the contract files we sampled. The benefits of these checklists were:

• recorded compliance to contracting policy and trade acts;
• verification of the completeness and accuracy of solicitation and contract documents;
• opportunity for performance feedback, corrective behaviour and practices, and acknowledgement of good files and initiatives; and
• development of increased procurement knowledge and skills for the responsible procurement officer and for the reviewer.

Procurement staff felt that the most common weaknesses were: the consumption of time and resources in a tight timeline; reviewers having insufficient procurement experience to make informed comments; and little performance incentive to change a laborious process.

A QC review of the solicitation document prior to public tendering helps with the clarity, accuracy, and completeness of the document, and increases the transparency, fairness and rigour of the procurement process. The two main QC groups within Headquarters, CRQC and Legal Services, felt that a weakness in QC was due to the timing of the review, as directed by the Supply Manual. A mandatory independent pre-award QC review was done, as the last step, prior to contract approval and signing. Once a competitive procurement was completed, it was almost impossible, to correct inappropriate contract terms and conditions, lack of performance criteria, or tailored specifications; even when the risks to the Crown were high. This limited the benefit of QC.

Section 2: Contract Quality Assurance

2.1 Five informal QA groups identified.

Within the Acquisitions business line, we identified a number of groups that provided an informal contract QA function. These included:

1. Post-award reviews. Two regional offices – Pacific and Western – continue to use a post-award QA program similar to one that was developed and reported to Acquisitions Headquarters until early 2000. Other regions and sectors used a similar program, but on an ad-hoc basis. The Pacific Region sampled contract files on a monthly basis, based on a random selection of completed files. The Western Region sampled contracts files twice a year, based on the dollar-value, buyer selected, and completed files. This Region assigned the review to the regional QA manager or to a team leader in another site office. The post-award QA program:
   • encouraged consistency among site offices;
   • determined the effectiveness of quality control measures that were in place;
   • provided an opportunity to review commodities from other site offices that may not be seen in their regular line of business; and
   • identified process improvements, training needs and resource distribution.

2. Networking of the CRQC and regional QA managers. In the past, the Headquarter Contract Quality Assurance Branch's mandate was to: facilitate and coordinate quality assurance activities throughout the department; coordinate and communicate departmental guidelines and best practices; and promote the development of education, training, and support services. The current role and responsibility of the CRQC is to: review of all contract documents that require a Senior Director level and above; review of solicitation documents on an ad-hoc basis; provide comments and advice on procurement plans and contracts; and/or comment on policy changes or procurement issues.

   Currently, regional QA managers and a CRQC manager hold regular teleconferences to discuss changes in policies, issues related to procurement process, or other related topics.

3. Other Oversight Groups. Legal Services provides information sessions on legal requirements and informal presentations on Canadian International Trade Tribunal decisions.
4. Automated Events Tracking and Warning System. In 2004-05, the Acquisitions Branch piloted an Automated Events Tracking and Warning System for the purpose of improved contract management. Over 900 procurement officers were trained to use the Automated Buyer Environment Milestones. This system replaces the practice of creating tasks or reminders in a procurement officer's personal e-mail account, and assists in timely actions, which ensure continued post-award contract quality control, including:
   • exercise of options;
   • renewal of contractual arrangements;
   • delivery and performance;
   • receipt of invoices, reports, progress meetings;
   • negotiation of rates; and
   • audit undertakings.
   Reporting occurs monthly or an ad-hoc basis. The use of the system for all new contracts was mandated by the ADM, Acquisitions Branch in 2004/05.

5. Cost Audit Program. The Contract Audit Group, within the Policy, Risk, Integrity and Strategic Management sector, administers the Cost Audit Program. The main objectives of Cost Audit Program are to provide assurance that the contracts under review do not contain a material amount of loss due to excessive profits or inaccurate or unreasonable costing practices; to deter contractors from initiating unacceptable practices by maintaining a sufficient level of audit presence; and to provide quality control information on the acquisitions process, including contract management.

   The Program continues to be an integral part of the Acquisitions business line and serves to protect the financial interests of the Crown through systematic audits of contracts that PWGSC has awarded. The cost audits are carried out by Audit Services Canada on a fee for service basis under a Memorandum of Agreement. The Program has four major components:

   • Sample – random sample of applicable contracts issued each fiscal year;
   • Priority Audits –contracts issued to specific contractors where it has been determined that there may be risk of making overpayments on contract activity;
   • Major Crown projects – cost reimbursable contracts are to be audited according to departmental policy; and
   • Other – ad-hoc requests, pre-negotiation reviews for cost analysts, termination claims, and audits on behalf of other North Atlantic Treaty Organization countries under reciprocal agreements.

   The advantages and impediments to each of the informal QA groups are discussed further in the next sections.

2.2 QA localized with limited sharing or legacy of improving procurements.

A contract QA program benefits from an effective contract QC program, in that QA systematically reviews and analyzes information generated by QC and provides feedback to improve the overall procurement function. A robust contract QA program is demonstrated through best practices, lessons learned, trend analysis and is reflective in training requirements, changes in policies or the Supply Manual, changes in templates or contract terms and conditions, and communication of contracting issues.

We expected that the Acquisitions business line would have a QA process that analyzes information generated by QC, and provided feedback and actions to improve the overall procurement function. As mentioned in the previous section, the informal QA processes we identified are pre and post award review, networking communication, other oversight groups, Automated Events Tracking and Warning System, and the Cost Audit Program.

2.2.1 Networking

Procurement staff felt that training or forums to discuss contracting issues such as QA are important to the development of procurement officers. However, we were told that such events were not provided on a regular basis.

Less formal site office meetings include a variety of contracting issues, such as:

• current Canadian International Trade Tribunal determinations;
• problems or lessons learned on a specific contract file;
• e-mails from regional offices or Headquarters regarding changes to policies or terms and conditions;
• changes to commodity strategies or practices; and
• information shared at management or director meetings.

Other sources of QA information included Real Property Branch's national contracting group meetings and best practices; mentors or coaches for new or less experienced procurement officers, internet web sites for Canadian International Trade Tribunal cases; and courses for certification in procurement.

Regional QA managers and a CQCRD manager felt that the major impediments to QA were:

• a laborious process with no limited results;
• limited availability of resources with contract QA experience, and a high learning curve;
• a shift in mindset to look at more than just straight compliance (details versus content);
• downsizing and re-organization, which diverted the duties of QA managers to other related areas, such as Automated Buyer Environment, policy implementation, and training; and
• no existing automated program or method to collect pertinent contract quality data or trend analysis. The software, developed in 1994, no longer functioned in the current technical environment.

We observed that the Acquisitions business line had no web sites or electronic bulletin boards to post lessons learned or to identify problems and suggested solutions. Within regional or sector groups there is person-to-person/group or e-mail traffic as a means of sharing and seeking responses, but rarely does the search extend beyond such a community. As these discussions are rarely documented, persons not attending do not benefit from these discussions.

2.2.2 Automated Events Tracking and Warning System

The main advantage of the Automated Events Tracking and Warning System is the early identification of actions that could increase economic savings, efficiencies, and effectiveness for client department programs. However, there has been limited uptake in the use of this system. As of the first quarter of 2006, just over half of these new contracts included in our sample had been assigned a Contract Management milestone.

2.2.3 Cost Audit Program

The main advantages of the Cost Audit Program cost audits include:

• protection of the Crown's interests;
• recovery of overpayments or excessive profit;
• ability to track good contractors who maintain documentation and controls;
• identification of potential compliance issues and procurement issues related to policies or contracting terms and conditions;
• assistance in studies for the negotiation of labour rates and profit ceilings; and
• deterence since the contractor knows at any time its financial/payroll records could be audited.

In any given year, there are few contracts that meet the requirements of high material value or cost reimbursable requiring an audit clause. The limiting factor on the number of cost audits undertaken in a given year was the budget of the Cost Audit Program. We noted that the Program had no systematic method, such as a code in the Acquisitions Information System to identify the total population of contracts that contained an audit clause. Instead, potential contracts were identified through an annual call letter to procurement officers and costs analysts. This does not result in the most efficient use of the cost auditors, in the initial stage of sampling. For example, if the total population of applicable contracts is known, cost auditors can examine a single contractor's records for a multiple of contracts. This would:

• provide a means of tracking good contractors;
• reduce annual reviews for known compliant contractors in the preparation of future cost audits; and
• reduce annual reviews for known compliant contractors in the preparation of future cost audits; and

We observed that each of the above-identified groups had a role in the function of a QA program. However, there was limited sharing of information or lessons learned among the Acquisitions business line as a whole. Procurement issues that were identified in post-audit reviews were discussed within the specific office/region/sector at the time of the review. Reports on the Automated Events Tracking and Warning System were provided to senior management. Procurement issues identified in cost audits were discussed with the procurement officer responsible for the specific contract and their managers, and incorporated into a trend report.

However, this information was not always shared with all procurement officers or changes made to policies or terms and conditions as a result of the QA.

One of the greatest challenges of the QA function was the limited documentation, hardcopy or electronic, to pass on to new or future procurement officers which could prevent the reoccurrence of similar types of procurement issues.

Conclusions

We reviewed a sample of 515 contract files in order to verify the integrity and completeness of the QC controls in monitoring of the quality of a contract, and to identify QA lessons learned or improvements in service quality and efficiency. Our conclusions relate only to the sample of 515 contracts we examined and should not be attributed to all contracts.

We concluded that a common and consistent process that formally identifies when, under what circumstances and how QC reviews are to be conducted could enhance the quality, rigour and efficiency of QC review. Further, to minimize the time and maximize the benefit of QC review, a balance should be struck between the number and structure of reviews versus the delivery of contracting services. We also concluded that QC reviews from groups such as Legal Services, the Office of the Chief Risk Officer, and QA managers were more valuable at the solicitation stage than at the contract approval stage, since limited modifications were possible at the latter stage due to a competitive process. Finally, we believe that pre and post award QC sampling and review of contracts, using a standard, computerized program that inputs into a QA program, would enhance the quality and rigour of contracts across the Acquisitions business line.

We also found that departmental actions for plain language templates, modifications to the Standard Acquisitions Contract Clause manual, and ongoing commodity strategies have enhanced the QC process and the efficiency of contracting.

We identified a number of informal QA programs, however, there was no centralized program that identified trend analysis, lessons learned, and best practices. We believe that formalization of the existing programs, and enhancement of the means to ensure the contracting process is improved as a result of what is learned through the QA programs would improve the effectiveness and efficiency of contracting.

Recommendations

We made two recommendations. The Assistant Deputy Minister, Acquisitions Branch needs to:

1. develop a common and consistent Quality Control review process from solicitation to contract approval in order to identify and document when, under what circumstances, and how Quality Control review is to be conducted; and
2. formalize a Quality Assurance program and incorporate lessons learned into the contracting process. The Quality Assurance program should include the development of: a mandated post award sampling review program; a centralized Quality Assurance program; a systematic means of identifying contracts with an audit clause and selecting contracts for cost audit; and a formal program to share lessons learned.

Management Response

The report's findings relate to a sample of 515 contracts awarded up to March 31, 2005, and not all contracts awarded by PWGSC. Management accepts these findings and agrees that a formal a common and consistent quality control review and quality assurance program are important.

Over the past two years during which the audit was conducted, the department had been taking actions to address the report's findings and recommendations. More specifically, the Associate Deputy Minister and responsible PWGSC senior management have examined and clarified the department's roles and responsibilities. Further, the Acquisitions Branch is in the process of and will continue to lead a significant change management initiative through procurement transformation which includes actions in six key areas – changes to procurement policy, streamlining military procurement, improvements and efficiencies to contracting processes, modify relationships with stakeholders, increase people capacity and training, and modernize acquisition tools. Finally, in May 2007, the Department launched a six-point action plan to strengthen procurement activities.

In addition to these actions, the Acquisitions Branch understands that the Audit and Evaluation Branch and Office of the Chief Risk Officer will be considering the audit observations and recommendations contained in other various audits and reviews related to procurement activities and will be identifying lessons learned. The Acquisitions Branch looks forward to the results of this exercise.

About the Audit

Objectives

The objective of this 2005 audit was to provide assurance on the controls and monitoring of quality of contracts within Acquisitions business line, which would support improvements in service quality and efficiency, strengthen governance and accountability, and provide sound stewardship of public resources. Specifically the audit was:

• To identify the various contract Quality Control (QC) and Quality Assurance (QA) groups within the department;
• To assess the strengths and weaknesses of the contract QC and QA programs;
• To identify significant gaps or risks in the contract QC and QA programs; and,
• To determine the effectiveness and efficiency of the contract QC and QA programs and the extent to which these activities meet Acquisitions Branch's objectives.

Scope and Approach

The detailed examination phase was conducted from June 2005 to May 2006. Work related to another audit - the Audit of Contract Management - Roles and Responsibilities was undertaken concurrently.

The observations on QC were determined through a review of a statistical sample of contract files for compliance to the procurement regulations and policies. The observations on QA were determined through interviews, documentation, and the lessons learned from the sample review. We selected the population of contract files from data in the Acquisitions Information System.

The parameters for the selection were:

• All commodities – goods, services, construction, telecommunications, architectural and engineering, repair and overhaul;
• All active contracts as of March 31, 2005, based on the expiry date, as well as, those contracts that may have expired but were awarded between January 01, 2004 to March 31, 2005;
• All original contracts that were valued at or above Schedule 3, Level 1 delegation of authority; and
• All standing offers (SO), supply arrangements (SA), and call-ups awarded from January 2004 to March 2005.

Contracts with delegated approval authority lower than Schedule 3, Level 1 were excluded from the audit's scope because most QC groups were not required to review and provide comments on contractual documents at that level. As well, we excluded contracts from the Communications commodity, since an Audit of Advertising was underway in 2005-2006. We also excluded contracts from the site offices of Germany and Great Britain, since a contract review had already been conducted in the first site, and the other office had been closed.

The total population was provided to a statistician from the former Consulting and Audit Canada. He selected an independent, statistically determined sample of 515 contract files. The sample was based on a 90 % confidence interval and a 10 % margin of error, and included contracts from the eight entities within the Acquisitions business line. These entities are: three Headquarters Sectors  - Land, Aerospace, Marine Systems (LAMS), Commercial Acquisition and Supply Management (CASMS), and Services and Technology Acquisition Management (STAMS); and five regions – Atlantic, Quebec, Ontario, Western, and Pacific. Although, Acquisitions Branch had reorganized its structure in 2005/06 to include an additional Sector – Services and Specialized Acquisitions Management - we reference the grouping consistent with the Acquisitions Information System contract data existing at the start of the audit.

We examined the sample of contract files and relevant documentation in the regional offices from June to August 2005, and in the Headquarters Sectors from January to May 2006. Interviews were held with more than 70 PWGSC directors, managers, supply specialists, and procurement officers from across all eight entities.

The audit program, completed for each contract, recorded the audit results from identification of a requirement by a client department until award of the contract. The audit program assessed  contracts in terms of compliance with the Supply Manual and government and departmental policies. Compliance was recorded as an affirmative or negative. Non-compliance was not weighted in terms of a minor or major impact/risk. All the sampled files we examined were under the custody of the responsible procurement officer.

We expected that procurement officers would comply with the PWGSC Supply Manual and Treasury Board Contracting Policy, which require that procurement files provide a complete audit trail that contains details of relevant communications and decisions. If however, documents were missing from the files the auditors asked or e-mailed the responsible procurement officer to validate if there was additional documentation. We did not request that client departments or Real Property Services property managers provide any additional or missing documentation from their files. 

One scope limitation was the evidence to determine the efficiency of contract QC review. The audit program for the first entity reviewed contained a requirement to indicate the level of effort or time to complete the QC review. The intent was to provide an average amount of time that was expended in the undertaking of a QC review. However, given a limitation of documentation on file and lack of recorded dates the requirement was removed from the audit program for the other entities and added to the interview questions.

We analyzed the audit results and consolidated into a summary report for each of the eight entities. The audit reported on significant observations and conclusions, which were inclusive to the Acquisitions business line and supported by relevant and specific examples. This audit was conducted in accordance with the standards of the Treasury Board Policy on Internal Audit, and with the Internal Auditing Standards for the Government of Canada.

We prepared and validated audit findings and conclusions with departmental senior management prior to preparation of a final report. Upon acceptance of the draft report, the final report was presented to the Deputy Minister, and the PWGSC Audit and Evaluation Committee.

Audit Criteria

The criteria used in this audit were:

• The contract files sampled should demonstrate compliance with the standards of the Government Contract Regulations, the Supply Manual, and government and department policies, such that they stand the test of public scrutiny in matters of prudence and probity, facilitate access, encourage competition, and reflect fairness in the spending of public funds.
• The contract QC programs should ensure the consistent application of standards to procurements; demonstrate mechanisms to detect and correct the underlying causes of errors; and, identify process improvements, training, and resource distribution needs.
• The contract QA programs should continually evaluate the adequacy and effectiveness of the QC program with a view to correcting the function of it, where necessary.
• The contract QC and QA programs should support the achievement of the Acquisitions Branch's targets of increased effectiveness and efficiency.

Audit work completed

Audit work for this report was substantially completed by May 2006.

Audit Team

Staff of the Audit and Evaluation Branch conducted the audit, under the direction of the Director, Internal Audit Services Directorate, and the Director General, Audit and Evaluation Branch, as Chief Audit Executive.