Evaluation of the Document Imaging Services Sub-program (1.7.4) (final report)
May 6, 2015
On this page
1. This report presents the results of the Evaluation of the Document Imaging Services sub-program located as item 1.7.4 on the 2014 to 2015 Program Alignment Architecture (PAA) for Public Works and Government Services Canada (PWGSC). This program first appeared in the PWGSC's PAA in 2008.
2. The Document Imaging Solutions Center (DISC) (the program) is situated in the Cheque Redemption Control Directorate (CRCD) located in the Accounting, Banking and Compensation Branch (ABCB).
3. This evaluation was included in the 2013 to 2018 Risk-Based Audit and Evaluation Plan. The Evaluation provides an assessment of the relevance and the performance of the Program for the period April 2012 to September 2014. Planning and research took place between May and December 2014, in accordance with the Standard on Evaluation for the Government of Canada and in conformity with the PWGSC Policy on Evaluation. More information on the approach, methodologies and limitations of this evaluation can be found in Appendix A: About the evaluation.
4. The Evaluation Directorate completed a risk-based calibration assessment to determine the best approach for the conduct of this evaluation. The assessment identified Document Imaging Services as an appropriate candidate for a streamlined evaluation for the following reasons:
- the Program represents less than 1% of PWGSC's annual spending
- the Program operates within a centralized delivery mechanism
- the Program experiences low sensitivity to public scrutiny
Why it is Important
5. In 2014, the Government of Canada launched "Digital Canada 150", a plan to guide its digital future. This future includes the digitization of government documents. For example, as of 2017, Library and Archives Canada will only accept from federal organizations, digital records and their associated metadata for archival purposes. It is estimated that up to 93.3 million pages of documentation are produced annually by federal departments and agencies. Digitization can assist federal departments and agencies in meeting their recordkeeping obligations. For these reasons, as well as to meet PWGSC's obligations under the Treasury Board Policy on Evaluation, it is important to periodically evaluate the relevance and performance of PWGSC's Document Imaging Services to support decision-making.
About the program
6. In 2001, using modern imaging technology, CRCD implemented an automated system in support of the Receiver General for Canada program that electronically reconciled paper cheques received from financial institutions. This automated system included a process of creating a digital copy (image) of the cheques (document) for the purposes of performing an online reconciliation within the accounting system. CRCD continues to perform this activity in-house for the Receiver General of Canada.
7. In 2002, CRCD began offering its document imaging services on a cost-recovery basis to other PWGSC programs, as well as to other federal departments and agencies, creating the standalone program called DISC. Document imaging (also referred to as electronic imaging or imaging in this report) is a technological process by which users scan hard copy documents (such as cheques, letters, forms, pictures, and/or microfilms) into a computer system and store them in digital format. The process includes indexing of the digital files and/or entering metadataFootnote 1 about the digital files into a computer system and saving the digital files on some form of storage technology (CDs, hard drives, servers) for query or retrieval purposes. The Program employs advanced imaging technology.
8. CRCD and DISC activities are performed in Matane, Quebec, using a common infrastructure of high volume production scanners. DISC is overseen by the Director General for CRCD and the Director located in Gatineau, Quebec, where there is a small project office for client engagement and partnership management.
9. DISC provides federal departments and agencies with a range of support services in the area of document imaging which are referred to as "Document Imaging Solutions". The document imaging process employed by DISC is detailed in Appendix B: Document imaging process. DISC initially works with its clients to identify the areas of support required (the business requirement definition phase, the analysis and configuration phase and the testing phase), and then provides the services selected by the client (the production phase).
10. DISC presents its services to its clients under the following 4 categories:
- Core: under this service level, there are no document preparation services provided in advance of scanning. Digital file processing is limited to a data capture of up to 8 characters. No quality assurance process is applied. Digital files are returned to the client by external media or an electronic file transfer
- Basic: under this service level, there are limited document preparation services provided in advance of scanning. There is greater data capture (that is file number and file name) than at the core service level. A quality assurance process is employed. DISC returns digital files to the client by external media or an electronic file transfer
- Enhanced: under this service level, the same services are provided as at the Basic level however there is also the possibility of receiving the client's documents from multiple platforms (scanner, fax, etc.). The documents go through a similar document preparation, scanning and quality assurance processes as under the Basic service level, however, more data is captured and data validation steps can be performed against data extracts provided by the client. At this service level, DISC can manage the digital files on behalf of the client whereby DISC gives authorized users access to the digital files via a secure web portal
- Comprehensive: under this service level, the same services are provided as at the enhanced level, however DISC can also provide support to its clients in the management of their own users. At the front end of its service delivery for this level, DISC can manage the receipt of documents from different technologies and from different users: documents can be received by DISC on behalf of the clients through a centralized PO Box, email and/or fax. At the tail end of its service delivery, DISC can interface with its clients' information technology (IT) when delivering digital files, a feature whereby DISC can directly enable its clients to manage their own client base
11. DISC has 3 established price points for 3 of its 4 service levels (basic, enhanced and comprehensive). DISC also offers other custom services for secret and archival documents for which a price-estimate can be provided.
12. Information on the type and volume of services conducted by the Program on behalf of its clients is detailed later in this report.
13. A logic model (Appendix C: Logic model) for DISC was prepared by the Evaluation team in consultation with the program. The logic model identifies DISC's intended outcomes and the activities undertaken by the program in support of their achievement. The following provides a brief description of the activities identified in the logic model:
- Outreach: consists primarily of:
- participation (through information kiosks or booths) at various government fairs and other events to promote PWGSC's services in this area
- maintenance of a presence in the federal government's record and information management community
- liaison with client service groups as services are included in Departmental Service Agreements and service standards are posted on the PWGSC internet website
- Client, costs, and performance management: consists mainly of:
- the conduct of client need assessments to assist federal departments in identifying their imaging requirements
- the development and management of the Memorandum of Understanding (MOU) between client departments and PWGSC
- the development and maintenance of the cost-recovery model according to the PWGSC cost recovery framework, the processing of account payables and receivables
- the development and monitoring of the service standards and performance measures
- the production of reports for different stakeholders
- Operations: includes processes, procedures and systems used to carry out the document imaging and related activities. The processes, procedures and systems are independently certified and DISC maintains an internal quality assurance function for its operations. The activities, which vary depending on the service levels chosen by the client, encompass the following categories:
- Centralized inbound mail management (including regular mail, courier and fax): receive correspondence and select, record track, open and prepare documents for scanning
- Scanning: scan documents into the file format selected by the client and check the quality of images with automated and manual tools
- Automated and manual data extraction and classification: index, classify and use automated software to extract key information (that is., metadata) from paper records
- Access, archiving and disposal: archive and provide record management services on a secure system. Images can be accessed through integration with client IT systems and online access can be provided using the government intranet. On-site physical document storage and shredding are also available to the client
14. In fiscal year 2013 to 2014, $11.45M of revenues and $11.42M of expenditures were reported; this included the salaries and associated costs of the program's 141 employees (approximately 107 Full-time employee (FTE)s).
15. DISC, as an in-house activity of the CRCD, was initiated under the authority of the Receiver General for Canada. No additional authority was sought for its provision of imaging services to other federal departments and agencies. PWGSC has the authority to charge and recover costs for services under the Department of Public Works and Government Services Act and the Treasury Board Common Services Policy. PWGSC's authority to spend the revenue generated by DISC is derived from the Treasury Board Policy on Special Revenue Spending Authority.
16. The results of the Evaluation research and analysis are presented in table format (see Section 1: Relevance and Section 2: Performance). The tables present the findings, evidence, conclusions and recommendations by evaluation issue. The indicators and data sources used to assess the evaluation issues are included in the table as well. This reporting format meets the requirements for evaluation reporting set out in Section 6.4.1a (i-ix) of the Treasury Board Standard on Evaluation for the Government of Canada.
17. There is a continuing need for DISC, and a growing need for advice and guidance in identifying document imaging and data capture needs in the emerging domain of document digitization. There is a strong current and projected need for document imaging services on the part of federal departments and agencies to meet their business and record keeping needs. DISC has responded to that need and is nearing maximum capacity.
18. Although some other government departments do their own imaging in-house, there are no other 'standalone' programs such as DISC within the Government of Canada. While similar services are available from the private sector, none of the firms on the Standing Offer provide the entire suite of services provided by DISC. Furthermore, the program is highly complementary with PWGSC's Receiver General for Canada and the Pension Programs. DISC's experience in defining imaging requirements and ability to provide an entire suite of services, combined with its secure operating environment, has laid the ground work for a new multi-channel enterprise solution to support the increased digitalization of Government of Canada documentation. The Evaluation found that DISC is currently in the process of identifying and analysing the business needs of federal departments with the intent of better defining its value proposition of the services it offers to clients.
19. While objectives for DISC have yet to be formalized the program was found to be aligned with government and departmental priorities, in particular those related to the achievement of efficiencies, the safeguarding of information and regional development. The provisions within the Public Works and Government Services Act provided a basis for the continuation of DISC as well as for the establishment of a common service within the Department to provide document imaging and data capture advice and services to other federal departments and agencies.
20. Performance results to-date for DISC are mixed. DISC services are compliant with international quality standards. The quality management system is International Organisation for Standardization (ISO) compliant and undergoes yearly audits to maintain the certification. The Evaluation determined that DISC has generally provided timely service to its clients; however, the Program has not collected data on all service standards related to timeliness. The Evaluation determined that DISC encountered challenges providing competitively priced services. Efficiency gains were observed in 2011 to 2012 and 2012 to 2013 fiscal years, however, we observed an increase in its average cost per page in 2013 to 2014. Program management is aware of its current barriers to competitiveness and is taking action to increase its efficiency in the future.
21. The Evaluation did not assess the extent to which there have been savings (economy) to the Government of Canada as a result of departments' and agencies' use of DISC service, nor through the provision of a common infrastructure for document imaging and data capturing services to other government departments and agencies.
22. Public Works and Government Services Canada (PWGSC) intends to transition from its current optional direct document imaging service fulfillment to an enterprise model that primarily uses the private sector to meet imaging needs, thus ensuring cost containment, best value for money as well as compliance with required Government of Canada standards and policies. Towards that end, a Request for Proposal (RFP) has recently been posted.
Recommendations and Management Action Plans
Recommendation 1: the Assistant Deputy Minister for Accounting, Banking and Compensation Branch (ABCB) should enhance the ongoing relevance of the Program by:
- formalizing the rationale and objectives for PWGSC's on-going provision of document imaging services to federal departments and agencies
- considering the value proposition offered by DISC in the context of the complementary and duplicative aspects of its services with the private sector
- Management action plan 1.1: A business case detailing the approach, rationale, objectives and benefits of the Enterprise Solution for Document Imaging has been developed. Communication materials highlighting the benefits and relevance of the Enterprise Document Imaging Solution will be developed
- Management action plan 1.2: ABCB will limit use of DISC internal services, conducting further analysis on those services once results of the RFP are known and vendors are on board
Recommendation 2: The Assistant Deputy Minister for Accounting, Banking and Compensation Branch should enhance the ability of the Program to achieve its outcomes related to pricing and efficiency by examining the potential for alternative delivery approaches.
- Management action plan 2.1: ABCB will be engaging client departments and confirming imaging volume/dollar commitments, driving volume price discounts
- Management action plan 2.2: ABCB to launch Enterprise Solution for Document Imaging that primarily uses the private sector to meet imaging needs. The RFP has been structured to enable competitive pricing and ensure departments pay only for the services they need. If approved, the mandatory aspect of the service will ensure volume discounts
Recommendation 3: The Assistant Deputy Minister for Accounting, Banking and Compensation Branch should enhance the performance measurement practices of the Program by streamlining the number of service standards in Memorandums of Understanding with clients and ensuring the on-going data collection and assessment of those found to be the most useful.
- Management action plan 3.1: ABCB will streamline the number of service standards in Memoranda of Understanding with clients, in support of the ongoing data collection and assessment of the service standards found to be the most useful
- Management action plan 3.2: ABCB will implement a quarterly dashboard to report on service standards
Section 1: Relevance
Continued need: The extent to which federal departments and agencies have a need for the type of services provided by Document Imaging Services.
|Current (2012–14) and projected (2014 and future exercices) need for document and data capturing services of federal departments and agencies.||
In 2007, a market segmentation study estimated that approximately 90 million pages of documents are produced annually by federal departments and agencies and that 650 million pages of legacy documents were yet to be imaged. A similar figure could not be found for the 2012–14 period.
According to a survey of 40 federal organisations conducted in July 2014 by the Program, the total need for document imaging services (based on current Business Plans from those organisations) is estimated at 1.65 billion pages and 227 million microfilms. However, not every document will undergo the imaging process since the final decision to digitize lies with the managers of federal programs; it is not clear when or if the remaining documents will be digitized.
As of 2017, Library and Archives Canada (LAC) will accept only digital records and their associated metadata for archival purposes. Departments will have to convert paper-based documents of enduring business value to digital images. Furthermore, the paper documents of business value currently in storage at LAC will be returned to departments. This will create additional pressure to image and to handle the returned documents on federal organizations in terms of record keeping and physical storage.
|Document review||There is a strong current and projected need for document imaging services on the part of federal departments and agencies to meet their business and recordkeeping needs.||N/A|
|Extent to which DISC has responded to a need for such services on behalf of federal departments and agencies.||In 2013 to 2014, DISC digitized 26.6 million pages for 8 federal departments and agencies, including PWGSC, at a cost of $11.42 million.||Document review||DISC has responded to a need for document imaging and data capturing services from federal programs within and outside of PWGSC. In fact, it is nearing its capacity to do so. Some federal programs outside of PWGSC have met their needs using in-house and private sector capacity. DISC redirects client requests to the RMSO that are not cost effective to the Program or outside of the Program capacity.|
|The Program reports that, as of 2014, it's on the verge of reaching its maximum capacity with its current infrastructure.||Interviews|
|The Evaluation found that some departments, such as Statistics Canada and Library and Archives Canada, meet part of their demand for digital imaging internally or through means other than DISC||Interviews|
|Exhibit 1: Document Imaging Solutions Center services in 2013–14 (in volume and percentage and Appendix D: Report exhibits provides detailed information on the volume per client department for 2013 to 2014. Over the course of 2009 to 2010 to 2013 to 2014, organizations within PWGSC have represented on average 25% of DISC' volume of work per year (pages imaged), as found in Exhibit 2: Document Imaging Solutions Center clientele distribution between Public Works and Government Services and other government departments and Appendix D: Report exhibits.|
|DISC redirects potential clients to a Regional Master Standing Offer (RMSO) with the private sector when the contract value is under $100,000. Under the RMSO, the private sector responded to 76 call-ups in 2013 to 2014, with a total value of $2 million for 18 federal organizations.||Procurement data review|
|DISC also redirects potential clients to the RMSO when they lack the necessary equipment (for instance digitizing microfilms). The Program was unable to identify how many potential clients they have redirected to other service providers since its establishment in 2005.|
Federal priorities and departmental strategic outcome: The extent to which the objectives of Document Imaging Services are aligned with federal government priorities and with PWGSC's Strategic outcome
|Existence of stated objectives for DISC||The Evaluation could not find evidence of formal objectives for PWGSC's provision of services in the area of document imaging. DISC was originally created to provide in-house services to the Receiver General for Canada. The extension of DISC's provision of services to other government departments and agencies was an innovative approach to make use of surplus capacity resulting from the financial industry's shift from paper to digital images in the clearing and settling of cheques which contributed to regional development and offset a forecasted decline in employment in Matane, Quebec.||Document review||While the Evaluation could not find evidence of formal objectives for DISC, the Program was found to be aligned with several Government priorities. Furthermore, DISC aligns with PWGSC's priority for the delivery of efficient and effective services.||The Assistant Deputy Minister for Accounting, Banking and Compensation Branch should enhance the relevance of the Program by:
|Extent to which DISC's objectives align with Government of Canada priorities for 2014 to 2015 and beyond.||Since the Evaluation could not find evidence of formal objectives, DISC activities were assessed for alignment with Government priorities. The Evaluation found DISC activities support the following Government of Canada priorities:
|Extent to which DISC's objectives align with PWGSC's strategic outcome for 2014–25 and beyond.||The PWGSC 2014 to 2015 Report on Plans and Priorities identified the delivery of efficient and effective services as one of its organizational priorities. The expansion of DISC was selected as a key initiative to support the Client Service and Integrity theme.||Document review|
Appropriate Role and Responsibility for the Federal Government: Does the provision of DISC services align with federal roles and responsibilities?
|Extent to which DISC complements, duplicates or overlaps with other current federal government programs.||
Program management reported that DISC services are connected to and consistent with federal government responsibility for managing documents securely and efficiently, although there is not an explicit linkage. For example, DISC provides an environment with controlled infrastructure where activities are carried out by public servants with valid security clearances. The evaluation found that a minority of DISC clients explicitly require such level of security to meet their service contracts.
Duplication or overlap
DISC directly complements 2 PWGSC programs (Receiver General for Canada Program and the Pension Program) while also supporting the data and document management requirements of other federal departments and agencies.
DISC is the only program offering imaging services to external clients. Some services offered by DISC, however, duplicate or overlap with the in-house imaging activities of other federal departments and agencies.
DISC's experience in defining imaging requirements could complement private sector services to fulfil those requirements. DISC's ability to provide an entire suite of services makes it unique from the private sector; however, there is a high degree of duplication and/or overlap with the individual services provided and the private sector. As a result there appears to be a value proposition for DISC services, but it needs to be clarified.
|Extent to which DISC complements, duplicates or overlaps with current services provided by the private sector.||
However, the Evaluation found that DISC has the experience and capacity to assist other government departments in identifying their imaging requirements. Further, PWGSC is uniquely placed to establish procurement instruments with the private sector. These 2 strengths could be further developed to allow DISC to advise other government departments on the establishment of requirements and to support them in having those requirements met by the private sector through PWGSC-established procurement instruments.
Duplication or overlap
It should be noted the existing "scanning and data conversion" RMSO only qualifies firms to provide a limited range of services up to a maximum of $100,000. However, federal organizations can contract a broader range of services from the private sector through other procurement vehicles such as a Request for Proposal.
|Evidence of policy and/or legislative authority for the provision of document imaging services.||
PWGSC has the authority to charge and recover costs for services under the Department of Public Works and Government Services Act and the Treasury Board Common Services Policy. Sections 5, 6, 7 and 15 of the PWGSC Act authorize the Department to operate as a common service agency for the Government of Canada; the Minister to acquire and provide services for departments; to develop services for increasing the efficiency and economy of federal public administration; and provide information management and information technology systems and services.
|Document review||The PWGSC Act provides the Department with the authority to provide information management services to other government departments, and to develop services for increasing the efficiency and economy of federal public administration.||N/A|
There is a continued need for document imaging services by federal departments and agencies. DISC lacks formal objectives but supports federal and departmental priorities. While there is duplication and overlap between the individual services provided by DISC and those of the private sector and/or internal operations within other government departments, DISC is a unique program in the Government of Canada and its services align with PWGSC's legislated authorities. PWGSC's experience in defining imaging requirements and ability to provide the entire suite of services may provide an indication of the value the Department can provide to its clients; however DISC's value proposition could be better defined.
Section 2: Performance
Outcome achievement: Immediate outcome—The extent to which DISC is offering federal institutions competitively priced document imaging and data capturing solutions.
|Comparison of DISC service range and pricing to other service providers||
The program, however, reports that exact service and pricing comparisons are difficult to obtain in the document imaging industry and that benchmarking averages may not provide an accurate portrait. The program references, as an example, a recent large project comparison it completed which assessed DISC pricing as 20% lower than the private sector for the services required.
Further, the program's obligation to operate on an annual, cost recovered, breakeven basis creates challenges in estimating the price of its services, which can have an impact on its clients. The MOU between DISC and its clients provides an estimated total cost per page (variable and fixed). Variable costs to clients are more easily defined than fixed costs, as the per-page fixed cost cannot be calculated until DISC's annual production volume is known at the end of the fiscal year.
In prior years, the Evaluation observed the actual cost per page was less than estimated cost, resulting in a reimbursement to clients. The Program reports that some clients were not receptive to reimbursement as it represented, for them, unspent funds at the end of the fiscal-year.
||DISC has encountered challenges providing competitively priced services. DISC provides, on an annual, cost-recovered, breakeven basis, a range of services similar to those available from the private sector. There is evidence that the services provided by DISC may be more costly than similar services provided by the private sector. Furthermore, the pricing strategy employed by the program to ensure a cost recovered, break even position on an annual basis, impacts its ability to accurately estimate pricing which has had an impact on some clients.||The Assistant Deputy Minister for Accounting, Banking and Compensation Branch enhance the ability of the Program to achieve its outcomes related to pricing and efficiency by examining alternative delivery approaches.|
Outcome achievement: Immediate outcome—The extent to which DISC offers document imaging and data capturing solutions in compliance with international quality standards.
|Extent to which DISC has received and maintained certification and/or recognition from international standard organizations||The quality management system implemented by the CRCD has been assessed and found fully compliant with the requirements of ISO 9001:2008. The certificate was issued by the "Bureau de Normalisation du Québec" and is valid until January 2016.||Document review||DISC is providing services which are compliant with international quality standards: its quality management system is ISO compliant and undergoes yearly audits to maintain the certification.||N/A|
|Extent to which DISC has implemented and maintained activities to ensure their compliance with international quality standards||
CRCD has a team of internal auditors who are trained and qualified to perform audits on all the elements of the ISO certification.
A yearly audit maintains the certification and a tri-annual audit renews the certification (the certification expires in January 2016). If recommendations arise from the internal or annual audits, DISC modifies its procedures accordingly.
ISO specific tests are performed daily on the scanners.
DISC also performs specific quality assurance checks at the request of clients.
Outcome achievement: Intermediate outcome—The extent to which Federal institutions receive efficient and timely document imaging and data capturing solutions.
|Extent to which the program has been able to manage business volumes to achieve economies of scale. (Efficiency)||
Exhibit 3: Total costs and imaging volumes compared to cost per page for fiscal years 2010 to 2011 to 2013 to 2014 (Appendix D) illustrates DISC's total cost and imaging volumes for fiscal years 2010 to 2011 to 2013 to 2014. The Evaluation's analysis of this data found that the volume of documents imaged by DISC increased by 418% from 2010 to 2011 to 2013 to 2014. Over the same period, total costs increased by 258%. The result was a decrease in the cost per page from $.70 in 2010 to 2011 to $.43 in 2013 to 2014. While the latter cost (as noted in the assessment of Immediate outcome 1) is still higher than the comparable private sector average cost, it represents a 39% decrease in the cost per page over this period for the Program.
The economy of scale realized by DISC has been passed on to its clients through reductions in the fees charged. While DISC can determine with reasonable precision the variable costs that should be charged to each client, at the time service is requested they are unable to allocate a portion of the program's fixed infrastructure costs until the annual volume of activity is known. Consequently, any economy of scale realised by DISC, based on total volumesprocessed by DISC, are applied at year-end, where they are allocated proportionately.
|Document review||DISC has achieved varying economies of scale in recent years of operation, which has contributed to efficient services to its clients. It has also resulted in cost savings to clients within the 2011 to 2012 and 2012 to 2013 fiscal years. The program has also seen its average cost per page rise in 2013 to 2014.||The Assistant Deputy Minister forAccounting, Banking and Compensation Branch enhance the ability of the program to achieve its outcomes related to pricing and efficiency by examining alternative delivery approaches.|
|Client perspectives on receipt of efficient services (efficiency)||A majority of clients (5/9) interviewed expressed a high level of satisfaction overall with the DISC services. Furthermore, clients indicated that: quality mattered more to them than price; and they did not actively look into the private sector offerings when deciding upon a service provider.||Interviews|
|Percentage of time-based service standards met as per service level agreements. (timeliness)||
The Evaluation reviewed 14 client service agreements, each containing up to 7 service standards related to response time and/or timeliness of service provision over the course of completing a contract for a client. The evaluation found that only one of these 7 indicators is being measured on a regular basis, and it serves as the main indicator of timely service.
With regards to this main indicator, DISC documentation indicates that it has exceeded its established target of meeting the standard: "provision of images to the client within a specified timeframe" 95% of the time over the last 4 years.
|Document review||DISC has provided images to its clients within specified timeframes most of the time within the last 4 years. The program, however, has not maintained data collection on all service standards related to timeliness.||The Assistant Deputy Minister for Accounting, Banking and Compensation Branch should enhance the performance measurement practices of the program by streamlining the number of service standards in Memorandums of Understanding with clients and ensuring the on-going data collection and assessment of those found to be the most useful.|
|Client perspectives on timeliness of service provision. (timeliness)||
Almost all clients interviewed (6/8) expressed satisfaction regarding the time taken by DISC to carry out imaging services (once the MOU was in place). Some interviewees also expressed satisfaction with DISC's proactive nature and its willingness to provide answers and fixes when an issue arose.
Nevertheless, some interviewees expressed concerns that the length of the project start-up phase was excessive, particularly in regard to the time it takes for the Program to present an offer to potential clients (MOU phases).
Conclusion: Performance (outcome achievement)
DISC has encountered challenges providing competitively priced services. The services meet international quality standards. DISC has made progress towards providing federal institutions with efficient document imaging and data capturing solutions as it has been able to achieve some economies of scale in its operations as volume of activity increases. DISC's performance measurement reporting indicates that it has succeed in generally providing timely services to clients, however, data collection and reporting on service standards addressing timeliness has been limited.
Efficiency and economy: is defined as the extent to which resources are utilized in relation to the production of outputs and outcomes. Efficiency refers to the extent to which resources are used such that a greater level of output is produced with the same level of input or, a lower level of input is used to produce the same level of output. Economy refers to minimizing the use of resources in the pursuit of outcomes. A program has high demonstrable economy and efficiency when resources maximize outputs at least cost and when there is a high correlation between minimum resources and outcomes achieved. Efficiency and economy was assessed through multiple lines of evidence including document review, data review and interviews.
In the course of evaluating DISC, efficiency and economy were evaluated as core components of the Program's immediate and intermediate outcomes. It was found that DISC had made progress towards providing federal institutions with efficient document imaging and data capturing solutions when it has been able to maintain economies of scale in its operations (efficiency) but had encountered challenges providing competitively priced services (economy). The Evaluation, however, did not identify the savings (economy) to the Government of Canada through the use of DISC services by government departments and agencies (such as through related to reduced or eliminated paper file storage costs as a result of document imaging and data capturing solutions).
DISC has achieved a level of efficiency as a result of applying economies of scale: it has reduced its average cost per page from $.70 in 2010 to 2011 to $.43 in 2013 to 2014.
DISC demonstrated a level of economy at the time of its establishment (pre-evaluation scope) through its re-use of existing government resources to implement this program for a cost-avoidance equivalent of $3.5 million (from 2003 to 2008), however, the Evaluation was unable to determine the extent of savings to the Government of Canada through the provision of a common infrastructure for document imaging and data capturing services. The program operates under the PWGSC Cost Management Framework, using the net voting funding mechanism, and therefore has the obligation to cost recover and break even at each fiscal year end. The Evaluation found that over the last 5 years, the Program has had a deficit once and a surplus on 4 occasions as show in Exhibit 4: Document Imaging Solutions Center Revenues, Expenditures and Balances, 2009 to 2010 through 2013 to 2014 (appendix D); this is impacting DISC's ability to demonstrate economy as well as indicating the impact of inconsistent unit pricing, an issue discussed within this report under the assessment of Immediate outcome 1.
Conclusion: Performance (efficiency and economy)
DISC had made progress towards providing federal institutions with efficient document imaging and data capturing solutions when it has been able to maintain economies of scale in its operations (efficiency), but has encountered challenges providing competitively priced services (economy). The program operates under a cost-recovery, break-even model, but has been unable to achieve a neutral balance at the end of recent fiscal years, having experienced both a deficit and surpluses. The Evaluation did not assess the extent to which there have been savings (economy) to the Government of Canada as a result of departments' and agencies' use of DISC service, nor through the provision of a common infrastructure for document imaging and data capturing services.
Appendix A: About the evaluation
This evaluation was included in the 2013 to 2018 Risk-Based Audit and Evaluation Plan.
The Evaluation examined Document Imagining Services, delivered by the Cheque Redemption and Control Directorate within PWGSC. This evaluation had 2 objectives:
- To determine the relevance of the program: the continued need for the program, its alignment with governmental priorities and departmental outcomes and its alignment with federal roles and responsibilities
- To determine the performance of the program: the achievement of its expected outcomes and a demonstration of efficiency and economy by the Program
Approach to the evaluation of Document Imaging Services
The Evaluation Directorate completed a risk-based calibration assessment to determine the best approach for the conduct of this evaluation. The assessment identified DISC as an appropriate candidate for a streamlined evaluation for the following reasons:
- the program represents less than 1% of PWGSC's annual spending
- the program operates within a centralized delivery mechanism
- the program experiences low sensitivity to public scrutiny
The level of effort allocated to the conduct of the Evaluation reflects the above factors. An evaluation matrix was developed using the Treasury Board's 5 core issues to be addressed in evaluations. The following data sources were used to inform the lines of evidence to answer the questions in the evaluation matrix:
- Program document review: the preliminary document review provided information on the Program and its context to assist in the planning of the Evaluation. A more comprehensive program document review was conducted to collect information to answer the evaluation questions. The program document review consisted of primary documents that pertained to DISC: client files (including agreements, contracts, invoicing and payment records); program administration, monitoring and reporting documents (such as internally- and externally-focused reports); meeting minutes, client surveys, program-led studies; and financial records
- Literature review: the preliminary literature review provided information on the Program and its context to assist in the planning of the Evaluation. A more comprehensive literature review was conducted to collect information to answer the evaluation questions. The literature review consisted of secondary documents that pertained to DISC: departmental documents (such as the Report on Plans and Priorities and the Departmental Performance Report); government documents (such as legislative and policy documents); and external publications
- Financial analysis: financial data related to the Program's budgets, revenues, and expenditures were reviewed, including contract information and service fee structures. This information informed the analysis of program efficiency and economy
- Interviews: a total of 14 structured interviews, using an interview guide, were held with key program staff (n=3), clients (n=9), and officials with other federal organizations (n=2)
Limitations of the evaluation
The planning of the Evaluation applied a risk-based assessment to determine the most appropriate approach and level of effort to be applied in order to ensure the availability of timely and objective information to meet the needs of senior management. In the application of the approach, several issues were encountered. None of these issues were significant enough to prevent evaluation reporting.
- Literature review: the Evaluation was unable to locate data for the total value of imaging work provided internally by other federal departments and agencies and the Evaluation had limited access to data for the work provided externally by the private sector. the parameters of the Evaluation did not permit the research necessary to establish such data. The Evaluation was also unable to find benchmarks to compare the quality of DISC and private sector services. Consequently, the Evaluation relied primarily on the views of the Program's clients to provide an indication of the relative quality of services
- Interviews: the evaluation's interview invitations had a 37% response rate (9 out of 24)
- Surveys: the parameters of the evaluation did not include the administration of a survey. The evaluation relied on existing data available through 2 Program-directed surveys conducted prior to this evaluation
- Financial analysis: documents on program revenues were provided at an aggregate level, which impeded the Evaluation's ability to analyze aspects of the Program's performance in a more detailed manner. Moreover, the Program changed its methodologies for the calculation of costs and revenues over the period of 2011 to 2014 which made comparisons difficult
Based on the lines of evidence, the Evaluation team prepared a Director's Draft Report presenting the findings, conclusions, and recommendations. This draft report was reviewed by the Quality Assessment and Practice Improvement Directorate within the Office of Audit and Evaluation and by the Chief Audit and Evaluation Executive. Subsequently, it was sent to senior program management for review and comments. Validation of facts and any other comments on the Director's Draft Report were assessed and, where appropriate, incorporated in the Chief Audit and Evaluation Executive's Draft Report. This report was sent to the Office of Primary Interest for final acceptance. A management response to the report and a management action plan in response to the recommendations was requested. The draft Final report, management response, and management action plan were tabled at the Audit and Evaluation Committee for the Deputy Minister's approval in March 2015. The Final report will be submitted to the Treasury Board Secretariat and posted on the PWGSC website.
The Evaluation was conducted by employees of the Office of Audit and Evaluation, overseen by the Director of Evaluation and conducted under the overall direction of the Chief Audit and Evaluation Executive. The evaluation was reviewed by the Quality Assessment and Practice Improvement Directorate of the Office of Audit and Evaluation.
Appendix B: Document imaging process
Document imaging process description
The image depicts the Document Imaging Process in sequential order. The image begins with Documents & Mail, which leads to Document Preparation, leading to Imaging, which then leads to Quality Control. Following Quality Control is Classification and Indexing, which leads off to 2 separate processes: Archiving, and Digital Support.
Both Archiving and Digital Support lead to Client Users. Archiving leads to Client Users through Image Transfer. On the other hand, Digital Support leads to Client Users first through Client Database, and then through Image Transfer. There is a 2-way relationship between Archiving and Client Users: once Archiving (through Image Transfer) proceeds to Client Users, it then comes back through a step entitled Queries. There is also a 2-way relationship between Client Database and Client Users: once Client Database (through Image Transfer) proceeds to Client Users, it also comes back through Queries.
Appendix C: Logic model
Logic model description
The logic model depicts the logic model of Document Imaging Services. From the top down, the model describes DISC's program in terms of its activities, outputs, immediate outcome, intermediate outcome, and PWGSC's strategic outcome, and the linkages between them.
The activities are separated into 3 categories:
- Outreach: Promote services at events and fairs; Liaise with the PWGSC Client Service group
- Client, Cost and Performance Management: Conduct client need assessments; Develop and maintain MOUs; Develop cost-recovery model; Process account payables/receivables; Monitor and report on performance; Develop and apply service standards
- Operations: Prepare documents and handle bulk mail; High-volume scanning; Capture, extract and index data; Provide secure storage of paper and digital images; Archive and dispose (secure destruction) of documents; Provide technical support and informatics; Monitor quality and compliance
Outreach has a one-way relationship with Client, Costs and Performance Management. Client, Costs and Performance Management has a 2-way relationship with Operations.
The activities under Client, Costs and Performance Management are linked to the first section of Outputs. The activities under Operations are linked to the second section of outputs.
The first section of the programs outputs are:
- cost-recovery model; Signed MOUs; Invoices, payments, and receipts; Client and departmental service standards; Performance reports
The second section of the program outputs are:
- imaged documents; Metadata; Client IT infrastructure interfaces; Stored and indexed information; Secure online client access; Secure destruction
The outputs are linked to the immediate outcome.
The program's immediate outcome is:
- document Imaging Services offers federal institutions competitively priced document imaging and data capturing solutions in compliance with international quality standards
The immediate outcome is linked to the intermediate outcome.
The program's intermediate outcome is:
- Federal institutions receive efficient and timely document imaging and data capturing solutions
The intermediate outcome is linked to PWGSC strategic outcome.
Public Works and Government Services Canada Strategic Outcome
PWGSC's strategic outcome is:
- high-quality, central services and programs that ensure sound stewardship on behalf of Canadians and meet the program needs of federal institutions
Appendix D: Report exhibits
Exhibit 1: Document Imaging Solutions Center services in 2013 to 2014 (in volume and percentage)
The exhibit depicts the volume in pages and percentage for "Other government departments (OGDs)"
|Other government departments||Volume (in pages)||Volume (in %)|
Veteran Affairs Canada
Department of National Defense
Privy Council Office
Library and Archives Canada
The exhibit depicts the volume in pages and percentage for "PWGSC programs".
|Public Works and Government Services Canada programs||Volume (in pages)||Volume (in %)|
PWGSC—Industrial Security Sector
PWGSC—Parliamentary Precinct Branch
PWGSC—Cheque Redemption Control Directorate
Grand DIS Total
Exhibit 2: Document Imaging Solutions Center clientele distribution between Public Works and Government Services and other government departments
The exhibit represents the Document Imaging Solutions Center clientele distribution between Public Works and Government Services and other government departments
|2009 to 2010||19%||81%|
|2010 to 2011||31%||69%|
|2011 to 2012||17%||83%|
|2012 to 2013||27%||73%|
|2013 to 2014||30%||70%|
Exhibit 3: Total costs and imaging volumes compared to cost per page for fiscal years 2010 to 2011 to 2013 to 2014
The exhibit depicts the total costs and imaging volumes compared to cost per page for fiscal years 2010 to 2011 to 2013 to 2014. On the left side of the table, vertically, are the variables Volumes (pages), and Total costs ($), descending from 30,000,000 to 0 in increments of 5,000,000. On the right side of the table, vertically, is the variable Cost per page, descending from $0.80 to $-, in $0.10 increments. Horizontally are the fiscal years categorized into 4 groups, which are 2010 to 2011, 2011 to 2012, 2012 to 2013, and 2013 to 2014. Directly below that indicates which color the line of each variable will be on the table.
The exhibit has 3 lines, one representing Total costs, another representing Volume (pages), and one representing Cost per page. For Total costs, the line increased from 2010 to 2011 to 2013 to 2014, starting at $4,420,165 and ending at $11,423,350. Volumes (pages) showed a significant increase from 2010 to 2011 to 2013 to 2014, starting at 6,349,868, and ending at 26,567,365. Finally, Cost per page saw a decline from 2010 to 2011 ($0.70) to 2012 to 2013 ($0.34), but began to slightly rise again by 2013 to 2014 ($0.43).
Exhibit 4: Document Imaging Solutions Center Revenues, Expenditures and Balances, 2009 to 2010 through 2013 to 2014
The exhibit depicts the revenues, expenditures and balances for DISC from Fiscal year (FY) 2009 to 2010 to 2013 to 2014.
|Fiscal year||2009 to 2010||2010 to 2011||2011 to 2012||2012 to 2013||2013 to 2014|
- Date modified: