2005-613 Evaluation Framework for PWGSC's Shared Travel Services Initiative (STSI), Final Report

September 8, 2006

Table of Contents

• Executive Summary
• 1. Introduction
• 2. Profile
  • 2.1 Background and History
  • 2.2 Governance
  • 2.3 Resources
  • 2.4 Delivery of Travel Services
  • 2.5 Expected Benefits
• 3. Logic Model
  • 3.1 Objectives
  • 3.2 Activities
  • 3.3 Outputs
  • 3.4 Immediate Outcomes
  • 3.5 Intermediate Outcomes
  • 3.6 Final Outcomes
• 4. Evaluation Strategy
  • 4.1 Evaluation Issues
  • 4.2 Methodology
  • 4.3 Subsequent Evaluations
• 5. Recommendation
• Appendix A - Principal Documents Consulted
• Appendix B - Interviews Conducted
• Appendix C - Description of a Logic Model

Please note that under the Access to Information Act a limited amount of text within these documents may not be disclosed, and will be shown as [ * ].

Executive Summary

This Evaluation Framework for Public Works and Government Services Canada (PWGSC) Shared Travel Services Initiative (STSI) was developed under the authority of the PWGSC 2005/06-2007/08 Multi Year Evaluation Plan.

The Government-Wide Travel Modernization Project was announced in 2001 as a Major Crown Project and as a joint effort between PWGSC and Treasury Board of Canada Secretariat (TBS). In 2002 a procurement process was initiated for the provision of an integrated "end-to-end" travel management service that would provide a range of travel services including a full-services travel agency, on-line reservation service, travel expense claim service, travel card service, traveler's reimbursement service, business intelligence capabilities by individual departments and by government as a whole, an employee portal, and an employee traveler service network. In December 2003, the Government-Wide Travel Modernization Office was transferred to PWGSC and was renamed the Shared Travel Services Initiative.

The objective of the Evaluation Framework is to provide management with recommendations for an evaluation study.

The Evaluation Issues are categorized within the following areas as per TBS policy:

• Relevance
• Design/Delivery
• Success/Impacts
• Cost-Effectiveness/Alternatives

Relevance

1. Is STSI consistent with PWGSC priorities and its Commodity Management role?

2. Is STSI a priority for TBS/Office of the Comptroller General?

Design/Delivery

3. What impact did contract design, bid solicitation process and evaluation have on the STSI project team's capacity to develop and implement a modern, cost-effective Government travel service?

4. What mechanisms do STSI have in place to manage and ensure a seamless transition to the new shared Travel Program?

5. Is the Business Case and underlying assumptions valid?

Success/Impacts

6. Are current and projected uptake levels capable of generating desired benefits as indicated in the current business plan?

7. Are departments and agencies engaged and exercising stewardship in regards to the transition to the new shared Travel Program?

8. To what extent has the STSI improved the travel services to government employees and enhanced the travel experience for public servants?

9. Is the increased travel information supporting better travel decisions within Other Government Departments (OGDs)?

10. To what extent has STSI increased transparency and accountability of travel carried out by employees within OGDs?

11. To what extent is STSI serving as a pathfinder or benchmark for all aspects of shared services model?

Cost-Effectiveness/Alternatives

12. To what extent has STSI enabled the Government to achieve net savings in actual travel costs and support Expenditure Review Committee (ERC)?

13. To what extent has the STSI reduced travellers' and OGDs' administrative costs?

14. Does STSI remain affordable?

PWGSC will require early feedback to ensure that the STSI is being carried out as designed and intended and that a smooth transition towards the new shared Travel Program is occurring. The required feedback can be obtained by conducting a formative evaluation. The estimated cost for the formative evaluation is $94,000 in professional services, and 70 OAE person-days. The formative evaluation will be conducted through the use of document review, selected interviews, focus groups, departmental case studies and data available through Travel AcXess Voyage (Contractor).

It is recommended that:

1. The CEO, Information Technology Services Branch, accept this evaluation framework and the strategy outlined as the basis for undertaking the formative evaluation to begin in November 2005.

1. Introduction

The Evaluation Directorate of Public Works and Government Services Canada (PWGSC) developed this Evaluation Framework for PWGSC's Shared Travel Services Initiative (STSI) with the assistance of Consulting and Audit Canada (CAC).

The Evaluation Framework was developed in consultation with the STSI project team. A review of documentation and existing reports (see Appendix A - Principal Documents Consulted) and key stakeholder interviews (see Appendix B - Interviews Conducted) were undertaken in order to identify evaluation issues that reflect areas of interest to management.

This document provides a profile of STSI, describes its goals, presents a logical sequence of activities and outcomes, identifies evaluation issues, and provides an evaluation strategy to measure the outcomes and impacts of key activities, as well as the proposed methodology to be employed. The objective of the Evaluation Framework is to provide management with recommendations for an evaluation study.

2. Profile

2.1 Background and History

The delivery of government programs and operations often depends on travel. The Treasury Board Travel Directive (October 2002) and the Travel Administration Guide^[1] govern travel by public servants employed by the Treasury Board. Responsibility for travel administration has been delegated to deputy heads of departments. The Travel Program for the Government of Canada (GoC), located within PWGSC, was initially a grouping of services provided to public service employees, which dealt with the provision of Travel Agency services along with a directory for Accommodation and Car rental businesses.

In 1995, in the Chapter on Travel and Hospitality, the Auditor General found that the management and accountability for travel within the GoC could be improved. It noted that with respect to the management of public servant travel, information was not organized for senior management to efficiently assess the need for and costs of travel; more emphasis on a values-driven system may lead to more cost-effective travel and better employee morale; opportunities for automation and streamlining of procedures would have the potential for improving control and reduce the cost of travel administration; and the pursuit of these opportunities required focused efforts and co-operation among central agencies and departments.

A review of government travel was undertaken by the Treasury Board Senior Advisory Committee (TBSAC) and, in September 2000, Treasury Board of Canada Secretariat's (TBS), Government Travel Modernization Office (GTMO) obtained the endorsement of the TBSAC for an 'end-to-end' solution that proposed a government-wide implementation strategy to modernize GoC travel processes, services and systems. The Government-Wide Travel Modernization Project, now called the Shared Travel Service Initiative (STSI), was announced in 2001 as a Major Crown Project and as a joint effort between PWGSC and TBS. The Initiative supports the mandate of TBS and aligns with other TBS led initiatives, including Human Resources (HR) Modernization (workplace improvement), Modern Comptrollership (stewardship) and Shared Services.

The GTMO would be working towards an e-commerce travel solution for Public Service employees intended to enhance accountability and save the government time and money in continued efforts to be more 'employee responsive' and focused on improving service to Canadians. TBS amended the Travel Policy in May 2001 to reflect the evolving travel environment and the new business principles in government travel management, which are: trust, respect, flexibility, transparency and valuing people. The contractual arrangement in place with Ryder for the delivery of travel services was ending in March 2002, with a one-year option remaining.

In 2002 a procurement process was initiated for the provision of an integrated "end-to-end" travel management service and in January 2003, a contract for the delivery of government travel services was awarded to a new supplier, Accenture, and its team of subcontractors, American Express, Concur Technologies and Bell Canada who collectively are known as Travel AcXess Voyage. It is a $275M 7-year contract with two option years to provide a range of travel services including a full-services travel agency, on-line reservation service, travel expense claim service, travel card service, traveler's reimbursement service, business intelligence capabilities by individual departments and by government as a whole, an employee portal, and an employee traveler service network. The goals and objectives, with performance targets, were established and indicators formed part of the contract.

The project office, staffed in early 2003, waited until December 2003 before the details of the contract with Accenture was finally negotiated and formally awarded. After the contract was signed, the contractor resources were brought on stream. In December 2003, GTMO was transferred to PWGSC and was renamed the Shared Travel Services Initiative although it remained a joint project between TBS and PWGSC.

The 2003 Business Case observed that total travel spend for the federal government has increased, on average, 8.3 per cent per year for the last five years. Over the course of the nine-year STSI contract (2003/04 -2011/12), total travel spend was estimated to reach close to $18.3 billion. The Business Case was revisited in September 2005 updating assumptions and using actuals to refine projections. The total travel spend is now projected to be $15.7 billion. In Budget 2005, the Expenditure Review Committee (ERC)^[2] identified savings to be realized from "Government procurement - Travel modernization" of $375M over the next five years.

The new Travel Program will include the total array of services from the Travel Agency, On-line Booking and Travel Card along with a global portal, training, reporting and expense management services.

2.2 Governance

PWGSC's STSI was initially located in the Service Integration Branch (SIB). As one of the key components of The Way Forward, it was moved to the Acquisitions Branch (AB), incorporating the PWGSC Travel Program. Since June 2005, PWGSC's STSI resides in the Information Technology Services Branch (ITSB), with the Travel Program group reporting both to AB and STSI.

The key committees guiding STSI are the following:

Senior Project Advisory Committee (SPAC): In June 2005, the Assistant Deputy Minister (ADM) Advisory Committee evolved to become the SPAC. It is comprised of 10 Senior Executives from small, medium and large departments and co-chaired by the Comptroller General of Canada and the CEO, ITSB. The Committee's roles include: ensuring effective strategic direction; engaging the travel industry to adopt best practices; providing recommendations in the areas of employee support, reducing the administrative burden and improving cost efficiencies; ensuring that STSI maintains strategic fit with HR modernization and ERC; providing advice to complement negotiations with unions and National Joint Council (NJC); and endorsing emerging plans, monitoring progress on goals and objectives and providing overall direction and leadership for the Initiative.

Travel Commodity Council^[3]: responsible for negotiating with travel providers for additional discounts in the areas of airlines, rail and bus; accommodations; car rental services; and meeting and conference planning services. Membership of this committee will be aligned with the SPAC.

STSI Executive Committee: responsible for the overall management of STSI, and its successful delivery across the GoC. This Committee should provide strategic leadership and direction as well as providing timely resolution of issues. It is comprised of senior executives from PWGSC, TBS, Accenture and other key stakeholders.

Interdepartmental Travel Users Working Group (ITUWG): comprised of DG/Director level and Senior Full-time Travel Officers (SFTOs). This committee supports the collaborative working relationships between the STSI and the departments.

2.3 Resources

As of August 2005, the STSI employed 43 FTEs of which 15 are from the Travel Operations Directorate, and 15 are full-time consultants. Actual level of staff is below the planned number for this period. It is expected that these numbers will be reduced by about 50% early in FY 2006/07 as many secondments are ending.

Exhibit 1

STSI and Travel AcXess Voyage Budget - (Cumulative up to June 30^th, 2005)

	Cumulative Data
	FY 2003-04 (J/F/M)	FY 2004-05	FY 2005-06 (A/M/J)	Total Project
STSI Project Office Team Budget	$2,065,532	$9,733,481	$8,017,427	$19,816,440
Travel AcXess Budget	$1,756,800	$22,643,200	$0	$24,400,000
Total Project Budget	$3,822,322	$32,376,681	$8,014,427	$44,216,440
Actual Expenditure STSI Project Office	$2,214,400	$7,518,071	$1,927,722	$11,660,193
Actual Expenditure Travel AcXess	$2,196,000	$7,200,000	$0	$9,396,000
Total Actual Expenditure	$4,410,400	$14,718,071	$1,927,722	$21,056,193
STSI Budget Variance	($148,868)	$2,215,410	$6,089,704	$8,156,247
Travel AcXess Budget Variance	($439,200)	$15,443,200	$0	$15,004,000
% STSI Budget Spent	107%	77%	24%	59%
% Travel AcXess Budget Spent	0%	32%	0%	39%

2.4 Delivery of Travel Services

The STSI is structured following from the decision to use the private sector to develop and deliver a range of travel services that include^[4]:

• Electronic travel and interoperability services, which will include:
  • Electronic travel portal services
  • Electronic travel planning and authorization (including travel authorization number issuance and traveler profile creation and maintenance)
  • Electronic self-service booking
  • Electronic travel expense management
  • Government travel analysis and reporting
  • Data interfaces with existing government financial systems
  • Connectivity to other government services, systems and capabilities
• Full-service travel and fulfillment services, which will include:
  • Full service travel agency services
  • Electronic fulfillment services (for electronic bookings through an electronic travel self-service booking tool)
• Travel financial services, which will include:
  • Travel financial instrument services (designated travel card and traveler's cheques)
  • Alternative approach to reimbursement of travelers (to be implemented at the discretion of the Crown)
• Employee travel support and training services, which will include:
  • Traveling employee support network
  • Employee familiarization and training

The Contractor

The roles of the private sector team, Travel AcXess Voyage, including the subcontractors, are described below:

Accenture Inc.: the prime contractor, responsible for overall program and project management, systems integration and development of the electronic travel services component of the project.

American Express: a sub-contractor, supplying a full suite of travel services, which includes the online booking tool, the travel agency, the corporate travel card and traveling employee support network.

Bell Nexxia: a sub-contractor, supplying infrastructure connectivity for the employee travel portal and the expense management application (including secure channel connectivity).

Concur: a sub-contractor, providing expense management tool and travel authorization software for the solution.

The STSI Project Team

The STSI Project Team is responsible for management and oversight of the travel services contract, for commodity management, for business intelligence and reporting to departments, and for providing the following services in support of the client departments and other stakeholders:

• Travel Program governance;
• Project Management;
• Travel Policy Interpretation;
• Risk and Issue Management;
• Client engagement activities, including implementation support, benefits management, and financial interface support;
• Change management activities, including training, process mapping and redesign, departmental readiness and knowledge transfer;
• Communications activities, including stakeholder status reporting, departmental communications, and marketing management;
• Technology implementation activities, including system development oversight, data conversion, acceptance testing, and business intelligence management; and
• Travel operations activities including contract monitoring, management of air and hotel agreements and travel services support.

Treasury Board of Canada Secretariat

TBS is responsible for the management of policy and the government side of labour relations in regards to the Travel Directive.

Public Works and Government Services Canada

Within PWGSC:

• Acquisition Branch is responsible for the procurement function and contract management. The contract authority reports both to Acquisition Branch and the STSI Project Executive Director.
• Services Integration Branch manages the relationship with departments.
• Corporate Services area reports on the The Way Forward savings.
• ITSB manages and delivers the services of the Secure Channel Network through a team of suppliers (Team Bell Canada Enterprises (BCE)) that includes BCE Emergis, Bell Nexxia, CGI, Cisco, Compaq, Entrust and IBM. Key security features provided by this government electronic service delivery include: confidentiality, authentication and integrity. To provide STSI's expense management and financial operability components, and raise adoption on key services, delivery through the Secure Channel Network is needed.

Key Stakeholders

Key stakeholder groups include: all government departments and agencies in the areas of travel administration (including SFTOs), HR, finance, systems and change management (corporate); National Joint Council (NJC); unions; TBS in the areas of travel and financial policy; private sector contractors involved; public servants (travellers); and the travel industry.

2.5 Expected Benefits

Qualitative benefits: The 2003 STSI Business Case indicated its intention to provide a substantial qualitative improvement over the current travel service baseline with the use of the currently provided services (travel agency, online booking tool and travel card). The quality and scope of improvements of the entire initiative over the current baseline include:

• Improved web-based travel services and travel information available from desktop, laptop, or personal digital assistant.
• Electronic submission of travel requests.
• Electronic routing of travel requests and expense claims.
• E-mail notification of status, approvals and payments.
• Electronic signatures and approvals.
• Electronic expense claim processing fully integrated with travel bookings.
• Automated policy compliance audit on electronic expense claim.
• Online business intelligence and standardized and online reporting capabilities.
• Online referral to policy and other entitlements.
• Cost-effective data interfaces with the eight financial systems found in the federal government.
• A bilingual help desk/call centre available 24/7.
• A travelling employee support network.
• A travel card that provides incentives for use, as well as expanded employee services while in travel status and which pre-populates the online travel expense claim form.
• Improved insurance coverage for travelers.
• An international scope of service, including improved health and safety advisories, visa and passport requirements, and knowledge of the travel whereabouts and status of each employee in travel status.

High levels of adoption and utilization of the new, modern tools by travelers and management are expected to result in significant benefits to the government.

Financial benefits (Government-wide travel spend): In 2003, with the STSI solution, it was estimated that the government would reduce overall travel spend over the nine years of the STSI contract by $811 million (net of the implementation costs) from the baseline scenario of $18.3 billion to $17.5 billion. The 2005 Business Case revision presents a baseline travel spend of $15.7 billion and forecasts a reduction of some $668 million (net of implementation costs) over the nine years of the STSI contract.

3. Logic Model

Exhibit 2 presents the STSI Logic Model that illustrates the objectives as well as the logical chain of activities, outputs and outcomes of STSI. A description of logic models can be found in Appendix C - Description of a Logic Model.

3.1 Objectives

The objectives of the STSI are to:

• Develop and manage a cost-effective, integrated, end-to-end government-wide modernized shared Travel Program that:
  • Improves services to government employees by easing operational context and improving travel experience;
  • Achieves savings on travel, in part to support the ERC;
  • Reduces administrative costs;
  • Improves management information; and
  • Increases transparency and accountability of government travel.
• Serves as a pathfinder/benchmark for all aspects of the shared services model.

"STSI will ensure the delivery of a modern, government-wide travel management system that improves services to employees and provides savings to departments and ultimately taxpayers. The solution will provide fully functional, comprehensive and seamlessly integrated "employee-friendly" travel services to government employees. The scope of the travel services includes a full-services travel agency, on-line reservation service, travel expense claim service, travel card service, traveler's reimbursement service, business intelligence capabilities, an employee portal, and employee traveler service network." ^[5]

3.2 Activities

There are four primary activities undertaken by the STSI Solution. These activities are described below.

Activity #1: Manage and negotiate contract to further automate, simplify, streamline and improve the economics of the Travel Program

This activity consists of managing and negotiating the contract with Accenture and its subcontractors to ensure the necessary provisions are in place to meet the required levels of operation of the suites of services and automated tools. STSI performs quality checks to ensure compliance by the contractor to negotiated service levels. The Contracting Negotiation Panel and the contracting authority, which report to the Acquisition Branch but reside within the STSI project structure, manage this activity.

Activity #2: Facilitate consistency with PWGSC, TBS and Other Government Departments (OGDs) HR, travel and financial policy frameworks

This activity consists of meeting with key stakeholders to make them aware of the issues around the variance between the direction of the new Travel Program and the existing policy and financial frameworks at TBS and within OGDs. STSI needs to combine the joint efforts of various parts of the government as a whole. STSI acts as a catalyst to engage key stakeholders to take action and make the necessary changes to ensure the consistency with the new approach.

STSI is working with the following stakeholders to address policy changes:

• Senior Project Advisory Committee (SPAC)
• TBS Policy, Comptrollership and Chief Information Officer branches
• NJC
• PWGSC Cabinet and Treasury Board Submission Services Directorate, Corporate Services
• STSI Design Working Group (Vanguard departments)

Activity #3: Manage transition to shared Travel Program by developing and disseminating information to influence OGD traveller and management behaviours

This activity describes the manner in which information is disseminated to departments and employees to promote awareness of the new travel program. It consists of developing a communication strategy and providing departments with ongoing updates on the project, as well as on their own performance in comparison to others. It involves regular meetings with senior departmental heads to discuss the benefits of the new Travel Program and the issues associated with its integration within each department. It also consists of monitoring the various aspects of the project such as key activities; timelines and budget; arising issues and risks; collecting specific information on a regular basis (such as uptake rates); and the development of status reports to senior management and key stakeholders.

Activity #4: Manage ongoing Travel commodity including negotiating air, hotel, car rental and other supplier prices

This activity describes PWGSC's role in regards to commodity management and the administration of the Travel Program.

3.3 Outputs

Four outputs flow from the activities.

Output #1: Contract with prime contractor / Payments against contract

Activity #1 should produce the following output: a contract, which permits the achievement of the STSI goals. It also provides the necessary mechanisms for the resolution of issues and for payment.

Output #2: Requests to PWGSC, TBS and OGDs to modernize HR, travel and financial policy frameworks

These documents should stem from activity #2

• Requests for meetings
• Debriefing to key stakeholders
• Record of Decisions/Resolutions
• List of outstanding items
• Presentations

Output #3: Training / Presentations / Information packages / Reports on transition

Flowing from activity #3, these are produced:

• Communication strategy
• Training packages for employees (classroom and e-learning)
• Information packages to employees and departments
• Presentations to senior management and employees
• Metrics on the project
• Metrics on performance
• Status reports
• Debriefings
• Business Intelligence
• Departmental reports

Output #4: Negotiated prices available to government employees

These should flow from Activity #4:

• List of hotel accommodations available (Canada and International)
• List of transportation options available:
  • Air (Canada and International)
  • Rail
  • Bus
  • Car rental
  • Ferry
• Conference and meeting planning rates

3.4 Immediate Outcomes

Immediate outcomes will happen at two levels. The first level includes:

Immediate outcome #1: Prime contractor manages overall project

Activity #1 with its related output should enable the prime contractor to ensure the development and rolling out of a suite of services and automated tools that are functional and satisfactory. The contractor is also able to make revisions and improve the tools following quality reviews by the client.

The contractor puts in place the following suite of tools:

• Travel Agency: provides reservation services as well as numerous benefits to travelers including worldwide services available 24/7/365. American Express provides these services.
• Online Booking Tool (OBT): enabled by the new web based portal, it is a tool with which all public servants can make their own reservations for air, rail, car and hotel. Bell Nexxia maintains the infrastructure for this service.
• Expense Management Tool (EMT): it will incorporate Electronic Travel Planning and Authorization functions and electronic Travel Expense Management and Travel Analysis and Reporting. The final products should have interoperability with existing government financial systems and with other government services, systems and capabilities (still in testing phase with vanguard departments). Concurr is developing this product.
• Travel Card: the new government American Express travel card offers a 24/7/365 employee support network, cash advances through ATMs worldwide, broad international acceptance, enhanced and new insurance options and access to beneficial loyalty programs.

Immediate outcome #2: Changes in relevant TBS and departmental policies and processes

Activity #2 should lead to adjustments of key policies in the area of travel and financial management.

Immediate outcome #3: Better informed and increased by-in by SFTOs / Senior Management

Flowing from the combined activities and related outputs #2 and #3, departmental managers should be well informed of what the new shared Travel Program consists and what are the long term benefits for the government and for them. Buy-in should therefore increase and employees can be given stronger direction. SFTOs should also be able to better manage the operational changes towards the new solution.

The second level of immediate outcomes includes:

Immediate outcome #4: Increased awareness and uptake of travel tools by government employees

All the activities, outputs and immediate outcomes (level 1) should lead to increased awareness of travel tools by public servants and increased uptake of the travel suite of tools. High adoption rates of the OBT, travel card and EMT are a critical success factor to the realization of the benefits of the STSI solution.

Immediate outcome #5: Transition toward modernized shared Travel Program

As well, all the activities, outputs and immediate outcomes (level 1) should lead to the transition towards the new shared Travel Program at PWGSC, within OGDs and across government as a whole.

3.5 Intermediate Outcomes

Flowing from the immediate outcomes (level 2) of increased uptake of travel tools and the transition toward a modernized shared Travel Program, there are five intermediate outcomes.

Intermediate outcome #1: Enhanced travel experience for employees

Public servants who travel will benefit from an end-to-end automated system where information is entered only once. Travelers will also benefit from a strengthened and more effective support system while traveling. The new government travel card will provide for better loyalty programs and insurance plans.

Intermediate outcome #2: Improved management and stewardship of travel within departments

In the past, travel expenditures were covered by PWGSC. The new shared Travel Program will provide OGDs with useful information to be able to manage their own travel and become more accountable for travel spending.

Intermediate outcome #3: Increased and better government-wide travel program information for negotiating with airlines, hotels, car rental agencies and other suppliers (improved commodity management)

Increased uptake of the suite of travel tools will increase government-wide Travel Program information available for the negotiation of rates for various services. Leveraging the government buying power and increasing capacity to use travel costs and information patterns strategically will provide managers with better information for decision-making and enhanced reporting capabilities. This outcome will also increase transparency and accountability of government.

Intermediate outcome #4: Reduced administrative costs

By eliminating a largely manual travel expense management process within departments, the new shared Travel Program aims to reduce administrative costs. By automating, simplifying and streamlining the total travel experience form the trip planning stage through the reconciliation, it is expected that efficiencies will be gained, employee time will be freed and therefore the administrative burden will be significantly reduced. The new system should be increasing productivity and flexibility.

Intermediate outcome #5: Increased benchmarking for all aspects of shared services model based on STSI

Shared services models are gaining popularity with the evolution of new technologies and advocated expected savings from such systems. STSI will be the first of its kind in the GoC using best industry practices, expertise, tools and services. Lessons learned from this Initiative will serve as a benchmark for future similar projects.

Intermediate outcome #6: Direct savings in travel costs

Improved travel management within departments, better information for decision-making and reduced administrative cost should, in turn, lead to direct savings in travel costs throughout the government as a whole.

3.6 Final Outcomes

Achievement of the intermediate outcomes, in turn, should lead to the following ultimate outcomes.

Ultimate outcome #1: Value-driven, modern, word class, government-wide travel management system and program with improved services to employees and savings to departments and taxpayers

Ultimate outcome #2: Shared services successfully introduced across government and inherent benefits achieved

Exhibit 2 (View detailed figure by clicking on link)

4. Evaluation Strategy

The purpose of the evaluation strategy is to identify issues and associated study questions that provide management with valuable information as to the outcomes and effectiveness of STSI activities. The evaluation issues were developed in accordance with the stated activities and outcomes, consultations with the STSI project team and Treasury Board Secretariat Evaluation Policy.

4.1 Evaluation Issues

The objective of the evaluation is to assess the success and effectiveness of STSI activities in order to provide management with information on the extent that STSI is achieving its objectives. Exhibit 3 presents the Evaluation Matrix, which outlines the evaluation issues along with indicators and identifies which issues are to be examined in a formative or summative evaluation.

4.2 Methodology

In order to assess the issues described in the evaluation matrix, the following methodologies would be used:

• Document Review: includes studies, planning documents, research reports, performance frameworks, progress reports, central agency documents, or any other pertinent documents.
• Data Review: includes data collected on a day-to-day basis by the initiative such as correspondence, individual or project file information, monitoring data and reports, financial data and reports, as well as data collected by the contractor.
• Consultations with stakeholders: includes interviews, focus group sessions, expert opinion and surveys. Consultations can occur with individuals and groups who have an interest in the evaluation or are affected by the evaluation. Key stakeholders may include personnel directly involved in the initiative, other government employees, decision-makers, central agency employees, members of the travel industry, and contractors.
• Departmental Case Studies: includes an in-depth exploration of one or more particular departments for the purpose of gaining depth of understanding into the issues being investigated.

Exhibit 3

Evaluation Matrix for the Shared Travel Services Initiative

Evaluation Issues	Indicators	Formative Evaluation	Summative Evaluation
Relevance
1. Is STSI consistent with PWGSC priorities and its Commodity Management role?	Referenced in Departmental Performance Reports Existence of MOU delineating TBS/PWGSC accountabilities Part of speeches made by the Deputy Minister/Minister Adequate management of the travel commodity		√
2. Is STSI a priority for TBS/Office of the Comptroller General?	Expressed commitment to mandatory use Policies and directives adjusted to encourage uptake by travellers Negotiations with Unions Advisory/Steering Committee providing active leadership		√
Design / Delivery
3. What impact did contract design, bid solicitation process and evaluation have on the STSI project team's capacity to develop and implement a modern, cost-effective Government travel service?	Timeliness of contractor's deliverables Number and type of contract amendments Fluid partnership with contractor Adequate resources in place (human, financial) to manage the contract Products are the best available Savings to government Expectations are clearly stated and shared with stakeholders Information is readily accessible and communicated Ethical considerations are embedded in decision-making aspects of governance		√
4. What mechanisms do STSI have in place to manage and ensure a seamless transition to the new shared Travel Program?	Adequate project/program structure, governance and project management frameworks in place Authorities, policies, infrastructure (Secure Channel), and other tools in place Adequate budget and staffing Adequate communication with OGDs Timeliness of implementing the new shared service Travel Program Variance on schedule of implementation Number and type of complaints Mechanism to effectively address complaints Comprehensive reviews of adapting existing travel systems were carried out	√
5. Is the Business Case and underlying assumptions valid?	2005 Business Case consistent with current activities and context (costs, benefits, savings, funding)	√	√
Success / Impact
6. Are current and projected uptake levels capable of generating desired benefits as indicated in the current business plan?	Uptake of travel tools Documented benefits 2005 Business Plan	√
7. Are departments and agencies engaged and exercising stewardship in regards to the transition to the new shared Travel Program?	Uptake of Travel Program tools within departments Departmental policies and procedures consistent with STSI implementation Deputy Minister/Assistant Deputy Minister attendance at relevant committees Referenced in Departmental Performance Reports Department integrating STSI solution SFTOs engaged Communiqués/info sessions to employees Change management strategy	√	√
8. To what extent has the STSI improved the travel services to government employees and enhanced the travel experience for public servants?	Awareness of travel suite of tools by employees High satisfaction rates of the suite of tools (user friendly) Number of complaints Employees spend less time arranging travel Uptake of tools Training strategy in place Number and type of complaints		√
9. Is the increased travel information supporting better travel decisions within OGDs?	Benefits, including reduced service fees, associated with better planning and decisions that are being made with improved data SFTOs and senior managers using improved information in decision-making Documented savings by OGDs		√
10. To what extent has STSI increased transparency and accountability of travel carried out by employees within OGDs?	Senior managers, SFTOs, travellers and travel administrators have a clearer understanding of the costs of traveling Rate of uptake of STSI tools Compliance with travel regulations Delinquency in card payment Reduced number of changes to travel plans and service fees Proportion of Canadians who believe that Government Travel is value driven Comparison to industry standards (based on Commodity Council and on best practices of Fortune 500) and other jurisdictions		√
11. To what extent is STSI serving as a pathfinder or benchmark for all aspects of shared services model?	Documented cases of other shared services benefiting from STSI's experience (sharing lessons learned) Presentations requested by other shared service initiatives, and/or inquiries received The risks of this form of travel program delivery are adequately defined and controllable	√	√
Cost-effectiveness / Alternatives
12. To what extent has STSI enabled the Government to achieve net savings in actual travel costs and support ERC?	Savings realized and acknowledged by ERC GoC has sufficient information to negotiate better prices from travel suppliers Canadians getting value for their tax dollars		√
13. To what extent has the STSI reduced travellers' and OGDs' administrative costs?	Amount of time spent by employees, travel administrators and financial officers on travel arrangements Uptake of STSI tools Documented savings	√	√
14. Does STSI remain affordable	Fair and reasonable cost structure Harvestable net benefits		√

4.3 Subsequent Evaluations

The STSI should be evaluated at critical points in its development. It is proposed that a formative and summative evaluation be conducted.

Formative Evaluation

PWGSC will require early feedback to ensure that STSI is being carried out as designed and intended and that a smooth transition towards the new shared Travel Program is occurring. A formative evaluation to be conducted starting in November 2005 will provide preliminary information on the Initiative's performance. It should focus on the identification of any problems being encountered, which may compromise the initiative's effectiveness, and determine corrective actions.

The formative evaluation will be conducted through the use of document review, selected interviews, focus groups, departmental case studies and data available through Travel AcXess Voyage.

The estimated cost for the formative evaluation is $94,000 in professional services, and 70 OAE person-days.

Summative Evaluation

The summative evaluation will address the long-term effectiveness of the new Travel Program, the achievement of objectives (success), whether its program rationale has continued relevance and whether there are more appropriate alternatives to consider in the delivery of the specified services. In order to support departmental analyses of the initiative and facilitate informed decisions about extending the contract for the optional two years period, a recommended start date for this evaluation is early in year 6.

5. Recommendation

It is recommended that:

1. The CEO, Information Technology Services Branch, accept this evaluation framework and the strategy outlined as the basis for undertaking the formative evaluation to begin in November 2005.

Appendix A - Principal Documents Consulted

1995 Report of the Auditor General of Canada - Chapter 7 - Travel and Hospitality

Budget 2005 - Expenditure Review for Sound Financial Management - Department of Finance

PWGSC, STSI, Project Management Plan, Release Version 5.02, 2004

Request for Proposal - Government Travel Modernization Services, 2002.

Toward a Stewardship Theory of Management. J. H. Davis, F.D. Shoorman & L. Donaldson. Academy of Management Review, 1997.

TB Submission, Major Crown Project, STSI, 2003

TBS, CIO Branch, Outcomes Management Project, The STSI Benefits Case, 2005 (Fujitsu Consulting)

Internet

TBS - Travel
TBS - Public Service Modernization
TBS - Expenditure Review

Appendix B - Interviews Conducted

• Bégin, Pierre, Manager, Accounting Operations, Corporate Services Sector, Financial Management Branch, NRCan
• Brennan, Shawn, Executive Director, STSI, PWGSC
• Corbett, Michael, Director, Release Management and Quality Assurance, STSI, PWGSC
• Draper, Walter, Director (SFTO), Financial Policies, Systems and Accounting Directorate, INAC
• Elias, Peter, Director, Client Services and Training, STSI, PWGSC
• Flynn-McRae, Mary, Director, Change Management, STSI, PWGSC
• Holt, Dawn, Financial Advisor, Financial Policies, Training and Communications, Financial Policies, Systems and Accounting Directorate, INAC
• Horsley, John, Director, Information Technology, STSI, PWGSC
• Job, Wayne, Senior Projects Coordinator, Strategic Systems Infrastructure Directorate Executive Director's Office, OCG, TBS
• Kehoe, Gary, Director, Design and Implementation, STSI, PWGSC
• Leclerc-Cowell, Lucie, Director, Travel Operations, STSI, PWGSC
• Poulin, Stéphane, Senior Policy Analyst, Strategic Systems Infrastructure Directorate Executive Director's Office, OCG, TBS
• Purnell, Ted, Contract Authority, AB, PWGSC
• [ * ], Accenture (Contractor)

Appendix C - Description of a Logic Model

A Logic Model is an essential tool in conducting an evaluation study. It is intended to show, in a succinct fashion, the logic of how a program, policy or initiative is expected to achieve its objectives. Generally, a Logic Model builds on the management process and graphically displays the linkage and causal connections between the program's (or policy's or initiative's) activities that are supported by inputs and the different levels of results - including outputs, strategic outcomes (immediate and intermediate) and ultimate results (or long-term outcomes).

It is useful to examine this process as a sequence of cause and effect relationships. For example, the inputs make it possible for the activities to be carried out; the activities then produce the outputs; the outputs, in the hands of the recipients, lead or are expected to lead to achieving immediate and intermediate outcomes, which generally correspond to desired changes in terms of knowledge, attitudes, skills and behaviours of recipients and any subsequent targeted people or organizations.

Generally, intermediate and ultimate outcomes clarify implicitly or explicitly who are the intended beneficiaries and targets of the program, policy or initiative. They can include specific groups of Canadians, Canadian or foreign businesses and non-profit organizations, a specific department(s) or jurisdiction(s), or the people of Canada in general. In turn, the intermediate and ultimate outcomes are intended to lead to achieving the ultimate outcomes or results, i.e., the expected long-term changes.

Developing a Logic Model facilitates the examination of the plausibility of the various relationships and provides an invaluable tool for the on-going and future performance monitoring and evaluation of the program, policy or initiative.

The key elements for developing a Logic Model for the Shared Travel Service Initiative include the following:

1. Activities: What key activities are being undertaken to contribute to the strategic STSI outcomes? What work is carried out to produce the outputs?
2. Outputs: What outputs need to be produced to achieve the expected strategic STSI outcomes
3. Immediate outcomes: What short-term consequences should be expected to result from the key STSI activities and outputs? How do recipients of outputs use the outputs to effect changes in their knowledge (including awareness), attitudes, skills, intentions, and behaviours of the target beneficiaries?
4. Intermediate outcomes: What strategic STSI outcomes are expected to be achieved (flowing from activities, outputs and immediate outcomes) that will demonstrate progress towards achieving the long-term outcomes and ultimate results? Are the targeted stakeholders being affected in the way that they were expected to be?
5. Long-term outcomes/Ultimate results: Why has the STSI been undertaken and what is it expected to ultimately achieve and who are the ultimate beneficiaries? Long-term outcomes (and sometimes intermediate outcomes, as well) may take a very long time to achieve and are generally subject to influences (both positive and negative) beyond the STSI.

In developing a Logic Model for the STSI, it was useful to begin by identifying and understanding its objectives.

Footnotes

^[1] TBS - Travel Directive

^[2] The TBS - Expenditure Review ERC is a cabinet-level committee is responsible for reviewing all federal spending.

^[3] Following the work of the Parliamentary Secretary's Task Force Government-wide Review of Procurement and PWGSC's The Way Forward, the Commodity Council management initiative was launched in May 2005. Commodity management is an approach to procurement based on a strategic decision making process for a particular group of goods or services. One such Commodity Council as been created for travel/transportation to enhance the new Travel Program.

^[4] Statement of Work, Annex 1 - Request for Proposal, Government Travel Modernization Services.

^[5] TBS, CIO Branch, Outcome Management Study, Fujitsu Consulting