Public Services and Procurement Canada
Internal Audit: Audit of stakeholder engagement in the PeopleSoft 9.2 upgrade—Stakeholder sign-off process (Phase 3)

Office of the Chief Audit, Evaluation and Risk Executive

August 4, 2021

Do not circulate/Do not copy

Internal Audit: Audit of stakeholder engagement in the PeopleSoft 9.2 upgrade—Stakeholder sign-off process (Phase 3) (PDF, 477KB)

Introduction

Public Services and Procurement Canada (PSPC) has a legal mandate, identified in sections 12 and 13 of the Department of Public Works and Government Services Canada Act (1996, chapter 16), to provide payroll services to the employees of the public service of Canada.

The PeopleSoft 9.1 Pay System, also known as Phoenix, has been in operation since 2016. Version 9.1 of PeopleSoft is an end-of-life commercial off-the-shelf product and standard support for the core system functionality has ended. The Government of Canada (GOC) and Oracle have agreed on extended support for version 9.1 until December 2021. After this date, PSPC is faced with 2 key operational risks:

• due to the lack of vendor support, there is a high risk of security vulnerabilities and service degradation impacting PSPC’s ability to produce accurate and timely pay
• failure to apply tax updates is a compliance risk and impacts PSPC’s ability to produce accurate pay related to deductions and tax slips

To address these risks, PSPC has developed a plan to upgrade version 9.1 to version 9.2 by July 2021. The Pay Solutions Branch (PSB) at PSPC is responsible for this upgrade, and it is being completed through the PeopleSoft Pay System 9.2 Upgrade Project (referred to as the Project hereafter).

The Project seeks to contribute to the achievement of Pay System Sustainability and Pay Stabilization Efforts. The scope of the upgrade focuses on a technical upgrade that will result in minimal changes to integration architecture, data entry, and business processes; however, there are additional potential benefits to the upgrade, which will be considered post-upgrade as part of PSPC’s continuous pay stabilization efforts.

Background

As a result of the risk assessment of PSPC’s audit universe, the Office of the Chief Audit, Evaluation and Risk Executive (OCAERE) determined assurance work related to stakeholder engagement within the PeopleSoft Pay System 9.2 Upgrade Project should be undertaken in 2019 to 2020 and 2020 to 2021.

Originally, the OCAERE audit plan consisted of multiple engagements related to stakeholder engagement throughout the project with regular reporting at key project milestones. The first report was issued in February 2020. During the completion of the first engagement, OCAERE noted significant overlap between its findings and recommendations and those issued by an independent third party, Gartner, engaged by the Project team to perform periodic assessments at key project milestones.

As a result, the OCAERE audit approach was adjusted to conduct assurance work related to the stakeholder sign-off process which is a critical stakeholder engagement activity that is key to the Project’s success. The stakeholder sign-off process is 1 of several measures in the Project plan that was informed by the Office of Auditor General recommendations, and a series of engagements completed with numerous stakeholders following the initial implementation of the Phoenix system. Developing and implementing the sign-off process is part of PSPC’s commitment to stakeholder engagement and represents a key element in the Project’s go-or-no-go decision process. The stakeholder sign-off is a 1-time sign-off based on results from the testing period of the upgrade.

Departments and agencies signed off to confirm:

• testing of the human resources (HR) system is complete
• tested transactions have successfully integrated with the 9.2 system
• technical resources have been, or will be, allocated to ensuring an HR system is available to connect with the Phoenix 9.2 environments during the dry-runs and deployment to production
• key HR transactions have been successfully tested
• the organization is equipped to support the new platform
• departmental and agency variance during the Project’s pay validation testing is within the approved threshold and tolerance level

Focus of the audit

This internal audit is included in the most recent Office of the Chief Audit, Evaluation and Risk Executive`s approved Risk-Based Audit Plan 2019 to 2020 and 2020 to 2023.

Importance

As outlined in the Project Charter, key stakeholders include:

• departments and agencies that use the Phoenix system
• partners who use Phoenix
• partners whose systems interface with Phoenix
• internal and external committees that are required for decisions, endorsement, oversight, project gating, and approvals

Adequate stakeholder engagement is 1 way to support the success of the Project included in the Project plan. The Project plan was informed by:

• the Goss Gilroy report
• the Office of Auditor General 2018 Report 1 Building and Implementing the Phoenix Pay System recommendations
• engagements completed with numerous stakeholders, following the initial implementation of the Phoenix system

The performance of independent assessments, along with addressing issues identified as a result of these assessments are part of PSPC’s commitment to stakeholder engagement.

Given the large number of stakeholder groups that the upgrade may affect, stakeholder engagement throughout the Project has been identified as a critical success factor in order to build trust, credibility and strong relationships. This helps facilitate buy-in and successful implementation of this upgrade, as well as future pay stabilization projects.

Objective

The objective of this audit was to assess the adequacy and effectiveness of stakeholder engagement in the Project’s lifecycle. The engagement was completed in multiple phases, with results reported in multiple engagement reports to ensure that audit findings and recommendations were provided to the Project team in a timely manner.

This report contains the findings from the third phase of the engagement. It assessed how the Project team executed the sign-off process. The audit specifically examined if the sign-off process activities were conducted according to the sign-off process approved by Project governance bodies and had been communicated to stakeholders.

Scope

The period covered in this audit engagement is from March 9, 2021 to June 15, 2021.

This assessment reviewed the Project team’s activities up to June 15, 2021. The audit assessed the degree to which stakeholder engagement activities have been performed to meet the needs of the stakeholders. More specifically, audit activities assessed:

• if the Project team engaged stakeholders as required by the approved stakeholder sign-off process
• whether escalation procedures were completed as outlined in the Project’s escalation process

The scope excluded an assessment of project activities not related to the engagement of stakeholders.

Audit criteria and methodology

The internal audit was conducted in accordance with the International Standards for the Professional Practice of Internal Auditing, as supported by the results of the quality assurance and improvement program.

Appendix A: Audit criteria and methodology recaps the audit criteria and methodology assessed in each phase of the audit.

Summary observations and findings

Overall, we found that the Project team followed the approved stakeholder sign-off process. Where escalation procedures were required to obtain sign-off, procedures were performed as designed and documented in the Project escalation process. As all audit criteria were met, we have not issued any recommendations.

Through interviews with stakeholders, feedback on the Project’s engagement of stakeholders through the sign-off process was generally positive; however, interviewees suggested improvements for management’s consideration when conducting similar projects in the future. To enable future projects to leverage the experiences of the Project team and the stakeholders that participated in the stakeholder sign-off process, there is an opportunity to solicit broader feedback from the stakeholder community to identify best practices and opportunities for improvement that may be taken into consideration for future projects.

Detailed observations and findings

The following outlines the observations and findings of the Project.

Sign-off process

During the first and second phases of this multi-phase audit, we completed assessments to confirm whether the Project team:

• identified stakeholders to whom the sign-off process is relevant
• consulted stakeholders on the design of the sign-off process
• incorporated stakeholder feedback into the process
• formalized and communicated the final sign-off process to stakeholders
• obtained acknowledgements of stakeholders’ understanding of the process and their roles and responsibilities
• developed escalation procedures in case any stakeholders did not sign off

This phase of the audit focused on the execution of the sign-off process, which commenced on May 7, 2021, as summarized in the table below.

For each phase in the sign-off process, audit fieldwork was performed to assess if the key activities were appropriately completed.

Sign-off process initiation

On May 7, 2021, the Project team sent email requests for sign-offs to 206 executives including Chief Information Officers (CIO), Chief Financial Officers (CFO), and Heads of human resources (HHR) for each stakeholder organization. Stakeholders were asked to email their sign-off to the Project team between May 10, 2021 and May 23, 2021. Stakeholder organizations that host an HR system that integrates or interfaces with Phoenix needed 3 sign-offs, 1 each from the CIO, CFO, and HHR. Departments). For departments that do not host a HR system and use a system hosted by another organization or are a direct entry department, 2 sign-offs (CFO and HHR) were requested.

Sign-off requests accompanied briefing packages which contained documentation to support each stakeholder’s sign-off decision. Through interviews with stakeholders that participated in the sign-off process, the majority noted that the information provided within the briefing packages was complete, clear, and effectively supported the executives in making their sign-off decision. While some stakeholders indicated that they required clarification of the information provided in the briefing packages, the stakeholders noted that the Project team provided the additional information required to complete and submit their sign-off.

Response tracking and reminder communications

Once stakeholders received requests for sign-off, the Project team tracked all sign-offs received along with key sign-off related metrics. Metrics tracked included the number of sign-offs received, the number of sign-offs received from each stakeholder organization versus the number of sign-offs required from each organization, and the number of sign-offs received against the minimum stakeholder sign-off threshold previously approved by Project governance and communicated to stakeholders.

As required in the escalation process, 1 week after the initial sign-off requests were sent to stakeholders, stakeholders who did not yet respond received a reminder that there were 7 calendar days remaining before the escalation process would be initiated. Stakeholders who did not respond received a second reminder 2 weeks after the initial sign-off requests notifying the stakeholders that did not submit their sign-off by May 23, 2021 that, if the Project team did not receive a sign-off within 3 calendar days (by May 27, 2021), the escalation process would be initiated.

Escalation

On May 27, 2021, signatures from 13 (out of 96) stakeholder organizations remained outstanding. Following the approved escalation process, the executives from these 13 organizations responsible for signing-off were contacted by senior officials within Treasury Board Secretariat (TBS) and PSPC to address any questions or concerns stakeholders may have had that was preventing them from completing and submitting their sign-off. Following the initiation of the escalation process, all remaining outstanding signatures were received by June 11, 2021.

Presentation of results to governance committees

As outlined in the escalation process, the results of the sign-off process would be considered on June 10, 2021 by the Upgrade Acceptance Board (UAB) when making its go-or-no-go decision. When the UAB met on June 10 to finalize the Go/No-go decision, 205 (out of 206) stakeholder sign-offs had been received (it was noted that the final sign-off was obtained by the Project team on June 11, 2021) and the minimum stakeholder sign-off threshold had been achieved. The results of the sign-off process were considered by the UAB in conjunction with the achievement of numerous other readiness criteria and checkpoints. As all checkpoints and criteria were met, the UAB’s approved the move to production between June 18 and June 22, 2021.

Stakeholder feedback on sign-off process

Between June 9 and June 15, 2021, the audit team conducted interviews with a sample of six (6) executives responsible for submitting a sign-off as part of the stakeholder sign-off process. Stakeholder feedback obtained through these interviews confirmed that the stakeholders were engaged to the extent agreed upon to meet the requirements of the sign-off process. The majority of interviewees provided positive feedback on the sign-off process.

Interviewees indicated that the Project team’s engagement of the stakeholder community prior to and during the completion of the sign-off process ensured that executives were aware of their responsibilities for signing off, understood the key sign-off activities and timelines, and had access to the supporting documentation required to make a sign off decision. Interviewees noted that the Project team responded to questions and concerns in a timely manner.

The majority of interviewees also noted that they support the use of stakeholder sign-off processes for future similar projects.

While interviewees’ comments were generally positive, they suggested several opportunities for improvements. Suggestions for changes to the sign-off process included:

• increasing the time that larger stakeholder organizations have to review supporting documentation before signing off
• giving greater clarity in communications regarding requirements that are applicable to a sub-set of organizations (for instance only CIOs from departments that host an HR system that is integrated or interfaces with Phoenix were required to sign off)
• clearly communicating the impact on the project if a stakeholder does not submit their sign-off

Conclusion

Conclusions are based on:

• the review of audit evidence
• interviews with the Project team
• interviews with a sample of stakeholder that participated in the sign-off process

These showed that stakeholders were engaged according to the requirements of the sign-off process. No recommendations have been issued.

The audit noted that, as part of its post go-live activities, the Project should get feedback from the broader stakeholder community to identify best practices and opportunities for improvements that may be taken into consideration for future projects.

Recommendations issued in the stakeholder engagement in the PeopleSoft 9.2 upgrade: Stakeholder sign-off process phase 2 audit report

The Stakeholder Engagement in the PeopleSoft 9.2 Upgrade—Stakeholder Sign-Off Process Phase 2 Audit Report (April 2021) was presented and approved at the May 2021 Departmental Audit Committee. It contained several recommendations to the Project team related to:

• stakeholder acknowledgements of their understanding of the sign-off process
• the development of the escalation process
• the implementation of the escalation process

Action plans were developed by the Project team and documented action plans within the Phase 2 audit report.

As part of the Stakeholder Engagement in the PeopleSoft 9.2 Upgrade—Stakeholder Sign-Off Process Phase 3 Audit Report, testing was performed to confirm that the action plans were appropriately implemented.

Stakeholder acknowledgement of understanding of sign-off process

The Stakeholder Engagement in the PeopleSoft 9.2 Upgrade—Stakeholder Sign-Off Process Phase 2 Audit Report (April 2021) recommended the following related to stakeholder acknowledgements of their understanding of the sign-off process:

The Project team should continue collecting formal acknowledgements of stakeholders’ understanding of the process. The Project team should define a deadline and an appropriate response rate to receive these acknowledgements.

Documentation (stakeholder communications, response tracking) and interviews with the Project team confirmed that recommendations were adequately addressed. Refer to Appendix B: Procedures performed to assess the implementation of recommendations issued in the phase 2 audit report for a listing of the performed procedures and examined documentation .

Escalation process

In the Stakeholder Engagement in the PeopleSoft 9.2 Upgrade—Stakeholder Sign-Off Process Phase 2 Audit Report (April 2021), the following recommendations related to the sign-off and escalation process were issued:

Recommendation 1

The Project team should clearly define how the departmental sign-off process will be finalized and evaluated for the purposes of the go-or-no-go decision. Specifically, it is recommended that the Project team:

• determine if each of the identified individuals in a department or agency (CFO, CIO and HHR) are required to sign off, in order for an organization to be deemed to have signed off
• determine if acceptance of the sign-off process by delegates of the CFO, CIO and HHR are acceptable responses from each organization
• ensure that mitigation measures are in place to manage the risk associated with acceptance of responses as “assumed” sign-offs
• clear communication to governance bodies with respect to the number of sign-offs received and relevant trends:
  • sign-off by type of executive (CFO, CIO, Head of HR)
  • number of assumed sign-offs
  • including communication to individuals that will be performing the go-or-no-go decision

Furthermore, we recommend that any updates to processes as a result of this recommendation be included in Project documentation and be communicated to stakeholders.

Recommendation 2

The Project team should continue refining the escalation procedures or other procedures as necessary, to be followed if the minimum threshold for departmental sign-off is not met and if one or more stakeholders decide not sign off. Considerations include:

• procedures for escalating beyond the executives responsible for signing off within their respective organizations or to central agencies to follow-up with respective departments (for example, OCHRO, GOC CIO, OCG)
• procedures to manage the potential that stakeholders may provide a no-go sign-off on the upgrade

Through the examination of documentation (communications to stakeholders, sign-off tracking sheets) and interviews with the Project team, the recommendations were confirmed to have been adequately addressed. Refer to Appendix B: Procedures performed to assess the implementation of recommendations issued in the phase 2 audit report for a listing of the procedures that were performed and the documentation that was examined.

Recommendations

As all audit criteria were met, no recommendations were issued.

Conclusion

Through the completion of an assessment, the Project team engaged stakeholders according to the stakeholder sign-off process. Where escalation procedures were required to obtain signatures, procedures followed the Project escalation process. The Project team received all stakeholder sign-offs before deploying the system upgrade. Feedback from a sample of stakeholders was positive.

However, there is an opportunity to issue a questionnaire to stakeholders to identify lessons learned and develop best practices for future projects.

Acknowledgement

In closing, we would like to acknowledge, recognize, and thank the Project team for the time dedicated and the information provided during the course of this engagement.

Audit team

Lori-Lee Flanagan

Director

Gilles Trahan

Audit Principal

Jhaal Fergus

Auditor

Carter Wilmsmeyer

Audit intern

Deloitte (LLP)

Consultant

Appendices

The following appendices detail criteria and procedures during phase 3 of the sign-off process.

Appendix A: Audit criteria and methodology

Please refer to the following audit criteria and methodology of phase 3 of the sign-off process.

Audit criteria

Based on the OCAERE risk assessment, the following lines of enquiry were identified. This tables also includes information on which report each criteria is addressed in.

Methodology

As part of OCAERE’s multi-phased audit approach , audit fieldwork for the current audit phase was performed between March 8, 2021 and June 15, 2021. During this period, the audit team continued attending, as an observer, relevant project committee meetings including the Upgrade Steering Committee (USC) and the Integrated Pay Advisory Board (IPAB). As well, the audit team conducted:

• reviews of audit evidence including meeting minutes, records of decisions, presentation materials, communications, responses, activity trackers, and other items developed by the Project team
• interviews with key Project team members and a sample of stakeholders that participated in the sign-off process

While a sample of 9 stakeholders were selected for interviews, due to the availability and the timing of the audit examination phase, 6 interviews were held. When identifying stakeholders to interview, the audit team considered:

• the systems that different stakeholders used
• the size of the stakeholder organizations
• the pay model at each organization
• other factors

1 CIO, 2 CFOs, and 3 HHR were interviewed.

Audit activities were performed concurrently with the Project’s activities. The audit team communicated audit findings to the Project team through real-time status reports (bi-weekly or more regularly) to facilitate timely implementation of corrective actions. The audit team remained independent from the Project team and was not responsible for management decision-making as it relates to the Project’s activities and deliverables.

During the examination phases of each assessment area identified above, key PSB personnel and external IBM consultants working with the PSPC Project team were interviewed. In addition, relevant documentation was reviewed in detail. At the conclusion of the examination phase, key members of the Project team were requested to provide validation of the findings.

During the reporting phase for each individual assessment area, the audit team documented the audit findings, conclusions, and recommendations in a draft audit report. The draft report includes the assessment results up to the end of the period for which the report relates. The draft report was submitted for acceptance to the ADM of PSB in a timely manner following the Chief Audit, Evaluation and Risk Executive’s acceptance. The draft final report will be tabled at the Departmental Audit Committee meeting in August 2021.

Appendix B: Procedures to assess recommendations in the phase 2 audit report

This table lists:

• the 3 stakeholder sign-off process recommendations issued in the Phase 2 audit report (April 2021) (column 1)
• the action plans outlined by the Project team (column 2)
• the procedures performed to assess if the recommendations were adequately addressed (column 3)