Evaluation of the Federal/Provincial/Territorial Group Purchasing Program for Drugs and Vaccines (2008-609)

Final Report

March 19, 2009

Table of Contents

• MAIN POINTS
• INTRODUCTION
• PROFILE
  • Background
  • Authority
  • Roles and Responsibilities
  • Stakeholders
  • Procurement Process
  • Resources
  • Logic Model
• FOCUS OF THE ENGAGEMENT
  • Objective
  • Scope
• FINDINGS AND CONCLUSIONS
  • Relevance
    • Findings Findings - Relevance
    • Conclusions Conclusions - Relevance
  • Performance
    • Findings Findings - Performance
    • Conclusions Conclusions - Performance
• GENERAL CONCLUSIONS
• MANAGEMENT RESPONSE
• RECOMMENDATIONS AND MANAGEMENT ACTION PLAN
• ABOUT THE EVALUATION
• Appendix A: Evaluation Matrix
• Appendix B: Principal Documents Consulted

Please note that under the Access to Information Act a limited amount of text within these documents may not be disclosed, and will be shown as [ * ].

MAIN POINTS

What was examined

i. The Federal/Provincial/Territorial (F/P/T) Group Purchasing Program for Drugs and Vaccines (the Program) is part of the Commercial Acquisitions and Supply Management Sector, Acquisitions Branch in Public Works and Government Services Canada (PWGSC). It includes the Influenza Vaccine Project and the Drugs, Vaccines and Biologics Division.

ii. The Program's mandate is to carry out the purchase of drugs and vaccines as an ongoing, voluntary arrangement on behalf of F/P/T governments, utilizing the procurement services of PWGSC. The Program works with provincial and territorial governments, as well as the Correctional Service of Canada, the Department of National Defence and Health Canada to acquire a safe supply of medications for Canadians in emergencies.

iii. The Program purchases the annual influenza vaccine, along with approximately 51 other vaccines, as well as a variety of drugs, including antivirals. In 2007-2008, the Program managed 64 contracts, comprising 25 vaccine contracts, ten drug standing offers, 28 drug contracts and one Prime Vendor standing offer, for a total value of $301.6M. Vaccines comprise approximately $254.5M, of which [ * ] was for [ * ] million doses of influenza vaccine. In addition, approximately 750 types of drugs are purchased.

Why it is important

iv. In the 1970s, the Federal and Provincial Deputy Ministers of Health approved the creation of a federal/provincial program for the bulk purchase of drugs and vaccines. This voluntary program has now been in place for over 30 years. In 2003, the National Immunization Strategy was developed to provide a comprehensive F/P/T strategy and a collaborative approach to address vaccine supply and achieve best value for vaccine procurement, long-term security of supply and quality of supply. The Program is a key component of both these initiatives.

What was found

v. The Program is relevant and consistent with departmental and government-wide policies and priorities. It is supported through legislation and policies. The Program's collaborative and nation-wide approach is consistent with the National Immunization Strategy and its aim of achieving best value for vaccines and long-term security of supply.

vi. As the federal department responsible for the acquisition of goods and services for the federal government and as it has been authorized to work on behalf of provincial and territorial jurisdictions, PWGSC has an appropriate role to procure drugs and vaccines on behalf of F/P/T jurisdictions.

vii. There is evidence of ongoing voluntary participation by all provinces and territories, along with substantial growth in procurement levels, that demonstrates a strong relevance for a nation-wide centralized procurement as is now provided through the Program. All stakeholders indicate there is a need for a centralized Canada-wide program as is currently provided through the Program.

viii. The Program is achieving its intended outcomes. F/P/T clients realize savings through cost-effective acquisitions and achieve value for money. Jurisdictions benefit from being part of a larger purchasing group. All clients are treated equally no matter their size or buying power.

ix. Clients have high overall confidence with the Program and are highly satisfied with the fair, efficient, knowledgeable and responsive acquisition services. Jurisdictions benefit from the Program's procurement expertise and are generally quite satisfied with the Program's efficient and timely contract administration.

x. The Program contributes to the security of supply in Canada through strategies such as dual sourcing and establishing a Canadian production capacity.

xi. The voluntary participation of all the jurisdictions is a measurement of the value and success of the Program. While the Program is in the process of delivering performance measurements on its outputs, the Program outputs, outcomes and client satisfaction are not being fully assessed through internal performance measurements.

xii. The Program provides a Canada-wide perspective and increases awareness of national immunization issues. The federal government benefits from participating in the national health community as an active and valued participant.

xiii. While the Program is providing excellent value for money with regard to the prices negotiated for drugs and vaccines, it is recovering insufficient administration fees to cover its costs for the work provided on provincial and territorial services. While proposals are being made to increase the fee, it could be two years before they are sufficient to cover the estimated costs. In addition, the blended price system is complex, resulting in a difficulty in establishing vaccine prices and accurate invoicing amounts.

xiv. Evidence was not found of more cost-effective alternatives to procure drugs and vaccines; rather, it appears that similar models are being implemented in other countries.

Management Response

xv. Acquisitions Branch considers the results of this evaluation accurately and fairly reflect the Federal, Provincial and Territorial Group Purchasing Program for Drugs and Vaccines, as of January 2009. Acquisition Branch accepts this evaluation report and intends to act on all recommendations of this evaluation by implementing a Management Action Plan.

Recommendations and Management Action Plans

xvi. Based on the key findings and conclusions contained in this report, the Office of Audit and Evaluation recommends that the Assistant Deputy Minister, Acquisitions Branch:

Recommendation 1: Develop performance measurements to fully track Program performance and success.

• Management Action Plan 1: The Program is in the process of delivering performance measures on its outputs, as described in the PWGSC Performance Measurement Framework 2008-2009 as follows:
  • Percentage of contracts issued on time,
  • Unit vaccine costs compared to US prices,
  • Percentage of shipments delivered on time,
  • Number of contracts issued and
  • Number of shipments negotiated.
• In addition to the above, an annual client survey will be conducted.
• This will be done by March 2010.

Recommendation 2: Conduct an analysis of the blended price mechanism to better establish pricing.

• Management Action Plan 2: The pricing mechanism was reviewed in late 2008 and the MOU was revised to include an annual price review and price adjustment.
• With each annual review the pricing mechanism will be reviewed in consultation with our financial advisors, to ascertain whether additional changes are needed.
• This will be done by March 2010.

INTRODUCTION

1. This report presents the results of the evaluation of the Federal/Provincial/Territorial (F/P/T) Group Purchasing Program for Drugs and Vaccines (the Program). The Audit and Evaluation Committee of Public Works and Government Services Canada (PWGSC) approved this Evaluation as part of the 2008-2011 Risk-Based Multi-Year Audit and Evaluation Plan.

PROFILE

Background

2. In 1973, the Federal and Provincial Deputy Ministers of Health proposed a program for the bulk purchase of drugs and vaccines, resulting in the creation of a working group. In 1976, the Conference of Deputy Ministers of Health approved the continuation of this working group as a program. Ongoing for over 30 years, the Program now provides services to all 13 provincial and territorial health ministries, the Correctional Service of Canada, the Department of National Defence, Health Canada and the Public Health of Agency of Canada (PHAC).

3. In 2003, the National Immunization Strategy was developed by the Government of Canada, provinces and territories to provide a comprehensive strategy to meet national immunization needs. It was a recognition that a collaborative approach was needed to address issues such as vaccine supply and access to publicly funded vaccine programs. The Strategy was designed to address a number of challenges to immunization faced by all Canadian jurisdictions, including: increased vaccine expenditures due to the introduction of expensive new vaccines, escalating prices of existing vaccines, concerns about security of supply and supply shortage, as well as the need for rapid and effective national interventions in emergency situations. The Strategy aims to achieve best value for vaccine procurement, long-term security of supply and quality of supply.

Authority

4. The Constitution Act (1867) assigns the provincial and territorial jurisdictions with the responsibility for health care and its delivery, including the purchase of drugs and vaccines to deliver their respective programs.

5. The Canada Health Act (1984) provides the federal government a role in health, which includes setting and administering national principles for the health system. In order for provinces and territories to deliver their programs, the federal government provides them with financial support.

6. The F/P/T Group Purchasing Program for Drugs and Vaccines operates under the following authorities:

• Department of Public Works and Government Services Act (1996)
• Order in Council 1986-1581 (1986)
• 1973 and 1976 Federal/Provincial agreements
• Memoranda of Understanding

7. Under the Department of Public Works and Government Services Act, the Minister's functions include "the acquisition and provision of articles, supplies, machinery, equipment and other materiel for departments." This allows the federal government to procure drugs and vaccines for those federal government departments and agencies that require them. In the context of the Program, this includes:

• Canadian Food Inspection Agency
• Citizenship and Immigration Canada (immigrants)
• Correctional Service of Canada (federal inmates)
• Foreign Affairs and International Trade (foreign service officers)
• Health Canada (First Nations people, Inuit and Aboriginal people)
• National Defence (Canadian Forces members)
• Royal Canadian Mounted Police (members)

8. A 1986 Order in Council enables the Minister of Supply and Services (now the Minister of Public Works and Government Services) to provide services to Crown corporations, federal departments and the provincial governments "where such corporations or governments request such services to be provided by the Minister of Supply and Services on its behalf."

9. The request for the Minister of Supply and Services to provide service to the provinces and territories dates from the 1973 and 1976 agreements between the federal government and the provincial/territorial governments to create the Program. In addition, all provinces and territories sign annual Memoranda of Understanding with the Program to agree to purchase vaccines and other drugs through the Program and its services.

10. In addition to the authorities listed above, and in addition to the policies and guidelines governing the operations of all federal government programs, the Common Services Policy also applies to the Program. The Policy is to "ensure that departments and agencies can acquire responsive, cost-effective support for their program delivery". The common service organization, in this case PWGSC, "may negotiate special arrangements to offer services to Crown corporations or non-federal organizations, provided that it is consistent with the mandate of the common service organization". It also states "optional services are funded mainly by full cost-recovery through a revolving fund or net-voting authority".

Roles and Responsibilities

11. The mandate of the Program is to carry out the purchase of drugs and vaccines as an ongoing, voluntary arrangement on behalf of F/P/T governments, utilizing the procurement services of PWGSC. The guiding principles of the Program are:

• Savings through combined buying power;
• Equality of all participants; and
• Provincial and territorial voluntary participation.

12. The Program procures the annual influenza vaccine and approximately 51 other vaccines, as well as approximately 750 drugs and antivirals. It does this on behalf of the F/P/T jurisdictions by negotiating and establishing procurement contracts, using two different procurement models. The Program provides procurement expertise and contract quality assurance, as well as a coordination and liaison role between the clients and the suppliers. The Program also coordinates back orders and shortages to ensure appropriate distribution amongst the jurisdictions.

13. In addition, the Program is involved in numerous inter-jurisdictional working groups such as the Vaccine Supply Working Group, Interdepartmental Vaccine Supply Committee and the Antiviral Stockpile Management Task Group. It participates in the health community, providing advice and expertise on vaccine and drug procurement issues. Finally, it plays a role in emergency preparedness.

14. Currently, the Program is comprised of the Influenza Vaccine Project and the Drugs, Vaccines and Biologics Division.

The Influenza Vaccine Project

15. In early 2007, the Influenza Vaccine Project was separated from the Drugs, Vaccines and Biologics Division into a distinct administrative unit. This was due to the growing size and complexity of the influenza vaccine contract (both cost and quantities) and to provide a focus on the new contract to be put in place by 2011.

16. Influenza vaccine purchasing and contract administration is the responsibility of the Project. Vaccines are purchased on behalf of the F/P/T clients who provide their purchasing requirements directly to PWGSC. Influenza vaccine purchases are one of the largest purchases of the overall Program, with the ten-year influenza contract (2001-2011) being valued at $600M, with a potential growth to $1.2B. The $600M ten-year estimated expenditure includes not only the purchase of the annual influenza virus vaccine requirements but also the costs involved with the preparation for pandemic influenza vaccine requirements as discussed below.

17. An influenza pandemic may occur when an influenza strain changes into a new strain to which humans have no immunity. If this new strain of influenza virus has the ability to spread easily from person to person, many people around the world could become ill and possibly die. The Project plays a role in preparing for such pandemics through capacity building and the purchase of equipment and facilities. Health Canada and F/P/T jurisdictions determine what production capacity is needed and can be afforded to produce the 8 million doses of vaccines required per month in the case of a pandemic. Health Canada sets out a "wish list" of requirements it would like to see met. The Project negotiates with suppliers to establish milestones to meet the requirements and then follows up with suppliers to ensure milestones are being met. Suppliers inform the Project about their equipment requirements to meet the milestones. To encourage suppliers to establish production capacity in Canada, and to ensure a Canadian capacity, the Government of Canada pays for the equipment and facilities, such as facilities to house the hens that produce the eggs used to make the vaccines. The Project inspects the equipment and facilities, approves them and then affixes a Government of Canada logo on the equipment to confirm it is Government of Canada-owned.

18. As of October 2008, the Project is also responsible for the procurement of antivirals. Established over the last four years, the first antivirals were purchased for Ontario and the National Emergency Stockpile. These antivirals have a five-year shelf life and have not yet reached their expiration date. The first review of these stockpiles is currently being undertaken by PHAC, who is responsible for the overall management of the National Emergency Stockpile.

The Drugs, Vaccines and Biologics Division

19. The Drugs, Vaccines and Biologics Division is responsible for the procurement and contracting arrangements for other non-influenza vaccines, drugs, medical supplies and biologics. These include drugs such as Carbamazepine, Dextromethorphan, Rifampin and Soflax and vaccines such as Human Papillomavirus Vaccin (HPV), Hepatitis "A"& "B" series and Typhoid.

Stakeholders

20. The vaccine process, especially for influenza, is a complicated long-term continuum, involving various levels of non-governmental organizations, government and manufacturers across all jurisdictions.

21. The process starts with the World Health Organization monitoring circulating influenza viruses. Annually, the three strains believed to be the most prevalent for the upcoming season are selected. This information is communicated worldwide to governments and vaccine manufacturers, enabling governments to begin to forecast their vaccine dose requirements and manufacturers to develop the vaccines.

22. Exhibit 1 illustrates the influenza vaccine supply continuum within Canada. The impetus for acquiring vaccines is provided through the role of the government to ensure the health of Canadians, a coordinated F/P/T approach to health care, a security of supply in Canada and a Canadian capacity to produce vaccines. The World Health Organization, Health Canada and the inter-jurisdictional working groups provide direction. The F/P/T, the Program and the suppliers then implement the acquisition of vaccines. This results in a coordinated response that aims to achieve a Canadian supply and capacity, thereby protecting the health of Canadians.

23. Health Canada's role is to maintain regulatory measures on vaccine strains, approve vaccine strains and set capacity requirements. Health Canada is also responsible for vaccine safety, along with PHAC.

24. PHAC's mission is "to promote and protect the health of Canadians through leadership, partnership, innovation and action in public health". It assists the provincial and territorial governments to set common policy and supports the process by which they purchase vaccines. PHAC works on vaccine recommendations with its technical body, the National Advisory Committee for Immunizations.

25. PHAC is directly involved in building capacity for the influenza pandemic supply. It works with PWGSC to test pandemic vaccines, undergo clinical trials and determine the requirements for equipment and facilities. A team within PHAC's Centre for Emergency Preparedness is responsible for acquiring vaccines and managing the National Emergency Stockpile.

26. The Vaccine Supply Working Group is a forum where all jurisdictions meet monthly to review Canadian vaccine supply issues. The Group makes recommendations to the Canadian Immunization Committee and PHAC on accessing a high quality and secure vaccine supply at the best international prices. Its objectives include making recommendations for contracts on vaccines and monitoring vaccine supply and prices. It also develops principles, guidelines and strategies for addressing vaccine supply issues, such as vaccine shortages.

27. The current Group was created in 2004, replacing the F/P/T Bulk Purchase Committee, which had been co-chaired by PWGSC and rotating regional representatives. The Group is supported administratively by PHAC, with co-chairs from PHAC and one rotating jurisdictional representative. PWGSC is represented on the Group by two non-voting regular members from the Program who have an advisory role and provide expertise on procurement issues. The Group is key to coordinating Canadian vaccine procurement, raising problems and resolving issues.

Exhibit 1: Influenza Vaccine Supply Continuum

A larger image and text description of Exhibit 1 are available on a separate page. Due to the size of the image, it may not display properly.

Procurement Process

28. The Program manages two different processes to procure drugs and vaccines. For influenza and the five other vaccines, PWGSC contracts with two suppliers on behalf of the F/P/Ts in order to ensure security of supply. PWGSC pays the suppliers their individual prices and subsequently invoices the F/P/T members a blended price. The other vaccines and drugs are procured traditionally with PWGSC issuing the contracts and the F/P/T members placing the orders and paying the suppliers directly.

Process for the Procurement of Influenza and Five Vaccines

29. For the influenza vaccine, the annual procurement process normally begins in December when the Influenza Vaccine Project requests from its clients a forecast of the quantities required for their respective influenza programs. In January, the clients provide their estimates. In May, the Project confirms with suppliers the quantities and contracts for that year's production, the shipment date and the delivery addresses of the F/P/T clients. All provincial and territorial jurisdictions acquire the influenza vaccine through the Project.

30. In response to client demands that influenza vaccines be purchased from multiple sources to ensure a security of supply, the Project negotiates with more than one influenza vaccine supplier. Once the contracts are in place, the Project orders the vaccines from the suppliers.

31. The vaccines are delivered directly to the F/P/T clients and shipping notices are sent to the Project. At no point does the Project take possession of the vaccines. The Project verifies the delivery and condition of the orders with the individual clients. The Project pays the supplier invoices, using the Optional Services Revolving Fund, and invoices the clients to recover the costs.

32. In the case of expired or surplus vaccines, the client returns them directly to the supplier and advises the Project, which verifies this with the supplier. In the case of surplus vaccines, the Project will initially attempt to facilitate shipping the vaccines to another jurisdiction that needs them, rather than returning them to the supplier. Similarly, should one jurisdiction have a shortage, the Project will become involved to help find another jurisdiction with a surplus or that can share some of its supply.

33. When jurisdictions return unused or expired vials to the supplier, the supplier applies a credit against the Program's account. Each contract has a maximum quantity of vials that can be returned for credit. For the influenza vaccine, [ * ] of what is sold can be returned within four months following the date of expiry. The supplier destroys returns that exceed [ * ] maximum.

34. For the five non-influenza vaccines—Measles, Mumps & Rubella (MMR), Chicken Pox, Rabies, Pneumococcal and Meningococcal—the process is the same as the influenza vaccine procurement process; however, it is managed by the Division. Clients have also identified these vaccines as needing multiple suppliers to ensure a security of supply. A blended price is also charged. These vaccines are purchased by the Division, the supplier is paid using the Optional Services Revolving Fund and clients are invoiced to recover the costs.

Blended Price

35. To charge the same price to all jurisdictions for the influenza and other five vaccines they procure, the Program uses a "blended price" formula. The cost of the higher priced vaccine is blended with the lower one, based upon the quantities obtained from each supplier. The blended price is calculated at the beginning of the year, based upon estimated quantities to be purchased from the respective suppliers.

36. Once the vaccines are delivered to the jurisdictions, the Program pays the supplier at the "real" cost and invoices the jurisdictions at the blended price. The Program adjusts the blended price throughout the year, based on changing quantities ordered at the different prices. The blended price is currently set to 3 decimal places to recover the product costs and achieve a breakeven objective.

Process for the Procurement of the Non-Influenza Vaccines and Drugs, including Antivirals

37. The Program uses the traditional PWGSC procurement model for the remaining vaccines and drugs, including antivirals. The Drugs, Vaccines and Biologics Division establishes contracts for the vaccines and drugs. The Division may conduct market research, produce product documents indicating availability and assist clients to define their requirements. Once the client decides on its requirements, the Division publishes a detailed request for proposal or a request for a standing offer and elicits, evaluates and selects a bid. Client requests are pooled and solicited together to acquire one contract at a better price than if they had gone as separate bids. Finally, it awards and administers the contracts. PWGSC does not order and invoice for these vaccines and drugs; clients order directly from the suppliers and the suppliers invoice the clients.

38. The Influenza Vaccine Project also uses the traditional procurement model for antivirals. The Project establishes and manages the contracts for clients to procure Relenza and Tamiflu. The actual ordering and purchasing is done directly by the clients.

Emergency Preparedness

39. To prepare for an influenza pandemic in Canada, the Project administers vaccine-related contracts to ensure the creation of a Canadian production capacity and assures the availability of the needed raw materials and supplies. This represents equipment and facilities, such as facilities to house the hens that produce the eggs used to make the vaccines. This also includes the development and implementation of a pandemic operations plan. In November 2005, a supplier contract was amended to include the requirement for the development, production and clinical trials of mock or pandemic-like vaccines.

40. In preparation for other possible pandemics, the Project's role is to establish and manage contracts for the procurement of antivirals. It also negotiates the supply of a large quantity of the antiviral Relenza for Canada's National Stockpile, which is managed by PHAC.

41. In addition, the Division contracts on behalf of federal departments to stockpile a supply of vaccines and other treatments for medical countermeasures. Medical countermeasures are taken proactively or reactively to maintain sustainability through the prevention and pre-treatment of injury from nuclear, biological and chemical agent hazards, such as anthrax and smallpox. As these are usually available from a sole supplier, a blended price is not used. PWGSC establishes the contract, followed by direct delivery and payment between the supplier and client.

Resources

42. In 2007-2008, the Program was responsible for total contract expenditures of $301.6M on behalf of the F/P/T jurisdictions, covering vaccines, drugs and antivirals. In 2007-2008, the Program managed 64 contracts, comprising 25 vaccine contracts, ten drug standing offers, 28 drug contracts and one Prime Vendor standing offer. Vaccines comprise approximately $254.5M, of which [ * ] was for [ * ] million doses of influenza vaccines.

43. Administrative costs, defined as covering 11 full-time staff, accommodation, overhead, regular operating expenses and corporate services, were $1.4M in 2007-2008. The A-base provides funding for the administrative costs that are associated with the services that are provided to the federal jurisdictions. An annual administration fee is charged to each province and territory to cover the costs associated with the work on provincial and territorial files.

44. The administration costs covered by the fee charged to the provinces and territories are based on calculations by the Program that approximately 60% of the level of effort by Program staff is to support provincial and territorial work and approximately 40% of the level of effort is to support federal work. PWGSC's Finance Branch agrees with these calculations. Therefore, the administration fee charged to the provinces and territories does not cover all costs to administer the Program, but only the portion calculated for the costs associated with work being done for them. The Program estimates that the level of effort on federal files is heavier for a considerably smaller amount of purchases than for larger non-federal files. This is due to different response styles by federal clients. Some sections of the Program contract solely on behalf of federal clients, such as the stockpile for medical countermeasures by the Department of National Defence.

45. In 2007-2008, the fee charged to recover Program costs was $10,000 per jurisdiction, producing $130,000 in revenue. The Program is currently proposing to increase the fee to $20,000 per jurisdiction in 2008-2009 and $60,000 in 2009-2010.

Logic Model

46. A logic model is an essential tool in conducting an evaluation study. It is a visual representation that links a program's activities, outputs and outcomes, provides a systematic and visual method of illustrating the program theory and shows the logic of how a program, policy or initiative is expected to achieve its objectives. It also provides the basis for developing the performance measurement and evaluation strategies.

47. A logic model of the Program is presented in Exhibit 2. It was developed based on a detailed document review, meetings with Program managers and interviews with key stakeholders. It was subsequently validated with Program staff.

Activities

48. The activities section of the logic model lists the actions the Program carries out that contribute to the achievement of the outputs and outcomes.

49. The activities of the Program include the procurement of vaccines, drugs and antivirals on behalf of F/P/T jurisdictions, contract administration, client services and outreach to the health and vaccine communities.

50. The activities resulting from the procurement role include selecting the procurement approach, preparing the evaluation criteria and contractor selection methodology, as well as selecting and awarding the contracts. Contract Administration includes verifying supplier fulfillment of contractual requirements, paying suppliers and invoicing clients as appropriate and providing contract quality assurance. The Program also provides client services, such as coordinating the Group Purchasing Program, developing MOUs with provincial and territorial governments, liaising with F/P/T clients, mediating between clients and suppliers and providing procurement advice and expertise to clients. The community outreach role includes participation in community committees and working groups, liaising with the health community and providing procurement expertise and advice.

Outputs

51. Outputs are concrete and direct results of the activities. Outputs resulting directly from the activities include contracts being awarded and managed, the required drugs, vaccines and antivirals being procured and procurement expertise being delivered. In addition, it includes the acquisition of a safe supply of medication in case of emergencies.

52. Outputs resulting from the contract administration activity include contracts being administered, goods being delivered and verification that suppliers respect their commitments. It also includes suppliers being paid and costs being recovered from clients as appropriate. Finally, the Program ensures that the quality of the contracts is maintained.

53. The outputs from the client services activity include coordinating the Group Purchasing Program and ensuring the MOUs with the clients are signed. The activities also result in ensuring contracts are functioning successfully and sharing information at the national level.

54. The community outreach outputs include participating in community committees and working groups, ensuring good interaction between PWGSC and the health community and ensuring transparency of activities.

Desired Outcomes

55. This logic model has three levels of outcomes: immediate, intermediate and ultimate. The Program has more direct influence over the immediate/short-term outcomes, which span approximately one to two years from the activities. This is followed by the intermediate/medium-term outcomes, which occur in the following three to five years. The ultimate/long-term outcomes take over five years or longer to achieve and generally are subject to influences beyond the Program.

56. At the immediate outcome stage, it is expected that the F/P/T jurisdictions realize savings through cost-effective acquisitions and that they have access to fair, efficient, knowledgeable and responsive acquisition services.

57. The intermediate outcomes flow from the activities, outputs and immediate outcomes and external influences. At this point, the Program outcome is for PWGSC to contribute to Canadians benefiting from an enhanced security of supply, obtaining best value for money and benefiting from a Canada-wide solution for the procurement of drugs and vaccines.

58. Ultimate outcomes result from the activities, outputs, immediate and intermediate outcomes and external influences outside the Program's control. The current desired outcome is that Canada is pandemic ready and fosters good public health.

Program Risks

59. As shown in Exhibit 2, risks to the Canadian vaccine supply can arise from issues associated with the security of the supply, such as the risk of an insufficient supply of vaccines, drugs and/or antivirals or the manufacturing of the wrong strain of vaccines. There are also risks related to the Canadian capacity to produce emergency supplies. Another possible risk is linked to the health community and its isolation from the multi-jurisdictional community, producing an uncoordinated F/P/T response to procurement and emergency preparedness. In addition, there is the risk of unknown pandemics, for which no antivirals or vaccines exist. The risks linked to the ultimate outcome include the F/P/T jurisdictions' ability to deliver their programs, the Government of Canada's preparedness for an emergency or pandemic and the Government of Canada's ability to ensure public health care resulting in increased illness or death.

Exhibit 2: Logic Model

A larger image of Exhibit 2 and a text alternative are available on a separate page. Due to the size of the image, it may not display properly.

FOCUS OF THE ENGAGEMENT

60. The evaluation assessed the relevance and performance of the F/P/T Group Purchasing Program for Drugs and Vaccines. An evaluation matrix, including evaluation issues, questions, indicators and data sources, was developed during the planning phase (Appendix A: Evaluation Matrix). More information on the approach and methodologies used to conduct this evaluation are in the section: About this Evaluation.

Objective

61. Evaluate the F/P/T Group Purchasing Program for Drugs and Vaccines in accordance with the Evaluation Policy in order to determine the Program's relevance¹ and performance² in achieving its planned outcomes. The evaluation also explored alternative ways of achieving the expected results.

Scope

62. The evaluation examined the F/P/T Group Purchasing Program for Drugs and Vaccines within the Commercial Acquisitions and Supply Management Sector of the Acquisitions Branch. The Program includes the Influenza Vaccine Project and the Drugs, Vaccines and Biologics Division, through which PWGSC provides contracting and related services for the procurement of drugs and vaccines by F/P/T governments. The evaluation assessed PWGSC's procurement of drugs and vaccines on behalf of the F/P/T governments, as well as the Program's other roles to support this activity.

FINDINGS AND CONCLUSIONS

63. The findings and conclusions presented below are based on multiple lines of evidence used during the evaluation.

Relevance

Evaluation Issue: Is the F/P/T Group Purchasing Program for Drugs and Vaccines consistent with departmental and government-wide policies and priorities; and does it address a continuing need?

Findings

Is the PWGSC F/P/T Group Purchasing Program for Drugs and Vaccines consistent with departmental/government-wide policies and priorities?

64. Legislation and policies exist that provide the mandate for PWGSC to procure drugs and vaccines for F/P/T jurisdictions. The Department of Public Works and Government Services Act and the Common Services Policy provide the basis for the Program to procure the drugs and vaccines as a centralized service. A 1986 Order in Council gives PWGSC the authority to provide non-federal organizations with services, upon request, that are consistent with its mandate. The Memoranda signed with the provincial and territorial jurisdictions request PWGSC to act on their behalf. The Program's objectives and intended outcomes support the requirements of the provincial and territorial jurisdictions to purchase drugs and vaccines to support delivery of their respective programs. The 2003 National Immunization Strategy demonstrates a Government of Canada priority to maintain a collaborative F/P/T approach to health care and to specifically address issues such as the achievement of best value for vaccine procurement, long-term security of supply and quality of supply.

To what extent is PWGSC's role in vaccine and drug supply and emergency preparedness appropriate to the Government of Canada?

65. The Health Act and the National Immunization Strategy both support a Government of Canada role in ensuring the delivery of health care programs. The Department of Public Works and Government Services Act, the 1986 Order in Council and the Common Services Policy make it appropriate for the Government of Canada to provide a centralized procurement role.

Is there an ongoing need for a centralized vaccine and drug procurement program, as is now provided through the PWGSC F/P/T Group Purchasing Program for Drugs and Vaccines?

66. The provinces and territories need to purchase vaccines, drugs and antivirals as part of their health mandate. To do this effectively and with value for money, they need to have bulk purchasing power and procurement expertise. None of the provinces or territories are required to participate in the Program, but all chose to. Voluntary participation in the Program and the tremendous growth of the Program (see paragraphs 76-77) provide evidence of a need for the services offered by the Program.

67. In interviews and the questionnaire, Program stakeholders indicated they believe there is an ongoing need for a Canada-wide centralized procurement approach for vaccines. Half the respondents believe there is also a need for a centralized procurement for antivirals, especially for emergency planning. This question did not apply to all respondents. Responses also indicate support to purchase drugs for stockpiling towards the pandemic influenza response.

68. Evidence from the interviews and questionnaires indicates support from all respondents for the federal government to be the centralized procurer for vaccines. Most respondents also support this for antivirals. It was noted that the centralized system ensures preferred dose pricing, an enhanced security of supply and a fair process for the distribution of vaccines.

69. Most of the respondents in both the interviews and the questionnaire note the benefits of a centralized program, including assistance in the reallocation of vaccine or drugs during shortages or overages. For example, when a jurisdiction will not use all of its vaccines or drugs, it contacts the Program to locate another jurisdiction that needs the same product, thereby facilitating the reallocation. During shortages, the Program has coordinated all the jurisdictions to assist with an equitable allocation based upon need as indicated by the jurisdictions.

70. Another benefit of a centralized program was noted during the interviews with Program stakeholders. Regardless of jurisdictional size and levels of economic benefits from the Program, some jurisdictions said the Program makes them good corporate citizens and increases their knowledge of issues being faced by other provinces and territories.

Conclusions

71. All lines of evidence indicate that the Program is relevant and consistent with departmental and government-wide policies and priorities. It is supported through legislation and policies. The Program's collaborative and nation-wide approach is consistent with the National Immunization Strategy and its aim of achieving best value for vaccines and long-term security of supply.

72. As the federal department responsible for the acquisition of goods and services for the federal government and as it has been authorized to work on behalf of provincial and territorial jurisdictions, PWGSC has an appropriate role to procure drugs and vaccines on behalf of F/P/T jurisdictions.

73. Evidence of ongoing voluntary participation by all provinces and territories, along with substantial growth in procurement levels, demonstrates a strong relevance for a nation-wide centralized procurement as is now provided through the Program. All stakeholders indicate there is a need for a centralized Canada-wide program as is currently provided through the Program.

Performance

Evaluation Issue: Is the F/P/T Group Purchasing Program for Drugs and Vaccines achieving its intended outcomes?

Findings

Do the F/P/T jurisdictions realize savings through cost-effective and value for money acquisitions?

74. Evidence from the document review, financial analysis, interviews and the questionnaire indicates that all levels of government realize savings through the use of the Program. As all jurisdictions pay the same prices for the influenza vaccines, regardless of the size of the order or the distance to ship, many smaller jurisdictions realize savings that would be impossible were they not part of the Program. If the Program were purchasing only for the federal jurisdictions, it would not benefit from the reduced costs from bulk purchasing.

75. Jurisdictions find that participation in the Program provides them with access to cost-effective acquisitions, which enables them to realize efficiencies, have easier access to goods and services and receive support for their program delivery. As a result of the Program, the majority of respondents indicate they spend fewer resources on procurement within their jurisdiction.

76. Since its creation in 1976, all provinces and territories have gradually entered into agreements with the Program. By 1999, all provinces and territories were purchasing influenza vaccines through PWGSC. Over the past ten years purchases have more than doubled, from [ * ] million doses in 1998-1999 to [ * ] million in 2007-2008 (or respectively from [ * ] to [ * ]). With the negotiation of the influenza contract in 2001, there was a considerable jump in one year in the total influenza doses purchased from [ * ] million in 1999-2000 to [ * ] million in 2000-2001 with purchase amounts increasing from [ * ] to [ * ] (Exhibit 3).

Exhibit 3: F/P/T Influenza Vaccine Purchases-Total Doses (in millions)

	Year
Total	1998- 1999	1999- 2000	2000 - 2001	2001- 2002	2002- 2003	2003- 2004	2004- 2005	2005- 2006	2006- 2007	2007- 2008
Doses	[ * ]	[ * ]	[ * ]	[ * ]	[ * ]	[ * ]	[ * ]	[ * ]	[ * ]	[ * ]
TotalCost	[ * ]	[ * ]	[ * ]	[ * ]	[ * ]	[ * ]	[ * ]	[ * ]	[ * ]	[ * ]
¤ Cost Per Dose	[ * ]	[ * ] [ * ]	[ * ] [ * ]	[ * ] [ * ] [ * ]	[ * ] [ * ] [ * ]	[ * ]	[ * ]	[ * ]	[ * ]	[ * ]

¤ Cost per dose: Where there is one amount, it is the blended price; where there are several prices, the blended price was not available and each individual contract price is provided.
Source: Federal/Provincial/Territorial Committee on Group Purchasing of Drugs and Vaccines Annual Reports from 1998-1999 to 2007-2008.

77. There has also been considerable growth in the purchase of vaccines, drugs and antivirals from 2001-2002 to 2007-2008, as indicated in the contract expenditure in exhibit 4 below. The most notable increase occurred in vaccines, other than influenza, which grew over 15 times in overall totals from $13.6M to $212.9M.

Exhibit 4: F/P/T Contract Expenditures (in millions of dollars)

Type	Jurisdiction
	P/T		Federal		Total
	2001-2002	2007-2008	2001-2002	2007-2008	2001-2002	2007-2008
Influenza Vaccine	Not Available	[ * ] [ * ]	Not Available	[ * ] [ * ]	[ * ]	[ * ]
Vaccines (excludes Influenza)	[ * ] [ * ]	[ * ] [ * ]	[ * ] [ * ]	[ * ] [ * ]	[ * ]	[ * ]
Antivirals	Not Available	[ * ] [ * ]	Not Available	[ * ] [ * ]	Not Available	[ * ]
Drugs	[ * ] [ * ]	[ * ] [ * ]	[ * ] [ * ]	[ * ] [ * ]	[ * ]	[ * ]
TOTALS		[ * ]		[ * ]		[ * ]

¤(%) Indicates the percentage the provincial and territorial or federal amount represented in the total figures for the given year.
Source: Federal/Provincial/Territorial Committee on Group Purchasing of Drugs and Vaccines Annual Reports from 2001-2002 to 2007-2008

78. The document review and interviews show that F/P/T clients find the Program provides them with value for money. Smaller provinces and territories have told the Program that they benefit anywhere from 10% to 50% on the purchase price of vaccines and/or drugs by purchasing through the Program. In a recent example, between the 2006-2007 and 2007-2008 prices negotiated in the contracts with two suppliers for the supply of pertussis-containing vaccines, the savings for the jurisdictions are estimated by the Program to be $52.1M, excluding savings for option years.

79. The negotiating power of the long-term influenza contract, along with the bulk purchasing power, enables the Program to achieve substantial value for money for the jurisdictions. In addition, some respondents indicate that because of their participation in the Program, jurisdictions do not have to compete with each other to establish contracts with suppliers, potentially driving up prices and lower efficiencies.

80. The Program reports that consolidation of the influenza vaccine purchases for all of Canada have resulted in the
[ * ]; however, these figures were unavailable to the evaluation for confirmation. More recently, it can be established that the[ * ].

81. In the United States, the Centers for Disease Control's [ * ]. Exhibit 5 shows the blended price charged by the Program to the F/P/T clients compared to the posted American price. The American price includes excise tax and indemnity insurance, which are not charged in the Canadian blended price.

Exhibit 5: Influenza Dose Prices - F/P/T blended Price Compared to the American Centers for Disease Control Prevention

Cost Per Dose	Year
	2003-2004	2004-2005	2005-2006	2006-2007	2007-2008
F/P/T	[ * ] [ * ]	[ * ]	[ * ]	[ * ]	[ * ]
Centers for Disease Control	$8.41 $11.46	$8.92	$11.36	$11.58	$10.10

Source: Federal/Provincial/Territorial Committee on Group Purchasing of Drugs and Vaccines Annual Report; information from 2003-2004 to 2007-2008; Centers for Disease Control List Vaccine Price from December 22, 2005. All Prices are in Canadian Dollars.

Program Costs

82. Based on the Common Services Policy, optional services should be fully cost-recovered. Therefore, the fees the Program collects from the provincial and territorial jurisdictions must cover the cost of providing these services. In 2007-2008, the fee charged to recover its costs was $10,000 per jurisdiction, producing $130,000 in revenue. The Program recognizes that this does not cover the current costs and has proposed to increase the fee to $20,000 per jurisdiction in 2008-2009 and $60,000 in 2009-2010.

83. In 2007-2008, the Program costs $1.4M to administer. To cover the level of effort done by Program staff to support provincial and territorial work, the Program should be recovering $780,000 from these jurisdictions or approximately $60,000 each. These calculations are based on a 2007-2008 desk audit conducted by the Program, which indicated the actual proportion of work completed on behalf of the provinces and territories was 56.9%.

Blended Price

84. Twelve out of the fourteen respondents to the questionnaire believe that blended pricing is the most appropriate pricing mechanism for the vaccines.

85. The difficulty with the blended price is that the actual quantities purchased from the different suppliers during the year do not always match the initial estimates used to establish the blended price. As jurisdictions are required to submit their estimated quantities with considerable lead-time, the actual quantity required may be quite different from the estimates. In addition, for various reasons such as production difficulties and unforeseen urgencies, it may be necessary to purchase more doses at higher or lower prices, depending on which supplier is able to meet the demand. The blended price usually underestimates the volume that the Program will buy from the lower cost supplier and assumes the Program will be buying more quantities from the higher cost supplier. This makes it difficult to finalize the annual blended price per dose and invoice the cost accurately.

86. In addition, returns create two difficulties. First, the Program has already invoiced the jurisdiction and received a payment, which does not include the credit subsequently applied to the account by the supplier. Second, the vials may be returned in a different year from when they were purchased, as the cycle for purchasing vaccines does not match the fiscal cycle, resulting in transactions occurring within different fiscal years.

87. The impact of the fluctuating blended price, the difficulty of fixing actual costs within a specific fiscal year and the credits applied to the account is that the Program can inadvertently recover more money from the jurisdictions than it paid the suppliers for the vaccines. This results in the Program showing a surplus in the Optional Services Revolving Fund, as calculated by the Finance Branch, including $960K in 2006-2007, $184K in 2005-2006 and $747K in 2004-2005. In 2007-2008, the vaccine program had a surplus of approximately $1.1M as the contractor did not bill for clinical trials as anticipated. This has been addressed with Finance and will be addressed in future years with credits to the F/P/T, should a similar situation arise. According to the Program, aside from this specific case, variances of +/- $300K are the norm—this is approximately .5% on this $60M program.

Are there more cost-effective alternatives for procuring drugs and vaccines for Canadian requirements?

88. Most respondents to the interview and questionnaire did not suggest any alternatives to the Program and believe that it is the most cost-effective way to procure drugs and vaccines. While some indicated that a possible alternative could be to have the provincial and territorial jurisdictions purchase their respective drugs and vaccines, they also discounted this as an unreasonable alternative that could hinder them from delivering their programs.

89. While a number of other jurisdictions were examined to locate alternative delivery models, the American and British models were of particular interest. Although the United States and the United Kingdom have different health care jurisdictional responsibilities from the Canadian system, evidence was found that they are moving towards centralized purchasing to leverage greater buying power and lower costs. Interviewees, including American officials, indicate that the United States is actively regarding the Vaccine Supply Working Group and the Program as possible models to adopt. Also, the Program has been asked by other countries to present on its model.

90. Each year, the Program conducts price comparisons with American prices listed by the Center for Disease Control and the Program prices always compare very favourably to the US prices. While the American prices include a "liability contingency surcharge" that is not included in Program prices, interviewees believe the price difference is due to the centralized nature of the Canadian system whereby the suppliers deal with one client. The prices for the United Kingdom are not publicly available.

United States

91. Annual influenza vaccine use in the United States has increased from about 10 million doses in 1976, at the time of the swine influenza scare, to about 80 million doses in 2003-2004. For the 2007-2008 influenza season, six American-licensed manufacturers were expected to produce an estimated 132 million doses of seasonal influenza vaccine. Influenza vaccine production and distribution in the United States are primarily private sector endeavours and vaccines are ordered and delivered at the local level to individual distributors, including doctors' offices and retail stores. There is currently no centralized government-led system comparable to PWGSC's Program.

92. However, in 2007, the National Center for Immunization and Respiratory Diseases created a consolidated national system to more efficiently inventory and distribute publicly purchased vaccines. The system relies on a single, centralized distributor and a network of existing vaccine depots to store and ship vaccines at the state and local level. By November 2007, more than half of the American Centers for Disease Control's 64 immunization program grantees were using the new system. This has reduced the number of depots by 36%, further improving distribution efficiency.

93. The Center is also developing a centralized vaccine ordering system to simplify and streamline how providers place orders. This will enable it to track vaccine use, forecast future vaccine needs and potential changes in program cost, as well as assist in future emergency response.

United Kingdom

94. The United Kingdom Department of Health is the largest purchaser of vaccines in Europe, spending approximately £200M annually on centralized purchase and distribution for national immunization programs. It uses the Purchasing and Supply Agency of the National Health Service to procure vaccines as part of its routine vaccination program. The Department invites tenders for vaccine contracts stipulating desirable criteria that the vaccine should satisfy, such as safety, efficacy and price. Vaccines are included in the main drug pricing negotiation between the Association of the British Pharmaceutical Industry and the Government.

95. Procurement arrangements within the Department were highly devolved, but in 2003 were being strengthened and improved. The Department recognized that there were tensions between achieving the benefits of local autonomy in purchasing decisions and leveraging the considerable negotiating power of a buyer as large as the National Health Service. It was decided that local bodies needed to be part of a national system to achieve the right balance.

Do F/P/T jurisdictions have access to fair, efficient, knowledgeable and responsive acquisition services?

96. Clients indicate in interviews and the questionnaire that they have high overall confidence with the Program and are highly satisfied with the services provided by the Program. Discussions during the interviews indicated satisfaction with Program services, with one respondent commenting that the Program staff are an "awesome team—many work above and beyond their call and duty".

97. Evidence from the interviews and the questionnaire demonstrates that the Program provides fair, efficient, knowledgeable and responsive services. Most respondents noted a key benefit of using the Program was the depth of knowledge they were able to access on procurement and vaccine- or drug-related issues. The Program is recognized for its rapid response to shortages or problems with suppliers. When a supplier is unable to provide the required doses, the Program locates a new supplier. The Program has demonstrated it provides responsive acquisition services, such as creating a dual sourcing on vaccines, in response to client demands.

98. From interviews and the questionnaire, all respondents indicate they are generally quite satisfied with the Program's contract administration. Most respondents believe that Program staff manage the contracts efficiently and respond to the jurisdictions' needs in a timely manner. Generally, respondents believe they have enough lead-time to provide orders to the Program and almost all of the respondents said the contracts meet their requirements. They find the contracts negotiated by the Program are easy to use and that Program staff are knowledgeable about contract details. It was noted that the Program is very inclusive in discussing the contracts and assists jurisdictions in looking at the options and impacts of the various contracts in a realistic manner.

99. With regards to complaints and issues with suppliers, most respondents find the Program resolves complaints or issues in a timely manner and most believe that resolutions are made to their satisfaction. There were a small number of respondents who expressed an interest to have feedback on the resolution of issues and a mechanism, such as problem reports, to bring forward specific issues.

100. While most find the procurement procedures easily accessible, a few indicated they would like more information about the procedures. It was noted by a few that there have been delays in issuing contracts, numerous contract amendments and a lack of translation of contracts.

Do Canadians benefit from an enhanced security of supply?

101. The literature review and interviews show that Canada faces several risks to the security of the vaccine and drug supply. Vaccine production is a complex and lengthy process, which impacts on the ability of suppliers to respond rapidly to requirements and emergencies. The global manufacturing base is consolidating and contracting, resulting in a small number of companies becoming responsible for most of the production. The complexity of vaccine production makes it less predictable compared to pharmaceutical production and spoiled or unusable batches can take months to replace. Additional risks include suppliers being unable to produce their drugs and vaccines, the wrong strains being selected for the vaccines, the inability to obtain drugs or vaccines in Canada in the case of an emergency and the long lead time to develop new drugs or vaccines.

102. For some drugs or vaccines, only one supplier has the capacity to meet the F/P/T requirements and Canada has to compete with international demands. Long lead times (12-18 months) required by manufacturers to manufacture drugs and vaccines make it difficult for the jurisdictions to accurately forecast their requirements, resulting in overages and shortages. In addition, newly developed vaccines are often more expensive, which results in escalating prices.

103. Evidence demonstrates that the Program has put in place dual award contracts for its vaccines and drugs to ensure a security of supply. The Program works with other jurisdictions to ensure a capacity in Canada to produce vaccines. Respondents note that the centralization of the Program ensures a rapid sharing of information to ensure everyone is aware of any issues or problems. All respondents agree that a federally centralized purchasing program contributes to a continuity of supply by coordinating the redistribution of vaccines and drugs in the cases of shortages, overages and back orders.

Do Canadians benefit from a Canada-wide solution to procure vaccines and drugs?

104. Stakeholders indicate that the Program provides a Canada-wide solution to vaccine issues. According to its mandate, all Program participants are equal and the size of the order or the location of the delivery does not impact on a participant's access to vaccines or drugs, either in the costs of the doses or the delivery. Representatives from a larger province indicated in an interview that being part of the Program had made them better "corporate citizens" from a Canada-wide perspective, while others noted the national scope of the Program enabled a better understanding of vaccine issues across the country. It was noted that there is less likely to be a conflict of interest amongst the provinces and territories when it is the federal government providing vaccine procurement services, rather than the provinces and territories.

105. Evidence also indicates that the Program contributes to the cohesion of the health community by its involvement in numerous inter-jurisdictional working groups such as the Vaccine Supply Working Group, Interdepartmental Vaccine Supply Committee and the Antiviral Stockpile Management Task Group. Most respondents indicated that a valued benefit of the Program in these meetings is its Canada-wide perspective, which enables the Program to put local issues into a national perspective and indicate when an issue might have a larger impact. By increasing awareness of the Canada-wide aspect, the Program raises issues that assist jurisdictions in having an improved national perspective, thereby increasing community cohesion.

Does Canada have pandemic/emergency preparedness?

106. The Program contributes to the continuity and security of supply for emergency preparedness by administering vaccine-related contracts to ensure the creation of a Canadian production capacity and assures the availability of raw materials and supplies. It also includes the development and implementation of a pandemic operations plan. The Program awards and administers contracts for the procurement of antivirals and contracts on behalf of federal departments to stockpile vaccines and other treatments for medical countermeasures.

107. Questionnaire respondents note that PWGSC contributes towards mitigating the risks of an emergency, especially through its use of multiple suppliers and managing dual award contracts. Most of the respondents agree that PWGSC contributes to emergency preparedness.

Does the Canadian Government foster good public health?

108. The Program enables the F/P/T jurisdictions to deliver their health programs through the cost-effective acquisition of vaccines, drugs and antivirals. The Program contributes towards an enhanced Canadian security of supply through its multiple suppliers. The Program provides a Canada-wide perspective and contributes towards emergency preparedness. All these factors contribute to enabling the Canadian Government to foster good public health.

What are the barriers to successful project delivery of the F/P/T Group Purchasing Program for Drugs and Vaccines?

109. Barriers to the successful delivery of the Program could include acquisitions no longer being cost-effective, a loss of supply, differences amongst the Program participants, participants leaving the Program or poor acquisition services.

110. The Program faces the ongoing challenge of increasing prices for drugs and vaccines, which is compounded by a lack of alternative manufacturers. When the current influenza vaccine contract terminates in March 2011, prices negotiated for the new contract will likely increase considerably, which could reduce the cost-effectiveness of the Program.

111. In creating contracts to enable different levels of government in different regions to deliver their respective programs, the Program works to coordinate and reconcile diverse groups with competing needs and expectations. It needs to balance the requirements and needs of federal and provincial/territorial clients that are vastly different in size and located in geographically diverse locations. These diverse interests and requirements could present a barrier to the successful delivery of the Program.

112. As a voluntary program, there is always a possibly that some participants could decide to opt out of the Program. This would create a barrier to successful delivery of the Program in that it would reduce cost-savings achieved through bulk purchasing and increase administrative costs as there would be fewer jurisdictions to pay the administrative fee.

113. A barrier to a program's success is a lack of measurement of its outputs and outcomes, as well as lack of reporting of problems or issues. The key performance measure for the Program is the continuing voluntary participation of all jurisdictions. The Program is in the process of delivering performance measures on its outputs, as described in the PWGSC Performance Measurement Framework 2008-2009. This requires performance measures of:

• Percentage of contracts issued on time,
• Unit vaccine costs compared to US prices,
• Percentage of shipments delivered on time,
• Number of contracts issued and
• Number of shipments negotiated.

114. The targets for these data are to be achieved by March 31, 2009, with some of them already in place. In addition, the Program documents contractual issues and their outcomes. However, these measures do not fully address outcomes, such as savings achieved through cost-effective acquisitions. Also, they do not address client satisfaction with Program services.

115. An additional form of performance measurement comes from the minutes of the Vaccine Supply Working Group meetings, which provide a record of technical issues raised and an indication of how they were resolved. The Group is formalizing a process to record problems and complaints. However, while these provide a record of any problems, especially with regard to suppliers, the focus is not specifically on the Program and the measurements are neither taken nor kept by the Program.

Conclusions

116. Based on evidence in document reviews, interviews and the questionnaire the Program is achieving its intended outcomes. F/P/T clients realize savings through cost-effective acquisitions and achieve value for money. Jurisdictions benefit from being part of a larger purchasing group. All clients are treated equally no matter their size or buying power.

117. Clients have high overall confidence with the Program and are highly satisfied with the fair, efficient, knowledgeable and responsive acquisition services. Jurisdictions benefit from the Program's procurement expertise and are generally quite satisfied with the Program's efficient and timely contract administration.

118. The Program contributes to the security of supply in Canada through strategies such as dual sourcing and establishing a Canadian production capacity.

119. The voluntary participation of all the jurisdictions is a measurement of the value and success of the Program. While the Program is in the process of delivering performance measurements on its outputs, the Program outputs, outcomes and client satisfaction are not being fully assessed through internal performance measurements.

120. The Program provides a Canada-wide perspective and increases awareness of national immunization issues. The federal government benefits from participating in the national health community as an active and valued participant.

121. While the Program is providing excellent value for money with regard to the prices negotiated for drugs and vaccines, it is recovering insufficient administration fees to cover its costs for the work provided on provincial and territorial services. While proposals are being made to increase the fee, it could be two years before they are sufficient to cover the estimated costs. In addition, the blended price system is complex, resulting in a difficulty in establishing vaccine prices and accurate invoicing amounts.

122. Evidence was not found of more cost-effective alternatives to procure drugs and vaccines; rather, it appears that similar models are being implemented in other countries.

GENERAL CONCLUSIONS

123. Relevance: The F/P/T Group PurchasingProgram for Drugs and Vaccines is consistent with departmental/government-wide policies/priorities, it is appropriate to the Government of Canada and it does address a continuing need.

124. Performance: The F/P/T Group Purchasing Program for Drugs and Vaccines is achieving its intended outcomes and is providing a cost-effective, value for money service. However, it is not fully recovering its costs due to insufficient administration fees being charged to the provincial and territorial jurisdictions. The complexity of the blended price mechanism is resulting in difficulty in establishing vaccine prices and accurate invoicing amounts.

MANAGEMENT RESPONSE

125. Acquisitions Branch considers the results of this evaluation accurately and fairly reflect the Federal, Provincial and Territorial Group Purchasing Program for Drugs and Vaccines, as of January 2009. Acquisition Branch accepts this evaluation report and intends to act on all recommendations of this evaluation by implementing a Management Action Plan.

RECOMMENDATIONS AND MANAGEMENT ACTION PLAN

126. Based on the key findings and conclusions contained in this report, the Office of Audit and Evaluation recommends that the Assistant Deputy Minister, Acquisitions Branch:

Recommendation 1: Develop performance measurements to fully track Program performance and success.

• Management Action Plan 1: The Program is in the process of delivering performance measures on its outputs, as described in the PWGSC Performance Measurement Framework 2008-2009 as follows:
  • Percentage of contracts issued on time,
  • Unit vaccine costs compared to US prices,
  • Percentage of shipments delivered on time,
  • Number of contracts issued and
  • Number of shipments negotiated.
• In addition to the above, an annual client survey will be conducted.
• This will be done by March 2010.

Recommendation 2: Conduct an analysis of the blended price mechanism to better establish vaccine pricing.

• Management Action Plan 2: The pricing mechanism was reviewed in late 2008 and the MOU was revised to include an annual price review and price adjustment.
• With each annual review the pricing mechanism will be reviewed in consultation with our financial advisors, to ascertain whether additional changes are needed.
• This will be done by March 2010.

ABOUT THE EVALUATION

The evaluation examined the F/P/T Group Purchasing Program, including the Influenza Vaccine Project and the Drugs, Vaccines and Biologics Division. The evaluation assessed the Program's procurement of vaccines, drugs and biologics on behalf of the F/P/T governments, as well as the Program's other roles to support this activity.

Approach

The evaluation was conducted in accordance with the Evaluation Standards of the Government of Canada and the Office of Audit and Evaluation of PWGSC. The evaluation took place between June 2008 and November 2008. To assess the evaluation issues and questions, the following lines of evidence were used:

Document Review: The initial document review was conducted and consisted of gaining an understanding of the Program and its context to assist in the planning phase. Over 50 documents were reviewed in this process. Data from these documents were collected in a matrix and subsequently analyzed. A second phase of the document review was conducted to collect and assess Program data (such as financial, performance measurement, other types of data already collected by the Program). The analysis of Program data contributed to assessing the success of the Program. A bibliography of the principle documents consulted is included in Appendix B: Principal Documents Consulted.

Literature Review: A literature review was conducted. The literature review focused on contextualizing the Program both nationally and internationally, as well as identifying alternative delivery models through an analysis of other jurisdictions. Additionally, the review identified key policy and supporting documentation regarding vaccine procurement issues. A bibliography of the principle documents consulted is included in Appendix B: Principal Documents Consulted.

Interviews: The evaluation team conducted interviews with Program managers and clients from federal, provincial and territorial governments (n=20). The qualitative analysis of the interviews provided information about the relevance and performance of the Program from the perspective of Program users. An interview guide was used. Interviews were also conducted with American vaccine program experts (n=4).

Questionnaire: A questionnaire was distributed to the Program clients, including all provincial and territorial jurisdictions, as well as six federal clients (n=19). Sixteen responses were received, including ten provinces, two territories and four federal jurisdictions. The qualitative analysis of the responses provided information about the relevance and performance of the Program from the perspective of Program users.

Limitations of the Methodology

Document Review: Efforts were made to ensure that all data provided about the Program were reviewed and documented. Due to the large volume of data provided and the complex nature of the vaccine environment, it is possible that elements of some documents were not integrated into the findings. However, every effort was made to systematically identify and categorize data from documents. There were no documents available documenting the creation of the Program in the 1970s and early financial and Program information was unavailable.

Literature Review: The great variety internationally in health systems and procurement models made it difficult to locate systems comparable to the Program. Efforts were made to include relevant material to enable an unbiased and neutral assessment.

Interviews: The sampling technique identified a range of Program clients so that the broadest range of experiences would be documented. A sample of clients was interviewed. As such, the results of the interviews are not necessarily statistically representative of all Program clients. In addition, these individuals represent their own opinions and experiences within the Program.

Questionnaires: Questionnaires were sent to all Program clients. While there was a response rate of 84%, the return rate does not enable a generalization to the overall client population.

The multiple lines of enquiry involved and the number of interviews and questionnaires conducted served to mitigate these limitations.

Reporting

We documented our findings and conclusions in a Director's Draft Report, which was internally cleared through Audit and Evaluation's Quality Assurance function. We provided the Program's Director General with the Director's Draft Report with a request to validate facts and comment on the Report. A Chief Audit Executive Draft Report was prepared and provided to the Office of Primary Interest's Assistant Deputy Minister for acceptance. The Office of Primary Interest has been requested to respond with a Management Action Plan. The Draft Final Report, including the PWGSC F/P/T Group Purchasing Program Management Action Plan, will be presented to PWGSC's Audit and Evaluation Committee for the Deputy Minister's approval in March 2009. Once finalized, the Final Report will be submitted to the Treasury Board and posted on the PWGSC website.

Evaluation Team

Staff of the Office of Audit and Evaluation developed this evaluation under the direction of the Director of Evaluation and the Chief Audit Executive of the Office of Audit and Evaluation, Oversight Branch. The PWGSC Library undertook additional research for the evaluation team.

Appendix A: Evaluation Matrix

Evaluation Questions	Indicators	Data Sources
Relevance: Is the Group Purchasing Program consistent with departmental/government-wide policies/priorities; does it address a continuing need?
1. Is the Group Purchasing Program consistent with PWGSC/Government of Canada (GC) policies and priorities?	Existence of legislation, policy and priority statements that identify the mandate for the Group Purchasing Program Extent of alignment between Group Purchasing Program activities, outputs and intended outcomes with PWGSC/GC policies and priorities Degree to which the Group Purchasing Program's objectives and intended outcomes support or complement other existing programs that are GC priorities (e.g., Health Canada, Public Health Agency Canada) Expressed support for the Group Purchasing Program by government senior management Evidence of attendance and active leadership of the Group Purchasing Program's senior management on Advisory/Steering Committees	Document Review Orders-in-Council Department of Public Works and Government Services Act Canada Health Act Public Health Agency of Canada Act Common Services Policy Canada Health Care Policy National Immunization Strategy Service agreements PWGSC Report on Plans and Priorities PWGSC Departmental Performance Report Acquisitions Business Plan Group Purchasing Program documents Other relevant documents, studies and reports Interviews PWGSC managers including the Program F/P/T representatives as required
2. To what extent is PWGSC's role in vaccine and drug supply and emergency preparedness appropriate to GC?	Documented evidence of legislative, regulatory, policy, mandate that is specifically incumbent upon PWGSC for the delivery of the Group Purchasing Program services Evidence of clearly defined roles and responsibilities with regard to the vaccine and drug supply Existence, usage and effectiveness of alternatives Evidence P/T clients find federal government involvement in the Group Purchasing Program appropriate/beneficial	Data and Document Review DPWGS Act Canada Health Act Public Health Agency of Canada Act Common Services Policy National Immunization Strategy Documents defining roles and responsibilities Performance reports Feedback reports Client testimonials Other relevant documents, studies and reports Interviews / Focus Groups PWGSC managers/staff including the Group Purchasing Program F/P/T staff
3. Is there an ongoing need for a centralized vaccine and drug procurement program, as is now provided through PWGSC's Group Purchasing Program?	Evidence that the Group Purchasing Program provides F/P/T clients with services that support their programs Evidence of an increase in the number of F/P/T jurisdictions that have used the Group Purchasing Program's services since its inception Evidence that the Group Purchasing Program services are still required by its clients Evidence that the Group Purchasing Program contributes to a coordinated response and community cohesion	Data and Document Review Existing client satisfaction feedback Performance/Status reports Statistical data Other relevant documents, studies and reports Service agreements Interviews/Focus Groups PWGSC managers/staff including the Group Purchasing Program F/P/T representatives involved in the Group Purchasing Program
Performance: Is the Group Purchasing Program achieving its intended outcomes?
4. To what extent is the Group Purchasing Program achieving its intended outcomes?	Evidence PWGSC is providing a Canada-wide solution for the procurement of vaccines and drugs Evidence the Group Purchasing Program is fulfilling its responsibility for acquiring the requested drugs, vaccines, antivirals and services on behalf of the F/P/T jurisdictions in support of their programs and services: Procuring the required supply of vaccines, drugs and antivirals Contributing to Canada's capacity to produce vaccines in Canada Enabling a security of supply in Canada in case of emergency	Data and Document Review Performance reports Operational reports Client satisfaction surveys Other relevant studies or reports Service agreements Interviews PWGSC managers/staff including the Group Purchasing Program Clients who have used the Group Purchasing Program for the provision of drugs, vaccines and antivirals
5. Are clients satisfied with Group Purchasing Program services?	Extent to which F/P/T clients are satisfied with the services provide by the Group Purchasing Program Number of products delivered on time and within the requirements of the agreements/contracts Evidence that the Group Purchasing Program provides F/P/T clients with value-added advice and guidance Evidence of Group Purchasing Program responsiveness to F/P/T needs F/P/T satisfaction with Group Purchasing Program services Extent to which the Group Purchasing Program supports F/P/T's ability to deliver on their programs Extent of Group Purchasing Program responsiveness to F/P/T jurisdictions' requirements Ease of doing business with the Group Purchasing Program Extent to which F/P/T clients agree that the Group Purchasing Program is fulfilling its responsibilities	Data and Document Review Roles and Responsibilities Accountability Framework Performance reports Operational reports Project delivery processes Client satisfaction surveys Other relevant studies or reports Interviews/Focus Groups PWGSC managers/staff including the Program F/P/T clients
6. What are the barriers to successful project delivery of Group Purchasing Program?	Evidence of dependencies/other factors/programs already under way that could affect the outcomes of the Group Purchasing Program Extent to which the procurement/delivery of orders are late and/or not within the requirements Existence of impediments to delivering on time and/or within the requirements Existence of complaints from F/P/T clients about Group Purchasing Program Services Existence of effective processes and procedures to deliver the Program Existence of actions taken to reduce/eliminate identified impediments	Data and Document Review Accountability/Strategic Framework Performance reports Operational reports Continual Improvement Framework Other relevant studies or reports Interviews/Focus Groups PWGSC managers/staff including the Group Purchasing Program F/P/T staff who have dealt with the Group Purchasing Program
7. What are the risks to the vaccine and drug supply?	Identification of the risks to the vaccine and drug supply Extent of risks to the security of the vaccine and drug supply Extent of risk to the capacity to produce vaccines in Canada	Data and Document Review Roles and Responsibilities Strategic Framework Documents identifying, discussing and analyzing potential risks and parallel mitigation plan(s) Continual Improvement Framework Other relevant studies or reports Interviews/Focus Groups PWGSC managers and staff including the Group Purchasing Program F/P/T staff
8. Have there been any unintended positive or negative outcomes attributable to the Group Purchasing Program?	Existence of unintended positive outcomes Existence of unintended negative outcomes	Data and Document Review Performance reports Client satisfaction surveys Other relevant studies or reports Interview PWGSC managers/staff including the Group Purchasing Program F/P/T staff
9. Is the Group Purchasing Program cost-effective from a government-wide perspective?	Extent to which the Group Purchasing Program achieves best value for the Crown by reducing per unit purchasing cost Extent to which the Group Purchasing Program has facilitated lower costs for F/P/T Extent to which the Group Purchasing Program provided better access for small/remote groups Evidence of level of efficiency of the Group Purchasing Program in procuring	Data and Document Review Program financial statements Costing and Charging Review Project Project Business Management System database Operational reports Financial reports and plans Budgetary allocations Project reports/updates of costs Industry benchmarks used for project costing Other relevant studies or reports Interviews PWGSC managers/staff including the Group Purchasing Program PWGSC financial advisor for the Group Purchasing Program Representatives from F/P/T jurisdictions
10. Are there more cost-effective alternatives for procuring drugs and vaccines for Canadian requirements?	Review Group Purchasing Program total delivery costs Extent to which the Group Purchasing Program is ensuring best-value for Canadians Difference in unit costs in bulk procurement model and in individual F/P/T procurement Comparisons to other jurisdictions and industry benchmarks Assessment of cost of not delivering the Group Purchasing Program	Data and Document Review Project reports/updates of costs and options Studies conducted by the Program Experiences of other jurisdictions Experiences of other countries Industry benchmarks used for project costing Other relevant documents, studies and reports Interviews PWGSC managers including the Group Purchasing Program Representatives from F/P/T jurisdictions

Appendix B: Principal Documents Consulted

Authority and Governance

1. Canada Health Act (1984).
2. Department of Public Works and Government Services Act (1996).
3. Department of Justice. Order in Council P.C. 1986-1581 (1986).
4. Public Works and Government Services Canada. Influenza Virus Vaccine Contracts and Memorandum of Understanding (MOU). 2000-2001.
5. Public Health Agency of Canada. National Immunization Strategy: Final Report, 2003.
6. Treasury Board of Canada Secretariat, Common Services Policy (August 2006).

Documents about PWGSC F/P/T Group Purchasing Program

1. Public Health Agency of Canada. Vaccine Supply in Canada and the National Immunization Strategy, 2007 09 26.
2. Public Health Agency of Canada. Vaccine Supply Working Group Meeting. 2004 12 02-03.
3. Public Works and Government Services Canada. Annual Report: Federal, Provincial, Territorial Committee on Group Purchasing of Drugs and Vaccines, 2007-2008.
4. Public Works and Government Services Canada. Departmental Performance Report, 2006-2007.
5. Public Works and Government Services Canada. Influenza Vaccine Final Report. 2008.
6. Public Works and Government Services Canada. Influenza Vaccine Program, 2008.
7. Public Works and Government Services Canada. Drugs, Vaccines, and Biologics: Meeting with Commodity Manager, 2007 02 21.
8. Public Works and Government Services Canada. Drugs, Vaccines, and Biologics: Presentation to the Supply Managers, 2008 03.
9. Public Works and Government Services Canada. Federal/Provincial/Territorial Committee on Group Purchasing of Drugs and Vaccines Annual Reports from 1992-1993 to 2007-2008.
10. Public Works and Government Services Canada. Procurement of Drugs, Vaccines, and Medical Supplies by the Government of Canada for Pandemic Purposes,  2006 06 06.
11. Public Works and Government Services Canada. PWGSC Annual Fee Proposal, 2008.
12. Public Works and Government Services Canada. Success Stories/Lessons Learned, 2007-2008.

Documents on Other Jurisdictions

1. Government Accountability Office. Influenza Vaccine: Issues Related to Production, Distribution, and Public Health Messages. GAO-08-27. 2008 10.
2. House of Commons Committee of Public Accounts. Procurement of vaccines by the Department of Health: Fifteenth Report of Session 2003-2004. 2004 03.
3. National Center for Immunization and Respiratory Diseases. Annual Report 2008.
4. National Vaccine Program Office. Strengthening the Nation's Influenza Vaccination System: An NVAC Assessment.  2004 12.
5. 24. Parliamentary Office of Science and Technology Postnote. UK Vaccine Capacity. 2008 08.

¹ Relevance is the extent to which a policy or program addresses a demonstrable need, is appropriate to the federal government and is responsive to the needs of Canadians.
² Performance is the extent to which the program achieves economy, efficiency and effectiveness.