Proficiency Testing: Diagnosing Threats to Quality
It happens all too often: a tired technician transposes two numbers, and a positive result becomes a negative. Or a piece of equipment goes too long without maintenance, and drifts out of calibration. As every scientist and laboratory technologist knows, laboratory medicine is only as valuable as it is reliable. But in the unpredictable world of real-time diagnostics, how do we ensure that our instruments and protocols remain on target?
Proficiency testing (PT) is a way to identify and prevent quality problems in the medical laboratory. PT programmes send characterised clinical specimens to participating laboratories on a regularly scheduled basis. These laboratories then process and analyse the specimens as if they were ordinary patient samples, and report their results to the organiser. By working with samples that have known values, laboratories can learn how their results compare to both the validated measurements from each sample, and to the average findings of many peer laboratories. The reports from PT programmes help participating laboratories to pinpoint areas where their testing is unreliable, and often include concrete suggestions for improvement.
PT is one element of the broader system of external quality assessment, EQA. Many authorities around the world use results from PT programmes to assess laboratory performance as part of registration and licensing, in conjunction with annual on-site reviews. EQA is also a requirement for every international accreditation process, an important consideration for laboratories seeking this high-level credential.
Proficiency testing: the basics
To learn how a PT programme is designed and administered, Lab Culture spoke to Dr. Jane Carter and Mr. Stephen Munene, who work at the African Medical and Research Foundation (AMREF), which has recently been rebranded to Amref Health Africa. Dr. Carter and Mr. Munene are based in Nairobi, Kenya, and coordinate the East African Regional External Quality Assessment Scheme (EA-REQAS) on behalf of the ministries of health. For the past six years, this PT programme has administered EQA testing targeting peripheral level laboratories in Burundi, Kenya, Tanzania, and Uganda.
Individual laboratories carry out many different types of testing, from yes-or-no diagnostics to more variable prognostics like viral load testing and qualitative examinations such as blood film morphology. Although some PT programmes focus on specific diagnostic tests, especially in high-priority disease areas like HIV and tuberculosis, many programmes cover a broader range of testing to evaluate laboratories on a more comprehensive level. [1]
To test laboratories’ proficiency in an assortment of procedures, PT programmes must maintain and characterise numerous types of samples. For example, the South African National Health Laboratory Service (NHLS) offers parasitology slides for a large panel of tests,[2] while EA-REQAS has included, among other samples, pus smears for Gram staining and peripheral blood films for blood cell morphology.[3] These samples may be derived from individual patients (e.g. parasitology slides), pooled samples, or samples synthesised specifically for use in PT. The more similar samples are to “clinically relevant challenges”, the better, so care is taken that test samples closely resemble real patient samples and should be processed in an identical manner to routine clinical samples.
At present, the pre-analytical phase of sample handling is not assessed via EQA, although many programmes include questionnaires to help recognise pre-analytical sources of error.[4] Some programmes, such as EA-REQAS, include clinical and public health questions as part of their PT surveys. These questions are designed to underscore the importance of laboratory confirmation of diseases of public health, ensure that clinicians understand the role of laboratory testing in overall patient diagnosis, and promote collaboration between these different health cadres.
In addition to participating laboratories, PT samples may also be sent to tertiary level validating laboratories to set target values. In the case of EA-REQAS, a collection of national-level laboratories in Kenya validate positive and negative results and, as a group, set the target values for each survey. Based on these results, the scheme coordinators create a marking key which resides in a central database for automated scoring of proficiency test results and generation of Immediate Feedback reports.
“A good PT programme is supportive rather than punitive and provides laboratories with opportunities for learning, and identifying and addressing gaps,” say Dr. Carter and Mr. Munene. The EA-REQAS PT programme provides a variety of suggestions on improving performance after each round of proficiency testing. The Immediate Feedback reports received by participating laboratories include their acceptable and unacceptable results, overall performance scores, and advice on how to improve their performance.
The programme’s coordinators also post learning materials on the EA-REQAS website, and have a help desk where participants can turn for advice at any time. “In a sense, a good PT programme provides a distance learning opportunity for even the most remote laboratories,” add Dr. Carter and Mr. Munene.
A guide for national action
As laboratories address the problems that PT has illuminated, the quality of day-to-day testing improves, with positive outcomes for patient care. Depending on the country, there may also be a requirement to follow-up on corrective actions that have been taken as a result of poor PT results. For example, South African laboratories with unacceptable results must submit corrective action reports to their regional managers. [1]
National ministries of health may also review composite reports on their laboratories’ PT performance. These include consolidated data on the laboratories that participate, which may or may not identify these laboratories by name, along with potential sources of error identified by the PT programme. EA-REQAS prepares composite reports after each survey with analysis of results from all countries, which include alerts or policy change suggestions for ministries focusing on laboratory procedures that appear to be suddenly or consistently underperforming.
In the case of EA-REQAS, performance results of named laboratories are sent to the Ministry of Health in which the laboratories are based, with coded information sent to other countries’ ministries; this was agreed by the regional committee that oversees the scheme. On the other hand, the National Quality Assessment Programme in Burkina Faso has maintained a nondisclosure policy, even when the Ministry of Health requested identifying information on the laboratories performing below standard, reasoning that the programme’s goal is quality improvement, not punishment. [2]
In addition to monitoring existing laboratory and diagnostic networks, PT may also be used as part of a post-marketing surveillance exercise during rollout of new diagnostic tests. A very successful case was described by Dr. Peter Fonjungo and colleagues, who used EQA as a way to confirm that six new laboratories across Ethiopia, which were initiating dried blood spot polymerase chain reaction for diagnosis of HIV in infants, were as effective as the national reference hospital where this test was first made available. [3]
PT in isolation, without further efforts or interventions, cannot sufficiently improve a laboratory’s performance. A survey of South African laboratories participating in the NHLS stool and blood parasitology PT programme from 2004 to 2010 found only a slight improvement in stool test accuracy, and no discernible change in blood test accuracy. [4] This reflects the fact that PT is designed to identify problems, not to fix them. On the other hand, analysis of EA-REQAS results has shown a direct correlation between numbers of times laboratories have participated and improved performance.
However, when used to diagnose a laboratory’s weaknesses as if they were a set of symptoms, PT can be a powerful tool. A group of laboratories at clinical trial centres in Burkina Faso and Ghana demonstrated the potential of PT when they undertook a study to find the reasons for poor PT performance and improve their results. [5] With the assistance of an external quality control advisor, and using records kept by laboratory technologists to track equipment maintenance, receipt of new reagents, and archival printouts of instrument readings, the advisory team was able to diagnose most of the problems that each laboratory had encountered. The bulk of errors were methodological, including problems with reagents, infrequent calibration or outdated standard operating procedures (SOPs). The advisory team also observed basic clerical errors, such as mistakes in transcribing data from an instrument onto the PT result form; technical errors such as incorrect in dilution or poor water quality; and, occasionally, problems with sample stability. Having determined these sources of error, the laboratories reduced their error rate by a third, and continue to participate in PT and quality management. An important lesson from this and other PT investigations is that errors in the course of testing are usually not caused by individual misconduct or carelessness, but instead emerge from several contributing features or events. [6]
In addition to finding quality issues in individual laboratories, PT can be a key tool for identifying more widespread technical problems. Dr. Carter and Mr. Munene described a high number of clinically dangerous results in one distribution of an HIV diagnostic PT sample. Although the sample was reactive, as generated by the PT providers at EA-REQAS and confirmed by validating laboratories, many primary level laboratories reported a negative result.
“On further analysis, we found that the erroneous results appeared to be linked to one type of testing kit,” say the EA-REQAS administrators. “This was a vital piece of information that may have impacted many patients, and which we reported to the relevant authorities. This is a good example of how PT schemes can play a major role in post-market surveillance of testing techniques and test kits and can impact policy change.”
Challenges to proficiency testing and ways forward
PT programmes may face difficulties delivering specimens to remote laboratories. Specimens often require cold chain transport, especially when made to resemble patient samples. Damage to samples en route, or improper storage between their receipt and testing, can affect test results and may alter a laboratory’s acceptable outcome rate.
To address this problem, EA-REQAS uses only materials that are stable at ambient temperature. To prevent delays in delivery, a courier system has been introduced with an additional return service so that laboratories may send paper-based results at no cost back to the EA-REQAS Coordinating Centres in Kenya, Tanzania and Uganda. Laboratories with internet access may also report results online. For laboratories that must depend on the postal service to deliver PT products, the advisory team in Burkina Faso and Ghana suggested ordering shelf-stable tests, carefully monitoring the shipping calendar, and where possible, providing postal carriers with refrigerators and detailed instructions on the importance of maintaining samples at the correct temperature. [1]
PT may present a financial burden to the testing laboratory, which may decrease compliance. Time and reagents spent testing PT samples cannot be used to test real patient samples. Some laboratories may be unable to process PT samples due to unserviceable equipment, shortage of reagents, and lack of staff. However, says Dr. Carter, “we have learnt that even in very small laboratories that may feel compromised in terms of good equipment and reagents, and staff shortages, participation in a PT programme can start the journey to better quality testing, if participants learn from the reports and from re-examining the PT samples from surveys.”
PT requires the participation of not just laboratory staff, but also other members of the clinical team. Laboratory staff members enrolled in the EA-REQAS programme have reported challenges in persuading clinicians and public health workers to complete their sections of surveys, leading to delayed return of results. Dr. Carter and Mr. Munene have observed that, “once they recognise the importance and value of EQA, [administrators, clinicians and public health workers] will readily participate in answering survey questions – this requires on-going sensitisation and information.”
How to implement proficiency testing
A PT plan for a participating laboratory should be relevant to the tests performed in that laboratory; offer all staff the opportunity to participate in the PT exercise and see the results; and include a plan to use the results and materials to improve performance. The PT plan should address all components of quality management systems, including staffing, infrastructure and utilities, equipment, supplies, record keeping and quality assurance.
Dr. Carter and Mr. Munene have the following advice for laboratories planning to enrol in a PT programme: “Identify an appropriate scheme for your own laboratory; fully participate in it; learn from it; ask questions; and use the results to systematically improve the quality of your work and provide continuing education for your laboratory and other health facility staff.”
It is important for the integrity of PT that the samples be treated as normal patient samples. If the test does not represent average analysis of an average sample, the results will be of little practical use. [1] When PT results are not acceptable, it is vital to look back, document the problems, and report on any corrective actions taken. Both staff members responsible for and familiar with the tests at issue, and health facility management and supervisors, should review PT reports to address technical and administrative issues. Dr. Carter and Mr. Munene also advise laboratories to keep the samples they are sent for re-examination after making the recommended changes.
There are many PT schemes available, and laboratories should take care in choosing a relevant and well-validated plan. This may not be a straightforward choice, as the field is crowded with international, governmental, and private options, both integrated and disease-specific. A good first step is to consult with the national ministry of health, which will likely have a complete list of appropriate programmes, including any available at national level. Among the national programmes are the Burkina Faso EQA programme; the Ethiopian Public Health Institute (www.ehnri.gov.et/); and the Medical Laboratory Science Council of Nigeria (www.mlscn.gov.ng). The US Centers for Disease Control and Prevention also provides laboratory quality assurance programmes that include PT services (www.cdc.gov/labstandards/index.html).
For laboratories located in EA-REQAS countries, more information about the programme can be found at www.eareqas.org. The programme is in the process of obtaining ISO 17043 accreditation and is scaling up its operations. Other internationally available PT programmes are administered by the South African NHLS, which offers a range of tests (www.nhls.ac.za/?page=proficiency_testing&id=42). The World Health Organization administers quality assessment schemes for HIV, HBsAg and HCV serology involving reference laboratories in both Francophone and Anglophone Africa, through the South African National Institute of Communicable Diseases and the Senegalese National Reference Laboratory (www.who.int/diagnostics_laboratory/quality/serology/en/).
New opportunities for quality control
Today, most laboratories that are enrolled in PT programmes are located at the national or reference level. Peripheral laboratories, which serve the majority of patient communities, have been slower to participate, in part because integrated programmes, which serve many disease areas, are not always available at the village or community level.
As a result, PT has not yet made its full impact felt at the local level, even under Ministries of Health that have embraced EQA as an essential part of a quality management system. EA-REQAS has been one organisation that has struggled to expand its influence, even after initial successes. According to Dr. Carter and Mr. Munene, “The main challenges for EA-REQAS have been in ensuring Ministry of Health ownership and buy-in; convincing peripheral laboratories of the importance and value in participating; producing materials for the region on an expanding scale; establishing an electronic database and a marking system that can cope with the large numbers of participants; and securing adequate funding for on-going operations.”
Sustainability at the national level is a concern for PT programmes. Currently, the East African scheme is free of charge to government laboratories, but to promote sustainability, and to remind administrators of the cost of quality, EA-REQAS is in the process of proposing a modest charge to all laboratories, however small. Similarly, the Burkina Faso EQA scheme is seeking a line item in the national budget.
The testing authorities also incur certain costs. For individuals who want to be involved in the launch of a PT programme, Dr. Carter advises that “to be a PT provider takes a huge investment, not so much in money, but in time, effort, diplomacy, advocacy and determination.” Among the resources required for starting a PT programme are a process, including SOPs, for production of test materials; staff to produce these materials, manage results, and answer questions from enrolled laboratories; a database for entering and analysing results; and good survey development and report-writing skills.
A model for expansion may emphasise local improvements, while building on the infrastructure already developed by successful authorities like EA-REQAS. International cooperation can save redundant efforts, while allowing individual health authorities to exercise some control over local challenges.
“We think regional hubs for EQA are a good idea,” say Dr. Carter and Mr. Munene. “For every individual country to create comprehensive EQA programmes is a huge task, but regional cooperation can work. We have made many mistakes and learnt from them, so there is no need for others to go through the same learning curve.”
This kind of effort will benefit from the assistance of organisations like ASLM, which has experience reaching out to individual laboratories and fostering collaboration across national borders. Dr. Carter and Mr. Munene believe ASLM can lay the groundwork for regional cooperation, while EA-REQAS can assist with advising potential PT providers on how to set up schemes and run them. A multi-organisational structure like this will build on the talents of proven leaders in enhancing laboratory quality, bringing PT to a wider set of laboratories with the ability to address local concerns.
Select “Download” (in the top right margin) to view the article in PDF format.
By: Laurel Oldach (Editorial Team); Contributors: Jane Carter, MBBS, FRCPC (Amref Health Africa, Headquarters), Stephen Munene, BSc (Amref Health Africa, Kenya), Julius Tome, HND (Amref Health Africa, Kenya), David Isaboke, HND (Amref Health Africa, Headquarters) and Jackson Songok, BSc (Amref Health Africa, Kenya); Editor: Aaron Krol, MFA (Editorial Team)
Originally published in the May 2014 issue of Lab Culture newsletter.
[1] Parkin, de Mendoza, Schuurman, Jennings, Bremer, Jordan, Bertagnolio, WHO DBS Genotyping Workshop. Evaluation of in-house genotyping assay performance using dried blood spot specimens in the global World Health Organization Laboratory Network. Clinical Infectious Diseases. 2012; 54(suppl4), S273.
[2] Poonsamy, Dini and Frean. Performance of clinical laboratories in South African parasitology proficiency testing surveys between 2004 and 2010. J Clinical Microbiology. 2012; 50(10), 3356-8.
[3] Proceedings of the first meeting of EA-REQAS: 3-4 April 2003. Accessed at eareqas.org.
[4] James et al J Clinical Pathology 2014