By Drs. Seif Elmankabadi, Kelvin Moore, Odi Ehie, Philip Bickler, Isabella Auchus, and Michael Lipnick
It has long been known that pulse oximeters may perform less accurately in patients with dark skin color. However, this was not widely known to many clinicians and it was only recently that these performance problems have been linked to potential health and healthcare disparities.
In recent years, the pulse oximeter has become a household name due to COVID-19 and has resulted in a rise in over-the-counter, inexpensive devices (which may have the greatest inaccuracies). Retrospective studies during COVID-19 highlighted for the first time that patients whose self-reported race was Black had a higher likelihood of inaccuracy with pulse oximeters (i.e. SpO2 overestimation of SaO2) and subsequent healthcare disparities.
Pulse oximetry performance has long been the focus of the UCSF Hypoxia Lab, started by John Severinghaus, MD. The lab was among the first to report controlled data demonstrating performance problems with pulse oximeters due to skin color nearly 20 years ago. During COVID-19, the Hypoxia Lab teamed up with the UCSF Center for Health Equity in Surgery and Anesthesia to launch The Open Oximetry Project, which aims to improve accuracy, access, and testing standards for pulse oximeters to improve access to safe pulse oximeters to people worldwide. This project has six pillars of work: laboratory and clinical studies, developing new in-vitro techniques, improving regulatory standards, open data, and multinational and multidisciplinary stakeholder collaboration.
Laboratory Studies
Laboratory studies are a core pillar of the Open Oximetry Project at the UCSF Hypoxia Lab; some of which the lab has pioneered for over 50 years and others which it is developing for this project. All oximeters developed for clinical use must go through standardized laboratory studies, both to accomplish calibration of the instrument and to establish the degree of compliance set for performance standards. However, these laboratory studies, which are invariably done on young healthy subjects, have significant limitations when translating results to real-world patient populations.
Two of the major limitations of existing regulatory standards include inadequate diversity of skin color and inadequate performance requirements during low perfusion conditions. Preliminary results from Open Oximetry have suggested that good performance in the lab, but poor performance in real world clinical settings may be due to a combination of dark skin pigment and low perfusion. Recent data from the lab found the rate of missed diagnoses of hypoxemia to be greater than 20% in subjects with dark skin pigment and low perfusion, even with some high-end pulse oximeters. The lab is working on new protocols to create a stress test that can better predict how devices will function in the real world.
The project is collecting data on many of the most popular devices to validate manufacturer claims and communicate these findings clearly to consumers and clinicians via OpenOximetry.org. Findings are also being shared closely with global regulatory bodies to help improve future iterations of regulatory standards.
Clinical Studies
Another foundational pillar of the project is the observational clinical study known as EquiOx. This study is conducted at San Francisco General Hospital (SFGH) and led by site PI Dr. Carolyn Hendrickson. The discrepancy between lab and clinical performance has further emphasized the need to address the pulse oximeter bias in the clinical setting. Thus, there is need for well-designed prospective clinical studies, especially in critically ill adults and pediatric patients.
Most studies have attempted to solve this issue via retrospective studies, which are inherently limited and prone to inaccuracies of the electronic medical record system. In addition, skin pigmentation is often characterized on the basis of race, which cannot be considered the same as skin color. EquiOx addresses this limitation through its diverse patient population, skin pigment measurements, and instantaneous SpO2 vs SaO2 readings
EquiOx enrolls a wide range of differently pigmented patients hospitalized in the intensive care unit. The skin pigmentation is measured using the Konica Minolta, Delfin Spectrophotometer, Von Luchen scale, and Fitzpatrick scale. The study pairs simultaneous arterial blood draws with pulse oximeters readings, allowing for real time SpO2 versus SaO2 comparison. This data is then correlated with how pigmented the patient’s skin is. In addition, the research team notes additional factors that may contribute to oximeter inaccuracies such as finger size, probe location, patient medications (vasopressors and paralytics), movement, ventilation settings, and comorbidities. The EquiOx study will help evaluate the effect of skin pigmentation on pulse oximeter performance in critically ill adults. Combining the controlled laboratory studies with the clinical data is a step towards eliminating bias in pulse oximeters.
New Techniques
The OpenOximetry Team is developing new in vitro techniques to test pulse oximeters. Human studies are expensive, time consuming, and often limited to healthy subjects. Furthermore, as discussed above, laboratory studies do not always alone predict real world performance. To address these challenges and evaluate device performance, the new in vitro techniques developed will help to quickly identify poor performing devices while also complementing laboratory and clinical testing.
New Standards
The UCSF Hypoxia Lab closely works with global regulatory bodies to identify ways to improve standards for pulse oximeters that will increase safety and performance, while emphasizing health equity and access. Upon first glance, the most technologically advanced and rigorously tested device seems to be the goal; however, this may have unintended consequences such as increasing cost and thereby decreasing access. Therefore, this effect would likely disproportionately impact resource-variable and low- and middle-income communities worldwide. We are proposing modifications of standards for measuring pulse oximeter accuracy and precision, while ensuring performance outliers are analyzed and reported to leading regulatory agencies. In addition, there is a need for improved diversity of study subject pools, an objective definition of skin pigmentation, and an initiative to document data transparently. These criteria aim to raise the bar for performance standards in oximeter certification and address issues relating to skin pigmentation and perfusion index.
Open Data
The OpenOximetry project aims to harmonize data collection protocols and create an open repository to share data transparently, effectively, and efficiently across teams. Many groups collect oximeter data, but these efforts are often siloed and with varying methods. Sharing and aggregating our historical dataset and new prospective data will accelerate progress and avoid redundant research efforts. The harmonized dataset will provide investigators with the opportunity to standardize data and analyses leading to clinically relevant information.
Collaboration
The Hypoxia Lab hopes to build a collaborative community of researchers worldwide to promote cross-disciplinary collaboration and include perspectives across diverse practice settings. In addition, we recognize, not all laboratories worldwide have equal experience and/or capacity. The Open Oximetry team envisions a collaborative community meeting quarterly to co-develop best practices, share data, and ultimately accelerate improved performance and equity in pulse oximetry.
What Now?
The OpenOx project is working diligently with catalytic support from Moore, McGovern, USAID/STAR, RWJF, and the FDA. Despite the pulse oximeter’s limitations, it remains one of the most effective and ubiquitous biomedical devices available to ensure patient safety worldwide. Our laboratory studies have tested nearly 20 pulse oximeters with insights into ways to ensure greater performance and improve standards. The EquiOx clinical study has enrolled nearly 200 critically ill patients at SFGH, and is planning to expand to UCSF’s Parnassasus campus. In addition, The Hypoxia Lab is collaborating with other research teams to create a harmonized dataset to direct national and global initiatives designed to reduce bias in pulse oximetry. Ultimately, a focus on diversity, equity, and inclusion is imperative to the current and future success of patient care and biomedical technology. In March, UCSF will host a joint meeting with the ISO Working Group on Pulse Oximeters, to redefine standards for pulse oximeter performance.
