Transparency in Healthcare AI Solutions: Balancing Performance & Explainability

The integration of Artificial Intelligence (AI) in healthcare is not new - however, as computational advancements continue to be made the number of potential solutions increases, not the least of which is general pre-trained transformers (GPTs). The need for transparency, trust, and ethical considerations is also not new and can not be overstated. Recent conversations around transparency sparked this article, patient data being utilized in healthcare models will continue to increase and both patient and provider will want to understand exactly what they are seeing. Additionally, institutions are trying to balance the convenience and speed of commercial solutions with the trust of creating LLMs and care and research models from existing open source libraries. The convenience of off-the-shelf AI solutions often introduces a "black box" scenario is generally the topic with significant concern around obscuring the understanding of how decisions are made. The implications of the path organizations choose and the opacity of their work anf solutions will be profound.

DALL-E generated image 02/26/24

The Drawbacks of "Black Box" AI Solutions

Commercial AI solutions, while offering rapid deployment and seemingly advanced capabilities, tend to operate as "black boxes." This label refers to systems and applications whose workings are not visible or understandable to their users. In healthcare, where decisions must be explainable and justifiable - not just in the clinical setting but also in the research and translational science space. The inability to scrutinize AI decision-making processes can have, even in the in the operational and supply chain verticals should be questioned. Hard conversations must begin at every level of healthcare systems:

• What are the organizational principles around AI utilization?
• What is the organizational strategy for ensuring fair AI adoption?
• How will the organization consider and document both primary and secondary model effects, both intended and potentially unitended?
• In what cases will limited transparency be acceptable?

Organizations that fail to have this conversation risk eroding patient, provider, and employee trust along with potentially finding their organization left behind in this transformative healthcare landscape.

It's important to recognize that reliance on purchased AI solutions will stifle skill development within healthcare organizations. It deprives healthcare systems of the opportunity to build the necessary skills for the next generation of provider and research relevance. I'm also acutely aware of the current profit and operational challenges that providers face in 2024 - especially community and regional health systems. While resources, talent, and strategy time are tight - even these systems must lean into maturing their organization in a manner that allows them to successfully build an AI strategy and thoughtfully create a balance of commercially available software and home-grown developed models that priortize transparency and explainability.

The Case for Building and Customizing Open Source Solutions

Building AI solutions with open-source tools offers an alternative that aligns more closely with the ethical and practical requirements of healthcare. Open-source platforms foster a culture of transparency, enabling healthcare organizations to develop AI models that are both interpretable and tailored to their unique contexts. This approach not only demystifies AI decision-making but also builds trust among clinicians and patients alike.

Moreover, engaging with open-source AI projects encourages continuous learning and skill acquisition among healthcare professionals. It cultivates a deeper understanding of AI technologies, empowering staff to contribute to AI solutions that enhance patient care.

Prioritizing Transparency over Results

In the quest to harness AI's potential, healthcare organizations must prioritize transparency over immediate results. Transparent AI models facilitate ethical decision-making, ensuring that AI-driven interventions are based on clear, understandable, and justifiable criteria. This transparency is crucial for maintaining patient trust and for the legal and ethical use of AI in healthcare.

Adopting AI solutions that emphasize transparency also positions healthcare organizations as leaders in ethical AI use. It demonstrates a commitment to patient safety, informed consent, and the responsible application of technology in sensitive areas.

Striking a Balance: Hybrid Approaches

For many healthcare organizations, a hybrid approach that combines the strengths of commercial and open-source AI solutions may offer the best path forward. This strategy involves leveraging the robustness and scalability of commercial products for certain operational needs while maintaining core systems built on open-source platforms for critical, decision-making processes.

Implementing a hybrid model requires a commitment to transparency from all stakeholders, including vendors. Healthcare organizations should advocate for and participate in the development of AI solutions, ensuring that commercial products adhere to the same standards of transparency and explainability as their open-source counterparts.

Conclusion

The deployment of AI in healthcare is not just a technological challenge; it is a moral and ethical imperative. As healthcare organizations navigate the complexities of integrating AI into their operations, they must remain vigilant about the trade-offs between convenience and transparency. By fostering a culture that values openness, trust, and continuous learning, the healthcare industry can leverage AI to deliver not just more efficient, but also more compassionate and equitable patient care.