In a recent WIRED article, a fascinating connection was made between the defense industry's eagerness to support Ukraine and an unexpected driver: combat data. Defense companies worldwide are actively engaging in Ukrainian support, not just to uphold democratic values in a shrinking world but to gain access to valuable combat data. The race for supremacy in the world of intelligent and autonomous weapons is increasingly reliant on data science – specifically, who has access to the most relevant training data (Meaker)
"The war in Ukraine presents an unprecedented opportunity for military tech companies. The scale of the fighting and the sheer number of weapons systems and high-tech sensors deployed have created a vast amount of data about how battles are fought and how people and machines behave under fire." - Morgan Meaker
This emphasizes a well-known truth, that vast datasets have applications (yes, beyond general pre-trained transformers); they hold the key to training models for specific tasks, and healthcare leaders are looking for ways to scale their efforts quickly.
Data's Role in Healthcare
Healthcare, much like the defense industry, recognizes the value of data. While healthcare data may not be as noisy as combat data, it presents its own challenges, such as interoperability and complex environments. Hospitals, researchers, and clinicians are in pursuit of a different goal compared to defense companies. They aim to become learning health systems, leveraging data to enhance care outcomes, intervention strategies, treatment plans, and precision medicine. Healthcare's future relies heavily on data, and the more data available, the greater the potential for health systems to provide better care and adapt to changing healthcare landscapes – perhaps, most notably, the shift from fee for service to value based care.
Healthcare's Complex Landscape
Healthcare is a substantial sector, but it lacks the luxury of narrowly focused use cases that allow allocating significant funding to fewer areas. Health systems often comprise numerous clinical and research units, each with their unique needs and goals. This complexity, combined with the highly regulated nature of healthcare data, creates challenges. Additionally, health systems grapple with technical debt accumulated over the years, resulting from acquisitions, slow cloud adoption, lack of platform level investments, and data silos. Despite these challenges, healthcare organizations, often regional institutions, are committed to ensuring the health of their communities.
The Emergence of Data Platforms
Healthcare organizations are now addressing their data challenges with cloud platforms like Databricks, Snowflake, and Microsoft Synapse. These platforms are designed to help unify data from various sources, including finance, insurance, electronic medical records (EMRs), and operations. However, many health systems get stuck in the pre-processing phase, hindered by the complexity of data engineering on regulated data. This phase often consumes a significant portion of data scientists' time. Many health systems are stuck in a slower adoption period than they would prefer – having identified critical data sources, selecting a cloud-based data platform, and generally employe talented data scientists. However, they have still struggled to create an operationally sustainable, secure, and cost-effective platform solution in Databricks, Snowflake, or Synapse. It’s a non-trivial task to complete complex data engineering on highly regulated data to unify finance, insurance, EMR, and operational data, so, slow progress continues as teams are mired in the pre-processing phase and may never realize how easy these platforms can make the post-processing work of business intelligence reporting and end to end machine learning and deep learning. It's no stretch to estimate that data scientists within a typical health system spend 70+% of their time doing manual data preprocessing work, that if platform investments were made would be minimized to less than 10% - that’s a serious uptick in business operations and care insights. And it’s possible right now to enjoy those benefits – but it does require cloud data platform and data engineering leadership with deep experience.
Data-Driven Transformation
Creating operational, secure, and cost-effective data platforms is a critical challenge for healthcare organizations. When implemented effectively, these platforms can significantly reduce the manual data preprocessing work that data scientists are burdened with. This translates to more time spent on valuable tasks like data analysis and modeling. The potential impact on business operations and care insights is substantial.
A High-Level Executive Plan
To harness valuable data assets within hospital systems effectively, consider the following steps:
1. Construct a New Approach: Start thinking of data as a product with a roadmap, user testing, delivery, and continuous improvement. Similar to clinical products or services, data has defined values to end users.
2. Focus on Data Delivery Methodology: Centralized layered data structures have proven to be winners of the enterprise data architecture race. That's not to say that organizations have not seen success with large data mesh approaches, but not at the same level as centralized platforms leveraging raw, cleansed, and curated data layers.
3. Evaluate User Self-Service Options: Examine current business intelligence tools in use and point them to curated (gold) data layers to facilitate trusted reporting. Tool rigidity can be limiting, instead increase flexibility at the department level and maintain rigidity on minimizing data movement and modeling within the application layer.
4. Bring Data Governance to Life: Draft critical data governance outcomes and enhance your organization's security posture by leveraging data governance tools available from your cloud provider. Cloud AE's (in our experience) are happy to work through proposed solutions that include both cloud level governance (Microsoft's Purview for example) and platform level governance (Databrick's Unity Catalogue for example).
5. The Data Science Opportunity: Recognize the role of data scientists in healthcare transformation. By optimizing data preprocessing, these professionals can dedicate more time to advanced analytics, predictive modeling, and AI-driven insights. This low hanging ROI opportunity can be the kickstarter to getting the funding you need to start your platform proof of concept efforts.
6. Collaborative Health Ecosystems: Foster partnerships and collaboration with other healthcare institutions and industry stakeholders to share data-driven insights, promote interoperability, and advance research in healthcare. What is the organization's sharing posture? Become familiar with data sharing options at the platform level - they are surprisingly easy and securely monitored.
The Data-Driven Future of Healthcare
The journey to becoming a data-driven healthcare system may be challenging, but the rewards are transformative. The ability to provide better care, adapt to value-based care models, and stay relevant in a dynamic healthcare landscape hinges on effectively leveraging data.
Citations: