Mastering Cost Prediction
Accurate cost prediction is paramount for healthcare businesses looking to optimize budgets, forecast expenditures, and enhance decision-making. We rigorously evaluate GenHealth.ai’s predictive model to ensure it exceeds the prior state-of-the-art in terms of accuracy and reliability. Here's a detailed look at how we assess our model's performance on cost predictions using various plots and metrics.
Accuracy in Cost Predictions
Our model's Normalized Mean Average Error (MAE) is about 84.5%. This indicates that the predictions are relatively close to the actual costs, considering the scale and complexity of the datasets. Normalization allows for the comparison of prediction accuracy across different datasets or models by adjusting for variations in scale and magnitude, scaling the MAE relative to the magnitude of the actual values. For instance, if the actual healthcare cost for a specific case is 8,450 of the actual cost. In comparison, models from the Society of Actuaries (SOA) reported MAE Norm values ranging from 91.2% to 96.7%, which translates to prediction errors between 9,670 for the same $10,000 cost. This demonstrates that our model achieves significantly lower error rates, underscoring its effectiveness and reliability in cost prediction.
Model Strengths and Opportunities through Cost Percentiles
The boxplot of real and predicted costs across different cost percentiles provides key insights into our model's performance. Cost percentiles represent the distribution of patient costs for the first year of historical data, segmented into groups. The model shows strong performance for lower and mid-range cost predictions, where the predicted and actual costs align closely. In the highest percentiles, particularly the 99-100th percentile, there is some divergence. Actual costs in this range exhibit greater variability and include a few outliers. This suggests that while the model is robust and accurate across most cost ranges, there is an opportunity to improve its predictive accuracy by addressing the outliers in the highest cost range.
Evaluating Top 1 Percent Highest Cost
Even when predicting higher costs, a notably challenging task for any model, our model demonstrates significant power in identifying the top 1 percent of future cost patients, achieving an Area Under the Curve (AUC) of 0.85. The AUC measures a model's ability to distinguish the top 1 percent of high-cost patients, with values ranging from 0 to 1. An AUC of 1.0 indicates a perfect model, while 0.5 signifies no discriminatory power, equivalent to random guessing. An AUC of 0.85 means our model has an 85 percent chance of correctly distinguishing between a randomly chosen top 1 percent high-cost patient and another patient.
When compared to the models from the SOA paper, the top-performing model in this category achieves an AUC of 0.879, while other models have AUCs of 0.871 and 0.842. Although our model does not surpass the highest AUC in this category, it performs comparably, highlighting its strong predictive capabilities in identifying high-cost patients.
Uncovering Hidden Biases
Gender and age can reveal subtle biases in predictive models. Our box plots of prediction errors for real and predicted costs across various percentiles by gender and age categories aren't just charts—they’re powerful diagnostic tools. By carefully analyzing these dimensions, we ensure our model excels not only in overall performance but also in fairness and equity. Identifying outliers and understanding variations across demographics help us refine our approach, making our predictions both inclusive and impressively accurate. This commitment to detailed analysis ensures our model is robust and fair, delivering reliable results for all groups.
Time-Tracing Predictions
The time series plot of error costs highlights the predictive performance of our model across different years, segmented by gender. This visualization provides a clear view of how prediction errors for male and female groups have evolved from 2017 to 2022. Notably, the majority of predictions for both genders remain close to zero, indicating accurate forecasting over time. However, we can observe some outliers, particularly in the years 2018 and 2020, where prediction errors spike. Overall, this time-tracing analysis reinforces the consistency of our model's predictions while also highlighting areas for potential improvement, ensuring our predictions remain robust and equitable across different demographics and over time.
The time series plot of actual costs provides a comprehensive view of the cost predictions where the majority of actual costs for both genders are clustered at lower values, with occasional spikes observed. Spikes and trends tell a story in the time series of predicted costs offers, reflecting the impact of global events like the COVID-19 pandemic on cost predictions.
These evaluations underscore the robustness, detail, and reliability of our approach, reinforcing our model's capability to provide precise and dependable cost predictions which are central to many organizations in healthcare. By verifying the model's statistical accuracy and assessing its practical application, we enable health plans, providers, and government to enhance patient care efficiently. With more reliable predictions than any other existing solution, healthcare organizations can support effective resource allocation and risk management better than ever.