Where Large Language Models Hallucinate or Lack Data, Large Medical Models can help in Healthcare

Large Language Models (LLMs) are significantly different from GenHealth’s Large Medical Model (LMMs). LLMs are not always the best tool in healthcare for many reasons and can be supplanted by the use of Large Medical Models. These are some of the major points where LLMs fall short.

1. LLMs are usually less specific than medical models because each inference response is more difficult to aggregate with one another. They will provide slightly different response which are difficult to aggregate.
2. LLMs have a tendency to hallucinate events that are impossible or will be wrong. For example they often miss ICD or other medical codes.
3. LLMs do not have all the necessary data for entities in healthcare or the understanding of the domain at the same level as our specialized large medical model.
4. LLMs are not trained on actual medical events, but what people say about medical events

Likewise, medical models do not do many things that LLMs do. Therefore this is not an either / or, instead this post is to demonstrate the areas where LLMs have short comings and where LMMs can better address those areas. Together, LLMs and LMMs have a power to transform the industry.

Example

To better illustrate these cases, here’s a transcript with ChatGPT for effectively the same prompt that we ran using GenHealth’s model here (https://genhealth.ai/#quality). We translated for a prompt that ChatGPT understands.

Prompt

Given a patient who is a 71 year old female patient with congestive heart failure and a treatment plan beginning with angiotensin receptor blocker (losartan), tell me what conditions, medications, and procedures this patient is likely to experience over the next 12 months. Give me a month by month account for months 1 - 12. Add probabilities to those events where possible

ChatGPT Response

I am not a healthcare professional, and I cannot provide medical advice or predict specific outcomes for individual patients. However, I can provide you with a general outline of the potential conditions, medications, and procedures that a patient with congestive heart failure might experience over a 12-month period, with some common considerations. Please keep in mind that actual experiences can vary widely based on the patient's specific medical history, the severity of their condition, and their response to treatment.

Month 1:

- Diagnosis: The patient is diagnosed with congestive heart failure (CHF).
- Medication: The patient begins treatment with losartan, an angiotensin receptor blocker (ARB).
- Lifestyle Changes: The patient is advised to make dietary changes, reduce sodium intake, and start a daily weight monitoring routine.

Month 2-3:

- Medication Adjustment: Depending on the patient's response to losartan, medication dosages may be adjusted.
- Education: The patient and their family receive education on managing CHF, including medication compliance, dietary restrictions, and symptom monitoring.

Month 4-6:

- Follow-up Appointments: Regular follow-up appointments with a cardiologist to monitor the patient's condition and adjust treatment as needed.
- Medication Review: Medication effectiveness and potential side effects are assessed.
- Dietary Modifications: Further dietary adjustments may be recommended.

Month 7-9:

- Progressive Symptoms: If the patient's CHF is not well controlled, they may experience worsening symptoms such as shortness of breath, fatigue, and fluid retention.
- Medication Optimization: Medication changes or additions (e.g., [beta](/blog/beta)-blockers or diuretics) may be considered to better manage symptoms.
- Cardiac Rehabilitation: Depending on the patient's physical condition, cardiac rehabilitation may be recommended.

Month 10-12:

- Hospitalization: In severe cases, the patient may experience exacerbations of CHF, leading to hospitalization.
- Procedures: If needed, procedures such as angioplasty or stent placement may be performed to improve blood flow to the heart.
- End-of-Life Planning: Depending on the patient's overall health and prognosis, discussions about advanced care planning and end-of-life care may occur.

Probabilities for specific events will depend on the patient's individual circumstances and response to treatment. It's essential for the patient to have ongoing communication with their healthcare team to monitor their progress and make necessary adjustments to their treatment plan.

Please note that this is a general overview and not a prediction for a specific patient. Patients with CHF should receive personalized care and guidance from their healthcare providers.

1. Lack of specificity

In the response above, we do not see specific medical codes or concepts. We do not see specific probabilities like we do in GenHealth’s response below. The months are grouped. With some better prompting work you can likely get a more desired output, but it’s still far from the detail in GenHealth’s response which includes all the raw data depicted in these graphs below. You can interact with it here (https://genhealth.ai/#quality).

Notice the hearing loss outcome which we’ll talk about more in point #4 below.

2. Tendency to Hallucinate

Hallucination in Large Language Models is defined as events where the LLMs make up an answer thinking that its correct. If we had asked for specific ICD codes in the above example, ChatGPT would be less specific and often gets medical terminology codes wrong. Here is a specific example of that where we ask for an ICD code “struck by orca”

Prompt

Provide me with the ICD code for someone who is struck by an orca.

ChatGPT Response

The International Classification of Diseases (ICD) is a system used for medical coding and classification of diseases and health conditions. While it does include codes for various types of injuries and accidents, it may not have a specific code for someone being struck by an orca because such incidents are relatively rare.

In the ICD-10 system, which was in use until my knowledge cutoff date in September 2021, injuries from marine animals were classified under different codes depending on the specific circumstances and injuries. You would typically find codes related to marine animal injuries under the "W" codes, which are used for external causes of morbidity and mortality. For example, a code like "W56.0" might be used for a person bitten by a sea lion.

However, since the ICD system is periodically updated, and my knowledge only goes up to September 2021, it's possible that there have been changes or updates to the codes. I recommend checking the latest version of the ICD system, such as ICD-11 or any subsequent revisions, for the most current and specific codes related to injuries caused by marine animals like orcas. You can consult with a medical coding professional or refer to the latest coding resources for precise and up-to-date information.

It’s close but not correct. Actual ICD code for struck by orca is W56.22 . The ChatGPT response suggests that there may not be as specific an ICD code. Clearly it’s uncertain in it’s response. GenHealth’s LMMs do not hallucinate in their response because we have the ability to reference real data in ICDs in both our training and in our responses. We specifically have individual tokens for the various parts of ICDs and therefore parts like the .22 are actually in the vocabulary and token set of our model.

3. Not all the necessary data

Large Language Models do not have all the data needed for certain use cases. For example they are missing really important aspects of healthcare like provider NPIs and Cost.

Prompt

provide a valid provider NPI Number

ChatGPT Response

I'm sorry, but as an artificial intelligence, I do not have the ability to generate or provide real National Provider Identifier (NPI) numbers. NPI numbers are unique 10-digit identification numbers issued to health care providers in the United States by the Centers for Medicare and Medicaid Services (CMS). They are used to identify providers in healthcare transactions. Providing or generating a real NPI number would be a breach of privacy and confidentiality, and it would be against OpenAI's use-case policy.

Clearly there is a lack of understanding that NPIs are public. Lastly, we asked it to identify the provider associated with a real NPI and it was not able to do so. You can see the rest of the transcript here where it is never able to provide an actual NPI. If you work in healthcare and have experienced provider referral management, provider search, provider network design or many other use cases, it’s quite important to have a proper understanding of NPIs and the ability to produce them.

4. Not trained on actual medical events

GenHealth’s AI model is trained on actual medical events from real patient histories. 40M of them so far. ChatGPT and other LLMs are trained on public information across the internet, and more specific models like MedPalm are fine tuned on additional research papers and domain specific text. That fine tuning definitely improves the model but does not provide novel insight that has not yet been published.

In our original example above, we see a side-effect of the medication Furosemide in older females as hearing loss. However there is no clear published research on that outcome. But our model trained on millions of patients does show that and may be something for a care team or health plan to look into more before prescribing the medication.

This goes beyond what doctors can do today provided published research to practice evidence based medicine. We believe our model output and evaluation metrics demonstrate an improvement upon today’s “evidence”.

Ultimately there is a place for both LLMs and LMMs in healthcare. LLMs excel at back and forth conversations in the medical domain. LMMs are best for use with structured input and output or population level use cases where human prompts are not ideal for each interaction. There are many new use cases that open up with the addition of GenHealth.ai’s Large Medical Models.