For rural docs without access to specialists or older docs who need to stay up to date this is great. It is 100% terrible for med students, residents, fellows, and young attendings ALL of whom need the practice and experience of unassisted diagnosis and treatment on the cases they see most commonly. It’s no surprise that it’s “better” for complex cases, but it’s most likely on par with a PCP or FP or Pedi doc or specialist seeing common issues they see everyday in their own practices. No doc will refuse a new toy, so I’m worried that this will lead to a deterioration of clinical expertise in the US. Any other docs have an opinion?
If I needed someone to build me a complicated application, I would choose the software developer with 20 years of experience before LLM's were capable of helping software developers that uses LLM's to assist over the software developer with 1 year of experience that never learned to code and just uses LLM's for everything but is clueless as to how any of it works.
But you’ve arrived at the ER and the attending staff are oversubscribed, so you have interns and residents primarily trying to keep you alive. Sure, if you are a VIP you’ll get extra special care, but most of us aren’t.
Most ER work is urgent but familiar even to junior staff, and a lot of it is not life-threatening either. If they need to 'keep you alive' then they are following established protocols, not asking an LLM for advice.
1. This a claim from Microsoft not a peer reviewed study on pubmed. So it's a marketing title nothing more
2. Even some ideal frontier model is only as accurate as the inputs given for diagnosis, and will typically bias towards said inputs.
3. Multi-modal can handle images but are terrible at image classification unless using dedicated models for each pathology.
4. The very large majority of traning corpus with these companies is in English. There is zero guarantee that said mythical performance is preserved across languages.
5. The very thought of companies with the moral compass of most AI companies accessing and training on your medical records should make anyone pause.
other industries by and large have been able to figure out how to separate novice practitioners away from actual public utility until they're ready to participate.
I don't claim to know whether that could be done feasibly for medicine, I only point out that medicine is uncommon in how it educates.
Medicine is uncommon because there are no practice pieces for a lot of things. This isn't about learning stitches, this is about learning how to interact with a human, figure out what is going on, what tests and checks and options to even order.
Which, coincidentally, won't be solved by AI anytime soon. You'd need a full anthropomorphic robot (it needs to look human for a lot of people to interact with it on a sufficient level), and the ability to distinguish bias from experience from prejudice. (A lot of humans suck at that as well, to be fair.)
Any medical AI at this level will only be as good as the human medical professional using it, and it will make preexistent human-caused bias/prejudice problems much worse.
(Largest one I'm aware of: not being male is an appallingly large handicap to getting properly and correctly diagnosed, even in Western countries.)
Way back in 1979, the MYCIN system was shown to be better at diagnosing than Doctors:
>...Eight independent evaluators with special expertise in the management of meningitis compared MYCIN's choice of antimicrobials with the choices of nine human prescribers for ten test cases of meningitis. MYCIN received an acceptability rating of 65% by the evaluators; the corresponding ratings for acceptability of the regimen prescribed by the five faculty specialists ranged from 42.5% to 62.5%. The system never failed to cover a treatable pathogen while demonstrating efficiency in minimizing the number of antimicrobials prescribed.
I watched my PCP read the Google AI summary to me in office when I asked a question about my visit about a year ago... I'd rather just use my Claude sub at this point than pay $180 for 10 minutes of interaction. AI should be disruptive in this space, but it's a walled garden and I feel like they will do everything to avoid disruption.
Healthcare is one of the most critical fields to our collective wellbeing while simultaneously being one of the most problematic ones, with doctors routinely disregarding, misdiagnosing, and otherwise failing to help or even harming patients, especially anyone who isn't a typical case of a typical issue.
I have more sympathy for almost every other profession affected than doctors.
Do you live in the USA? Living in Europe, my experiences as well as what I'm hearing have mostly been pretty good. Not sure I even know anyone with a major misdiagnosis.
(Everything is way over capacity though. Can't recall a specific one but I do think there's a few "took too long, worse off due to that". But that's a scale & system-wide finance issue, not quality.)
> The system achieved its strongest results using OpenAI’s GPT-5, with additional testing showing improvements using GPT-5.5.
If it's basically dumping details into ChatGPT, I'd wonder why the hospital would bother paying Microsoft. They can ask it to simulate a panel of physicians too.
One thing missing from the comments so far is that this system only gets to see answers and ask specific questions. It misses out on some things from the environment and is tested in a way that gives it advantage.
It won't notice that the patient's hand is trembling. It won't be aware of issues common in some areas. It won't notice that the patient always comes in with a partner and is nervous about some answers. Etc. It's amazing that it beats doctors on applying a turn-by-turn decision tree, but that's not what a typical visit looks like. (Yes yes, there are terrible doctors who don't care, that's besides the point here)
As a healthcare provider (albeit paramedic), I don't think the issue is AI's encyclopedic knowledge of disease, but rather, how to draw out a good, structured patient interview and exam, and cross-reference details. I could regale you with patients who only remembered their serious penicillin allergy when we got to the hospital, or forget their shoulder pain when describing chest pressure and so on and so on and so on.
However, this is something an AI can be good at. And well, GIGO, be your provider human or AI. I can't act on things you don't tell me. Well, somewhat - if I'm looking at your lower legs and seeing severe edema, I might be skeptical of your claim that "other than what's going on today, perfectly healthy". So maybe we need realtime video too.
Relatedly, I was talking to my immigration attorney, who has been very good, on an area where immigration law intersects some other topics, and she was fairly obviously using AI on our video call to guide her on many areas she wasn't as familiar with.
Agree. I have been an EMT and I have been a patient in a teaching hospital. You would need video, and then a robot arm to test if that is pitting edema, continuing your illustration.
^ this. If Microsoft says it, it's true. Who posts this stuff? Real issues in diagnosis after the first follow up question (microsoft doesn't talk about that).
Echoing others, but not even rare diseases. I had perioral dermatitis, first result on google for the symptoms. I went to 4 dermatologists over 4-5 months who failed to diagnose it, assuring me it absolutely wasn't a fungus, instead telling me it was eczema and prescribing topical steroids (as is their wont). The real solution was washing my face a bit more.
The first time I heard "When you hear hoofbeats, think horses, not zebras" I was appalled. How about we just empirically determine where the hoofbeats are coming from before eliminating the killer zebras as a possibility?
Because we have limited resources, time and every intervention can cause more harm. For example, you don't want to get an MRI every time you get a headache - it would be counterproductive to you, drain resources from a system which needs to handle more patients, and regularly make you visit an environment with potential infectious diseases. Where we have a quick and simple empirical test, we often apply it and the available options are growing - for example we have quick tests for a number of respiratory diseases, which wasn't available earlier.
AI is going to be both a boon and a bane for doctors. Its "good enough" behavior will lead to complacency like it does any place where the automated system is "good enough" but the system still wants human validation. This will without a doubt result in some preventable tragedies that will end with more work and check lists (and probably not enough accountability). But there are other places where it will absolutely free up mental energy to focus more on the patient.
I was talking to my own doctor and their experience with a new note taking / transcription AI app they've been using, having worked on medical software before. One of the problems I noticed when designing software then was that software likes its categorized fields, with data entered just the right way and in just the right order. But medical work is almost never that neat and tidy. Patients and other medical professionals give you information in whatever order things come to their mind in, or in whatever order their own system and paperwork displays it. A simple prescription consisting of a patient's name, DOB, medication, medication form, dispensing quantity, refills and dosing instructions might be given to a provider in any order of those fields and no matter what order your software chooses, it won't be the same order as someone else giving it to you. Most providers that used the software I worked on resorted to hand writing everything down and filling it all in later, or having to ask for repetition over and over as they clicked through various fields trying to remember everything before it flicked out of their short term memory or the other person moved on. And as more and more systems and regulations require everything to be entered into electronic records, this has become more of a problem. To the point where you go to some appointments and you often don't even see your doctor's face because they're stuck behind a terminal nodding along as they try to capture everything you say into the specific fields in the system. AI tooling has the potential (and at least from my own doctor's experience, is already showing that potential) to free up the mental effort spent on getting the right data into the right fields. AI can do a great job of understanding that a free text field that says:
Fucitol tab 25mg qid
John Smith 01/01/75
#30 3 months
and a free text field that says:
Fucitol 25mg tablet
30 tabs, 4x per day
3 refills
Smith, John 1975-01-01
say the same things, and get those things into the right fields in the record for you. They still need to go back and check it (and see the previous caveat about "good enough" systems) but by freeing the doctor from the mental load of intentionally categorizing all of the data they're hearing while simultaneously trying to reason through what all that data means and enter it at the same time they have more mental room to actually pay attention to the things the patient is saying, the things they're doing and the things the doctor is seeing.
And one thing I hope is that it makes switching doctors if you need to safer and more effective. The sheer volume of information a new patient can bring with them if they have all their medical records is massive and sorting through it all is likely something we'll never give our providers enough time and money to do. Things will get missed, or doctors will defer (consciously or not) to the diagnosis already listed, even if a more thorough review might reveal the diagnosis was wrong. An AI system designed to review an influx of data with an adversarial eye, or at least an eye towards looking for missed possibilities or potential reasons for the patient's dissatisfaction with the previous provider could go a long way towards helping end the sorts of "I went through 10 doctors all of which told me it was all in my head/I should try losing weight/I can't be experiencing this issue at my age, and then finally the last doctor had just finished reading a paper about how XYZ can cause these symptoms and now I'm finally getting treatment" stories anyone with friends or family with chronic health problems probably has heard more than once.
The problem is that insurance won't authorize unless the doctor has a good reason to suspect a rare decease with expensive tests. Even then they will verify that the doctor has followed protocol and gone thru all the cheap tests first.
Here in the UK, they will try their hardest to not refer to the consultant, will do the basic blood tests and tell you that you are all right, take pain killers or anti allergy medicine. Its such a terrible system
People gotta get over it. Just like we had to accept AI can code better than us, we have to accept that AI will just be far better at dispensing medical advice and enabling us to truly divulge all our medical history without being dismissed or judged by a doctor who sees so many patients they can’t possible care deeply about each and every single one.
The system is already broken, so turning over control to AI companies where no human ever sees the patient is a no brainer. There's no tradeoff to consider here, after all the our entire economy is riding on the success of AI, so any concern about human factors that can't be measured by economists is just a distraction from what really matters.
I'm not sure what AI companies have to do with it. Open weight models are available and work great, literally today kimi K3.1 pro was announced and goes toe to toe with closed models. You are tying AI usage to bigAI and that is unfounded.
Medical doctors are probably of the easiest profession to be replaced with Ai when the time comes but not in all fields yet. General practice is probably easy to replace with Ai; next is oncologists and any narrow field of specialty that requires simple examination and a lot of testing. Doctors have done everything possible to avoid touching and examining patients closely in recent times and the field is Ai ready. Some blood and urine test, Xrays and MRI and voila... diagnosis.
You already blew way past budget, and worse, availability, than most patients can afford. And you want to minimise unnecessary radiation exposure. Those things are ordered when you have indications they will help (with diagnosis or care), not as scattershot to get diagnostic data.
I disagree, I produced a reasoning. Medical fields of practice that have little hands on interaction are easiest to duplicate with ai because diagnosis is based on tests, conversation and observation.
For rural docs without access to specialists or older docs who need to stay up to date this is great. It is 100% terrible for med students, residents, fellows, and young attendings ALL of whom need the practice and experience of unassisted diagnosis and treatment on the cases they see most commonly. It’s no surprise that it’s “better” for complex cases, but it’s most likely on par with a PCP or FP or Pedi doc or specialist seeing common issues they see everyday in their own practices. No doc will refuse a new toy, so I’m worried that this will lead to a deterioration of clinical expertise in the US. Any other docs have an opinion?
This is the same story for software development, and it’s going to continue to spread to specialist fields.
As a consumer, if the AI was truly better why would you want to accept substandard care because younger docs need “practice” ?
If I needed someone to build me a complicated application, I would choose the software developer with 20 years of experience before LLM's were capable of helping software developers that uses LLM's to assist over the software developer with 1 year of experience that never learned to code and just uses LLM's for everything but is clueless as to how any of it works.
Same thing with doctors.
But you’ve arrived at the ER and the attending staff are oversubscribed, so you have interns and residents primarily trying to keep you alive. Sure, if you are a VIP you’ll get extra special care, but most of us aren’t.
Most ER work is urgent but familiar even to junior staff, and a lot of it is not life-threatening either. If they need to 'keep you alive' then they are following established protocols, not asking an LLM for advice.
1. This a claim from Microsoft not a peer reviewed study on pubmed. So it's a marketing title nothing more
2. Even some ideal frontier model is only as accurate as the inputs given for diagnosis, and will typically bias towards said inputs.
3. Multi-modal can handle images but are terrible at image classification unless using dedicated models for each pathology.
4. The very large majority of traning corpus with these companies is in English. There is zero guarantee that said mythical performance is preserved across languages.
5. The very thought of companies with the moral compass of most AI companies accessing and training on your medical records should make anyone pause.
How do you make corn when you’ve eaten all the seed?
other industries by and large have been able to figure out how to separate novice practitioners away from actual public utility until they're ready to participate.
I don't claim to know whether that could be done feasibly for medicine, I only point out that medicine is uncommon in how it educates.
Medicine is uncommon because there are no practice pieces for a lot of things. This isn't about learning stitches, this is about learning how to interact with a human, figure out what is going on, what tests and checks and options to even order.
Which, coincidentally, won't be solved by AI anytime soon. You'd need a full anthropomorphic robot (it needs to look human for a lot of people to interact with it on a sufficient level), and the ability to distinguish bias from experience from prejudice. (A lot of humans suck at that as well, to be fair.)
Any medical AI at this level will only be as good as the human medical professional using it, and it will make preexistent human-caused bias/prejudice problems much worse.
(Largest one I'm aware of: not being male is an appallingly large handicap to getting properly and correctly diagnosed, even in Western countries.)
You make more of the artificial one.
I'm not sure about the 100% terrible bit as I presume they'll try diagnosing as well and then check with what the robot says and maybe learn from it.
Why can't they practice by having an AI and the student both make a diagnosis independently, and comparing them?
Way back in 1979, the MYCIN system was shown to be better at diagnosing than Doctors:
>...Eight independent evaluators with special expertise in the management of meningitis compared MYCIN's choice of antimicrobials with the choices of nine human prescribers for ten test cases of meningitis. MYCIN received an acceptability rating of 65% by the evaluators; the corresponding ratings for acceptability of the regimen prescribed by the five faculty specialists ranged from 42.5% to 62.5%. The system never failed to cover a treatable pathogen while demonstrating efficiency in minimizing the number of antimicrobials prescribed.
https://pubmed.ncbi.nlm.nih.gov/480542/
[dead]
I watched my PCP read the Google AI summary to me in office when I asked a question about my visit about a year ago... I'd rather just use my Claude sub at this point than pay $180 for 10 minutes of interaction. AI should be disruptive in this space, but it's a walled garden and I feel like they will do everything to avoid disruption.
Healthcare is one of the most critical fields to our collective wellbeing while simultaneously being one of the most problematic ones, with doctors routinely disregarding, misdiagnosing, and otherwise failing to help or even harming patients, especially anyone who isn't a typical case of a typical issue.
I have more sympathy for almost every other profession affected than doctors.
Do you live in the USA? Living in Europe, my experiences as well as what I'm hearing have mostly been pretty good. Not sure I even know anyone with a major misdiagnosis.
(Everything is way over capacity though. Can't recall a specific one but I do think there's a few "took too long, worse off due to that". But that's a scale & system-wide finance issue, not quality.)
> The system achieved its strongest results using OpenAI’s GPT-5, with additional testing showing improvements using GPT-5.5.
If it's basically dumping details into ChatGPT, I'd wonder why the hospital would bother paying Microsoft. They can ask it to simulate a panel of physicians too.
One thing missing from the comments so far is that this system only gets to see answers and ask specific questions. It misses out on some things from the environment and is tested in a way that gives it advantage.
It won't notice that the patient's hand is trembling. It won't be aware of issues common in some areas. It won't notice that the patient always comes in with a partner and is nervous about some answers. Etc. It's amazing that it beats doctors on applying a turn-by-turn decision tree, but that's not what a typical visit looks like. (Yes yes, there are terrible doctors who don't care, that's besides the point here)
As a healthcare provider (albeit paramedic), I don't think the issue is AI's encyclopedic knowledge of disease, but rather, how to draw out a good, structured patient interview and exam, and cross-reference details. I could regale you with patients who only remembered their serious penicillin allergy when we got to the hospital, or forget their shoulder pain when describing chest pressure and so on and so on and so on.
However, this is something an AI can be good at. And well, GIGO, be your provider human or AI. I can't act on things you don't tell me. Well, somewhat - if I'm looking at your lower legs and seeing severe edema, I might be skeptical of your claim that "other than what's going on today, perfectly healthy". So maybe we need realtime video too.
Relatedly, I was talking to my immigration attorney, who has been very good, on an area where immigration law intersects some other topics, and she was fairly obviously using AI on our video call to guide her on many areas she wasn't as familiar with.
Agree. I have been an EMT and I have been a patient in a teaching hospital. You would need video, and then a robot arm to test if that is pitting edema, continuing your illustration.
Title is: Microsoft Says AI Is More Than Twice as Good at Diagnosing Diseases as Doctors
^ this. If Microsoft says it, it's true. Who posts this stuff? Real issues in diagnosis after the first follow up question (microsoft doesn't talk about that).
Twice as many hallucinations too?
Often doctors miss rare diseases and gaslight patients that they don't have them.
AI is great for this since it has encyclopedic knowledge.
The diseases don't even have to be rare:
* https://pmc.ncbi.nlm.nih.gov/articles/PMC9802334/
* https://time.com/6074224/gender-medicine-history/
Echoing others, but not even rare diseases. I had perioral dermatitis, first result on google for the symptoms. I went to 4 dermatologists over 4-5 months who failed to diagnose it, assuring me it absolutely wasn't a fungus, instead telling me it was eczema and prescribing topical steroids (as is their wont). The real solution was washing my face a bit more.
Which they (probably?) should otherwise they will overtreat their patients for false positives.
Averages are not much use if you are the patient whose diagnosis was missed
The first time I heard "When you hear hoofbeats, think horses, not zebras" I was appalled. How about we just empirically determine where the hoofbeats are coming from before eliminating the killer zebras as a possibility?
Because we have limited resources, time and every intervention can cause more harm. For example, you don't want to get an MRI every time you get a headache - it would be counterproductive to you, drain resources from a system which needs to handle more patients, and regularly make you visit an environment with potential infectious diseases. Where we have a quick and simple empirical test, we often apply it and the available options are growing - for example we have quick tests for a number of respiratory diseases, which wasn't available earlier.
I wonder if any doctors feel depressed about losing "creativity", or are instead yeah I can help diagnose and cure more people.
AI is going to be both a boon and a bane for doctors. Its "good enough" behavior will lead to complacency like it does any place where the automated system is "good enough" but the system still wants human validation. This will without a doubt result in some preventable tragedies that will end with more work and check lists (and probably not enough accountability). But there are other places where it will absolutely free up mental energy to focus more on the patient.
I was talking to my own doctor and their experience with a new note taking / transcription AI app they've been using, having worked on medical software before. One of the problems I noticed when designing software then was that software likes its categorized fields, with data entered just the right way and in just the right order. But medical work is almost never that neat and tidy. Patients and other medical professionals give you information in whatever order things come to their mind in, or in whatever order their own system and paperwork displays it. A simple prescription consisting of a patient's name, DOB, medication, medication form, dispensing quantity, refills and dosing instructions might be given to a provider in any order of those fields and no matter what order your software chooses, it won't be the same order as someone else giving it to you. Most providers that used the software I worked on resorted to hand writing everything down and filling it all in later, or having to ask for repetition over and over as they clicked through various fields trying to remember everything before it flicked out of their short term memory or the other person moved on. And as more and more systems and regulations require everything to be entered into electronic records, this has become more of a problem. To the point where you go to some appointments and you often don't even see your doctor's face because they're stuck behind a terminal nodding along as they try to capture everything you say into the specific fields in the system. AI tooling has the potential (and at least from my own doctor's experience, is already showing that potential) to free up the mental effort spent on getting the right data into the right fields. AI can do a great job of understanding that a free text field that says:
and a free text field that says: say the same things, and get those things into the right fields in the record for you. They still need to go back and check it (and see the previous caveat about "good enough" systems) but by freeing the doctor from the mental load of intentionally categorizing all of the data they're hearing while simultaneously trying to reason through what all that data means and enter it at the same time they have more mental room to actually pay attention to the things the patient is saying, the things they're doing and the things the doctor is seeing.And one thing I hope is that it makes switching doctors if you need to safer and more effective. The sheer volume of information a new patient can bring with them if they have all their medical records is massive and sorting through it all is likely something we'll never give our providers enough time and money to do. Things will get missed, or doctors will defer (consciously or not) to the diagnosis already listed, even if a more thorough review might reveal the diagnosis was wrong. An AI system designed to review an influx of data with an adversarial eye, or at least an eye towards looking for missed possibilities or potential reasons for the patient's dissatisfaction with the previous provider could go a long way towards helping end the sorts of "I went through 10 doctors all of which told me it was all in my head/I should try losing weight/I can't be experiencing this issue at my age, and then finally the last doctor had just finished reading a paper about how XYZ can cause these symptoms and now I'm finally getting treatment" stories anyone with friends or family with chronic health problems probably has heard more than once.
The problem might be more that doctors don’t want to order expensive tests for those rare conditions .
AI might have similar reservations depending on standard of care
The problem is that insurance won't authorize unless the doctor has a good reason to suspect a rare decease with expensive tests. Even then they will verify that the doctor has followed protocol and gone thru all the cheap tests first.
It sounds like both of you are looking at this with the US healthcare system in mind, and that non-reviewed "study" probably does too.
…which actually makes sense. "The AI said so" might be a magic trick in that absolute hellscape of bad care and financial racketeering.
Here in the UK, they will try their hardest to not refer to the consultant, will do the basic blood tests and tell you that you are all right, take pain killers or anti allergy medicine. Its such a terrible system
People gotta get over it. Just like we had to accept AI can code better than us, we have to accept that AI will just be far better at dispensing medical advice and enabling us to truly divulge all our medical history without being dismissed or judged by a doctor who sees so many patients they can’t possible care deeply about each and every single one.
Yes, let's just get over it.
The system is already broken, so turning over control to AI companies where no human ever sees the patient is a no brainer. There's no tradeoff to consider here, after all the our entire economy is riding on the success of AI, so any concern about human factors that can't be measured by economists is just a distraction from what really matters.
I'm not sure what AI companies have to do with it. Open weight models are available and work great, literally today kimi K3.1 pro was announced and goes toe to toe with closed models. You are tying AI usage to bigAI and that is unfounded.
Isn’t Kimi K3.1 pro developed by an AI company?
> You are tying AI usage to bigAI and that is unfounded.
Tell me a single competitive AI model that hasn’t been created by a large AI company. I don’t care if weights are open or not.
That's moving the goal post. I was responding to this line in your comment:
> so turning over control to AI companies where no human ever sees the patient is a no brainer
Using an open weight model turns no control over to an AI company, even if that model was created by some AI company.
Medical doctors are probably of the easiest profession to be replaced with Ai when the time comes but not in all fields yet. General practice is probably easy to replace with Ai; next is oncologists and any narrow field of specialty that requires simple examination and a lot of testing. Doctors have done everything possible to avoid touching and examining patients closely in recent times and the field is Ai ready. Some blood and urine test, Xrays and MRI and voila... diagnosis.
> Xrays and MRI
You already blew way past budget, and worse, availability, than most patients can afford. And you want to minimise unnecessary radiation exposure. Those things are ordered when you have indications they will help (with diagnosis or care), not as scattershot to get diagnostic data.
Not exactly…
Not exactly what?! Say that you don't agree and then produce an argument as to why!
"Up and to the right": https://news.ycombinator.com/item?id=50220850
You gave no argument as to why either
I disagree, I produced a reasoning. Medical fields of practice that have little hands on interaction are easiest to duplicate with ai because diagnosis is based on tests, conversation and observation.
For pure diagnosis, maybe.
Not for surgery, at least not yet. The robotics isn’t quite good enough
The time when a robot surgeon outperforms a human one probably isn’t that far off, I expect China will come out with something in 5-10 years.
Obviously you are right, not surgery yet. I am talking about hands off diagnosis and treatments.
Actually it is unfortunate we do not have mostly automated radiological interpretation, likely to do with data hoarding by incumbents