Show HN: Using GPT-3 and Whisper to save doctors’ time

117 points by ar7hur 3 years ago | 137 comments

Hey HN,

We're Alex, Martin and Laurent. We previously founded Wit.ai (W14), which we sold to Facebook in 2015. Since 2019, we've been working on Nabla (https://nabla.com), an intelligent assistant for health practitioners.

When GPT-3 was released in 2020, we investigated it's usage in a medical context[0], to mixed results.

Since then we’ve kept exploring opportunities at the intersection of healthcare and AI, and noticed that doctors spend am awful lot of time on medical documentation (writing clinical notes, updating their EHR, etc.).

Today, we're releasing Nabla Copilot, a Chrome extension generating clinical notes from video consultations, to address this problem.

You can try it out, without installation nor sign up, on our demo page: https://nabla.com/copilot-demo/

Here’s how it works under the hood:

- When a doctor starts a video consultation, our Chrome extension auto-starts itself and listens to the active tab as well as the doctor’s microphone.

- We then transcribe the consultation using a fine-tuned version of Whisper. We've trained Whisper with tens of thousands of hours of medical consultation and medical terms recordings, and we have now reached an error rate which is 3× lower than Google's Speech-To-Text.

- Once we have the transcript, we feed it to a heavily trained GPT-3, which generates a clinical note.

- We finally return the clinical note to the doctor through our Chrome extension, the doctor can copy it to their EHR, and send a version to the patient.

This allows doctors to be fully focused on their consultation, and saves them a lot time.

Next, we want to make this work for in-person consultation.

We also want to extract structured data (in the FHIR standard) from the clinical note, and feed it to the doctor’s EHR so that it is automatically added to the patient's record.

Happy to further discuss technical details in comments!

---

[0]: https://nabla.com/blog/gpt-3/

swatcoder 3 years ago |

I don't spend much time worrying about AI ethics, but bringing AI close to patient interactions and record-keeping in healthcare seems grossly premature and irresponsible. There are countless brilliant applications of AI and the last one we need is a mediocre automated transcriptionist that distances a doctor from their responsibility to engage with the patient and can bear no accountability for error.

This is a task that perhaps can be supported with AI some day, but there are fields that deserve the application of a mature technology, not the gold rush rush game of integrating today's hottest thing.

D13Fd 3 years ago | |

The transcription part doesn't bother me. That's just advanced speech-to-text.

The summarization part, though, is dangerous. It's a very quick path to us losing all faith in our own medical records. Any way you slice it, no matter how much you train it, it's still going to be vulnerable to hallucination errors that slip by the reviewing doctor and become part of the patient's medical history.

ar7hur 3 years ago | |

I understand your concerns. Our objectives are, on the contrary, to reduce the doctor/patient distance that was created by EHRs and all the required administrative processes. We already measure that when AI takes care of this stuff, doctors do better engage with patients.

ttpphd 3 years ago | |

"I don't spend much time worrying about AI ethics"

Maybe you should re-evaluate whether that's the right choice based on your own comment, especially when ethics and safety experts are being fired by big tech co.

JohnFen 3 years ago |

As a patient, this is an application that I would be extremely wary of, personally. I don't want the details of my conversations with my doctor to be sent to a third party in the first place, and I wouldn't trust the results of the transcription to be correct.

My doctor could vet it for accuracy, I suppose, but why? He's already putting his notes in my records anyway.

1123581321 3 years ago | |

Most people doing a video meeting with their doctor are already trusting their information to third parties in some way. I understand that you probably are in a different situation. Perhaps services like these are on the path to your more privacy-oriented doctor having a tool that helps them and is comfortable for knowledgeable patients.

JohnFen 3 years ago | | |

Yes, if you're doing telehealth, you're already taking a risk in terms of privacy. But the service provider doesn't need to actually pay attention to what you and your doctor are talking about to provide the service.

This is different, in that this service requires that a third party pay attention to the content of your discussion in order to work. That's significantly more intrusive.

l5t 3 years ago | |

It takes 40% of your doctor time to put his notes in your medical records. We have been testing with doctors to reduce this time drastically so that they can spend more quality time with you. We also interviewed a lot of patients who take notes to be able to remember what’s been discussed during a consultation. So it could also benefit the patient

Veen 3 years ago | | |

Your response completely ignores the concerns raised in the comment you are replying to. "It saves your doctor 40% of their time" is not a reasonable response to "I don't want my private medical details divulged to a third party".

burntcookie90 3 years ago | | |

> that they can spend more quality time with you

Maybe its just the doc offices i've been to, but i think this would actually just increase the patient churn in a hospital. There's no way a doc is going to increase their patient time by the 40% saved if the hospital can toss them in front of another patient.

JohnFen 3 years ago | | |

I can absolutely see the benefits of this, that's not the question. The question is whether or not the benefit justifies the risk.

mannyv 3 years ago | |

Many practices today have a transcriber with every doc, and that person is their personal scribe. You trust them, right?

In any case HIPAA will protect you. Your provider will have a business associate contract with them, so they're subject to huge fines if they violate that.

This is a demo. In real life this'll be wrapped with a whole lot of legal contracts.

c0m 3 years ago | | |

I trust a scribe because they are a person with years of specialised training and experience who can be held accountable.

> in any case, HIPAA will protect you

It protects you insofar as it disincentives honest actors from doing sketchy things with your data. It's punishment for orgs, not protection for patients, like how laws against murder punishes the murderer rather than protecting the victim.

The best thing a patient can do to protect their privacy is to be actively avoid of medical practitioners that, for example, use tools like this which send your private medical consultation transcripts to God-knows-where.

SkyPuncher 3 years ago | |

> I don't want the details of my conversations with my doctor to be sent to a third party in the first place

Then you simply cannot use modern healthcare. Unless you're doctor is cash-pay, off the cuff, they're _legally_ using third-parties to perform their services.

----

If your doctor/healthcare provider is a covered entity, they're bound by HIPAA. That means they're free to establish relationships with Business Associates that are part of delivering healthcare. Those Business Associates must also be HIPAA compliant.

v4dok 3 years ago |

I like the idea, but I would like it even more if it was on-prem. The doctors (at least in EU) will be very wary of having their client meeting essentially recorded by a third party. With this as a cloud SaaS, patient confidentiality is essentially broken since the raw data is available to you while you transcribe it. I understand that you compete with "google speech-to-text" but this is not a feature meant to be used by doctors (even if they "illegally" do).

Obviously the business model is harder with on-prem, but cloud-first for doctor notes is in the long run much harder.

priyanmuthu 3 years ago |

My worry is verification of facts. What if the model summarized incorrect facts, and it gets added to the medical history? How can doctors easily verify and fix things?

jjoonathan 3 years ago | |

Have you had a medical billing experience recently? They don't care about facts, they care about maximizing the bill subject to plausibility constraints.

I fear that LLMs will kick this to a whole new level :(

sp332 3 years ago | |

It’s easier to revise than to write a first draft. It’s possible to miss mistakes on review, but relatively, I think it’s more likely that a distracted human will mess something up on the first draft.

D13Fd 3 years ago | | |

That's exactly the thought process that leads to us trusting LLMs in places where they shouldn't be trusted.

qgin 3 years ago | |

A good benchmark could be to compare the accuracy of generated notes to the accuracy of other currently accepted practices like human medical scribes and assistants.

ar7hur 3 years ago | |

Thanks for your comment. We clearly show a diff to the doctor before updating patient records. We are very explicit that they should check it (and edit if necessary).

avgDev 3 years ago |

I feel uncomfortable knowing doctor would use anything like this.

How are 2 party consent states handled?

Is this HIPPA compliant?

NorthOf33rd 3 years ago |

This is the scariest real life attempt to use the bullshit machine I’ve seen so far.

Nothing like “subtly wrong” clinical notes to affect positive medical outcomes. And time pressed medical professionals are certainly well suited to vigilance tasks like correcting machine generated errors. /s.

I sincerely hope this never sees the light of day.

manv1 3 years ago | |

If you think this is scary you should see how it's done today without AI.

Doctor's handwriting is notoriously bad. Now try putting that into an EMR. Good luck with that! You think you're going to get someone who gets paid $2000/hr to type shit up? Yeah, guess again.

Pigalowda 3 years ago | | |

We don’t get paid even close to $2k/hr. Where is that number from? Many physicians type notes, others use dragon dictation, or medical assistants/scribes.

I used to type, now I dictate.

Quality of note is provider dependent though. If an LLM can improve quality of notes across the board through simple summary that’s interesting. But the input matters - is the provider actually asking the right questions? Did they look at past relevant history and other notes and put it in their current note?

The LLM should be in a HIPAA compliant environment and have access to the EMR. Then it provides a tidy summary for that. Second it should have relevant questions for the provider to ask during the interview but also a real time/dynamic component which generates relevant follow up questions dependent on patient responses.

Lastly it should put together the old and the new and generate a new note which can then be edited by the provider. Editing notes is much easier than generation of a new high quality note.

urduntupu 3 years ago |

Technically interesting but solving the wrong problem.

Doctor's already today spent too little time with their patients to understand diseases holistically enough.

Adding technically between these 2 will make treatments in most of the cases worse, not better.

vsareto 3 years ago | |

Nothing's going to fix that but having more doctors around.

Going from a recorded transcript to a summary note, extracted structured data, and diagnostic codes is a big time saver.

nharada 3 years ago | |

> solving the wrong problem

You seem quite confident about this, but based on the doctors I know writing notes is a real pain and mostly seen as (important) scutwork. They're only given a set amount of time to see a patient (including notes), and if you can reduce notes they'd actually get more time talking and seeing the patient.

alach11 3 years ago | | |

> if you can reduce notes they'd actually get more time talking and seeing the patient

You're not thinking like a hospital administrator. Sounds like these doctors can handle 40% more patients to me!

SkyPuncher 3 years ago | | |

You're both correct. My wife is a psychiatrist. I'm friends with several doctors through her.

Everyone hates notes. However, there are already a bunch of dictation services that help write notes more efficiently. They absolutely work well enough.

Adoption and funding is the primary blocker. When my wife works at facilities that have M-Modal, she's 80%+ faster. However, she works at many, who despite knowing the benefits, simply don't care to spend the time, money, or resources on implementing scribe/dictation services.

thfuran 3 years ago | | |

What percent of that scutwork is actually note taking rather than clicking a bunch of checkboxes or combo boxes or otherwise navigating through EHR UI?

JohnFen 3 years ago | | |

The last time I visited a doctor, he had an assistant with him that took all the notes.

stopachka 3 years ago | |

Right now, drs type notes as you speak to them. If this works, they'd have more time to spend actually listening and talking to you.

hartleybrody 3 years ago | |

Wow, lots of haters in this thread. Having doctors spend less time with clerical/administrative work needed to record conversations will free up more time to focus on patients. I assume the generated note text can be easily modified by the doc if necessary, but saves them the bulk of having to type things up and remember everything that was discussed.

orcajerk 3 years ago |

Last time I went to the doctor, he charged me $300 to google symptoms on a computer. Now you're saying I get to pay $300 for him to ask ChatGPT? No thanks. We all know that is where this is going - doctor decision support system, i.e. the AI has access to information locked behind expensive journals that the patient does not.

D13Fd 3 years ago |

The "feed it to a heavily trained GPT-3" part is backwards. The transcribed notes should be stored and retained. The LLM should be applied later, in real time, when someone is referring back to the notes.

That way you still get the benefit of summaries, without polluting the medical records with hallucinated AI nonsense.

Terretta 3 years ago |

Ignore the nay-sayers in this thread. I doubt the ones commenting have spent many days personally observing or personally doing the problem scenario you are solving for.

This is spectacular for telemedicine in the post-covid regulation environment.

The new model is an onsite medical assistant taking patient vitals and setting up the Zoom, the patient then talks to an APRN (Advanced Practice Registered Nurse) in the patient's field (e.g. subacute care, gerontology, etc.).

The APRNs spend 30 minutes reading the file, tele-consult with a patient for a half hour. Medicare pays for the half hour. Then the APRN has to spend ~60 minutes turning that session into the EHR notes. Many just don't, they make an ultra short note, and hope Medicare doesn't challenge it. Patient loses. If too strapped for time, they also may not pre-read, because they only get paid for the video conf time. Patient loses.

Downside I see with this is the APRNs doing telemedicine are making about 1/4 of the rate they should be, and can't afford an expensive tool. The telehealth provider is not going to pay for it, they get paid for the 1/2 hour, the wasted time is not out of their pocket, so they won't spend to make it up. This is going to be on the nurse, who likely can't afford it.

On the plus side: time saved with this tool would go directly to ability to pre-read, or ability to see a patient an hour instead of one per two or three hours. It would also permit a higher caliber of expertise nurse to do tele-medicine visits in off days, because the time saved could bring comp back more in line with the expertise.

Go for it!

908B64B197 3 years ago |

> This allows doctors to be fully focused on their consultation

Here's the thing: rigorous medical documentation is not a burden on the practitioner, but rather an integral part of treating someone. You need detailed historical data to diagnose some tricky conditions and treat them correctly.

What's the tool doing to prevent a practitioner from simply blindly copy-pasting the output without verifying it?

> and saves them a lot time.

Which means more billable. Have you considered charging per medical act performed?

khyryk 3 years ago | |

Unfortunately I've had interactions with the medical system where the doctors wrote their after meeting notes with severe errors and pieces of fiction, as well as specialists who almost ran out of the room after 5 minutes of an appointment they billed 45 minutes for to the insurance. I doubt such cases are rare and I suspect this sort of product, which I'm a priori neutral about, would change the situation much for the worse.

norgie 3 years ago |

Is Whisper really up to this? When I've used it, it's...fine. But it also randomly dreams up weird transcriptions that have nothing to do with what's actually being said, and with perfect audio too. Like the audio will be a presentation at a tech conference, and Whisper will predict "check out our site at randomlink.com", when no one said anything close to that.

I'd be pretty concerned if my doctor relied on this to be honest.

l5t 3 years ago | |

Good point, this is why we finetune Whisper with our own medical dataset to improve the performance

ysavir 3 years ago |

I'd prefer to have 0 AI interpreting my session with a doctor. Will there be any way for me to identify which physicians use this service so that I can avoid using them?

narwally 3 years ago | |

Hell, it's hard enough these days to identify which physicians are actually physicians. I recently had an NP introduce herself to me as "Dr. So-and-so."

rubatuga 3 years ago |

I did some work at a startup that tried to solve a similar problem and currently finishing medical school. Nobody here is discussing the paperwork and burnout problem that exists in medicine that will make these tools valuable. Fortunately, language models used for summarization based on a transcript are much less likely to hallucinate compared to upscaling/denoising image algorithms. Recent LMs are scoring top marks on LSAT, so they are maintaining accuracy as they get larger. If doctors can save time, this will be useful for the patient in the long-run.

On-prem stuff is more tricky, but useful for privacy. Nvidia is supporting the space with their health AGX series.

JohnFen 3 years ago | |

> On-prem stuff is more tricky, but useful for privacy.

An on-prem installation, assuming that it doesn't phone home to the mothership, is something I would entertain. But one that requires disclosure of medical information to a service provider would be a lot harder for me to accept.

hartleybrody 3 years ago |

I used to work in a hospital and one thing that was already tough was when doctors would copy massive (multiple page) notes that were mostly boilerplate, probably stored as some template in the EHR. After seeing the same note a few times, you started to learn which page to scroll to in order to view the relevant section for the patient. I would hope this system only generates clinically-relevant text for that patient, usually only 2-3 paragraphs for a standard consultation.

lukah 3 years ago |

This is really interesting work, congratulations! A lot of negativity in the comments seems to stem from cynicism around doctors and their billing, but in a country like the UK I know removing essentially "admin tasks" from the GP would be very welcome and give time back to the clinician, allowing them to spend more time focussing on the patient and outcomes.

cinntaile 3 years ago |

> We then transcribe the consultation using a fine-tuned version of Whisper. We've trained Whisper with tens of thousands of hours of medical consultation and medical terms recordings, and we have now reached an error rate which is 3× lower than Google's Speech-To-Text.

I think the general idea is interesting but this is the wrong benchmark to convince people imo. What is the actual error rate of Google's Speech-to-Text on this your data? How convinced are you that it's properly labeled? That it's 3x lower isn't necessarily impressive. Then a follow up question... What I as a patient would want to know is something else, how does it compare against medical secretaries that usually transcribe documents like these?

CiceroCiceronis 3 years ago |

I'm a medical student and one of my main takeaways from spending time in clinical environments is that a lot of doctors and other staff seem to either not realise or not care about how much time is wasted due to slow, inefficient processes and technology. I was wondering to myself about what it might take to train a model to recognise medical jargon to help doctors rapidly transcribe audio recordings of their consults/spoken recounts of their notes. Not surprised that somebody is looking to make a commercial product of it, there's definitely space for it.

tomasyany 3 years ago |

It looks pretty neat! Just wondering, what other languages are supported? English seems pretty natural as it is where GPT-3 thrives, but have you been able to support other languages as well?

l5t 3 years ago | |

We currently support English and French and we will be supporting more soon

retox 3 years ago |

How tied are you to the name, it's almost Nambla.

whitethunder 3 years ago | |

That's all I see when I see this company's name. Unfortunate choice.

gigel82 3 years ago |

What is OpenAI's privacy policy? How do you ensure the transcription doesn't get stored by OpenAI or used in training their future LLM versions?

I'd recommend looking at a fully self-hosted LLM alternative or ensuring the GPT-3 endpoint you're using is compliant (I've heard Azure has a GPT-3 API that is supposedly more "compliant").

ar9av 3 years ago |

Maybe the future of the first line of healthcare will start to use AI with the assistance of a specialized nurse to follow your health overtime and only fall back to a real doctor when a real problem occurs.

This could elevate a lot of problems we have with our public healthcare(Canada).

tzm 3 years ago |

Love this and have had discussions w/ doctors and practitioners about this exact problem. I think extracting to FHIR is key, along with making it searchable as a longitudinal data store for a more robust personal record. Clear need. Would love to help.

stopachka 3 years ago |

This is really cool! What surprised you guys about fine-tuning whisper + gpt-3?

mraison 3 years ago | |

Domain specificity is a killer advantage when fine-tuning! For Whisper in particular, fine-tuning with actual medical speech gave us an immediate strong advantage over competing APIs.

phren0logy 3 years ago |

Side-note: your email field to sign up for your newsletter is very wonky with both Safari and Firefox (at least for me). It only lets me type one letter at a time, for some reason.

liautaud 3 years ago | |

Thanks for the report, it should be fixed now.

mitchellpkt 3 years ago |

Just read the blog post, very interesting. I thought sections about changing the temperature and repetition penalty were particularly neat, thanks for including examples.

Banjor 3 years ago |

One thing that a lot of folks still don't get is that some problems are bad, while others are not-so bad.

SkyPuncher 3 years ago |

Pretext: My wife is a psychiatrist. Via her, I'm am close friends with many doctors. Also, I know it's super easy to be critical, but I'm going to save you an insane amount of effort and heartache.

This is actually a solved problem in medicine, already. Maybe not solved, but absolutely "good enough". M-Modal, Dragon, and Telephone services (yes, call someone and have them translate) all exist. They all translate notes well enough. For most doctors, it's a huge improvement over typing.

The problem is primarily access. Most institutions simply do not want to pay for this. They don't view it as necessary. Further complicating the issue. Unless a physician is in private practice, they have little to no control over their ability to use a tool like this. HIPAA is far too high of a barrier.

You're trying to implement a technical solution to what's ultimately a politics/human problem.

----

I've spent a bunch of time in the space, both as a MedSpouse and the CTO of a HealthTech company. I wish you the best of luck as I'd love to see life be easier for my wife, but this is _extremely_ far from being MVP viable.

* HIPAA - You absolutely, 100%, without a doubt need to be HIPAA compliant. You likely want to be HITRUST compliant as well. You're very, very clearly a Business Associate (HIPAA term) and processing PHI. Without that, you're dead in the water.

* Chrome extension - This is not going to cut it. At baseline, it requires copy and pasting between Chrome and the EHR system. Bad UX.

    * More tactically, most EHR access occurs via remote access, like Citrix. Often from a thin-terminal on the hospital floor or at a med station. These are locked down environments that have a high likely hood of not having Chrome installed _and_ not being allow to install _any_ software (including Chrome extensions).

* Doctors need real-time feedback and the ability to correct. Real-time feedback is critical because carrying a patient takes mental load. Part of the purpose of notes is to create a written record that "closes" a patient for the time being. An async service isn't really helpful since it generates an interrupt after the physician has already moved on.

* Accuracy is important. Accuracy is even more important with _extremely_ similar and _extremely_ uncommon words. For example, can this system differential between prednisone and prednilisone? In fact, can it even identify those words? Can it do it with an accent?

* FHIR is only as much of a standard as tables are to relational databases. The hard part is the thousands of custom fields that are different between health systems.

* Structured data really isn't used or useful in medicine. You might have a medication list that can be structured as data, but actual notes include a lot of prose. I believe part of this is necessary to capture the nuance/minutiae of some healthcare. Part of it stems from the lack of any standard.

* Again, this is all possible already. Tele-notes have been done since the introduction of EHR systems.

* Nobody is going to use a Chrome extension.

egorfine 3 years ago | |

Right now I am working on an MVP of a product just like this one for a private psychiatry practice.

> HIPAA - You absolutely, 100%, without a doubt need to be HIPAA compliant

This is incorrect: kindly reminder that HIPAA is a specific law applicable only to a single particular country. The vast majority of the world does not need to be compliant to it. Although similar laws exists elsewhere, they are not at all insurmountable.

> More tactically, most EHR access occurs via remote access

I have never seen a system where EHR access is used via remote desktop. (Although I admit that would be a good security measure). But then again, I'm not in the US.

> can this system differential between prednisone and prednilisone

Whisper is nothing short of magic in this regard. Yes it can - out of the box. Even more: it can easily differentiate lots of medical terms in languages other than english. Even more: it can easily parse compound terms like "hypothalamic–pituitary–gonadal axis" in other languages where first two words sounds very very similar.

I reckon that differentiating prednisone and prednisolone in doctor's writing is a harder problem than differentiating those from voice.

Having said that, your concerns and remarks are very valid. This is why we are not focusing on replacing written records/EHR fill out. There are other ways AI and large language models could deliver help for psych practice and the patient. And safer.

ouid 3 years ago |

it sounds like you're removing a checklist from doctors.

BCM43 3 years ago |

Will you sign a BAA for HIPAA covered entities?

doctoring 3 years ago | |

They list it under their "legal documents" page: https://www.nabla.com/docs/business-associate-agreement

ceejayoz 3 years ago | | |

If I were a provider, I'd probably want to see their BSA with OpenAI.

bitshiftfaced 3 years ago | | |

They're saying that they will keep the information private, but I'm wondering about whether GPT will uphold this standard as well.

ar7hur 3 years ago | |

Yes absolutely.

rubenstern 3 years ago |

Looks very interesting! Are you recruiting ?

ar7hur 3 years ago | |

Yes we are! https://www.nabla.com/careers/

kgiddens1 3 years ago |

Congrats on the launch Nabla!

mjdowney 3 years ago |

That's super cool!

futureisvintage 3 years ago |

Amazing idea!!

avipars 3 years ago |

how can this be compliant with HIPAA?

dragonwriter 3 years ago | |

It can be HIPAA compliant by every company in the chain of use of the data signing a BAA with every one of the companies they deal directly with that handles PHI, and then following the terms of HIPAA in dealing with the data.

ismosoft 3 years ago |

this is amazing. great great

ismosoft 3 years ago |

amazing. you're ROCKKKKKKKKKK