Show HN: Medical LLM API on par with Google Med-PaLM 2. 92% USMLE accuracy

Show HN: Medical LLM API on par with Google Med-PaLM 2. 92% USMLE accuracy(medisearch.io)

17 points by mpmisko 3 years ago | 13 comments

Hello HN!

I’d like to share a medical question-answering API which has state-of-the-art performance on the USMLE self-assessment exam. You can try out the MediSearch API in 10 seconds in this Colab [https://tinyurl.com/medisearch-colab], or test the search engine live at MediSearch [https://medisearch.io]. See our API page [https://medisearch.io/api] to apply for access. Some technical highlights:

1. State-of-the-art accuracy of 92% on the USMLE self-assessment exam [https://tinyurl.com/medisearch-eval]. 2. Ranked 2nd on MedQA benchmark, closely tailing Google's Med-PaLM 2 (within ~2%). 3. The API provides article references as an output.

We’d love to hear feedback from this community! We’ve seen people use the MediSearch API to implement a chat bot or search field within their app.

Please note that MediSearch does not have a 100% accuracy. Therefore, a human professional should always make a final decision on the information suggested by MediSearch.

You can use the MediSearch API similarly to the OpenAI API [https://pypi.org/project/medisearch-client/]. If you’d like access to the MediSearch API, please email michal@medisearch.io or visit our API page [https://medisearch.io/api]. Please specify your use case in the application.

gardenfelder 3 years ago |

I asked it a simple question: "what is known about PSC?" and it returned two citations.

Interesting.

Right now, nothing is known about the business model behind the product. What are the plans for it?

mpmisko 3 years ago | |

Thanks for giving it a go! We will charge for the API and we may have a pro subscription with some nice features :).

mackopes 3 years ago |

Having an API with medical information is amazing!

Just out of curiosity, is a React client or even a javascript library on your roadmap?

mpmisko 3 years ago | |

Hey, a React client is definitely on our roadmap! The demo (https://tinyurl.com/medisearch-docs) in our docs is written in JS, so it should be useful out of the gate! A proper client is coming in the next few weeks.

zopper 3 years ago |

This looks really cool. Does your API also support follow up questions and a longer conversations?

mpmisko 3 years ago | |

Glad that you like it! We do support follow-up questions. Implementing a chat-style interaction should be very straightforward :) (see our Python client for examples https://pypi.org/project/medisearch-client/).

hzoli42 3 years ago |

Wow, really cool to see citation support! What are the different sources that are cited?

mpmisko 3 years ago | |

We primarily cite medical scientific papers and health guidelines. We occasionally cite health blogs (such as healthline.com), but this is mostly for simple health questions (e.g., "tell me some nutrition tips").

brandonb 3 years ago |

Awesome! Can you talk a bit about how you trained or fine-tuned MediSearch's LLM?

eoravkin 3 years ago | |

Thanks! Glad that you like it:)

We haven't trained or fine-tuned on neither of USMLE or MedQA. We use several LLMs in our system, and honestly I don't think there is any single thing that I can point to which is different to how other people do it. I guess it's a combination of many small improvements.

Sorry for the vague answer. We are thinking about publishing a report at some point.

eoravkin 3 years ago |

We'd be interested to hear examples or ideas of how this could be useful for you!

dredev 3 years ago |

This is amazing! It almost never hallucinates. How did you guys accomplish that?

eoravkin 3 years ago | |

Thanks a lot! We put a lot of energy into solving hallucination. The tldr is that we have a eval set to test hallucination and tweak stuff to optimize for performance on this eval set.