I, and my advisor's lab[1] work in this area. I'm going to focus on the technical aspects of evidence synthesis, as opposed to the business aspects.

There's a difference between symptoms, diagnosis, treatment options, evidence for those, and the audiences for whom these are written. WebMD and friends target a broader market, scientific studies target...scientists and doctors.

I think the hard parts of building a better WebMD (along the article's lines) are:

  - screening articles for relevance. This is more than mere search or finding potentially relevant articles, but also making a decision to include them
  - extracting _structured_ information from the articles. Frequently we talk about Populations/Problems, Interventions, Comparators (interventions), and (medical) Outcomes, collectively PICOs. Extraction of each component is easy. Assembling them is surprisingly hard. Finding equivalencies between them within documents is surprisingly hard. Finding facets or different parts of a treatment (how do you handle combined treatments for a single study? how about when some studies use them and some don't?)
  - establishing equivalencies for your evidence synthesis between documents is even harder: do you care about dosage? combination with other treatments? what happens when a slightly different formulation of a treatment gets used, or a treatment is administered poorly? I don't know of anyone within scholarly document processing community working on analysis of medical methods (possibly ignorance on my part!).

It would be nice if trial preregistration had all these details, but not all trials are preregistered, nor all outcomes published, the aim of the trial can shift, and there's a large pile of literature where this information isn't available.

I think real time updates are the smallest problem among these: solving extraction (including the big structured objects) mostly solves this; the statistics of a meta-analysis are not complicated. To be clear, this is still an issue. I have a figure[2] highlighting the lag.

Presenting the evidence in a digestible, meaningful way, seems like a hard HCI problem to do right, and easy to do poorly. Merely giving PICOs (structured!) and findings is easy and a bad UX for the non-technical, a narrative summary is interesting to provide and hard to do well, and a subgroup analysis can suffer problems from differing group sizes and effectiveness findings (tailoring is tricky).

There are several organizations[3] working on these sorts of problems.

[1] https://www.byronwallace.com/

[2] https://www.semanticscholar.org/paper/MS%CB%862%3A-Multi-Doc...

[3] https://community.cochrane.org/organizational-info/people/st...

user:	jaydepun
created:	October 8, 2021
karma:	2
about:	PhD Student in Computer Science studying machine learning and model interpretability, in the context of natural language processing over medical- (and other) domain documents.