By: Sam Finlayson, MD, PhD and Travis Zack, MD, PhD
Today, we are releasing EvidenceGrade, a new feature on OpenEvidence that provides a real-time assessment of the certainty of evidence behind each answer.
One of our most requested features is a systematic assessment of the strength of the evidence surfaced in our answers. “How strong is the evidence in the literature backing this recommendation by OE?” Done manually, grading the strength of medical evidence is intensive and challenging work – formal appraisals typically involve expert panels, carefully framed clinical questions, and substantial time. As a result, formal evidence appraisals are only performed for a fraction of the questions clinicians ask in practice. In their absence, individual clinicians are forced to informally approximate this process under the significant time and resource constraints imposed by their daily practice. EvidenceGrade is our first attempt to bring the transparency and structure of explicit evidence grading to our platform without sacrificing the speed clinicians depend on at the point of care.
EvidenceGrade combines the key principles of the GRADE framework — the methodology behind Cochrane, WHO, and most major clinical guidelines — with the unique strengths of OpenEvidence, the most trusted clinical decision support platform. In this post, we share our approach, the compromises required to grade evidence without delaying answers, and how clinicians using OpenEvidence can help us improve it.
Grading evidence in real time runs into several constraints at once. The most obvious is speed. A formal GRADE assessment is deliberative: a panel frames the question, appraises each study, and reaches consensus over weeks or months. A clinician working through a question in practice has far less time, and cannot realistically be expected either to wait for lengthy processes to run or to engage in the exhaustive back and forth needed to fully scope the question and its search criteria.
Time and use-case constraints also change the nature of the retrieval itself, and therefore the evidence body we are grading. In a real-time setting, clinicians typically do not want every paper that could bear on a question, but a well-curated set of the most relevant ones. The retrieval system powering OpenEvidence is optimized for exactly this: surfacing the most relevant, informative, and complementary papers and combining them into a set that answers the question completely. This differs from the exhaustive predefined search behind a systematic review. Retrieval is also inevitably shaped by the clinical questions themselves, which are rarely as neatly scoped as the questions a systematic review is built around, but rather range from broad and general to narrow and highly complicated. The evidence that comes back spans the full range of source types as well, from practice guidelines and randomized trials to observational studies, case reports, and preclinical work. EvidenceGrade needs to operate with all this messiness in mind.
Finally, "best evidence" is not a fixed ladder but varies greatly based on question and topic. The best evidence for therapeutic interventions comes from well powered, randomized trials, while prospective cohorts represent the best design for questions around prognosis, a diagnostic accuracy study for diagnosis etcSometimes randomization or even enrollment for a given question is neither feasible nor ethical. The nature of the sort of evidence that can be practically achievable even under best case scenarios also varies by field. Rare disease genetics – one of our own specialties – will never run a trial that reaches the sizes seen routinely in adult cardiology, and indeed for many rare conditions, a seemingly small study may include many or all patients ever diagnosed. A scheme that simply rewarded large randomized trials would penalize whole areas of medicine for limits no researcher can remove. EvidenceGrade therefore grounds itself relative to the best achievable study design for the question at hand, while still being forthright about key limitations. This will at times produce gentler grading than we suspect a formal Cochrane review would give the same question, but it reflects our attempt to align EvidenceGrade with the practical realities of day-to-day clinical medicine.
Our system produces a single letter grade (A through D, with +/− modifiers, plus U when a grade cannot be assigned) reflecting the strength of the best evidence available for a specific answer.
Before grading, we first assess whether the question is amenable to a single evidence grade. This step filters out simple definitional lookups, queries focused on tasks like summarization or clinical reasoning, and complex case discussions that don't boil down neatly to a clear evidentiary question.
When a question is gradeable, the system works in two phases. In the first phase, each retrieved paper is scored individually on three axes. In the second phase, the body of evidence as a whole is scored using a GRADE-inspired approach, setting a ceiling from the strongest relevant papers and then applying upgrades and downgrades.
Phase 1: Per-paper scoring
After OpenEvidence gathers the relevant literature that will be used to answer the question, each retrieved paper is scored on three axes:
Phase 2: Body-level grading
After per-paper scoring, the system grades the body of evidence as a whole:
While we have worked hard to make this approach rigorous, EvidenceGrade is not a substitute for a full systematic review or meta-analysis. We see it as a complement to traditional evidence synthesis, and we are excited to continue partnering with medical societies on innovative approaches to keeping clinical guidelines current.
The final grades are intended to reflect the following principles:
Grade A — Strong Evidence. Supported by the best-achievable study designs for the question — RCTs or rigorous SRs, well-conducted prospective cohorts or registries, or current evidence-based guidelines with strong recommendations. Findings are precise, consistent, and directly applicable. Further research is least likely to change the conclusion.
Grade B — Moderate Evidence. Supported by appropriate study designs with notable limitations on one or more axes — modest sample sizes, surrogate outcomes, observational evidence applied to a therapy question, or moderate imprecision. Future findings could meaningfully refine this conclusion.
Grade C — Limited Evidence. Rests on weaker designs (narrative reviews, expert opinion) or stronger designs with serious limitations across multiple axes. Future findings could materially change this conclusion.
Grade D — Minimal Evidence. Rests on case reports, case series, preclinical or animal data, or mechanistic extrapolation. Best understood as a directional signal.
Grade U — Unable to Grade. Available evidence is insufficient to judge. Distinct from D, which applies when relevant but very weak evidence exists. This is also distinct from questions to OpenEvidence that are not amenable to evidence based grading (e.g. factual questions like “What is the dose for Tucatinib in metastatic breast cancer?” or non-evidence based questions like “Write me a prior authorization letter for Tucatinib”)
+/− Modifiers. Indicate the evidence sits near the boundary between two grades, tied to a specific reason on one of the three axes (Quality, Certainty, or Relevance).
We randomly sampled ~4500 user questions from OpenEvidence. Overall, the average EvidenceGrade in this sample falls just above the cutoff for a B, with sizable variation between specialties. Matching our own intuition around modern medicine, fields whose questions most frequently mapped onto strong evidence bases including oncology, pharmacy, endocrinology, cardiology, and general medicine; procedural specialties and other fields like geriatrics, genetics, and dermatology tended to map onto weaker strength of evidence.
Diving in a bit deeper, we can layer on additional information about how often different specialties questions activate the grading system at all, and how grade distributions change when specialties are asking about their own specialty (in-domain questions) vs when they are asking about other fields (general). Again unsurprisingly to us, specialists asking about their own fields of practice tend to drive at topics with less established evidence bases, resulting in lower grades:
This is only a first look at how evidence strength varies across specialties, and we are excited to further explore this landscape and use EvidenceGrade to identify unmet needs in the published literature.
Our design of EvidenceGrade is informed by our own clinical practice and how we intend to use the feature.
One of us, Sam, is a pediatrician and medical geneticist. Wearing my pediatrics hat, I'm acutely aware in day-to-day practice of how much of our clinical evidence relies on extrapolation. More than 80% of pediatric drug and device approvals are based on extrapolation from adult data, a pattern that holds across many other aspects of daily practice and is borne out in the chart above. With features like EvidenceGrade, I hope we at OpenEvidence can help users see more clearly how precisely the available evidence maps onto the question at hand, and use that clarity to adapt it when needed, delivering personalized care to the patient in front of them. As a medical geneticist, my specialty practice sits even closer to the frontier of the literature: roughly half of the 6,000+ genetic disorders we treat were discovered in the past 15 years. Tools like EvidenceGrade help us respect the limitations of the evidence we're working with, while calibrating its grades against the best achievable standard for the field—recognizing that the small, heterogeneous patient groups here constrain any study's size and rigor, no matter how well designed.
The other, Travis, is an oncologist. Where Sam's world is often defined by evidence deserts — rare conditions where extrapolation is unavoidable — mine sits at the opposite extreme. Oncology attracts some of the highest research funding, and the resulting volume of literature is a genuine luxury. But abundance brings its own hazards. The first is that a well-traveled main road, with its clear signposts of practice-changing trials, can lull you into a false sense of security that every decision rests on equally strong evidence. In that world, you often don't realize the moment you've stepped off the beaten path onto less stable ground. The second is the mirror image of the evidence desert: not a scarcity requiring extrapolation, but an evidence jungle, where a question has been studied many times over and the studies point in conflicting directions, each suggesting something a little different. In both situations, a gauge on the strength of the underlying evidence — and on how much the sources actually agree — is exactly the kind of orientation I want at the point of care.
EvidenceGrade is now live. We view this release as a starting point. Evidence grading at this scale and speed is inherently imperfect, and we expect the system to make mistakes. Some grades will feel too generous, others too conservative, and some questions will surface edge cases we haven't yet anticipated.
We invite you to provide feedback on this system. Every evidence grade now includes the option to upvote or downvote the rating. These votes go directly into our evaluation pipeline and help us identify where the system is falling short. Over time, this feedback will allow us to calibrate the system against the judgment of the clinical community it is designed to serve.