Validating VRS Variant Resolution Against Real AACR GENIE Data

Last month I wrote up the variant-identity resolver we built on GA4GH VRS: computed, sequence-derived identifiers instead of HGVS string matching, tested against a corpus of 185 real variants pulled from 95 manufacturer-published sample reports. That corpus is real, but it's small, and it's curated in the sense that a vendor sample report only shows variants worth showing. I wanted to know what happens when the same resolver meets something bigger and messier: real-world data at a scale no sample-report corpus reaches, with no editorial hand picking which variants make the cut.

AACR Project GENIE is that dataset. It's a real-world, de-identified genomic and clinical registry pooled across contributing cancer centers and released publicly on Synapse. The current public release, v20.0-public, carries 289,869 samples across 242,866 patients and a mutation-call file with 3,191,158 real variant rows. We pulled it, ran it through the resolver, and I'm writing up what we found the same way we wrote up the first corpus: the real numbers, not the ones that would look best in a deck.

A reproducible sample, not a cherry-picked one

We drew a reservoir sample of 2,000 variants from the full 3.19-million-row mutation file, seeded so the exact sample is replayable, and ran every one through the resolver in vrs_only mode against the same pinned reference data as the first corpus run: ga4gh.vrs 2.3.3, biocommons.seqrepo 0.6.11, the 2024-12-20 SeqRepo snapshot. The one change was scope: the first corpus run's MANE Select disambiguation table was scoped to the 82 genes that actually showed up in those 95 reports, which would have been a quiet form of selection bias against a genuinely random real-world sample. For this run we built the full-genome table, all 19,363 MANE Select genes, from the same pinned NCBI MANE release.

1,804 of 2,000 (90.2%) resolved cleanly to a computed VRS identifier. That's higher than the 68.6% we measured on the vendor-report corpus, and it has a real explanation rather than being a number to just quote.

GENIE's coding and protein HGVS fields come out of a single VEP-based annotation pipeline, applied the same way across every contributing center, and an explicit RefSeq transcript accession is present on about 89% of real rows. The vendor corpus, by contrast, drew from nine different manufacturers' PDF and free-text formats, and 43.8% of its extracted coding-HGVS values carried no stated transcript at all, which forces the resolver to fall back on MANE Select inference far more often, with everything that entails. Uniform machine annotation resolves more cleanly than heterogeneous human-authored report text. That's not a surprising result once you say it out loud, but it's a different claim than "the resolver works better on GENIE," and worth being precise about.

The remaining 9.8% split across the same failure-mode categories the first corpus surfaced, but not in the same proportions. 127 of the 196 unresolved cases, about two-thirds of all failures, were intronic positions, a real and documented limitation of VRS and vrs-python: a VRS Allele is defined relative to a transcript's mature or coding sequence, not its introns. That's a much bigger share than the vendor corpus showed, and the reason is the data itself: GENIE's mutation file is unfiltered, capturing every sequencing-pipeline call, while a vendor's clinical PDF report only ever surfaces the subset a lab considers actionable enough to print. An unbiased, at-scale sample runs into intronic and splice-adjacent calls a curated report simply never contains.

The four variant-class checks, on real cases this time

The narrower and more important question is whether the resolver's central claim holds on real-world cases specifically. We pulled every real GENIE point-mutation call for four genes central to precision oncology (EGFR, BRAF, KRAS, NTRK1) and every real NTRK1 structural-variant call, and checked four pairs:

• EGFR L858R vs. T790M. The activating mutation and the resistance mutation that emerges after first-generation TKI therapy. Different clinical meaning, same gene, adjacent in the protein.
• BRAF V600E vs. a Class III residue. V600 is the well-known hotspot. Class III mutations (RAS-dependent, kinase-impaired, per Yao et al., Cancer Cell 2017) are a mechanistically distinct category that needs different treatment logic entirely, and we cross-referenced the 24 specific Class II/III residues from that paper against real GENIE BRAF calls: all 24 turned up in the data, with real case counts from single digits up into the hundreds.
• KRAS G12C vs. a non-G12C hotspot (G12D). G12C has an FDA-approved targeted therapy. The other common KRAS hotspots don't, or not the same one.
• NTRK1 gene fusion vs. NTRK1 point mutation. A fusion isn't a single sequence alteration, so it can't be a VRS Allele by construction; the resolver has to route it to the typed graph instead of forcing it into the wrong model.

All three point-mutation pairs resolved to genuinely distinct VRS identifiers on real GENIE-sourced cases, not the synthetic fixtures we used to validate this logic originally. And all 371 real NTRK1 fusion calls in the GENIE release correctly routed to the typed-graph path rather than being coerced into an Allele identifier that would have been meaningless for a fusion. That's the full population, not a sample; the fusion-type guard fires before any resolution attempt, so checking every real case cost nothing extra.

What this doesn't show

GENIE arrives as pre-extracted, structured variant calls: gene symbol, HGVS strings, transcript accession, sample metadata, already parsed and normalized by the consortium's own pipeline before it ever reaches us. None of this exercises the part of UNMIRI's stack that has to read an actual Foundation Medicine, Tempus, Caris, or Guardant PDF and pull a variant table out of inconsistent formatting, vendor-specific shorthand, and OCR noise. That's a real and different problem, and the only thing that's actually tested it so far is the 95-report vendor corpus from last month's post. A strong result on GENIE is not evidence about parsing accuracy on messy real-world report formats, and we're not going to let one get read as the other.

What this run does add: the same resolver, tested against roughly eleven times as many real-world variant calls as the first corpus, pulled from a real GENIE mutation file that runs into the millions of rows, on data nobody curated for us in advance, with the same falsification discipline. The four variant-class distinctions the whole identity layer depends on held on real cases. The failure modes are named, counted, and explained rather than absorbed into a vague miss rate. That's what "tested," not just "built," is supposed to mean.

Umair Khan

Founder and CTO, UNMIRI

Building UNMIRI, a precision oncology infrastructure company with four product surfaces: cross-vendor NGS interpretation, genomics-aware decision support, oncology literature intelligence, and a free cross-vendor unification tool for clinicians. Writing here on architecture, clinical data, and HIPAA-ready AI.

See the NGS Interpretation API →

Advisors: Somdutta Saha, PhD, advises on technical architecture and bioinformatics methods as Technical and Scientific Advisor. On the clinical side, UNMIRI is in ongoing conversations with multiple board-certified pathologists about advisory roles. Public introductions land on the About page once each engagement is formalized and the advisor approves being named.