How the suite runs, and what it depends on

The suite has five stages. On every proposed change, CI runs all of them, together with a build of the app and its end-to-end browser tests, and keeps the report as an artifact of the run. Anyone can run the same five stages with one command, npm run verify, which takes about ten minutes and stops with an error if any check fails. The report is written to docs/verification/; this page shows the copy saved with the code, updated by hand from a passing run. Every CI run on master keeps its own report, as a download named verification-report on its page on GitHub.

The warehouse is this tool's own database: a copy of the public studies in cBioPortal, loaded into ClickHouse, which is the database every analysis in the app runs against. The suite only reads from it. In CI, the read-only login is supplied as a secret.

The studies the suite uses, chosen because each exercises something that is easy to get wrong:

One known source of false alarms. The cBioPortal comparison needs both sides to hold the same release of each study. If cBioPortal updates a study before the warehouse is reloaded, the comparison fails even though nothing in the code is wrong. That failure is intended: the suite reports the difference rather than tolerating it, and someone has to check whether the cause is the data or the code.

1 Are the patient and mutation counts the same as cBioPortal's?

cBioPortal has been used and checked by the cancer genomics community for over a decade, and it was built by a different team on separate code. We ask both tools the same questions about the same studies and require exactly the same answer. A count either matches or it does not, so no tolerance is allowed. The questions cover cohort sizes, how often each top gene is mutated, amplified, deleted or fused (both the number of altered samples and the number of samples sequenced for that gene), clinical counts, and filtered cohorts such as “patients with a TP53 mutation”.

Last run October 23, 2026 against https://www.cbioportal.org (public API). Frequencies are shown as altered / sequenced for that gene.

2 Does a cohort contain the patients its definition says?

No reference program can tell us who belongs in “never-smokers with a TP53 mutation who received immunotherapy”. What we can test is logic that must hold whatever the right answer is. For example, “A and B” must be exactly the patients found in both A and B, and a cohort plus its opposite must make up the whole study. The tool computes each side separately on real data, and the check compares the actual patient lists, not just the totals.

For the simpler conditions (a mutation or copy-number change in a gene, a clinical value or number, a list of patients) there is a second way to get the answer: fetch the study's raw records and work out each patient's membership by hand, in a separate program written from the rules in the documentation. A random generator then writes a hundred definitions combining those conditions with and, or and not, and each must pick exactly the same patients both ways.

• ✓each test condition picks out a real, partial group of patientsmsk_chord_2024 · 13,105, 12,168 and 3,341 of 24,950 patientsbrca_tcga · 317, 229 and 513 of 1,101 patients
• ✓"A and B" is exactly the patients in both A and Bmsk_chord_2024 · A and B: 6,241 patients, identicalbrca_tcga · A and B: 46 patients, identical
• ✓the order conditions are written in does not change the cohortmsk_chord_2024 · B and A vs A and B: 6,241 patients, identicalbrca_tcga · B and A vs A and B: 46 patients, identical
• ✓adding a condition can only narrow a cohort, never grow itmsk_chord_2024 · A 13,105 ⊇ A and B 6,241 ⊇ A and B and C 439brca_tcga · A 317 ⊇ A and B 46 ⊇ A and B and C 22
• ✓three conditions combine the same way two domsk_chord_2024 · A and B and C: 439 patients, identicalbrca_tcga · A and B and C: 22 patients, identical
• ✓"A or B" is exactly the patients in eithermsk_chord_2024 · A or B: 19,032 patients, identicalbrca_tcga · A or B: 500 patients, identical
• ✓counts add up: |A or B| = |A| + |B| − |A and B|msk_chord_2024 · 19,032 = 13,105 + 12,168 − 6,241brca_tcga · 500 = 317 + 229 − 46
• ✓"not A" is every other patient in the studymsk_chord_2024 · not A: 11,845 patients, identicalbrca_tcga · not A: 784 patients, identical
• ✓A and "not A" share nobody and together are the whole studymsk_chord_2024 · 13,105 + 11,845 = 24,950brca_tcga · 317 + 784 = 1,101
• ✓"not (A or B)" equals "not A and not B" (De Morgan's law)msk_chord_2024 · not (A or B): 5,918 patients, identicalbrca_tcga · not (A or B): 601 patients, identical
• ✓KRAS G12C patients are all among KRAS-mutated patientsmsk_chord_2024 · 1,113 ⊆ 7,115
• ✓KRAS driver mutations are a subset of all KRAS mutationsmsk_chord_2024 · 7,082 ⊆ 7,115
• ✓"KRAS: MUT AMP" is exactly mutated or amplifiedmsk_chord_2024 · MUT AMP: 7,287 patients, identical
• ✓PIK3CA H1047R patients are all among PIK3CA-mutated patientsbrca_tcga · 126 ⊆ 317
• ✓PIK3CA driver mutations are a subset of all PIK3CA mutationsbrca_tcga · 307 ⊆ 317
• ✓"PIK3CA: MUT AMP" is exactly mutated or amplifiedbrca_tcga · MUT AMP: 351 patients, identical
• ✓each patient has exactly one GENDER: the values share nobody and cover everyone recordedmsk_chord_2024 · 13,874 + 11,073 + 3 = 24,950; recorded 24,950; overlap 0
• ✓CURRENT_AGE_DEID below 65 and at-or-above 65 share nobody and cover everyone with a valuemsk_chord_2024 · 10,654 + 14,293 = 24,947
• ✓the engine reports that the split's children add up to the parentmsk_chord_2024 · 0 + 916 + 1,278 + 5,615 = 7,809 of 7,809msk_chord_2024 · 0 + 1,908 + 0 + 1,543 + 1,917 = 5,368 of 5,368
• ✓every arm holds exactly the patients a by-hand first-match-wins pass puts theremsk_chord_2024 · G12C 916/916, other KRAS 1278/1278, rest 5615/5615msk_chord_2024 · PIK3CA 1908/1908, TP53 1543/1543, rest 1917/1917
• ✓profiled for MUTATION in every one of KRAS, NRAS, BRAF, NTRK3, SOX9 is the intersection of each gene's profiled patientsmsk_chord_2024 · gene list: 24,948 patients, identical · each gene: 24,950 / 24,950 / 24,950 / 24,950 / 24,948
• ✓profiled for CNA in every one of KRAS, TP53, PIK3CA is the intersection of each gene's profiled patientsbrca_tcga · gene list: 1,080 patients, identical · each gene: 1,080 / 1,080 / 1,080
• ✓everyone who reached a later line of therapy also reached every earlier onemsk_chord_2024 · line 1: 21,444 ⊇ line 2: 7,824 ⊇ line 3: 3,088 ⊇ line 4: 1,412
• ✓discovery and validation halves share nobody, cover everyone, and are close to 50/50msk_chord_2024 · 12,514 + 12,436 = 24,950 (50.2% discovery)
• ✓the halves are the same every time they are drawnmsk_chord_2024 · discovery, twice: 12,514 patients, identical

Last run October 23, 2026.

3 Are the statistics computed correctly?

Every statistical method the tool reports is checked against the library statisticians already use for it: lifelines, statsmodels and scipy, which are widely used in published analyses. Both are given the same datasets (135 test cases in all). The test data is chosen to be awkward: tied death times, empty groups, very small and very large counts, patients who enter the study late. Our numbers must match the library's to the precision shown below, and the check runs before every release.

4 Can I check a specific result myself?

Yes. Every result in the analysis app has a How this was computed panel. It lists each step from the database rows to the number on screen, the exact database queries that ran, and a short Python script that recomputes the result with the standard libraries. You can send that script to a statistician or run it yourself.

The scripts are checked too. Before every release, the analyses in the app's Examples gallery (99 of them) are run, and every script must reproduce the numbers the app showed. If the code changes how a result is computed but the script is not updated to match, the check fails.

5 Does it find what published papers found?

We took 11 published cancer genomics papers whose data is in cBioPortal and re-derived their main findings from scratch with this tool. 5 reproduce fully, 5 partly, and the rest are blocked by data the study does not include. Each one opens in the app as a runnable analysis, with the paper's claim written beside our result. One reproduction found a bug in this tool: a sarcoma study with no copy-number data reported MDM2 amplification as 0% instead of saying the data was missing. That bug has been fixed.

See the paper reproductions →

6 Would these checks notice a mistake?

A check that always passes proves nothing. To show these checks would notice a real error, we planted 18 realistic bugs in the code, one at a time, and reran the checks after each. These are bugs that produce believable but wrong numbers rather than crashes. 18 of 18 were caught. One is a real bug this project once shipped: brca_tcga MYC amplification read 10.6% where cBioPortal says 21.2%. The cBioPortal comparison now catches it.

Last run October 23, 2026.

Choices a statistician should review

Where there is more than one defensible method, this is the one the tool uses. Each is pinned by the checks in section 3.

Tied death times in Cox regression use Efron's method, not Breslow's.

Efron is the default in R's survival package and in lifelines, and is more accurate with many ties (common when follow-up is recorded in whole months).

Fisher's exact test is two-sided, summing every table no more likely than the observed one.

This is scipy's and R's definition. A one-sided test would halve many p-values.

The χ² test has no continuity correction; the Cochran–Mantel–Haenszel test has one.

Each matches the reference default (scipy without correction, R's mantelhaen.test with correction).

Group means are compared with Welch's test, which does not assume equal variances.

Clinical values rarely have equal spread across cohorts; Student's t would be wrong when they do not.

Multiple-testing correction is Benjamini–Hochberg (false discovery rate), not Bonferroni.

The usual choice when screening many genes: it controls the share of false positives among the hits rather than the chance of any false positive at all.

Survival with delayed entry (left truncation) removes patients from the at-risk group until they were observable.

Patients sequenced years after diagnosis would otherwise make early survival look better than it was.

A gene's frequency divides by samples whose panel covers that gene, not by every sample.

A sample whose panel omits a gene is not profiled for it, so counting it as unaltered makes the gene look rarer. This matches cBioPortal.

Running the suite

With network access, the warehouse login in .env.local, and uv for Python:

npm run verify          # every stage below; writes the report to docs/verification/

npm test                # statistics (no database or network needed)
npm run check:portal    # counts against cBioPortal
npm run check:cohorts   # cohort logic
npm run check:methods   # every example's Python script reproduces its numbers
npm run check:mutants   # plant each bug, confirm a check catches it (about three minutes)

The reference answers for the statistics are generated by the Python scripts in python/golden/, with library versions pinned, so a statistician can regenerate them independently.