phenoforge.eval

Ontology-aware metrics and the benchmark harness, scoring retrieval and decomposition against the bundled curated cohorts as ground truth. See the README’s Validation and limitations section for what’s actually been measured.

phenoforge.eval.models

Pure result types for the eval harness. No DB dependency.

Gives metrics.py and harness.py a shared vocabulary of typed results without either depending on the other’s I/O — metrics.py stays a pure, DB-free module that property-based tests can target cheaply.

class phenoforge.eval.models.CohortEvalResult(**data)[source]

Bases: BaseModel

Per-cohort scoring result for one retrieval method against curated ground truth.

Variables:

  • cohort_id – OHDSI Phenotype Library cohort id.

  • cohort_name – Cohort display name — also the query fed to retrieval methods that generated predicted_codes.

  • method – Label for the method under test, e.g. "bm25", "dense", "hybrid", "expand_descendants".

  • predicted_codes – ICD-10-CM codes the method produced.

  • ground_truth_codes – ICD-10-CM codes from the curated cohort.

  • coverage – Ground-truth -> predicted direction score, [0, 1] — did the whole curated set get assembled.

  • over_inclusion_penalty – Predicted -> ground-truth direction score, [0, 1] — how much of what was predicted has no good match anywhere in the curated set.

  • hierarchical_score – Harmonic mean of coverage and 1 - over_inclusion_penalty; a single combined quality number.

  • unresolvable_ground_truth – Curated codes not found in the loaded vocabulary (a data-quality escape hatch, not expected to be nonempty in normal operation).

Parameters:
cohort_id: str
cohort_name: str
method: str
predicted_codes: list[str]
ground_truth_codes: list[str]
coverage: float
over_inclusion_penalty: float
hierarchical_score: float
unresolvable_ground_truth: list[str]
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class phenoforge.eval.models.DecompositionCaseResult(**data)[source]

Bases: BaseModel

One test case’s scores: retrieval alone vs. decomposition + retrieval.

Both scores are measured against the same cohort_id’s curated ground truth, isolating what the decompose step costs (or doesn’t) on top of retrieval quality that’s already measured separately by run_benchmark.

Variables:

  • cohort_id – Bundled OHDSI Phenotype Library cohort id used as ground truth.

  • cohort_name – Cohort display name.

  • population_description – The free-text input fed to decompose.

  • seed_terms – What decompose actually extracted.

  • baseline_hierarchical_score – Hybrid search’s score using the cohort’s own display name as the query directly — “if decomposition were skipped entirely.”

  • decomposed_hierarchical_score – Score from running the real decompose → per-term curated/generated resolution → assemble pipeline, starting from population_description.

  • gap – baseline_hierarchical_score - decomposed_hierarchical_score. Positive means decomposition cost accuracy relative to querying with the right term directly; negative means decomposition (e.g. splitting into multiple terms) did better than the single-query baseline.

Parameters:

  • cohort_id (str)

  • cohort_name (str)

  • population_description (str)

  • seed_terms (list[str])

  • baseline_hierarchical_score (float)

  • decomposed_hierarchical_score (float)

  • gap (float)

cohort_id: str
cohort_name: str
population_description: str
seed_terms: list[str]
baseline_hierarchical_score: float
decomposed_hierarchical_score: float
gap: float
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class phenoforge.eval.models.DecompositionReport(**data)[source]

Bases: BaseModel

Aggregate report across every decomposition test case.

Variables:

  • results – One result per test case.

  • mean_gap – Mean DecompositionCaseResult.gap across cases — decomposition’s average accuracy cost on this test set.

Parameters:
results: list[DecompositionCaseResult]
mean_gap: float
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class phenoforge.eval.models.BenchmarkReport(**data)[source]

Bases: BaseModel

Aggregate report across all cohorts and methods.

Variables:
Parameters:
results: list[CohortEvalResult]
mean_coverage_by_method: dict[str, float]
mean_over_inclusion_by_method: dict[str, float]
mean_hierarchical_score_by_method: dict[str, float]
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

phenoforge.eval.metrics

Ontology-aware scoring: tree distance, partial credit, coverage, over-inclusion.

Pure functions, no duckdb import anywhere in this file. Everything takes already-computed ancestor chains or scores as plain list[int] / float inputs, so the property-based tests in tests/eval/test_metrics.py never touch a database — chain construction lives in phenoforge.engine.expansion.ancestor_chain_ids(), and DB-backed orchestration lives in phenoforge.eval.harness.

Standard exact-match recall is the wrong metric for this task: retrieving E11.21 when the target is E11.9 is a near-miss under the same parent, while retrieving a circulatory-chapter code is a total miss — both score zero under exact match today. Every score here is instead derived from tree_distance(), so near-misses earn partial credit and total misses don’t.

phenoforge.eval.metrics.tree_distance(chain_a, chain_b)[source]

Distance between two concepts via their lowest common ancestor.

chain_a/chain_b are each [self_id, parent_id, ..., root_id], self-inclusive and nearest-first, as returned by phenoforge.engine.expansion.ancestor_chain_ids(). Walking chain_a outward from self, the first id also present in chain_b is the lowest common ancestor — the closest shared node, since chain_a is ordered nearest-to-farthest.

Parameters:

  • chain_a (list[int]) – Ancestor chain of the first concept.

  • chain_b (list[int]) – Ancestor chain of the second concept.

Returns:

Tree-edge distance (sum of each concept’s distance to the LCA), or None if the chains share no common ancestor (different chapters or vocabularies entirely — a true miss, not a near-miss).

Return type:

int | None

phenoforge.eval.metrics.distance_weighted_score(distance, max_distance=6)[source]

Convert a tree distance into partial credit in [0, 1].

distance=0 (exact match) scores 1.0. Score decays linearly to 0.0 at distance >= max_distance. None (no common ancestor — a total miss) scores 0.0. Linear decay is chosen over exponential for interpretability at prototype scale: a near-miss like E11.21 vs. E11.9 (distance 2, common parent E11) should land clearly above zero and clearly below 1.0, not be swamped by a steep curve.

Parameters:

  • distance (int | None) – Tree distance from tree_distance(), or None.

  • max_distance (int) – Distance at or beyond which score floors to 0.0.

Returns:

Score in [0.0, 1.0].

Return type:

float

phenoforge.eval.metrics.set_coverage(predicted_scores, n_ground_truth)[source]

Mean of each ground-truth code’s best-match score against predicted codes.

The caller has already computed one best distance_weighted_score() per ground-truth code (0.0 if nothing predicted is even a near-miss) — see phenoforge.eval.harness.match_scores().

Parameters:

  • predicted_scores (list[float]) – One best-match score per ground-truth code, each in [0, 1].

  • n_ground_truth (int) – Number of ground-truth codes; must equal len(predicted_scores).

Returns:

Mean coverage in [0, 1]. 1.0 (vacuously) if n_ground_truth == 0.

Return type:

float

Raises:

ValueError – If len(predicted_scores) != n_ground_truth.

phenoforge.eval.metrics.over_inclusion_penalty(predicted_scores, n_predicted)[source]

Fraction of predicted codes with no good match anywhere in ground truth.

The caller has already computed one best distance_weighted_score() per predicted code against ground truth. A predicted code close to something in ground truth — a legitimately-missing sibling, given the curated set is a lower bound (see phenoforge.engine.curated.resolve_curated_concepts()) — earns partial credit rather than being flatly penalized as a false positive.

Parameters:

  • predicted_scores (list[float]) – One best-match score per predicted code, each in [0, 1].

  • n_predicted (int) – Number of predicted codes; must equal len(predicted_scores).

Returns:

Penalty in [0, 1] — mean(1 - score) over predicted codes. 0.0 (vacuously) if n_predicted == 0.

Return type:

float

Raises:

ValueError – If len(predicted_scores) != n_predicted.

phenoforge.eval.metrics.hierarchical_score(coverage, over_inclusion_penalty)[source]

Harmonic mean of coverage and precision (1 - over_inclusion_penalty).

An F1-style single quality number: coverage plays the recall role (did we get the whole curated set), 1 - over_inclusion_penalty plays the precision role (is what we got actually relevant). Harmonic rather than arithmetic mean so a method can’t hide a bad score on one axis by being good on the other — cohort definitions fail on false positives too, not just missed codes.

Parameters:
Returns:

Harmonic mean in [0, 1]. 0.0 if both coverage and precision are 0.0.

Return type:

float

phenoforge.eval.harness

Benchmark harness: score generated retrieval against curated ground truth.

Orchestration layer with DB I/O — the counterpart to the pure functions in phenoforge.eval.metrics. For each bundled OHDSI Phenotype Library cohort, loads the curated concept set as ground truth (free — no hand-annotation required, since curated value sets are already the labels) and scores each retrieval method’s output against it.

phenoforge.eval.harness.match_scores(con, from_codes, to_codes)[source]

Best distance-weighted score from each of from_codes to any of to_codes.

For each code in from_codes, computes tree_distance() (via ancestor_chain_ids()) against every code in to_codes and keeps the best (lowest-distance) match, then applies distance_weighted_score(). Ancestor chains are fetched once per unique code and memoized for the call’s duration — cohorts run to the tens of codes, so simple O(n*m) with memoized lookups beats a precomputed global distance matrix at this scale.

Parameters:

  • con (DuckDBPyConnection) – Open connection to vocab.duckdb.

  • from_codes (list[str]) – Codes to score. Caller decides direction: ground truth codes here (matched against predicted) computes coverage; predicted codes here (matched against ground truth) computes over-inclusion.

  • to_codes (list[str]) – Codes to match against.

Returns:

One score per from_codes entry, same order. 0.0 per entry if to_codes is empty.

Return type:

list[float]

phenoforge.eval.harness.score_cohort(con, cohort_id, cohort_name, ground_truth_codes, predicted_codes, method)[source]

Score one method’s predictions for one cohort against curated ground truth.

Computes coverage (ground-truth -> predicted direction) and over-inclusion (predicted -> ground-truth direction) via two match_scores() calls, then set_coverage(), over_inclusion_penalty(), and hierarchical_score().

Parameters:

  • con (DuckDBPyConnection) – Open connection to vocab.duckdb.

  • cohort_id (str) – OHDSI Phenotype Library cohort id.

  • cohort_name (str) – Cohort display name, echoed into the result.

  • ground_truth_codes (list[str]) – ICD-10-CM codes from the curated concept set.

  • predicted_codes (list[str]) – ICD-10-CM codes from the method under test.

  • method (str) – Label for the method under test, e.g. "hybrid".

Returns:

The scored result.

Return type:

CohortEvalResult

phenoforge.eval.harness.run_benchmark(con, library_dir, cohort_ids, methods=None, dense=None, k=25, fanout_threshold=100, dense_weight=0.5)[source]

Run every requested method against every cohort, aggregate the results.

Per cohort id: loads curated ground truth via load_curated_concept_set(), generates predictions per requested method (cohort_name — the cohort’s own display name — is the query fed to "bm25"/"dense"/"hybrid"; "expand_descendants" seeds from the shortest ground-truth code, a deliberately weak baseline rather than a competitive method — the least-specific curated code is the natural entry point a human would expand from), scores each with score_cohort(), and aggregates.

Parameters:

  • con (DuckDBPyConnection) – Open connection to vocab.duckdb.

  • library_dir (Path) – Directory of fetched OHDSI Phenotype Library cohorts (scripts/fetch_phenotype_library.py output).

  • cohort_ids (list[str]) – Cohort ids to benchmark.

  • methods (list[str] | None) – Methods to run; defaults to ["bm25", "hybrid", "expand_descendants"]. "dense" only runs if dense is given; a requested method needing dense when none is given is skipped for that cohort, not errored — matching search_concepts’s existing graceful-degradation behavior.

  • dense (DenseRetriever | None) – Pre-built dense retriever, or None to skip dense-dependent methods. Tests inject one built with the fake embedder from tests/engine/fake_embedder.py so this function’s own tests never download a real model.

  • k (int) – Result count passed to each retrieval method.

  • fanout_threshold (int) – Passed through to curated ground-truth loading.

  • dense_weight (float) – Passed through to hybrid_search() for the "hybrid" method — dense’s share of the fused RRF score. See scripts/sweep_hybrid_weights.py for sweeping this against curated ground truth.

Returns:

Full benchmark report across all cohorts and methods.

Return type:

BenchmarkReport

phenoforge.eval.decomposition

Evaluate the agent’s decompose step, isolated from retrieval quality.

run_benchmark (harness.py) measures retrieval given an already-correct search term. Nothing measures the step before that: given a free-text population description, does decompose (a real Claude call) extract the right seed terms? Before this module, that had been exercised exactly once, by hand, in notebooks/agent_walkthrough.ipynb.

Scoring term strings directly would need a new, somewhat arbitrary similarity metric and a hand-labeled “correct terms” dataset — and would measure something beside the point, since nobody cares about the terms themselves, only the codes that come out the other end. Instead, this reuses the bundled cohorts’ existing ground truth and score_cohort unchanged: for each test case, compare “hybrid search using the cohort’s own name directly” (decomposition skipped entirely) against “the real decompose step, then per-term curated/generated resolution” (mirroring agent/nodes.py’s check_curated/generate, minus the human confirmation gate — this measures decomposition and retrieval together, not human review). The gap between those two scores, against the same ground truth, is what decomposition costs — or saves, if splitting a description into multiple terms does better than one query.

class phenoforge.eval.decomposition.DecompositionTestCase(**data)[source]

Bases: BaseModel

A free-text population description mapped to the cohort it should resolve to.

Variables:

  • population_description – Phrased like a real study population description, not copied from the cohort’s own bundled display name — otherwise decomposition would be trivially unnecessary to get right.

  • cohort_id – Bundled OHDSI Phenotype Library cohort id this description should resolve to; its curated concept set is ground truth.

  • source – Where the phrasing pattern came from, for traceability — not used by scoring, just documentation.

Parameters:

  • population_description (str)

  • cohort_id (str)

  • source (str)

population_description: str
cohort_id: str
source: str
model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

phenoforge.eval.decomposition.DEFAULT_TEST_CASES: list[DecompositionTestCase] = [DecompositionTestCase(population_description='adults with type 2 diabetes mellitus', cohort_id='503', source='ClinicalTrials.gov NCT03985293, "...in Adults With Type 2 Diabetes Mellitus"'), DecompositionTestCase(population_description='patients with type 2 diabetes or a documented history of diabetes', cohort_id='40', source='common trial inclusion phrasing combining current diagnosis with history of diabetes'), DecompositionTestCase(population_description='type 2 diabetes mellitus confirmed by diagnosis, treatment, or laboratory testing', cohort_id='288', source='phrasing pattern from diagnosis/treatment/lab-indexed diabetes ascertainment algorithms'), DecompositionTestCase(population_description='adult patients with type 1 diabetes mellitus', cohort_id='499', source='ClinicalTrials.gov NCT01421147, "A Study in Adults With Type 1 Diabetes"'), DecompositionTestCase(population_description='patients presenting to the emergency department with diabetic ketoacidosis', cohort_id='611', source='common DKA trial inclusion phrasing (ED presentation with primary DKA diagnosis)'), DecompositionTestCase(population_description='pregnant women diagnosed with gestational diabetes mellitus', cohort_id='619', source='common GDM trial inclusion phrasing, e.g. ClinicalTrials.gov NCT02244814'), DecompositionTestCase(population_description='adults with diabetic retinopathy', cohort_id='647', source='common diabetic-retinopathy trial inclusion phrasing'), DecompositionTestCase(population_description='patients with chronic kidney disease', cohort_id='687', source='common CKD trial inclusion phrasing framed around a CKD diagnosis'), DecompositionTestCase(population_description='adult patients hospitalized with acute myocardial infarction', cohort_id='510', source='common AMI trial phrasing (diagnosis of acute MI, hospitalized within 48h of onset)'), DecompositionTestCase(population_description='adults diagnosed with atrial fibrillation', cohort_id='513', source='common AF trial inclusion phrasing, e.g. ClinicalTrials.gov NCT04357288'), DecompositionTestCase(population_description='adult patients with a diagnosis of heart failure', cohort_id='519', source='common heart failure trial inclusion phrasing'), DecompositionTestCase(population_description='adults hospitalized with community-acquired pneumonia', cohort_id='507', source='common CAP trial inclusion phrasing (radiographic infiltrate plus symptoms)'), DecompositionTestCase(population_description="adults with a physician's diagnosis of asthma", cohort_id='521', source='common asthma trial inclusion phrasing'), DecompositionTestCase(population_description='adult patients with major depressive disorder', cohort_id='594', source='ClinicalTrials.gov NCT00795821, "A Study in Adult Patients With Major Depressive Disorder"'), DecompositionTestCase(population_description='adults with generalized anxiety disorder', cohort_id='602', source='common GAD trial inclusion phrasing, e.g. ClinicalTrials.gov NCT03829241'), DecompositionTestCase(population_description='adult patients with a diagnosis of rheumatoid arthritis', cohort_id='457', source='common RA trial inclusion phrasing (ACR/EULAR 2010 classification criteria)'), DecompositionTestCase(population_description='adults with a diagnosis of epilepsy', cohort_id='549', source='common epilepsy trial inclusion phrasing'), DecompositionTestCase(population_description='adults with a history of migraine', cohort_id='524', source='common migraine trial inclusion phrasing (ICHD-3 diagnostic criteria)'), DecompositionTestCase(population_description='adult patients presenting to the emergency department with suspected sepsis', cohort_id='463', source='common sepsis trial inclusion phrasing (ED presentation, suspected infection)'), DecompositionTestCase(population_description='adults with a new diagnosis of inflammatory bowel disease', cohort_id='775', source="common IBD trial inclusion phrasing (Crohn's disease or ulcerative colitis)"), DecompositionTestCase(population_description='adult patients with a diagnosis of liver cirrhosis', cohort_id='626', source='common cirrhosis trial inclusion phrasing'), DecompositionTestCase(population_description='adults with a diagnosis of hyperlipidemia', cohort_id='558', source='common hyperlipidemia/dyslipidemia trial inclusion phrasing'), DecompositionTestCase(population_description='adults with obesity', cohort_id='563', source='common obesity trial inclusion phrasing (BMI-based eligibility)')]

Phrasing patterns drawn from real ClinicalTrials.gov study titles and inclusion-criteria language for each bundled cohort’s condition — not the cohort’s own bundled display name, and not invented from scratch, so these test decomposition against how population descriptions are actually phrased in practice, not against this project’s own wording conventions.

phenoforge.eval.decomposition.score_decomposition(con, library_dir, decompose_fn, dense=None, k=10, fanout_threshold=100, test_cases=None)[source]

Score decompose_fn against each test case’s bundled cohort ground truth.

Parameters:

  • con (DuckDBPyConnection) – Open connection to vocab.duckdb.

  • library_dir (Path) – Directory of fetched OHDSI Phenotype Library cohorts.

  • decompose_fn (Callable[[str], list[str]]) – Decomposition function under test — pass default_decomposer()’s result for a real Claude call, or a fake for testing.

  • dense (DenseRetriever | None) – Pre-built dense retriever, or None to degrade to BM25-only, matching search_concepts’s existing fallback.

  • k (int) – Result count passed to each retrieval call. Defaults to 10 to match agent/nodes.py’s check_curated/generate and engine/hybrid.hybrid_search’s own defaults — this function’s whole premise is comparing against what the real pipeline does, so its depth has to match that pipeline’s, not an arbitrary wider value (a mismatch here previously existed: this defaulted to 25 while production defaults to 10, silently measuring a more generous search than what actually ships).

  • fanout_threshold (int) – Passed through to curated ground-truth loading and curated resolution.

  • test_cases (list[DecompositionTestCase] | None) – Cases to run; defaults to DEFAULT_TEST_CASES.

Returns:

Per-case scores and the mean gap.

Return type:

DecompositionReport