LLM-based literature assessment system for rare disease gene curation. It screens papers for relevance and extracts evidence against PanelApp Australia diagnostic criteria. It then groups each gene's evidence into gene-disease-MoI associations and rates each one. The report shows every association next to what PanelApp Australia already curates, with the panels it matches.
# Installation (all pipeline stages; LLM calls go to the Claude API)
uv sync
# With ML dependencies (only for the PubMed baseline screening classifier)
uv sync --extra ml
# The report's PDF viewer page (Node.js 24+; rebuild after viewer/ changes)
npm --prefix viewer ci && npm --prefix viewer run build
Install the NCBI EDirect tools:
sh -c "$(curl -fsSL https://ftp.ncbi.nlm.nih.gov/entrez/entrezdirect/install-edirect.sh)"
Enable the local pre-commit hook (runs the same lint set as CI):
uv run pre-commit install
The system uses multiple databases:
data/db.sqlite): Per-run database, created from schema.sql, seeded from the ledger by palit ingest-pubmeddata/pubmed_ingestion_ledger.sqlite): Created from ledger_schema.sql by palit ledger init. The single canonical dedup/disposition memory across all runs (see "PubMed ingestion ledger" below)data/pubmed_baseline_screening.sqlite): Created from schema.sql by palit screen-pubmeddata/screening_classifier/training.sqlite): Created from src/palit/screening_classifier/training.sql (only needed for training)Both main and screening workflows use the same schema for consistency, allowing the same tools (e.g., assess-relevance) to work on both databases.
Stages that read the GenCC submissions or the MONDO ontology download gencc_submissions.tsv and mondo.obo into the run database's directory (e.g. data/) when the files are missing or older than 7 days.
LLM stages call the Claude API through the Anthropic SDK, authenticated with an ant OAuth profile instead of an API key. Log in once per profile (brew install anthropics/tap/ant first if needed):
ant auth login --profile <profile>
Palit reads the profile name from PALIT_ANTHROPIC_PROFILE, set either in the environment or in .env (see .env.example). The variable is required: an LLM stage stops with a validation error naming it before sending any request. Palit deliberately ignores ANTHROPIC_PROFILE and ANTHROPIC_API_KEY, which Claude Code sessions may export for their own workspace. The SDK refreshes the access token itself. The refresh token eventually expires, so when a previously working profile starts failing authentication, run ant auth login --profile <profile> again. Run one palit process per profile: the SDK serialises token refreshes within a process only.
Requests go to Claude Opus 5.5 (MODEL in src/palit/llm.py). When Opus 5.5 refuses a paper, gene or association in a stage, the stage's next attempt sends that subject to Claude Sonnet 5.5 (FALLBACK_MODEL) with the same prompt, schema, effort and token limit. The multi-round stages (extract-evidence, map-mondo) restart the subject's conversation from round 1 on Sonnet 5.5, since thinking blocks cannot be replayed to another model, and the tournament resends a refused prompt on Sonnet 5.5 within its round. A subject is refused for good only when Sonnet 5.5 refuses it too; this is what "refused" means in the stage summaries and the report, and stages skip such subjects. The fallback attempt runs in the same invocation and counts toward --max-retries. With --limit, a stage runs one attempt, so the fallback waits for the next invocation. --retry-refused on assess-relevance and extract-evidence sends the subjects refused for good in earlier invocations through both models once more. The fallback is palit's own because the Batches API rejects the server-side fallbacks parameter. assess-relevance --model runs the relevance screen on Sonnet 5.5 or Haiku 5.5 instead, for comparing models; only Opus 5.5 has a fallback, which is always a lower tier, so these models' refusals are final.
A stage also gives up on a subject once it has rejected two of its answers, or seen them cut off at max_tokens, since the subject's latest accepted answer (MAX_FAILED_ANSWERS in src/palit/llm.py); later invocations skip it as failed for good. --retry-failed on assess-relevance, extract-evidence, assess-genes, map-mondo and match-panels sends such subjects again, until they fail that often within the invocation.
Stages send their requests as Message Batches, except that a model's requests in a round go out as immediate requests when there are fewer than 10 of them (MIN_BATCH_REQUESTS), since a batch can take hours however few requests it holds. map-mondo and match-panels send all of their requests immediately. A map-mondo mapping takes several tool rounds. Every match-panels request starts with the same system prompt, the list of all panels, of about 61,000 tokens. Once the first request to a model has written that prefix to the prompt cache, the model's other immediate requests read it from the cache at a fraction of the batch price. In a batch, most requests write the cache instead of reading it.
assess-relevance settles a paper refused for good as not relevant, so that ledger writeback marks it settled and later runs do not ingest it again. Its relevance_assessment_json records the refusal in place of a screen:
{"relevant": false, "screen": null,
"refused": {"model": "claude-sonnet-5-5", "category": "bio"}}
Every invocation first settles the papers refused for good that still have no result, since a run database's recorded refusals may lack one. With --retry-refused, it reopens the settled papers instead, and any refused for good again are settled again. The report shows these papers as refused: they are kept out of the screen misses and the low-confidence review list, and the panel-publication sensitivity leaves them out.
uv run palit llm costs --db-path data/db.sqlite shows requests, refusals, rejected answers, tokens, and USD cost per stage, model and service tier for a run database. A rejected answer failed the stage's schema or structural checks, or was cut off at max_tokens, and is asked for again; llm_requests.rejection records why. Every stage prints the same summary for itself when it finishes, listing each refused request; uv run palit llm refusals --db-path data/db.sqlite lists all refusals with their safety-classifier category and the subject's outcome: recovered (answered after the refusal), refused for good (Sonnet 5.5 refused it last), or not answered yet. It then lists the subjects failed for good with their last rejection, as does each stage's summary. uv run pytest -m api checks that every stage's structured-output configuration still compiles on the API for both models (it needs the profile above).
Evidence extraction (palit extract-evidence) requires a running variant-lookup service. Copy .env.example to .env and set both:
VARIANT_LOOKUP_BASE_URL=https://<host>:<port>
VARIANT_LOOKUP_API_KEY=<bearer-token>
.env is gitignored. The command exits immediately on startup if either variable is missing.
# Configuration. PANEL_DATE selects the PanelApp snapshot the run is assessed
# against. Date it on the day the run is set up, after the literature window has
# closed, so that a gene a curator added during the window is already on its
# panel when the run assesses it.
PANEL_DATE=2025-10-20
START_DATE=2025-10-01
END_DATE=2025-10-15
LEDGER=data/pubmed_ingestion_ledger.sqlite
# 0. One-time: create the ledger — the dedup/disposition memory shared across all
# runs (see "PubMed ingestion ledger" below). Seed it from existing run DBs with
# `palit ledger seed` if you have prior corpora.
uv run palit ledger init --ledger $LEDGER
# 0a. Build the PanelApp snapshot of PANEL_DATE, which the stages from literature
# expansion on read: each public panel at its version on that date, with the reviews of its
# genes. PanelApp serves reviews only as they are now and edits them in
# place, so each panel records when its reviews were captured
# (evaluations_fetched_at); build the snapshot on PANEL_DATE so that the
# two agree. About 35,000 review requests, roughly ten minutes. A snapshot
# built without reviews gets them with `palit panelapp add-reviews`.
uv run palit panelapp snapshot --panel-date $PANEL_DATE
# 1. Ingest papers for the window through the ledger. The ledger replaces the old
# buffer window + --previous-db: late-indexed stragglers arrive via the FTP
# update-file sync (constant cost, unbounded horizon), and papers already settled
# (assessed not relevant, or downloaded) are never reconsidered while
# relevant-not-downloaded papers are re-emitted for a download retry. Preprints
# first so their metadata (version) survives for automatic PDF download;
# ingest-pubmed backfills PMIDs into preprint rows without overwriting them.
uv run palit ingest-preprints --ledger $LEDGER $START_DATE $END_DATE
uv run palit ingest-pubmed --ledger $LEDGER $START_DATE $END_DATE
# 2. Assess the relevance of every title and abstract with the scope screen,
# sent as Message Batches. A paper is relevant when it reports new patients
# with pathogenic variants, or functional tests of disease variants, for a
# specific gene within scope; the screen lists its gene-disease associations.
# Relevance only decides whether a paper gets a full-text read. Whether an
# association is already curated is not judged from an abstract, so papers on
# curated genes are read too. Safe to interrupt and re-run: it re-attaches to
# batches still in flight. Papers Opus 5.5 refuses go to Sonnet 5.5 (see
# "Claude API" above); papers both refuse are settled as not relevant, and
# --retry-refused sends them once more.
uv run palit assess-relevance
# 2a. (Optional) Screen the PubMed baseline with the retrospective prompt.
uv run palit assess-relevance --db-path data/pubmed_baseline_screening.sqlite --prompt-path prompts/retrospective_screening_prompt.txt
# 3. Download full-text papers (automated PMC + preprints, manual fallback)
uv run palit download-papers attempt-pmc
uv run palit download-papers download-preprints
uv run palit download-papers open-browser
# ... manually download PDFs to data/papers/ ...
uv run palit download-papers register
# 4. Extract evidence from the PDFs: Claude reads each PDF, looks up its genes
# (HGNC) and variants (gnomAD v4.1, via the variant-lookup service; requires
# VARIANT_LOOKUP_* env vars, see Setup) in one round, and cites verbatim
# quotes. Two batch rounds; safe to interrupt and re-run. Papers Opus 5.5
# refuses are extracted by Sonnet 5.5; re-runs skip papers both refused. Add
# --retry-refused to send each of them once more, since the safety
# classifier does not refuse the same paper every time.
# A PDF too long for the context window, often an article with all its
# supplements in one file, is sent with only the leading pages that fit,
# and the model is told so. Those pages leave room for 128,000 output
# tokens, the models' maximum; a page count stored with less room is
# counted again. Quotes are located in the pages that were sent, and the
# report shows the full PDF.
# Quotes never reject an answer. A quote the PDF can't place is kept, and
# the report and its viewer mark it as not located. Only placeholder
# citations that quote nothing (an empty quote, "x", "placeholder") are
# dropped.
# The extraction lists, for each gene with evidence in the paper, one disease
# entity per disease and MoI. Each entity has its own variants, family
# counts, criteria, summary and quality concerns. A disease whose only
# evidence in the paper is out of scope (large CNVs, gene fusions, digenic
# inheritance, somatic tumour variants, mechanistic context) gets no entity,
# even when another disease of the same gene gets one; a gene without an
# entity is left out. A disease for which the paper reports no new patients
# but functional or animal-model tests of specific disease variants (patient
# cells, variant knock-in models, measured structure or activity of the
# variant proteins; in-silico modelling alone does not count) gets an entity with 0 patients, no family counts and the tested
# variants; aggregation counts its experiments toward the association of that
# disease. The gene's variants are looked up in gnomAD once each, however
# many of its entities list them.
# Each disease entity carries three family counts. The reported count is
# every family the paper reports. The qualifying count (`family_count`) is
# the families whose genotype passes the qualifying variant gate. The
# independent count drops families that share the variant's ancestral
# origin; criteria A to C use it. For registries and case series without
# family structure, the qualifying and independent counts are derived from
# the patients who carry a qualifying genotype.
# An initial paper is extracted only when assess-relevance found it relevant
# or PanelApp cites it for one of the run's genes (step 6), even if a PDF is
# on file; expansion papers are always extracted. The genes with evidence
# from a relevant paper are this run's genes: steps 5, 6 and 9 work on them
# alone. Only relevant papers add run genes, however they entered the
# corpus, so a cited paper the screen rejected adds none.
uv run palit extract-evidence
# 5. Discover papers referenced in evidence (citation-based expansion)
uv run palit discover-citations discover
# 5a. Optionally add papers manually that weren't found automatically
uv run palit discover-citations add --gene GENE_SYMBOL PMID1 PMID2 ...
# 6. Expand literature beyond citations. Tournament selection over the screened
# baseline, bounded by --cutoff-date to the literature preceding the window,
# then unconditional seeding of the publications PanelApp already cites for
# each gene, on any panel of the --panel-date snapshot, copied from the
# baseline where it holds them (see "PanelApp Publication Seeding" below).
uv run palit expand-literature --cutoff-date $START_DATE --panel-date $PANEL_DATE
# 7. Download expansion papers and the papers PanelApp cites for the run's
# genes (same workflow as step 3)
uv run palit download-papers attempt-pmc
uv run palit download-papers download-preprints
uv run palit download-papers open-browser --expansion-only
# ... manually download PDFs to data/papers/ ...
uv run palit download-papers register
# 8. Extract evidence from the papers steps 5 to 7 added (same as step 4)
uv run palit extract-evidence
# 9. Aggregate each gene's evidence across papers into gene-disease-MoI
# associations, one request per gene. The model groups the papers' disease
# entities into associations, anchored on what PanelApp Australia already
# curates for the gene: its entries and reviews on every panel that holds
# it, from the snapshot. Each association records its relation to PanelApp
# Australia (existing, new disease or new MoI), the rating PanelApp gives it
# as judged from those entries and reviews, and its rating from the evidence
# in this run's corpus. GenCC's disease terms for the gene, from every
# submitter, are offered as names; GenCC classes are not shown. A gene is
# skipped when the target panels have already reviewed all of its papers:
# their entries for the gene and those entries' reviews cite every paper (by
# PMID, or by DOI in any letter case), and every entry of it there is known
# to have reviews. A gene is aggregated however many of its papers are cited
# when one of its entries on a target panel has no review, or reviews the
# snapshot does not hold: a gene entry whose reviews PanelApp could not
# serve, or a repeat-expansion entry, whose reviews the snapshot does not
# fetch. Safe to interrupt and re-run: it re-attaches to batches still in
# flight and skips genes that already have an aggregation.
uv run palit assess-genes --panel-date $PANEL_DATE
# 10. Map each association that reuses no GenCC disease term onto a
# MONDO term. Claude searches the local MONDO release with tools and
# answers with a term and a match type: exact when the term is for this
# gene's form of the disease (its definition or OMIM cross-reference names
# the gene), broader when MONDO has no such term and this is the most
# specific term that includes the disease. The report marks broader terms. Requests go out immediately, not
# as batches, because a mapping takes several tool rounds. Each
# association's conversation runs on its own, with up to 50 requests in
# flight across them (DEFAULT_IMMEDIATE_WORKERS), and its mapping is stored
# when the conversation ends. Re-runs map only the associations still
# without a term.
uv run palit map-mondo
# 11. Match each association to diagnostic panels by its disease, MoI and
# summary, and by the extractions of its papers: every disease each paper
# reports for the gene, with its MoI, patient count, summary and quotes
# with commentary. A paper's other diseases of the gene are included, with
# the disease and MoI of the gene's other associations; the model judges
# which diseases belong to the association. The extractions must have the
# per-disease shape. Requests go out immediately, not as batches, so
# that they read the shared panel list from the prompt cache, and each
# association's matches are stored as they arrive. Re-runs match only the
# associations still without matches.
uv run palit match-panels --panel-date $PANEL_DATE
# 12. Generate the report package: index.html, the PDF viewer page, and each
# cited paper's PDF (symlinked) with its quote highlights (citations/*.json).
# Each gene shows one block per association, with its relation to PanelApp
# Australia and its rating in this corpus. An association whose papers the
# target panels have all reviewed already, by the rule of step 9, is no
# finding whatever its rating, and says so; the statistics count those that
# would otherwise be findings. Each gene has a PanelApp prefill: a review
# form on the first target panel holding it, an add form on the first target
# panel otherwise. It describes the gene's findings. A gene without findings
# is prefilled with its other associations, or, when all are already
# reviewed, at its rating on the panel. A review replaces the curator's
# evaluation of the gene's entry, so its MoI, phenotypes and publications
# combine the entry's with those of the report's associations. The union is
# with the panel entry, not with the curator's own earlier review: a
# phenotype or publication that the review had and the entry lacks is
# replaced on submit, so the curator adds it back in the form. The comment
# is added to the earlier ones, so it describes only the prefilled
# associations: each gets a heading line with its finding (new disease, new
# MoI, or upgrade from the rating it starts from), disease, MoI and rating,
# then its summary. An association without a finding gets the heading
# without a finding kind. When the target panels have already reviewed the
# papers of every association, the comment is empty. An add form's comment
# heads each association with its disease, MoI and rating, and is the
# summary alone for a single association. PanelApp records citations per
# gene entry, not per disease,
# so a paper cited for one of the gene's diseases counts as reviewed for all
# of them. A gene on a target panel without findings goes under "Unreviewed
# on target panel" when any of its entries on the target panels has no
# review, and under "Already reviewed" when they all have reviews, or
# reviews the snapshot does not hold. A gene unreviewed on a target panel
# that has no association in the report is listed with the papers the run
# read for it: papers with a full-text extraction that the relevance screen
# named it in, that PanelApp cites for it, or whose extraction holds a
# disease of it. The list
# separates genes without such papers, genes whose papers gave no disease
# entity, and genes whose papers did but which have no association.
# `aws s3 sync` uploads the symlink targets.
uv run palit generate-report --report-id report_mendeliome --panel-date $PANEL_DATE
# To view a report locally, serve its folder and open http://localhost:8000/.
# Opening index.html as a file does not work: the viewer page fetches
# citations/*.json and papers/*.pdf, which browsers block for file:// pages.
# uv run python -m http.server 8000 -d reports/report_mendeliome
# 13. Fold this run's dispositions back into the ledger so future runs skip the
# papers settled here and resume any relevant-not-downloaded ones. Covers
# expansion/discovered-citation papers too (keyed by DOI). Separate from, and
# run alongside, the baseline-screening update below.
uv run palit ledger writeback --db-path data/db.sqlite --run-id report_mendeliome --ledger $LEDGER
A gene in the run is assessed on all its evidence, including every paper PanelApp Australia cites for it on any panel. Tournament selection keeps a minimal, non-redundant evidence set, so it routinely discards papers a curator has already cited (a single-family report loses to a comprehensive cohort by design), the relevance screen may reject a cited paper, and many older cited papers were never in the screened literature. If the run never read them, the report's "new evidence" would rest on papers it cannot compare with.
Seeding takes, for each of the run's genes, the publications PanelApp cites for it in the run's snapshot: the publication lists of its gene and repeat-expansion entries on every panel, and those of its gene entries' reviews. It needs a snapshot with reviews. Then:
data/pubmed_baseline_screening.sqlite, or --baseline-db-path) if the baseline holds it, matched by PMID or by DOI in any letter case, and fetched from PubMed (batched efetch) or CrossRef otherwise. It is stored as an expansion paper with source_details = 'panelapp:{hgnc_id}', the lowest HGNC ID citing it, and scheduled for download. Since the baseline takes in every PanelApp citation (see "Updating the Baseline"), fetching is the exception.panelapp_citations gets one row per run gene and held paper PanelApp cites for it, whether the run held the paper before seeding or seeding added it. A linked paper without a download status, such as an initial paper the relevance screen rejected, is scheduled for download.A paper linked to one of the run's genes is in the corpus whatever its relevance, so it is downloaded and extracted with the expansion papers; download-papers open-browser --expansion-only lists it too. The run's genes come from relevant papers only, so the extraction of a cited paper the screen rejected adds no genes. A cited paper the screen passed adds its genes like any relevant paper, and it may be extracted only after seeding (for example when reduce-literature had unscheduled it and seeding scheduled it again). Seeding runs once per run, so a gene that first enters the run through such a paper is aggregated without the papers PanelApp cites for it.
In the monthly workflow expand-literature seeds. It does so after --force-all's expansion wipe, so seeded papers survive it, and before the tournament's early return, so a rerun still picks up publications added to a panel since. Panel-scoped runs have no expand-literature and seed with palit panelapp seed-citations (see "Panel-Specific Curation"), which runs the same seeding.
Seeding can be rerun: papers already held are counted and skipped, and the table is rebuilt from the snapshot each time. Publications that never resolve to a DOI, and those whose PDF cannot be obtained, are logged and stay out of the corpus, so seeding widens coverage without guaranteeing it.
PubMed keeps indexing records weeks-to-months after their create date (CRDT), so a
fixed fetch window silently loses thousands of late-indexed papers every month. The
ingestion ledger (data/pubmed_ingestion_ledger.sqlite) is a single canonical
database that makes ingestion robust to this lag. It records, per DOI, the refreshed
bibliographic metadata plus the terminal disposition each run wrote back, and is the
dedup/disposition memory that replaces the old per-run buffer window + --previous-db
set-difference.
Two sources feed it, complementary by recency:
https://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/) — applied
incrementally by file number. A late-indexed straggler arrives in whatever file
first adds it, so old-date papers are caught at constant per-run cost without
re-fetching old days. Revised records (e.g. a late-attached abstract) reappear in
later files and refresh the row.Each run partitions previously-seen DOIs into settled (assessed not relevant, including papers both models refused to assess, or downloaded — never reconsidered) and actionable (not downloaded, and either never assessed or relevant — re-emitted into the run). A downloaded paper is settled whatever its relevance: the expansion papers a run writes back are never assessed for relevance, and they must not return as new papers. A CRDT month is finalised and dropped from the actionable set after a 6-month closure horizon, which bounds the work set.
LEDGER=data/pubmed_ingestion_ledger.sqlite
# Create the ledger once.
uv run palit ledger init --ledger $LEDGER
# Seed it from existing run databases (cross-era: DOI- and PMID-keyed corpora).
uv run palit ledger seed --ledger $LEDGER --db data/db_2026_april.sqlite --db data/db_2026_may.sqlite ...
# Apply any new FTP update files to the ledger (also done inside ingest-pubmed).
uv run palit ledger sync --ledger $LEDGER
# Per run: ingest-pubmed seeds the run DB from the ledger's actionable set;
# at run end, fold dispositions back.
uv run palit ledger writeback --db-path data/db.sqlite --run-id <run> --ledger $LEDGER
# 1. Install ML dependencies and setup W&B
uv sync --extra ml
wandb login
# 2. Create training database
sqlite3 data/screening_classifier/training.sqlite < src/palit/screening_classifier/training.sql
# 3. Extract positive PMIDs from main workflow database
uv run palit screening-classifier extract-pmids
# 4. Prepare training data (fetches negatives from PubMed, assigns train/val/test splits)
uv run palit screening-classifier prepare-data
# 5. Train classifier
uv run palit screening-classifier train
# 6. Evaluate classifier
uv run palit screening-classifier evaluate
Model outputs saved to outputs/best_model/ (HuggingFace format + optimal threshold).
Once trained, use the classifier to screen PubMed baseline XML files:
# Download PubMed baseline (all XML files + checksums, ~47GB compressed)
mkdir -p data/pubmed_baseline
cd data/pubmed_baseline
for kind in baseline updatefiles
do
curl -s https://ftp.ncbi.nlm.nih.gov/pubmed/$kind/ | \
grep -oP '(?<=href=")[^"]*\.(xml\.gz|md5)' | \
parallel --bar -j 8 "if [ ! -f {} ]; then curl -s -O \"https://ftp.ncbi.nlm.nih.gov/pubmed/$kind/{}\"; else echo \"{} exists, skipping.\"; fi"
done
cd ../..
# Screen baseline files with trained classifier
uv run palit screen-pubmed \
--checkpoint outputs/best_model \
--baseline-dir data/pubmed_baseline \
--output-db data/pubmed_baseline_screening.sqlite
Relevant papers are stored in pubmed_baseline_screening.sqlite. Processing progress is tracked in data/screening_progress.json for resumability.
For historical baseline screening (2000-2025), use the retrospective screening prompt which evaluates papers based on the evidence they provide rather than novelty:
# Retrospective mode: evaluates historical evidence value, not novelty
uv run palit assess-relevance \
--db-path data/pubmed_baseline_screening.sqlite \
--prompt-path prompts/retrospective_screening_prompt.txt
Key difference from standard relevance assessment:
relevance_assessment_prompt.txt): the scope screen of the monthly run, which asks whether a paper reports new patients or functional tests of disease variants for a specific gene, without judging whether the association is already known; optimized for recent literatureretrospective_screening_prompt.txt): Asks "Does this provide SUBSTANTIAL evidence for gene-disease relationships?" - optimized for historical baseline screeningThe retrospective prompt evaluates papers in their historical context, accepting important early descriptions of gene-disease associations even if those genes are now well-established. This ensures comprehensive coverage across 25 years of literature for downstream tournament selection and analysis.
After each fortnightly processing run completes, feed the papers its scope screen passed back into the baseline screening DB so it grows as a comprehensive repository:
FORTNIGHTLY_DB=data/db_2026_february_h1.sqlite
sqlite3 -cmd "ATTACH '$FORTNIGHTLY_DB' AS source" data/pubmed_baseline_screening.sqlite <<'SQL'
CREATE TEMP TABLE relevant_dois AS
SELECT doi FROM source.papers
WHERE relevance_assessment_json IS NOT NULL
AND json_extract(relevance_assessment_json, '$.screen.relevant') = 1;
INSERT OR IGNORE INTO papers
(doi, pmid, title, abstract, authors, journal, source_date,
source, source_metadata, source_type, source_details, download_status,
relevance_assessment_raw, relevance_assessment_json)
SELECT doi, pmid, title, abstract, authors, journal, source_date,
source, source_metadata, source_type, source_details, 'scheduled',
relevance_assessment_raw, relevance_assessment_json
FROM source.papers WHERE doi IN relevant_dois;
INSERT OR IGNORE INTO gene_mentions
(hgnc_id, paper_gene_symbol, paper_doi, source)
SELECT hgnc_id, paper_gene_symbol, paper_doi, source
FROM source.gene_mentions
WHERE source = 'relevance_assessment'
AND paper_doi IN relevant_dois;
DROP TABLE relevant_dois;
DETACH source;
SQL
The heredoc is quoted so the shell leaves the JSON paths alone; -cmd attaches the run database first, with the shell variable expanded. $.screen.relevant is the screen's decision in databases of either relevance shape.
The baseline also takes in every publication PanelApp Australia cites, on any panel, so that a panel-scoped run starts from PanelApp's citations as well as the screened literature. Once the run's snapshot holds reviews (built by palit panelapp snapshot, or completed with add-reviews for an older one), run:
uv run palit panelapp ingest-citations --panel-date $PANEL_DATE
It reads the PMIDs and DOIs from the publication lists of every gene and repeat-expansion entry and of each gene review, skips those the baseline already holds (by PMID, or by DOI in any letter case), fetches the rest from PubMed in batches of 200 or from CrossRef, and stores their metadata with source_type = 'panelapp_citation' and source_details = 'panelapp-snapshot:{panel_date}'. A DOI is stored as PanelApp spells it, and a held paper that lacks a PMID gets the one PanelApp cites it by. It writes no gene mentions: which genes a paper concerns is decided by the run that reads it. A PMID PubMed gives no usable record for, or a DOI CrossRef cannot resolve, is counted and skipped. If PubMed stays unreachable after retries, the command stops with an error; the papers stored until then are kept, and a rerun carries on from there.
For curating literature for a specific panel (e.g., Arthrogryposis):
# Configuration. As above, PANEL_DATE is the day the run is set up.
PANEL_DATE=2025-10-20
PANEL_ID=47 # Arthrogryposis panel ID
PANEL_NAME=arthrogryposis
# 0. Build the PanelApp snapshot, as in the main workflow
uv run palit panelapp snapshot --panel-date $PANEL_DATE
# 1. Copy the baseline's papers to a new database: the screened literature and
# every publication PanelApp cites (see "Updating the Baseline"). Only the
# paper metadata is copied: step 2 assesses relevance afresh and schedules
# the relevant papers for download.
sqlite3 data/$PANEL_NAME.sqlite < schema.sql
sqlite3 data/$PANEL_NAME.sqlite "ATTACH 'data/pubmed_baseline_screening.sqlite' AS source; INSERT INTO papers (doi, pmid, title, abstract, authors, journal, source, source_date, source_metadata, source_type, source_details) SELECT doi, pmid, title, abstract, authors, journal, source, source_date, source_metadata, 'initial', source_details FROM source.papers"
# 2. Assess relevance scoped to the panel: the screen gets the panel description
# and passes papers with human genetic evidence within the panel's phenotypic
# scope, whether or not the panel already lists the gene
uv run palit assess-relevance \
--db-path data/$PANEL_NAME.sqlite \
--panel-date $PANEL_DATE \
--scope-panel-id $PANEL_ID \
--prompt-path prompts/panel_relevance_assessment_prompt.txt
# 3. (Optional) Reduce literature for well-researched genes
# Genes with hundreds of papers make the aggregation prompt long and slow. For panels with well-researched genes (e.g., POLG
# with 200+ papers), use tournament selection to keep only the most informative.
# It counts and selects among the papers step 2 assessed relevant:
uv run palit reduce-literature --db-path data/$PANEL_NAME.sqlite
# 4. Download full-text papers (now reduced set if step 3 was run)
uv run palit download-papers attempt-pmc --db-path data/$PANEL_NAME.sqlite
uv run palit download-papers download-preprints --db-path data/$PANEL_NAME.sqlite
uv run palit download-papers open-browser --db-path data/$PANEL_NAME.sqlite
# ... manually download PDFs ...
uv run palit download-papers register --db-path data/$PANEL_NAME.sqlite
# 5. Extract evidence, scoped to the panel. Scope is decided for each disease
# entity (one disease with one MoI), on the patients the paper reports for
# it: the disease gets an entity only when those individuals have features
# that would lead a clinician to order the panel, judged from what the paper
# reports about them, with any exclusions the panel's description makes. A
# paper saying they have the panel's disorder is enough, and a feature it
# does not mention is unknown, not absent. Another disease of the same gene
# can be left out. An entity without patients is judged on what the paper
# says about the patients who carry its tested variants (their diagnosis or
# features, also when cited from earlier reports). Each entity records why it fits in
# `scope_rationale`: the reported features with how many of its individuals
# have them. The genes with evidence from a relevant paper are the run's
# genes.
uv run palit extract-evidence --db-path data/$PANEL_NAME.sqlite --panel-date $PANEL_DATE --scope-panel-id $PANEL_ID
# 6. Seed the publications PanelApp cites for the run's genes, on any panel
# (see "PanelApp Publication Seeding"). Most are already in the database,
# copied from the baseline in step 1; those the screen rejected are scheduled
# for download all the same. The rest are copied from the baseline or
# fetched.
uv run palit panelapp seed-citations --db-path data/$PANEL_NAME.sqlite --panel-date $PANEL_DATE
# 7. Download and extract the seeded papers, as in steps 4 and 5
uv run palit download-papers attempt-pmc --db-path data/$PANEL_NAME.sqlite
uv run palit download-papers download-preprints --db-path data/$PANEL_NAME.sqlite
uv run palit download-papers open-browser --db-path data/$PANEL_NAME.sqlite --expansion-only
# ... manually download PDFs ...
uv run palit download-papers register --db-path data/$PANEL_NAME.sqlite
uv run palit extract-evidence --db-path data/$PANEL_NAME.sqlite --panel-date $PANEL_DATE --scope-panel-id $PANEL_ID
# A relevant paper extracted in this round (one reduce-literature had left
# out, then seeded because PanelApp cites it) can add run genes. Step 6 runs
# once, so these genes are aggregated without the papers PanelApp cites for
# them.
# 8. Assess genes against the panel, which is the only target panel. Each
# association gets a `scope_rationale` condensed from those of the
# paper-level entities it groups, with the paper each count comes from.
# assess-genes stops when a paper's extraction was not made with the same
# scoping: one without `scope_rationale` here, or one with it in an
# aggregation without --scope-panel-id.
uv run palit assess-genes --db-path data/$PANEL_NAME.sqlite --panel-date $PANEL_DATE --scope-panel-id $PANEL_ID
# 9. Map associations without a GenCC term onto MONDO terms, as in the main
# workflow. Panel-scoped runs do not run match-panels.
uv run palit map-mondo --db-path data/$PANEL_NAME.sqlite
# 10. Generate report package against the panel: every association of a gene
# not on it is a finding, and an upgrade starts from the gene's rating on it.
# Each association shows its `scope_rationale` as "Panel fit". A gene
# unreviewed on the panel without an association is listed as in the main
# workflow; the group whose papers gave no disease entity reads "Papers
# read, none fit the panel". That judgement covers those papers' patients
# only; whether the entry belongs on the panel is the curator's call.
uv run palit generate-report \
--report-id panel_$PANEL_NAME \
--db-path data/$PANEL_NAME.sqlite \
--panel-date $PANEL_DATE \
--scope-panel-id $PANEL_ID
LLM-based literature assessment system for rare disease gene curation. It screens papers for relevance and extracts evidence against PanelApp Australia diagnostic criteria. It then groups each gene's evidence into gene-disease-MoI associations and rates each one. The report shows every association next to what PanelApp Australia already curates, with the panels it matches.
# Installation (all pipeline stages; LLM calls go to the Claude API)
uv sync
# With ML dependencies (only for the PubMed baseline screening classifier)
uv sync --extra ml
# The report's PDF viewer page (Node.js 24+; rebuild after viewer/ changes)
npm --prefix viewer ci && npm --prefix viewer run build
Install the NCBI EDirect tools:
sh -c "$(curl -fsSL https://ftp.ncbi.nlm.nih.gov/entrez/entrezdirect/install-edirect.sh)"
Enable the local pre-commit hook (runs the same lint set as CI):
uv run pre-commit install
The system uses multiple databases:
data/db.sqlite): Per-run database, created from schema.sql, seeded from the ledger by palit ingest-pubmeddata/pubmed_ingestion_ledger.sqlite): Created from ledger_schema.sql by palit ledger init. The single canonical dedup/disposition memory across all runs (see "PubMed ingestion ledger" below)data/pubmed_baseline_screening.sqlite): Created from schema.sql by palit screen-pubmeddata/screening_classifier/training.sqlite): Created from src/palit/screening_classifier/training.sql (only needed for training)Both main and screening workflows use the same schema for consistency, allowing the same tools (e.g., assess-relevance) to work on both databases.
Stages that read the GenCC submissions or the MONDO ontology download gencc_submissions.tsv and mondo.obo into the run database's directory (e.g. data/) when the files are missing or older than 7 days.
LLM stages call the Claude API through the Anthropic SDK, authenticated with an ant OAuth profile instead of an API key. Log in once per profile (brew install anthropics/tap/ant first if needed):
ant auth login --profile <profile>
Palit reads the profile name from PALIT_ANTHROPIC_PROFILE, set either in the environment or in .env (see .env.example). The variable is required: an LLM stage stops with a validation error naming it before sending any request. Palit deliberately ignores ANTHROPIC_PROFILE and ANTHROPIC_API_KEY, which Claude Code sessions may export for their own workspace. The SDK refreshes the access token itself. The refresh token eventually expires, so when a previously working profile starts failing authentication, run ant auth login --profile <profile> again. Run one palit process per profile: the SDK serialises token refreshes within a process only.
Requests go to Claude Opus 5.5 (MODEL in src/palit/llm.py). When Opus 5.5 refuses a paper, gene or association in a stage, the stage's next attempt sends that subject to Claude Sonnet 5.5 (FALLBACK_MODEL) with the same prompt, schema, effort and token limit. The multi-round stages (extract-evidence, map-mondo) restart the subject's conversation from round 1 on Sonnet 5.5, since thinking blocks cannot be replayed to another model, and the tournament resends a refused prompt on Sonnet 5.5 within its round. A subject is refused for good only when Sonnet 5.5 refuses it too; this is what "refused" means in the stage summaries and the report, and stages skip such subjects. The fallback attempt runs in the same invocation and counts toward --max-retries. With --limit, a stage runs one attempt, so the fallback waits for the next invocation. --retry-refused on assess-relevance and extract-evidence sends the subjects refused for good in earlier invocations through both models once more. The fallback is palit's own because the Batches API rejects the server-side fallbacks parameter. assess-relevance --model runs the relevance screen on Sonnet 5.5 or Haiku 5.5 instead, for comparing models; only Opus 5.5 has a fallback, which is always a lower tier, so these models' refusals are final.
A stage also gives up on a subject once it has rejected two of its answers, or seen them cut off at max_tokens, since the subject's latest accepted answer (MAX_FAILED_ANSWERS in src/palit/llm.py); later invocations skip it as failed for good. --retry-failed on assess-relevance, extract-evidence, assess-genes, map-mondo and match-panels sends such subjects again, until they fail that often within the invocation.
Stages send their requests as Message Batches, except that a model's requests in a round go out as immediate requests when there are fewer than 10 of them (MIN_BATCH_REQUESTS), since a batch can take hours however few requests it holds. map-mondo and match-panels send all of their requests immediately. A map-mondo mapping takes several tool rounds. Every match-panels request starts with the same system prompt, the list of all panels, of about 61,000 tokens. Once the first request to a model has written that prefix to the prompt cache, the model's other immediate requests read it from the cache at a fraction of the batch price. In a batch, most requests write the cache instead of reading it.
assess-relevance settles a paper refused for good as not relevant, so that ledger writeback marks it settled and later runs do not ingest it again. Its relevance_assessment_json records the refusal in place of a screen:
{"relevant": false, "screen": null,
"refused": {"model": "claude-sonnet-5-5", "category": "bio"}}
Every invocation first settles the papers refused for good that still have no result, since a run database's recorded refusals may lack one. With --retry-refused, it reopens the settled papers instead, and any refused for good again are settled again. The report shows these papers as refused: they are kept out of the screen misses and the low-confidence review list, and the panel-publication sensitivity leaves them out.
uv run palit llm costs --db-path data/db.sqlite shows requests, refusals, rejected answers, tokens, and USD cost per stage, model and service tier for a run database. A rejected answer failed the stage's schema or structural checks, or was cut off at max_tokens, and is asked for again; llm_requests.rejection records why. Every stage prints the same summary for itself when it finishes, listing each refused request; uv run palit llm refusals --db-path data/db.sqlite lists all refusals with their safety-classifier category and the subject's outcome: recovered (answered after the refusal), refused for good (Sonnet 5.5 refused it last), or not answered yet. It then lists the subjects failed for good with their last rejection, as does each stage's summary. uv run pytest -m api checks that every stage's structured-output configuration still compiles on the API for both models (it needs the profile above).
Evidence extraction (palit extract-evidence) requires a running variant-lookup service. Copy .env.example to .env and set both:
VARIANT_LOOKUP_BASE_URL=https://<host>:<port>
VARIANT_LOOKUP_API_KEY=<bearer-token>
.env is gitignored. The command exits immediately on startup if either variable is missing.
# Configuration. PANEL_DATE selects the PanelApp snapshot the run is assessed
# against. Date it on the day the run is set up, after the literature window has
# closed, so that a gene a curator added during the window is already on its
# panel when the run assesses it.
PANEL_DATE=2025-10-20
START_DATE=2025-10-01
END_DATE=2025-10-15
LEDGER=data/pubmed_ingestion_ledger.sqlite
# 0. One-time: create the ledger — the dedup/disposition memory shared across all
# runs (see "PubMed ingestion ledger" below). Seed it from existing run DBs with
# `palit ledger seed` if you have prior corpora.
uv run palit ledger init --ledger $LEDGER
# 0a. Build the PanelApp snapshot of PANEL_DATE, which the stages from literature
# expansion on read: each public panel at its version on that date, with the reviews of its
# genes. PanelApp serves reviews only as they are now and edits them in
# place, so each panel records when its reviews were captured
# (evaluations_fetched_at); build the snapshot on PANEL_DATE so that the
# two agree. About 35,000 review requests, roughly ten minutes. A snapshot
# built without reviews gets them with `palit panelapp add-reviews`.
uv run palit panelapp snapshot --panel-date $PANEL_DATE
# 1. Ingest papers for the window through the ledger. The ledger replaces the old
# buffer window + --previous-db: late-indexed stragglers arrive via the FTP
# update-file sync (constant cost, unbounded horizon), and papers already settled
# (assessed not relevant, or downloaded) are never reconsidered while
# relevant-not-downloaded papers are re-emitted for a download retry. Preprints
# first so their metadata (version) survives for automatic PDF download;
# ingest-pubmed backfills PMIDs into preprint rows without overwriting them.
uv run palit ingest-preprints --ledger $LEDGER $START_DATE $END_DATE
uv run palit ingest-pubmed --ledger $LEDGER $START_DATE $END_DATE
# 2. Assess the relevance of every title and abstract with the scope screen,
# sent as Message Batches. A paper is relevant when it reports new patients
# with pathogenic variants, or functional tests of disease variants, for a
# specific gene within scope; the screen lists its gene-disease associations.
# Relevance only decides whether a paper gets a full-text read. Whether an
# association is already curated is not judged from an abstract, so papers on
# curated genes are read too. Safe to interrupt and re-run: it re-attaches to
# batches still in flight. Papers Opus 5.5 refuses go to Sonnet 5.5 (see
# "Claude API" above); papers both refuse are settled as not relevant, and
# --retry-refused sends them once more.
uv run palit assess-relevance
# 2a. (Optional) Screen the PubMed baseline with the retrospective prompt.
uv run palit assess-relevance --db-path data/pubmed_baseline_screening.sqlite --prompt-path prompts/retrospective_screening_prompt.txt
# 3. Download full-text papers (automated PMC + preprints, manual fallback)
uv run palit download-papers attempt-pmc
uv run palit download-papers download-preprints
uv run palit download-papers open-browser
# ... manually download PDFs to data/papers/ ...
uv run palit download-papers register
# 4. Extract evidence from the PDFs: Claude reads each PDF, looks up its genes
# (HGNC) and variants (gnomAD v4.1, via the variant-lookup service; requires
# VARIANT_LOOKUP_* env vars, see Setup) in one round, and cites verbatim
# quotes. Two batch rounds; safe to interrupt and re-run. Papers Opus 5.5
# refuses are extracted by Sonnet 5.5; re-runs skip papers both refused. Add
# --retry-refused to send each of them once more, since the safety
# classifier does not refuse the same paper every time.
# A PDF too long for the context window, often an article with all its
# supplements in one file, is sent with only the leading pages that fit,
# and the model is told so. Those pages leave room for 128,000 output
# tokens, the models' maximum; a page count stored with less room is
# counted again. Quotes are located in the pages that were sent, and the
# report shows the full PDF.
# Quotes never reject an answer. A quote the PDF can't place is kept, and
# the report and its viewer mark it as not located. Only placeholder
# citations that quote nothing (an empty quote, "x", "placeholder") are
# dropped.
# The extraction lists, for each gene with evidence in the paper, one disease
# entity per disease and MoI. Each entity has its own variants, family
# counts, criteria, summary and quality concerns. A disease whose only
# evidence in the paper is out of scope (large CNVs, gene fusions, digenic
# inheritance, somatic tumour variants, mechanistic context) gets no entity,
# even when another disease of the same gene gets one; a gene without an
# entity is left out. A disease for which the paper reports no new patients
# but functional or animal-model tests of specific disease variants (patient
# cells, variant knock-in models, measured structure or activity of the
# variant proteins; in-silico modelling alone does not count) gets an entity with 0 patients, no family counts and the tested
# variants; aggregation counts its experiments toward the association of that
# disease. The gene's variants are looked up in gnomAD once each, however
# many of its entities list them.
# Each disease entity carries three family counts. The reported count is
# every family the paper reports. The qualifying count (`family_count`) is
# the families whose genotype passes the qualifying variant gate. The
# independent count drops families that share the variant's ancestral
# origin; criteria A to C use it. For registries and case series without
# family structure, the qualifying and independent counts are derived from
# the patients who carry a qualifying genotype.
# An initial paper is extracted only when assess-relevance found it relevant
# or PanelApp cites it for one of the run's genes (step 6), even if a PDF is
# on file; expansion papers are always extracted. The genes with evidence
# from a relevant paper are this run's genes: steps 5, 6 and 9 work on them
# alone. Only relevant papers add run genes, however they entered the
# corpus, so a cited paper the screen rejected adds none.
uv run palit extract-evidence
# 5. Discover papers referenced in evidence (citation-based expansion)
uv run palit discover-citations discover
# 5a. Optionally add papers manually that weren't found automatically
uv run palit discover-citations add --gene GENE_SYMBOL PMID1 PMID2 ...
# 6. Expand literature beyond citations. Tournament selection over the screened
# baseline, bounded by --cutoff-date to the literature preceding the window,
# then unconditional seeding of the publications PanelApp already cites for
# each gene, on any panel of the --panel-date snapshot, copied from the
# baseline where it holds them (see "PanelApp Publication Seeding" below).
uv run palit expand-literature --cutoff-date $START_DATE --panel-date $PANEL_DATE
# 7. Download expansion papers and the papers PanelApp cites for the run's
# genes (same workflow as step 3)
uv run palit download-papers attempt-pmc
uv run palit download-papers download-preprints
uv run palit download-papers open-browser --expansion-only
# ... manually download PDFs to data/papers/ ...
uv run palit download-papers register
# 8. Extract evidence from the papers steps 5 to 7 added (same as step 4)
uv run palit extract-evidence
# 9. Aggregate each gene's evidence across papers into gene-disease-MoI
# associations, one request per gene. The model groups the papers' disease
# entities into associations, anchored on what PanelApp Australia already
# curates for the gene: its entries and reviews on every panel that holds
# it, from the snapshot. Each association records its relation to PanelApp
# Australia (existing, new disease or new MoI), the rating PanelApp gives it
# as judged from those entries and reviews, and its rating from the evidence
# in this run's corpus. GenCC's disease terms for the gene, from every
# submitter, are offered as names; GenCC classes are not shown. A gene is
# skipped when the target panels have already reviewed all of its papers:
# their entries for the gene and those entries' reviews cite every paper (by
# PMID, or by DOI in any letter case), and every entry of it there is known
# to have reviews. A gene is aggregated however many of its papers are cited
# when one of its entries on a target panel has no review, or reviews the
# snapshot does not hold: a gene entry whose reviews PanelApp could not
# serve, or a repeat-expansion entry, whose reviews the snapshot does not
# fetch. Safe to interrupt and re-run: it re-attaches to batches still in
# flight and skips genes that already have an aggregation.
uv run palit assess-genes --panel-date $PANEL_DATE
# 10. Map each association that reuses no GenCC disease term onto a
# MONDO term. Claude searches the local MONDO release with tools and
# answers with a term and a match type: exact when the term is for this
# gene's form of the disease (its definition or OMIM cross-reference names
# the gene), broader when MONDO has no such term and this is the most
# specific term that includes the disease. The report marks broader terms. Requests go out immediately, not
# as batches, because a mapping takes several tool rounds. Each
# association's conversation runs on its own, with up to 50 requests in
# flight across them (DEFAULT_IMMEDIATE_WORKERS), and its mapping is stored
# when the conversation ends. Re-runs map only the associations still
# without a term.
uv run palit map-mondo
# 11. Match each association to diagnostic panels by its disease, MoI and
# summary, and by the extractions of its papers: every disease each paper
# reports for the gene, with its MoI, patient count, summary and quotes
# with commentary. A paper's other diseases of the gene are included, with
# the disease and MoI of the gene's other associations; the model judges
# which diseases belong to the association. The extractions must have the
# per-disease shape. Requests go out immediately, not as batches, so
# that they read the shared panel list from the prompt cache, and each
# association's matches are stored as they arrive. Re-runs match only the
# associations still without matches.
uv run palit match-panels --panel-date $PANEL_DATE
# 12. Generate the report package: index.html, the PDF viewer page, and each
# cited paper's PDF (symlinked) with its quote highlights (citations/*.json).
# Each gene shows one block per association, with its relation to PanelApp
# Australia and its rating in this corpus. An association whose papers the
# target panels have all reviewed already, by the rule of step 9, is no
# finding whatever its rating, and says so; the statistics count those that
# would otherwise be findings. Each gene has a PanelApp prefill: a review
# form on the first target panel holding it, an add form on the first target
# panel otherwise. It describes the gene's findings. A gene without findings
# is prefilled with its other associations, or, when all are already
# reviewed, at its rating on the panel. A review replaces the curator's
# evaluation of the gene's entry, so its MoI, phenotypes and publications
# combine the entry's with those of the report's associations. The union is
# with the panel entry, not with the curator's own earlier review: a
# phenotype or publication that the review had and the entry lacks is
# replaced on submit, so the curator adds it back in the form. The comment
# is added to the earlier ones, so it describes only the prefilled
# associations: each gets a heading line with its finding (new disease, new
# MoI, or upgrade from the rating it starts from), disease, MoI and rating,
# then its summary. An association without a finding gets the heading
# without a finding kind. When the target panels have already reviewed the
# papers of every association, the comment is empty. An add form's comment
# heads each association with its disease, MoI and rating, and is the
# summary alone for a single association. PanelApp records citations per
# gene entry, not per disease,
# so a paper cited for one of the gene's diseases counts as reviewed for all
# of them. A gene on a target panel without findings goes under "Unreviewed
# on target panel" when any of its entries on the target panels has no
# review, and under "Already reviewed" when they all have reviews, or
# reviews the snapshot does not hold. A gene unreviewed on a target panel
# that has no association in the report is listed with the papers the run
# read for it: papers with a full-text extraction that the relevance screen
# named it in, that PanelApp cites for it, or whose extraction holds a
# disease of it. The list
# separates genes without such papers, genes whose papers gave no disease
# entity, and genes whose papers did but which have no association.
# `aws s3 sync` uploads the symlink targets.
uv run palit generate-report --report-id report_mendeliome --panel-date $PANEL_DATE
# To view a report locally, serve its folder and open http://localhost:8000/.
# Opening index.html as a file does not work: the viewer page fetches
# citations/*.json and papers/*.pdf, which browsers block for file:// pages.
# uv run python -m http.server 8000 -d reports/report_mendeliome
# 13. Fold this run's dispositions back into the ledger so future runs skip the
# papers settled here and resume any relevant-not-downloaded ones. Covers
# expansion/discovered-citation papers too (keyed by DOI). Separate from, and
# run alongside, the baseline-screening update below.
uv run palit ledger writeback --db-path data/db.sqlite --run-id report_mendeliome --ledger $LEDGER
A gene in the run is assessed on all its evidence, including every paper PanelApp Australia cites for it on any panel. Tournament selection keeps a minimal, non-redundant evidence set, so it routinely discards papers a curator has already cited (a single-family report loses to a comprehensive cohort by design), the relevance screen may reject a cited paper, and many older cited papers were never in the screened literature. If the run never read them, the report's "new evidence" would rest on papers it cannot compare with.
Seeding takes, for each of the run's genes, the publications PanelApp cites for it in the run's snapshot: the publication lists of its gene and repeat-expansion entries on every panel, and those of its gene entries' reviews. It needs a snapshot with reviews. Then:
data/pubmed_baseline_screening.sqlite, or --baseline-db-path) if the baseline holds it, matched by PMID or by DOI in any letter case, and fetched from PubMed (batched efetch) or CrossRef otherwise. It is stored as an expansion paper with source_details = 'panelapp:{hgnc_id}', the lowest HGNC ID citing it, and scheduled for download. Since the baseline takes in every PanelApp citation (see "Updating the Baseline"), fetching is the exception.panelapp_citations gets one row per run gene and held paper PanelApp cites for it, whether the run held the paper before seeding or seeding added it. A linked paper without a download status, such as an initial paper the relevance screen rejected, is scheduled for download.A paper linked to one of the run's genes is in the corpus whatever its relevance, so it is downloaded and extracted with the expansion papers; download-papers open-browser --expansion-only lists it too. The run's genes come from relevant papers only, so the extraction of a cited paper the screen rejected adds no genes. A cited paper the screen passed adds its genes like any relevant paper, and it may be extracted only after seeding (for example when reduce-literature had unscheduled it and seeding scheduled it again). Seeding runs once per run, so a gene that first enters the run through such a paper is aggregated without the papers PanelApp cites for it.
In the monthly workflow expand-literature seeds. It does so after --force-all's expansion wipe, so seeded papers survive it, and before the tournament's early return, so a rerun still picks up publications added to a panel since. Panel-scoped runs have no expand-literature and seed with palit panelapp seed-citations (see "Panel-Specific Curation"), which runs the same seeding.
Seeding can be rerun: papers already held are counted and skipped, and the table is rebuilt from the snapshot each time. Publications that never resolve to a DOI, and those whose PDF cannot be obtained, are logged and stay out of the corpus, so seeding widens coverage without guaranteeing it.
PubMed keeps indexing records weeks-to-months after their create date (CRDT), so a
fixed fetch window silently loses thousands of late-indexed papers every month. The
ingestion ledger (data/pubmed_ingestion_ledger.sqlite) is a single canonical
database that makes ingestion robust to this lag. It records, per DOI, the refreshed
bibliographic metadata plus the terminal disposition each run wrote back, and is the
dedup/disposition memory that replaces the old per-run buffer window + --previous-db
set-difference.
Two sources feed it, complementary by recency:
https://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/) — applied
incrementally by file number. A late-indexed straggler arrives in whatever file
first adds it, so old-date papers are caught at constant per-run cost without
re-fetching old days. Revised records (e.g. a late-attached abstract) reappear in
later files and refresh the row.Each run partitions previously-seen DOIs into settled (assessed not relevant, including papers both models refused to assess, or downloaded — never reconsidered) and actionable (not downloaded, and either never assessed or relevant — re-emitted into the run). A downloaded paper is settled whatever its relevance: the expansion papers a run writes back are never assessed for relevance, and they must not return as new papers. A CRDT month is finalised and dropped from the actionable set after a 6-month closure horizon, which bounds the work set.
LEDGER=data/pubmed_ingestion_ledger.sqlite
# Create the ledger once.
uv run palit ledger init --ledger $LEDGER
# Seed it from existing run databases (cross-era: DOI- and PMID-keyed corpora).
uv run palit ledger seed --ledger $LEDGER --db data/db_2026_april.sqlite --db data/db_2026_may.sqlite ...
# Apply any new FTP update files to the ledger (also done inside ingest-pubmed).
uv run palit ledger sync --ledger $LEDGER
# Per run: ingest-pubmed seeds the run DB from the ledger's actionable set;
# at run end, fold dispositions back.
uv run palit ledger writeback --db-path data/db.sqlite --run-id <run> --ledger $LEDGER
# 1. Install ML dependencies and setup W&B
uv sync --extra ml
wandb login
# 2. Create training database
sqlite3 data/screening_classifier/training.sqlite < src/palit/screening_classifier/training.sql
# 3. Extract positive PMIDs from main workflow database
uv run palit screening-classifier extract-pmids
# 4. Prepare training data (fetches negatives from PubMed, assigns train/val/test splits)
uv run palit screening-classifier prepare-data
# 5. Train classifier
uv run palit screening-classifier train
# 6. Evaluate classifier
uv run palit screening-classifier evaluate
Model outputs saved to outputs/best_model/ (HuggingFace format + optimal threshold).
Once trained, use the classifier to screen PubMed baseline XML files:
# Download PubMed baseline (all XML files + checksums, ~47GB compressed)
mkdir -p data/pubmed_baseline
cd data/pubmed_baseline
for kind in baseline updatefiles
do
curl -s https://ftp.ncbi.nlm.nih.gov/pubmed/$kind/ | \
grep -oP '(?<=href=")[^"]*\.(xml\.gz|md5)' | \
parallel --bar -j 8 "if [ ! -f {} ]; then curl -s -O \"https://ftp.ncbi.nlm.nih.gov/pubmed/$kind/{}\"; else echo \"{} exists, skipping.\"; fi"
done
cd ../..
# Screen baseline files with trained classifier
uv run palit screen-pubmed \
--checkpoint outputs/best_model \
--baseline-dir data/pubmed_baseline \
--output-db data/pubmed_baseline_screening.sqlite
Relevant papers are stored in pubmed_baseline_screening.sqlite. Processing progress is tracked in data/screening_progress.json for resumability.
For historical baseline screening (2000-2025), use the retrospective screening prompt which evaluates papers based on the evidence they provide rather than novelty:
# Retrospective mode: evaluates historical evidence value, not novelty
uv run palit assess-relevance \
--db-path data/pubmed_baseline_screening.sqlite \
--prompt-path prompts/retrospective_screening_prompt.txt
Key difference from standard relevance assessment:
relevance_assessment_prompt.txt): the scope screen of the monthly run, which asks whether a paper reports new patients or functional tests of disease variants for a specific gene, without judging whether the association is already known; optimized for recent literatureretrospective_screening_prompt.txt): Asks "Does this provide SUBSTANTIAL evidence for gene-disease relationships?" - optimized for historical baseline screeningThe retrospective prompt evaluates papers in their historical context, accepting important early descriptions of gene-disease associations even if those genes are now well-established. This ensures comprehensive coverage across 25 years of literature for downstream tournament selection and analysis.
After each fortnightly processing run completes, feed the papers its scope screen passed back into the baseline screening DB so it grows as a comprehensive repository:
FORTNIGHTLY_DB=data/db_2026_february_h1.sqlite
sqlite3 -cmd "ATTACH '$FORTNIGHTLY_DB' AS source" data/pubmed_baseline_screening.sqlite <<'SQL'
CREATE TEMP TABLE relevant_dois AS
SELECT doi FROM source.papers
WHERE relevance_assessment_json IS NOT NULL
AND json_extract(relevance_assessment_json, '$.screen.relevant') = 1;
INSERT OR IGNORE INTO papers
(doi, pmid, title, abstract, authors, journal, source_date,
source, source_metadata, source_type, source_details, download_status,
relevance_assessment_raw, relevance_assessment_json)
SELECT doi, pmid, title, abstract, authors, journal, source_date,
source, source_metadata, source_type, source_details, 'scheduled',
relevance_assessment_raw, relevance_assessment_json
FROM source.papers WHERE doi IN relevant_dois;
INSERT OR IGNORE INTO gene_mentions
(hgnc_id, paper_gene_symbol, paper_doi, source)
SELECT hgnc_id, paper_gene_symbol, paper_doi, source
FROM source.gene_mentions
WHERE source = 'relevance_assessment'
AND paper_doi IN relevant_dois;
DROP TABLE relevant_dois;
DETACH source;
SQL
The heredoc is quoted so the shell leaves the JSON paths alone; -cmd attaches the run database first, with the shell variable expanded. $.screen.relevant is the screen's decision in databases of either relevance shape.
The baseline also takes in every publication PanelApp Australia cites, on any panel, so that a panel-scoped run starts from PanelApp's citations as well as the screened literature. Once the run's snapshot holds reviews (built by palit panelapp snapshot, or completed with add-reviews for an older one), run:
uv run palit panelapp ingest-citations --panel-date $PANEL_DATE
It reads the PMIDs and DOIs from the publication lists of every gene and repeat-expansion entry and of each gene review, skips those the baseline already holds (by PMID, or by DOI in any letter case), fetches the rest from PubMed in batches of 200 or from CrossRef, and stores their metadata with source_type = 'panelapp_citation' and source_details = 'panelapp-snapshot:{panel_date}'. A DOI is stored as PanelApp spells it, and a held paper that lacks a PMID gets the one PanelApp cites it by. It writes no gene mentions: which genes a paper concerns is decided by the run that reads it. A PMID PubMed gives no usable record for, or a DOI CrossRef cannot resolve, is counted and skipped. If PubMed stays unreachable after retries, the command stops with an error; the papers stored until then are kept, and a rerun carries on from there.
For curating literature for a specific panel (e.g., Arthrogryposis):
# Configuration. As above, PANEL_DATE is the day the run is set up.
PANEL_DATE=2025-10-20
PANEL_ID=47 # Arthrogryposis panel ID
PANEL_NAME=arthrogryposis
# 0. Build the PanelApp snapshot, as in the main workflow
uv run palit panelapp snapshot --panel-date $PANEL_DATE
# 1. Copy the baseline's papers to a new database: the screened literature and
# every publication PanelApp cites (see "Updating the Baseline"). Only the
# paper metadata is copied: step 2 assesses relevance afresh and schedules
# the relevant papers for download.
sqlite3 data/$PANEL_NAME.sqlite < schema.sql
sqlite3 data/$PANEL_NAME.sqlite "ATTACH 'data/pubmed_baseline_screening.sqlite' AS source; INSERT INTO papers (doi, pmid, title, abstract, authors, journal, source, source_date, source_metadata, source_type, source_details) SELECT doi, pmid, title, abstract, authors, journal, source, source_date, source_metadata, 'initial', source_details FROM source.papers"
# 2. Assess relevance scoped to the panel: the screen gets the panel description
# and passes papers with human genetic evidence within the panel's phenotypic
# scope, whether or not the panel already lists the gene
uv run palit assess-relevance \
--db-path data/$PANEL_NAME.sqlite \
--panel-date $PANEL_DATE \
--scope-panel-id $PANEL_ID \
--prompt-path prompts/panel_relevance_assessment_prompt.txt
# 3. (Optional) Reduce literature for well-researched genes
# Genes with hundreds of papers make the aggregation prompt long and slow. For panels with well-researched genes (e.g., POLG
# with 200+ papers), use tournament selection to keep only the most informative.
# It counts and selects among the papers step 2 assessed relevant:
uv run palit reduce-literature --db-path data/$PANEL_NAME.sqlite
# 4. Download full-text papers (now reduced set if step 3 was run)
uv run palit download-papers attempt-pmc --db-path data/$PANEL_NAME.sqlite
uv run palit download-papers download-preprints --db-path data/$PANEL_NAME.sqlite
uv run palit download-papers open-browser --db-path data/$PANEL_NAME.sqlite
# ... manually download PDFs ...
uv run palit download-papers register --db-path data/$PANEL_NAME.sqlite
# 5. Extract evidence, scoped to the panel. Scope is decided for each disease
# entity (one disease with one MoI), on the patients the paper reports for
# it: the disease gets an entity only when those individuals have features
# that would lead a clinician to order the panel, judged from what the paper
# reports about them, with any exclusions the panel's description makes. A
# paper saying they have the panel's disorder is enough, and a feature it
# does not mention is unknown, not absent. Another disease of the same gene
# can be left out. An entity without patients is judged on what the paper
# says about the patients who carry its tested variants (their diagnosis or
# features, also when cited from earlier reports). Each entity records why it fits in
# `scope_rationale`: the reported features with how many of its individuals
# have them. The genes with evidence from a relevant paper are the run's
# genes.
uv run palit extract-evidence --db-path data/$PANEL_NAME.sqlite --panel-date $PANEL_DATE --scope-panel-id $PANEL_ID
# 6. Seed the publications PanelApp cites for the run's genes, on any panel
# (see "PanelApp Publication Seeding"). Most are already in the database,
# copied from the baseline in step 1; those the screen rejected are scheduled
# for download all the same. The rest are copied from the baseline or
# fetched.
uv run palit panelapp seed-citations --db-path data/$PANEL_NAME.sqlite --panel-date $PANEL_DATE
# 7. Download and extract the seeded papers, as in steps 4 and 5
uv run palit download-papers attempt-pmc --db-path data/$PANEL_NAME.sqlite
uv run palit download-papers download-preprints --db-path data/$PANEL_NAME.sqlite
uv run palit download-papers open-browser --db-path data/$PANEL_NAME.sqlite --expansion-only
# ... manually download PDFs ...
uv run palit download-papers register --db-path data/$PANEL_NAME.sqlite
uv run palit extract-evidence --db-path data/$PANEL_NAME.sqlite --panel-date $PANEL_DATE --scope-panel-id $PANEL_ID
# A relevant paper extracted in this round (one reduce-literature had left
# out, then seeded because PanelApp cites it) can add run genes. Step 6 runs
# once, so these genes are aggregated without the papers PanelApp cites for
# them.
# 8. Assess genes against the panel, which is the only target panel. Each
# association gets a `scope_rationale` condensed from those of the
# paper-level entities it groups, with the paper each count comes from.
# assess-genes stops when a paper's extraction was not made with the same
# scoping: one without `scope_rationale` here, or one with it in an
# aggregation without --scope-panel-id.
uv run palit assess-genes --db-path data/$PANEL_NAME.sqlite --panel-date $PANEL_DATE --scope-panel-id $PANEL_ID
# 9. Map associations without a GenCC term onto MONDO terms, as in the main
# workflow. Panel-scoped runs do not run match-panels.
uv run palit map-mondo --db-path data/$PANEL_NAME.sqlite
# 10. Generate report package against the panel: every association of a gene
# not on it is a finding, and an upgrade starts from the gene's rating on it.
# Each association shows its `scope_rationale` as "Panel fit". A gene
# unreviewed on the panel without an association is listed as in the main
# workflow; the group whose papers gave no disease entity reads "Papers
# read, none fit the panel". That judgement covers those papers' patients
# only; whether the entry belongs on the panel is the curator's call.
uv run palit generate-report \
--report-id panel_$PANEL_NAME \
--db-path data/$PANEL_NAME.sqlite \
--panel-date $PANEL_DATE \
--scope-panel-id $PANEL_ID