Methodology

How answers are generated

• Mosaic side. The question is run through Mosaic's MCP tools over the oncology knowledge graph. The exact tool calls and their structured outputs are captured verbatim (scripts/capture_demo.py) and shown in the "How Mosaic answered this" panel.
• LLM baseline side. The same question is run through general-purpose models with web search enabled. Prompts are kept minimal — no prompt-engineering competition — and recorded verbatim alongside the answer (scripts/capture_baselines.py).
• Identical question, identical prompt. Both baselines receive the same prompt; only the model differs. We do not curate retries or pick the best of N — the first response is what ships.

Prompts shown verbatim

For egfr-resistance-bypass (one example; every demo carries its own prompts in the JSON):

You are a pre-clinical drug discovery analyst. Answer concisely (under 250 words). Cite the sources you used. If you do not know, say so rather than guess.

Question: Which targets are implicated in resistance to EGFR inhibitors, and which of those are still development whitespace?

You are a pre-clinical drug discovery analyst. Answer concisely (under 250 words). Cite the sources you used. If you do not know, say so rather than guess.

Question: Which targets are implicated in resistance to EGFR inhibitors, and which of those are still development whitespace?

Claude additionally enables the web_search_20250305 tool with max_uses=5; GPT-4o uses its built-in search via the -search-preview model variant. No system prompt, no retrieval prefixes, no few-shots.

Versions & dates

KG version 107 · Mosaic run 2026-07-18 · baselines 2026-06-02 · claude-sonnet-4-6 · gpt-4o-search-preview

Each demo carries its own timestamps; re-capture cadence is quarterly. Stale demos are flagged by scripts/validate_demos.py.

BENCH — quantitative with-vs-without-MCP

The on-page demo carousel shows qualitative side-by-sides. The paper's headline benchmark is a separate, larger run on a 12-question hand-curated pool, scored by an LLM judge plus mechanical identifier validation. The treated arm calls Mosaic MCP tools; the baseline arm uses the same model with no tools.

The headline finding is a methods contribution, not a marketing one: LLM judges with training cutoffs that predate the KG's coverage window systematically penalise real-but-recent identifiers as "fabricated future dates". A four-line judge-prompt fix (pass today's date + state the KG's coverage window) flipped 4 of 12 questions and raised the average grounding score by +0.92. Mechanical NCT/PMID validation found zero invalid identifiers in either arm at any point — the bias is verification-affordance, not fabrication.

Full write-up: eval/results/bench_findings.md in the repo, §5.3 and §7.3 of the manuscript.

Limitations

• Pre-recorded, small curated sample — not a live benchmark. The on-page demo is N=1 today; the full benchmark above is N=12.
• Oncology only; results do not generalise to other therapy areas.
• Several Mosaic layers are partial/heuristic (semantic relations, resistance, modality, synthetic-lethal proxy) and labelled as such in tool output — treat demos as capability illustrations, not evidence.
• BENCH pool is 1st-order lookup-style questions where training-data recall is most competitive. KG-native showcase tools (synthetic-lethal whitespace, resistance bypass maps, modality gaps) are not exercised by the headline benchmark yet — that's future work (M3.5).
• Where an LLM baseline answers well, it is shown unedited — the comparison is not curated to favour Mosaic.

Reproducing this

The repo ships a Docker reproducibility package (paper/reproducibility/) and a deterministic capture toolchain. From a fresh checkout: scripts/capture_demo.py for the Mosaic side, scripts/capture_baselines.py for the LLM side, scripts/bench_with_without_mcp.py for the 12-question benchmark. All outputs are committed JSON; a re-run with the same KG snapshot is bit-identical aside from model nondeterminism in the prose answers.