How Alive evaluates, and what it will not claim

1. Adaptive intake: what you have, and what we actually read

Alive starts from whatever you bring. For every source it records whether it was processed, unsupported, excluded for privacy, or failed, and it never claims to have read something it did not. Each statement about your project is labelled as observed (shown by an artifact such as an API contract), user declared (stated in your materials), inferred, or unknown, and linked to where it came from. AI may draft this profile; Alive checks every citation and quote against the source text and you confirm the result.

2. When Alive chooses not to simulate

A rule-based planner decides which evaluations apply. An idea is evaluated with an evidence review, not a simulation: simulating something that does not yet have a defined workflow would produce numbers with nothing behind them. The planner explains every tool it recommends, skips, or cannot offer yet, and never recommends a tool that does not exist.

3. Evidence reviews and citations

Large language models frequently cite sources that do not support their statements, even with retrieval: one automated evaluation found 50–90% of medical answers were not fully supported by their own citations.^[6] Alive therefore never lets a model write references. It searches bibliographic registries itself, checks each record for retractions and corrections, and lets the model cite only those records. A claim counts as supported only if its quote appears verbatim in openly licensed text and a second, independent check agrees. Every review dimension is labelled an AI hypothesis until a named person confirms it.

4. What a synthetic world is, and is not

Alive's worlds are simulation models: explicit, inspectable descriptions of people, staff, processes and systems, composed from a versioned catalog. They are not digital twins. The National Academies define a digital twin as a virtual representation that is continuously updated with data from a specific real system and used to inform decisions about it.^[1] Alive's worlds are built from synthetic data and published assumptions, are connected to no real clinic, patient or device, and are meant for exploring “what if”, not for predicting what will happen in a particular place.

Synthetic populations can reproduce broad demographics while getting outcomes wrong. A published validation of Synthea, a widely used generator, found for example 0% blood pressure control among synthetic patients against 74.5% in real Massachusetts care data; its authors describe the data as suited to non-clinical uses.^[2][3] Alive reports results as comparisons between scenarios under labelled assumptions, not as estimates of real-world rates.

5. Model primitives and provenance

Every world is built from six kinds of component: actors, organizations, environments, processes, systems and evaluators. Each model has a card stating its purpose, boundary, assumptions and known failure regimes (see models). Every quantitative parameter records its provenance: measured, public evidence (with citations), expert assumption, or synthetic assumption. An AI can never be a provenance source. Model locks pin the exact code of every model used in a run.

6. Verified results, modeled outcomes and hypotheses

Results are always typed. A verified failure is an assertion that failed in a run's event trace, with the event ids to prove it. A modeled outcome is a metric or comparison that depends on the world's assumptions and says how many it rests on. An AI hypothesis is a reasoned starting point to test. An expert assessment is a named person's judgement. There is no overall readiness score.

7. Replication, baselines and uncertainty

Every scenario is run many times with independent random seeds, and results are reported as means with 95% confidence intervals, never a single run.^[5] Comparisons use a declared baseline and common random numbers: within a world, every variant sees the same synthetic people and the same per-person random draws, so the difference reflects the change rather than chance. When an interval includes zero, the result says the variants cannot be distinguished. Runs are deterministic: the same world, model lock and runtime reproduce the event trace byte for byte, with a published digest per replication.

8. Evidence grades

Grades apply to a metric for a stated use, following simulation validation practice that a model is valid only for a purpose, never in general.^[4] L1: executed and tested. L2: internally verified. L3: calibrated against documented external data. L4: externally validated with disclosed prediction errors. These are Alive's internal terms, not an external certification. Every current model is L1. The compiler refuses any world that claims a higher grade than its models have earned. See validation.

9. Use of AI judges

AI models draft reviews and check whether quotes support claims. Their agreement with human experts varies widely across tasks, so Alive trusts them asymmetrically: an AI verdict that a citation does not support a claim is accepted, while a verdict that it does requires a verbatim quote and a second independent check. Before any AI judgement is used without human review, its agreement with human reviewers will be measured on Alive's own cases and published.

10. What Alive is not

• Not a regulatory certification, and not evidence of clinical benefit or safety.
• Not a diagnostic or clinical decision support system; it makes no clinical recommendations.
• Not a substitute for research with real patients and users.
• Not connected to any production health system; it works with synthetic data only.

11. Ownership, privacy and research sharing

Projects are private to your organization. Sponsors who fund credits cannot see the projects they fund. Research sharing is off by default; you may opt in to aggregate or expanded sharing and revoke it later. Alive does not train models on your submissions. See privacy.

12. Versions and change

Schemas, models and the tool registry are versioned, and every run records the exact versions it used. Changes to public contracts are listed in the changelog.

References

1. National Academies of Sciences, Engineering, and Medicine (2024). Foundational Research Gaps and Future Directions for Digital Twins. doi.org/10.17226/26894
2. Chen J, Chun D, Patel M, Chiang E, James J (2019). The validity of synthetic clinical data: a validation study of a leading synthetic data generator (Synthea) using clinical quality measures. BMC Med Inform Decis Mak. doi.org/10.1186/s12911-019-0793-0
3. Walonoski J, Kramer M, Nichols J, et al. (2018). Synthea: an approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record. JAMIA. doi.org/10.1093/jamia/ocx079
4. Eddy DM, Hollingworth W, Caro JJ, et al. (2012). Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Value in Health. doi.org/10.1016/j.jval.2012.04.012
5. Law AM (2023). Design and analysis of simulation experiments using three simple statistical formulas. Proc. Winter Simulation Conference. www.informs-sim.org/wsc23papers/124.pdf
6. Wu K, Wu E, Wei K, et al. (2025). An automated framework for assessing how well LLMs cite relevant medical references. Nature Communications. doi.org/10.1038/s41467-025-58551-6