Reproduce and compare runs

Reproduce a run

Every replication writes an event trace whose SHA-256 digest is recorded in manifest.json. Running the same WorldSpec with the same model lock on the same runtime (Python, numpy and image versions are in manifest.runtime) produces identical digests.

uv run alive run --world fixtures/worlds/consumer-app.json --output /tmp/alive-a.zip --json > /dev/null
uv run alive run --world fixtures/worlds/consumer-app.json --output /tmp/alive-b.zip --json > /dev/null
unzip -p /tmp/alive-a.zip manifest.json | python3 -c "import json,sys;print([t['digest'] for t in json.load(sys.stdin)['trials']][:2])" > /tmp/alive-a.txt
unzip -p /tmp/alive-b.zip manifest.json | python3 -c "import json,sys;print([t['digest'] for t in json.load(sys.stdin)['trials']][:2])" > /tmp/alive-b.txt
diff /tmp/alive-a.txt /tmp/alive-b.txt && echo "identical"

Identity across different operating systems or library versions is not promised: numerical libraries may change how random numbers are transformed between releases.

Use completed trials for this comparison. Wall-clock limits can stop a run at different points even with the same configuration, so timed-out runs may contain different completed trial sets. Archive timestamps, measured wall time and ZIP container bytes also vary; compare canonical event-trace digests rather than requiring the entire archive to be byte-identical.

Compare a change fairly

Put the alternatives in one WorldSpec as variants rather than running two separate worlds. Within a world, every variant sees the same synthetic people and the same per-person random draws, so the paired comparison isolates the effect of the change. For example, to test a third navigator, add a treatment variant with an override:

{
  "id": "three_navigators",
  "role": "treatment",
  "label": "Three navigators",
  "overrides": [{ "target": "navigators.staff", "value": 3 }]
}

The findings report the paired difference with an interval. If the interval includes zero, the finding says the difference is not distinguishable from simulation noise; add replications before reading anything into it.

Read a hosted comparison

After a run completes, POST /v1/experiments/compare accepts {"run_id": "run_…"} with a read-scoped bearer token. It returns run_id, world_revision_id, status, assumption_parameters and metric summaries containing the stored baseline/treatment pairs. This request reads existing results; it does not run an experiment or reserve credits.

The alternatives must have run as variants of one WorldSpec with the same model lock, seed and runtime. Comparisons between unrelated run ids are not supported. For a timed-out run, inspect n_pairs and interval: too few defined pairs leave the interval null. No stored comparison, or a queued/running/failed/canceled run, returns CONFLICT (409). An inaccessible run returns NOT_FOUND (404). The same comparisons appear in the run results and ZIP export.