VALIDATION · CALIBRATION

Is the inference
actually correct?

Simulation-Based Calibration: draw θ from the prior, simulate data, run inference, rank the truth among posterior draws. A calibrated procedure produces uniform ranks. Pick an engine below; each tab shows its KS distance to uniform per model cell and quantity, ranked by PIT.

How to read it

PIT RANKING

Scalar hyperparameters ranked by their PIT, cdf(marginal, truth). Required for grid/Laplace engines — INLA draws θ from a few integration-grid points, so naive sample ranks pick up a spurious staircase.

VERDICT (effect-size tiered)

within band = passes the Säilynoja et al. (2022) 95% simultaneous-band ECDF test. At ~10³ replicates that test detects even tiny error, so a fail is tiered by KS: minor ≤ 0.10 (approximation-level, fine), substantial > 0.10. The sparkline plots each cell's rank ECDF minus uniform against that band — a curve hugging zero inside the shaded band is calibrated.

REGIMES

Well-identified (more data informs the variance component) vs weak-identification stress (heavy prior, little data). Calibration is expected to degrade under stress.

:loglik

A joint data-dependent test quantity — the observation log-likelihood at (θ, x) — catching joint miscalibration the per-hyperparameter ranks miss.

Full nested-Laplace over a hyperparameter grid. The flagship — calibrated when the model is well-posed.

Well-identified

n_nodes 100 · PC u 1 · 500 replicates · 4 within band · 7 minor · 3 substantial

cell	quantity	KS	verdict
poisson_iid	τ	0.076	minor
poisson_iid	loglik	0.058	within band
poisson_rw1	τ	0.054	minor
poisson_rw1	loglik	0.028	within band
bernoulli_iid	τ	0.057	minor
bernoulli_iid	loglik	0.058	within band
bernoulli_rw1	τ	0.065	minor
bernoulli_rw1	loglik	0.034	within band
normal_iidnon-id	τ	0.401	substantial
normal_iidnon-id	σ	0.333	substantial
normal_iidnon-id	loglik	0.305	substantial
normal_rw1	τ	0.063	minor
normal_rw1	σ	0.058	minor
normal_rw1	loglik	0.047	minor

Weak-identification stress

n_nodes 30 · PC u 1 · 1000 replicates · 4 within band · 7 minor · 3 substantial

cell	quantity	KS	verdict
poisson_iid	τ	0.070	minor
poisson_iid	loglik	0.038	within band
poisson_rw1	τ	0.073	minor
poisson_rw1	loglik	0.026	within band
bernoulli_iid	τ	0.069	minor
bernoulli_iid	loglik	0.027	within band
bernoulli_rw1	τ	0.072	minor
bernoulli_rw1	loglik	0.029	within band
normal_iidnon-id	τ	0.302	substantial
normal_iidnon-id	σ	0.267	substantial
normal_iidnon-id	loglik	0.220	substantial
normal_rw1	τ	0.075	minor
normal_rw1	σ	0.067	minor
normal_rw1	loglik	0.046	minor

Notes

SBC ranked by PIT (cdf(marginal, truth)) for INLA/TMB; required because INLA's posterior θ is grid-quantized.
Verdict is effect-size tiered. ‘within band’ = passes the Säilynoja, Bürkner & Vehtari (2022) ECDF test with 95% SIMULTANEOUS confidence bands (rank ECDF inside the band at every point at once). At n≈10³ replicates that test detects even tiny error, so a cell failing it is tiered by KS effect size: ‘minor’ = KS ≤ 0.10 (≤10% max-CDF deviation — approximation-level, practically fine), ‘substantial’ = KS > 0.10. This separates an approximate-but-usable method from a genuinely-off one; a pure significance verdict would flag every approximation at this n.
hmc_laplace runs a leaner NUTS chain per replicate (offline cost); the well-identified regime (n=100 nodes) is prohibitively slow for per-replicate NUTS and is omitted.
Gaussian-IID (tagged ‘non-identified’) is a deliberate stress case: y~N(x,σ), x~N(0,1/τ) ⇒ only σ²+1/τ is identified. SBC against the EXACT posterior here is uniform (reference + harness validated), and a faithful sampler (hmc_laplace) recovers it (KS ~0.06); INLA's grid integration of the degenerate ridge does not (a finer grid barely helps), an inherent limit of grid-based hp exploration, not an implementation error. RW1 structure breaks the degeneracy and all engines recover.

Generated 2026-06-07 01:19 · benchmark/render_validation.jl

References

SBC

Validating Bayesian Inference Algorithms with Simulation-Based Calibration

S. Talts, M. Betancourt, D. Simpson, A. Vehtari & A. Gelman

arXiv preprint · 2018 · arXiv:1804.06788

Introduces SBC: under exact inference, the rank of each prior-drawn parameter within its posterior is uniform, so non-uniform ranks diagnose miscalibration of any Bayesian computation.

Calibration test

Graphical test for discrete uniformity and its applications in goodness-of-fit evaluation and multiple sample comparison

T. Säilynoja, P.-C. Bürkner & A. Vehtari

Statistics and Computing · 2022 · doi:10.1007/s11222-022-10090-6

The ECDF-difference test with exact simultaneous confidence bands used here for the verdict — controlling family-wise error across the whole curve, the principled way to test SBC rank uniformity.