Wang Ma马旺

NeurIPS 2026First author

Black-Box Uncertainty Quantification for Large Language Models via Ensemble-of-Ensembles

Wang Ma, Debarun Bhattacharjya, Junkyu Lee, Nhan H Pham, Harsha Kokel, Qiang Ji

TL;DRA two-level ensemble estimates an LLM's uncertainty from its text outputs alone and splits it into aleatoric and epistemic parts. No weights or logits are needed, and AUROC is comparable to or better than white-box baselines.

Uncertainty quantification (UQ) is essential for building reliable and trustworthy systems with large language models (LLMs). However, conventional Bayesian or ensemble-based UQ methods are computationally intractable at the scale of modern LLMs and often require white-box access to model parameters or logits. This paper introduces a two-level ensemble framework for black-box uncertainty estimation that operates entirely at inference time and is theoretically grounded in the law of total variance, decomposing total predictive uncertainty into aleatoric and epistemic components. The inner ensemble captures stochasticity and ambiguity through repeated stochastic decoding, while the outer ensemble approximates parameter-driven uncertainty via semantically perturbed prompts that act as proxy samples from an implicit posterior over plausible inputs. By measuring variability in a continuous embedding space, our framework yields interpretable and scalable uncertainty estimates across diverse LLMs. We further provide finite-sample guarantees for our uncertainty estimators, and bound the gap with Bayesian estimators. Experiments on standard benchmarks show that our black-box estimator achieves AUROC comparable to or surpassing state-of-the-art white-box baselines, while providing a meaningful decomposition that distinguishes linguistic ambiguity from knowledge uncertainty.

CVPR 2026HighlightFirst author

Towards Knowledge-augmented Bayesian Deep Learning For Computer Vision

Wang Ma, Hanjing Wang, Yufei Zhang, Darsha Udayanga, Qiang Ji

TL;DRDomain knowledge enters Bayesian deep learning twice: first as an informative prior, then as an adaptive likelihood during inference. This improves accuracy, robustness, adaptation and uncertainty estimation over Bayesian and knowledge-augmented baselines.

Bayesian deep learning (BDL) integrates Bayesian inference with deep learning, improving predictive performance while enabling principled uncertainty quantification. However, existing BDLs often rely on non-informative random priors, limiting the benefits of Bayesian inference. In contrast, knowledge-augmented deep learning explicitly injects domain knowledge during training, yet lacks a probabilistic foundation. In this paper, we propose a knowledge-augmented BDL framework that integrates domain knowledge both as an informative prior and as an adaptive likelihood under a unified two-stage hybrid formulation. In the first stage, we learn a knowledge-informed prior p(θ|K) by pre-training a model to satisfy domain-specific constraints. In the second stage, we perform Bayesian inference on task data with an adaptive knowledge likelihood p(K|θ,D), which dynamically enforces these constraints during optimization. This unified framework enables knowledge to guide both initialization and training, significantly improving prediction accuracy, robustness, adaptation and uncertainty estimation. Experiments on various computer vision tasks, including semi-synthetic and real-knowledge scenarios, demonstrate that our two-stage framework consistently outperforms state-of-the-art Bayesian and knowledge-augmented baselines.