Boosting Certified Robustness of Deep Networks via a Compositional Architecture

A core challenge with existing certified defense mechanisms is that while they improve certified robustness, they also tend to drastically decrease standard accuracy, making it difficult to use these methods in practice. In this work, we propose a new architecture which addresses this challenge and enables one to boost the certified robustness of any state-of-the-art deep network, while controlling the overall accuracy loss, without requiring retraining. The key idea is to combine this model with a (smaller) certified network where at inference time, an adaptive selection mechanism decides on the network to process the input sample. The approach is compositional: one can combine any pair of state-of-the-art (e.g., EfficientNet or ResNet) and certified networks, without restriction. The resulting architecture enables much higher standard accuracy than previously possible with certified defenses alone, while substantially boosting the certified robustness of deep networks. We demonstrate the effectiveness of this adaptive approach on a variety of datasets and architectures. For instance, on CIFAR-10 with an $\ell_\infty$ perturbation of 2/255, we are the first to obtain a high standard accuracy (91.6%) with non-trivial certified robustness (22.8%). Notably, prior state-of-the-art methods incur a substantial drop in accuracy (77.4%) for a similar certified robustness (16.5%).