Learning-based bayesian inference for testing of autonomous systems

REALM researchers propose learning-based models for fast and efficient discovery of failures in autonomous systems prior to deployment, validated by demonstrations on two robotic system platforms.

Authors: Anjali Parashar, Ji Yin, Charles Dawson, Panagiotis Tsiotras, Chuchu Fan 
Citation: IEEE Robotics and Automation Letters (RA-L), Volume 9, Issue 10

Abstract:
For the safe operation of robotic systems, it is important to accurately understand its failure modes using prior testing. Hardware testing of robotic infrastructure is known to be slow and costly. Instead, failure prediction in simulation can help to analyze the system before deployment. Conventionally, large-scale naïve Monte Carlo simulations are used for testing; however, this method is only suitable for testing average system performance. For safety-critical systems, worst-case performance is more crucial as failures are often rare events, and the size of test batches increases substantially as failures become rarer. Rare-event sampling methods can be helpful; however, they exhibit slow convergence and cannot handle constraints. 

This research introduces a novel sampling-based testing framework for autonomous systems which bridges these gaps by utilizing a discretized gradient-based second-order Langevin algorithm combined with learning-based techniques for constrained sampling of failure modes. Our method can predict more diverse failures by exploring the search space efficiently and ensures feasibility with respect to temporal and implicit constraints. We demonstrate the use of our testing methodology on two categories of testing problems, via simulations and hardware experiments. Our method discovers up to 2X failures compared to naïve Random Walk sampling, with only half of the sample size.