Research

Learning-based techniques are a powerful and efficient way to design versatile controllers that are able to adapt to complex dynamic environments rapidly. In contrast to classical control approaches, learning does not require comprehensive domain knowledge but distills this knowledge from data. Thus, learning techniques are generalizable to a broader range of problems. Yet, learning-based techniques usually lack theoretical guarantees that are essential in safety-critical robotic applications.

In my research, I aim to unlock the potential of learning-based techniques for cyber-physical systems by incorporating safety guarantees into learning of controllers for autonomous systems. I currently focus on integrating safety specifications formulated via formal methods – e.g., temporal logic and reachability analysis – into deep reinforcement learning. To evaluate the performance of my theoretical approaches, I validate their effectiveness and efficiency in various motion-planning tasks, including autonomous driving, unmanned aerial vehicles, and mobile robots.

In a nutshell, I work at the intersection of reinforcement learning, formal methods, and robotics. This work can be clustered into three strands:

1. Provably safe reinforcement learning: Develop reinforcement learning algorithms that provide absolute guarantees with respect to safety requirements.
2. Formal methods for system safety: Design a language for formal safety specifications that is versatile and well-suited for cyber-physical systems.
3. Motion planning for cyber-physical systems: Validate theoretical results on safety-critical motion planning tasks using real-world data or physical experiments on autonomous systems.

Selected Publications

2024

1. 

   Provable Traffic Rule Compliance in Safe Reinforcement Learning on the Open Sea

   Hanna Krasowski, and Matthias Althoff

   IEEE Transactions on Intelligent Vehicles, 2024

   Abs arXiv Bib PDF

   For safe operation, autonomous vehicles have to obey traffic rules that are set forth in legal documents formulated in natural language. Temporal logic is a suitable concept to formalize such traffic rules. Still, temporal logic rules often result in constraints that are hard to solve using optimization-based motion planners. Reinforcement learning (RL) is a promising method to find motion plans for autonomous vehicles. However, vanilla RL algorithms are based on random exploration and do not automatically comply with traffic rules. Our approach accomplishes guaranteed rule-compliance by integrating temporal logic specifications into RL. Specifically, we consider the application of vessels on the open sea, which must adhere to the Convention on the International Regulations for Preventing Collisions at Sea (COLREGS). To efficiently synthesize rule-compliant actions, we combine predicates based on set-based prediction with a statechart representing our formalized rules and their priorities. Action masking then restricts the RL agent to this set of verified rule-compliant actions. In numerical evaluations on critical maritime traffic situations, our agent always complies with the formalized legal rules and never collides while achieving a high goal-reaching rate during training and deployment. In contrast, vanilla and traffic rule-informed RL agents frequently violate traffic rules and collide even after training.

   @article{Krasowski2024.safeRLautonomousVessels,
     archiveprefix = {arXiv},
     title = {Provable Traffic Rule Compliance in Safe Reinforcement Learning on the Open Sea},
     author = {Krasowski, Hanna and Althoff, Matthias},
     year = {2024},
     journal = {IEEE Transactions on Intelligent Vehicles},
     volume = {},
     pages = {1--18},
     doi = {10.1109/TIV.2024.3400597},
     issn = {2379-8904},
   }

2. 

   Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking

   Roland Stolz*, Hanna Krasowski*, Jakob Thumm, Michael Eichelbeck, Philipp Gassert, and Matthias Althoff

   In Proc. of the Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024

   Abs arXiv PDF Code Video

   Continuous action spaces in reinforcement learning (RL) are commonly defined as interval sets. While intervals usually reflect the action boundaries for tasks well, they can be challenging for learning because the typically large global action space leads to frequent exploration of irrelevant actions. Yet, little task knowledge can be sufficient to identify significantly smaller state-specific sets of relevant actions. Focusing learning on these relevant actions can significantly improve training efficiency and effectiveness. In this paper, we propose to focus learning on the set of relevant actions and introduce three continuous action masking methods for exactly mapping the action space to the state-dependent set of relevant actions. Thus, our methods ensure that only relevant actions are executed, enhancing the predictability of the RL agent and enabling its use in safety-critical applications. We further derive the implications of the proposed methods on the policy gradient. Using Proximal Policy Optimization (PPO), we evaluate our methods on three control tasks, where the relevant action set is computed based on the system dynamics and a relevant state set. Our experiments show that the three action masking methods achieve higher final rewards and converge faster than the baseline without action masking.

2023

1. 

   Provably Safe Reinforcement Learning via Action Projection Using Reachability Analysis and Polynomial Zonotopes

   Niklas Kochdumper*, Hanna Krasowski*, Xiao Wang*, Stanley Bak, and Matthias Althoff

   IEEE Open Journal of Control Systems, 2023

   Abs arXiv Bib PDF Code Video

   While reinforcement learning produces very promising results for many applications, its main disadvantage is the lack of safety guarantees, which prevents its use in safety-critical systems. In this work, we address this issue by a safety shield for nonlinear continuous systems that solve reach-avoid tasks. Our safety shield prevents applying potentially unsafe actions from a reinforcement learning agent by projecting the proposed action to the closest safe action. This approach is called action projection and is implemented via mixed-integer optimization. The safety constraints for action projection are obtained by applying parameterized reachability analysis using polynomial zonotopes, which enables to accurately capture the nonlinear effects of the actions on the system. In contrast to other state-of-the-art approaches for action projection, our safety shield can efficiently handle input constraints and dynamic obstacles, eases incorporation of the spatial robot dimensions into the safety constraints, guarantees robust safety despite process noise and measurement errors, and is well suited for high-dimensional systems, as we demonstrate on several challenging benchmark systems.

   @article{Kochdumper2023.safeRLReachabilityAnalysis,
     author = {Kochdumper*, Niklas and Krasowski*, Hanna and Wang*, Xiao and Bak, Stanley and Althoff, Matthias},
     journal = {IEEE Open Journal of Control Systems},
     title = {Provably Safe Reinforcement Learning via Action Projection Using Reachability Analysis and Polynomial Zonotopes},
     year = {2023},
     volume = {2},
     pages = {79-92},
     doi = {10.1109/OJCSYS.2023.3256305},
   }

2. 

   Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking

   Hanna Krasowski*, Jakob Thumm*, Marlon Müller, Lukas Schäfer, Xiao Wang, and Matthias Althoff

   Transactions on Machine Learning Research, 2023

   Abs arXiv Bib PDF Code

   Ensuring the safety of reinforcement learning (RL) algorithms is crucial to unlock their potential for many real-world tasks. However, vanilla RL and most safe RL approaches do not guarantee safety. In recent years, several methods have been proposed to provide hard safety guarantees for RL, which is essential for applications where unsafe actions could have disastrous consequences. Nevertheless, there is no comprehensive comparison of these provably safe RL methods. Therefore, we introduce a categorization of existing provably safe RL methods, present the conceptual foundations for both continuous and discrete action spaces, and empirically benchmark existing methods. We categorize the methods based on how they adapt the action: action replacement, action projection, and action masking. Our experiments on an inverted pendulum and a quadrotor stabilization task indicate that action replacement is the best-performing approach for these applications despite its comparatively simple realization. Furthermore, adding a reward penalty, every time the safety verification is engaged, improved training performance in our experiments. Finally, we provide practical guidance on selecting provably safe RL approaches depending on the safety specification, RL algorithm, and type of action space.

   @article{Krasowski2023b.ProvablySafeRLSurvey,
     author = {Krasowski*, Hanna and Thumm*, Jakob and Müller, Marlon and Schäfer, Lukas and Wang, Xiao and Althoff, Matthias},
     title = {Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking},
     year = {2023},
     journal = {Transactions on Machine Learning Research},
     issn = {2835-8856},
   }