Real-time reinforcement learning with online training for large-scale facilities

by Luca Scomparin

Friday Mar 14, 2025, 2:00 PM → 4:30 PM Europe/Berlin

413 (IPE Bldg. 242)

Luca Scomparin will present a preliminary version of his doctoral thesis defense on Real-time reinforcement learning with online training for large-scale facilities

Reinforcement learning (RL) offers powerful solutions for control problems, but training agents in real-world environments is challenging due to data requirements and strict real-time constraints. To address this, the experience accumulator architecture was developed, allowing real-time agents to interact with their environment while recording data for offline training. This reduces real-time computational demands and increases system flexibility by enabling the exploration of different reward functions. Additionally, this approach allows the reuse of standard RL algorithm implementations, avoiding the need for specialized real-time adaptations.

To facilitate RL deployment in large-scale facilities, the KINGFISHER platform was designed and implemented on AMD/Xilinx Versal hardware. This modular system provides low-latency processing and standardized interfaces for integration with experimental setups. KINGFISHER was tested at the KARA synchrotron, where it successfully controlled betatron oscillations, first validating a conventional feedback controller and then deploying RL-based controllers that often outperformed traditional methods. The system also tackled the more complex challenge of microbunching instability, demonstrating both the feasibility and adaptability of RL-based control in high-speed, real-world environments.