Can reinforcement learning adapt seamlessly to the complexities of continuous environments? This research introduces a novel framework for reinforcement learning (RL) that operates directly in continuous time and space, eliminating the need for prior discretization. This is crucial for tasks where precision and real-time adaptability are paramount. The core of the method lies in minimizing a continuous-time form of the temporal difference (TD) error, derived from the Hamilton-Jacobi-Bellman (HJB) equation. The authors develop update methods using backward Euler approximation and exponential eligibility traces, drawing parallels with traditional algorithms like residual gradient and TD(λ). They also formulate two policy improvement approaches: a continuous actor-critic method and a value-gradient-based greedy policy. These algorithms are valuable tools for various control and optimization problems. Simulations on pendulum swing-up and cart-pole swing-up tasks demonstrate the superiority of the proposed algorithms, particularly the value-gradient-based policy with a learned dynamic model, in terms of both trial count and efficiency. The results suggest potential applications in robotics, autonomous systems, and other fields requiring precise control and real-time adaptation. This research paves the way for more efficient and robust RL solutions in complex, continuous environments, pushing the boundaries of what autonomous agents can achieve.
Published in Neural Computation, a journal covering computational and mathematical approaches to understanding the brain and nervous system, this paper directly contributes to the journal's focus on reinforcement learning algorithms. By developing methods applicable to continuous-time dynamical systems, the research addresses key challenges in neural computation and control systems.