Continuous-time adaptive critics

T. Hanselmann; L. Noakes; A. Zaknich

doi:10.1109/TNN.2006.889499

Back

Journal article

Open access

Peer reviewed

Continuous-time adaptive critics

T. Hanselmann, L. Noakes and A. Zaknich

IEEE Transactions on Neural Networks, Vol.18(3), pp.631-647

2007

DOI: https://doi.org/10.1109/TNN.2006.889499

Files and links (2)

pdf

continuous-time_adaptive_critics.pdfDownload View

Published (Version of Record) Open Access

url

Link to Published Version *Subscription may be requiredView

Abstract

A continuous-time formulation of an adaptive critic design (ACD) is investigated. Connections to the discrete case are made, where backpropagation through time (BPTT) and real-time recurrent learning (RTRL) are prevalent. Practical benefits are that this framework fits in well with plant descriptions given by differential equations and that any standard integration routine with adaptive step-size does an adaptive sampling for free. A second-order actor adaptation using Newton's method is established for fast actor convergence for a general plant and critic. Also, a fast critic update for concurrent actor-critic training is introduced to immediately apply necessary adjustments of critic parameters induced by actor updates to keep the Bellman optimality correct to first-order approximation after actor changes. Thus, critic and actor updates may be performed at the same time until some substantial error build up in the Bellman optimality or temporal difference equation, when a traditional critic training needs to be performed and then another interval of concurrent actor-critic training may resume

Details

Title: Continuous-time adaptive critics
Authors/Creators: T. Hanselmann (Author/Creator) - The University of Melbourne
L. Noakes (Author/Creator) - School of Mathematics and Statistics
A. Zaknich (Author/Creator) - Murdoch University
Publication Details: IEEE Transactions on Neural Networks, Vol.18(3), pp.631-647
Publisher: IEEE
Identifiers: 991005544378507891
Murdoch Affiliation: School of Engineering
Language: English
Resource Type: Journal article

Metrics

450 File views/ downloads

85 Record Views

97 Times Cited - Web of Science

InCites Highlights

These are selected metrics from InCites Benchmarking & Analytics tool, related to this output

Collaboration types: Domestic collaboration
Citation topics: 4 Electrical Engineering, Electronics & Computer Science; 4.116 Robotics; 4.116.862 Reinforcement Learning
Web Of Science research areas: Computer Science, Artificial Intelligence; Computer Science, Hardware & Architecture; Computer Science, Theory & Methods; Engineering, Electrical & Electronic
ESI research areas: Engineering