2024-06-21
このページでは や で出てくる連続時間のTD誤差について考えていきます.
連続時間のTD誤差は以下の数式で表されます.このページではこの式の導出までを行います. \begin{align} \delta_{TD} = \frac{d}{dt} V^\pi(s(t)) – \frac{1}{\tau_r} V^\pi(s(t)) + r(s(t),a(t)) \end{align}
連続時間のTD誤差について考えることは連続な時間の中で神経細胞のネットワークがどのように学習するのかに関連するのではないかと思います.
離散時間の場合
まず復習から始めます.強化学習の基本のキである離散時間のTD誤差の計算は以下のような式で書かれます.
\begin{align} \delta_{TD} = r_t + \gamma V(s_{t+1}) -V(s_t) \end{align}
このとき\(V(s_t)\)は価値関数と呼ばれ,時刻\(t\)において状態\(s_t\)にいたときに将来得られると予想していた報酬の割引累積和の期待値です.このとき,将来どのような報酬が得られるかは将来とる行動に依存するので,行動の決定規則(方策)\(\pi\)を明示して以降は\(V^\pi\)のように書きます.
そのため\(V^\pi(s_t)\)は以下のように定義されています.
\begin{align} V^\pi(s_t) := E[r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \gamma^3 r_{t+3} + \cdots]_\pi \end{align}
少し工夫すると一つ後の時刻の価値\(V^\pi(s_{t+1})\)を使って以下のように書けます.
\begin{align} V^\pi(s_t) &= E[r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \gamma^3 r_{t+3} + \cdots]_\pi \\ &= r_t + \gamma (E[r_{t+1} + \gamma r_{t+2} + \gamma^2 r_{t+3} + \cdots]_\pi )\\ &= r_t + \gamma V^\pi(s_{t+1}) \end{align}
つまり\(V^\pi(s_t)\)が正しい関数の場合,上記の式の\(V^\pi(s_t)\)を右辺に移項すると
\begin{align} 0 = r_t + \gamma V^\pi(s_{t+1}) – V^\pi(s_t) \end{align}
となり,時刻\(t\)で予想していたその状態\(s_t\)の価値と実際に状態\(s_{t+1}\)に遷移して報酬\(r_t\)を受け取った際にもう一度計算した価値が一致します.
しかし,実際の学習の初期にはある状態がどの程度の価値を持っているのか正しく予測できないので実際遷移した後に計算した価値\(r_t + \gamma V^\pi(s_{t+1})\)と遷移前に予想していた価値\(V^\pi(s_t)\)の間に差\(\delta_{TD}\)が生じます.
この誤差のことをTemporal Difference error (TD誤差)と呼びます.
TD誤差が正であれば遷移の結果が予想よりも良かったことを意味し,負であれば予想よりも悪かったことを意味します.そのため以下のように価値関数\(V\)を更新することで次第に正確な価値関数に近づいていきます.
\begin{align} V^\pi_{i+1}(s_t) \leftarrow V^\pi_{i}(s_t) + \alpha \delta_{TD} \end{align}
ここで\(\alpha\)は学習率,添え字の\(i\)は更新回数を表しています.
連続時間の場合
連続時間の場合,方策\(\pi\)に従う価値関数は以下のように定義されます.
\begin{align} V^\pi(s(t)) := E\left [ \int_t^\infty e^{-\frac{(t’ – t)}{\tau_r} } r(s(t’),a(t’)) dt’ \right ]_\pi \end{align}
離散の際には和でしたが,連続の場合には積分になっています.そして\(\gamma\)の代わりに指数関数による割引\(e^{-\frac{(t’ – t)}{\tau_r}}\)を報酬\(r(s(t’),a(t’))\)にかけています.時定数\(\tau_r\)が小さいほど報酬は多く割り引かれます.
そして連続時間なのでTD誤差を計算する際に1ステップ後との差を計算するのではなく微分します.
連続時間の価値関数を微分すると
\begin{align} &\frac{d}{dt} V^\pi(s(t)) \\ &= \frac{d}{dt} E\left [ \int_t^\infty e^{-\frac{(t’ – t)}{\tau_r} }r(s(t’),a(t’)) dt’ \right ]_\pi\\ &= \frac{d}{dt} E\left [ e^{\frac{t}{\tau_r}} \int_t^\infty e^{-\frac{t’}{\tau_r} } r(s(t’),a(t’)) dt’ \right ]_\pi\\ &= E\left [ \left ( \frac{d}{dt}e^{\frac{t}{\tau_r}}\right ) \int_t^\infty e^{-\frac{t’}{\tau_r} } r(s(t’),a(t’)) dt’ + e^{\frac{t}{\tau_r}} \left ( \frac{d}{dt}\int_t^\infty e^{-\frac{t’}{\tau_r} } r(s(t’),a(t’)) dt’\right ) \right ]_\pi\\ &= E\left [ \frac{1}{\tau_r}e^{\frac{t}{\tau_r}} \int_t^\infty e^{-\frac{t’}{\tau_r} } r(s(t’),a(t’)) dt’ + e^{\frac{t}{\tau_r}} \left ( 0 – e^{-\frac{t}{\tau_r} } r(s(t),a(t))\right ) \right ]_\pi\\ &= E\left [ \frac{1}{\tau_r} \int_t^\infty e^{-\frac{t’-t}{\tau_r} } r(s(t’),a(t’)) dt’ – r(s(t),a(t)) \right ]_\pi\\ &= \frac{1}{\tau_r} V^\pi(s(t)) – r(s(t),a(t)) \end{align}
よって\(V^\pi(s_t)\)は以下の等式を満たす必要があります.
\begin{align} 0 = \frac{d}{dt} V^\pi(s(t)) – \frac{1}{\tau_r} V^\pi(s(t)) + r(s(t),a(t)) \end{align}
しかし,学習の際には\(V^\pi(s(t))\)は正確には計算できないので誤差が生じます.その誤差が以下のように連続時間でのTD誤差となります.
\begin{align} \delta_{TD} = \frac{d}{dt} V^\pi(s(t)) – \frac{1}{\tau_r} V^\pi(s(t)) + r(s(t),a(t)) \end{align}
また,この式のままでは意味が理解しにくいかもしれませんがオイラー後進差分で微分を離散化すると
\begin{align} \delta_{TD} &= \frac{d}{dt} V^\pi(s(t)) – \frac{1}{\tau_r} V^\pi(s(t)) + r(s(t),a(t)) \\ &\simeq \frac{V^\pi(s(t)) – V^\pi(s(t-\Delta t))}{\Delta t} – \frac{1}{\tau_r} V^\pi(s(t)) + r(s(t),a(t))\\ &= \frac{1}{\Delta t}\left( V^\pi(s(t)) – V^\pi(s(t-\Delta t)) – \frac{\Delta t}{\tau_r} V^\pi(s(t)) +\Delta t \times r(s(t),a(t)) \right)\\ &= \frac{1}{\Delta t}\left( (1-\frac{\Delta t}{\tau_r})V^\pi(s(t)) – V^\pi(s(t-\Delta t)) +\Delta t \times r(s(t),a(t)) \right)\\ &= \frac{1}{\Delta t}\left(\Delta t \times r(s(t),a(t)) + \gamma V^\pi(s(t)) – V^\pi(s(t-\Delta t)) \right) \end{align}
のように離散時間の場合のTD誤差と同じような意味を持った数式になっていることが分かります.
このとき\(\gamma=1-\frac{\Delta t}{\tau_r}\)と置いています.
参考文献
{8834464:4JXK5XJ3};{8834464:2IJ68CR3}
apa
default
asc
0
2354
%7B%22status%22%3A%22success%22%2C%22updateneeded%22%3Afalse%2C%22instance%22%3Afalse%2C%22meta%22%3A%7B%22request_last%22%3A850%2C%22request_next%22%3A50%2C%22used_cache%22%3Atrue%7D%2C%22data%22%3A%5B%7B%22key%22%3A%222IJ68CR3%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Doya%22%2C%22parsedDate%22%3A%222000-01-01%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3EDoya%2C%20K.%20%282000%29.%20Reinforcement%20Learning%20in%20Continuous%20Time%20and%20Space.%20%3Ci%3ENeural%20Computation%3C%5C%2Fi%3E%2C%20%3Ci%3E12%3C%5C%2Fi%3E%281%29%2C%20219%26%23x2013%3B245.%20%3Ca%20href%3D%27https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1162%5C%2F089976600300015961%27%3Ehttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1162%5C%2F089976600300015961%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Reinforcement%20Learning%20in%20Continuous%20Time%20and%20Space%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kenji%22%2C%22lastName%22%3A%22Doya%22%7D%5D%2C%22abstractNote%22%3A%22This%20article%20presents%20a%20reinforcement%20learning%20framework%20for%20continuous-time%20dynamical%20systems%20without%20a%20priori%20discretization%20of%20time%2C%20state%2C%20and%20action.%20Basedonthe%20Hamilton-Jacobi-Bellman%20%28HJB%29%20equation%20for%20infinite-horizon%2C%20discounted%20reward%20problems%2C%20we%20derive%20algorithms%20for%20estimating%20value%20functions%20and%20improving%20policies%20with%20the%20use%20of%20function%20approximators.%20The%20process%20of%20value%20function%20estimation%20is%20formulated%20as%20the%20minimization%20of%20a%20continuous-time%20form%20of%20the%20temporal%20difference%20%28TD%29%20error.%20Update%20methods%20based%20on%20backward%20Euler%20approximation%20and%20exponential%20eligibility%20traces%20are%20derived%2C%20and%20their%20correspondences%20with%20the%20conventional%20residual%20gradient%2C%20TD%20%280%29%2C%20and%20TD%20%28%5Cu03bb%29%20algorithms%20are%20shown.%20For%20policy%20improvement%2C%20two%20methods%5Cu2014a%20continuous%20actor-critic%20method%20and%20a%20value-gradient-based%20greedy%20policy%5Cu2014are%20formulated.%20As%20a%20special%20case%20of%20the%20latter%2C%20a%20nonlinear%20feedback%20control%20law%20using%20the%20value%20gradient%20and%20the%20model%20of%20the%20input%20gain%20is%20derived.%20The%20advantage%20updating%2C%20a%20model-free%20algorithm%20derived%20previously%2C%20is%20also%20formulated%20in%20the%20HJB-based%20framework.%5Cn%20%20%20%20%20%20%20%20%20%20%20%20The%20performance%20of%20the%20proposed%20algorithms%20is%20first%20tested%20in%20a%20nonlinear%20control%20task%20of%20swinging%20a%20pendulum%20up%20with%20limited%20torque.%20It%20is%20shown%20in%20the%20simulations%20that%20%281%29%20the%20task%20is%20accomplished%20by%20the%20continuous%20actor-critic%20method%20in%20a%20number%20of%20trials%20several%20times%20fewer%20than%20by%20the%20conventional%20discrete%20actor-critic%20method%3B%20%282%29%20among%20the%20continuous%20policy%20update%20methods%2C%20the%20value-gradient-based%20policy%20with%20a%20known%20or%20learned%20dynamic%20model%20performs%20several%20times%20better%20than%20the%20actor-critic%20method%3B%20and%20%283%29%20a%20value%20function%20update%20using%20exponential%20eligibility%20traces%20is%20more%20efficient%20and%20stable%20than%20that%20based%20on%20Euler%20approximation.%20The%20algorithms%20are%20then%20tested%20in%20a%20higher-dimensional%20task%3A%20cart-pole%20swing-up.%20This%20task%20is%20accomplished%20in%20several%20hundred%20trials%20using%20the%20value-gradient-based%20policy%20with%20a%20learned%20dynamic%20model.%22%2C%22date%22%3A%222000-01-01%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1162%5C%2F089976600300015961%22%2C%22ISSN%22%3A%220899-7667%2C%201530-888X%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fdirect.mit.edu%5C%2Fneco%5C%2Farticle%5C%2F12%5C%2F1%5C%2F219-245%5C%2F6324%22%2C%22collections%22%3A%5B%22TDE3Q4T2%22%5D%2C%22dateModified%22%3A%222024-06-20T07%3A44%3A48Z%22%7D%7D%2C%7B%22key%22%3A%22EDSHNPW3%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Yamanaka%20et%20al.%22%2C%22parsedDate%22%3A%222002-02-01%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3EYamanaka%2C%20K.%2C%20Kimura%2C%20T.%2C%20Miyazaki%2C%20M.%2C%20Kawashima%2C%20N.%2C%20Nozaki%2C%20D.%2C%20Nakazawa%2C%20K.%2C%20Yano%2C%20H.%2C%20%26amp%3B%20Yamamoto%2C%20Y.%20%282002%29.%20Human%20cortical%20activities%20during%20Go%5C%2FNoGo%20tasks%20with%20opposite%20motor%20control%20paradigms.%20%3Ci%3EExperimental%20Brain%20Research%3C%5C%2Fi%3E%2C%20%3Ci%3E142%3C%5C%2Fi%3E%283%29%2C%20301%26%23x2013%3B307.%20%3Ca%20href%3D%27https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2Fs00221-001-0943-2%27%3Ehttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2Fs00221-001-0943-2%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Human%20cortical%20activities%20during%20Go%5C%2FNoGo%20tasks%20with%20opposite%20motor%20control%20paradigms%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kentaro%22%2C%22lastName%22%3A%22Yamanaka%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Toshitaka%22%2C%22lastName%22%3A%22Kimura%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Makoto%22%2C%22lastName%22%3A%22Miyazaki%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Noritaka%22%2C%22lastName%22%3A%22Kawashima%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Daichi%22%2C%22lastName%22%3A%22Nozaki%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kimitaka%22%2C%22lastName%22%3A%22Nakazawa%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hideo%22%2C%22lastName%22%3A%22Yano%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yoshiharu%22%2C%22lastName%22%3A%22Yamamoto%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222002-2-1%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1007%5C%2Fs00221-001-0943-2%22%2C%22ISSN%22%3A%220014-4819%2C%201432-1106%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Flink.springer.com%5C%2F10.1007%5C%2Fs00221-001-0943-2%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-06-18T10%3A15%3A53Z%22%7D%7D%2C%7B%22key%22%3A%224GX2UUKW%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Pfurtscheller%20et%20al.%22%2C%22parsedDate%22%3A%221993-12-24%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3EPfurtscheller%2C%20G.%2C%20Neuper%2C%20C.%2C%20%26amp%3B%20Kalcher%2C%20J.%20%281993%29.%2040-Hz%20oscillations%20during%20motor%20behavior%20in%20man.%20%3Ci%3ENeuroscience%20Letters%3C%5C%2Fi%3E%2C%20%3Ci%3E164%3C%5C%2Fi%3E%281%29%2C%20179%26%23x2013%3B182.%20%3Ca%20href%3D%27https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2F0304-3940%2893%2990886-P%27%3Ehttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1016%5C%2F0304-3940%2893%2990886-P%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%2240-Hz%20oscillations%20during%20motor%20behavior%20in%20man%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gert%22%2C%22lastName%22%3A%22Pfurtscheller%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Christa%22%2C%22lastName%22%3A%22Neuper%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Joachim%22%2C%22lastName%22%3A%22Kalcher%22%7D%5D%2C%22abstractNote%22%3A%2240-Hz%20oscillations%20were%20measured%20during%20finger%2C%20toe%20and%20tongue%20movement%20using%20an%20electrode%20array%20of%2056%20electrodes%20over%20the%20pre%20and%20post%20central%20areas.%20Each%20movement%20was%20made%20150%20times%20in%20intervals%20of%2012%20s.%20The%20average%20power%20increase%20in%20narrow%20frequency%20bands%20between%208%20and%2040%20Hz%20was%20then%20calculated%20and%20the%20topographical%20distribution%20studied.%2040-Hz%20oscillations%20were%20only%20found%20contralateral%20over%20the%20hand%20area%20for%20finger%20movements%2C%20close%20to%20the%20vertex%20for%20toe%20movements%20and%20bilateral%20symmetrical%20for%20tongue%20movements.%20The%20maxima%20of%20the%2040-Hz%20oscillations%20occurred%20before%20movement%20onset.%22%2C%22date%22%3A%221993-12-24%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.1016%5C%2F0304-3940%2893%2990886-P%22%2C%22ISSN%22%3A%220304-3940%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.sciencedirect.com%5C%2Fscience%5C%2Farticle%5C%2Fpii%5C%2F030439409390886P%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-06-18T07%3A40%3A46Z%22%7D%7D%2C%7B%22key%22%3A%22F47S6QYT%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%26%23x9762%3B%26%23x63A5%3B%26%23x9078%3B%26%23x8003%3B%26%23x3067%3B%26%23x306F%3B%26%23x3001%3B%26%23x7814%3B%26%23x7A76%3B%26%23x63D0%3B%26%23x6848%3B%26%23x8005%3B%26%23x3054%3B%26%23x672C%3B%26%23x4EBA%3B%26%23x306B%3B%26%23x7814%3B%26%23x7A76%3B%26%23x69CB%3B%26%23x60F3%3B%26%23x306E%3B%26%23x8AAC%3B%26%23x660E%3B%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22KWYNAF7S%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5Cu9762%5Cu63a5%5Cu9078%5Cu8003%5Cu3067%5Cu306f%5Cu3001%5Cu7814%5Cu7a76%5Cu63d0%5Cu6848%5Cu8005%5Cu3054%5Cu672c%5Cu4eba%5Cu306b%5Cu7814%5Cu7a76%5Cu69cb%5Cu60f3%5Cu306e%5Cu8aac%5Cu660e%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2220%22%2C%22annotationSortIndex%22%3A%2200019%7C000676%7C00609%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A19%2C%5C%22rects%5C%22%3A%5B%5B73.824%2C219.353%2C325.094%2C232.49%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-17T02%3A34%3A49Z%22%7D%7D%2C%7B%22key%22%3A%22XKY74TFP%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%26%23x6240%3B%26%23x5C5E%3B%26%23x7814%3B%26%23x7A76%3B%26%23x6A5F%3B%26%23x95A2%3B%26%23x306B%3B%26%23x304A%3B%26%23x3044%3B%26%23x3066%3B%26%23x7814%3B%26%23x7A76%3B%26%23x502B%3B%26%23x7406%3B%26%23x6559%3B%26%23x80B2%3B%26%23x306B%3B%26%23x95A2%3B%26%23x3059%3B%26%23x308B%3B%26%23x30D7%3B%26%23x30ED%3B%26%23x30B0%3B%26%23x30E9%3B%26%23x30E0%3B%26%23x3092%3B%26%23x3042%3B%26%23x3089%3B%26%23x304B%3B%26%23x3058%3B%26%23x3081%3B%26%23x4FEE%3B%26%23x4E86%3B%26%23x3057%3B%26%23x3066%3B%26%23x3044%3B%26%23x308B%3B%26%23x3053%3B%26%23x3068%3B%26%23x3002%3B%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22KWYNAF7S%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5Cu6240%5Cu5c5e%5Cu7814%5Cu7a76%5Cu6a5f%5Cu95a2%5Cu306b%5Cu304a%5Cu3044%5Cu3066%5Cu7814%5Cu7a76%5Cu502b%5Cu7406%5Cu6559%5Cu80b2%5Cu306b%5Cu95a2%5Cu3059%5Cu308b%5Cu30d7%5Cu30ed%5Cu30b0%5Cu30e9%5Cu30e0%5Cu3092%5Cu3042%5Cu3089%5Cu304b%5Cu3058%5Cu3081%5Cu4fee%5Cu4e86%5Cu3057%5Cu3066%5Cu3044%5Cu308b%5Cu3053%5Cu3068%5Cu3002%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2219%22%2C%22annotationSortIndex%22%3A%2200018%7C000070%7C00125%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A18%2C%5C%22rects%5C%22%3A%5B%5B70.224%2C703.363%2C518.558%2C716.5%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-17T02%3A33%3A29Z%22%7D%7D%2C%7B%22key%22%3A%22YXUFEZF5%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E2024%20%26%23x5E74%3B%204%20%26%23x6708%3B%26%23xFF11%3B%26%23x65E5%3B%26%23x6642%3B%26%23x70B9%3B%26%23x3067%3B%26%23x3001%3B%26%23x535A%3B%26%23x58EB%3B%26%23x53F7%3B%26%23x53D6%3B%26%23x5F97%3B%26%23x5F8C%3B%2010%20%26%23x5E74%3B%26%23x672A%3B%26%23x6E80%3B%26%23xFF08%3B2014%20%26%23x5E74%3B%204%20%26%23x6708%3B%202%20%26%23x65E5%3B%26%23x4EE5%3B%26%23x964D%3B%26%23x306B%3B%26%23x535A%3B%26%23x58EB%3B%26%23x53F7%3B%26%23x53D6%3B%26%23x5F97%3B%26%23xFF09%3B%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22KWYNAF7S%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%222024%20%5Cu5e74%204%20%5Cu6708%5Cuff11%5Cu65e5%5Cu6642%5Cu70b9%5Cu3067%5Cu3001%5Cu535a%5Cu58eb%5Cu53f7%5Cu53d6%5Cu5f97%5Cu5f8c%2010%20%5Cu5e74%5Cu672a%5Cu6e80%5Cuff082014%20%5Cu5e74%204%20%5Cu6708%202%20%5Cu65e5%5Cu4ee5%5Cu964d%5Cu306b%5Cu535a%5Cu58eb%5Cu53f7%5Cu53d6%5Cu5f97%5Cuff09%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2218%22%2C%22annotationSortIndex%22%3A%2200017%7C000799%7C00609%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A17%2C%5C%22rects%5C%22%3A%5B%5B76.104%2C219.353%2C522.58%2C232.49%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-17T02%3A33%3A09Z%22%7D%7D%2C%7B%22key%22%3A%22SNLBNVFQ%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%26%23x6700%3B%26%23x5927%3B%20200%20%26%23x4EF6%3B%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22KWYNAF7S%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5Cu6700%5Cu5927%20200%20%5Cu4ef6%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2217%22%2C%22annotationSortIndex%22%3A%2200016%7C000112%7C00191%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A16%2C%5C%22rects%5C%22%3A%5B%5B158.661%2C637.363%2C217.73%2C650.5%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-17T02%3A32%3A03Z%22%7D%7D%2C%7B%22key%22%3A%22GDZTZMVA%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E5%20%26%23x5343%3B%26%23x4E07%3B%26%23x5186%3B%28%26%23x76F4%3B%26%23x63A5%3B%26%23x7D4C%3B%26%23x8CBB%3B%26%23x3002%3B%26%23x901A%3B%26%23x671F%3B%26%23xFF1B%3B%26%23x7814%3B%26%23x7A76%3B%26%23x671F%3B%26%23x9593%3B%26%23xFF15%3B%26%23x5E74%3B%26%23x9593%3B%29%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22KWYNAF7S%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%225%20%5Cu5343%5Cu4e07%5Cu5186%28%5Cu76f4%5Cu63a5%5Cu7d4c%5Cu8cbb%5Cu3002%5Cu901a%5Cu671f%5Cuff1b%5Cu7814%5Cu7a76%5Cu671f%5Cu9593%5Cuff15%5Cu5e74%5Cu9593%29%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2216%22%2C%22annotationSortIndex%22%3A%2200015%7C000806%7C00697%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A15%2C%5C%22rects%5C%22%3A%5B%5B262.01%2C131.363%2C481.107%2C144.5%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-17T02%3A31%3A21Z%22%7D%7D%2C%7B%22key%22%3A%2293FA455D%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%26%23x7814%3B%26%23x7A76%3B%26%23x671F%3B%26%23x9593%3B%26%23x306F%3B%26%23x3001%3B%26%23x539F%3B%26%23x5247%3B%26%23x3068%3B%26%23x3057%3B%26%23x3066%3B%26%23x7814%3B%26%23x7A76%3B%26%23x958B%3B%26%23x59CB%3B%26%23xFF08%3B2025%20%26%23x5E74%3B%201%20%26%23x6708%3B%26%23x4EE5%3B%26%23x964D%3B%26%23xFF09%3B%26%23x304B%3B%26%23x3089%3B%205%20%26%23x5E74%3B%26%23x9593%3B%26%23x3067%3B%26%23x3059%3B%26%23x3002%3B%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22KWYNAF7S%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5Cu7814%5Cu7a76%5Cu671f%5Cu9593%5Cu306f%5Cu3001%5Cu539f%5Cu5247%5Cu3068%5Cu3057%5Cu3066%5Cu7814%5Cu7a76%5Cu958b%5Cu59cb%5Cuff082025%20%5Cu5e74%201%20%5Cu6708%5Cu4ee5%5Cu964d%5Cuff09%5Cu304b%5Cu3089%205%20%5Cu5e74%5Cu9593%5Cu3067%5Cu3059%5Cu3002%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2216%22%2C%22annotationSortIndex%22%3A%2200015%7C000678%7C00587%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A15%2C%5C%22rects%5C%22%3A%5B%5B70.584%2C241.313%2C433.51%2C254.45%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-17T02%3A31%3A03Z%22%7D%7D%2C%7B%22key%22%3A%22HD996C7B%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%26%23x6A5F%3B%26%23x68B0%3B%26%23x5B66%3B%26%23x7FD2%3B%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22KWYNAF7S%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5Cu6a5f%5Cu68b0%5Cu5b66%5Cu7fd2%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2216%22%2C%22annotationSortIndex%22%3A%2200015%7C000314%7C00279%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A15%2C%5C%22rects%5C%22%3A%5B%5B408.19%2C549.373%2C452.229%2C562.51%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-17T02%3A30%3A46Z%22%7D%7D%2C%7B%22key%22%3A%22NITMJAAU%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%26%23x30ED%3B%26%23x30DC%3B%26%23x30C6%3B%26%23x30A3%3B%26%23x30AF%3B%26%23x30B9%3B%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22KWYNAF7S%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5Cu30ed%5Cu30dc%5Cu30c6%5Cu30a3%5Cu30af%5Cu30b9%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2216%22%2C%22annotationSortIndex%22%3A%2200015%7C000300%7C00279%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A15%2C%5C%22rects%5C%22%3A%5B%5B255.854%2C549.373%2C321.851%2C562.51%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-17T02%3A30%3A42Z%22%7D%7D%2C%7B%22key%22%3A%22QRWKU3TP%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%26%23x30A2%3B%26%23x30FC%3B%26%23x30AD%3B%26%23x30C6%3B%20%26%23x30AF%3B%26%23x30C1%3B%26%23x30E3%3B%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22KWYNAF7S%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5Cu30a2%5Cu30fc%5Cu30ad%5Cu30c6%20%5Cu30af%5Cu30c1%5Cu30e3%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2216%22%2C%22annotationSortIndex%22%3A%2200015%7C000278%7C00257%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A15%2C%5C%22rects%5C%22%3A%5B%5B497.501%2C571.363%2C541.418%2C584.5%5D%2C%5B59.544%2C549.373%2C92.664%2C562.51%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-17T02%3A30%3A38Z%22%7D%7D%2C%7B%22key%22%3A%22R7QTJV8Q%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%26%23x30A2%3B%26%23x30EB%3B%26%23x30B4%3B%26%23x30EA%3B%26%23x30BA%3B%26%23x30E0%3B%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22KWYNAF7S%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5Cu30a2%5Cu30eb%5Cu30b4%5Cu30ea%5Cu30ba%5Cu30e0%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2216%22%2C%22annotationSortIndex%22%3A%2200015%7C000267%7C00257%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A15%2C%5C%22rects%5C%22%3A%5B%5B377.929%2C571.363%2C443.926%2C584.5%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-17T02%3A30%3A33Z%22%7D%7D%2C%7B%22key%22%3A%22WVUKG9TI%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%26%23x6570%3B%26%23x7406%3B%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22KWYNAF7S%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5Cu6570%5Cu7406%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2216%22%2C%22annotationSortIndex%22%3A%2200015%7C000247%7C00257%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A15%2C%5C%22rects%5C%22%3A%5B%5B160.821%2C571.363%2C182.78%2C584.5%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-17T02%3A30%3A29Z%22%7D%7D%2C%7B%22key%22%3A%22VMHKW6X5%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%26%23x6700%3B%26%23x9577%3B%26%23xFF11%3B%26%23x5E74%3B%26%23x9593%3B%26%23x3001%3B%26%23x63A1%3B%26%23x629E%3B%26%23x8005%3B%26%23x306E%3B%26%23x8CC7%3B%26%23x683C%3B%26%23x3092%3B%26%23x6301%3B%26%23x3063%3B%26%23x305F%3B%26%23x307E%3B%26%23x307E%3B%26%23x7814%3B%26%23x7A76%3B%26%23x958B%3B%20%26%23x59CB%3B%26%23x3092%3B%26%23x7336%3B%26%23x4E88%3B%26%23x3057%3B%26%23x307E%3B%26%23x3059%3B%26%23x3002%3B%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22KWYNAF7S%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5Cu6700%5Cu9577%5Cuff11%5Cu5e74%5Cu9593%5Cu3001%5Cu63a1%5Cu629e%5Cu8005%5Cu306e%5Cu8cc7%5Cu683c%5Cu3092%5Cu6301%5Cu3063%5Cu305f%5Cu307e%5Cu307e%5Cu7814%5Cu7a76%5Cu958b%20%5Cu59cb%5Cu3092%5Cu7336%5Cu4e88%5Cu3057%5Cu307e%5Cu3059%5Cu3002%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2212%22%2C%22annotationSortIndex%22%3A%2200011%7C000081%7C00125%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A11%2C%5C%22rects%5C%22%3A%5B%5B303.29%2C702.883%2C535.832%2C716.02%5D%2C%5B59.664%2C680.923%2C148.205%2C694.06%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-17T02%3A29%3A28Z%22%7D%7D%2C%7B%22key%22%3A%22KGNKZBR6%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%26%23x30AF%3B%26%23x30ED%3B%26%23x30B9%3B%26%23x30A2%3B%26%23x30DD%3B%26%23x30A4%3B%26%23x30F3%3B%26%23x30C8%3B%26%23x30E1%3B%26%23x30F3%3B%26%23x30C8%3B%26%23x3092%3B%26%23x3057%3B%26%23x305F%3B%26%23x3053%3B%26%23x3068%3B%26%23x306B%3B%26%23x3088%3B%26%23x308B%3B%26%23x5E74%3B%26%23x53CE%3B%26%23x306E%3B%26%23x5408%3B%26%23x8A08%3B%26%23x304C%3B%26%23x3001%3B%26%23x30AF%3B%26%23x30ED%3B%26%23x30B9%3B%26%23x30A2%3B%20%26%23x30DD%3B%26%23x30A4%3B%26%23x30F3%3B%26%23x30C8%3B%26%23x30E1%3B%26%23x30F3%3B%26%23x30C8%3B%26%23x5B9F%3B%26%23x65BD%3B%26%23x524D%3B%26%23x306E%3B%26%23x5E74%3B%26%23x53CE%3B%26%23x3092%3B%26%23x8D85%3B%26%23x3048%3B%26%23x308B%3B%26%23x91D1%3B%26%23x984D%3B%26%23x3068%3B%26%23x306A%3B%26%23x308B%3B%26%23x3053%3B%26%23x3068%3B%26%23x304C%3B%26%23x5FC5%3B%26%23x8981%3B%26%23x3067%3B%26%23x3059%3B%26%23x3002%3B%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22KWYNAF7S%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5Cu30af%5Cu30ed%5Cu30b9%5Cu30a2%5Cu30dd%5Cu30a4%5Cu30f3%5Cu30c8%5Cu30e1%5Cu30f3%5Cu30c8%5Cu3092%5Cu3057%5Cu305f%5Cu3053%5Cu3068%5Cu306b%5Cu3088%5Cu308b%5Cu5e74%5Cu53ce%5Cu306e%5Cu5408%5Cu8a08%5Cu304c%5Cu3001%5Cu30af%5Cu30ed%5Cu30b9%5Cu30a2%20%5Cu30dd%5Cu30a4%5Cu30f3%5Cu30c8%5Cu30e1%5Cu30f3%5Cu30c8%5Cu5b9f%5Cu65bd%5Cu524d%5Cu306e%5Cu5e74%5Cu53ce%5Cu3092%5Cu8d85%5Cu3048%5Cu308b%5Cu91d1%5Cu984d%5Cu3068%5Cu306a%5Cu308b%5Cu3053%5Cu3068%5Cu304c%5Cu5fc5%5Cu8981%5Cu3067%5Cu3059%5Cu3002%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%229%22%2C%22annotationSortIndex%22%3A%2200008%7C000505%7C00389%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A8%2C%5C%22rects%5C%22%3A%5B%5B209.002%2C439.333%2C541.41%2C452.47%5D%2C%5B54%2C417.373%2C386.242%2C430.51%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-17T02%3A27%3A41Z%22%7D%7D%2C%7B%22key%22%3A%22RLX8BR59%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Huh%20and%20Sejnowski%22%2C%22parsedDate%22%3A%222017-06-19%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3EHuh%2C%20D.%2C%20%26amp%3B%20Sejnowski%2C%20T.%20J.%20%282017%29.%20%3Ci%3EGradient%20Descent%20for%20Spiking%20Neural%20Networks%3C%5C%2Fi%3E%20%28No.%20arXiv%3A1706.04698%29.%20arXiv.%20%3Ca%20href%3D%27http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1706.04698%27%3Ehttp%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1706.04698%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Gradient%20Descent%20for%20Spiking%20Neural%20Networks%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dongsung%22%2C%22lastName%22%3A%22Huh%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Terrence%20J.%22%2C%22lastName%22%3A%22Sejnowski%22%7D%5D%2C%22abstractNote%22%3A%22Much%20of%20studies%20on%20neural%20computation%20are%20based%20on%20network%20models%20of%20static%20neurons%20that%20produce%20analog%20output%2C%20despite%20the%20fact%20that%20information%20processing%20in%20the%20brain%20is%20predominantly%20carried%20out%20by%20dynamic%20neurons%20that%20produce%20discrete%20pulses%20called%20spikes.%20Research%20in%20spike-based%20computation%20has%20been%20impeded%20by%20the%20lack%20of%20ef%5Cufb01cient%20supervised%20learning%20algorithm%20for%20spiking%20networks.%20Here%2C%20we%20present%20a%20gradient%20descent%20method%20for%20optimizing%20spiking%20network%20models%20by%20introducing%20a%20differentiable%20formulation%20of%20spiking%20networks%20and%20deriving%20the%20exact%20gradient%20calculation.%20For%20demonstration%2C%20we%20trained%20recurrent%20spiking%20networks%20on%20two%20dynamic%20tasks%3A%20one%20that%20requires%20optimizing%20fast%20%28%5Cu2248%20millisecond%29%20spike-based%20interactions%20for%20ef%5Cufb01cient%20encoding%20of%20information%2C%20and%20a%20delayed-memory%20XOR%20task%20over%20extended%20duration%20%28%5Cu2248%20second%29.%20The%20results%20show%20that%20our%20method%20indeed%20optimizes%20the%20spiking%20network%20dynamics%20on%20the%20time%20scale%20of%20individual%20spikes%20as%20well%20as%20the%20behavioral%20time%20scales.%20In%20conclusion%2C%20our%20result%20offers%20a%20general%20purpose%20supervised%20learning%20algorithm%20for%20spiking%20neural%20networks%2C%20thus%20advancing%20further%20investigations%20on%20spike-based%20computation.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A1706.04698%22%2C%22date%22%3A%222017-06-19%22%2C%22DOI%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1706.04698%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%22TDE3Q4T2%22%5D%2C%22dateModified%22%3A%222024-06-07T02%3A09%3A01Z%22%7D%7D%2C%7B%22key%22%3A%22RZSXIISJ%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Huh%20and%20Sejnowski%22%2C%22parsedDate%22%3A%222017-06-14%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3EHuh%2C%20D.%2C%20%26amp%3B%20Sejnowski%2C%20T.%20J.%20%282017%2C%20June%2014%29.%20%3Ci%3EGradient%20Descent%20for%20Spiking%20Neural%20Networks%3C%5C%2Fi%3E.%20ArXiv.Org.%20%3Ca%20href%3D%27https%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1706.04698v2%27%3Ehttps%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1706.04698v2%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22webpage%22%2C%22title%22%3A%22Gradient%20Descent%20for%20Spiking%20Neural%20Networks%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dongsung%22%2C%22lastName%22%3A%22Huh%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Terrence%20J.%22%2C%22lastName%22%3A%22Sejnowski%22%7D%5D%2C%22abstractNote%22%3A%22Much%20of%20studies%20on%20neural%20computation%20are%20based%20on%20network%20models%20of%20static%20neurons%20that%20produce%20analog%20output%2C%20despite%20the%20fact%20that%20information%20processing%20in%20the%20brain%20is%20predominantly%20carried%20out%20by%20dynamic%20neurons%20that%20produce%20discrete%20pulses%20called%20spikes.%20Research%20in%20spike-based%20computation%20has%20been%20impeded%20by%20the%20lack%20of%20efficient%20supervised%20learning%20algorithm%20for%20spiking%20networks.%20Here%2C%20we%20present%20a%20gradient%20descent%20method%20for%20optimizing%20spiking%20network%20models%20by%20introducing%20a%20differentiable%20formulation%20of%20spiking%20networks%20and%20deriving%20the%20exact%20gradient%20calculation.%20For%20demonstration%2C%20we%20trained%20recurrent%20spiking%20networks%20on%20two%20dynamic%20tasks%3A%20one%20that%20requires%20optimizing%20fast%20%28~millisecond%29%20spike-based%20interactions%20for%20efficient%20encoding%20of%20information%2C%20and%20a%20delayed%20memory%20XOR%20task%20over%20extended%20duration%20%28~second%29.%20The%20results%20show%20that%20our%20method%20indeed%20optimizes%20the%20spiking%20network%20dynamics%20on%20the%20time%20scale%20of%20individual%20spikes%20as%20well%20as%20behavioral%20time%20scales.%20In%20conclusion%2C%20our%20result%20offers%20a%20general%20purpose%20supervised%20learning%20algorithm%20for%20spiking%20neural%20networks%2C%20thus%20advancing%20further%20investigations%20on%20spike-based%20computation.%22%2C%22date%22%3A%222017%5C%2F06%5C%2F14%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1706.04698v2%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%22TDE3Q4T2%22%5D%2C%22dateModified%22%3A%222024-06-07T02%3A06%3A09Z%22%7D%7D%2C%7B%22key%22%3A%223AQXQ247%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3EIt%20can%20in%20principle%20be%20estimated%26%23x2014%3Bin%20spite%20of%20the%20fact%20that%20the%20implicit%20discrete%20variable%20zjt%20is%20non-differentiable%26%23x2014%3Bwith%20the%20help%20of%20a%20suitable%20pseudo%20derivative%20for%20spikes%20as%20in%20refs.%203%2C4.%3C%5C%2Fi%3E%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%2232QGAAVI%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22It%20can%20in%20principle%20be%20estimated%5Cu2014in%20spite%20of%20the%20fact%20that%20the%20implicit%20discrete%20variable%20zjt%20is%20non-differentiable%5Cu2014with%20the%20help%20of%20a%20suitable%20pseudo%20derivative%20for%20spikes%20as%20in%20refs.%203%2C4.%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%222%22%2C%22annotationSortIndex%22%3A%2200001%7C005780%7C00600%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A1%2C%5C%22rects%5C%22%3A%5B%5B352.573%2C172.465%2C551.644%2C181.996%5D%2C%5B303.591%2C159.929%2C551.707%2C172.415%5D%2C%5B303.59%2C149.448%2C551.701%2C160.106%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-07T02%3A05%3A00Z%22%7D%7D%2C%7B%22key%22%3A%22KT5BGJ7Z%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3Ein%20the%20model%20striatum%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%222WMQQ9AN%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22in%20the%20model%20striatum%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2211621%22%2C%22annotationSortIndex%22%3A%2200001%7C004526%7C00613%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A1%2C%5C%22rects%5C%22%3A%5B%5B147.633%2C162.634%2C219.663%2C169.413%5D%5D%7D%22%2C%22dateModified%22%3A%222024-06-03T01%3A34%3A28Z%22%7D%7D%2C%7B%22key%22%3A%22HF4FX4DA%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22McCarthy%20et%20al.%22%2C%22parsedDate%22%3A%222011-07-12%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3EMcCarthy%2C%20M.%20M.%2C%20Moore-Kochlacs%2C%20C.%2C%20Gu%2C%20X.%2C%20Boyden%2C%20E.%20S.%2C%20Han%2C%20X.%2C%20%26amp%3B%20Kopell%2C%20N.%20%282011%29.%20Striatal%20origin%20of%20the%20pathologic%20beta%20oscillations%20in%20Parkinson%26%23x2019%3Bs%20disease.%20%3Ci%3EProceedings%20of%20the%20National%20Academy%20of%20Sciences%3C%5C%2Fi%3E%2C%20%3Ci%3E108%3C%5C%2Fi%3E%2828%29%2C%2011620%26%23x2013%3B11625.%20%3Ca%20href%3D%27https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1073%5C%2Fpnas.1107748108%27%3Ehttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1073%5C%2Fpnas.1107748108%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Striatal%20origin%20of%20the%20pathologic%20beta%20oscillations%20in%20Parkinson%27s%20disease%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22M.%20M.%22%2C%22lastName%22%3A%22McCarthy%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22C.%22%2C%22lastName%22%3A%22Moore-Kochlacs%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22X.%22%2C%22lastName%22%3A%22Gu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22E.%20S.%22%2C%22lastName%22%3A%22Boyden%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22X.%22%2C%22lastName%22%3A%22Han%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22N.%22%2C%22lastName%22%3A%22Kopell%22%7D%5D%2C%22abstractNote%22%3A%22Enhanced%20oscillations%20at%20beta%20frequencies%20%288%5Cu201330%20Hz%29%20are%20a%20signature%20neural%20dynamic%20pathology%20in%20the%20basal%20ganglia%20and%20cortex%20of%20Parkinson%27s%20disease%20patients.%20The%20mechanisms%20underlying%20these%20pathological%20beta%20oscillations%20remain%20elusive.%20Here%2C%20using%20mathematical%20models%2C%20we%20find%20that%20robust%20beta%20oscillations%20can%20emerge%20from%20inhibitory%20interactions%20between%20striatal%20medium%20spiny%20neurons.%20The%20interaction%20of%20the%20synaptic%20GABAa%20currents%20and%20the%20intrinsic%20membrane%20M-current%20promotes%20population%20oscillations%20in%20the%20beta%20frequency%20range.%20Increased%20levels%20of%20cholinergic%20drive%2C%20a%20condition%20relevant%20to%20the%20parkinsonian%20striatum%2C%20lead%20to%20enhanced%20beta%20oscillations%20in%20the%20striatal%20model.%20We%20show%20experimentally%20that%20direct%20infusion%20of%20the%20cholinergic%20agonist%20carbachol%20into%20the%20striatum%2C%20but%20not%20into%20the%20neighboring%20cortex%2C%20of%20the%20awake%2C%20normal%20rodent%20induces%20prominent%20beta%20frequency%20oscillations%20in%20the%20local%20field%20potential.%20These%20results%20provide%20evidence%20for%20amplification%20of%20normal%20striatal%20network%20dynamics%20as%20a%20mechanism%20responsible%20for%20the%20enhanced%20beta%20frequency%20oscillations%20in%20Parkinson%27s%20disease.%22%2C%22date%22%3A%222011-07-12%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1073%5C%2Fpnas.1107748108%22%2C%22ISSN%22%3A%220027-8424%2C%201091-6490%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fpnas.org%5C%2Fdoi%5C%2Ffull%5C%2F10.1073%5C%2Fpnas.1107748108%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-06-03T01%3A28%3A28Z%22%7D%7D%2C%7B%22key%22%3A%22NKEIS5SQ%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%5B20%2C21%5D%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22N2XCZ62R%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5B20%2C21%5D%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%222%22%2C%22annotationSortIndex%22%3A%2200001%7C000597%7C00431%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A1%2C%5C%22rects%5C%22%3A%5B%5B108.396%2C351.28%2C134.721%2C359.502%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-28T05%3A42%3A55Z%22%7D%7D%2C%7B%22key%22%3A%2262TWLWM5%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%5B16%5D%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22N2XCZ62R%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5B16%5D%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%221%22%2C%22annotationSortIndex%22%3A%2200000%7C005118%7C00672%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A0%2C%5C%22rects%5C%22%3A%5B%5B359.093%2C110.331%2C374.21%2C118.553%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-28T05%3A42%3A49Z%22%7D%7D%2C%7B%22key%22%3A%22ZBFFKJKQ%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%5BDDM21%5D%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%2276ELJ5IY%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5BDDM21%5D%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ff6666%22%2C%22annotationPageLabel%22%3A%2252%22%2C%22annotationSortIndex%22%3A%2200051%7C000191%7C00288%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A51%2C%5C%22rects%5C%22%3A%5B%5B143.776%2C542.067%2C183.616%2C553.664%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-17T05%3A09%3A52Z%22%7D%7D%2C%7B%22key%22%3A%22QLUAP56M%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3EIn%20%5BDDM21%5D%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%2276ELJ5IY%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22In%20%5BDDM21%5D%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ff6666%22%2C%22annotationPageLabel%22%3A%2248%22%2C%22annotationSortIndex%22%3A%2200047%7C002091%7C00550%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A47%2C%5C%22rects%5C%22%3A%5B%5B192.731%2C276.036%2C253.906%2C291.046%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-17T05%3A09%3A31Z%22%7D%7D%2C%7B%22key%22%3A%22QDSBHJFJ%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Chen%20et%20al.%22%2C%22parsedDate%22%3A%222020-05-11%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3EChen%2C%20B.%2C%20Xu%2C%20M.%2C%20Li%2C%20L.%2C%20%26amp%3B%20Zhao%2C%20D.%20%282020%29.%20%3Ci%3EDelay-Aware%20Model-Based%20Reinforcement%20Learning%20for%20Continuous%20Control%3C%5C%2Fi%3E%20%28No.%20arXiv%3A2005.05440%29.%20arXiv.%20%3Ca%20href%3D%27http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2005.05440%27%3Ehttp%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2005.05440%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Delay-Aware%20Model-Based%20Reinforcement%20Learning%20for%20Continuous%20Control%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Baiming%22%2C%22lastName%22%3A%22Chen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mengdi%22%2C%22lastName%22%3A%22Xu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Liang%22%2C%22lastName%22%3A%22Li%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ding%22%2C%22lastName%22%3A%22Zhao%22%7D%5D%2C%22abstractNote%22%3A%22Action%20delays%20degrade%20the%20performance%20of%20reinforcement%20learning%20in%20many%20real-world%20systems.%20This%20paper%20proposes%20a%20formal%20de%5Cufb01nition%20of%20delay-aware%20Markov%20Decision%20Process%20and%20proves%20it%20can%20be%20transformed%20into%20standard%20MDP%20with%20augmented%20states%20using%20the%20Markov%20reward%20process.%20We%20develop%20a%20delayaware%20model-based%20reinforcement%20learning%20framework%20that%20can%20incorporate%20the%20multi-step%20delay%20into%20the%20learned%20system%20models%20without%20learning%20effort.%20Experiments%20with%20the%20Gym%20and%20MuJoCo%20platforms%20show%20that%20the%20proposed%20delay-aware%20model-based%20algorithm%20is%20more%20ef%5Cufb01cient%20in%20training%20and%20transferable%20between%20systems%20with%20various%20durations%20of%20delay%20compared%20with%20off-policy%20model-free%20reinforcement%20learning%20methods.%20Codes%20available%20at%3A%20https%3A%5C%2F%5C%2Fgithub.com%5C%2Fbaimingc%5C%2Fdambrl.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2005.05440%22%2C%22date%22%3A%222020-05-11%22%2C%22DOI%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2005.05440%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-05-17T04%3A55%3A59Z%22%7D%7D%2C%7B%22key%22%3A%22SKDRWZKE%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3Eaugmented%20state%20x%20%26%23x201C%3B%20ps%2C%20a1%2C%20.%20.%20.%20%2C%20a%26%23x2206%3Bq%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%2276ELJ5IY%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22augmented%20state%20x%20%5Cu201c%20ps%2C%20a1%2C%20.%20.%20.%20%2C%20a%5Cu2206q%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ff6666%22%2C%22annotationPageLabel%22%3A%2245%22%2C%22annotationSortIndex%22%3A%2200044%7C001975%7C00566%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A44%2C%5C%22rects%5C%22%3A%5B%5B271.286%2C262.582%2C450.615%2C275.334%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-17T04%3A03%3A35Z%22%7D%7D%2C%7B%22key%22%3A%22USWC9PXZ%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3Ememoryless%20approach%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%2276ELJ5IY%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22memoryless%20approach%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ff6666%22%2C%22annotationPageLabel%22%3A%2245%22%2C%22annotationSortIndex%22%3A%2200044%7C001124%7C00359%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A44%2C%5C%22rects%5C%22%3A%5B%5B227.778%2C471.51%2C332.732%2C482.198%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-17T04%3A03%3A35Z%22%7D%7D%2C%7B%22key%22%3A%22Q2SYIBKH%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3Eaugmented%20state%20approach%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%2276ELJ5IY%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22augmented%20state%20approach%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ff6666%22%2C%22annotationPageLabel%22%3A%2245%22%2C%22annotationSortIndex%22%3A%2200044%7C001083%7C00345%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A44%2C%5C%22rects%5C%22%3A%5B%5B411.146%2C485.458%2C510.231%2C496.146%5D%2C%5B96.378%2C471.51%2C128.908%2C482.198%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-17T04%3A03%3A35Z%22%7D%7D%2C%7B%22key%22%3A%223TZB6NF2%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3EdSARSA%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%2276ELJ5IY%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22dSARSA%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2247%22%2C%22annotationSortIndex%22%3A%2200046%7C001515%7C00422%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A46%2C%5C%22rects%5C%22%3A%5B%5B96.378%2C409.017%2C140.887%2C419.705%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-17T04%3A03%3A35Z%22%7D%7D%2C%7B%22key%22%3A%225B4NDDZ8%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3ERTAC%20%5BRP19%5D%2C%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%2276ELJ5IY%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22RTAC%20%5BRP19%5D%2C%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2247%22%2C%22annotationSortIndex%22%3A%2200046%7C000539%7C00193%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A46%2C%5C%22rects%5C%22%3A%5B%5B394.636%2C637.942%2C464.909%2C648.63%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-17T04%3A03%3A35Z%22%7D%7D%2C%7B%22key%22%3A%22TANBVEET%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3EDelayCorrecting%20Actor-Critic%20%28DCAC%29%20builds%20on%20SAC%20%5BHZAL18%5D%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%2276ELJ5IY%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22DelayCorrecting%20Actor-Critic%20%28DCAC%29%20builds%20on%20SAC%20%5BHZAL18%5D%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%2247%22%2C%22annotationSortIndex%22%3A%2200046%7C000269%7C00133%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A46%2C%5C%22rects%5C%22%3A%5B%5B477.711%2C693.733%2C510.241%2C708.743%5D%2C%5B96.378%2C679.785%2C381.139%2C690.473%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-17T04%3A03%3A35Z%22%7D%7D%2C%7B%22key%22%3A%22G76W3L96%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3Emodel-based%20approach%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%2276ELJ5IY%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22model-based%20approach%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ff6666%22%2C%22annotationPageLabel%22%3A%2245%22%2C%22annotationSortIndex%22%3A%2200044%7C001162%7C00359%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A44%2C%5C%22rects%5C%22%3A%5B%5B449.152%2C471.51%2C510.231%2C482.198%5D%2C%5B96.378%2C457.563%2C140.194%2C468.251%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-17T04%3A03%3A35Z%22%7D%7D%2C%7B%22key%22%3A%22Y3X3JP4V%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3EWe%20start%20by%20describing%20the%20three%20main%20approaches%20from%20the%20literature%20to%20tackle%20constant%20delays%20in%20state%20observation%20or%20action%20execution.%3C%5C%2Fi%3E%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%2276ELJ5IY%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22We%20start%20by%20describing%20the%20three%20main%20approaches%20from%20the%20literature%20to%20tackle%20constant%20delays%20in%20state%20observation%20or%20action%20execution.%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%235fb236%22%2C%22annotationPageLabel%22%3A%2245%22%2C%22annotationSortIndex%22%3A%2200044%7C000956%7C00317%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A44%2C%5C%22rects%5C%22%3A%5B%5B494.594%2C513.354%2C510.231%2C524.042%5D%2C%5B96.378%2C499.406%2C510.231%2C510.094%5D%2C%5B96.378%2C485.458%2C330.281%2C496.146%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-17T04%3A03%3A35Z%22%7D%7D%2C%7B%22key%22%3A%22IMSBL3NB%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Cho%20et%20al.%22%2C%22parsedDate%22%3A%222014-09-02%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3ECho%2C%20K.%2C%20van%20Merrienboer%2C%20B.%2C%20Gulcehre%2C%20C.%2C%20Bahdanau%2C%20D.%2C%20Bougares%2C%20F.%2C%20Schwenk%2C%20H.%2C%20%26amp%3B%20Bengio%2C%20Y.%20%282014%29.%20%3Ci%3ELearning%20Phrase%20Representations%20using%20RNN%20Encoder-Decoder%20for%20Statistical%20Machine%20Translation%3C%5C%2Fi%3E%20%28No.%20arXiv%3A1406.1078%29.%20arXiv.%20%3Ca%20href%3D%27http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1406.1078%27%3Ehttp%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1406.1078%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Learning%20Phrase%20Representations%20using%20RNN%20Encoder-Decoder%20for%20Statistical%20Machine%20Translation%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kyunghyun%22%2C%22lastName%22%3A%22Cho%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Bart%22%2C%22lastName%22%3A%22van%20Merrienboer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Caglar%22%2C%22lastName%22%3A%22Gulcehre%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dzmitry%22%2C%22lastName%22%3A%22Bahdanau%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fethi%22%2C%22lastName%22%3A%22Bougares%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Holger%22%2C%22lastName%22%3A%22Schwenk%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Yoshua%22%2C%22lastName%22%3A%22Bengio%22%7D%5D%2C%22abstractNote%22%3A%22In%20this%20paper%2C%20we%20propose%20a%20novel%20neural%20network%20model%20called%20RNN%20Encoder%5Cu2013Decoder%20that%20consists%20of%20two%20recurrent%20neural%20networks%20%28RNN%29.%20One%20RNN%20encodes%20a%20sequence%20of%20symbols%20into%20a%20%5Cufb01xedlength%20vector%20representation%2C%20and%20the%20other%20decodes%20the%20representation%20into%20another%20sequence%20of%20symbols.%20The%20encoder%20and%20decoder%20of%20the%20proposed%20model%20are%20jointly%20trained%20to%20maximize%20the%20conditional%20probability%20of%20a%20target%20sequence%20given%20a%20source%20sequence.%20The%20performance%20of%20a%20statistical%20machine%20translation%20system%20is%20empirically%20found%20to%20improve%20by%20using%20the%20conditional%20probabilities%20of%20phrase%20pairs%20computed%20by%20the%20RNN%20Encoder%5Cu2013Decoder%20as%20an%20additional%20feature%20in%20the%20existing%20log-linear%20model.%20Qualitatively%2C%20we%20show%20that%20the%20proposed%20model%20learns%20a%20semantically%20and%20syntactically%20meaningful%20representation%20of%20linguistic%20phrases.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A1406.1078%22%2C%22date%22%3A%222014-09-02%22%2C%22DOI%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1406.1078%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-05-17T03%3A51%3A25Z%22%7D%7D%2C%7B%22key%22%3A%22QHB4EFYS%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3EThis%20suggests%20that%20we%20compare%20the%20predictive%20and%20non-predictive%20agents%20more%20directly%2C%20by%20having%20them%20train%20against%20each%20other.%3C%5C%2Fi%3E%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22BQWXKNXX%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22This%20suggests%20that%20we%20compare%20the%20predictive%20and%20non-predictive%20agents%20more%20directly%2C%20by%20having%20them%20train%20against%20each%20other.%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ff6666%22%2C%22annotationPageLabel%22%3A%227%22%2C%22annotationSortIndex%22%3A%2200006%7C000485%7C00237%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A6%2C%5C%22rects%5C%22%3A%5B%5B328.488%2C545.287%2C504.004%2C554.194%5D%2C%5B108%2C534.378%2C444.032%2C543.285%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-17T03%3A14%3A11Z%22%7D%7D%2C%7B%22key%22%3A%22WEJQGJ3F%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3EIn%20order%20to%20be%20Markovian%2C%20we%20must%20augment%20the%20original%20space%20S%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22BQWXKNXX%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22In%20order%20to%20be%20Markovian%2C%20we%20must%20augment%20the%20original%20space%20S%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%224%22%2C%22annotationSortIndex%22%3A%2200003%7C001009%7C00363%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A3%2C%5C%22rects%5C%22%3A%5B%5B206.255%2C419.559%2C467.853%2C428.625%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-17T01%3A04%3A44Z%22%7D%7D%2C%7B%22key%22%3A%2223D55HJH%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Firoiu%20et%20al.%22%2C%22parsedDate%22%3A%222017%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3EFiroiu%2C%20V.%2C%20Whitney%2C%20W.%20F.%2C%20%26amp%3B%20Tenenbaum%2C%20J.%20B.%20%282017%29.%20%3Ci%3EBeating%20the%20World%26%23x2019%3Bs%20Best%20at%20Super%20Smash%20Bros.%20Melee%20with%20Deep%20Reinforcement%20Learning%3C%5C%2Fi%3E.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Beating%20the%20World%5Cu2019s%20Best%20at%20Super%20Smash%20Bros.%20Melee%20with%20Deep%20Reinforcement%20Learning%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vlad%22%2C%22lastName%22%3A%22Firoiu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22William%20F%22%2C%22lastName%22%3A%22Whitney%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Joshua%20B%22%2C%22lastName%22%3A%22Tenenbaum%22%7D%5D%2C%22abstractNote%22%3A%22There%20has%20been%20a%20recent%20explosion%20in%20the%20capabilities%20of%20game-playing%20arti%5Cufb01cial%20intelligence.%20Many%20classes%20of%20RL%20tasks%2C%20from%20Atari%20games%20to%20motor%20control%20to%20board%20games%2C%20are%20now%20solvable%20by%20fairly%20generic%20algorithms%2C%20based%20on%20deep%20learning%2C%20that%20learn%20to%20play%20from%20experience%20with%20minimal%20knowledge%20of%20the%20speci%5Cufb01c%20domain%20of%20interest.%20In%20this%20work%2C%20we%20will%20investigate%20the%20performance%20of%20these%20methods%20on%20Super%20Smash%20Bros.%20Melee%20%28SSBM%29%2C%20a%20popular%20console%20%5Cufb01ghting%20game.%20The%20SSBM%20environment%20has%20complex%20dynamics%20and%20partial%20observability%2C%20making%20it%20challenging%20for%20human%20and%20machine%20alike.%20The%20multi-player%20aspect%20poses%20an%20additional%20challenge%2C%20as%20the%20vast%20majority%20of%20recent%20advances%20in%20RL%20have%20focused%20on%20single-agent%20environments.%20Nonetheless%2C%20we%20will%20show%20that%20it%20is%20possible%20to%20train%20agents%20that%20are%20competitive%20against%20and%20even%20surpass%20human%20professionals%2C%20a%20new%20result%20for%20the%20multi-player%20video%20game%20setting.%22%2C%22date%22%3A%222017%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22url%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-05-17T00%3A18%3A32Z%22%7D%7D%2C%7B%22key%22%3A%22BE5IZTAF%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Firoiu%20et%20al.%22%2C%22parsedDate%22%3A%222018-10-16%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3EFiroiu%2C%20V.%2C%20Ju%2C%20T.%2C%20%26amp%3B%20Tenenbaum%2C%20J.%20%282018%29.%20%3Ci%3EAt%20Human%20Speed%3A%20Deep%20Reinforcement%20Learning%20with%20Action%20Delay%3C%5C%2Fi%3E%20%28No.%20arXiv%3A1810.07286%29.%20arXiv.%20%3Ca%20href%3D%27http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1810.07286%27%3Ehttp%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1810.07286%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22At%20Human%20Speed%3A%20Deep%20Reinforcement%20Learning%20with%20Action%20Delay%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vlad%22%2C%22lastName%22%3A%22Firoiu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tina%22%2C%22lastName%22%3A%22Ju%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Josh%22%2C%22lastName%22%3A%22Tenenbaum%22%7D%5D%2C%22abstractNote%22%3A%22There%20has%20been%20a%20recent%20explosion%20in%20the%20capabilities%20of%20game-playing%20arti%5Cufb01cial%20intelligence.%20Many%20classes%20of%20tasks%2C%20from%20video%20games%20to%20motor%20control%20to%20board%20games%2C%20are%20now%20solvable%20by%20fairly%20generic%20algorithms%2C%20based%20on%20deep%20learning%20and%20reinforcement%20learning%2C%20that%20learn%20to%20play%20from%20experience%20with%20minimal%20prior%20knowledge.%20However%2C%20these%20machines%20often%20do%20not%20win%20through%20intelligence%20alone%20%5Cu2013they%20possess%20vastly%20superior%20speed%20and%20precision%2C%20allowing%20them%20to%20act%20in%20ways%20a%20human%20never%20could.%20To%20level%20the%20playing%20%5Cufb01eld%2C%20we%20restrict%20the%20machine%5Cu2019s%20reaction%20time%20to%20a%20human%20level%2C%20and%20%5Cufb01nd%20that%20standard%20deep%20reinforcement%20learning%20methods%20quickly%20drop%20in%20performance.%20We%20propose%20a%20solution%20to%20the%20action%20delay%20problem%20inspired%20by%20human%20perception%20%5Cu2013%20to%20endow%20agents%20with%20a%20neural%20predictive%20model%20of%20the%20environment%20which%20%5Cu201cundoes%5Cu201d%20the%20delay%20inherent%20in%20their%20environment%20%5Cu2013%20and%20demonstrate%20its%20ef%5Cufb01cacy%20against%20professional%20players%20in%20Super%20Smash%20Bros.%20Melee%2C%20a%20popular%20console%20%5Cufb01ghting%20game.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A1810.07286%22%2C%22date%22%3A%222018-10-16%22%2C%22DOI%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1810.07286%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-05-17T00%3A08%3A49Z%22%7D%7D%2C%7B%22key%22%3A%22C4HCLHVX%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Schuitema%20et%20al.%22%2C%22parsedDate%22%3A%222010-10%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3ESchuitema%2C%20E.%2C%20Bu%26%23x15F%3Boniu%2C%20L.%2C%20Babu%26%23x161%3Bka%2C%20R.%2C%20%26amp%3B%20Jonker%2C%20P.%20%282010%29.%20Control%20delay%20in%20Reinforcement%20Learning%20for%20real-time%20dynamic%20systems%3A%20A%20memoryless%20approach.%20%3Ci%3E2010%20IEEE%5C%2FRSJ%20International%20Conference%20on%20Intelligent%20Robots%20and%20Systems%3C%5C%2Fi%3E%2C%203226%26%23x2013%3B3231.%20%3Ca%20href%3D%27https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FIROS.2010.5650345%27%3Ehttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FIROS.2010.5650345%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Control%20delay%20in%20Reinforcement%20Learning%20for%20real-time%20dynamic%20systems%3A%20A%20memoryless%20approach%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Erik%22%2C%22lastName%22%3A%22Schuitema%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lucian%22%2C%22lastName%22%3A%22Bu%5Cu015foniu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Robert%22%2C%22lastName%22%3A%22Babu%5Cu0161ka%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pieter%22%2C%22lastName%22%3A%22Jonker%22%7D%5D%2C%22abstractNote%22%3A%22Robots%20controlled%20by%20Reinforcement%20Learning%20%28RL%29%20are%20still%20rare.%20A%20core%20challenge%20to%20the%20application%20of%20RL%20to%20robotic%20systems%20is%20to%20learn%20despite%20the%20existence%20of%20control%20delay%20-%20the%20delay%20between%20measuring%20a%20system%27s%20state%20and%20acting%20upon%20it.%20Control%20delay%20is%20always%20present%20in%20real%20systems.%20In%20this%20work%2C%20we%20present%20two%20novel%20temporal%20difference%20%28TD%29%20learning%20algorithms%20for%20problems%20with%20control%20delay.%20These%20algorithms%20improve%20learning%20performance%20by%20taking%20the%20control%20delay%20into%20account.%20We%20test%20our%20algorithms%20in%20a%20gridworld%2C%20where%20the%20delay%20is%20an%20integer%20multiple%20of%20the%20time%20step%2C%20as%20well%20as%20in%20the%20simulation%20of%20a%20robotic%20system%2C%20where%20the%20delay%20can%20have%20any%20value.%20In%20both%20tests%2C%20our%20proposed%20algorithms%20outperform%20classical%20TD%20learning%20algorithms%2C%20while%20maintaining%20low%20computational%20complexity.%22%2C%22date%22%3A%222010-10%22%2C%22proceedingsTitle%22%3A%222010%20IEEE%5C%2FRSJ%20International%20Conference%20on%20Intelligent%20Robots%20and%20Systems%22%2C%22conferenceName%22%3A%222010%20IEEE%5C%2FRSJ%20International%20Conference%20on%20Intelligent%20Robots%20and%20Systems%22%2C%22language%22%3A%22%22%2C%22DOI%22%3A%2210.1109%5C%2FIROS.2010.5650345%22%2C%22ISBN%22%3A%22%22%2C%22url%22%3A%22%22%2C%22collections%22%3A%5B%22ELKC2S9S%22%5D%2C%22dateModified%22%3A%222024-05-16T04%3A26%3A33Z%22%7D%7D%2C%7B%22key%22%3A%22TXEIGDRK%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Liotet%22%2C%22parsedDate%22%3A%222023-09-20%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3ELiotet%2C%20P.%20%282023%29.%20%3Ci%3EDelays%20in%20Reinforcement%20Learning%3C%5C%2Fi%3E%20%28No.%20arXiv%3A2309.11096%29.%20arXiv.%20%3Ca%20href%3D%27http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2309.11096%27%3Ehttp%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2309.11096%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22preprint%22%2C%22title%22%3A%22Delays%20in%20Reinforcement%20Learning%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Pierre%22%2C%22lastName%22%3A%22Liotet%22%7D%5D%2C%22abstractNote%22%3A%22Delays%20are%20inherent%20to%20most%20dynamical%20systems.%20Besides%20shifting%20the%20process%20in%20time%2C%20they%20can%20significantly%20affect%20their%20performance.%20For%20this%20reason%2C%20it%20is%20usually%20valuable%20to%20study%20the%20delay%20and%20account%20for%20it.%20Because%20they%20are%20dynamical%20systems%2C%20it%20is%20of%20no%20surprise%20that%20sequential%20decision-making%20problems%20such%20as%20Markov%20decision%20processes%20%28MDP%29%20can%20also%20be%20affected%20by%20delays.%20These%20processes%20are%20the%20foundational%20framework%20of%20reinforcement%20learning%20%28RL%29%2C%20a%20paradigm%20whose%20goal%20is%20to%20create%20artificial%20agents%20capable%20of%20learning%20to%20maximise%20their%20utility%20by%20interacting%20with%20their%20environment.%20RL%20has%20achieved%20strong%2C%20sometimes%20astonishing%2C%20empirical%20results%2C%20but%20delays%20are%20seldom%20explicitly%20accounted%20for.%20The%20understanding%20of%20the%20impact%20of%20delay%20on%20the%20MDP%20is%20limited.%20In%20this%20dissertation%2C%20we%20propose%20to%20study%20the%20delay%20in%20the%20agent%27s%20observation%20of%20the%20state%20of%20the%20environment%20or%20in%20the%20execution%20of%20the%20agent%27s%20actions.%20We%20will%20repeatedly%20change%20our%20point%20of%20view%20on%20the%20problem%20to%20reveal%20some%20of%20its%20structure%20and%20peculiarities.%20A%20wide%20spectrum%20of%20delays%20will%20be%20considered%2C%20and%20potential%20solutions%20will%20be%20presented.%20This%20dissertation%20also%20aims%20to%20draw%20links%20between%20celebrated%20frameworks%20of%20the%20RL%20literature%20and%20the%20one%20of%20delays.%22%2C%22genre%22%3A%22%22%2C%22repository%22%3A%22arXiv%22%2C%22archiveID%22%3A%22arXiv%3A2309.11096%22%2C%22date%22%3A%222023-09-20%22%2C%22DOI%22%3A%22%22%2C%22citationKey%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2309.11096%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-05-15T05%3A50%3A55Z%22%7D%7D%2C%7B%22key%22%3A%22VZIQ9D3W%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3Eextended%20STBP%20%5B18%5D%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22PKJ34UR6%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22extended%20STBP%20%5B18%5D%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%224%22%2C%22annotationSortIndex%22%3A%2200003%7C000012%7C00066%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A3%2C%5C%22rects%5C%22%3A%5B%5B184.517%2C764.195%2C279.306%2C775.548%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-14T10%3A16%3A08Z%22%7D%7D%2C%7B%22key%22%3A%22JIHPFMIX%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3EPoPSAN%20%5B17%5D%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22PKJ34UR6%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22PoPSAN%20%5B17%5D%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%223%22%2C%22annotationSortIndex%22%3A%2200002%7C001229%7C00722%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A2%2C%5C%22rects%5C%22%3A%5B%5B424.344%2C108.229%2C488.407%2C119.582%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-14T10%3A11%3A09Z%22%7D%7D%2C%7B%22key%22%3A%22B4RA8AI6%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3ESpatio-Temporal%20Backpropagation%28STBP%29%20%26%23x6CD5%3B%20%5B14%5D%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22PKJ34UR6%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22Spatio-Temporal%20Backpropagation%28STBP%29%20%5Cu6cd5%20%5B14%5D%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%223%22%2C%22annotationSortIndex%22%3A%2200002%7C000406%7C00455%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A2%2C%5C%22rects%5C%22%3A%5B%5B212.261%2C375.283%2C291.681%2C386.636%5D%2C%5B72%2C356.389%2C223.561%2C367.742%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-14T10%3A07%3A21Z%22%7D%7D%2C%7B%22key%22%3A%22EDB5CT59%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3E%5B9%5D%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%22PKJ34UR6%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22%5B9%5D%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%222%22%2C%22annotationSortIndex%22%3A%2200001%7C000379%7C00387%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A1%2C%5C%22rects%5C%22%3A%5B%5B191.103%2C442.989%2C202.619%2C454.342%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-14T10%3A04%3A37Z%22%7D%7D%2C%7B%22key%22%3A%22QLKBVJU6%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Naya%20et%20al.%22%2C%22parsedDate%22%3A%222021%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3ENaya%20K.%2C%20Kutsuzawa%20K.%2C%20Owaki%20D.%2C%20%26amp%3B%20Hayashibe%20M.%20%282021%29.%20Spiking%20Neural%20Network%20Discovers%20Energy-Efficient%20Hexapod%20Motion%20Deep%20Reinforcement%20Learning.%20%3Ci%3EIEEE%20Access%3C%5C%2Fi%3E%2C%20%3Ci%3E9%3C%5C%2Fi%3E%2C%20150345%26%23x2013%3B150354.%20%3Ca%20href%3D%27https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FACCESS.2021.3126311%27%3Ehttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FACCESS.2021.3126311%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Spiking%20Neural%20Network%20Discovers%20Energy-Efficient%20Hexapod%20Motion%20in%20Deep%20Reinforcement%20Learning%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Katsumi%22%2C%22lastName%22%3A%22Naya%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kyo%22%2C%22lastName%22%3A%22Kutsuzawa%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dai%22%2C%22lastName%22%3A%22Owaki%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mitsuhiro%22%2C%22lastName%22%3A%22Hayashibe%22%7D%5D%2C%22abstractNote%22%3A%22%22%2C%22date%22%3A%222021%22%2C%22language%22%3A%22ja%22%2C%22DOI%22%3A%2210.1109%5C%2FACCESS.2021.3126311%22%2C%22ISSN%22%3A%222169-3536%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fieeexplore.ieee.org%5C%2Fdocument%5C%2F9606760%5C%2F%22%2C%22collections%22%3A%5B%22TDE3Q4T2%22%5D%2C%22dateModified%22%3A%222024-05-14T10%3A03%3A20Z%22%7D%7D%2C%7B%22key%22%3A%22YY3EQ7I8%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Terman%20et%20al.%22%2C%22parsedDate%22%3A%222002-04-01%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3ETerman%2C%20D.%2C%20Rubin%2C%20J.%20E.%2C%20Yew%2C%20A.%20C.%2C%20%26amp%3B%20Wilson%2C%20C.%20J.%20%282002%29.%20Activity%20Patterns%20in%20a%20Model%20for%20the%20Subthalamopallidal%20Network%20of%20the%20Basal%20Ganglia.%20%3Ci%3EJournal%20of%20Neuroscience%3C%5C%2Fi%3E%2C%20%3Ci%3E22%3C%5C%2Fi%3E%287%29%2C%202963%26%23x2013%3B2976.%20%3Ca%20href%3D%27https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1523%5C%2FJNEUROSCI.22-07-02963.2002%27%3Ehttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1523%5C%2FJNEUROSCI.22-07-02963.2002%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Activity%20Patterns%20in%20a%20Model%20for%20the%20Subthalamopallidal%20Network%20of%20the%20Basal%20Ganglia%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22D.%22%2C%22lastName%22%3A%22Terman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22J.%20E.%22%2C%22lastName%22%3A%22Rubin%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22A.%20C.%22%2C%22lastName%22%3A%22Yew%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22C.%20J.%22%2C%22lastName%22%3A%22Wilson%22%7D%5D%2C%22abstractNote%22%3A%22Based%20on%20recent%20experimental%20data%2C%20we%20have%20developed%20a%20conductance-based%20computational%20network%20model%20of%20the%20subthalamic%20nucleus%20and%20the%20external%20segment%20of%20the%20globus%20pallidus%20in%20the%20indirect%20pathway%20of%20the%20basal%20ganglia.%20Computer%20simulations%20and%20analysis%20of%20this%20model%20illuminate%20the%20roles%20of%20the%20coupling%20architecture%20of%20the%20network%2C%20and%20associated%20synaptic%20conductances%2C%20in%20modulating%20the%20activity%20patterns%20displayed%20by%20this%20network.%20Depending%20on%20the%20relationships%20of%20these%20coupling%20parameters%2C%20the%20network%20can%20support%20three%20general%20classes%20of%20sustained%20firing%20patterns%3A%20clustering%2C%20propagating%20waves%2C%20and%20repetitive%20spiking%20that%20may%20show%20little%20regularity%20or%20correlation.%20Each%20activity%20pattern%20can%20occur%20continuously%20or%20in%20discrete%20episodes.%20We%20characterize%20the%20mechanisms%20underlying%20these%20rhythms%2C%20as%20well%20as%20the%20influence%20of%20parameters%20on%20details%20such%20as%20spiking%20frequency%20and%20wave%20speed.%20These%20results%20suggest%20that%20the%20subthalamopallidal%20circuit%20is%20capable%20both%20of%20correlated%20rhythmic%20activity%20and%20of%20irregular%20autonomous%20patterns%20of%20activity%20that%20block%20rhythmicity.%20Increased%20striatal%20input%20to%2C%20and%20weakened%20intrapallidal%20inhibition%20within%2C%20the%20indirect%20pathway%20can%20switch%20the%20behavior%20of%20the%20circuit%20from%20irregular%20to%20rhythmic.%20This%20may%20be%20sufficient%20to%20explain%20the%20emergence%20of%20correlated%20oscillatory%20activity%20in%20the%20subthalamopallidal%20circuit%20after%20destruction%20of%20dopaminergic%20neurons%20in%20Parkinson%27s%20disease%20and%20in%20animal%20models%20of%20parkinsonism.%22%2C%22date%22%3A%222002%5C%2F04%5C%2F01%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1523%5C%2FJNEUROSCI.22-07-02963.2002%22%2C%22ISSN%22%3A%220270-6474%2C%201529-2401%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.jneurosci.org%5C%2Fcontent%5C%2F22%5C%2F7%5C%2F2963%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222024-05-10T01%3A35%3A36Z%22%7D%7D%2C%7B%22key%22%3A%22KU5AHFZW%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Mazzoni%20et%20al.%22%2C%22parsedDate%22%3A%222015-12-14%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3EMazzoni%2C%20A.%2C%20Lind%26%23xE9%3Bn%2C%20H.%2C%20Cuntz%2C%20H.%2C%20Lansner%2C%20A.%2C%20Panzeri%2C%20S.%2C%20%26amp%3B%20Einevoll%2C%20G.%20T.%20%282015%29.%20Computing%20the%20Local%20Field%20Potential%20%28LFP%29%20from%20Integrate-and-Fire%20Network%20Models.%20%3Ci%3EPLOS%20Computational%20Biology%3C%5C%2Fi%3E%2C%20%3Ci%3E11%3C%5C%2Fi%3E%2812%29%2C%20e1004584.%20%3Ca%20href%3D%27https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1371%5C%2Fjournal.pcbi.1004584%27%3Ehttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1371%5C%2Fjournal.pcbi.1004584%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Computing%20the%20Local%20Field%20Potential%20%28LFP%29%20from%20Integrate-and-Fire%20Network%20Models%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alberto%22%2C%22lastName%22%3A%22Mazzoni%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Henrik%22%2C%22lastName%22%3A%22Lind%5Cu00e9n%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hermann%22%2C%22lastName%22%3A%22Cuntz%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Anders%22%2C%22lastName%22%3A%22Lansner%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stefano%22%2C%22lastName%22%3A%22Panzeri%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gaute%20T.%22%2C%22lastName%22%3A%22Einevoll%22%7D%5D%2C%22abstractNote%22%3A%22Leaky%20integrate-and-fire%20%28LIF%29%20network%20models%20are%20commonly%20used%20to%20study%20how%20the%20spiking%20dynamics%20of%20neural%20networks%20changes%20with%20stimuli%2C%20tasks%20or%20dynamic%20network%20states.%20However%2C%20neurophysiological%20studies%20in%20vivo%20often%20rather%20measure%20the%20mass%20activity%20of%20neuronal%20microcircuits%20with%20the%20local%20field%20potential%20%28LFP%29.%20Given%20that%20LFPs%20are%20generated%20by%20spatially%20separated%20currents%20across%20the%20neuronal%20membrane%2C%20they%20cannot%20be%20computed%20directly%20from%20quantities%20defined%20in%20models%20of%20point-like%20LIF%20neurons.%20Here%2C%20we%20explore%20the%20best%20approximation%20for%20predicting%20the%20LFP%20based%20on%20standard%20output%20from%20point-neuron%20LIF%20networks.%20To%20search%20for%20this%20best%20%5Cu201cLFP%20proxy%5Cu201d%2C%20we%20compared%20LFP%20predictions%20from%20candidate%20proxies%20based%20on%20LIF%20network%20output%20%28e.g%2C%20firing%20rates%2C%20membrane%20potentials%2C%20synaptic%20currents%29%20with%20%5Cu201cground-truth%5Cu201d%20LFP%20obtained%20when%20the%20LIF%20network%20synaptic%20input%20currents%20were%20injected%20into%20an%20analogous%20three-dimensional%20%283D%29%20network%20model%20of%20multi-compartmental%20neurons%20with%20realistic%20morphology%2C%20spatial%20distributions%20of%20somata%20and%20synapses.%20We%20found%20that%20a%20specific%20fixed%20linear%20combination%20of%20the%20LIF%20synaptic%20currents%20provided%20an%20accurate%20LFP%20proxy%2C%20accounting%20for%20most%20of%20the%20variance%20of%20the%20LFP%20time%20course%20observed%20in%20the%203D%20network%20for%20all%20recording%20locations.%20This%20proxy%20performed%20well%20over%20a%20broad%20set%20of%20conditions%2C%20including%20substantial%20variations%20of%20the%20neuronal%20morphologies.%20Our%20results%20provide%20a%20simple%20formula%20for%20estimating%20the%20time%20course%20of%20the%20LFP%20from%20LIF%20network%20simulations%20in%20cases%20where%20a%20single%20pyramidal%20population%20dominates%20the%20LFP%20generation%2C%20and%20thereby%20facilitate%20quantitative%20comparison%20between%20computational%20models%20and%20experimental%20LFP%20recordings%20in%20vivo.%22%2C%22date%22%3A%222015%5C%2F12%5C%2F14%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1371%5C%2Fjournal.pcbi.1004584%22%2C%22ISSN%22%3A%221553-7358%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fjournals.plos.org%5C%2Fploscompbiol%5C%2Farticle%3Fid%3D10.1371%5C%2Fjournal.pcbi.1004584%22%2C%22collections%22%3A%5B%22TDE3Q4T2%22%5D%2C%22dateModified%22%3A%222024-05-02T08%3A46%3A01Z%22%7D%7D%2C%7B%22key%22%3A%223SCUBUZ8%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3Et%3A%2010%26%23x2013%3B50%20ms%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%2233ZVYUT2%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22t%3A%2010%5Cu201350%20ms%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%22922%22%2C%22annotationSortIndex%22%3A%2200000%7C002426%7C00430%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A0%2C%5C%22rects%5C%22%3A%5B%5B290.041%2C344.388%2C335.667%2C352.856%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-02T06%3A22%3A46Z%22%7D%7D%2C%7B%22key%22%3A%22PXX5E6MR%22%2C%22library%22%3A%7B%22id%22%3A8834464%7D%2C%22meta%22%3A%7B%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3E%3Ci%3EOur%20community%20has%20moved%20from%20the%20early%20critical%20questions%20%28%26%23x2018%3B%26%23x201C%3BDo%20oscillations%20exist%3F%26%23x201D%3B%26%23x2019%3B%20or%20%26%23x2018%3B%26%23x201C%3BDo%20network%20oscillations%20assist%20brain%20computation%3F%26%23x201D%3B%26%23x2019%3B%29%20to%20%26%23x2018%3B%26%23x201C%3Bhow%26%23x201D%3B%26%23x2019%3B%20neuronal%20oscillations%20contribute%20to%20circuit%20operations%20and%20behavior.1%3C%5C%2Fi%3E.%20%28n.d.%29.%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22parentItem%22%3A%2233ZVYUT2%22%2C%22itemType%22%3A%22annotation%22%2C%22annotationType%22%3A%22highlight%22%2C%22annotationText%22%3A%22Our%20community%20has%20moved%20from%20the%20early%20critical%20questions%20%28%5Cu2018%5Cu2018Do%20oscillations%20exist%3F%5Cu2019%5Cu2019%20or%20%5Cu2018%5Cu2018Do%20network%20oscillations%20assist%20brain%20computation%3F%5Cu2019%5Cu2019%29%20to%20%5Cu2018%5Cu2018how%5Cu2019%5Cu2019%20neuronal%20oscillations%20contribute%20to%20circuit%20operations%20and%20behavior.1%22%2C%22annotationComment%22%3A%22%22%2C%22annotationColor%22%3A%22%23ffd400%22%2C%22annotationPageLabel%22%3A%22922%22%2C%22annotationSortIndex%22%3A%2200000%7C001356%7C00507%22%2C%22annotationPosition%22%3A%22%7B%5C%22pageIndex%5C%22%3A0%2C%5C%22rects%5C%22%3A%5B%5B114.631%2C268.031%2C215.148%2C275.847%5D%2C%5B59.754%2C257.089%2C215.154%2C264.905%5D%2C%5B59.754%2C246.09%2C215.175%2C253.907%5D%2C%5B59.754%2C235.149%2C215.146%2C242.965%5D%2C%5B59.754%2C224.207%2C215.145%2C232.023%5D%2C%5B59.754%2C213.265%2C186.284%2C222.808%5D%5D%7D%22%2C%22dateModified%22%3A%222024-05-02T06%3A17%3A42Z%22%7D%7D%5D%7D
Doya, K. (2000). Reinforcement Learning in Continuous Time and Space.
Neural Computation ,
12 (1), 219–245.
https://doi.org/10.1162/089976600300015961
Yamanaka, K., Kimura, T., Miyazaki, M., Kawashima, N., Nozaki, D., Nakazawa, K., Yano, H., & Yamamoto, Y. (2002). Human cortical activities during Go/NoGo tasks with opposite motor control paradigms.
Experimental Brain Research ,
142 (3), 301–307.
https://doi.org/10.1007/s00221-001-0943-2
Pfurtscheller, G., Neuper, C., & Kalcher, J. (1993). 40-Hz oscillations during motor behavior in man.
Neuroscience Letters ,
164 (1), 179–182.
https://doi.org/10.1016/0304-3940(93)90886-P
面接選考では、研究提案者ご本人に研究構想の説明 . (n.d.).
所属研究機関において研究倫理教育に関するプログラムをあらかじめ修了していること。 . (n.d.).
2024 年 4 月1日時点で、博士号取得後 10 年未満(2014 年 4 月 2 日以降に博士号取得) . (n.d.).
5 千万円(直接経費。通期;研究期間5年間) . (n.d.).
研究期間は、原則として研究開始(2025 年 1 月以降)から 5 年間です。 . (n.d.).
最長1年間、採択者の資格を持ったまま研究開 始を猶予します。 . (n.d.).
クロスアポイントメントをしたことによる年収の合計が、クロスア ポイントメント実施前の年収を超える金額となることが必要です。 . (n.d.).
Huh, D., & Sejnowski, T. J. (2017).
Gradient Descent for Spiking Neural Networks (No. arXiv:1706.04698). arXiv.
http://arxiv.org/abs/1706.04698
Huh, D., & Sejnowski, T. J. (2017, June 14).
Gradient Descent for Spiking Neural Networks . ArXiv.Org.
https://arxiv.org/abs/1706.04698v2
It can in principle be estimated—in spite of the fact that the implicit discrete variable zjt is non-differentiable—with the help of a suitable pseudo derivative for spikes as in refs. 3,4. (n.d.).
in the model striatum . (n.d.).
McCarthy, M. M., Moore-Kochlacs, C., Gu, X., Boyden, E. S., Han, X., & Kopell, N. (2011). Striatal origin of the pathologic beta oscillations in Parkinson’s disease.
Proceedings of the National Academy of Sciences ,
108 (28), 11620–11625.
https://doi.org/10.1073/pnas.1107748108
Chen, B., Xu, M., Li, L., & Zhao, D. (2020).
Delay-Aware Model-Based Reinforcement Learning for Continuous Control (No. arXiv:2005.05440). arXiv.
http://arxiv.org/abs/2005.05440
augmented state x “ ps, a1, . . . , a∆q . (n.d.).
memoryless approach . (n.d.).
augmented state approach . (n.d.).
DelayCorrecting Actor-Critic (DCAC) builds on SAC [HZAL18] . (n.d.).
model-based approach . (n.d.).
We start by describing the three main approaches from the literature to tackle constant delays in state observation or action execution. (n.d.).
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014).
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation (No. arXiv:1406.1078). arXiv.
http://arxiv.org/abs/1406.1078
This suggests that we compare the predictive and non-predictive agents more directly, by having them train against each other. (n.d.).
In order to be Markovian, we must augment the original space S . (n.d.).
Firoiu, V., Whitney, W. F., & Tenenbaum, J. B. (2017). Beating the World’s Best at Super Smash Bros. Melee with Deep Reinforcement Learning .
Firoiu, V., Ju, T., & Tenenbaum, J. (2018).
At Human Speed: Deep Reinforcement Learning with Action Delay (No. arXiv:1810.07286). arXiv.
http://arxiv.org/abs/1810.07286
Schuitema, E., Buşoniu, L., Babuška, R., & Jonker, P. (2010). Control delay in Reinforcement Learning for real-time dynamic systems: A memoryless approach.
2010 IEEE/RSJ International Conference on Intelligent Robots and Systems , 3226–3231.
https://doi.org/10.1109/IROS.2010.5650345
Liotet, P. (2023).
Delays in Reinforcement Learning (No. arXiv:2309.11096). arXiv.
http://arxiv.org/abs/2309.11096
extended STBP [18] . (n.d.).
Spatio-Temporal Backpropagation(STBP) 法 [14] . (n.d.).
Naya K., Kutsuzawa K., Owaki D., & Hayashibe M. (2021). Spiking Neural Network Discovers Energy-Efficient Hexapod Motion Deep Reinforcement Learning.
IEEE Access ,
9 , 150345–150354.
https://doi.org/10.1109/ACCESS.2021.3126311
Terman, D., Rubin, J. E., Yew, A. C., & Wilson, C. J. (2002). Activity Patterns in a Model for the Subthalamopallidal Network of the Basal Ganglia.
Journal of Neuroscience ,
22 (7), 2963–2976.
https://doi.org/10.1523/JNEUROSCI.22-07-02963.2002
Mazzoni, A., Lindén, H., Cuntz, H., Lansner, A., Panzeri, S., & Einevoll, G. T. (2015). Computing the Local Field Potential (LFP) from Integrate-and-Fire Network Models.
PLOS Computational Biology ,
11 (12), e1004584.
https://doi.org/10.1371/journal.pcbi.1004584
Our community has moved from the early critical questions (‘“Do oscillations exist?”’ or ‘“Do network oscillations assist brain computation?”’) to ‘“how”’ neuronal oscillations contribute to circuit operations and behavior.1 . (n.d.).
ディスカッション
コメント一覧
まだ、コメントがありません