From f3385168b1e7ad1c7ef5d420fdff99689664f73e Mon Sep 17 00:00:00 2001 From: Melmon Date: Thu, 4 May 2023 17:36:21 +0100 Subject: [PATCH] Anything is possible with enough gin. --- .gitignore | 2 + writeup/Drescher-DGD-dissertation-2022-23.tex | 127 ++++++++++-------- writeup/img/interval_score_ddqn_noisy.png | Bin 0 -> 17982 bytes writeup/img/losses_ddqn_noisy.png | Bin 0 -> 15558 bytes 4 files changed, 75 insertions(+), 54 deletions(-) create mode 100644 writeup/img/interval_score_ddqn_noisy.png create mode 100644 writeup/img/losses_ddqn_noisy.png diff --git a/.gitignore b/.gitignore index 581e706..45029fe 100644 --- a/.gitignore +++ b/.gitignore @@ -538,7 +538,9 @@ logs/ logs_dueling/ logs_dueling_per/ logs_rainbow/ +logs_noisy/ training_dueling/ training_rainbow/ +training_noisy/ code-submission/ fil-result/ \ No newline at end of file diff --git a/writeup/Drescher-DGD-dissertation-2022-23.tex b/writeup/Drescher-DGD-dissertation-2022-23.tex index 9ceb021..dbde20d 100644 --- a/writeup/Drescher-DGD-dissertation-2022-23.tex +++ b/writeup/Drescher-DGD-dissertation-2022-23.tex @@ -260,12 +260,11 @@ Multi-step Learning, Distributional RL and Noisy Networks. Prioritised Experience Replay~\citep{schaul16} is a technique where experiences in the replay buffer are prioritised in terms of importance and how valuable experiences are to training. - This way, experiences with higher priority are sampled more often when training happens, allowing for the agent to learn more efficiently. + This way, experiences with higher learning opportunity are sampled more often when training happens, allowing for the agent to learn more efficiently. Priorities of experiences are adjusted over time so that the agent does not overfit with certain experiences. Multistep Learning~\citep[chap.~7.1]{sutton18} is a technique in reinforcement learning that uses sequences of actions and rewards rather than just an individual transition for learning. This is in contrast to traditional Q-learning which only takes into account an individual transition for training and calculating action values (a Markov Decision Process framework). - In multistep learning Distributional reinforcement learning~\citep{bellemare17} differs from traditional RL by evaluating a distribution of a random return, rather than a single value for expected returns. The goal is to estimate the probability distribution of an expected reward. @@ -337,18 +336,11 @@ The objective of the neural network for chizuru-rogue is to take in the observed dungeon state as inputs and return an action that will maximise the expected reward as output as if it were maximising an action-value function. - \subsection{Player Action}\label{subsec:action} - Every action in Rogue is available to the player from the start of the game. - - When the player uses an action that utilises an item, the game will await the player to input a key. - Every item in the player's inventory maps to one key on the keyboard. - The player may input \texttt{*} to see what legal items they may choose and their corresponding key. - Additionally, the player may see what the item-key mapping is by viewing their inventory with the \texttt{i} key at any time. - \subsection{Neural Network}\label{subsec:neural-network} % TODO How are you going to compare Dueling DQN and Rainbow DQN? Is there a metric or multiple metrics? The tradeoff between metrics? Are they standard or ad hoc? - The agent in our experiments will first utilise a base Dueling DQN, which will then be extended with Prioritised Experience Replay and finally extended with Noisy Networks and Multi-step Learning. - Should time allow we will also introduce Distributional RL in order to create a full Rainbow DQN algorithm. + We aim to evaluate the performance of our agent by implementing and analysing the base Dueling DQN algorithm and two improvements, each applied in turn. + We will first implement a base Dueling DQN, then we will extend the algorithm with Prioritised Experience Replay and finally extend with Noisy Networks. + Should time allow we will also introduce Distributional RL and Multi-step Learning in order to create a full Rainbow DQN algorithm. % TODO write about the benefits and drawbacks of both dueling and rainbow @@ -360,6 +352,9 @@ easy-to-use tools for defining, tuning and training neural network models. The agent will use Rogue-gym~\citep{kanagawa19} as an environment to interact with. + This is because it provides us with a OpenAI Gym environment for Rogue. + An OpenAI Gym environment is an environment that is wrapped with OpenAI Gym, providing tools for AI agents to interact with an environment. + We chose to use an existing environment over creating a new one as it would streamline development of our program, allowing us more time to focus on the AI. \subsection{Experiments}\label{subsec:experiments} % TODO experiment discussion @@ -388,7 +383,7 @@ From the results that can be seen in Figure~\ref{fig:ddqn_interval_score}, we were unable to extract a satisfactory result. Our model's average reward per interval\footnote{One interval is 10000 steps.} stagnated around 0.02, without increasing except one outlier of Interval 3, with an average reward of 0.049. Since the model could not increase its average reward, we set out to improve our model by configuring our hyperparameters and integrating Prioritised Experience Replay~\citet{schaul16} into our Dueling DQN for our second experiment. - \begin{figure}[h] + \begin{figure}[H] \caption[DDQN: Average reward per interval.]{Average reward per interval. One interval is 10000 steps.} \centering \includegraphics[scale=0.5]{interval_score_ddqn} @@ -399,8 +394,8 @@ In our second experiment, we integrated Prioritised Experience Replay, another improvement to the DQN algorithm. As shown in Figure~\ref{fig:ddqn_per_interval_score}, we were also unable to extract a satisfactory result, with the average reward per interval stagnating over the entire training period. - \begin{figure}[h] - \caption[DDQN with PER: Average reward per interval.]{Average reward per interval. One interval is 10000 steps.} + \begin{figure}[H] + \caption[DDQN/PER: Average reward per interval.]{Average reward per interval. One interval is 10000 steps.} \centering \includegraphics[scale=0.5]{interval_score_ddqn_per} \label{fig:ddqn_per_interval_score} @@ -408,8 +403,15 @@ - \subsection{Dueling DQN with Prioritised Experience Replay, Noisy Networks and Multi-step Learning}\label{subsec:dueling-dqn-with-prioritised-experience-replay-and-noisy-networks} - \textbf{Need to be added.} + \subsection{Dueling DQN with Prioritised Experience Replay and Noisy Networks}\label{subsec:dueling-dqn-with-prioritised-experience-replay-and-noisy-networks} % TODO + In our third experiment we integrated Noisy Networks to our network architecture. + We did this by utilising the Keras layer \texttt{GaussianNoise()} to add a small amount of Gaussian Noise after our convolutional layers as can be seen in Appendix~\ref{lst:ddqnnoisy}. + \begin{figure}[H] + \caption[DDQN/PER/NN: Average reward per interval.]{Average reward per interval. One interval is 10000 steps.} + \centering + \includegraphics[scale=0.5]{interval_score_ddqn_noisy} + \label{fig:ddqn_noisy_interval_score} + \end{figure} \section{Conclusion}\label{sec:conclusion} In this paper we have set out to improve upon to~\citet{asperti18} and ~\citet{kanagawa19}`s tests by utilising extensions to the DQN algorithm @@ -454,7 +456,7 @@ In future work investigation into why these memory issues occur should be performed and how to mitigate them. \begin{figure}[h] - \caption[Fil Profiler result for DDQN with PER.]{Fil memory profiler result for our DDQN with PER. Larger bars with a deeper shade of red means the line took up more memory.} + \caption[Fil Profiler result for DDQN with PER.]{Fil memory profiler result for our DDQN with PER. Larger bars with a deeper shade of red means the code line took up more memory.} \centering \includegraphics[scale=0.3]{fil} \label{fig:fil} @@ -477,6 +479,8 @@ %%%%% Everything after here is not counted in the word count. %%%%% + + \textbf{Total word count: xxx} \medskip \bibliographystyle{agsm} @@ -503,23 +507,7 @@ \subsection{Hyperparameters}\label{subsec:hyperparameters} \subsubsection{Dueling DQN - First Run} - \begin{lstlisting}[label={lst:ddqn1hyperparameters}] -GAMMA = 0.99 -NUM_ITERATIONS = 20000 -MAX_TURNS_IN_EPISODE = 1000 -BATCH_SIZE = 32 -BUFFER_SIZE = 200000 -MIN_REPLAY_SIZE = 400 -EPSILON_START = 1.0 -EPSILON_END = 0.01 -EPSILON_DECAY = 150000 -LEARNING_RATE = 0.00001 -LEARNING_FREQUENCY = 1000 -TARGET_UPDATE_FREQUENCY = 1000 - \end{lstlisting} - - \subsubsection{Dueling DQN - Second Run} - \begin{lstlisting}[label={lst:ddqn2hyperparameters}] + \begin{lstlisting}[label={lst:ddqnhyperparameters}] GAMMA = 0.99 NUM_ITERATIONS = 20000 MAX_TURNS_IN_EPISODE = 1250 @@ -556,37 +544,68 @@ PRIORITY_SCALE = 0.7 \subsection{Network Architecture}\label{subsec:network-architecture} \subsubsection{Dueling DQN} \begin{lstlisting}[label={lst:dueling}] - net_input = tf.keras.Input(shape=(h, w, 4)) - net_input = tf.keras.layers.Lambda(lambda layer: layer / 255)(net_input) +net_input = tf.keras.Input(shape=(h, w, 4)) - conv1 = tf.keras.layers.Conv2D(32, (3, 3), strides=2, activation="relu")(net_input) - conv2 = tf.keras.layers.Conv2D(64, (3, 3), strides=1, activation="relu")(conv1) - conv3 = tf.keras.layers.Conv2D(64, (3, 3), strides=1, activation="relu")(conv2) +conv1 = tf.keras.layers.Conv2D(32, (3, 3), strides=2, activation="relu")(net_input) +conv2 = tf.keras.layers.Conv2D(64, (3, 3), strides=1, activation="relu")(conv1) +conv3 = tf.keras.layers.Conv2D(64, (3, 3), strides=1, activation="relu")(conv2) - val, adv = tf.keras.layers.Lambda(lambda ww: tf.split(ww, 2, 3))(conv3) +val, adv = tf.keras.layers.Lambda(lambda ww: tf.split(ww, 2, 3))(conv3) - val = tf.keras.layers.Flatten()(val) - val = tf.keras.layers.Dense(1)(val) +val = tf.keras.layers.Flatten()(val) +val = tf.keras.layers.Dense(1)(val) - adv = tf.keras.layers.Flatten()(adv) - adv = tf.keras.layers.Dense(len(ACTIONS))(adv) +adv = tf.keras.layers.Flatten()(adv) +adv = tf.keras.layers.Dense(len(ACTIONS))(adv) - reduced = tf.keras.layers.Lambda(lambda ww: tf.reduce_mean(ww, axis=1, keepdims=True)) +reduced = tf.keras.layers.Lambda(lambda ww: tf.reduce_mean(ww, axis=1, keepdims=True)) - output = tf.keras.layers.Add()([val, tf.keras.layers.Subtract()([adv, reduced(adv)])]) +output = tf.keras.layers.Add()([val, tf.keras.layers.Subtract()([adv, reduced(adv)])]) - final_model = tf.keras.Model(net_input, output) +final_model = tf.keras.Model(net_input, output) - final_model.compile( - optimizer=tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE), - loss=tf.keras.losses.MeanSquaredError(), - metrics=[tf.keras.metrics.SparseCategoricalAccuracy()] - ) +final_model.compile( + optimizer=tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE), + loss=tf.keras.losses.MeanSquaredError(), + metrics=[tf.keras.metrics.SparseCategoricalAccuracy()] +) - return final_model +return final_model \end{lstlisting} \subsubsection{Dueling DQN/Noisy Networks} + \begin{lstlisting}[label={lst:ddqnnoisy}] +net_input = tf.keras.Input(shape=(h, w, HISTORY_LEN)) + +conv1 = tf.keras.layers.Conv2D(32, (3, 3), strides=2, activation="relu", use_bias=False, + kernel_initializer=tf.keras.initializers.VarianceScaling(scale=2.))(net_input) +conv2 = tf.keras.layers.Conv2D(64, (3, 3), strides=1, activation="relu", use_bias=False, + kernel_initializer=tf.keras.initializers.VarianceScaling(scale=2.))(conv1) +conv3 = tf.keras.layers.Conv2D(64, (3, 3), strides=1, activation="relu", use_bias=False, + kernel_initializer=tf.keras.initializers.VarianceScaling(scale=2.))(conv2) +noise = tf.keras.layers.GaussianNoise(0.1)(conv3) + +val, adv = tf.keras.layers.Lambda(lambda ww: tf.split(ww, 2, 3))(noise) + +val = tf.keras.layers.Flatten()(val) +val = tf.keras.layers.Dense(1, kernel_initializer=tf.keras.initializers.VarianceScaling(scale=2.))(val) + +adv = tf.keras.layers.Flatten()(adv) +adv = tf.keras.layers.Dense(len(ACTIONS), kernel_initializer=tf.keras.initializers.VarianceScaling(scale=2.))(adv) + +reduced = tf.keras.layers.Lambda(lambda ww: tf.reduce_mean(ww, axis=1, keepdims=True)) + +output = tf.keras.layers.Add()([val, tf.keras.layers.Subtract()([adv, reduced(adv)])]) + +final_model = tf.keras.Model(net_input, output) + +final_model.compile( + optimizer=tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE), + loss=tf.keras.losses.MeanSquaredError() +) + +return final_model + \end{lstlisting} \end{document} diff --git a/writeup/img/interval_score_ddqn_noisy.png b/writeup/img/interval_score_ddqn_noisy.png new file mode 100644 index 0000000000000000000000000000000000000000..5e11814fe89e1a91c8aa9bf4b60783f2f77f5eb4 GIT binary patch literal 17982 zcmc({2T)X7w=UY|0E(DEC8-1jNfHGb5R|wHA~}jk&XOA%P!UlP1SKbtoF(TV2uN&l zrpY-;Xpkn|vE2Xu&$;K`d+NS=_3BwxL9bqG&N=29bA)evgSU$El4nj|J&i)4&PYo= zP(q;&6Hur_sV9%amG{p}@4}Cxc4E@XC*hyl$!9+B`HKBRb^FKGM)pqn&ka$=R@Rn= ztab*^4GpdAOswr!j#LW6P0Ywm;?E8B?Me z2;Ais;O3+gd#rrbaV!>vqC-hPxUcLSxipM*RwhIp9H?1+J}T|?;Kfn(s|;V3*p-ip zl}f9zFrH-o>6oE5oS|8op_5@*tCP&y@rAX(8CCT9&!oYc! z%ir4f4xO^N$bI~z^iR^_n8@ut@>Y)PPIh)itt~>dz1?Y)-jmq7pFZ_&H}9+qn|^=7 zaSJ3sWd>O}9YTH?7f(I`KeYW6$>2v3RV)qsu(>COh99HPy)f{jYD$}8UJoPEc>1S)cnEaMLj);2eQKFf%pHE+uWc;vHK!;y$>1_xyNQ z%46c_E3!qlo7d0M2qm|+%9l2M)0UKcbCz1*nbVb1w!6P=qXZlpk$bCN99bLhS_}@p zT)u^OMTT^6a9G=RLuQSRrYYZ9E=4YW`0ycCq?E1e2&|6M=f7L|NqFNZ&nEJDMA(p< zf1Y5I{J#+S|Ivfz;?OPb=#j42@(Q+6S(xfJZ0C=$q!t}w#4Fl<9Z)za+kfrlNrI?v zY6Lb`WW{uIQ2i8qAtuw7fZ?p-wB`}w!@)`vvK~PgM%5RH7niAGmEFMlAI;~ z7G3-j?PQU-FH)&*$J4VmE_fD&a@Of$A(;1Ob{AF8ogv~!0)maUtl$yN-gMUQtwB7 z6jN09Qnp@Ztu;QB#QznID+(3Ko~k&~-=E^`?X5Vvzi2FycI3oaQq0i)-j2f_p;tCA zvvMD^H>cncNUvkp!c#KgCrB@S?AS3GS=q;+$}hivX^lpwrKhJm`yG=y%{3W3HPfdJ zFITiOdB9pzOY6kZ72AorIXDZiXq@J|QWRz7B_zBLzwi_g6coR&7wEXy7dt-wR4$VL z5i2X}+Rk1Hp+{?KYbW1r`#BZ_rYi8Os1PP>Zc02ZnSf(({jJE#lr=vrw$G7Kl4mj4vMLC9Qd!RZnv4@F5&^#5UJV z{zM=VcT;ya%4mLR=}dk%=AoWmj4|%|_a|+iX3niUmpN)H9W1n8R7oZ~{_04@_34IL zvGVfr)9_(maQ6+dEmnNsoBpc;i~bua2jinDNzqC!F2w@_1NJjfPG-wP#mwyNv^jpA zNeXHC`8?qr)Hn{(OP4RtH-~Yp?UZ*fpCdhPNa_8@5Zn|b0s7#JMvcC;PvoM~dI zzjb?qYpAO$UL1|zcHzQlSXx^0sy!{o<8`}xdtKQcgx#-^712us_1f65=i%Y$*LgPu zTX2?E*uv>bFc+yfh>@FBpP9*qixQ9DUyq}R;i}=_c9>U|e&kKu-F4e`o8fEE&(FWj z&Tic{v9`9>)zg#s^5x66OBu!0BW-P8tlcKA5m)^^(vgdma9eLErvbO^_ybn$B1L$x z#+xYpi81BdHx7_3tGrxiu)y-MscA~0LX4HuyD-oIaBt3Cf|{wlpk+-13l z^wqH8t0L}YjU^Gv!;2x-G69V`L&3R8{p?uvw47b41c#WQi(T>3pSWG8FP6&YHm5vJ z>KY!-Fzrl4L`1A8oG?RU<3XKw2?7EF_KSU-I#@!tW60CuiF$t)78cnHO!9f97n`X0 zZA2Lu7@|phK&#Bm%^yE{6aXeWGc!~E6O)3fl~pEuN=-_V1X-AG-_C|dczAie_o-AA zu&S!Tr%6Z%(-GD(fsV^VDaCd(GV|6gbIZ%Y^eqv53RxAzRQkF6Ufs^fdv^2$h1%kk zik&>GVFB#!1T7gBzqhls5be7Bi0+)Cf`b3%(qPgRE~6i|KBD8H*QA@)a4{Q=j2nd! zocF*w_FbP32v!hU^{QBpT1hS~)l$=SNuY86tuNj2NkKTpJhNB=+)3$j<$*hT_g777 za&lXd?L?mKgoJcQr-DcZNWIW5NqYucA-G&54z43>|JNIl*~ADdB@RmEhYw$8mM#mt zKT9qeU(GIKq!-C}dp0ZuT)DGGS8PRm{DZ8rwZ@Wd0%_u!J*#n9(o#~O7-gsOn2UKC3{L1#MKbAwG8 zzB_?@_v;ElcVg2v5`~Jti#(XsoJyOT4-SGK{E3euvlTs|@ipQD!2R7B-_#)JTCpBR zRV{!+@=~&t0Ecfw2IIs^X(|p0q`4RM+$gNv0SlIjEbgl5&D%HW4xu~|mtcZCH-$V` zRu*9YLvX4jFX1H0QyJED=f_DY6zU1Gpcl%KkF1}@fZuIJW}NIX<@_@v-6e20C!Jkx z-Y#E*h=UduC(4apvl>cfhPUqO)u9Nb0uW?uIVKS8G4!HgI#rZmh@gR%*#vv4) z3QYY7rcUV{c@2*5&qv>a0Q+0%7DrH?&u(-bIW=0WL7)+YD5%U^ku08rz3}oO)JK_< zgsNLgE`K%&Wc!HHjZ#_3pE!i7Vu5c?mI+fcb~s&EZvda!5BqYs$Uzm^PbM)m$6J&X6^{MDXFZfnOeG1p%ut&*m&{76)p*CYHHP><8-7eVjw0u zOcnbTbCj<>ybAh^wP4j6u|SU1}&e&VzG_Q&7butqXi2@=|)p0P9HrS82^NW->l{P6PvEy ziI+onr1$g(mRIgO$R3>DiJAFuE&Fl}2fb;VL8Sei?7O5jDp~a={{#VZ5Cq&1d{d-h zZJpK8(Xq%@qXaWVqu(5TbzdP`ym?}Ez*c%MOY)OptoyBiUAi|S)8O_oYYj~KhE2ib z2Q71*N!mPZ=)YkV;P? zj)j;JA`)dq#d9U^S{fU#{t7*Hf$`^zphypHW*vOGLSWZ_arNYtJ>Y$n)zwW@=KR|5 z_)N>ef|R^GWr$-6OQ~eH-GrlrT$E~SYaK(*2gJFWGi(z=XxV!fA89JjFD&r5@BVIE zaOO9R%1^c0Y~oA!*;7EFm#b%%+6_*N+jTQtpVpYt+sw=ieqa3Vu{|a%eU-SPJqpp` z{PJ>7o@pl{;0ZDs}4FY6HURaojHb0^wM*4b6_W~ZiMof_n5SV^QZzenx;=pz`eM_}0l`}4nl{$mQ+z{N2JpjRBWK$$tPO%1^$|B zUJ20ousK4?wm>+de8K1Kp^!z~sc7cW^Y@1I6_7J_523xriJ2dzD*__JMF@C@XmuFK?TMw~<02SqYEGe^G}5a$&3##rT+huPVsH{5Wfg|?nL#Sc$;mDs zh;`Bip($f&ZnO7y>Ib6JqW^dSaOuA|c(V_qK3+XZkaB6TqRlo?#ooZb{_%mdwTeeDvkXm5#R$;Mm*0aBJ3 zm|DgQQ&pq>@C1sE=iGciM<6;5|9P>2^DTzrhK4J6uBx9PzGT0{_ru@}{S_h2T!iv$gl3*+s<4Shur2Mv9IuZ`+`F^|9Qw1+(+Ly69VqoXmLL3UgUUawL(HR8!; zB^5R=(t@1aV(8IFj>vX+PTuIcY1ofmYbQw?(hVC2CpieQ23gL|OC$)PfDLI{p5sV= z4l7E7og>RJShxHtfxMn}kT@rjV~w*tcmC5V+figg`}`GG=nl5Nu40yzONR}sjzrr0 zS^t|qm&Z?GGE-jhdehrh|CCf{PTi{F1b^b_g*j22kfd^c?BuH5A0?5Z5i5T+)Zr=- zm^Wq;*AnC$^H@R{qmKk4-)Lj;`pczm>GY;pomVYERT+yVPsXZYH+Ig=Q@`sb{>X8a zC~qLio$vRBmjGLI)^6m7teA&UzOZdsS?c0vlFKO6ME zB$wIC24P1)erl8M=G%cL-|xHqYyGbjR}`ns(5(D#$KnJ0n_VFOU419eRk_|Os99p4 z0)BHNKMO18elS`%q@}63k-(0X21lz|;a2QNb?-c}6iFclNA?R;GqvxHj@;2HQMQ|I zZikd3%{;4I)w-OZO8RC3o!)a((^31I`mh)X^!0*6vd3hAWF7?XO9T0`kmmGEw?v#k zPxl+gbPpBV<=f534?B%uHA@{c=uLYwbui1tGu%ealu>BGZ!23HbDbD)?e?Qphmhnc zAt8a2CXg>Z{2s%_&*~Y4!H@rmz{fACB(E$kcD=p8#BGfilJ1b>8;<`sXrQ8@aa$VF z6h%2Xxmg@eG9d(Y;Rq$HF!^4fOt$uTKWz1c{_xr@X;9bPNhET}WP-1=bPlymn zMp9f{3{cvYRog39uE;1Ts2Lm2e2Xp5&sWjW(a|h(&UO=y5#B1snby9$RMXSm+na>P z<4t6o$^a!WG%{-B@NLV^xD}p!M)I1<@M7-*la@;GOe^{87otz9Z2OnC(i^wOGWYfy>Yiw6-7Qvd8kWbhQRS{Y;-E5 zm6d5j7GDAypaS6%WZc(jJ%}5o`2_`a%2|&gUZ}1Xi@IlLR3-en)UG4H-`d1{*tM~g zdpW7;GVGnV{w%~Ejv+Us2fDis#L=}QU&Y^yyHdl-w)fn5cr|mkuH#qQdkXe{Jsuh) z-+&;O!UnFAs)VGX`mT8&v-oe2%hzfD9Z*iIMPfY2|586SHIgND%>`Li{J;NGym3SNYGI-3{N}LDX3MAZ}iE_CJU?AkI`0(MwXmi8G+1a*igL-?25CCSF@}m{WOi#Z9 z09E_4(j6x_9zmZ!JG0RhT`!(I+6BX5cQ#F7C)+<+koY1uFv3lkz-r?p2B>8ce`Y{WZ@}4 zm9;@x0#77&9R7#IV-)I~e{w{gtc=V?BTE)3@i)vO=F!^J)QN;4i!#(YQNm zz@nN?X=`V}ti@v|bE_>Y%lzc9?d776!xVSAn^Pb!ygc-%hoSj0)sK2p@G7c#S$!@R z@|a$59m)@qole;~L|>)?N2~3lxN^fH^g~R4uGQu@!Pmtday#Ixe2&H9A2we;qq<;V zTu>M~BFmn=^Rm`rr@!VkqKmySuI z1!P(34h)v9e_7t3_DmjN0v0C;Z1rZ?$wiuN{;NjaAcQ*kyiTLV_W<<_|IZBk&=zbEc^N?&Uxv!thjFxt`urW0hS%20_H{bMt7=)!;b0>Yw#e zKki=cV66v8={$H(a^FI#D>rH3o)GeR^vg53c=DyASD19Z{m5a0z5W4y_qs(WxNIpo zFAUk!e0PqdW?TOUc@c8M_7)KFah@d5RD1Wwa;|Z@Jnf%_xL{l|d5e;_xkQf}JEx5V zMsF2Y?)^Ok8X2NyQPe78f<}(y{RMy4fjRKKbnbr-O~>@XEiXL8{|q2Seu^uP(tAoH zBGgLbqVNzTAj9&jE1!!N3Uuuy92=Ng|jY$e1CaT8js{7pX^{~Uby zTH#WszpzPrl3!l`bK9}00Dmis7I^&a{p2lIK+cSa2R zPHMxfyx+{gJt~OnbXVCk-Pz%nl`Xi_I!`S42SW3D$eCJU&**dKqc9fj&Hdoyd+54c zhf>2Hi!EJwtA3r9wOLQSY5U1Rz*IhIZIj(lxf`;#0$#+6AAN#`VykzXNQ};0nXU<= zxYFI?K42p;Hmw0KnGW!X+jMR<7G;NQam5@o_}1L#5-(I+BG0d?*TPc73G@Peorj;2 zu>s-)@f!9W$Dri>gvDY2eE6_ zRaiFnK;!S%eEtv$(RJ}z^Q_zD*gA3~4uD=tkDE0rnw} z6|U7T((h^I@X*&W`lcu0fjt>_9iSU}c=jhLE_+CbIUv1aCnLnTi>nn*o7c^*q%S$U zLx(K++wt&S0ppiS{Hgj&EQRs`S4>4fUBBrj4j-f2hHX)fj)}|J)s{St520wjYV#>g z9gqICM&-ngoOhoyEQH(5Z(r!j4Ap7y8Pwc{IvW(1{C?E+yNk;e+RCz=jc^0oK_iBy zVu}8cs~a{UtYlp>h-{(y8$~k7_YM>3$bqe4t)K3ZtN0@fjUNSG&w7g5c=#nRVxX?V z!|Y|1!X5B@^u+R%0jOp6^{$#r$B*y$TVyg8n;3vb#{mLUn3p#zt49n(kRT|CP~Ut0 ztj&P521y{ns3TT?X|V9kz7=>2;$~rGMi7gdrpkGtx*>k8;YHf)GxeZ#aH5THA`|?u zYM~~vyh!G7S|Km3i=jnL4E#Lv}dR z=1@)n*I<b=&=LQVUvid&wJ%5ttx%7lL6HpeGKIIWb>3v-~7<-1P zJn>A;uSs0k?Q^_0fH*H3od z108;cgdqNQ^2EJh+RcBJeS!n2n*MXyCqL)Pzso-TGi4e|kL62vch0a9SDZFH*eY*r zIF*JIJxr2*YuV2#KGkD-Au8%#sRuRRQ+oo5PuX4>;F*$Qn64~?eVPtE0p8D@W}Ql8sQiP7Z;aD9vT0t_$-G<#s4=I zAJ)Axk&k&UgTmeRC-TuOdTB=T0rVf{)pzfKY5T(k7}R2M*m0=6q-NsylJ9d*Z?F08 z?^Vunzqo(Jab_huYV#Mc7T)^FiHQ=q#yG!~3)musz{v=u$x!mhB>5;MC#S+hxd+5VpNDuAJR4+8RF1HqtOtjsz^(}g#rcSRg+2D& zmWGNSg8~ZJj(g!f4zNwHXq1|o#D{EoFHIMfPW26OB!F9B|8}Qq#N%KOLeWP6>+)I- z!~Htp_c+(n^%`J~mBo|a7>BZK>RuF1i&lk5%qG^$FYjpJ($ukBq5f%d_rJyt0SF-aG=YX@ zqE>XpX!@9x!-57%&QBF6~- z_xSl~wz(fWW^856%g-V9tCK;aiiV<%L@4A@AIG6QGx{9vqL(gVOgs2oQIYw*74G5% zXafe>)s|NJv+C8vUn<0V2K)&}L{rE4z&wW*%jgBi#NF!;n-T6YF_uQ5t^sgCe+)g+ zL!y_UcP~!SF6W=SJ-wJkoYrrvw>+?jY<=B|-ssk^Ru=vWtyg;X%b; z1oBCdZ}I)jY8A{U7xIUCXjJ@JF|?-_O9Y2}$M%OPAOCxG&x>?1>pN#qQq%xO*#xw! z&M{d1K88v`*dT0o2bjz4-Ftpn~K{;+55r6@JgKtKRN9j0B|=y0O_+zZlpVUYWrUZ&&W2;-LV7;@AGoyeD<- z8T${pN)(*~Y*W=(wc4EIAl)^;og;t4NK&U&KNwInG4FYLD8TxiX<}(aWs1`_)GsJ+d4A4L>dz7(mgcWm`14X(=9NbJggVrf$lwI&|KMn- z!0J%6?^w zw2d)%EhuXM^Q2Bqr*7>=at5}J1a+r;$5Rrw6L8Y@Vt!x_JCmw~JKJ}`AhX(k6LWf`^ZvIJkg|}` znKm-|K6c2}>u#xt_nJ~SDkmeQqfHU=P!%&P{aAcBJXr;!x=;R|;BbaKtI^-<2^l>N z8kw(0TF}S?ODHy_X|Sna>L$Q+4RNovM&cW!f0Hp zFzK*A@#>n^)ez7m8@A+2YpD1Zs#cw@$!OH-_f~}f5kkY$a*Zig=(G%yz;!FiQHOs& z6BMHrws@fZHfn{&IwRmN(>zBO%@~_@P{?ul>};Y~C+VirL8+;%CCyC|=VL4cRAAju zx+$KN`BAl`c-;l5>K)IJ@1D}zZDdrJqBzd*)G~H_$WrBawdWm2`}!rCWmD7* zEbFL-;bkv%u2K5bN@0z(4Mx7r?t{_t$CzIaLek6}Q`?Q2nC#ZgT-^N%*g`K2ROZ-Q zv+_wE8LyH7jpI~%XE0oK^>*5;=gwr`o(;i*^!Xis)f_x) z_CV*S+>H+xbej2Tze};K1+G^bYazQms!!xM-qzIc+z(LF2GqJ@a^?etuxlH(u`M^G{N0lTkFYau8;mLgUi=jz>PL> zGSk)Z+Q9e(*^qgtyDDmGhSLKDMUI)3HGlNnq4;;tnq9#$&@Gof_0H-TBrj+d+!2U! z8Ot3vxR6l5*fhdw8;71*{<%1PX}jG^1lP3LBapa7Yg3XezIW4K=qr%M%%BDZ)te4L zXbFTY4{Vh|GjWShOz>mm*X$U~tV&Hhd~sDj2jMcIY)- z?8_Fw67%rR6K8-AqvdFWQj)!k>0&P%#@^AdN$-3&CzLJKW76(?#$gPU|8o(0>QEi& zns~YRSP-2@EQv>JwaD=&CXez*>>V8u3R^gzbu|4P5Jx(otacoYq!g*?>7VIUl$DVv zEHKY{M0nVF;sb)o-oO6>LN5EspX3lzIygC9n0;|kkyk%5{)dJ0Up@2dkODuo(m$n_ zpfG*B-^OY-jm)EWs&k_L?C(-#lbU`zD=#xJc1IZ5rI#%;-l{sY_+4VP_UwI_uQbxp zara9%BOjj{2txhdS5%;-a#-d0J-4eCqv!ivN2om=s5?MR)Gkq^&ftVKycg z+Brz2Wq#U*#SG5(K7bN&>N=E&J6{f^>x&fXQwF;&d6aIRmcy&W0eTGdto{AQ2J*d9 zg5=d3sa&crMCrcbwr5->HYA!f(6+x{`Isv(uJ7c*OTTpTD1DR%IumebG^@L0FKx;% zJ1K`IjlEbxdUU$Ty;)likJ|YC12REXzDOrXB*ZQAv(+DQKLW*TAbxsukj@|jRmUWc zq=%TZwio_`ezvs{0Wk6#S8Y&1C@Cf-#kgw%&I%RHPSEg_jt-@ukPxzMYxx-AjGmsJ zP;3dq6(@gwA zc)IVtf#4PuwZLThj2ZpkC2hZ*^`Sc{w-=2wQc|S1w9Q6-3D>DdhO-(#d?m(!9Bou7QmX=H3V%k;KU;A@2idlD~ zjVjHA(}U=Tc@93^6yE%g;`aD{iO5IpxuKI4Z^AL%WzX&TF-SvkybB6R4AL1H8BnkP-?8DO5o8(gZQv;;A%OJW z)`*&|ZT9BMNEX6`1^t5rYXCq3qvlZZgMh>|K}HPZL9qy4%Z_j!b9!mMA)8mgPs}%IzcGy68b&BL zj*bPwKz03th=V1}b6$Pg0-S(e9jB<*Z%$P;Ef#T^^RF-XUE~hrwBpiyTVX7E=WF@R zZ(4jU;aUB~A=>d-3RhM_d%Ze?W@boFZpv=;3}};&Qa$2JfMURGkecq5ja7$*p4a=4 zAt6Z=y3YD-29T=*+QKtAqm%u#)+fT?kn^ihd#%*7uh=QA?}wX#Q1b5@iHq{AG_cdRvWyy7DZ9k-sx{wojxBMHFLzd z@tvWn+-mo%eD6rO*B{A^D3G7TB)VJ`KgZYJ)63H*+|fnS`~L0(AZ@?wT@YWv>6)ys?Pg-dq352-&&K{PCF ztioP+L9L2?=;w^r=9d8elq})s_c6bmWFjdSA-k*Lm7(BY%O8a$0`m4qvPd!&xL7tB zCOcW5zAt8lG;0k-+|RFS&G8QUDYz?9^X@%uuAwpHvXa{IUL+pWMX@U*mdy?P(+^XN z-}abh9ia<=kTb9~mF|k{fx}&|tg8_`0_dOp!)n&Gc%K&(%CsNWW+zXLP%dorzK950 z;vXyD1ZRAHIp8ilr`_=O=8c-Lv5Y$+^_y~KH*Gcbuw^@EBY8{lLzG&MY&JJmPi_^p zeL8Rzvv#@y^m`i+F>|u3C20&nHnnhjxHF5Vg;ZBQ*fb{g|N4+E?;eG@G)4nFoT-#8 z`MKCcU7W}Y^323WOA;ef@8xB;w2x>k0ePZ2QG>Tm;KZ14e%MTi>vJxl13RhY6Cdr3 z$lS`XL-JPU_b1P1^HsO@6I@$gOT2}_#flWrVtm zEjVCP%`{wO$o?=@6An~(Nb}7hSLNR;?ru1q^xFtjNh%N=n^;Jh13|Y{OnNiO&PwFRyW#3X^yCe@XMPvxnA}O0hjC z+VcB#?3J?F3(zc^Gij2v<7K}tEj{BDcl^rUb{<_CEzO62>b;ZmjrV?ad1?GOBxlJd zo?KX`FygMHkFpO@%#wCIC%1V_^9Enb-MPAmYNjcvnVLO4+Fhxg0awe+vo+G)Dwwyh zArEKO`n>x-Fi{j$1S(n(5CeoKrEp#YE2wG;6{`-DLqtN&E21Yj| zX>V+w=ZtXYR>fuV@zud3JJmH)<3!mR;E11?hbAhkIL$bdth@M6aq(z5WaB;0AR zdwg|k;eS!XNg&SnZ;6aJ4)u}Vl%4u}F$|${G0qJ{{D8{~qleezf7huq4#y{E=?1el zL|-e=H%Y0GVTbIsNc49j+j*9DEJ41=MP0~}-?h~;_;aZJr^X7*c3=DmG9uyc5gCor zBKSC6K&c3fg7Mtr*kvTaL%r~&QPsj2i-aNeUvss#N@4(N= zw~c*w+^-ws`c1|IvUtz9dxnO5L7^0is&76&!zI$n#YY&`Fb#j>yiD;lAFrF<5d$Qm zQFVE(ez{9;PY*jbC(^c*J!`E z|D54#7QTUx?3!r;k6Xu?*h4k?o#|-GuL!Hi)1bY^!vDcWx8QcjiVaJKEbbUa^$_`~ z0h_@0PR~z(N_~qezQP;%NzL)`iy$n)D)fo{m`qwRX94(4Ff%#D0!2_2XqcLOeYg}f zwa+LXZzThiiS`p;_^_8gpXN09ZlLkXYct)u{?dFt8oz6=rxAMW2JbP1 zAtG`OnQ^3Xv~mpCf0g351eI+5SXHO~qsFZ~SJN8pEtf-=3s-xt0Oh^$Dt8}ws9P2t)lr7ICe8THJ2!1RH!l}C7)EN3{xrPHwl zp%5tti}3ujX&Z6K86(xSj%4boz(mV>@=~#6OK||78dZPNGL`B+h~)W`Qf_}G*ZV0Y{qMtvCm*Cs_SlsAwiTgJ_%XTDm+i< zjC$++)}<$rML3KJ8LK{xWa+5D*r}YUav7T`abDj0Xr`XvkMvV<+{ksHgYO2CJ-=7; zwba?Iyiv}n$b6wfyajK9|2UJ?mH{uCLx{IU;*FC{96)HJl#xYIw}pIsy$c&(1Z11? z-kp9(MshR&dCe*s?pCun_})5R_lZ&GkvIcK>@%1N(aj8~>p+Dr8tih@vJ%HjDa;Fy z@qssQWI(nI?KKcza`eZNdvoH68S{S*^g2SCMuNkWjRUtwJ_7KWepBH^j(5@3|%py~R zHZV9D41O+Pj5rUA`0++AH@!pa1;BGYKNTxYkW2pkgnusez5f2Hf5VeU$;A23m=yAv}h9}kUTtX%&l!s!LRG>a3v6kIE)(P(-32YRk zHD@D50o%BogF;boc+9Et?d0G09X*n+8XUtyZQ11=_kp zLAOOj^ngU6A}(%LwsaI_^_W+GXj^2xiSnB^b8{wHOk{1fCl6J-duMldf-GAx{l-X$ zE+z%~T;5>Ia5J3`DRe&jb_6B@85mX%9V5UC%>k0BgT4LiVt;P5Ae*#+Yk|k7Cy*Jn z1gW!7lBVY7c7DQXXlX@YFc={h3xNC0tgL#VGvz$dK{!C)Yk#-9sg7~iwI6y$pkAUN zYz*}ipjZII1mSoavC*|fU zl+Hs;ih)x2@|{Y$^JNy;Zq(tkkb@Yl&Hr)KD0F>yL|Y8CIg!Z5Wk^t|1<+t4U& zy($8OzOBdUQh$8Gd7DZJgloec9L(b$d={r`E%f;?y0PKP{?brj;Q6VU8K$_r_&FeHX<3&i0PVwcZL+~J z#2^JE$j{3&fj%sRv{F|mfpoI{=uNu@c|0{5qx^^2`?MIc$7JY@`KOmg(D5%|lga;< zDW}(=C3dEJd1NdvTSf8Pib0`E;dif00jR+os7}xC)d^_$c`oTk)$V(ROQ=-II@6D_ zgAlk$EZLCs7+1}%0SYnT6-*h|#upw3u5(eJJZcYZkUF-tm!!8b4P6}_=ZU$XWe_W; z%+H%aINz|cl?FVUPmFR`H&=liRbA}BZrM4s;U_Un}+D-AR0pnn<~8t^@t@Ndx&J@?`n zF)XyG0TL9r89DsC*j`qtB2ZaPEvexX6AK?7pR^w5J($oQ*jozwk;e|dXp5mHpNFO( zGo*DBGaY99IXXIX(5h69LeDLG*bqv&sLW9x#hBpW#0cw({^h$!RYdLi4=GvXmp>*4 zS{%W7!l(bf9TkiLRBZk~ZR#w}d%{7PSL$O@292(cA;=2(tCtE#*cx8{Y0j(?%^N{3 z{`c$6BqhMM|DEjt>wfw9I2<{`-in7F8G7TRYKCS4Y-rl?#sF#w+c7p=63Wy=SXd`>ePCD5A*`=j! z=(HkCv3qD7?BWinh(-UkjpACGZwv6cHo6`w%t3D#%`aQ_NChM@wf5K<)GU5J_0gje znz&$4js}(5EYP{i-6?j5g^a?~%uK7Q5rh>PbFe$3qN|$;y}S|3zQZ-&n-U>4D$K41gFIG@#!X z9{lz{XiL|~-Ke46S8NJl?_I}*`w>hNv%fuqUf28_k7$1-NRa2Spo+ANAZK{ReM_*k>0Lk4X$JK5WN5{J zVrQ)X3cg$mdNBvG4c_otnMjvf#o8tGuc{*hxyJs$ zlDAC1i>WE6VK61kcOxyAz*k5B8`m4fu9m5tlI2cFLzYponV)3}B>2D~kF%@WZW!WR zQoiGo=H4WeNIP^iY^)sYuLQ=+MexoAYgx4emkmkoL7ggdiU(%x9UN{$M+>y1l+1p8 zUq(LP+0pTbU%yrzx(7WuCIHq3#sk;}{EHvc@6r+0Daecv{sNV?wW&%pH(!GCDjaOb z9Hh9el>gCLWUJHEvQ+U7&=y%@GpuMbDHv=HJq2;tNf;mnM-KX!3`CO@@S$^$6g|T z7XWw(=(V3WF^k#tLaSG~!4K`>4tD*KzRSi&nSYw57ulN5Bc{0mPUHet%(gF(ioXCJ z%xurO!~keiV?)EKf0~Fa(tlpn6^@bW@OR(1IgDAzAokS@BTNAdRMxwp;YG`;_+cR) zlZKEnOJ#GKFDMAbBK=Dc(h%V+eaUj0VsQ+)7*x5xP^k)@fZh~(E@DY23S-0>{|fz5 z`TKq4$F7mYBG;{DMcD|K>>JXRJHso=y1G_cQQ1YHL1Kh!-ki7kN_o4kdB;MBPKxLy zt7H}qzMWR5inxW$Qh6YAL&vGC1yF5bb|;y{)pm8;AxMJ$UhFhiOgrFbDypi9P$n4A z31$%9{UvVMpY!lwZ{#3@5AQ4v_$G2Th7B2x&0H{c1*xEA+Ip@@Z$Ibnze->VpeAN} zDumEy;6E2JvYQC{Hf&hSN=a(govMOat)<9!TG7SMKaT0}_t-T6HEfV&FOM%NDIwSM zutL8PT5sk8uCqB0tn(1<0hsM9aHRr9LeziJ$pqTm}&C; z%hCFjVD7o<8Ko73x8cP6DW*VAjv+Kke??3sP!!q`1`^Pg7FO$N3ro|BARJ&}Vv>A+ znKh7p*loQ9A*{5hXc#~N0=Te#Y?ZsuKTr#*LMRYTO1QxI*GJq8$zCCD23W_@A1-M$ zy`T96j`k6lwUVh|E&CDdR!VHF{!d3TzHkoxmzP;JlCy*@;n1yo`v#}MF; zI-EUSYxRWEyQH-Ajdqu#VRFc1BhyoUh?|gzi$R(=L3@x4aaX@~fCC~r(1FpbKQ3gJ zchEM%2!D81g9UPALccsOv+Jfy(qg;7JVTRz<{!KgsyZL;vrt*_7}Sv>fBFH)Sc+ga z@R$YGOB57?I>A%ZxR|UcL8nx($V~E)Z96NKD`68G^V`pAyw4D`4v6ev-sN!Ut;oc` z!oGU!^?Kx&-jQ_!)6fmxPD^@k=Oesrq>m~h?cE5R@|9KYgCN89jF(X z%h~3<*u6n9u*S+7)cLmeR_;5%>H7hef1`W_tKhL| z?7{owVjq<2##5YblqII7${hS6JNN=d+qBy2)IgDK+%SU%{NaP&Uc#^<^SO?M!1$b; zyR3Q@L({40PBmxvOmNZY*zR_)&1(B=CFf=?I1#HUD!$aydT4|CK<^;ray ziM`PiB5fc@GDP-3zVD4

>%g=i%d9u->1x1{}T6Dry0D9oSDep>q?doW~#jg4D>*RPJaM|Zr`rKAO( pA1a=sx8P3%HT(Z}#5Z(6cI?Hn_zK}53W?~XAId+-6npaWe*il%J7)j@ literal 0 HcmV?d00001 diff --git a/writeup/img/losses_ddqn_noisy.png b/writeup/img/losses_ddqn_noisy.png new file mode 100644 index 0000000000000000000000000000000000000000..969759ce5464f9db0ca3bb04aada01347d4bf4fb GIT binary patch literal 15558 zcmeIZXHZmKw>8>`ii(1Yq69%DNf1y_kSr?DWQhVoBSA7KQDUQlibO$j77)ofMQ%(kG759z&r} zC(tr?RZyq{YbX>+`jNx%jjwIRbvPWflSHc>fq!mCjNZU!8vA<>>{V?Z+dDn7HAb0O z+gKT2wKKFeHnz4iwXt6ytr3HhIFOU>*cv~wH?y(6sAgtmj8ZdoxX8|BXXF8MaL}K_3-&3xp;?pnaRSrj- zl)WUeV))QeoaW%vIi}~nx3z~n^a~~Wsc%Y}^ib!Y{1M&geN*8Iscd;ye#SPp_X(k3 zR;`J+sk%^ zHp)8OABGug9e|h6&*Z8gha-wt%*zE1wNGxP?RlE=r)Oqw7VANnaSXq1ov17$ zDk{ps&0Spa{;7q9MNcw1gx`7Ds6vjWBuP3DSJEeJPHJH)tGpE>Z5)Q!CB+nQ(GI0EI%l znC_uNhv4nF3me++>dcBn_EWP3uinXdBA6K$$AI1?5vQl1oPHNhp!<-WoxQnGuSW@^ z`lFYnDpgEe9P?a40)~Y zGqh4SoP6)zy_XCf?d=+9GBIluwz=}qB&ir-w`Tw~CuVk!v zLg+SK;f#LBXbQZFAVngB)Ta%n^fAWVisSHw^yOYsIJ{!&t-G>jb-O|iK0Kk96orGn zprokZmr_`~o}d5goAWAOxbp8ik!N-dWuQBdw;-n z{ZB5~)KM1rfuL-_1xCW}5GC{Zkut(kAT~2Bj9S-YYqFex%Lok*H<7*tXL)v69$mX} zOCKCDSh{#X41$L{nm7 z>QfFLAlY)O=GO>!>oQeKNlC%Y&9%KT&g0_DYCU!Q@B}g2Y8CA-=00a%va{!Q^~qG= zmiwK=pz?|ee+*l2@91dO`nu!V>gphb2ZZ9EKaa~Eazki^KM{;MwO!dePLb+bqZInr zs=p#2FaRsOA)r`L-PAYMr+Co$z;n+^y@jBT@6@;s!HMH@V^ul})!W(D)mx8ur?7je zALS#17uUj!Q}W7t-`uGE$RlS-(_KcN*Ab<;*?cmqDd6y`CLdQD5B+v-*5GzjwIZ`? zzg+r*uoP4Ns8c43*NtQAjlSeV;8I%U@pVy0H-sr;G&NK0Iz%=39)IiHnu_NMY)Yzo zO=-~{bL&p^*qmcVW+qpgI;$r0>8@(`ZGMY>u8DA#{?^d;$tPr2({6p@RNVOVcwOE~ zXfT=7zxt^ST~<~WZhD&6BUWa&>P|yL1IN{?MG(2djR4uG!f{0ddUB4))YN(fh0F@Lt#(OIuDKLB;58sD%^KdG zO;1k`9-d=ZEQk1=8!k}_VN+6cJQ{J4L-v^Z0_CT%@if^$*WVbp!dGQ9Y!cWX!T{|_`w6P@S3@AWPRxRBr7M^H#{7Ds?|VCLpz*_ob)hBqB9qen+a&24Nnm6QTul|~zDnOj?jwNB>4`ab%) zjwY`|Zg=iU`Np=|3?8ovYt+`x4q5H9roO7)#eT)ipHIAs7V4o+VLpJxFM2Y4KVx(b7nlRfv8-YDvhi54#lG z`t+Ac!=C#uVv}}j^Mdhp;eQ2rdM4LO&Sb}XhtnZPTO|Pr_W{u|?7bJ^<44v}=c(AC zatDhqNj@ztvXfI&ctHD}Uk$!}ot;6vy%nR@p`oEEA9@RY?p_`$v57}QY$w0or3_Nc z)@xD~UM@%u`GHLx6)(ej>FEd7yY?}gVq(CjcU)NQ;ls@CZWV+`7&Zko|+!96g-{l6Hly<;GfELd}rd}q)%#rZ~ z^=-N%>i~pbF7?(uI2(5vt>Uhu%()@(X>Wp8DHp<$>cl^Zh9)sC&KXkA^HaEY_c=tY z_mNaa7$EcgUPVj)w(vfJUUdI&&iec0|M>%CaiD2VBTrw4aqFfx_@2X;O~n-? z9XCoiru2?Et@^5|9YPUiLJXT;ivhF)XN0Am5fv3xXwYyB*hKT3?eysGZ;^bV@$oqmO@UO%ka=FqW$ES7s}$T@h!u9iy&&iA^rPWBT`dN2JGGLM zRQTRUDhVJ>-Nwd7ip$qntC*`vCVqx;RVMh$&&5wTa&Jl8{QONoKNUzkL&L*;Jw1j_ z>=Tc*!lI(1CvD4SAs>AC`9x}VMFlHtIp7g*>F(}sZf17h+BzFSzF4cL`CWezYv3Zz zRb!XpffzMaZqHY^Z2o+`_KmN@Z&SN9zgzjfc>}F}PdH2QaYwEuQ*pb(wFW&yg3!BJ zU7lvR9b~rr{BJd3;;Q{vpzSRs#WlqriLp|FT0StDO#G@tiEG)lT^jfcX?$+S{-9yk%80(7&aB`g7VX+ zd5}RN&5ZNzYHn+EAqTgV(n^-(oYs(wsnd#}y)tMf3Wo3OB5Cb`# zn9G0D^}1NA+{w<2sw&-*agPcHm+4FHgr4}~B#0-}MZ>{v)f&i)?(p{IK!$z=X6aRF zmyFo@UYLZxqoiPXa&$@*A$$ual8`C%Gu%T6xL;k%?uq4D!303Vc@>oYxBoXw_Un~U zh7#tqxU?BSv4fKTBlwA+u)HY{0U)!rrT;|ezsKwTr^!KGR1k;5QErcn)zY4a5ZDst zkQ$z*w^pTFt(x`iz4Gc9_!bxWCmjYr<+UPfL}l;im}Kp;D=;>6t$bpMqa02OdrNklgv#r~kxI zJkN)b)mnPF2);6DBFTc}ra%@w%Y6uj<_diI0=`7Ye`A6%_w0gN;J$%C#D@~#Qu?c< zTW9Dey}hTJ3no0x>D*Tbfq(g(7F~N1Gj%R56=%EP*#zmXDtA#Nngi1KlbbHMpMo&* z9HQ2J-{Q=i*pUf_GC?Lo2&-S8MjHMx4W9e@Jz2?Jd`?X^a0gCW|McQw$FBpK56e_u zQ!)Y^2Q!QAZBA(C>fh#QA^CAKn&PHEPTNO!i+c`z2Qr+|+vL{-QP*{(a)vB{6txkH z4p*;UMIet=DK2LAG)hp7t25rMt-d+OIipF%-9njYLg=**Jq#}x%z(>CPgjHTHYHCMK!b z74t$_x)nEn;z^%-x``E}3i$ndR$B#&@JDUrC0V7*Lkk!N~~;6R6;! z+Wl(roMBhf4y#Q0x~|S+{O6MgkWzl)h2z2qDlGIACiP-GYbAAQLkm4wIXO!3f-P-r z7Z#so>hj*>82)9Sbk&iyHNSUcgr1SZDqY^fw`wf0!*TdDj{LtRA~l9u8nm49{{JE( zXItOl7FgH!*mz#f;P1-LxzOY^1vug)cr)YQEA{MrseEMNCwgueBR|#aa9txZm&<3! zpr=1HWMfP9p3ba%m@zx8KL=;)4oxSYaqRJ?;_o&boSaBH?okw006wR2=gyrk$_tJa zK#Ict91Gz*?PEG#?H+Z*B5bMkG=5SRXItrfH&|tFv=m_^em1KA_tnSkWlFMS{Ra$z z(#DMAHw52tz6hh>C-nYk=_z`Z^Z%5oI*GMSbAe`93zcr_VK1O?|DNMu(5TS(L`^%p zg5hI%iP{_d1^Jsf9x$(8UzF1yLXNcxTSyTeXD#=F8NkmM`q0DK}l?NpubQ1T@ z5%|aP(z26v)~EW9SzhrqW?$$ z@NkF{@b`iB{ouU_zk&G@P!u%gZO%X8<3%NDHuqMd=wB#P>RjvbslpAv@Hzz%j{1ii zMbyR&$}!Tl=jMc70-4ZHBVhCpD)P_orJ1}SkPm}6yj(8&+`M3^0xw)^2uue_9IE#s zj=&w1@1ZXZ)1c(_kJpd{C$}@cAtM|M)H%zz(K&FF5TSdRy}%ax61PjzS8}KbQ8@rm zEG3mhgN)8Hl;}Np7MHB#GLPSlJzjTPwm#eU(*~K8ga5&*EmTk>U`Y_kU+MS+?VdwE zp;&(>fvWlb&!-tpP#utNxi(br?}7lCL}(Q{SdTU(G5O-`5>y+^c3#-nGEg? z*>0~mYhK5XFMN-Xpkz)+iqa?jD`ETH{X`bbe3N_h{Y$FGu4tZ{B55!9E*8I?TbWfw z&DPHfNW8s9N)hvTa_<$1xMq0xHU48z87hgQ`AZW@+H9lWwxEF5s^depQI>M^(Vx&u5m8(22KUr#mn0@}KPM|L7Q%xYoC>m&|au7tDlRawJ+sk^s6944f-on8bw5X=X)J{ zfh?8v=0WK?xA+bokwAS)RPNbC2AP-ke@yzouy1G+WEIb`?TGY=R~aK0JN`^(K#{cj0@1w35h@7qGaq z$Cst)WV{EtL?U^M#i;cZ#FWu2)#p&2R}-t#(`A*!wvws2ADT8bkrvAgocLsL{gIo! z<>pCIsA!INyt#&Oak2RCRsR33l3B>No}zfog7&DvR6kMPlTg|z)8~!oUuc|KVc##J zh1ug>m-Az}95py-T5t8b{)1=nZmAgIQAEHhoShsG*z}m$oR{9Fv0F)Y_I4IJdAzBE z|C`g4MSzRaR#oF=$F_SP9B#ci`~J~|bWr{KRXdf<6&#D;6?f0W)6R?33(ip?I8 zvJ+;$v8<6+d84n)eHD!VGZj}ErA}47crs`Je_{C+lVnwI&7(-}epGZdSDx|tR@<0O zVK$;ic}+BxI1f#{@JK$oyYh=0nxu+)whQCGvZ6=6UO&E4Z!EYa9HSYLEzJBbf<{xDwi(3YmBNf^#y!v~($cR#D#d5Nc<~~6 zKD_w>W2%9ORbCHs!PVknWvje21FNTw>nE-9ZX+|%1NFb(ku}ew26t}8+zfLsb%_u? zg!25RzqfU!_9T^AfEy%yz2f!WoCkT5oEn+A1qTIXERRV6zQoW%1X8I4AIp|=_8cLt-W1-g^z+758qrMw}Q|)<3Gtw73>(tJ&cn8l$u88+D zefxGFh584%f(|+`jw`3X;DZ9P_VC|588`@%nCL?F3_7^k8S=MzKaRlj9rtpnl~`kB zJi}{ovXQg+M4w&4QIynwdI(lTp9|Y2M@euzk=FWghMAUJi5K;4S$6yB=;9x;rFcVu zQ=}+W$oTiuSAfSqGaAfmI!Ic(HFo5l{nR$246f!`)rVx|8wDmqI%iORREZ4dQywHJ z@%LT&jN+B^Zps3bHgTds1)cG7`Ef=aQa%3oFAkA=DoINFGCc_(;=?OQg9&8Fz(U(E zJGc?lvB)n+x2aIIH0M1?{Z5l`6hKZ-9+JZBm^5mVqU5Cj7NzpOKaDuhqsK<(l15Ch zn%p~#uy%bn07KSHoNW#aX>*N19PvW`vjH*&$FxzA-aiFo$_97l;g$$! zshEFyMA1PfXnm{yWu$3A!$};}ae%N(s0Qzk@Vn-Q z2$$r0-l;O73nYUOaQBK?2?|PT-JNwgs9LrC zS^7zj(!Y#cQq_8}z9A$s(19Gq3wm#@R|q> zDgTdON$Dk{?%RAQ{T&z{UPT@*5hc#oM^>Tyn_u)^O;b8FP2v##@JhU;x~t zFP5s`SNSv~6_3!LEt=P$l}oYBB#P1co7cTMk)D$i-a6S0ifPJZYdEFLXiF%k!{*{6 z=w_h1K}pX(L{80IL7#jZt7J#ayu?On_KBN^DI(yaqZHG+Zluc-Xd0=y3G=F3QqzGr z7qY5cu9X=Fn_vQAM+wHctP#nt~q3%>H6@y#eA8rixN^Eon z$(sE?bziDl4^=!zw_%x2Ja*O;`-50BpXrrz^zh3iBAcNw;7nUv+tN?xJ2dzyKQ%m@ z8Z4%FZsLZ3ri;4Iz~nP#y9)X?0{OuM`CP5nwzn)8d5CUK)q9C`G356^R^&;`wUP$C9H*~S-L{Fd z&2)DjJxVOL{Qd!yWe0ov=|(!6UJ%*Tw6r1@MY&pK9U|G!S$Ha7!UnI`hZ*WUs=nW9 zZ;+LT4g9!glv%Om?My&#@sD;7UjMNKwIiLH+=mXi)tzSiq}op3jE`#yhJ1Y*>dJuw zq5z)Zi#Jr3ix0KtpYfLh$$C@5ATyq?dn5W-+D@iX0Jn3X@rCpn z6ou#buj*AkIC0{{(K}Vca~fJ&TKRGc88XcKOSV5#tC2?z1ZfyJ`-_9)6IJHx@q;>@ z>@DK-K`$OTi!g+iy7YB*U7HnJ`j>(hl0UD3UZz}$EklY{`O;A~muk=kpjUQh-e%(I zuMPDueVGAK}6rWH0Bv2G#KI^NZlIMb6#S>E5;9@TicpIX>B)=%2P0{;dEK&6j*_|ahT;Kjdi znvA-HAZOrrbI4kkG~)~`iTBL9 zj^B-=#Ie<3S9Lx@YO&bc+0rgFy`L{f5_B-{WhB+YGmHGeGap7C9zzA8gT;1OA+F&f zjH{b8CNYUst{aV5J;))s=b8>ssqbfCqK&`X;`TVd@;#kj54rXbBq+fP|CETza?dvSzm=t6CXZq{t$r^H0ErAnBAW)8%1ZJp{(Zh6C$IK&b1qJ>j)qZ2@vrahlo z5AdV(B`^1yyiq-|3hM8ptbkA5b(;pMX;;{1G2;HxYQutywhM(-f8OxwS?{*+(K{6NLL?~tS%aig+??E< zKRw6)n5zT|w>^7-_nUq;qe0>@dWla}cVO2J)>wf2z{lv*zEH=v7Zr zl!~u%;Is2@y_rx?gi;y5kw}!o)7Sc_68!I3exj%NKe z)2HVA=%+J!n?ZBV=EoO(lnkE>?$S&M=?lQ`Rk((NRLMA|Nc{^IG{RIh1#YZ`?6zWx)MpTmg?}}#I=e#K9 zo2P_Tj;y^Jw$Ejsi!aMrn|S(-idW|IEy3u6sPoNtTR?EBg926!8nxvk(eD{51{>p> zrAtRFTP^bn&>#*(#2!I;20l&9{Hz$W^^#oY+Wkrpc)1%7qJA)YxxDPVx!!S3^qt~W z7S$^8gsUO>&>~6tHMaLi9@K=!D|A2p){94tlNDmcJF$;nkkX(eV0pgmYjVZ)2(Rj6 zT*I%%M{^-Mv4}O0nW3THF`~ZQCnC{b%sR#}K^}wgu|6`1xyI8L>X!*j8AumyHirU4X#)qZ_w~h z2kr$;LR6FCrb1M2$zcaxi*C;*2mr%!Cds8#5oEsR!ZzNWa)ovm8|0&}KC zcA1nf^;H#ccU+iEei$o$a$%IA1&Cb|#k#|ugq!J+&wMvF_1-BWTD>6tG~ak_npDAV zK)+QFD?a@L%8Hj)s@RJ_N=7l=!kME6VlY@q+J*33RU2cjQNm8y$W=ekHPy$++K4*6 zJ9NFt^zj3rxNP;kvAi9%_=?7J^{%e2YesT14GklOTv`Tou1$O^8ocMb-eYP~TT}#0 z@=p8!N+i4ST0>J4pWn*e111JebopHI@TxbD+wE|S$5z6C^?0=|cuCat^z_PG<=+y; zV=}NatxAOLv3SOAlYAyL7K1LI-<+m6A9s}(`?h*)Oz+<1QwmZ1X1~t~KfH2A1)6-e zw%=;@M9&{LZ+XuSW;4XK297wJ^#=r^n#e3(elQ$*+sweXYN(p~$JgW)rP0BS>!TOM zJsyUzDyV2^Ec0;ES#3w6t-sE6;VQ?TRy9S7hB3T3dH$VS2lYXKZs5Ejrl4en;6xXXw6T(zuV~^d$@P)9EjIYpj`PUd&EMi~~P3 z>rO_)+YsVqG-QR?RhxaR?s9xTjgym=?fw3phFDfmpbBoPqZndjhntkK@t?M?o!#9N zDjm~27j3)E9ErRgAe_&{I*oNToWsNvyQoZt2)D!;L!;0UbU&~IQdaOp{%R8#@AW>X z_vt@cj2Fb*9@zc)M&lTN{dvb!h_P0Zxi3+^TR=sxYp@D>ra~*<_*DBX!m}M>&8(g} zOTX^#rWS5#wptn=e%SFK7Fccjl$_fM->a(H?V!n5I4x*^)2S;-dhIhA1W>j?N{fWc*T>E#Q zE9~L78sewRfaaFON08u5pv}5VSaWo5AeS>SYiDoY_ZJ?2E!=`w z8o*WuG5!A^U@*%!_{ffy78O0c zIIdRO)u@q?hX;^m1^9^;(!kouaI&Audjr0O3!#HsCuwi~Alo+xB0koCmp)rJCxepr z8+G%4Sy+p}e3+Wr1O}M4e6AY;0(96Pj1;f=4C2EH!$0M}FKiPCz1`ggJ3&fA1|}b_ zI>sR#IK=x`mYG=Ib-@U_laAr=EGWS*YXcbZ-xC3ph@TYos_A_g91u&!e}4cGEzCF# zx_;0pgzO&DE8g)0gt)d)e?{K&y_DF^JEN`(RprFJ?Y`b#BPTSI(6hw^#4wX07j8b; z66&%&SHiN^;adso8oU!`&A}`~Wp=5|W-EW%hAN#4Vp#%B!6hTKJzKEWB))e)cy)c) z#sYo_wwi8e&z(7PkQp6M+?iU#W#kNLnM+nxRSlIsO%el_^UP8cW1u*g2_3I$<=@rR zb55z=s9I=X6o-q=U9r21-K=A@-(4H90FzlcEte8ldv--Ze!9+Y7g+6 zD!!YRT}?2*Ao7fN?-_U!_Lg~wDso|5U68sn02pirn__dvcYac0JDCDSK4fHNeYgKu*nG`FzdJn1~Ho~2>2B5n-Y1Mr*JInV1jgZc0TsH?#@_Zf3xHKNZRtUt?1^Dm-|Dy+M_hA+x;dzBLlgY z*+90gcEMx+{wy7~H1YF{f5|l6JxbzPYxn{D=F-GXP2qfS(1|X8Ws16PEW49uPY9yoz?Aj5^FBaPg|?HXc+FHiM@V0*-Z2@`Qs zlRqE2LweYC6D%t_O4QYH=hG7OrOTIFI6*XF0jo|yyAbDC$Z1toRlWp)Z$}y27H^L_ z4qGjWzW_fxT)D5(d4(B`G^*)rDzE);0(&wNaiYYX6|)?tE1&h1JH&~V<`x#}4<4My z?!YYd1AJ#9TLdO2XO>%e25dZ5qDgADx5tTkKmchAvuCe6;M5=~qj8vXpm+pJmAH$7 z#GqYduD-J|w+JIPW3%_i=I~Bl5W)3rIG5ICG?G(vY^rX|PO#-0)+{{}aasRv;!t-& z;P^8)7$w4bNljK|r3i9Ma7U?KJo#kRLlyiP$gGi{!H&Nt{;r;aqEa)G8OA&K2&HHoXpoDVpsdQ5lh^_W`M#3kG0gK_|5Q~-m^q7C)V9w z*#1J!W7-sOfj!$J0hml^SXiT-wWDJ(SL@qAwdU4Vy1wo6XK^We`dEXGdtfz%QHKtS z*soK5x-UNKS3YfTX#p&<-I+^*RmB4?OP%0Vc=39=Nf0f|M&~#2e`%Jq#I1_)-E0V_ zqDOVFR;fk&fW>NASy`3;iL&?=SX5k0Q9%y?QNGr%Q)J%9k)>V4k+sI!-PxItrka}m z@uSp!a^9V@;e_oIWr~Wk@8J0>%z0 zB1xHPBO3O(An66EOu#@p5g1)Mz5{ctuBsZcx6?sW^x7^p%s?h_bwV1+N8=sm+7-a3C;$`~LW!P$9; zdaoRxtE~{OpO8H_eVQ1DZ-gUd;L&6 z3VH05nwy(L`@mgxe6vwtRf)K&w8W526|En>^xy#m25wpoCVctwedg-jh1O-0)LGDZgb*ThhgT=Jx3`nD zc6D`~!ce5uq%MDb#{_^{kTiPKFF1T%}oOj?G)UipDtH-_x2_W zFI1}Plvsxj;6szWhP!08-;NFsC}YP{>RYFO&shu?TOp+%j0iJY$Za#1fagE}ZFapewX(qTxd1`j@9cKp;g++Zo#@%&8)~n@EhF-9Hw{>eOhKe|kU5Oqy1)M+@X72C+?6}DRj@3Hnh|gVdTYa(x2MIk(qkPpe7WzVTA zZClpWwZgh)*HLt}^A2G(Rr~^0q|1@NyV_vtUkJCnz6NiN2c8BR(dgNQ#aM9Q!FYdM za9?mG%sbU&_;Q!aK!2bLF}FcipeZL7a)p!qpGcF&aXd0&2qz!kTQ9j2P^44@+ddFc zr1raaOboqN*jaIXp*%y^D~-o%5cdZ2>HB(mV+n^i*x8XvoXihx8;rCHV4V@#ovM=n z>!o9(6o2cC6T)^pusdHhgAvcL?$d^|5P!ZK)LEDsF)=X(Tn%cOOnmeCd^w(bcR9R< z-q!Sus^v)OedIY1(b2B`mPu#YIKrARd&}VfFiDKW&8C~`TRw0PgzOUVHG3A`l9zPt zyY3O-5}EM*sQbpKc9lyJvl$d9OG};Ckl9)TU1~ZlHSTSoZV|GJ0NIO_8s{Pll1(?X zI>13aCl?Gv3Bg%)HMQfLFBcACwuc7nH%6VnPAco|?Y%TJcUqKTqHv$M4&@oXrlqA_ z1(GO>BfczgUU@vR9(@;|ODkRS*hu~pQ=zEHHvnBOk{4aR8yomq+ literal 0 HcmV?d00001 -- 2.45.2