15 files changed, 588 insertions(+), 0 deletions(-)

A 2022/2022-10-07-tio-go/_figs/daqmon-l1evts.png
A 2022/2022-10-07-tio-go/_figs/daqmon-l1rate.png
A 2022/2022-10-07-tio-go/_figs/daqmon-laser-lost-emits.png
A 2022/2022-10-07-tio-go/_figs/daqmon-laser-main-hover.png
A 2022/2022-10-07-tio-go/_figs/daqmon-laser-main.png
A 2022/2022-10-07-tio-go/_figs/daqmon-laser-tt-recv.png
A 2022/2022-10-07-tio-go/_figs/daqmon-main-hover.png
A 2022/2022-10-07-tio-go/_figs/daqmon-main.png
A 2022/2022-10-07-tio-go/_figs/daqmon-tile-ctpin.png
A 2022/2022-10-07-tio-go/_figs/daqmon-tile-dqflag.png
A 2022/2022-10-07-tio-go/_figs/daqmon-tile-eba.png
A 2022/2022-10-07-tio-go/_figs/lbc59-pmt.png
A 2022/2022-10-07-tio-go/_figs/ps-dt2018.png
A 2022/2022-10-07-tio-go/talk.slide
A 2022/2022-10-07-tio-go/talk.tex

@@ 0,0 1,174 @@
+# New plugins for Tile-In-One
+TileCal Week
+7 Oct 2022
+
+Sebastien Binet
+CNRS/IN2P3/LPC-Clermont
+https://github.com/sbinet
+@0xbins
+sebastien.binet@clermont.in2p3.fr
+
+## New (Go) plugins for Tile-In-One
+
+Working (10-20% FTE) on TileCal monitoring duties for LPC-Clermont, 2 prongs:
+
+- [git:tio-hvmon](https://gitlab.cern.ch/tile-in-one/tio-0021), _a.k.a_ `tio-0021`
+- [git:tio-daqmon](https://gitlab.cern.ch/tile-in-one/tio-0022), _a.k.a_ `tio-0022`
+
+## tio-hvmon
+
+## tio-hvmon
+
+`tio-hvmon` is supposed to (eventually) display monitoring informations for the HVs.
+
+` `
+
+`tio-hvmon` was the initial work item.
+
+- integrate D. Calvet's code from [HVAnaMon](https://gitlab.cern.ch/atlas-clermont/tile/HV/HVAnaMon) into `TiO`
+- `HVAnaMon` was initially written for _"offline"_ analysis of HVs
+- improved `HVAnaMon` data storage (about `3x` **smaller**)
+- improved `HVAnaMon` data analysis time (about `2x` **faster**)
+- `HVAnaMon` needs a bit of re-engineering for a _"quasi-online"_ workload
+
+## HVAnaMon
+
+[`HVAnaMon`](https://gitlab.cern.ch/atlas-clermont/tile/HV/HVAnaMon) is a set of ([Go-based](https://golang.org)) tools to:
+
+- ingest [DDV](https://atlas-ddv.cern.ch/DDV.html) (_DCS Data Viewer_) data
+- analyse data over multiple periods and/or modules (`all`, `power` or `plotonly`)
+- monitor stability of HV channels
+- predict/infer future possible issues
+- plan for repairs
+- devise strategies in advance when boards come back from CERN to Clermont
+
+.image _figs/lbc59-pmt.png
+
+## tio-hvmon
+
+- `HVAnaMon` worked by pulling data off `atlas-ddv.cern.ch`
+- data retrieval speed not sufficient for a daily monitoring (`~12h` for a single partition)
+- current strategy is to directly retrieve data from `ATONR_ADG`, in a similar manner to what [TileDCSDataGrabber.py](https://gitlab.cern.ch/atlas/athena/-/blob/master/TileCalorimeter/TileCoolDcs/python/TileDCSDataGrabber.py) does (thanks Sasha!)
+- initial tests with `tio-hvmon-db-pull` reduced data retrieval to less than an `1h`
+  (for all partitions)
+
+` `
+
+Eventually, this will all land under:
+
+- [https://tio.cern.ch/hvmon]()
+
+## tio-hvmon - TODO
+
+Still need to:
+
+- gain experience with automatic monitoring
+	- run every day? every hour? twice a week?
+- assess disk storage needs
+- wire up `HVAnaMon` backend with `tio-hvmon`
+- provide a web output (in addition to PDF+PNG)
+
+.image _figs/ps-dt2018.png 400 _
+
+## tio-daqmon
+
+## tio-daqmon
+
+Migrated the (`python` based) [TileDAQmon](https://gitlab.cern.ch/atlas-tile-online/TileDAQmon) program from D. Calvet to [TiO](https://tio.cern.ch).
+
+` `
+
+`TileDAQmon` monitors the status of the Tile partitions and the LASER, scrapping data off [atlasop.cern.ch](https://atlasop.cern.ch), and presenting the aggregated data as a static HTML page (auto-refreshed every `120s`).
+
+- all presented data generated _via_ `acron`
+- authentication done with [authzsvc/auth-get-sso-cookie](https://gitlab.cern.ch/authzsvc/tools/auth-get-sso-cookie)
+
+## tio-daqmon
+
+In order to integrate `TileDAQmon` with `TiO`:
+
+- developed a ([Go](https://golang.org)) library to automatically renew Kerberos credentials (via `keytab`)
+	- inspired from CERN IT's [authzsvc/auth-get-sso-cookie](https://gitlab.cern.ch/authzsvc/tools/auth-get-sso-cookie)
+- needed to query [https://atlasop.cern.ch/info/current/ATLAS/is](https://atlasop.cern.ch/info/current/ATLAS/is)
+
+.image _figs/daqmon-main.png 350 _
+
+.link https://tio.cern.ch/daqmon/
+
+## tio-daqmon
+
+`tio-daqmon` is a complete ([Go](https://golang.org)) `HTTP` server, with multiple end-points:
+
+- `/` displays a `TiO`-compatible (static) web page
+- `/tile` displays TileDAQ-related informations
+- `/laser` displays LASER-related informations
+- `/plot/<xyz>` display PNG plots
+
+All the work is performed _"server side"_:
+
+- very lightweight HTML page
+- next to no Javascript
+- data is collected from `atlasop` every 2mn
+- PNGs are re-generated every 2mn
+- PNGs are served from a local cache
+
+## tio-daqmon - Tile
+
+.image _figs/daqmon-main.png 600 _
+
+## tio-daqmon - Tile
+
+.image _figs/daqmon-main-hover.png 600 _
+
+## tio-daqmon - Tile
+
+.image _figs/daqmon-l1evts.png 600 _
+
+## tio-daqmon - Tile
+
+.image _figs/daqmon-l1rate.png 600 _
+
+## tio-daqmon - Tile
+
+.image _figs/daqmon-tile-dqflag.png 600 _
+
+## tio-daqmon - Tile
+
+.image _figs/daqmon-tile-eba.png 600 _
+
+## tio-daqmon - Tile
+
+.image _figs/daqmon-tile-ctpin.png 600 _
+
+## tio-daqmon - LASER
+
+.image _figs/daqmon-laser-main.png 600 _
+
+## tio-daqmon - LASER
+
+.image _figs/daqmon-laser-main-hover.png 600 _
+
+## tio-daqmon - LASER
+
+.image _figs/daqmon-laser-lost-emits.png 600 _
+
+## tio-daqmon - LASER
+
+.image _figs/daqmon-laser-tt-recv.png 600 _
+
+## Conclusions
+
+[tio/daqmon](https://tio.cern.ch/daqmon) is live and ready.
+
+` `
+
+- ready for feedback and/or (constructive) criticism
+- **integrate `tio-daqmon` into the usual DAQ on-call shifters tour?**
+
+---
+
+[tio/hvmon](https://tio-dev.cern.ch/hvmon) is still a _WIP_.
+
+- eventually, should migrate from `tio-dev.cern.ch` to `tio.cern.ch`
+- eventually, should be also integrated into the usual DQ shifters tour
+

@@ 0,0 1,414 @@
+\documentclass[9pt]{beamer}
+
+\usepackage[utf8]{inputenc}
+\usepackage{colortbl}
+\usepackage[english]{babel}
+
+\newcommand{\colhref}[3][blue]{\href{#2}{\color{#1}{#3}}}%
+
+
+\newcommand{\myblue} [1] {{\color{blue}#1}}
+\newcommand{\newauthor}[4]{
+  \parbox{0.26\textwidth}{
+    \texorpdfstring
+      {
+        \centering
+        #1 \\
+        \colhref{#2}{\texttt{#3}} \\
+        #4 \\
+      }
+      {#1}
+  }
+}
+
+
+
+% beamer template
+\beamertemplatetransparentcovereddynamic
+\usetheme{Madrid}
+
+\hypersetup{%
+  pdftitle={New plugins for Tile-In-One},%
+  pdfauthor={Sebastien Binet},%
+%
+}
+
+\title[New plugins for Tile-In-One]{New plugins for Tile-In-One}
+\author[Sebastien Binet]{
+ \parbox{0.26\textwidth}{
+	\texorpdfstring
+	  {
+		\centering
+ 		Sebastien Binet \\
+ 		CNRS/IN2P3/LPC-Clermont \\
+ 		\colhref{https://github.com/sbinet}{\texttt{https://github.com/sbinet}} \\
+ 		\colhref{http://twitter.com/0xbins}{\texttt{@0xbins}} \\
+ 		\colhref{mailto:sebastien.binet@clermont.in2p3.fr}{\texttt{sebastien.binet@clermont.in2p3.fr}} \\
+ 	  }
+	{Sebastien Binet}
+}
+ }
+
+\subtitle{TileCal Week}
+\date{2022-10-07}
+
+\begin{document}
+
+\frame{\titlepage
+}
+
+\part<presentation>{Main Talk}
+
+\section[slides]{slides}
+
+
+\begin{frame}[fragile]
+\frametitle{New (Go) plugins for Tile-In-One}
+
+Working (10-20\% FTE) on TileCal monitoring duties for LPC-Clermont, 2 prongs:
+
+
+\begin{itemize}
+\item \colhref{https://gitlab.cern.ch/tile-in-one/tio-0021}{\texttt{git:tio-hvmon}}, \emph{a.k.a} \texttt{tio-0021}
+\item \colhref{https://gitlab.cern.ch/tile-in-one/tio-0022}{\texttt{git:tio-daqmon}}, \emph{a.k.a} \texttt{tio-0022}
+\end{itemize}
+
+
+\end{frame}
+
+\begin{frame}[fragile]
+  \begin{columns}
+    \begin{column}{0.7\textwidth}
+      \begin{block}{}
+        \begin{center}
+          tio-hvmon\ldots
+        \end{center}
+      \end{block}
+    \end{column}
+  \end{columns}
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-hvmon}
+
+\texttt{tio-hvmon} is supposed to (eventually) display monitoring informations for the HVs.
+
+\texttt{ }
+
+\texttt{tio-hvmon} was the initial work item.
+
+
+\begin{itemize}
+\item integrate D. Calvet's code from \colhref{https://gitlab.cern.ch/atlas-clermont/tile/HV/HVAnaMon}{\texttt{HVAnaMon}} into \texttt{TiO}
+\item \texttt{HVAnaMon} was initially written for \emph{"offline"} analysis of HVs
+\item improved \texttt{HVAnaMon} data storage (about \texttt{3x} \textbf{smaller})
+\item improved \texttt{HVAnaMon} data analysis time (about \texttt{2x} \textbf{faster})
+\item \texttt{HVAnaMon} needs a bit of re-engineering for a \emph{"quasi-online"} workload
+\end{itemize}
+
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{HVAnaMon}
+
+\colhref{https://gitlab.cern.ch/atlas-clermont/tile/HV/HVAnaMon}{\texttt{\texttt{HVAnaMon}}} is a set of (\colhref{https://golang.org}{\texttt{Go-based}}) tools to:
+
+
+\begin{itemize}
+\item ingest \colhref{https://atlas-ddv.cern.ch/DDV.html}{\texttt{DDV}} (\emph{DCS Data Viewer}) data
+\item analyse data over multiple periods and/or modules (\texttt{all}, \texttt{power} or \texttt{plotonly})
+\item monitor stability of HV channels
+\item predict/infer future possible issues
+\item plan for repairs
+\item devise strategies in advance when boards come back from CERN to Clermont
+\end{itemize}
+
+
+\begin{figure}[h]
+\begin{center}
+\includegraphics[width=8cm,height=3cm]{_figs/lbc59-pmt.png}
+\end{center}
+\end{figure}
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-hvmon}
+
+
+\begin{itemize}
+\item \texttt{HVAnaMon} worked by pulling data off \texttt{atlas-ddv.cern.ch}
+\item data retrieval speed not sufficient for a daily monitoring (\texttt{$\sim$12h} for a single partition)
+\item current strategy is to directly retrieve data from \texttt{ATONR\_ADG}, in a similar manner to what \colhref{https://gitlab.cern.ch/atlas/athena/-/blob/master/TileCalorimeter/TileCoolDcs/python/TileDCSDataGrabber.py}{\texttt{TileDCSDataGrabber.py}} does (thanks Sasha!)
+\item initial tests with \texttt{tio-hvmon-db-pull} reduced data retrieval to less than an \texttt{1h}
+(for all partitions)
+\end{itemize}
+\texttt{ }
+
+Eventually, this will all land under:
+
+
+\begin{itemize}
+\item \colhref{}{\texttt{https://tio.cern.ch/hvmon}}
+\end{itemize}
+
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-hvmon - TODO}
+
+Still need to:
+
+
+\begin{itemize}
+\item gain experience with automatic monitoring
+
+\begin{itemize}
+\item run every day? every hour? twice a week?
+\end{itemize}
+\item assess disk storage needs
+\item wire up \texttt{HVAnaMon} backend with \texttt{tio-hvmon}
+\item provide a web output (in addition to PDF+PNG)
+\end{itemize}
+
+
+\begin{figure}[h]
+\begin{center}
+\includegraphics[width=5cm,height=5cm]{_figs/ps-dt2018.png}
+\end{center}
+\end{figure}
+
+\end{frame}
+
+\begin{frame}[fragile]
+  \begin{columns}
+    \begin{column}{0.7\textwidth}
+      \begin{block}{}
+        \begin{center}
+          tio-daqmon\ldots
+        \end{center}
+      \end{block}
+    \end{column}
+  \end{columns}
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-daqmon}
+
+Migrated the (\texttt{python} based) \colhref{https://gitlab.cern.ch/atlas-tile-online/TileDAQmon}{\texttt{TileDAQmon}} program from D. Calvet to \colhref{https://tio.cern.ch}{\texttt{TiO}}.
+
+\texttt{ }
+
+\texttt{TileDAQmon} monitors the status of the Tile partitions and the LASER, scrapping data off \colhref{https://atlasop.cern.ch}{\texttt{atlasop.cern.ch}}, and presenting the aggregated data as a static HTML page (auto-refreshed every \texttt{120s}).
+
+
+\begin{itemize}
+\item all presented data generated \emph{via} \texttt{acron}
+\item authentication done with \colhref{https://gitlab.cern.ch/authzsvc/tools/auth-get-sso-cookie}{\texttt{authzsvc/auth-get-sso-cookie}}
+\end{itemize}
+
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-daqmon}
+
+In order to integrate \texttt{TileDAQmon} with \texttt{TiO}:
+
+
+\begin{itemize}
+\item developed a (\colhref{https://golang.org}{\texttt{Go}}) library to automatically renew Kerberos credentials (via \texttt{keytab})
+
+\begin{itemize}
+\item inspired from CERN IT's \colhref{https://gitlab.cern.ch/authzsvc/tools/auth-get-sso-cookie}{\texttt{authzsvc/auth-get-sso-cookie}}
+\end{itemize}
+\item needed to query \colhref{https://atlasop.cern.ch/info/current/ATLAS/is}{\texttt{https://atlasop.cern.ch/info/current/ATLAS/is}}
+\end{itemize}
+
+
+\begin{figure}[h]
+\begin{center}
+\includegraphics[width=5cm,height=4cm]{_figs/daqmon-main.png}
+\end{center}
+\end{figure}
+\colhref{https://tio.cern.ch/daqmon/}{\texttt{tio.cern.ch/daqmon/}}
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-daqmon}
+
+\texttt{tio-daqmon} is a complete (\colhref{https://golang.org}{\texttt{Go}}) \texttt{HTTP} server, with multiple end-points:
+
+
+\begin{itemize}
+\item \texttt{/} displays a \texttt{TiO}-compatible (static) web page
+\item \texttt{/tile} displays TileDAQ-related informations
+\item \texttt{/laser} displays LASER-related informations
+\item \texttt{/plot/<xyz>} display PNG plots
+\end{itemize}
+All the work is performed \emph{"server side"}:
+
+
+\begin{itemize}
+\item very lightweight HTML page
+\item next to no Javascript
+\item data is collected from \texttt{atlasop} every 2mn
+\item PNGs are re-generated every 2mn
+\item PNGs are served from a local cache
+\end{itemize}
+
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-daqmon - Tile}
+
+\begin{figure}[h]
+\begin{center}
+\includegraphics[width=9cm,height=8cm]{_figs/daqmon-main.png}
+\end{center}
+\end{figure}
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-daqmon - Tile}
+
+\begin{figure}[h]
+\begin{center}
+\includegraphics[width=9cm,height=8cm]{_figs/daqmon-main-hover.png}
+\end{center}
+\end{figure}
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-daqmon - Tile}
+
+\begin{figure}[h]
+\begin{center}
+\includegraphics[width=10cm,height=8cm]{_figs/daqmon-l1evts.png}
+\end{center}
+\end{figure}
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-daqmon - Tile}
+
+\begin{figure}[h]
+\begin{center}
+\includegraphics[width=9cm,height=8cm]{_figs/daqmon-l1rate.png}
+\end{center}
+\end{figure}
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-daqmon - Tile}
+
+\begin{figure}[h]
+\begin{center}
+\includegraphics[width=7cm,height=8cm]{_figs/daqmon-tile-dqflag.png}
+\end{center}
+\end{figure}
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-daqmon - Tile}
+
+\begin{figure}[h]
+\begin{center}
+\includegraphics[width=8cm,height=8cm]{_figs/daqmon-tile-eba.png}
+\end{center}
+\end{figure}
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-daqmon - Tile}
+
+\begin{figure}[h]
+\begin{center}
+\includegraphics[width=10cm,height=8cm]{_figs/daqmon-tile-ctpin.png}
+\end{center}
+\end{figure}
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-daqmon - LASER}
+
+\begin{figure}[h]
+\begin{center}
+\includegraphics[width=9cm,height=8cm]{_figs/daqmon-laser-main.png}
+\end{center}
+\end{figure}
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-daqmon - LASER}
+
+\begin{figure}[h]
+\begin{center}
+\includegraphics[width=9cm,height=8cm]{_figs/daqmon-laser-main-hover.png}
+\end{center}
+\end{figure}
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-daqmon - LASER}
+
+\begin{figure}[h]
+\begin{center}
+\includegraphics[width=8cm,height=8cm]{_figs/daqmon-laser-lost-emits.png}
+\end{center}
+\end{figure}
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{tio-daqmon - LASER}
+
+\begin{figure}[h]
+\begin{center}
+\includegraphics[width=9cm,height=8cm]{_figs/daqmon-laser-tt-recv.png}
+\end{center}
+\end{figure}
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{Conclusions}
+
+\colhref{https://tio.cern.ch/daqmon}{\texttt{tio/daqmon}} is live and ready.
+
+\texttt{ }
+
+
+\begin{itemize}
+\item ready for feedback and/or (constructive) criticism
+\item \textbf{integrate \texttt{tio-daqmon} into the usual DAQ on-call shifters tour?}
+\end{itemize}
+
+\vspace{1em}
+\hrule
+\vspace{1em}
+\colhref{https://tio-dev.cern.ch/hvmon}{\texttt{tio/hvmon}} is still a \emph{WIP}.
+
+
+\begin{itemize}
+\item eventually, should migrate from \texttt{tio-dev.cern.ch} to \texttt{tio.cern.ch}
+\item eventually, should be also integrated into the usual DQ shifters tour
+\end{itemize}
+
+
+\end{frame}
+
+\end{document}