~sbinet/talks

0adacd63af6c88e712ab2ecfe1e8582359329ade — Sebastien Binet 1 year, 10 months ago 9cd2d34
2022-09-22-calorado: first import

Signed-off-by: Sebastien Binet <binet@cern.ch>
A 2022/2022-09-22-calorado/README.md => 2022/2022-09-22-calorado/README.md +3 -0
@@ 0,0 1,3 @@
# 2022-09-22-calorado

`go-present` link: [slides](https://talks.sbinet.org/2022/2022-09-22-calorado/talk.slide)

A 2022/2022-09-22-calorado/_figs/collab-calorado.png => 2022/2022-09-22-calorado/_figs/collab-calorado.png +0 -0
A 2022/2022-09-22-calorado/_figs/detector.png => 2022/2022-09-22-calorado/_figs/detector.png +0 -0
A 2022/2022-09-22-calorado/_figs/grafana-logo.png => 2022/2022-09-22-calorado/_figs/grafana-logo.png +0 -0
A 2022/2022-09-22-calorado/_figs/plot.png => 2022/2022-09-22-calorado/_figs/plot.png +0 -0
A 2022/2022-09-22-calorado/_figs/screen-crop.png => 2022/2022-09-22-calorado/_figs/screen-crop.png +0 -0
A 2022/2022-09-22-calorado/talk.slide => 2022/2022-09-22-calorado/talk.slide +252 -0
@@ 0,0 1,252 @@
# CaloRado
LPC Dev
22 Sep 2022

Sebastien Binet
CNRS/IN2P3/LPC-Clermont
https://github.com/sbinet
@0xbins
sebastien.binet@clermont.in2p3.fr

## CaloRado

Since ~June/July, started working on `CaloRado`:

  _"A detector to measure the radioactivity of volcanic gases"_

LPC is working on the development of a gamma calorimeter for continuous radon monitoring in volcanic plumes.

.image _figs/collab-calorado.png 350 _

## Apparatus

.image _figs/detector.png 450 _

## DAQ

Alex tricked me into having a look at the "DAQ".

- detector is sending UDP packets as fast as it can
- a dedicated LPC machine is listening for those and storing them
- initial `python` script by Magali was listening for packets, decoding them and storing data as `.csv` files
- initial `python` script by Magali was missing packets

So:
- could `Go` do better ?

  Well _of course!_
  Let me try.

.link https://gitlab.in2p3.fr/sbinet/calorado-daq

## 

The detector is sending UDP binary packets:


```
-- header: 14 bytes (multi-bytes encoded as big-endian)
 0 -  4: uint32: type of frames
 4 -  6: uint16: rate
 6 -  8: uint16: temperature
 8 - 14:         [RESERVED]
-- n-frames of 16 bytes
 0 -  4: uint32: boundary (==0xaaaaaaaa)
 4 -  8: uint32: trigger number
 8 - 10: uint16: amplitude
10 - 16: uint48: timestamp (relative)
-- footer: 2 bytes
 0 -  2: uint16: boundary (==0xaaaa)
```

_ie:_ a 14-bytes header, followed by `n` frames of 16-bytes data, and a final 2-bytes footer.

## How it started

```go

conn, err := net.ListenUDP("udp", &net.UDPAddr{
	Port: port,
	IP:   net.ParseIP(ip),
})
if err != nil {
	log.Fatalf("could not create UDP listener: %+v", err)
}
defer conn.Close()

buf := make([]byte, 1024)
for {
	n, err := conn.Read(buf)
	if err != nil {
		log.Printf("could not read data from UDP socket: %+v", err)
		continue
	}

	err = process(buf[:n])
	if err != nil {
		log.Printf("could not process trigger data: %+v", err)
		continue
	}
}
```

## How it started

Pretty simple:

```shell

$> go build -o calo-daq ./main.go
$> calo-daq
```

`calo-daq` is:

- (still) listening on UDP
- stores data as binary
- every hour, commits data to a binary file
- every minute, commits summary/monitoring data to a `JSON` file (timestamp, rate, temperature)

## And now...

## 

A slew of commands to handle the DAQ+Reco workflow:

```

$> go build ./cmd/calo-daq      # DAQ
$> go build ./cmd/calo-dump     # displays CALO files
$> go build ./cmd/calo-janitor  # concatenates CALO files (1/day) + gzip
$> go build ./cmd/calo-mon      # monitors CALO files for missing frames
$> go build ./cmd/calo-srv      # schedules calo-mon jobs, creates histograms (soon)
$> go build ./cmd/calo-cnv      # converts CALO files to ROOT
```

## calo-dump

```shell

$> calo-dump ./2022092100.raw.gz
Event #0 - 2022-09-21 00:00:00.0791 UTC - rate= 175.77 temp=14.22 kind=1 frames=30
  217614721    179  12602452132343
  217614722    611  12602452155334
  217614723   3692  12602452193631
  217614724    299  12602452222285
  217614725    518  12602452284647
  217614726    112  12602452339360
  217614727    247  12602452393412
  217614728    531  12602452411598
  217614729   1784  12602452426692
  217614730    445  12602452477254
  217614731   2520  12602452517602
[...]
Event #1 - 2022-09-21 00:00:00.2557 UTC - rate= 175.77 temp=14.22 kind=1 frames=30
  217614751    480  12602453509912
  217614752   2012  12602453513271
  217614753    136  12602453535683
  217614754    444  12602453558193
[...]
```

## calo-mon

`calo-mon` reads in a collection of `CALO` files and analyzes them to find missing frames:

```

  $> calo-mon ./data/20220907*.raw.gz
  ./data/2022090700.raw: ok (triggers=640530)
  ./data/2022090701.raw: lost triggers=  30 frames=   1
  ./data/2022090702.raw: ok (triggers=649170)
  ./data/2022090703.raw: ok (triggers=653100)
  ./data/2022090704.raw: ok (triggers=657930)
  ./data/2022090705.raw: ok (triggers=662490)
  ./data/2022090706.raw: lost triggers=  30 frames=   1
  ./data/2022090707.raw: lost triggers=  60 frames=   2
  ./data/2022090708.raw: ok (triggers=649950)
  ./data/2022090709.raw: ok (triggers=645240)
  ./data/2022090710.raw: ok (triggers=636510)
  ./data/2022090711.raw: ok (triggers=630060)
  ./data/2022090712.raw: lost triggers=  30 frames=   1
  ./data/2022090713.raw: ok (triggers=650280)
  [...]
```

Analysis of the origin(s) of lost triggers is still pending.

(most probably _"just"_ network effects (UDP))

## calo-cnv

`calo-cnv` reads in a collection of `CALO` files and can produce a single output ROOT file containing a `"calorado"` `TTree`:

```shell

  $> calo-cnv -o out.root ./20220907*.raw.gz
  calo-cnv: processing "./2022090700.raw.gz"...
  calo-cnv: processing "./2022090701.raw.gz"...
  [...]
  calo-cnv: processing "./2022090722.raw.gz"...
  calo-cnv: processing "./2022090723.raw.gz"...
  
  $> ll out.root
  -rw-r--r-- 1 binet binet 133M Sep 21 17:58 out.root

  $> root-ls -t ./out.root 
  === [./out.root] ===
  version: 62600
    TTree       calorado              (entries=15473430)
      trig      "trig/i"      TBranch
      ampl      "ampl/s"      TBranch
      timestamp "timestamp/l" TBranch
      temp      "temp/F"      TBranch
```

## Grafana

Alex is collecting the `JSON` sampling data files and the `CSV` output of `calo-mon` into [Grafana](https://grafana.com/)

   Grafana allows you to query, visualize, alert on and understand
   your metrics no matter where they are stored. Create, explore, and
   share beautiful dashboards with your team and foster a data driven
   culture.

.link https://grafana.com/oss/

We have a local instance that presents this data.

.image _figs/grafana-logo.png 150 _

##

.image _figs/screen-crop.png 510 _

## Physics

With the produced "reco" ROOT file (`calo-cnv` processes the `CALO` file and reconstruct the absolute timestamp for each frame, from the relative one), one can start actual physics:

- energy calibration of the detector

This will need to:

- build the distribution of the amplitude (in bins of temperature)
- fit the "obvious" "known" peaks at "known" energies
- extract a `F(ampl, temp) -> E`

```
  $> ll ./data/root/202208.root
  -rw-r--r-- 1 binet binet 5.5G Sep 19 14:28 ./data/root/202208.root
```

##

.image _figs/plot.png 600 _

## Next steps

- Luca will fit the various peaks and extract a calibration function `F(temp)`
- `calo-srv` will apply the calibration "on the fly" on the reconstructed data
- `calo-srv` will serve energy spectrum plots as data quality checks