From bd9d711825f962a4845704f3b36ccadc4b56a006 Mon Sep 17 00:00:00 2001 From: Sebastien Binet Date: Mon, 23 Aug 2021 14:24:07 +0200 Subject: [PATCH] all: migrate to markdown This CL used x/tools/cmd/present2md@d4cc65f0 to automatically convert the old "slide" format to a markdown-based one. --- 2015/20150612-lpc-si/talk.slide | 293 ++++--- 2015/20150929-gopy-lyon/gopy-lyon.slide | 559 +++++++------ 2015/20151001-docker-ecole-ci/talk.slide | 730 ++++++++--------- 2015/20151109-gopy-dotgo/gopy-dotgo.slide | 147 ++-- 2015/20151214-alice-parcxx/alice-parcxx.slide | 453 ++++++----- 2015/20151217-loops-docker/talk.slide | 758 +++++++++-------- 2016/20160125-alice-tbb/alice-tbb.slide | 442 +++++----- 2016/20160204-hep-strikes-back/hep-sw.slide | 674 ++++++++-------- .../new-languages.slide | 78 +- 2016/20160211-lpc-svc-info/lpc.slide | 168 ++-- 2016/20160512-lsst-fcs/fcs.slide | 61 +- .../alice-muon-intro-para.slide | 335 ++++---- 2016/20160531-rkt-audaces/talk.slide | 587 +++++++------- 2016/20160616-docker-webinaire/talk.slide | 561 +++++++------ 2016/20160630-go-hep-aristote/talk.slide | 627 +++++++------- 2016/20160704-lpc-svc-info-avirm/talk.slide | 37 +- 2016/20160928-ji-docker/talk.slide | 753 +++++++++-------- 2016/20160928-ji-go-polymer/talk.slide | 175 ++-- 2016/20161009-meetup-go-polymer/talk.slide | 175 ++-- 2016/20161010-dotgo-pygo/talk.slide | 141 ++-- .../talk.slide | 647 ++++++++------- .../talk.slide | 661 ++++++++------- 2016/20161130-envol-go/talk.slide | 611 +++++++------- 2016/20161202-envol-docker/talk.slide | 737 +++++++++-------- 2016/20161205-alice-fer/talk.slide | 103 ++- 2017/2017-07-07-go-conc-jdev/talk.slide | 311 ++++--- 2017/2017-08-31-go-gridka/talk.slide | 370 +++++---- 2017/2017-12-08-solid-daq/talk.slide | 253 +++--- 2017/2017-12-14-lsst-hubble-fit/talk.slide | 133 ++- 2017/20170123-go-hep-lpc/talk.slide | 314 ++++---- 2017/20170213-go-diana-hep/talk.slide | 446 +++++----- 2017/20170217-go-ccin2p3/talk.slide | 614 +++++++------- 2017/20170331-alice-fer/talk.slide | 213 +++-- 2017/20170523-hsf-lang-survey/talk.slide | 311 ++++--- 2017/20170601-go-conc-audaces/talk.slide | 311 ++++--- 2017/20170619-alice-update/talk.slide | 60 +- 2017/20170620-alice-mrrtf-update/talk.slide | 186 +++-- 2017/20170623-go-hep-annecy/talk.slide | 703 ++++++++-------- .../talk.slide | 257 +++--- 2018/2018-02-20-alice-mrrtf-update/talk.slide | 53 +- 2018/2018-03-28-matheq/talk.slide | 28 +- 2018/2018-04-03-go-hep-jlab/talk.slide | 762 +++++++++--------- 2018/2018-05-23-solid-raspi-srv/talk.slide | 163 ++-- 2018/2018-06-15-alice-ff/talk.slide | 264 +++--- 2018/2018-07-09-chep-alice-fer/talk.slide | 479 ++++++----- 2018/2018-07-11-lpc-go-hep/talk.slide | 29 +- 2018/2018-09-04-conc-mcnet/talk.slide | 575 +++++++------ 2018/2018-10-03-ji-fer/talk.slide | 481 ++++++----- 2018/2018-10-03-ji-gohep/talk.slide | 663 ++++++++------- 2018/2018-10-03-ji-gsoc-rx/talk.slide | 122 ++- 2018/2018-10-03-ji-solid-mon-rpi/talk.slide | 172 ++-- 2018/2018-10-19-lpcitt-go-hep/talk.slide | 670 ++++++++------- .../talk.slide | 494 ++++++------ 2019/2019-07-04-lpcitt-go-hep/talk.slide | 146 ++-- 2019/2019-10-22-golab-io-gohep/talk.slide | 760 +++++++++-------- .../talk.slide | 291 ++++--- 2020/2020-01-28-mim-daq/talk.slide | 256 +++--- 2020/2020-04-09-lpc-dev/talk.slide | 145 ++-- 2020/2020-06-15-lpc-dev-rootio/talk.slide | 236 +++--- 2020/2020-06-26-lpc-dev-groot/talk.slide | 194 ++--- 2020/2020-08-18-mim-daq/talk.slide | 88 +- 2020/2020-12-14-lpc-dev-path/talk.slide | 134 ++- 2020/orig-present.slide | 80 +- 2021/2021-01-15-gioui-path-p5/talk.slide | 153 ++-- 2021/2021-02-16-gioui-latex/talk.slide | 73 +- 2021/2021-03-30-gmni/talk.slide | 187 +++-- 66 files changed, 11127 insertions(+), 11566 deletions(-) diff --git a/2015/20150612-lpc-si/talk.slide b/2015/20150612-lpc-si/talk.slide index c9cc620..36a6efa 100644 --- a/2015/20150612-lpc-si/talk.slide +++ b/2015/20150612-lpc-si/talk.slide @@ -1,36 +1,35 @@ -Concurrency in HEP +# Concurrency in HEP LPC-SI, 2015/06/12 Sebastien Binet CNRS/IN2P3 -* Ma vie, mon oeuvre +## Ma vie, mon oeuvre -- 2002-2006: Thesis in ATLAS, LPC-Clermont-Fd -- 2006-2009: Post-doc in ATLAS, LBL-Berkeley -- 2009-2011: Post-doc in ATLAS, LAL-Orsay -- 2011-2015: IR, LAL-Orsay (ATLAS,LHCb,LSST) -- 2015-....: IR, LPC-Clermont-Fd (LSST,Alice) + - 2002-2006: Thesis in ATLAS, LPC-Clermont-Fd + - 2006-2009: Post-doc in ATLAS, LBL-Berkeley + - 2009-2011: Post-doc in ATLAS, LAL-Orsay + - 2011-2015: IR, LAL-Orsay (ATLAS,LHCb,LSST) + - 2015-....: IR, LPC-Clermont-Fd (LSST,Alice) -* ATLAS +## ATLAS - -* ATLAS detector (44m x 25m) +## ATLAS detector (44m x 25m) .image _figs/atlas-detector.gif 600 800 -* Main steps of data analysis/massaging +## Main steps of data analysis/massaging -#Monte-Carlo techniques +//Monte-Carlo techniques -- *Generation:* production of a single physics event (_e.g.:_ a collision and its decay products) -- *Simulation:* modelling interactions between particles and detector material -- *Reconstruction:* building physics objects (electrons, photons, ...) from the detector signals (energy deposits) -- *Analysis:* testing hypotheses against the reconstruction output + - **Generation:** production of a single physics event (_e.g.:_ a collision and its decay products) + - **Simulation:** modelling interactions between particles and detector material + - **Reconstruction:** building physics objects (electrons, photons, ...) from the detector signals (energy deposits) + - **Analysis:** testing hypotheses against the reconstruction output .image _figs/schema-general-flux-all-en.png 300 300 -* Software in HEP - some numbers +## Software in HEP - some numbers An LHC experiment (_e.g._ ATLAS, CMS) is ~3000 physicists but only a fraction of those is developing code. @@ -39,51 +38,46 @@ Reconstruction frameworks grew from ~3M SLOC to ~5M Summing over all HEP software stack for e.g. ATLAS: -- event generators: ~1.4M SLOC (C++, FORTRAN-77) -- I/O libraries ~1.7M SLOC (C++) -- simulation libraries ~1.2M SLOC (C++) -- reconstruction framework ~5M SLOC (C++) + steering/configuration (~1M SLOC python) (want to have a look at the [[http://acode-browser.usatlas.bnl.gov/lxr/source/][ATLAS code]]? [[https://github.com/cms-sw/cmssw][CMS code]]?) - - - + - event generators: ~1.4M SLOC (C++, FORTRAN-77) + - I/O libraries ~1.7M SLOC (C++) + - simulation libraries ~1.2M SLOC (C++) + - reconstruction framework ~5M SLOC (C++) + steering/configuration (~1M SLOC python) (want to have a look at the [ATLAS code](http://acode-browser.usatlas.bnl.gov/lxr/source/)? [CMS code](https://github.com/cms-sw/cmssw)?) -*GCC:* ~7M SLOC +**GCC:** ~7M SLOC -*Linux* *kernel* *3.6:* 15.9M SLOC +**Linux** **kernel** **3.6:** 15.9M SLOC - -* Software development cycle +## Software development cycle VCS (CVS, then SVN. GIT: not yet) Nightlies (Jenkins or homegrown solution) -- need a sizeable cluster of build machines (distcc, ccache, ...) -- builds the framework stack in ~8h -- produces ~2000 shared libraries -- installs them on AFS (also creates RPMs and tarballs) + - need a sizeable cluster of build machines (distcc, ccache, ...) + - builds the framework stack in ~8h + - produces ~2000 shared libraries + - installs them on AFS (also creates RPMs and tarballs) Devs can then test and develop off the nightly _via_ AFS Every 6 months or so a new production release is cut, validated (then patched) and deployed on the World Wide LHC Computing Grid (WLCG). -Release size: *~5Gb* - -- binaries, libraries (externals+framework stack) -- extra data (sqlite files, physics processes' modelisation data, ...) +Release size: **~5Gb** + - binaries, libraries (externals+framework stack) + - extra data (sqlite files, physics processes' modelisation data, ...) -* Software runtime ? +## Software runtime ? Big science, big data, big software, big numbers -- ~1min to initialize the application -- loading >500 shared libraries -- connecting to databases (detector description, geometry, ...) -- instantiating ~2000 C++ components -- 2Gb/4Gb memory footprint per process + - ~1min to initialize the application + - loading >500 shared libraries + - connecting to databases (detector description, geometry, ...) + - instantiating ~2000 C++ components + - 2Gb/4Gb memory footprint per process -* ATLAS, General context +## ATLAS, General context Data analysis for publication - collect and massage raw data from detector (ADC, currents, ...) @@ -94,180 +88,177 @@ Reconstruction software: `Athena` .image _figs/data-flux-summary-all.png 300 600 -* Athena: a sophisticated framework +## Athena: a sophisticated framework -- multi-language (`C++`, `python`, `FORTRAN`, `Java`) -- ~300 recurrent developers, -- ~1000 "one-time" developers -- 2000 packages & 5000 components + - multi-language (`C++`, `python`, `FORTRAN`, `Java`) + - ~300 recurrent developers, + - ~1000 "one-time" developers + - 2000 packages & 5000 components .image _figs/athena-component-model.png 300 600 -* Athena@ATLAS +## Athena@ATLAS Worked on `PyAthena`, a `CPython` interface layer to `Athena`: -- write algorithms in `python` -- faster to prototype, slower to execute + - write algorithms in `python` + - faster to prototype, slower to execute Worked on `perfmon`, a performance (`CPU`, `VMem`, `I/O`) monitor for `Athena`: -- minimal overhead (compared to `cachegrind`) -- "turn-key" solution for physicists -- debug `C++` performance issues and memory leaks + - minimal overhead (compared to `cachegrind`) + - "turn-key" solution for physicists + - debug `C++` performance issues and memory leaks .image _figs/fullmon-rdotoesd-perfmon.png 200 500 -* Software development in HEP +## Software development in HEP + + - `C++`: **slow** (very slow?) to compile/develop, **fast** to execute + - `python`: **fast** development cycle (no compilation), **slow** to execute -- `C++`: *slow* (very slow?) to compile/develop, *fast* to execute -- `python`: *fast* development cycle (no compilation), *slow* to execute -# (can be mitigated if leveraging/rewriting(parts in) `C++`. more work) +// (can be mitigated if leveraging/rewriting(parts in) `C++`. more work) .image _figs/xkcd-compiling.png 400 400 Are those our only options ? -* Moore's law +## Moore's law .image _figs/cpu-free-lunch.png 550 550 -* Moore's law +## Moore's law -- Moore's law still observed at the hardware level -- *However* the _effective_ perceived computing power is mitigated + - Moore's law still observed at the hardware level + - **However** the _effective_ perceived computing power is mitigated _"Easy_ _life"_ during the last 20-30 years: -- Moore's law translated into *doubling* compute capacity every ~18 months (_via_ clock frequency) -- *Concurrency* and *parallelism* necessary to efficiently harness the compute power of our new multi-core CPU architectures. + - Moore's law translated into **doubling** compute capacity every ~18 months (_via_ clock frequency) + - **Concurrency** and **parallelism** necessary to efficiently harness the compute power of our new multi-core CPU architectures. _But_ our current software isn't prepared for parallel/concurrent environments. -* AthenaMP@ATLAS +## AthenaMP@ATLAS Worked on `AthenaMP`: a multi-process/multi-core version of `Athena`. -- master process `fork()` `n` sub-processes -- shares memory _via_ `Copy-On-Write` (`COW`) -- uses `n` cores of your multi-core machine + - master process `fork()` `n` sub-processes + - shares memory _via_ `Copy-On-Write` (`COW`) + - uses `n` cores of your multi-core machine .image _figs/athena-mp-seq.png 300 600 -* Interlude: concurrency & parallelism +## Interlude: concurrency & parallelism -* Interlude: concurrency & parallelism +## Interlude: concurrency & parallelism -- *Concurrency* is about _dealing_ with lots of things at once. -- *Parallelism* is about _doing_ lots of things at once. -- Not the same, but related. -- Concurrency is about _structure_, parallelism is about _execution_. + - **Concurrency** is about _dealing_ with lots of things at once. + - **Parallelism** is about _doing_ lots of things at once. + - Not the same, but related. + - Concurrency is about _structure_, parallelism is about _execution_. .image _figs/conc-para.png 200 600 Concurrency is a way to structure a program by breaking it into pieces that can be executed independently. Communication is the means to coordinate the independent executions. -* Concurrency in HEP software +## Concurrency in HEP software .image _figs/conc-level.png 600 400 -* Concurrency in HEP software - II +## Concurrency in HEP software - II .image _figs/levels-of-conc.png -# Graal of concurrent software: +// Graal of concurrent software: .image _figs/gaudi-hive-2.png 250 350 -* +## .image _figs/conc-para-mt-mp.png 600 1000 - -* Time for a new language ? +## Time for a new language ? .image _figs/new-lang.png 600 800 -* Candidates +## Candidates -- python/pypy -- FORTRAN-2008 -- Vala -- Swift -- Rust -- Go -- Chapel -- Scala -- Haskell -- Clojure + - python/pypy + - FORTRAN-2008 + - Vala + - Swift + - Rust + - Go + - Chapel + - Scala + - Haskell + - Clojure -* Why not Go ? +## Why not Go ? -# .image _figs/hello-go.png 600 900 +// .image \_figs/hello-go.png 600 900 .play _code/hello.go - $ go run hello.go - Hello from Go + $ go run hello.go + Hello from Go A nice language with a nice mascot. .image _figs/golang-logo.png 200 400 -* Go in a nutshell +## Go in a nutshell -[[https://golang.org][Go]] is a new, general-purpose programming language. +[Go](https://golang.org) is a new, general-purpose programming language. -- Compiled -- Statically typed -- Concurrent -- Simple -- Productive + - Compiled + - Statically typed + - Concurrent + - Simple + - Productive "Go is a wise, clean, insightful, fresh thinking approach to the greatest-hits subset of the well understood." - Michael T. Jones +## History -* History - -- Project starts at Google in 2007 (by Griesemer, Pike, Thompson) -- Open source release in November 2009 -- More than 250 contributors join the project -- Version 1.0 release in March 2012 -- Version 1.1 release in May 2013 -- Version 1.2 release in December 2013 -- Version 1.3 release in June 2014 -- Version 1.4 release in December 2014 - + - Project starts at Google in 2007 (by Griesemer, Pike, Thompson) + - Open source release in November 2009 + - More than 250 contributors join the project + - Version 1.0 release in March 2012 + - Version 1.1 release in May 2013 + - Version 1.2 release in December 2013 + - Version 1.3 release in June 2014 + - Version 1.4 release in December 2014 -* Elements of Go +## Elements of Go -- Founding fathers: Russ Cox, Robert Griesemer, Ian Lance Taylor, Rob Pike, Ken Thompson + - Founding fathers: Russ Cox, Robert Griesemer, Ian Lance Taylor, Rob Pike, Ken Thompson + - Concurrent, garbage-collected + - An Open-source general progamming language (BSD-3) + - feel of a **dynamic** **language**: limited verbosity thanks to the _type_ _inference_ _system_, map, slices + - safety of a **static** **type** **system** + - compiled down to machine language (so it is fast, goal is ~10% of C) + - **object-oriented** but w/o classes, **builtin** **reflection** + - first-class functions with **closures** + - implicitly satisfied **interfaces** -- Concurrent, garbage-collected -- An Open-source general progamming language (BSD-3) -- feel of a *dynamic* *language*: limited verbosity thanks to the _type_ _inference_ _system_, map, slices -- safety of a *static* *type* *system* -- compiled down to machine language (so it is fast, goal is ~10% of C) -- *object-oriented* but w/o classes, *builtin* *reflection* -- first-class functions with *closures* -- implicitly satisfied *interfaces* +## Concurrency -* Concurrency +## Goroutines -* Goroutines + - The _go_ statement launches a function call as a goroutine -- The _go_ statement launches a function call as a goroutine go f() go f(x, y, ...) -- A goroutine runs concurrently (but not necessarily in parallel) -- A goroutine has its own (growable/shrinkable) stack + - A goroutine runs concurrently (but not necessarily in parallel) + - A goroutine has its own (growable/shrinkable) stack - -* A simple example +## A simple example .code _code/concurrency1.go /f START/,/f END/ @@ -275,55 +266,55 @@ Function f is launched as 3 different goroutines, all running concurrently: .play _code/concurrency1.go /main START/,/main END/ -* go in HEP +## go in HEP -* go in HEP +## go in HEP -- Created an `Athena` like concurrent framework, -- Built `fads` on top of it: a [[https://github.com/go-hep/fads]["FAst Detector Simulation"]] toolkit + - Created an `Athena` like concurrent framework, + - Built `fads` on top of it: a ["FAst Detector Simulation"](https://github.com/go-hep/fads) toolkit Designed with concurrency in mind (`CPU` and `I/O`) Results very encouraging: -- Compared with a [[https://root.cern.ch][ROOT]] -based `C++` fast simulation tool ([[https://cp3.irmp.ucl.ac.be/projects/delphes][Delphes]]) + - Compared with a [ROOT](https://root.cern.ch) -based `C++` fast simulation tool ([Delphes](https://cp3.irmp.ucl.ac.be/projects/delphes)) -* Results - testbenches +## Results - testbenches -- Linux: Intel(R) Core(TM)2 Duo CPU @ 2.53GHz, 4GB RAM, 2 cores -- MacOSX-10.6: Intel(R) Xeon(R) CPU @ 2.27GHz, 172GB RAM, 16 cores -- Linux: Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz, 100GB RAM, 40 cores + - Linux: Intel(R) Core(TM)2 Duo CPU @ 2.53GHz, 4GB RAM, 2 cores + - MacOSX-10.6: Intel(R) Xeon(R) CPU @ 2.27GHz, 172GB RAM, 16 cores + - Linux: Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz, 100GB RAM, 40 cores -* Linux (40 cores) testbench: memory +## Linux (40 cores) testbench: memory .image _figs/lhcb3-rss.png 550 800 -* Linux (40 cores) testbench: CPU +## Linux (40 cores) testbench: CPU .image _figs/lhcb3-cpu.png 550 800 -* Linux (40 cores) testbench: event throughput +## Linux (40 cores) testbench: event throughput .image _figs/lhcb3-hz.png 550 800 -* Conclusions +## Conclusions -[[https://golang.org][Go]] is really nice and can address the challenges of multicore machines. +[Go](https://golang.org) is really nice and can address the challenges of multicore machines. Can't wait to see it take `HEP` (and astro/cosmo) by storm. Working on providing useful software stacks for these communities: -- [[https://go-hep.github.io][go-hep]] -- [[https://github.com/astrogo][astrogo]] + - [go-hep](https://go-hep.github.io) + - [astrogo](https://github.com/astrogo) -[[https://golang.org][Go]] is also a nice platform for DevOps: [[https://docker.io][Docker]] and [[https://coreos.com][CoreOS]] +[Go](https://golang.org) is also a nice platform for DevOps: [Docker](https://docker.io) and [CoreOS](https://coreos.com) (_but_ _that's_ _for_ _next_ _time..._) -* Backup +## Backup -* Communication via channels +## Communication via channels A channel type specifies a channel value type (and possibly a communication direction): @@ -343,7 +334,7 @@ A channel permits _sending_ and _receiving_ values: Channel operations synchronize the communicating goroutines. -* Communicating goroutines +## Communicating goroutines Each goroutine sends its results via channel ch: @@ -352,5 +343,3 @@ Each goroutine sends its results via channel ch: The main goroutine receives (and prints) all results from the same channel: .play _code/concurrency2.go /main START/,/main END/ - - diff --git a/2015/20150929-gopy-lyon/gopy-lyon.slide b/2015/20150929-gopy-lyon/gopy-lyon.slide index 5e21458..1eb7b03 100644 --- a/2015/20150929-gopy-lyon/gopy-lyon.slide +++ b/2015/20150929-gopy-lyon/gopy-lyon.slide @@ -1,56 +1,53 @@ -gopy: extend CPython with Go +# gopy: extend CPython with Go Golang-Lyon, 2015/09/29 Sebastien Binet CNRS/IN2P3 -* Go in High Energy Physics (HEP) + +## Go in High Energy Physics (HEP) .image _figs/mad-scientist-gopher.jpg -* HEP (High Energy Physics) +## HEP (High Energy Physics) Field of physics which studies the fundamental laws of Nature and the properties of the constituents of matter. Many labs working on HEP around the world. -But, perhaps one of the most famous ones is [[http://cern.ch][CERN]]. - +But, perhaps one of the most famous ones is [CERN](http://cern.ch). -* CERN +## CERN .image _figs/cernaerial.jpg 500 700 - -* CERN-LHC +## CERN-LHC LHC: Large Hadron Collider. A proton-proton collider of 27km of circumference. .image _figs/cernring-l.jpg 450 700 - -* ATLAS installation +## ATLAS installation .image _figs/pit.png -* Software in HEP - -Historically, software in HEP has been written in *FORTRAN-77*. +## Software in HEP -- HEP people even wrote compilers -- HEP community even defined a few extensions (*MORTRAN*) +Historically, software in HEP has been written in **FORTRAN-77**. -Mid-90's: migration to *C++* + - HEP people even wrote compilers + - HEP community even defined a few extensions (**MORTRAN**) -Mid-2000's: *Python* gained tremendous mindshare +Mid-90's: migration to **C++** -- first thru the steering of *C++* binaries -- then as complete analyses glueing *C++* libraries together +Mid-2000's: **Python** gained tremendous mindshare + - first thru the steering of **C++** binaries + - then as complete analyses glueing **C++** libraries together -* Software in HEP - some numbers +## Software in HEP - some numbers An LHC experiment (_e.g._ ATLAS, CMS) is ~3000 physicists but only a fraction of those is developing code. @@ -59,400 +56,384 @@ Reconstruction frameworks grew from ~3M SLOC to ~5M Summing over all HEP software stack for e.g. ATLAS: -- event generators: ~1.4M SLOC (C++, FORTRAN-77) -- I/O libraries ~1.7M SLOC (C++) -- simulation libraries ~1.2M SLOC (C++) -- reconstruction framework ~5M SLOC (C++) + steering/configuration (~1M SLOC python) (want to have a look at the [[http://acode-browser.usatlas.bnl.gov/lxr/source/][ATLAS code]]? [[https://github.com/cms-sw/cmssw][CMS code]]?) - + - event generators: ~1.4M SLOC (C++, FORTRAN-77) + - I/O libraries ~1.7M SLOC (C++) + - simulation libraries ~1.2M SLOC (C++) + - reconstruction framework ~5M SLOC (C++) + steering/configuration (~1M SLOC python) (want to have a look at the [ATLAS code](http://acode-browser.usatlas.bnl.gov/lxr/source/)? [CMS code](https://github.com/cms-sw/cmssw)?) +**GCC:** ~7M SLOC +**Linux** **kernel** **3.6:** 15.9M SLOC -*GCC:* ~7M SLOC - -*Linux* *kernel* *3.6:* 15.9M SLOC - -* People committing code to VCS per month +## People committing code to VCS per month Variety of skills Huge turn-around Once the physics data is pouring, people go doing physics instead of software - .image _figs/cmssw-commits.png 400 600 - -* Software developers +## Software developers ~300 active developers (per experiment) ~1000 different developers integrated over the lifetime of a single LHC experiment. -- few "real" s/w experts -- some physicists with strong skill set in s/w -- many with some experience in s/w development -- some with *no* experience in s/w development + - few "real" s/w experts + - some physicists with strong skill set in s/w + - many with some experience in s/w development + - some with **no** experience in s/w development A multi-timezone environment -- Europe, North-America, Japan, Russia + - Europe, North-America, Japan, Russia Many communities (core s/w people, generators, simulation, ...) Development and infrastructures usually CERN-centric -Heavily Linux based ([[http://cern.ch/linux][Scientific Linux CERN]]) - +Heavily Linux based ([Scientific Linux CERN](http://cern.ch/linux)) -* Software development cycle +## Software development cycle VCS (CVS, then SVN. GIT: not yet) Nightlies (Jenkins or homegrown solution) -- need a sizeable cluster of build machines (distcc, ccache, ...) -- builds the framework stack in ~8h -- produces ~2000 shared libraries -- installs them on AFS (also creates RPMs and tarballs) + - need a sizeable cluster of build machines (distcc, ccache, ...) + - builds the framework stack in ~8h + - produces ~2000 shared libraries + - installs them on AFS (also creates RPMs and tarballs) Devs can then test and develop off the nightly _via_ AFS Every 6 months or so a new production release is cut, validated (then patched) and deployed on the World Wide LHC Computing Grid (WLCG). -Release size: *~5Gb* +Release size: **~5Gb** -- binaries, libraries (externals+framework stack) -- extra data (sqlite files, physics processes' modelisation data, ...) + - binaries, libraries (externals+framework stack) + - extra data (sqlite files, physics processes' modelisation data, ...) - -* Software runtime ? +## Software runtime ? Big science, big data, big software, big numbers -- ~1min to initialize the application -- loading >500 shared libraries -- connecting to databases (detector description, geometry, ...) -- instantiating ~2000 C++ components -- 2Gb/4Gb memory footprint per process + - ~1min to initialize the application + - loading >500 shared libraries + - connecting to databases (detector description, geometry, ...) + - instantiating ~2000 C++ components + - 2Gb/4Gb memory footprint per process -* (obligatory xkcd reference) +## (obligatory xkcd reference) .image _figs/xkcd-compiling.png 550 550 - -* +## We learn to love hating our framework. (every step of the way) And even more so in the future: -- work to make our software stack thread-safe -- or at least parts of it multithread friendly to harness multicore machines -- quite a lot of fun ahead - + - work to make our software stack thread-safe + - or at least parts of it multithread friendly to harness multicore machines + - quite a lot of fun ahead -* Remember Go ? +## Remember Go ? -- compiles quickly (no warnings, imports) -- enforces coherent coding rules (across projects) -- builtin test/benchmark/documentation facilities -- deploys easily, cross-compiles easily -- installs easily (also 3rd-party packages: _"go_get"_) -- fast to pick up, not as complicated as *C++* -- builtin reflection system -- builtin (de)serialization capabilities -- concurrency support -- garbage collected + - compiles quickly (no warnings, imports) + - enforces coherent coding rules (across projects) + - builtin test/benchmark/documentation facilities + - deploys easily, cross-compiles easily + - installs easily (also 3rd-party packages: _"go get"_) + - fast to pick up, not as complicated as **C++** + - builtin reflection system + - builtin (de)serialization capabilities + - concurrency support + - garbage collected +**Perfect** **match** for many HEP use cases. -*Perfect* *match* for many HEP use cases. +## Migrating to Go ? (evil plan for (HEP) world domination) -* Migrating to Go ? (evil plan for (HEP) world domination) +Migrating ~5M SLOC of C++ code to Go, during data taking, **unfortunately**, won't fly. -Migrating ~5M SLOC of C++ code to Go, during data taking, *unfortunately*, won't fly. +Creating new applications for data massaging or post-processing **might**. -Creating new applications for data massaging or post-processing *might*. - -Creating a new concurrent and parallel framework for the next accelerator *might*. +Creating a new concurrent and parallel framework for the next accelerator **might**. Need to build a critical mass of Go HEP enthusiasts - So far: -- building the packages to read/write data in HEP formats (see under [[http://github.com/go-hep][go-hep]]) -- built a proof of concept of a concurrent framework: [[http://github.com/go-hep/gaudi-fwk][go-hep/gaudi-fwk]] -- now building the real thing [[http://github.com/go-hep/fwk][go-hep/fwk]] -- building a physics simulation detector app on top of go-hep/fwk: [[http://github.com/go-hep/fads][go-hep/fads]] -- building a package of data analysis facilities + - building the packages to read/write data in HEP formats (see under [go-hep](http://github.com/go-hep)) + - built a proof of concept of a concurrent framework: [go-hep/gaudi-fwk](http://github.com/go-hep/gaudi-fwk) + - now building the real thing [go-hep/fwk](http://github.com/go-hep/fwk) + - building a physics simulation detector app on top of go-hep/fwk: [go-hep/fads](http://github.com/go-hep/fads) + - building a package of data analysis facilities +And, of course... [gopy](https://github.com/go-python/gopy)! -And, of course... [[https://github.com/go-python/gopy][gopy]]! - -* gopy +## gopy With `go-1.5`, introduction of a new feature: the ability to build a `C` shared library from a `Go` package. -- enabled with `-buildmode=c-shared` -- described in the [[https://docs.google.com/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ/edit]["execution modes"]] design document -- leveraged by the `gomobile` command and sub-commands to build applications for mobile platforms (Android + iOS) + - enabled with `-buildmode=c-shared` + - described in the ["execution modes"](https://docs.google.com/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ/edit) design document + - leveraged by the `gomobile` command and sub-commands to build applications for mobile platforms (Android + iOS) `gopy` is heavily inspired from `gomobile` (and reuses some of the code/design of `gomobile`) +## gopy - II -* gopy - II - -[[https://github.com/go-python/gopy][gopy]] automates the drudgery work of +[gopy](https://github.com/go-python/gopy) automates the drudgery work of creating a `python` C-extension module out of a `Go` package: -- inspects a `Go` package -- extracts the exported `types`, `funcs`, `vars` and `consts` -- creates a `Go` package that `cgo` exports the exported entities -- creates a `C` extension module, using the `CPython` API, calling these `cgo` exported entities -- compiles everything together into a `.so` shared object + - inspects a `Go` package + - extracts the exported `types`, `funcs`, `vars` and `consts` + - creates a `Go` package that `cgo` exports the exported entities + - creates a `C` extension module, using the `CPython` API, calling these `cgo` exported entities + - compiles everything together into a `.so` shared object -* gopy - installation +## gopy - installation `gopy` is a pure-Go command, so the installation is as simple as: - $ go get github.com/go-python/gopy/... + $ go get github.com/go-python/gopy/... And voila. Done. - For `gopy` to create the `.so` module, it will need: -- the `python-dev` package (python development headers) -- `pkg-config` and the `python2` config files (to correctly detect the `-I`, `-L` and `-l` incantations) -- a `$CC` compiler (obviously) + - the `python-dev` package (python development headers) + - `pkg-config` and the `python2` config files (to correctly detect the `-I`, `-L` and `-l` incantations) + - a `$CC` compiler (obviously) -* gopy - hello world +## gopy - hello world Consider this simple package `github.com/me/hello`: .code _code/hello.go -* gopy - hello world - II +## gopy - hello world - II To create a `CPython` C-extension module out of it: - $ cd somewhere # anywhere in $PYTHONPATH - $ gopy bind github.com/me/hello - 2015/09/28 15:12:44 work: /tmp/gopy-655556935 + $ cd somewhere # anywhere in $PYTHONPATH + $ gopy bind github.com/me/hello + 2015/09/28 15:12:44 work: /tmp/gopy-655556935 + + $ ls + hello.so + + $ python2 + >>> import hello + >>> dir(hello) + ['Hello', '__doc__', '__file__', '__name__', '__package__'] - $ ls - hello.so + >>> print hello.Hello("Golang-Lyon") + hello "Golang-Lyon" from Go - $ python2 - >>> import hello - >>> dir(hello) - ['Hello', '__doc__', '__file__', '__name__', '__package__'] - - >>> print hello.Hello("Golang-Lyon") - hello "Golang-Lyon" from Go + >>> print hello.__doc__ + hello is a simple package - >>> print hello.__doc__ - hello is a simple package + >>> print hello.Hello.__doc__ + Hello(str name) str - >>> print hello.Hello.__doc__ - Hello(str name) str + Hello greets someone. - Hello greets someone. - - -* gopy - hello world - III + -- extracts the `Go` doc-strings, attachs them onto their `python` counterpart -- creates a function, translates the `Go` native types into their `python` counterpart +## gopy - hello world - III + + - extracts the `Go` doc-strings, attachs them onto their `python` counterpart + - creates a function, translates the `Go` native types into their `python` counterpart Let's see what `gopy` has actually generated: -#.code _code/hello-cgo.go - - $ ls /tmp/gopy-655556935 - hello.c hello.go hello.h - - $ cat /tmp/gopy-655556935/hello.go - package main - - [...] - - //export cgo_func_hello_Hello - // cgo_func_hello_Hello wraps hello.Hello - func cgo_func_hello_Hello(name string) (string) { - _gopy_000 := hello.Hello(name) - return _gopy_000 - } - -* gopy - hello world - IV - - $ cat /tmp/gopy-655556935/hello.c - [...] - - #include "Python.h" - - #if PY_VERSION_HEX > 0x03000000 - # error "Python-3 is not yet supported by gopy" - #endif - - [...] - - /* pythonization of: hello.Hello */ - static PyObject* - cpy_func_hello_Hello(PyObject *self, PyObject *args) { - GoString c_name; - GoString c_gopy_ret; - - if (!PyArg_ParseTuple(args, "O&", cgopy_cnv_py2c_string, &c_name)) { - return NULL; - } - - c_gopy_ret = cgo_func_hello_Hello(c_name); - - return Py_BuildValue("O&", cgopy_cnv_c2py_string, &c_gopy_ret); - } - -* gopy - hello world - V - - $ cat /tmp/gopy-655556935/hello.c - [...] - - /* functions for package hello */ - static PyMethodDef cpy_hello_methods[] = { - {"Hello", cpy_func_hello_Hello, METH_VARARGS, "Hello(str name) str\n\nHello greets someone.\n"}, - {NULL, NULL, 0, NULL} /* Sentinel */ - }; - - PyMODINIT_FUNC - inithello(void) - { - PyObject *module = NULL; - - /* make sure Cgo is loaded and initialized */ - cgo_pkg_hello_init(); - - module = Py_InitModule3("hello", cpy_hello_methods, "hello is a simple package\n"); - - } - - -* gopy - implementation notes +//.code \_code/hello-cgo.go -`gopy` binds together 2 languages with a garbage collector. + $ ls /tmp/gopy-655556935 + hello.c hello.go hello.h + + $ cat /tmp/gopy-655556935/hello.go + package main + + [...] + + //export cgo_func_hello_Hello + // cgo_func_hello_Hello wraps hello.Hello + func cgo_func_hello_Hello(name string) (string) { + _gopy_000 := hello.Hello(name) + return _gopy_000 + } + +## gopy - hello world - IV + + $ cat /tmp/gopy-655556935/hello.c + [...] + + #include "Python.h" + + #if PY_VERSION_HEX > 0x03000000 + # error "Python-3 is not yet supported by gopy" + #endif + + [...] -- the bindings need to tell each side when a given value is not used anymore -- `gopy` notes when a `Go` value goes to the `python` side and increments a counter -- `Go` values that are returned and/or passed to `python` need to be on the heap -- `gopy` will do that for you via the `cgo` exported package + /* pythonization of: hello.Hello */ + static PyObject* + cpy_func_hello_Hello(PyObject *self, PyObject *args) { + GoString c_name; + GoString c_gopy_ret; + + if (!PyArg_ParseTuple(args, "O&", cgopy_cnv_py2c_string, &c_name)) { + return NULL; + } + + c_gopy_ret = cgo_func_hello_Hello(c_name); + + return Py_BuildValue("O&", cgopy_cnv_c2py_string, &c_gopy_ret); + } - // Call sequence (lots of hand-waving) +## gopy - hello world - V - -> python: hello.Hello("foo") - -> cpython: cpy_func_hello_Hello(...) - -> cgo: cgo_func_hello_Hello(...) - -> go: hello.Hello(...) + $ cat /tmp/gopy-655556935/hello.c + [...] + /* functions for package hello */ + static PyMethodDef cpy_hello_methods[] = { + {"Hello", cpy_func_hello_Hello, METH_VARARGS, "Hello(str name) str\n\nHello greets someone.\n"}, + {NULL, NULL, 0, NULL} /* Sentinel */ + }; -* gopy - wrapping a struct + PyMODINIT_FUNC + inithello(void) + { + PyObject *module = NULL; + + /* make sure Cgo is loaded and initialized */ + cgo_pkg_hello_init(); + + module = Py_InitModule3("hello", cpy_hello_methods, "hello is a simple package\n"); + + } + +## gopy - implementation notes + +`gopy` binds together 2 languages with a garbage collector. + + - the bindings need to tell each side when a given value is not used anymore + - `gopy` notes when a `Go` value goes to the `python` side and increments a counter + - `Go` values that are returned and/or passed to `python` need to be on the heap + - `gopy` will do that for you via the `cgo` exported package + + // Call sequence (lots of hand-waving) + + -> python: hello.Hello("foo") + -> cpython: cpy_func_hello_Hello(...) + -> cgo: cgo_func_hello_Hello(...) + -> go: hello.Hello(...) + +## gopy - wrapping a struct Now consider: .code _code/structs.go -* gopy - wrapping a struct - - $ gopy bind github.com/me/structs - - $ python2 - >>> import structs as st - >>> s = st.NewStruct(1) - >>> print s - structs.Struct{private:2, Public:1} - >>> s.Public = 3 - >>> print s - structs.Struct{private:2, Public:3} - >>> print s.Public - 3 - >>> u = st.Struct(42) - >>> print u - structs.Struct{private:0, Public:42} - - >>> u = st.Struct() - >>> print u - structs.Struct{private:0, Public:0} - >>> u.private = 42 - Traceback (most recent call last): - File "", line 1, in - AttributeError: 'structs.Struct' object has no attribute 'private' - -* gopy - wrapping a struct +## gopy - wrapping a struct + + $ gopy bind github.com/me/structs + + $ python2 + >>> import structs as st + >>> s = st.NewStruct(1) + >>> print s + structs.Struct{private:2, Public:1} + >>> s.Public = 3 + >>> print s + structs.Struct{private:2, Public:3} + >>> print s.Public + 3 + >>> u = st.Struct(42) + >>> print u + structs.Struct{private:0, Public:42} + + >>> u = st.Struct() + >>> print u + structs.Struct{private:0, Public:0} + >>> u.private = 42 + Traceback (most recent call last): + File "", line 1, in + AttributeError: 'structs.Struct' object has no attribute 'private' + +## gopy - wrapping a struct `gopy` generated a `CPython` `PyObject` like so: - /* --- decls for struct structs.Struct --- */ - typedef void* cgo_type_structs_Struct; + /* --- decls for struct structs.Struct --- */ + typedef void* cgo_type_structs_Struct; - /* Python type for struct structs.Struct - */ - typedef struct { - PyObject_HEAD - cgo_type_structs_Struct cgopy; /* unsafe.Pointer to structs.Struct */ - } cpy_type_structs_Struct; + /* Python type for struct structs.Struct + */ + typedef struct { + PyObject_HEAD + cgo_type_structs_Struct cgopy; /* unsafe.Pointer to structs.Struct */ + } cpy_type_structs_Struct; -* gopy - wrapping a struct +## gopy - wrapping a struct `gopy` generated a `getter/setter` for the public fields of the `struct`: - /* tp_getset for structs.Struct */ - static PyGetSetDef cpy_type_structs_Struct_getsets[] = { - {"Public", - (getter)cpy_func_structs_Struct_getter_2, - (setter)cpy_func_structs_Struct_setter_2, - "doc for Public", NULL}, - {NULL} /* Sentinel */ - }; + /* tp_getset for structs.Struct */ + static PyGetSetDef cpy_type_structs_Struct_getsets[] = { + {"Public", + (getter)cpy_func_structs_Struct_getter_2, + (setter)cpy_func_structs_Struct_setter_2, + "doc for Public", NULL}, + {NULL} /* Sentinel */ + }; (but not for the private fields) `gopy` also generated a `__str__` method (but it would have wired it to `String()` if the type were to implement the `Stringer` interface) -* gopy - other features (and missing features) +## gopy - other features (and missing features) -- translates the comma-error `Go` idiom into the equivalent `python` idiom (raises an `Exception`) -- slices and arrays implement the sequence protocol (but not yet subslices assignment) -- implements the buffer protocol for slices and arrays -- `maps` aren't supported yet -- `interfaces` aren't supported yet (except for `error`) -- functions/methods taking pointers to values are not supported yet -- implementing a `Go` interface from `python` isn't supported yet either -- channels are not supported yet -- packages exposing other packages in their API are not supported (yet?) + - translates the comma-error `Go` idiom into the equivalent `python` idiom (raises an `Exception`) + - slices and arrays implement the sequence protocol (but not yet subslices assignment) + - implements the buffer protocol for slices and arrays + - `maps` aren't supported yet + - `interfaces` aren't supported yet (except for `error`) + - functions/methods taking pointers to values are not supported yet + - implementing a `Go` interface from `python` isn't supported yet either + - channels are not supported yet + - packages exposing other packages in their API are not supported (yet?) -* gopy - gopy.py +## gopy - gopy.py `gopy` also ships a little `gopy.py` module that calls into the `gopy` command to generate and compile a `Go` package from the `python` prompt: - >>> import gopy - >>> hi = gopy.load("github.com/go-python/gopy/_examples/hi") - gopy> inferring package name... - gopy> loading 'github.com/go-python/gopy/_examples/hi'... - gopy> importing 'github.com/go-python/gopy/_examples/hi' - - >>> print hi - + >>> import gopy + >>> hi = gopy.load("github.com/go-python/gopy/_examples/hi") + gopy> inferring package name... + gopy> loading 'github.com/go-python/gopy/_examples/hi'... + gopy> importing 'github.com/go-python/gopy/_examples/hi' - >>> print hi.__doc__ - package hi exposes a few Go functions to be wrapped and used from Python. + >>> print hi + + >>> print hi.__doc__ + package hi exposes a few Go functions to be wrapped and used from Python. -* Conclusions +## Conclusions Thanks to the simple rules and formalism of `Go` it is reasonnably easy to automate: -- the extraction of the exported API -- the generation of bindings to a (number of) language(s) + - the extraction of the exported API + - the generation of bindings to a (number of) language(s) `gopy` is released under `BSD-3` and accepts pull requests (eagerly!) Come join the fun of the complete world "Gomination" via `CPython`. @@ -461,19 +442,17 @@ Come join the fun of the complete world "Gomination" via `CPython`. .link https://github.com/go-python/gopy/issues .link https://github.com/go-python/gopy/pulls +## Prospects -* Prospects - -- [[https://github.com/golang/go/issues/12416][new rules]] for `cgo` in `go-1.6` and the improved garbage collector will constrain the way `C` extension modules and `.so` can be built. -- will need to reuse `gomobile` 's IPC (I=intra) [[https://godoc.org/golang.org/x/mobile/bind/seq][mobile/bind/seq]] protocol -- implement the missing features -- handle cycles between values, across language boundaries -- support for `python-3` -- support for `cffi` (and perhaps target `PyPy`, and/or other platforms using `cffi`, _e.g._ `IronPython`) + - [new rules](https://github.com/golang/go/issues/12416) for `cgo` in `go-1.6` and the improved garbage collector will constrain the way `C` extension modules and `.so` can be built. + - will need to reuse `gomobile` 's IPC (I=intra) [mobile/bind/seq](https://godoc.org/golang.org/x/mobile/bind/seq) protocol + - implement the missing features + - handle cycles between values, across language boundaries + - support for `python-3` + - support for `cffi` (and perhaps target `PyPy`, and/or other platforms using `cffi`, _e.g._ `IronPython`) Grand (ambitious) vision: -- perhaps extract some of the bits and pieces of `gopy` and create -- [[https://github.com/go-binder][go-binder]] to generate and compile (idiomatic) bindings for `Ruby`, `Perl`, `R`, `C++`, ... -- ie: a `SWIG` for `Go` ? - + - perhaps extract some of the bits and pieces of `gopy` and create + - [go-binder](https://github.com/go-binder) to generate and compile (idiomatic) bindings for `Ruby`, `Perl`, `R`, `C++`, ... + - ie: a `SWIG` for `Go` ? diff --git a/2015/20151001-docker-ecole-ci/talk.slide b/2015/20151001-docker-ecole-ci/talk.slide index 3dd771f..d0dede3 100644 --- a/2015/20151001-docker-ecole-ci/talk.slide +++ b/2015/20151001-docker-ecole-ci/talk.slide @@ -1,478 +1,466 @@ -Docker - CI +# Docker - CI Ecole IN2P3, 2015/10/01 Sebastien Binet CNRS/IN2P3 -* Docker origins +## Docker origins -* The container revolution +## The container revolution Before 1960, cargo transport looked like: .image _figs/transport-pre-1960.png -* MxN combinatorics: matrix from Hell +## MxN combinatorics: matrix from Hell .image _figs/transport-mxn-matrix.png - -* Solution: Intermodal shipping container +## Solution: Intermodal shipping container .image _figs/transport-mxn-solved.png +## Containers - analysis -* Containers - analysis - -- enables seamless shipping on roads, railways and sea (intermodal) -- standardized dimensions -- opaque box convenient for all types of goods (privacy) + - enables seamless shipping on roads, railways and sea (intermodal) + - standardized dimensions + - opaque box convenient for all types of goods (privacy) .image _figs/cargo.jpg -* What is Docker? - +## What is Docker? -* Application deployment +## Application deployment .image _figs/docker-nn-matrix.png _Note:_ a 3rd dimension (OS/platform) could be considered -* Docker: an application container +## Docker: an application container .image _figs/docker-container-code.png -* Docker: no combinatorics no more +## Docker: no combinatorics no more .image _figs/docker-nn-matrix-solved.png -* Docker +## Docker `Docker` is an open source project to pack ship and run any application as a -lightweight container: [[http://www.docker.io][docker.io]] - - +lightweight container: [docker.io](http://www.docker.io) _Note:_ Although `docker` is primarily (ATM) Linux-oriented, it supports other OSes (Windows+MacOSX) at the price of a thin `Linux` VM which is automatically installed (and managed) on these systems. -See [[https://docs.docker.com/installation/][docker installation]] +See [docker installation](https://docs.docker.com/installation/) -* Docker +## Docker `Docker` is an open source project to pack ship and run any application as a -lightweight container: [[http://www.docker.io][docker.io]] +lightweight container: [docker.io](http://www.docker.io) High-level description: -- kind of like a lightweight VM -- runs in its own process space -- has its own network interface -- can run stuff as `root` + - kind of like a lightweight VM + - runs in its own process space + - has its own network interface + - can run stuff as `root` Low-level description: -- `chroot` on steroids -- container `==` isolated process(es) -- share kernel with host -- no device emulation + - `chroot` on steroids + - container `==` isolated process(es) + - share kernel with host + - no device emulation -* Docker: why? +## Docker: why? -- same use cases than for VMs (for `Linux` centric workloads) -- *speed*: boots in (milli)seconds -- *footprint*: 100-1000 containers on a single machine/laptop, small disk requirements + - same use cases than for VMs (for `Linux` centric workloads) + - **speed**: boots in (milli)seconds + - **footprint**: 100-1000 containers on a single machine/laptop, small disk requirements .image _figs/vm-lxc.png 350 600 -* Docker: why? +## Docker: why? -*Efficiency*: _almost_ no overhead +**Efficiency**: _almost_ no overhead -- processes are isolated but run straight on the host -- `CPU` performance = *native* performance -- memory performance = a few % shaved off for (optional) accounting -- network performance = small overhead + - processes are isolated but run straight on the host + - `CPU` performance = **native** performance + - memory performance = a few % shaved off for (optional) accounting + - network performance = small overhead -* Docker: why? +## Docker: why? -*Efficiency*: storage friendly +**Efficiency**: storage friendly -- unioning filesystems -- snapshotting filesystems -- copy-on-write + - unioning filesystems + - snapshotting filesystems + - copy-on-write -* Docker: why? +## Docker: why? -- provisionning takes a few milliseconds -- ... and a few kilobytes -- creating a new container/base-image takes a few seconds + - provisionning takes a few milliseconds + - ... and a few kilobytes + - creating a new container/base-image takes a few seconds -* Why are Docker containers lightweight? +## Why are Docker containers lightweight? .image _figs/vm-vs-containers.png 600 1000 -* Separation of concerns +## Separation of concerns Tailored for the dev team: -- my code -- my framework -- my libraries -- my system dependencies -- my packaging system -- my distro -- my data + - my code + - my framework + - my libraries + - my system dependencies + - my packaging system + - my distro + - my data Don't care where it's running or how. -* Separation of concerns +## Separation of concerns Tailored for the ops team: -- logs -- backups -- remote access -- monitoring -- uptime + - logs + - backups + - remote access + - monitoring + - uptime Don't care what's running in it. -* Docker: blueprint +## Docker: blueprint -* Docker: blueprint +## Docker: blueprint _Build,_ _ship_ and _run_ any application, _anywhere_. - `Docker` uses a client/server architecture: -- the `docker` _client_ talks to -- a `docker` _daemon_ via sockets or a RESTful API. + - the `docker` _client_ talks to + - a `docker` _daemon_ via sockets or a RESTful API. .image _figs/architecture.svg 350 600 - -* Docker: basics of the system +## Docker: basics of the system .image _figs/docker-system.png 550 1000 -* Docker: the CLI +## Docker: the CLI The `docker` client ships with many a subcommand: - $ docker help - Usage: docker [OPTIONS] COMMAND [arg...] - docker daemon [ --help | ... ] - docker [ -h | --help | -v | --version ] + $ docker help + Usage: docker [OPTIONS] COMMAND [arg...] + docker daemon [ --help | ... ] + docker [ -h | --help | -v | --version ] - A self-sufficient runtime for containers. + A self-sufficient runtime for containers. - [...] + [...] - Commands: - attach Attach to a running container - build Build an image from a Dockerfile - commit Create a new image from a container's changes - cp Copy files/folders from a container to a HOSTDIR or to STDOUT - images List images - import Import the contents from a tarball to create a filesystem image - info Display system-wide information - [...] + Commands: + attach Attach to a running container + build Build an image from a Dockerfile + commit Create a new image from a container's changes + cp Copy files/folders from a container to a HOSTDIR or to STDOUT + images List images + import Import the contents from a tarball to create a filesystem image + info Display system-wide information + [...] -* Docker: the CLI +## Docker: the CLI - $ docker version - Client: - Version: 1.8.1 - API version: 1.20 - Go version: go1.4.2 - Git commit: d12ea79 - Built: Sat Aug 15 17:29:10 UTC 2015 - OS/Arch: linux/amd64 + $ docker version + Client: + Version: 1.8.1 + API version: 1.20 + Go version: go1.4.2 + Git commit: d12ea79 + Built: Sat Aug 15 17:29:10 UTC 2015 + OS/Arch: linux/amd64 - Server: - Version: 1.8.1 - API version: 1.20 - Go version: go1.4.2 - Git commit: d12ea79 - Built: Sat Aug 15 17:29:10 UTC 2015 - OS/Arch: linux/amd64 + Server: + Version: 1.8.1 + API version: 1.20 + Go version: go1.4.2 + Git commit: d12ea79 + Built: Sat Aug 15 17:29:10 UTC 2015 + OS/Arch: linux/amd64 -* Interlude: docker configuration @CC.in2p3 +## Interlude: docker configuration @CC.in2p3 -* Linux +## Linux Just use the default config. -* MacOSX/Windows +## MacOSX/Windows Same thing, but on `MacOSX` and `Windows`, let's create a new Linux VM and call it `vm-ecole`: - $ docker-machine create -d virtualbox \ - vm-ecole - - $ eval $(docker-machine env vm-ecole) ## MacOSX - $ eval $(./docker-machine.exe env --shell=bash vm-ecole) ## Win + $ docker-machine create -d virtualbox \ + vm-ecole + $ eval $(docker-machine env vm-ecole) ## MacOSX + $ eval $(./docker-machine.exe env --shell=bash vm-ecole) ## Win -* Hello World +## Hello World Fetch a `docker` image from the `docker` registry: - $ docker pull busybox - Using default tag: latest - latest: Pulling from library/busybox - cf2616975b4a: Pull complete - 6ce2e90b0bc7: Pull complete - 8c2e06607696: Already exists - library/busybox:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security. - Digest: sha256:38a203e1986cf79639cfb9b2e1d6e773de84002feea2d4eb006b52004ee8502d - Status: Downloaded newer image for busybox:latest + $ docker pull busybox + Using default tag: latest + latest: Pulling from library/busybox + cf2616975b4a: Pull complete + 6ce2e90b0bc7: Pull complete + 8c2e06607696: Already exists + library/busybox:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security. + Digest: sha256:38a203e1986cf79639cfb9b2e1d6e773de84002feea2d4eb006b52004ee8502d + Status: Downloaded newer image for busybox:latest - $ docker images - REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE - busybox latest 8c2e06607696 4 months ago 2.43 MB + $ docker images + REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE + busybox latest 8c2e06607696 4 months ago 2.43 MB Now, run a command inside the image: - $ docker run busybox echo "Hello World" - Hello World - + $ docker run busybox echo "Hello World" + Hello World -* Docker basics +## Docker basics -- Run a container in detached mode: + - Run a container in detached mode: - $ docker run -d busybox sh -c \ - 'while true; do echo "hello"; sleep 1; done;' + $ docker run -d busybox sh -c \ + 'while true; do echo "hello"; sleep 1; done;' -- Retrieve the container id: + - Retrieve the container id: - $ docker ps - CONTAINER ID IMAGE COMMAND CREATED STATUS - 321c1aa5bcd4 busybox "sh -c 'while true; d" 3 seconds ago Up 2 seconds + $ docker ps + CONTAINER ID IMAGE COMMAND CREATED STATUS + 321c1aa5bcd4 busybox "sh -c 'while true; d" 3 seconds ago Up 2 seconds -- Attach to the running container: + - Attach to the running container: - $ docker attach 321c1aa5bcd4 - hello - hello - [...] + $ docker attach 321c1aa5bcd4 + hello + hello + [...] -- Start/stop/restart container - - $ docker stop 321c1aa5bcd4 - $ docker restart 321c1aa5bcd4 + - Start/stop/restart container + + $ docker stop 321c1aa5bcd4 + $ docker restart 321c1aa5bcd4 - -* Docker: public index (aka registry, aka the Hub) +## Docker: public index (aka registry, aka the Hub) `Docker` containers may be published and shared on a public registry, the `Hub`. -- It is searchable: + - It is searchable: - $ docker search apache2 - NAME STARS OFFICIAL AUTOMATED - rootlogin/apache2-symfony2 7 [OK] - reinblau/php-apache2 6 [OK] - tianon/apache2 4 [OK] - [...] - $ docker pull tianon/apache2 + $ docker search apache2 + NAME STARS OFFICIAL AUTOMATED + rootlogin/apache2-symfony2 7 [OK] + reinblau/php-apache2 6 [OK] + tianon/apache2 4 [OK] + [...] + $ docker pull tianon/apache2 -- Run the image and check the ports + - Run the image and check the ports - $ docker run -d -p 8080:80 tianon/apache2 - $ docker ps - CONTAINER ID IMAGE COMMAND PORTS - 49614161f5b7 tianon/apache2 "apache2 -DFOREGROUND" 0.0.0.0:8080->80/tcp + $ docker run -d -p 8080:80 tianon/apache2 + $ docker ps + CONTAINER ID IMAGE COMMAND PORTS + 49614161f5b7 tianon/apache2 "apache2 -DFOREGROUND" 0.0.0.0:8080->80/tcp The registry is also available from the browser: -- [[https://hub.docker.com][hub.docker.com]] + - [hub.docker.com](https://hub.docker.com) + +## Docker: creating a customized image -* Docker: creating a customized image + - run `docker` interactively: -- run `docker` interactively: + $ docker run -it ubuntu bash + root@524ef6c2e4ce:/# apt-get install -y memcached + [...] + root@524ef6c2e4ce:/# exit - $ docker run -it ubuntu bash - root@524ef6c2e4ce:/# apt-get install -y memcached - [...] - root@524ef6c2e4ce:/# exit - - $ docker commit `docker ps -q -l` binet/memcached - 4242210aba21641013b22198c7bdc00435b00850aaf9ae9cedc53ba75794891d + $ docker commit `docker ps -q -l` binet/memcached + 4242210aba21641013b22198c7bdc00435b00850aaf9ae9cedc53ba75794891d - $ docker run -d -p 11211 -u daemon binet/memcached memcached - a84e18168f1473a338f9ea3473dd981bf5e3dc7e41511a1252f7bb216d875860 + $ docker run -d -p 11211 -u daemon binet/memcached memcached + a84e18168f1473a338f9ea3473dd981bf5e3dc7e41511a1252f7bb216d875860 - $ docker ps - CONTAINER ID IMAGE COMMAND PORTS - a84e18168f14 binet/memcached "memcached" 0.0.0:32768->11211/tcp - -* Docker: creating a customized image + $ docker ps + CONTAINER ID IMAGE COMMAND PORTS + a84e18168f14 binet/memcached "memcached" 0.0.0:32768->11211/tcp -- interactive way is fine but not scalable -- enter `Dockerfiles` -- recipes to build an image -- start `FROM` a base image -- `RUN` commands on top of it -- easy to learn, easy to use +## Docker: creating a customized image -* Docker: Dockerfile + - interactive way is fine but not scalable + - enter `Dockerfiles` + - recipes to build an image + - start `FROM` a base image + - `RUN` commands on top of it + - easy to learn, easy to use + +## Docker: Dockerfile .code _code/dockerfile-nginx -* Docker: Dockerfile-II +## Docker: Dockerfile-II -- run in the directory holding that `Dockerfile` + - run in the directory holding that `Dockerfile` - $ docker build -t /server . - $ docker run -d -P /server + $ docker build -t /server . + $ docker run -d -P /server -- retrieve the port number: + - retrieve the port number: - $ docker ps - 34dc03cdbae8 binet/server "/bin/sh -c 'nginx -g" 0.0.0.0:32770->80/tcp + $ docker ps + 34dc03cdbae8 binet/server "/bin/sh -c 'nginx -g" 0.0.0.0:32770->80/tcp or: - $ docker inspect -f '{{.NetworkSettings.Ports}}' 34dc03cdbae8 + $ docker inspect -f '{{.NetworkSettings.Ports}}' 34dc03cdbae8 and then: - $ curl localhost:32770 - Hi, I am in your container! + $ curl localhost:32770 + Hi, I am in your container! -* Docker: Dockerfile-III +## Docker: Dockerfile-III -*NOTE:* for Windows(TM) and MacOSX(TM) users, a thin `Linux` VM is sitting +**NOTE:** for Windows(TM) and MacOSX(TM) users, a thin `Linux` VM is sitting between your machine and the container. The container is running inside that VM so you need to replace `localhost` with the `IP` of that VM: - $ docker-machine ip vm-ecole - 192.168.59.103 + $ docker-machine ip vm-ecole + 192.168.59.103 and then: - $ curl 192.168.59.103:32770 - Hi, I am in your container! - -* docker build + $ curl 192.168.59.103:32770 + Hi, I am in your container! + +## docker build -- takes a snapshot after each step -- re-uses those snapshots in future builds -- doesn't re-run slow steps when it isn't necessary (cache system) + - takes a snapshot after each step + - re-uses those snapshots in future builds + - doesn't re-run slow steps when it isn't necessary (cache system) .image _figs/docker-changes-updates.png 350 800 -* Docker Hub +## Docker Hub -- `docker` `push` an image to the Hub -- `docker` `pull` an image from the Hub to any machine + - `docker` `push` an image to the Hub + - `docker` `pull` an image from the Hub to any machine This brings: -- reliable deployment -- consistency + - reliable deployment + - consistency -- images are self-contained, independent from host -- if it works locally, it will work on the server -- _exact_ _same_ _behavior_ -- regardless of versions, distros and dependencies + - images are self-contained, independent from host + - if it works locally, it will work on the server + - _exact_ _same_ _behavior_ + - regardless of versions, distros and dependencies -* Docker for the developer +## Docker for the developer -- manage and *control* dependencies -- if it works on my machine, it works on the cluster -- reproducibility -- small but durable recipes + - manage and **control** dependencies + - if it works on my machine, it works on the cluster + - reproducibility + - small but durable recipes Never again: -- juggle with 3 different incompatible FORTRAN compilers -- voodoo incantations to get that exotic library to link with IDL -- figure out which version of LAPACK works with that code -- ... and what obscure flag coaxed it into compiling last time + - juggle with 3 different incompatible FORTRAN compilers + - voodoo incantations to get that exotic library to link with IDL + - figure out which version of LAPACK works with that code + - ... and what obscure flag coaxed it into compiling last time -* Development workflow +## Development workflow -- Fetch code (`git`, `mercurial`, ...) + - Fetch code (`git`, `mercurial`, ...) - $ git clone git@gitlab.in2p3.fr:EcoleInfo2015/TP.git - $ cd TP + $ git clone git@gitlab.in2p3.fr:EcoleInfo2015/TP.git + $ cd TP -- Edit code -- Mount code inside a `build` container -- Build+test inside that container + - Edit code + - Mount code inside a `build` container + - Build+test inside that container We'll test this workflow in the remainder of the hands-on session... -* Create a base container - -- create a directory `docker-web-base` to hold the `Dockerfile` for the base container -- create the `Dockerfile` and choose your favorite `Linux` distro (say, `centos`) as a base image, -- install the needed dependencies for the web-app (`maven` on `centos`) -- run: +## Create a base container - $ docker build -t /web-base . + - create a directory `docker-web-base` to hold the `Dockerfile` for the base container + - create the `Dockerfile` and choose your favorite `Linux` distro (say, `centos`) as a base image, + - install the needed dependencies for the web-app (`maven` on `centos`) + - run: + $ docker build -t /web-base . -* Create a base container - solution +## Create a base container - solution (see next slide) -* Create a base container - II +## Create a base container - II .code _code/base-dockerfile -* Base container for development +## Base container for development -- One could create a new container with all the development tools (editor, completion, ...) -- But you'd need to carry over the configuration (`ssh` keys, editor, ...) + - One could create a new container with all the development tools (editor, completion, ...) + - But you'd need to carry over the configuration (`ssh` keys, editor, ...) -Probably easier to just mount the sources *inside* the base container: +Probably easier to just mount the sources **inside** the base container: + + $ docker run -it -v `pwd`:/opt/in2p3/tp -p 8080:8080 /web-base bash + [root@48b2c74a5004 tp]# ./bin/compile.sh + [root@48b2c74a5004 tp]# java -jar /opt/in2p3/tp/target/webserver-1.0-SNAPSHOT-jar-with-dependencies.jar + 2015-09-10 14:13:25.467:INFO::main: Logging initialized @623ms + 2015-09-10 14:13:25.719:INFO:oejs.Server:main: jetty-9.3.z-SNAPSHOT + 2015-09-10 14:13:25.918:INFO:oejsh.ContextHandler:main: Started o.e.j.s.ServletContextHandler@4645801a{/,null,AVAILABLE} + 2015-09-10 14:13:25.933:INFO:oejs.ServerConnector:main: Started ServerConnector@62b60fbb{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} + 2015-09-10 14:13:25.934:INFO:oejs.Server:main: Started @1093ms - $ docker run -it -v `pwd`:/opt/in2p3/tp -p 8080:8080 /web-base bash - [root@48b2c74a5004 tp]# ./bin/compile.sh - [root@48b2c74a5004 tp]# java -jar /opt/in2p3/tp/target/webserver-1.0-SNAPSHOT-jar-with-dependencies.jar - 2015-09-10 14:13:25.467:INFO::main: Logging initialized @623ms - 2015-09-10 14:13:25.719:INFO:oejs.Server:main: jetty-9.3.z-SNAPSHOT - 2015-09-10 14:13:25.918:INFO:oejsh.ContextHandler:main: Started o.e.j.s.ServletContextHandler@4645801a{/,null,AVAILABLE} - 2015-09-10 14:13:25.933:INFO:oejs.ServerConnector:main: Started ServerConnector@62b60fbb{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} - 2015-09-10 14:13:25.934:INFO:oejs.Server:main: Started @1093ms - In another terminal: - $ curl localhost:8080 -

Bienvenue à l'école informatique IN2P3 2015

- Analyse de données - session=1kdgwh0cue0efokpe5t1gcvtj + $ curl localhost:8080 +

Bienvenue à l'école informatique IN2P3 2015

+ Analyse de données + session=1kdgwh0cue0efokpe5t1gcvtj -* Base container for dev - II +## Base container for dev - II -- On windows, the correct `-v` syntax is like: + - On windows, the correct `-v` syntax is like: - $ docker run -it -v //c/Users/username/some/path:/opt/in2p3/tp ... + $ docker run -it -v //c/Users/username/some/path:/opt/in2p3/tp ... .link https://github.com/docker/docker/issues/12590#issuecomment-96767796 - -* Create the final container +## Create the final container Now that we know the base image "works", we'll automatize the build part as yet another `Dockerfile`: -- create a new `Dockerfile` file (at the root of the `git` repository) based on the `web-base` image, with the correct build+run instructions -- make sure you can `docker` `build` it and tag it as `web-app` -- make sure that you can still access the web server when you run: + - create a new `Dockerfile` file (at the root of the `git` repository) based on the `web-base` image, with the correct build+run instructions + - make sure you can `docker` `build` it and tag it as `web-app` + - make sure that you can still access the web server when you run: - $ docker run -d -p 8080:8080 /web-app + $ docker run -d -p 8080:8080 /web-app _Hint:_ `ADD` @@ -480,125 +468,125 @@ _Hint:_ `CMD` .link https://docs.docker.com/reference/builder/ -* Create the final container - solutions +## Create the final container - solutions (see next slide) -* Create the final container - II +## Create the final container - II .code _code/webapp-dockerfile -* Create the final container - III +## Create the final container - III -- `CMD` describes the command to be run by default when the container is started -- `ADD` copies files, directories or URLs into the container's filesystem -- `VOLUME` creates a volume mount point inside the container which can contain data from the host or from other containers -- `USER` defines the user (or `UID`) with whom to run the various commands inside the container + - `CMD` describes the command to be run by default when the container is started + - `ADD` copies files, directories or URLs into the container's filesystem + - `VOLUME` creates a volume mount point inside the container which can contain data from the host or from other containers + - `USER` defines the user (or `UID`) with whom to run the various commands inside the container -* Create multiple versions of an image +## Create multiple versions of an image At times, it might be very useful to test 2 versions of an application and run them concurrently (to debug discrepancies.) Let's do just that. -- tag the last `web-app` image as `v1` + - tag the last `web-app` image as `v1` - $ docker tag \ - /web-app \ - /web-app:v1 + $ docker tag \ + /web-app \ + /web-app:v1 -- modify `src/main/java/fr/in2p3/informatique/ecole2015/web/MyServlet.java` to print a different welcome message -- containerize the new version as `.../web-app:v2` + - modify `src/main/java/fr/in2p3/informatique/ecole2015/web/MyServlet.java` to print a different welcome message + - containerize the new version as `.../web-app:v2` -* Create multiple versions of an image - II +## Create multiple versions of an image - II -- run the container `v2` on port `8082` + - run the container `v2` on port `8082` - $ docker run -p 8082:8080 \ - --name=web-app-v2 \ - /web-app:v2 + $ docker run -p 8082:8080 \ + --name=web-app-v2 \ + /web-app:v2 -- run the container `v1` on port `8080` + - run the container `v1` on port `8080` - $ docker run -p 8080:8080 \ - --name=web-app-v1 \ - /web-app:v1 + $ docker run -p 8080:8080 \ + --name=web-app-v1 \ + /web-app:v1 -- make sure the servers on ports `8080` and `8082` display the correct welcome messages. + - make sure the servers on ports `8080` and `8082` display the correct welcome messages. -* Continuous Integration +## Continuous Integration Modify the `Jenkins` build to: -- build the final container -- run the tests suite -- push the result as a tagged image + - build the final container + - run the tests suite + - push the result as a tagged image -* Sharing images +## Sharing images Up to now, the images you've been creating have been put on your local registry. But there is another registry instance available at: - cc-ecole2015-docker.in2p3.fr:5000 + cc-ecole2015-docker.in2p3.fr:5000 Let's try to package the previous `web-app:v2` and `web-app:v1` images and put them on that new registry: - $ docker tag \ - /web-app \ - cc-ecole2015-docker.in2p3.fr:5000//web-app + $ docker tag \ + /web-app \ + cc-ecole2015-docker.in2p3.fr:5000//web-app Now, try to `pull` the `web-app` image of your friend and run it. -* Inspecting logs +## Inspecting logs `docker` is nice enough to let us inspect what (running) containers are generating as logs. For a single container, it is as simple as: - $ docker logs - $ docker logs + $ docker logs + $ docker logs -- inspect the logs of your `web-app-v2` container -- insepct the logs of the container running your local registry + - inspect the logs of your `web-app-v2` container + - insepct the logs of the container running your local registry -* Inspecting logs - II +## Inspecting logs - II _e.g.:_ - $ docker logs web-app-v2 - { - "timeMillis" : 1443615891663, - "thread" : "main", - "level" : "INFO", - "loggerName" : "org.eclipse.jetty.server.Server", - "message" : "Started @1343ms", - "endOfBatch" : false, - "loggerFqcn" : "org.eclipse.jetty.util.log.Slf4jLog", - "contextMap" : [ ] - } + $ docker logs web-app-v2 + { + "timeMillis" : 1443615891663, + "thread" : "main", + "level" : "INFO", + "loggerName" : "org.eclipse.jetty.server.Server", + "message" : "Started @1343ms", + "endOfBatch" : false, + "loggerFqcn" : "org.eclipse.jetty.util.log.Slf4jLog", + "contextMap" : [ ] + } - $ docker logs ecole-registry - [...] - 30/Sep/2015:12:20:13 +0000 DEBUG: args = {'tag': u'latest', 'namespace': u'binet', 'repository': u'web-app'} - 30/Sep/2015:12:20:13 +0000 DEBUG: [get_tag] namespace=binet; repository=web-app; tag=latest - 30/Sep/2015:12:20:13 +0000 DEBUG: api_error: Tag not found - 172.17.42.1 - - [30/Sep/2015:12:20:13 +0000] "GET /v1/repositories/binet/web-app/tags/latest HTTP/1.1" 404 26 "-" "docker/1.8.2 go/go1.5.1 git-commit/0a8c2e3-dirty kernel/4.2.1-1-ARCH os/linux arch/amd64" + $ docker logs ecole-registry + [...] + 30/Sep/2015:12:20:13 +0000 DEBUG: args = {'tag': u'latest', 'namespace': u'binet', 'repository': u'web-app'} + 30/Sep/2015:12:20:13 +0000 DEBUG: [get_tag] namespace=binet; repository=web-app; tag=latest + 30/Sep/2015:12:20:13 +0000 DEBUG: api_error: Tag not found + 172.17.42.1 - - [30/Sep/2015:12:20:13 +0000] "GET /v1/repositories/binet/web-app/tags/latest HTTP/1.1" 404 26 "-" "docker/1.8.2 go/go1.5.1 git-commit/0a8c2e3-dirty kernel/4.2.1-1-ARCH os/linux arch/amd64" -* Inspecting logs - III +## Inspecting logs - III -- launch a container in interactive mode -- start a bash shell -- run inside that container: + - launch a container in interactive mode + - start a bash shell + - run inside that container: - docker> logger -i -s plop + docker> logger -i -s plop -- in another terminal: + - in another terminal: - $ docker logs + $ docker logs -* Creation of a build+target container pair +## Creation of a build+target container pair So far, we have been building containers where the intermediate results leading to the final binary (or set of binaries) are left inside the image. @@ -608,8 +596,8 @@ resource heavy. The usual solution is to have a 2-step process: -- a container in which the binaries are built -- a container in which the binaries are directly copied from the first + - a container in which the binaries are built + - a container in which the binaries are directly copied from the first Let's do that. @@ -617,33 +605,33 @@ _Hint:_ `docker` `export` _Hint:_ `docker` `import` _Hint:_ `docker` `cp` -* Creation of a build+target container pair +## Creation of a build+target container pair (solution on next slide) -* Creation of a build+target container pair +## Creation of a build+target container pair Extract the root fs from the build image: - $ mkdir rootfs && cd rootfs - $ docker run -d -p 8080:8080 --name=web-app-v2 \ - localhost:5000//web-app:v2 - $ docker export web-app-v2 | tar xf - - $ ls opt/in2p3/tp - bin/ .git/ sonar-project.properties - doc/ .gitignore src/ - Dockerfile pom.xml target/ - docker-web-base/ README.md + $ mkdir rootfs && cd rootfs + $ docker run -d -p 8080:8080 --name=web-app-v2 \ + localhost:5000//web-app:v2 + $ docker export web-app-v2 | tar xf - + $ ls opt/in2p3/tp + bin/ .git/ sonar-project.properties + doc/ .gitignore src/ + Dockerfile pom.xml target/ + docker-web-base/ README.md Another way is to use `docker` `cp`: - $ docker cp web-app-v2:/opt/in2p3/tp tp - + $ docker cp web-app-v2:/opt/in2p3/tp tp + The binaries are under `target`. If they were static libraries, you could just create a very slim container with them, using `docker` `import`. -* Running GUIs +## Running GUIs The application we have been currently "dockerizing" doesn't need any graphics per se. @@ -652,64 +640,61 @@ Many do, though. Let's try to run a simple graphics-enabled application from within a `docker` container: - $ docker run -it --rm centos bash - docker> yum install -y xclock - docker> xclock - + $ docker run -it --rm centos bash + docker> yum install -y xclock + docker> xclock If the network is too slow, you can try to pull: - $ docker pull \ - cc-ecole2015-docker.in2p3.fr:5000/centos-xclock + $ docker pull \ + cc-ecole2015-docker.in2p3.fr:5000/centos-xclock where `xclock` and its dependencies have been already installed. -* Running GUIs - II +## Running GUIs - II Running GUIs is a bit more involved than just running your simple "from the mill" CLI application. There are many options to enable graphics: -- `ssh` into a container with `X11` forwarding -- `VNC` -- sharing the `X11` socket - + - `ssh` into a container with `X11` forwarding + - `VNC` + - sharing the `X11` socket .link http://fabiorehm.com/blog/2014/09/11/running-gui-apps-with-docker/ .link https://blog.docker.com/2013/07/docker-desktop-your-desktop-over-ssh-running-inside-of-a-docker-container/ .link http://wiki.ros.org/docker/Tutorials/GUI -* Running GUIs - III +## Running GUIs - III Let's try the most direct (albeit a bit insecure) one: sharing the `X11` socket. First, allow all `X11` connections (that's the insecure part): - $ xhost + + $ xhost + Then: - $ docker run -ti --rm \ - -e DISPLAY=$DISPLAY \ - -v /tmp/.X11-unix:/tmp/.X11-unix \ - centos bash - docker> yum install -y xclock && xclock + $ docker run -ti --rm \ + -e DISPLAY=$DISPLAY \ + -v /tmp/.X11-unix:/tmp/.X11-unix \ + centos bash + docker> yum install -y xclock && xclock Don't forget to re-enable `X11` access control afterwards: - $ xhost - + $ xhost - +## Conclusions -* Conclusions + - `docker` is a rather good tool to deploy applications in containers + - eases the life of developers and sysadmins (devops) + - `docker` isn't the only game in town + - [rkt](https://coreos.com/rkt/docs) (`rocket`) from `CoreOS` + - [systemd-nspawn](http://0pointer.de/public/systemd-man/systemd-nspawn.html), now part of `systemd` -- `docker` is a rather good tool to deploy applications in containers -- eases the life of developers and sysadmins (devops) -- `docker` isn't the only game in town -- [[https://coreos.com/rkt/docs][rkt]] (`rocket`) from `CoreOS` -- [[http://0pointer.de/public/systemd-man/systemd-nspawn.html][systemd-nspawn]], now part of `systemd` - -* References +## References .link http://www.slideshare.net/jpetazzo/introduction-to-docker-december-2014-tour-de-france-bordeaux-special-edition .link http://www.slideshare.net/dotCloud/docker-intro-november @@ -719,4 +704,3 @@ Don't forget to re-enable `X11` access control afterwards: .link http://kubernetes.io/ .link http://mesos.apache.org/ .link https://coreos.com/rkt/docs - diff --git a/2015/20151109-gopy-dotgo/gopy-dotgo.slide b/2015/20151109-gopy-dotgo/gopy-dotgo.slide index a08eea8..7d57ca7 100644 --- a/2015/20151109-gopy-dotgo/gopy-dotgo.slide +++ b/2015/20151109-gopy-dotgo/gopy-dotgo.slide @@ -1,4 +1,4 @@ -gopy: extend CPython with Go +# gopy: extend CPython with Go DotGo, 2015/11/09 Sebastien Binet @@ -8,113 +8,113 @@ https://github.com/sbinet -* github.com/go-python/gopy +## github.com/go-python/gopy -[[https://github.com/go-python/gopy][gopy]] automates the drudgery work of +[gopy](https://github.com/go-python/gopy) automates the drudgery work of creating a `python` C-extension module out of a `Go` package: -- inspects a `Go` package -- extracts the exported `types`, `funcs`, `vars` and `consts` -- creates a `Go` package that `cgo` exports the exported entities -- creates a `C` extension module, using the `CPython` API, calling these `cgo` exported entities -- compiles everything together into a `.so` shared object + - inspects a `Go` package + - extracts the exported `types`, `funcs`, `vars` and `consts` + - creates a `Go` package that `cgo` exports the exported entities + - creates a `C` extension module, using the `CPython` API, calling these `cgo` exported entities + - compiles everything together into a `.so` shared object -* gopy - hello world +## gopy - hello world Consider this simple package `github.com/me/hello`: .code _code/hello.go -* gopy - hello world - II +## gopy - hello world - II To create a `CPython` C-extension module out of it: - $ cd somewhere # anywhere in $PYTHONPATH - $ gopy bind github.com/me/hello - 2015/09/28 15:12:44 work: /tmp/gopy-655556935 + $ cd somewhere # anywhere in $PYTHONPATH + $ gopy bind github.com/me/hello + 2015/09/28 15:12:44 work: /tmp/gopy-655556935 - $ ls - hello.so + $ ls + hello.so - $ python2 - >>> import hello - >>> dir(hello) - ['Hello', '__doc__', '__file__', '__name__', '__package__'] - - >>> print hello.Hello("dotgo.io") - hello "dotgo.io" from Go + $ python2 + >>> import hello + >>> dir(hello) + ['Hello', '__doc__', '__file__', '__name__', '__package__'] - >>> print hello.__doc__ - hello is a simple package + >>> print hello.Hello("dotgo.io") + hello "dotgo.io" from Go - -* gopy - hello world - III + >>> print hello.__doc__ + hello is a simple package -- extracts the `Go` doc-strings, attachs them onto their `python` counterpart -- creates a function, translates the `Go` native types into their `python` counterpart + + +## gopy - hello world - III + + - extracts the `Go` doc-strings, attachs them onto their `python` counterpart + - creates a function, translates the `Go` native types into their `python` counterpart Let's see what `gopy` has actually generated: -#.code _code/hello-cgo.go +//.code \_code/hello-cgo.go - $ ls /tmp/gopy-655556935 - hello.c hello.go hello.h - - $ cat /tmp/gopy-655556935/hello.go - package main + $ ls /tmp/gopy-655556935 + hello.c hello.go hello.h - [...] + $ cat /tmp/gopy-655556935/hello.go + package main - //export cgo_func_hello_Hello - // cgo_func_hello_Hello wraps hello.Hello - func cgo_func_hello_Hello(name string) (string) { - _gopy_000 := hello.Hello(name) - return _gopy_000 - } + [...] -* gopy - hello world - IV + //export cgo_func_hello_Hello + // cgo_func_hello_Hello wraps hello.Hello + func cgo_func_hello_Hello(name string) (string) { + _gopy_000 := hello.Hello(name) + return _gopy_000 + } - $ cat /tmp/gopy-655556935/hello.c - [...] +## gopy - hello world - IV - /* pythonization of: hello.Hello */ - static PyObject* - cpy_func_hello_Hello(PyObject *self, PyObject *args) { - GoString c_name; - GoString c_gopy_ret; - - if (!PyArg_ParseTuple(args, "O&", cgopy_cnv_py2c_string, &c_name)) { - return NULL; - } - - c_gopy_ret = cgo_func_hello_Hello(c_name); - - return Py_BuildValue("O&", cgopy_cnv_c2py_string, &c_gopy_ret); - } + $ cat /tmp/gopy-655556935/hello.c + [...] + /* pythonization of: hello.Hello */ + static PyObject* + cpy_func_hello_Hello(PyObject *self, PyObject *args) { + GoString c_name; + GoString c_gopy_ret; + + if (!PyArg_ParseTuple(args, "O&", cgopy_cnv_py2c_string, &c_name)) { + return NULL; + } + + c_gopy_ret = cgo_func_hello_Hello(c_name); + + return Py_BuildValue("O&", cgopy_cnv_c2py_string, &c_gopy_ret); + } -* gopy - other features (and missing features) +## gopy - other features (and missing features) -- wraps named types (+ methods) into `python` classes -- translates the comma-error `Go` idiom into the equivalent `python` idiom (raises an `Exception`) -- slices and arrays implement the sequence protocol (but not yet subslices assignment) -- implements the buffer protocol for slices and arrays + - wraps named types (+ methods) into `python` classes + - translates the comma-error `Go` idiom into the equivalent `python` idiom (raises an `Exception`) + - slices and arrays implement the sequence protocol (but not yet subslices assignment) + - implements the buffer protocol for slices and arrays -*Missing* *features:* +**Missing** **features:** -- `maps` and `interfaces` aren't supported yet (except for `error`) -- functions/methods taking pointers to values are not supported yet -- implementing a `Go` interface from `python` isn't supported yet either -- channels are not supported yet -- packages exposing other packages in their API are not supported (yet?) + - `maps` and `interfaces` aren't supported yet (except for `error`) + - functions/methods taking pointers to values are not supported yet + - implementing a `Go` interface from `python` isn't supported yet either + - channels are not supported yet + - packages exposing other packages in their API are not supported (yet?) -* Conclusions +## Conclusions Thanks to the simple rules and formalism of `Go` it is reasonnably easy to automate: -- the extraction of the exported API -- the generation of bindings to a (number of) language(s) + - the extraction of the exported API + - the generation of bindings to a (number of) language(s) `gopy` is released under `BSD-3` and accepts pull requests (eagerly!) Come join the fun of the complete world "Gomination" via `CPython`. @@ -123,5 +123,4 @@ Come join the fun of the complete world "Gomination" via `CPython`. .link https://github.com/go-python/gopy/issues .link https://github.com/go-python/gopy/pulls -A longer version of this talk is available [[http://talks.sbinet.org/2015/20150929-gopy-lyon/gopy-lyon.slide][here]]. - +A longer version of this talk is available [here](http://talks.sbinet.org/2015/20150929-gopy-lyon/gopy-lyon.slide). diff --git a/2015/20151214-alice-parcxx/alice-parcxx.slide b/2015/20151214-alice-parcxx/alice-parcxx.slide index 745b009..0c3f535 100644 --- a/2015/20151214-alice-parcxx/alice-parcxx.slide +++ b/2015/20151214-alice-parcxx/alice-parcxx.slide @@ -1,15 +1,15 @@ -Introduction to parallel programming +# Introduction to parallel programming MRRTF, 2015/12/14 Sebastien Binet CNRS/IN2P3/LPC binet@cern.ch -* Disclaimer +## Disclaimer I used to be a `C++` programmer, then a `python` programmer. -Nowadays, I am a [[https://golang.org][Go]] programmer at heart, enjoying blazing fast compilation time, builtin concurrency constructs, easy installation and deployment, etc... +Nowadays, I am a [Go](https://golang.org) programmer at heart, enjoying blazing fast compilation time, builtin concurrency constructs, easy installation and deployment, etc... I really learned parallel/concurrent programming with `Go`. My views on parallel programming with `C++` are thus somewhat skewed. @@ -17,53 +17,53 @@ My views on parallel programming with `C++` are thus somewhat skewed. .image figs/gopher.png .caption The Go mascott: a gopher -* Parallel programming +## Parallel programming Parallel programming is hard. Parallel programming is harder in `C++`. -Actually, why do we have to do *parallel* programming ? +Actually, why do we have to do **parallel** programming ? -* Moore's law +## Moore's law .image figs/cpu-free-lunch.png _ 550 -* The hardware/software contract +## The hardware/software contract .image figs/par-prog-old-days.png _ 850 -* +## .image figs/par-prog-power.png _ 850 -* +## .image figs/par-prog-cores.png _ 850 -* Many cores: we are all doing it +## Many cores: we are all doing it .image figs/par-prog-hw.png _ 800 -* Hardware diversity: basic building blocks +## Hardware diversity: basic building blocks .image figs/par-prog-building-blocks.png -* Hardware diversity: combining building blocks +## Hardware diversity: combining building blocks .image figs/par-prog-heterogeneous.png -* Hardware diversity: CPUs +## Hardware diversity: CPUs .image figs/par-prog-cpu.png _ 800 -* Parallel programming +## Parallel programming Ok, so we have to do parallel programming to properly utilize our new hardware... -Actually, are we really sure we want to do *parallel* programming ? +Actually, are we really sure we want to do **parallel** programming ? -* Concurrency vs Parallelism +## Concurrency vs Parallelism _Concurrency:_ programming as the composition of independently executing processes/tasks. @@ -71,144 +71,144 @@ _Parallelism:_ programming as the simultaneous execution of (possibly related) c .image figs/conc-vs-par.png 350 _ -* Concurrency vs Parallelism +## Concurrency vs Parallelism Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once. -Concurrency is about (program) *structure*. -Parallelism is about (program) *execution*. +Concurrency is about (program) **structure**. +Parallelism is about (program) **execution**. .image figs/conc-vs-par-prog.png 350 _ -* Concurrency vs Parallelism +## Concurrency vs Parallelism Concurrency and parallelism are related. Concurrency isn't parallelism (it's better!) Parallelizing an application is done by: -- finding concurrency in the problem -- exposing the concurrency in the source code -- exploiting the exposed concurrency to complete the job in less time. + - finding concurrency in the problem + - exposing the concurrency in the source code + - exploiting the exposed concurrency to complete the job in less time. .image figs/conc-vs-par-decomp.png -* Decomposition in parallel programs +## Decomposition in parallel programs Every parallel program is based on concurrency i.e: tasks defined by an application that can run at the same time. -*EVERY* parallel program requires a _task_decomposition_ and a _data_decomposition_: +**EVERY** parallel program requires a _task decomposition_ and a _data decomposition_: -- Task decomposition: break the application down into a set of tasks that can execute concurrently. -- Data decomposition: How must the data be broken down into chunks and associated with threads/processes to make the parallel program run efficiently. + - Task decomposition: break the application down into a set of tasks that can execute concurrently. + - Data decomposition: How must the data be broken down into chunks and associated with threads/processes to make the parallel program run efficiently. -* Goldilocks +## Goldilocks Parallel approaches have to find the "sweet spot" between two extremes. Too fine grained: -- Data: computation dominated by overhead -- Tasks: context switching overhead + - Data: computation dominated by overhead + - Tasks: context switching overhead Too coarse grained: -- Data: load balancing problems -- Tasks: insufficient items to keep processes busy + - Data: load balancing problems + - Tasks: insufficient items to keep processes busy -* Parallel programming: multi-process vs multi-threaded +## Parallel programming: multi-process vs multi-threaded Multi-process: -- resource requirements are multiplied w/ nbr of process instances -- (clever) use of `fork(2)` can mitigate this issue (but not in a `MT` environment) -- one process can not corrupt the memory of another process -- overhead of pushing data from one process to the other + - resource requirements are multiplied w/ nbr of process instances + - (clever) use of `fork(2)` can mitigate this issue (but not in a `MT` environment) + - one process can not corrupt the memory of another process + - overhead of pushing data from one process to the other Multi-threaded: -- small context switch times (wrt an OS' process) -- automatic sharing of many hardware resources (memory, fds, sockets...) -- thread-safety of external libraries? -- one thread can corrupt another thread + - small context switch times (wrt an OS' process) + - automatic sharing of many hardware resources (memory, fds, sockets...) + - thread-safety of external libraries? + - one thread can corrupt another thread `FairRoot` design can cater for both (`MP` _via_ `ZeroMQ` or `NanoMQ` message queues) -* Multi-threading & Multi-processing in a C++ world +## Multi-threading & Multi-processing in a C++ world Modern architectures impose massive challenges on programmability in the context of performance portability. -- massive increase in on-node parallelism -- deep memory hierarchies + - massive increase in on-node parallelism + - deep memory hierarchies -Only *portable* parallelization solution for `C++` programmers (today?): *OpenMP* & *MPI* +Only **portable** parallelization solution for `C++` programmers (today?): **OpenMP** & **MPI** -- hugely successful for years -- widely used and supported -- simple use for simple cases -- very portable -- highly optimized + - hugely successful for years + - widely used and supported + - simple use for simple cases + - very portable + - highly optimized _Which_ `C++` BTW ? -* C++ timeline +## C++ timeline `C++` is a wide and complex language. -Know your `C++` and the (subset of?) `C++` you are *allowed* to write! +Know your `C++` and the (subset of?) `C++` you are **allowed** to write! -- `C++03`? -- `C++11`? -- `C++14`? + - `C++03`? + - `C++11`? + - `C++14`? .image figs/wg21-timeline.png 300 _ -* Parallelism in C++ +## Parallelism in C++ `C++11` introduced lower level abstractions: -- `std::thread`, `std::mutex`, `std::future`, ... -- fairly limited, more is needed -- `C++` needs stronger support for higher-level parallelism + - `std::thread`, `std::mutex`, `std::future`, ... + - fairly limited, more is needed + - `C++` needs stronger support for higher-level parallelism Several proposals to the Standardization Committee are accepted or under consideration: -- Technical Specification: Concurrency -- Technical Specification: Parallelism -- Other smaller proposals: resumable functions, task regions, executors + - Technical Specification: Concurrency + - Technical Specification: Parallelism + - Other smaller proposals: resumable functions, task regions, executors -* Parallelism in C++ +## Parallelism in C++ Currently, there is no overarching vision related to higher-level parallelism -- goal is to standardize on a "big story" by 2020 -- no need for OpenMP, OpenACC, OpenCL, etc... + - goal is to standardize on a "big story" by 2020 + - no need for OpenMP, OpenACC, OpenCL, etc... But for the moment, `C++` programmers are stuck with `C++11/14`... -* Interlude: memory model & tools +## Interlude: memory model & tools -* Memory model +## Memory model With `C++11`, finally `C++` has a memory model that contemplates a multi-threaded execution of a program. A thread is a single flow of control within a program -- Every thread can potentially access every object and function in the program -- The interleaving of each thread's instructions is undefined + - Every thread can potentially access every object and function in the program + - The interleaving of each thread's instructions is undefined .image figs/thread-exec-race.png -* Memory model +## Memory model -`C++` guarantees that two threads can update and access *separate* memory locations without interfering with each other. +`C++` guarantees that two threads can update and access **separate** memory locations without interfering with each other. -- For all other situations updates and accesses have to be properly synchronized (_ie:_ atomics, locks, memory fences) -- If updates and accesses to the same location by multiple threads are not properly synchronized, there is a data race (_ie:_ undefined behavior) -- Data races can be made visible by transformations applied by the compiler or by the processor for performance reasons + - For all other situations updates and accesses have to be properly synchronized (_ie:_ atomics, locks, memory fences) + - If updates and accesses to the same location by multiple threads are not properly synchronized, there is a data race (_ie:_ undefined behavior) + - Data races can be made visible by transformations applied by the compiler or by the processor for performance reasons -* Tools to support parallel programming development/debugging +## Tools to support parallel programming development/debugging .link http://valgrind.org/docs/manual/drd-manual.html .link http://valgrind.org/docs/manual/hg-manual.html @@ -216,248 +216,247 @@ A thread is a single flow of control within a program Detect races at runtime: -- needs a test case and workload that triggers a race -- no false positive, but can not detect all possible races + - needs a test case and workload that triggers a race + - no false positive, but can not detect all possible races Needs a recompilation (`TSan`), (dramatic) increase of resources requirements at runtime (CPU, VMem) b/c of code instrumentation and bookkeeping. -* C++ parallel programming: building blocks +## C++ parallel programming: building blocks -* C++11 std::thread +## C++11 std::thread `C++11` (finally!) standardized threads -- `std::thread` is now part of the standard `C++` library -- `std::thread` is an abstraction and maps to local platform threads (`POSIX`, `Windows(TM)`, ...) + - `std::thread` is now part of the standard `C++` library + - `std::thread` is an abstraction and maps to local platform threads (`POSIX`, `Windows(TM)`, ...) -* C++11 std::thread +## C++11 std::thread .code code/hello-par.cxx - $> g++ -o hello-par --std=c++11 -pthread hello-par.cxx - $> ./hello-par - ** inside thread 139800850654976! - -* C++11 std::thread - avoiding errors + $> g++ -o hello-par --std=c++11 -pthread hello-par.cxx + $> ./hello-par + ** inside thread 139800850654976! + +## C++11 std::thread - avoiding errors .code code/hello-2.cxx -*(1)* Thread function must do *exception*handling*: unhandled exception => program termination +**(1)** Thread function must do **exception handling**: unhandled exception => program termination -*(2)* Must join with _thread_ *before* handle goes out of scope, otherwise: +**(2)** Must join with _thread_ **before** handle goes out of scope, otherwise: program termination. -* Programming style +## Programming style -- Old school: thread functions (what we just saw) -- Middle school: function objects (functors) + - Old school: thread functions (what we just saw) + - Middle school: function objects (functors) .code code/par-functors.cxx -* Programming style +## Programming style -- New school: `C++11` lambda functions (aka anonymous functions) + - New school: `C++11` lambda functions (aka anonymous functions) It's all about trade-offs... -*Lambda*functions:* +**Lambda functions:** -- easier and more readable (once your brained has been trained) -- code remains inline -- potentially more *dangerous* (`[&]` captures everything by reference) + - easier and more readable (once your brained has been trained) + - code remains inline + - potentially more **dangerous** (`[&]` captures everything by reference) -*Functions:* +**Functions:** -- more efficient: lambdas involve class, function objects -- potentially *safer*: requires explicit variable scoping -- more cumbersome + - more efficient: lambdas involve class, function objects + - potentially **safer**: requires explicit variable scoping + - more cumbersome -* Example: matrix multiply +## Example: matrix multiply .image figs/matrix-multiply.png -* Sequential version +## Sequential version .image figs/matrix-multiply-seq.png -* Structured (fork-join) parallelism +## Structured (fork-join) parallelism A common pattern when creating multiple threads .image figs/matrix-multiply-fork-join.png -* Parallel solution +## Parallel solution .image figs/matrix-multiply-par.png _ 850 -* Types of parallelism +## Types of parallelism Most common types: -- Data -- Task -- Embarrassingly parallel -- Dataflow + - Data + - Task + - Embarrassingly parallel + - Dataflow -* Data parallelism +## Data parallelism .image figs/par-types-data.png -* Task parallelism +## Task parallelism .image figs/par-types-task.png -* Embarrassingly parallel +## Embarrassingly parallel .image figs/par-types-emb-par.png -* Dataflow +## Dataflow .image figs/par-types-dataflow.png -* C++ concurrency features +## C++ concurrency features .image figs/cxx-features.png -* Futures +## Futures -*Futures* provide a higher level of abstraction +**Futures** provide a higher level of abstraction -- you start an asynchronous/parallel operation -- you are returned a handle to wait for the result -- thread creation, join and exceptions are handled for you + - you start an asynchronous/parallel operation + - you are returned a handle to wait for the result + - thread creation, join and exceptions are handled for you -* std::async + std::future +## std::async + std::future .image figs/future-async.png -* async operations +## async operations .image figs/future-async-2.png -* (no) Conclusions +## (no) Conclusions That's all for today. Many more to cover, though: -- SIMD -- Thread Building Blocks (`Intel` `TBB`) -- `FPGA` -- (auto-)vectorization -- `OpenCL`/`OpenMP`/`OpenACC` -- ... + - SIMD + - Thread Building Blocks (`Intel` `TBB`) + - `FPGA` + - (auto-)vectorization + - `OpenCL / OpenMP / OpenACC` + - ... -* Parallel programming: references +## Parallel programming: references .link https://web2.infn.it/esc15 .link http://concur.rspace.googlecode.com/hg/talk/concur.html .link http://sc13.supercomputing.org/sites/default/files/prog105/prog105.pdf - -* FairMQ + docker +## FairMQ + docker Running the `FairMQ` example 2 with `docker` (you need 3 terminals): - term-1> docker run -it --rm --name=ex2-fair hepsw/alice-fair bash - term-2> docker exec -it ex2-fair bash - term-3> docker exec -it ex2-fair bash + term-1> docker run -it --rm --name=ex2-fair hepsw/alice-fair bash + term-2> docker exec -it ex2-fair bash + term-3> docker exec -it ex2-fair bash -In `term-1`, create `/data/ex2-sample-process-sink.json` from [[https://github.com/FairRootGroup/FairRoot/blob/master/examples/MQ/2-sampler-processor-sink/ex2-sampler-processor-sink.json][ex2-sample-process-sink.json]]: +In `term-1`, create `/data/ex2-sample-process-sink.json` from [ex2-sample-process-sink.json](https://github.com/FairRootGroup/FairRoot/blob/master/examples/MQ/2-sampler-processor-sink/ex2-sampler-processor-sink.json): - term-1> curl -L \ - https://raw.githubusercontent.com/FairRootGroup/FairRoot/master/examples/MQ/2-sampler-processor-sink/ex2-sampler-processor-sink.json \ - > /data/ex2-sampler-processor-sink.json + term-1> curl -L \ + https://raw.githubusercontent.com/FairRootGroup/FairRoot/master/examples/MQ/2-sampler-processor-sink/ex2-sampler-processor-sink.json \ + > /data/ex2-sampler-processor-sink.json Then: - term-1> ex2-sampler --id sampler1 \ - --config-json-file /data/ex2-sampler-processor-sink.json - term-2> ex2-sink --id sink1 \ - --config-json-file /data/ex2-sampler-processor-sink.json - term-3> ex2-processor --id processor2 - --config-json-file /data/ex2-sampler-processor-sink.json - -* FairMQ - term-1 - - term-1> ex2-sampler --id sampler1 \ - --config-json-file /data/ex2-sampler-processor-sink.json - [16:22:24][STATE] Entering FairMQ state machine - [...] - [16:22:24][STATE] Entering INITIALIZING DEVICE state - [16:22:24][DEBUG] Validating channel "data-out[0]"... VALID - [16:22:24][DEBUG] Initializing channel data-out[0] (push) - [16:22:24][DEBUG] Binding channel data-out[0] on tcp://*:5555 - [16:22:25][STATE] Entering DEVICE READY state - [16:22:25][STATE] Entering INITIALIZING TASK state - [16:22:25][STATE] Entering READY state - [16:22:25][STATE] Entering RUNNING state - [16:22:25][INFO] DEVICE: Running... - [16:22:26][INFO] Sending "Hello" - -* FairMQ - term-2 - - term-2> ex2-sink --id sink1 --config-json-file /data/ex2-sampler-processor-sink.json - [16:24:33][STATE] Entering FairMQ state machine - [...] - [16:24:33][STATE] Entering INITIALIZING DEVICE state - [16:24:33][DEBUG] Validating channel "data-in[0]"... VALID - [16:24:33][DEBUG] Initializing channel data-in[0] (pull) - [16:24:33][DEBUG] Binding channel data-in[0] on tcp://*:5556 - [16:24:34][STATE] Entering DEVICE READY state - [16:24:34][STATE] Entering INITIALIZING TASK state - [16:24:34][STATE] Entering READY state - [16:24:34][STATE] Entering RUNNING state - [16:24:34][INFO] DEVICE: Running... - -* FairMQ - term-3 - - term-3> ex2-processor --id processor2 --config-json-file /data/ex2-sampler-processor-sink.json - [16:26:30][STATE] Entering FairMQ state machine - [...] - [16:26:30][STATE] Entering INITIALIZING DEVICE state - [16:26:30][DEBUG] Validating channel "data-out[0]"... VALID - [16:26:30][DEBUG] Initializing channel data-out[0] (push) - [16:26:30][DEBUG] Connecting channel data-out[0] to tcp://localhost:5556 - [16:26:30][DEBUG] Validating channel "data-in[0]"... VALID - [16:26:30][DEBUG] Initializing channel data-in[0] (pull) - [16:26:30][DEBUG] Connecting channel data-in[0] to tcp://localhost:5555 - [16:26:31][STATE] Entering DEVICE READY state - [16:26:31][STATE] Entering INITIALIZING TASK state - [16:26:31][STATE] Entering READY state - [16:26:31][STATE] Entering RUNNING state - [16:26:31][INFO] DEVICE: Running... - [16:26:31][INFO] Received data, processing... - [16:26:31][INFO] Received data, processing... - [16:26:32][INFO] Received data, processing... - [16:26:33][INFO] Received data, processing... - [16:26:34][INFO] Received data, processing... - ^C - [16:26:35][INFO] Caught signal 2 - [16:26:35][DEBUG] Closed all sockets! - [16:26:35][INFO] Exiting. - -* FairMQ + term-1> ex2-sampler --id sampler1 \ + --config-json-file /data/ex2-sampler-processor-sink.json + term-2> ex2-sink --id sink1 \ + --config-json-file /data/ex2-sampler-processor-sink.json + term-3> ex2-processor --id processor2 + --config-json-file /data/ex2-sampler-processor-sink.json + +## FairMQ - term-1 + + term-1> ex2-sampler --id sampler1 \ + --config-json-file /data/ex2-sampler-processor-sink.json + [16:22:24][STATE] Entering FairMQ state machine + [...] + [16:22:24][STATE] Entering INITIALIZING DEVICE state + [16:22:24][DEBUG] Validating channel "data-out[0]"... VALID + [16:22:24][DEBUG] Initializing channel data-out[0] (push) + [16:22:24][DEBUG] Binding channel data-out[0] on tcp://*:5555 + [16:22:25][STATE] Entering DEVICE READY state + [16:22:25][STATE] Entering INITIALIZING TASK state + [16:22:25][STATE] Entering READY state + [16:22:25][STATE] Entering RUNNING state + [16:22:25][INFO] DEVICE: Running... + [16:22:26][INFO] Sending "Hello" + + +## FairMQ - term-2 + + term-2> ex2-sink --id sink1 --config-json-file /data/ex2-sampler-processor-sink.json + [16:24:33][STATE] Entering FairMQ state machine + [...] + [16:24:33][STATE] Entering INITIALIZING DEVICE state + [16:24:33][DEBUG] Validating channel "data-in[0]"... VALID + [16:24:33][DEBUG] Initializing channel data-in[0] (pull) + [16:24:33][DEBUG] Binding channel data-in[0] on tcp://*:5556 + [16:24:34][STATE] Entering DEVICE READY state + [16:24:34][STATE] Entering INITIALIZING TASK state + [16:24:34][STATE] Entering READY state + [16:24:34][STATE] Entering RUNNING state + [16:24:34][INFO] DEVICE: Running... + +## FairMQ - term-3 + + term-3> ex2-processor --id processor2 --config-json-file /data/ex2-sampler-processor-sink.json + [16:26:30][STATE] Entering FairMQ state machine + [...] + [16:26:30][STATE] Entering INITIALIZING DEVICE state + [16:26:30][DEBUG] Validating channel "data-out[0]"... VALID + [16:26:30][DEBUG] Initializing channel data-out[0] (push) + [16:26:30][DEBUG] Connecting channel data-out[0] to tcp://localhost:5556 + [16:26:30][DEBUG] Validating channel "data-in[0]"... VALID + [16:26:30][DEBUG] Initializing channel data-in[0] (pull) + [16:26:30][DEBUG] Connecting channel data-in[0] to tcp://localhost:5555 + [16:26:31][STATE] Entering DEVICE READY state + [16:26:31][STATE] Entering INITIALIZING TASK state + [16:26:31][STATE] Entering READY state + [16:26:31][STATE] Entering RUNNING state + [16:26:31][INFO] DEVICE: Running... + [16:26:31][INFO] Received data, processing... + [16:26:31][INFO] Received data, processing... + [16:26:32][INFO] Received data, processing... + [16:26:33][INFO] Received data, processing... + [16:26:34][INFO] Received data, processing... + ^C + [16:26:35][INFO] Caught signal 2 + [16:26:35][DEBUG] Closed all sockets! + [16:26:35][INFO] Exiting. + +## FairMQ When `processor2` is run, the chain is complete and processing can start: - [term-1] - [16:22:25][INFO] DEVICE: Running... - [16:22:26][INFO] Sending "Hello" - [16:26:31][INFO] Sending "Hello" - [16:26:32][INFO] Sending "Hello" - [16:26:33][INFO] Sending "Hello" - [16:26:34][INFO] Sending "Hello" - [16:26:35][INFO] Sending "Hello" - - [term-2] - [16:26:31][INFO] Received message: "Hello (modified by processor2)" - [16:26:31][INFO] Received message: "Hello (modified by processor2)" - [16:26:32][INFO] Received message: "Hello (modified by processor2)" - [16:26:33][INFO] Received message: "Hello (modified by processor2)" - [16:26:34][INFO] Received message: "Hello (modified by processor2)" - -- 6 "Hello" data packets are send over the wire -- 5 packets are processed by `processor2` (before `^C` was hit) -- 5 packets are displayed by `sink1` - + [term-1] + [16:22:25][INFO] DEVICE: Running... + [16:22:26][INFO] Sending "Hello" + [16:26:31][INFO] Sending "Hello" + [16:26:32][INFO] Sending "Hello" + [16:26:33][INFO] Sending "Hello" + [16:26:34][INFO] Sending "Hello" + [16:26:35][INFO] Sending "Hello" + + [term-2] + [16:26:31][INFO] Received message: "Hello (modified by processor2)" + [16:26:31][INFO] Received message: "Hello (modified by processor2)" + [16:26:32][INFO] Received message: "Hello (modified by processor2)" + [16:26:33][INFO] Received message: "Hello (modified by processor2)" + [16:26:34][INFO] Received message: "Hello (modified by processor2)" + + - 6 "Hello" data packets are send over the wire + - 5 packets are processed by `processor2` (before `^C` was hit) + - 5 packets are displayed by `sink1` diff --git a/2015/20151217-loops-docker/talk.slide b/2015/20151217-loops-docker/talk.slide index 7d7aff0..07c71c6 100644 --- a/2015/20151217-loops-docker/talk.slide +++ b/2015/20151217-loops-docker/talk.slide @@ -1,406 +1,397 @@ -Introduction to Docker +# Introduction to Docker Reseau LoOPS, 2015/12/17 Sebastien Binet CNRS/IN2P3 -* Docker origins +## Docker origins -* The container revolution +## The container revolution Before 1960, cargo transport looked like: .image _figs/transport-pre-1960.png -* MxN combinatorics: matrix from Hell +## MxN combinatorics: matrix from Hell .image _figs/transport-mxn-matrix.png - -* Solution: Intermodal shipping container +## Solution: Intermodal shipping container .image _figs/transport-mxn-solved.png +## Containers - analysis -* Containers - analysis - -- enables seamless shipping on roads, railways and sea (intermodal) -- standardized dimensions -- opaque box convenient for all types of goods (privacy) + - enables seamless shipping on roads, railways and sea (intermodal) + - standardized dimensions + - opaque box convenient for all types of goods (privacy) .image _figs/cargo.jpg -* What is Docker? - +## What is Docker? -* Application deployment +## Application deployment .image _figs/docker-nn-matrix.png _Note:_ a 3rd dimension (OS/platform) could be considered -* Docker: an application container +## Docker: an application container .image _figs/docker-container-code.png -* Docker: no combinatorics no more +## Docker: no combinatorics no more .image _figs/docker-nn-matrix-solved.png -* Docker +## Docker `Docker` is an open source project to pack ship and run any application as a -lightweight container: [[http://www.docker.io][docker.io]] - - +lightweight container: [docker.io](http://www.docker.io) _Note:_ Although `docker` is primarily (ATM) Linux-oriented, it supports other OSes (Windows+MacOSX) at the price of a thin `Linux` VM which is automatically installed (and managed) on these systems. -See [[https://docs.docker.com/installation/][docker installation]] +See [docker installation](https://docs.docker.com/installation/) -* Docker +## Docker `Docker` is an open source project to pack ship and run any application as a -lightweight container: [[http://www.docker.io][docker.io]] +lightweight container: [docker.io](http://www.docker.io) High-level description: -- kind of like a lightweight VM -- runs in its own process space -- has its own network interface -- can run stuff as `root` + - kind of like a lightweight VM + - runs in its own process space + - has its own network interface + - can run stuff as `root` Low-level description: -- `chroot` on steroids -- container `==` isolated process(es) -- share kernel with host -- no device emulation + - `chroot` on steroids + - container `==` isolated process(es) + - share kernel with host + - no device emulation -* Docker: why? +## Docker: why? -- same use cases than for VMs (for `Linux` centric workloads) -- *speed*: boots in (milli)seconds -- *footprint*: 100-1000 containers on a single machine/laptop, small disk requirements + - same use cases than for VMs (for `Linux` centric workloads) + - **speed**: boots in (milli)seconds + - **footprint**: 100-1000 containers on a single machine/laptop, small disk requirements .image _figs/vm-lxc.png 350 600 -* Docker: why? +## Docker: why? -*Efficiency*: _almost_ no overhead +**Efficiency**: _almost_ no overhead -- processes are isolated but run straight on the host -- `CPU` performance = *native* performance -- memory performance = a few % shaved off for (optional) accounting -- network performance = small overhead + - processes are isolated but run straight on the host + - `CPU` performance = **native** performance + - memory performance = a few % shaved off for (optional) accounting + - network performance = small overhead -* Docker: why? +## Docker: why? -*Efficiency*: storage friendly +**Efficiency**: storage friendly -- unioning filesystems -- snapshotting filesystems -- copy-on-write + - unioning filesystems + - snapshotting filesystems + - copy-on-write -* Docker: why? +## Docker: why? -- provisionning takes a few milliseconds -- ... and a few kilobytes -- creating a new container/base-image takes a few seconds + - provisionning takes a few milliseconds + - ... and a few kilobytes + - creating a new container/base-image takes a few seconds -* Why are Docker containers lightweight? +## Why are Docker containers lightweight? .image _figs/vm-vs-containers.png 600 1000 -* Separation of concerns +## Separation of concerns Tailored for the dev team: -- my code -- my framework -- my libraries -- my system dependencies -- my packaging system -- my distro -- my data + - my code + - my framework + - my libraries + - my system dependencies + - my packaging system + - my distro + - my data Don't care where it's running or how. -* Separation of concerns +## Separation of concerns Tailored for the ops team: -- logs -- backups -- remote access -- monitoring -- uptime + - logs + - backups + - remote access + - monitoring + - uptime Don't care what's running in it. -* Docker: blueprint +## Docker: blueprint -* Docker: blueprint +## Docker: blueprint _Build,_ _ship_ and _run_ any application, _anywhere_. - `Docker` uses a client/server architecture: -- the `docker` _client_ talks to -- a `docker` _daemon_ via sockets or a RESTful API. + - the `docker` _client_ talks to + - a `docker` _daemon_ via sockets or a RESTful API. .image _figs/architecture.svg 350 600 - -* Docker: basics of the system +## Docker: basics of the system .image _figs/docker-system.png 550 1000 -* Docker: the CLI +## Docker: the CLI The `docker` client ships with many a subcommand: - $ docker help - Usage: docker [OPTIONS] COMMAND [arg...] - docker daemon [ --help | ... ] - docker [ -h | --help | -v | --version ] - - A self-sufficient runtime for containers. - - [...] - - Commands: - attach Attach to a running container - build Build an image from a Dockerfile - commit Create a new image from a container's changes - cp Copy files/folders from a container to a HOSTDIR or to STDOUT - images List images - import Import the contents from a tarball to create a filesystem image - info Display system-wide information - [...] - -* Docker: the CLI - - $ docker version - Client: - Version: 1.9.1 - API version: 1.21 - Go version: go1.5.1 - Git commit: a34a1d5-dirty - Built: Sun Nov 22 00:15:15 UTC 2015 - OS/Arch: linux/amd64 - - Server: - Version: 1.9.1 - API version: 1.21 - Go version: go1.5.1 - Git commit: a34a1d5-dirty - Built: Sun Nov 22 00:15:15 UTC 2015 - OS/Arch: linux/amd64 - -* Whirlwind tour of docker features - -* Hello World + $ docker help + Usage: docker [OPTIONS] COMMAND [arg...] + docker daemon [ --help | ... ] + docker [ -h | --help | -v | --version ] -Fetch a `docker` image from the `docker` registry: + A self-sufficient runtime for containers. - $ docker pull busybox - Using default tag: latest - latest: Pulling from library/busybox - cf2616975b4a: Pull complete - 6ce2e90b0bc7: Pull complete - 8c2e06607696: Already exists - library/busybox:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security. - Digest: sha256:38a203e1986cf79639cfb9b2e1d6e773de84002feea2d4eb006b52004ee8502d - Status: Downloaded newer image for busybox:latest + [...] - $ docker images - REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE - busybox latest 8c2e06607696 4 months ago 2.43 MB + Commands: + attach Attach to a running container + build Build an image from a Dockerfile + commit Create a new image from a container's changes + cp Copy files/folders from a container to a HOSTDIR or to STDOUT + images List images + import Import the contents from a tarball to create a filesystem image + info Display system-wide information + [...] -Now, run a command inside the image: +## Docker: the CLI - $ docker run busybox echo "Hello World" - Hello World + $ docker version + Client: + Version: 1.9.1 + API version: 1.21 + Go version: go1.5.1 + Git commit: a34a1d5-dirty + Built: Sun Nov 22 00:15:15 UTC 2015 + OS/Arch: linux/amd64 + Server: + Version: 1.9.1 + API version: 1.21 + Go version: go1.5.1 + Git commit: a34a1d5-dirty + Built: Sun Nov 22 00:15:15 UTC 2015 + OS/Arch: linux/amd64 + +## Whirlwind tour of docker features + +## Hello World + +Fetch a `docker` image from the `docker` registry: + + $ docker pull busybox + Using default tag: latest + latest: Pulling from library/busybox + cf2616975b4a: Pull complete + 6ce2e90b0bc7: Pull complete + 8c2e06607696: Already exists + library/busybox:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security. + Digest: sha256:38a203e1986cf79639cfb9b2e1d6e773de84002feea2d4eb006b52004ee8502d + Status: Downloaded newer image for busybox:latest + + $ docker images + REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE + busybox latest 8c2e06607696 4 months ago 2.43 MB + +Now, run a command inside the image: -* Docker basics + $ docker run busybox echo "Hello World" + Hello World -- Run a container in detached mode: +## Docker basics - $ docker run -d busybox sh -c \ - 'while true; do echo "hello"; sleep 1; done;' + - Run a container in detached mode: -- Retrieve the container id: + $ docker run -d busybox sh -c \ + 'while true; do echo "hello"; sleep 1; done;' - $ docker ps - CONTAINER ID IMAGE COMMAND CREATED STATUS - 321c1aa5bcd4 busybox "sh -c 'while true; d" 3 seconds ago Up 2 seconds + - Retrieve the container id: -- Attach to the running container: + $ docker ps + CONTAINER ID IMAGE COMMAND CREATED STATUS + 321c1aa5bcd4 busybox "sh -c 'while true; d" 3 seconds ago Up 2 seconds - $ docker attach 321c1aa5bcd4 - hello - hello - [...] + - Attach to the running container: -- Start/stop/restart container - - $ docker stop 321c1aa5bcd4 - $ docker restart 321c1aa5bcd4 + $ docker attach 321c1aa5bcd4 + hello + hello + [...] + - Start/stop/restart container + + $ docker stop 321c1aa5bcd4 + $ docker restart 321c1aa5bcd4 -* Docker: public index (aka registry, aka the Hub) +## Docker: public index (aka registry, aka the Hub) `Docker` containers may be published and shared on a public registry, the `Hub`. -- It is searchable: + - It is searchable: - $ docker search apache2 - NAME STARS OFFICIAL AUTOMATED - rootlogin/apache2-symfony2 7 [OK] - reinblau/php-apache2 6 [OK] - tianon/apache2 4 [OK] - [...] - $ docker pull tianon/apache2 + $ docker search apache2 + NAME STARS OFFICIAL AUTOMATED + rootlogin/apache2-symfony2 7 [OK] + reinblau/php-apache2 6 [OK] + tianon/apache2 4 [OK] + [...] + $ docker pull tianon/apache2 -- Run the image and check the ports + - Run the image and check the ports - $ docker run -d -p 8080:80 tianon/apache2 - $ docker ps - CONTAINER ID IMAGE COMMAND PORTS - 49614161f5b7 tianon/apache2 "apache2 -DFOREGROUND" 0.0.0.0:8080->80/tcp + $ docker run -d -p 8080:80 tianon/apache2 + $ docker ps + CONTAINER ID IMAGE COMMAND PORTS + 49614161f5b7 tianon/apache2 "apache2 -DFOREGROUND" 0.0.0.0:8080->80/tcp The registry is also available from the browser: -- [[https://hub.docker.com][hub.docker.com]] + - [hub.docker.com](https://hub.docker.com) -* Docker: creating a customized image +## Docker: creating a customized image -- run `docker` interactively: + - run `docker` interactively: - $ docker run -it ubuntu bash - root@524ef6c2e4ce:/# apt-get install -y memcached - [...] - root@524ef6c2e4ce:/# exit - - $ docker commit `docker ps -q -l` binet/memcached - 4242210aba21641013b22198c7bdc00435b00850aaf9ae9cedc53ba75794891d + $ docker run -it ubuntu bash + root@524ef6c2e4ce:/# apt-get install -y memcached + [...] + root@524ef6c2e4ce:/# exit - $ docker run -d -p 11211 -u daemon binet/memcached memcached - a84e18168f1473a338f9ea3473dd981bf5e3dc7e41511a1252f7bb216d875860 + $ docker commit `docker ps -q -l` binet/memcached + 4242210aba21641013b22198c7bdc00435b00850aaf9ae9cedc53ba75794891d - $ docker ps - CONTAINER ID IMAGE COMMAND PORTS - a84e18168f14 binet/memcached "memcached" 0.0.0:32768->11211/tcp - -* Docker: creating a customized image + $ docker run -d -p 11211 -u daemon binet/memcached memcached + a84e18168f1473a338f9ea3473dd981bf5e3dc7e41511a1252f7bb216d875860 -- interactive way is fine but not scalable -- enter `Dockerfiles` -- recipes to build an image -- start `FROM` a base image -- `RUN` commands on top of it -- easy to learn, easy to use + $ docker ps + CONTAINER ID IMAGE COMMAND PORTS + a84e18168f14 binet/memcached "memcached" 0.0.0:32768->11211/tcp -* Docker: Dockerfile +## Docker: creating a customized image + + - interactive way is fine but not scalable + - enter `Dockerfiles` + - recipes to build an image + - start `FROM` a base image + - `RUN` commands on top of it + - easy to learn, easy to use + +## Docker: Dockerfile .code _code/dockerfile-nginx -* Docker: Dockerfile-II +## Docker: Dockerfile-II -- run in the directory holding that `Dockerfile` + - run in the directory holding that `Dockerfile` - $ docker build -t /server . - $ docker run -d -P /server + $ docker build -t /server . + $ docker run -d -P /server -- retrieve the port number: + - retrieve the port number: - $ docker ps - 34dc03cdbae8 binet/server "/bin/sh -c 'nginx -g" 0.0.0.0:32770->80/tcp + $ docker ps + 34dc03cdbae8 binet/server "/bin/sh -c 'nginx -g" 0.0.0.0:32770->80/tcp or: - $ docker inspect -f '{{.NetworkSettings.Ports}}' 34dc03cdbae8 + $ docker inspect -f '{{.NetworkSettings.Ports}}' 34dc03cdbae8 and then: - $ curl localhost:32770 - Hi, I am in your container! + $ curl localhost:32770 + Hi, I am in your container! -* Docker: Dockerfile-III +## Docker: Dockerfile-III -*NOTE:* for Windows(TM) and MacOSX(TM) users, a thin `Linux` VM is sitting +**NOTE:** for Windows(TM) and MacOSX(TM) users, a thin `Linux` VM is sitting between your machine and the container. The container is running inside that VM so you need to replace `localhost` with the `IP` of that VM: - $ docker-machine ip default - 192.168.59.103 + $ docker-machine ip default + 192.168.59.103 and then: - $ curl 192.168.59.103:32770 - Hi, I am in your container! - -* docker build + $ curl 192.168.59.103:32770 + Hi, I am in your container! + +## docker build -- takes a snapshot after each step -- re-uses those snapshots in future builds -- doesn't re-run slow steps when it isn't necessary (cache system) + - takes a snapshot after each step + - re-uses those snapshots in future builds + - doesn't re-run slow steps when it isn't necessary (cache system) .image _figs/docker-changes-updates.png 350 800 -* Docker Hub +## Docker Hub -- `docker` `push` an image to the Hub -- `docker` `pull` an image from the Hub to any machine + - `docker` `push` an image to the Hub + - `docker` `pull` an image from the Hub to any machine This brings: -- reliable deployment -- consistency + - reliable deployment + - consistency -- images are self-contained, independent from host -- if it works locally, it will work on the server -- _exact_ _same_ _behavior_ -- regardless of versions, distros and dependencies + - images are self-contained, independent from host + - if it works locally, it will work on the server + - _exact_ _same_ _behavior_ + - regardless of versions, distros and dependencies -* Docker for the developer +## Docker for the developer -- manage and *control* dependencies -- if it works on my machine, it works on the cluster -- reproducibility -- small but durable recipes + - manage and **control** dependencies + - if it works on my machine, it works on the cluster + - reproducibility + - small but durable recipes Never again: -- juggle with 3 different incompatible FORTRAN compilers -- voodoo incantations to get that exotic library to link with IDL -- figure out which version of LAPACK works with that code -- ... and what obscure flag coaxed it into compiling last time + - juggle with 3 different incompatible FORTRAN compilers + - voodoo incantations to get that exotic library to link with IDL + - figure out which version of LAPACK works with that code + - ... and what obscure flag coaxed it into compiling last time -* Development workflow +## Development workflow -- Fetch code (`git`, `mercurial`, ...) + - Fetch code (`git`, `mercurial`, ...) - $ git clone git@github.com:sbinet/loops-20151217-tp - $ cd loops-20151217-tp + $ git clone git@github.com:sbinet/loops-20151217-tp + $ cd loops-20151217-tp -- Edit code -- Mount code inside a `build` container -- Build+test inside that container + - Edit code + - Mount code inside a `build` container + - Build+test inside that container We'll test this workflow in the remainder of the hands-on session... -* Deployment workflow +## Deployment workflow -- Pull binary (possibly from a private registry), packaged as a container -- Deploy container on cloud or local cluster + - Pull binary (possibly from a private registry), packaged as a container + - Deploy container on cloud or local cluster Many tools exist to ease the deployment of multi-containers applications _eg_: an application consisting of a web-server and a database, where each @@ -413,16 +404,15 @@ Configuration(s) also need to be provided to these containers... .link https://coreos.com/etcd/docs/latest/ .link https://docs.docker.com/compose/ +## Conclusions -* Conclusions + - `docker` is a rather good tool to deploy applications in containers + - eases the life of developers and sysadmins (devops) + - `docker` isn't the only game in town + - [rkt](https://coreos.com/rkt/docs) (`rocket`) from `CoreOS` + - [systemd-nspawn](http://0pointer.de/public/systemd-man/systemd-nspawn.html), now part of `systemd` -- `docker` is a rather good tool to deploy applications in containers -- eases the life of developers and sysadmins (devops) -- `docker` isn't the only game in town -- [[https://coreos.com/rkt/docs][rkt]] (`rocket`) from `CoreOS` -- [[http://0pointer.de/public/systemd-man/systemd-nspawn.html][systemd-nspawn]], now part of `systemd` - -* References +## References .link http://www.slideshare.net/jpetazzo/introduction-to-docker-december-2014-tour-de-france-bordeaux-special-edition .link http://www.slideshare.net/dotCloud/docker-intro-november @@ -433,93 +423,89 @@ Configuration(s) also need to be provided to these containers... .link http://mesos.apache.org/ .link https://coreos.com/rkt/docs -* Hands-on session +## Hands-on session -* Development workflow +## Development workflow -- Fetch the code from `github`: + - Fetch the code from `github`: - $ git clone git://github.com/sbinet/loops-20151217-tp - $ cd loops-20151217-tp + $ git clone git://github.com/sbinet/loops-20151217-tp + $ cd loops-20151217-tp We'll see _a_ use of `docker` as part of a development workflow: - -- Edit code -- Mount code inside a `build` container -- Build+test inside that container -We'll test this workflow in the remainder of the hands-on session... + - Edit code + - Mount code inside a `build` container + - Build+test inside that container +We'll test this workflow in the remainder of the hands-on session... -* Create a base container - -- create a directory `docker-web-base` to hold the `Dockerfile` for the base container -- create the `Dockerfile` and choose your favorite `Linux` distro (say, `debian`) as a base image, -- install the needed dependencies for the web-app (`go` and `pkg-config`) -- run: +## Create a base container - $ docker build -t /web-base . + - create a directory `docker-web-base` to hold the `Dockerfile` for the base container + - create the `Dockerfile` and choose your favorite `Linux` distro (say, `debian`) as a base image, + - install the needed dependencies for the web-app (`go` and `pkg-config`) + - run: + $ docker build -t /web-base . _Hint:_ The image `golang`, an official image from Docker Inc., is based on `debian` and has `go` installed... - -* Create a base container - solution +## Create a base container - solution (see next slide) -* Create a base container - II +## Create a base container - II .code _code/base-dockerfile -* Base container for development +## Base container for development + + - One could create a new container with all the development tools (editor, completion, ...) + - But you'd need to carry over the configuration (`ssh` keys, editor, ...) -- One could create a new container with all the development tools (editor, completion, ...) -- But you'd need to carry over the configuration (`ssh` keys, editor, ...) +Probably easier to just mount the sources **inside** the base container: -Probably easier to just mount the sources *inside* the base container: + $ docker run -it -v `pwd`:/go/src/github.com/sbinet/loops-20151217-tp \ + -p 8080:8080 /web-base bash + [root@48b2c74a5004 /go]# go run ./src/github.com/sbinet/loops-20151217-tp/web-app/main.go + 2015/12/16 14:13:13 listening on: http://localhost:8080 - $ docker run -it -v `pwd`:/go/src/github.com/sbinet/loops-20151217-tp \ - -p 8080:8080 /web-base bash - [root@48b2c74a5004 /go]# go run ./src/github.com/sbinet/loops-20151217-tp/web-app/main.go - 2015/12/16 14:13:13 listening on: http://localhost:8080 + - In another terminal: - $ curl localhost:8080 - hello LoOPS 20151217! - - - --- running external command... - - >>> pkg-config --cflags python2 - Package python2 was not found in the pkg-config search path. - Perhaps you should add the directory containing `python2.pc' - to the PKG_CONFIG_PATH environment variable - No package 'python2' found - error: exit status 1 + $ curl localhost:8080 + hello LoOPS 20151217! -* Base container for dev - II + --- running external command... -- On windows, the correct `-v` syntax is like: + >>> pkg-config --cflags python2 + Package python2 was not found in the pkg-config search path. + Perhaps you should add the directory containing `python2.pc' + to the PKG_CONFIG_PATH environment variable + No package 'python2' found + error: exit status 1 - $ docker run -it -v //c/Users/username/some/path:/go/src/ ... +## Base container for dev - II -.link https://github.com/docker/docker/issues/12590#issuecomment-96767796 + - On windows, the correct `-v` syntax is like: + + $ docker run -it -v //c/Users/username/some/path:/go/src/ ... +.link https://github.com/docker/docker/issues/12590#issuecomment-96767796 -* Create the final container +## Create the final container Now that we know the base image "works", we'll automatize the build part as yet another `Dockerfile`: -- create a new `Dockerfile` file (at the root of the `git` repository) based on the `web-base` image, with the correct build+run instructions -- make sure you can `docker` `build` it and tag it as `web-app` -- make sure that you can still access the web server when you run: + - create a new `Dockerfile` file (at the root of the `git` repository) based on the `web-base` image, with the correct build+run instructions + - make sure you can `docker` `build` it and tag it as `web-app` + - make sure that you can still access the web server when you run: - $ docker run -d -p 8080:8080 /web-app + $ docker run -d -p 8080:8080 /web-app _Hint:_ `ADD` @@ -527,54 +513,54 @@ _Hint:_ `CMD` .link https://docs.docker.com/reference/builder/ -* Create the final container - solutions +## Create the final container - solutions (see next slide) -* Create the final container - II +## Create the final container - II .code _code/webapp-dockerfile -* Create the final container - III +## Create the final container - III -- `CMD` describes the command to be run by default when the container is started -- `ADD` copies files, directories or URLs into the container's filesystem -- `VOLUME` creates a volume mount point inside the container which can contain data from the host or from other containers -- `USER` defines the user (or `UID`) with whom to run the various commands inside the container + - `CMD` describes the command to be run by default when the container is started + - `ADD` copies files, directories or URLs into the container's filesystem + - `VOLUME` creates a volume mount point inside the container which can contain data from the host or from other containers + - `USER` defines the user (or `UID`) with whom to run the various commands inside the container -* Create multiple versions of an image +## Create multiple versions of an image At times, it might be very useful to test 2 versions of an application and run them concurrently (to debug discrepancies.) Let's do just that. -- tag the last `web-app` image as `v1` + - tag the last `web-app` image as `v1` - $ docker tag \ - /web-app \ - /web-app:v1 + $ docker tag \ + /web-app \ + /web-app:v1 -- modify `sbinet/loops-20151217-tp/web-app/main.go` to print a different welcome message -- containerize the new version as `.../web-app:v2` + - modify `sbinet/loops-20151217-tp/web-app/main.go` to print a different welcome message + - containerize the new version as `.../web-app:v2` -* Create multiple versions of an image - II +## Create multiple versions of an image - II -- run the container `v2` on port `8082` + - run the container `v2` on port `8082` - $ docker run -p 8082:8080 \ - --name=web-app-v2 \ - /web-app:v2 + $ docker run -p 8082:8080 \ + --name=web-app-v2 \ + /web-app:v2 -- run the container `v1` on port `8080` + - run the container `v1` on port `8080` - $ docker run -p 8080:8080 \ - --name=web-app-v1 \ - /web-app:v1 + $ docker run -p 8080:8080 \ + --name=web-app-v1 \ + /web-app:v1 -- make sure the servers on ports `8080` and `8082` display the correct welcome messages. + - make sure the servers on ports `8080` and `8082` display the correct welcome messages. -* Sharing images +## Sharing images Up to now, the images you've been creating have been put on your local disk. But there is this public registry instance available at: @@ -584,40 +570,40 @@ But there is this public registry instance available at: Let's try to package the previous `web-app:v2` and `web-app:v1` images and put them on that registry: - $ docker push /web-app:v1 - $ docker push /web-app:v2 + $ docker push /web-app:v1 + $ docker push /web-app:v2 Now, try to `pull` the `web-app` image of your friend and run it. -* Inspecting logs +## Inspecting logs `docker` is nice enough to let us inspect what (running) containers are generating as logs. For a single container, it is as simple as: - $ docker logs - $ docker logs + $ docker logs + $ docker logs -- inspect the logs of your `web-app-v2` container + - inspect the logs of your `web-app-v2` container _e.g.:_ - $ docker logs web-app-v2 - 2015/12/16 14:53:56 listening on: http://localhost:8080 + $ docker logs web-app-v2 + 2015/12/16 14:53:56 listening on: http://localhost:8080 -* Inspecting logs - II +## Inspecting logs - II -- launch a container in interactive mode -- start a bash shell -- run inside that container: + - launch a container in interactive mode + - start a bash shell + - run inside that container: - docker> logger -i -s plop + docker> logger -i -s plop -- in another terminal: + - in another terminal: - $ docker logs + $ docker logs -* Creation of a build+target container pair +## Creation of a build+target container pair So far, we have been building containers where the intermediate results leading to the final binary (or set of binaries) are left inside the image. @@ -627,8 +613,8 @@ resource heavy. The usual solution is to have a 2-step process: -- a container in which the binaries are built -- a container in which the binaries are directly copied from the first + - a container in which the binaries are built + - a container in which the binaries are directly copied from the first Let's do that. @@ -636,51 +622,51 @@ _Hint:_ `docker` `export` _Hint:_ `docker` `import` _Hint:_ `docker` `cp` -* Creation of a build+target container pair +## Creation of a build+target container pair (solution on next slide) -* Creation of a build+target container pair +## Creation of a build+target container pair Extract the root fs from the build image: - $ mkdir rootfs && cd rootfs - $ docker run -d -p 8080:8080 --name=web-app-v2 \ - /web-app:v2 - $ docker export web-app-v2 | tar xf - - $ ls go/src/github.com/sbinet/loops-20151217-tp/ - Dockerfile docker-web-base hello README.md web-app - $ ls go/bin - web-app + $ mkdir rootfs && cd rootfs + $ docker run -d -p 8080:8080 --name=web-app-v2 \ + /web-app:v2 + $ docker export web-app-v2 | tar xf - + $ ls go/src/github.com/sbinet/loops-20151217-tp/ + Dockerfile docker-web-base hello README.md web-app + $ ls go/bin + web-app Another way is to use `docker` `cp`: - $ docker cp web-app-v2:/go/bin/web-app web-app - + $ docker cp web-app-v2:/go/bin/web-app web-app + The binaries are under `/go/bin`. As `go` usually creates static libraries, you can just create a very slim container with them, using `docker` `import`. -* Creation of a build+target container pair +## Creation of a build+target container pair - $ mkdir rootfs && cd rootfs - $ docker cp web-app-v2:/go/bin/web-app web-app - $ cat > Dockerfile << EOF - FROM ubuntu - ADD web-app /usr/bin/web-app - CMD web-app - EOF - $ docker build -t slim-web-app . + $ mkdir rootfs && cd rootfs + $ docker cp web-app-v2:/go/bin/web-app web-app + $ cat > Dockerfile << EOF + FROM ubuntu + ADD web-app /usr/bin/web-app + CMD web-app + EOF + $ docker build -t slim-web-app . Compare the size: - $ docker images - ubuntu 14.04 187.9 MB - slim-web-app latest 196.6 MB - binet/web-app latest 749.3 MB - golang 1.5.2 703.8 MB + $ docker images + ubuntu 14.04 187.9 MB + slim-web-app latest 196.6 MB + binet/web-app latest 749.3 MB + golang 1.5.2 703.8 MB -* Running GUIs +## Running GUIs The application we have been currently "dockerizing" doesn't need any graphics per se. @@ -689,57 +675,54 @@ Many do, though. Let's try to run a simple graphics-enabled application from within a `docker` container: - $ docker run -it --rm ubuntu bash - docker> apt-get update -y && apt-get install -y x11-apps - docker> xclock + $ docker run -it --rm ubuntu bash + docker> apt-get update -y && apt-get install -y x11-apps + docker> xclock - -* Running GUIs - II +## Running GUIs - II Running GUIs is a bit more involved than just running your simple "from the mill" CLI application. There are many options to enable graphics: -- `ssh` into a container with `X11` forwarding -- `VNC` -- sharing the `X11` socket - + - `ssh` into a container with `X11` forwarding + - `VNC` + - sharing the `X11` socket .link http://fabiorehm.com/blog/2014/09/11/running-gui-apps-with-docker/ .link https://blog.docker.com/2013/07/docker-desktop-your-desktop-over-ssh-running-inside-of-a-docker-container/ .link http://wiki.ros.org/docker/Tutorials/GUI -* Running GUIs - III +## Running GUIs - III Let's try the most direct (albeit a bit insecure) one: sharing the `X11` socket. First, allow all `X11` connections (that's the insecure part): - $ xhost + + $ xhost + Then: - $ docker run -ti --rm \ - -e DISPLAY=$DISPLAY \ - -v /tmp/.X11-unix:/tmp/.X11-unix \ - ubuntu bash - docker> apt-get update -y && apt-get install -y xclock && xclock + $ docker run -ti --rm \ + -e DISPLAY=$DISPLAY \ + -v /tmp/.X11-unix:/tmp/.X11-unix \ + ubuntu bash + docker> apt-get update -y && apt-get install -y xclock && xclock Don't forget to re-enable `X11` access control afterwards: - $ xhost - + $ xhost - +## Conclusions -* Conclusions + - `docker` is a rather good tool to deploy applications in containers + - eases the life of developers and sysadmins (devops) + - `docker` isn't the only game in town + - [rkt](https://coreos.com/rkt/docs) (`rocket`) from `CoreOS` + - [systemd-nspawn](http://0pointer.de/public/systemd-man/systemd-nspawn.html), now part of `systemd` -- `docker` is a rather good tool to deploy applications in containers -- eases the life of developers and sysadmins (devops) -- `docker` isn't the only game in town -- [[https://coreos.com/rkt/docs][rkt]] (`rocket`) from `CoreOS` -- [[http://0pointer.de/public/systemd-man/systemd-nspawn.html][systemd-nspawn]], now part of `systemd` - -* References +## References .link http://www.slideshare.net/jpetazzo/introduction-to-docker-december-2014-tour-de-france-bordeaux-special-edition .link http://www.slideshare.net/dotCloud/docker-intro-november @@ -749,4 +732,3 @@ Don't forget to re-enable `X11` access control afterwards: .link http://kubernetes.io/ .link http://mesos.apache.org/ .link https://coreos.com/rkt/docs - diff --git a/2016/20160125-alice-tbb/alice-tbb.slide b/2016/20160125-alice-tbb/alice-tbb.slide index a713555..ab40e6b 100644 --- a/2016/20160125-alice-tbb/alice-tbb.slide +++ b/2016/20160125-alice-tbb/alice-tbb.slide @@ -1,4 +1,4 @@ -Introduction to Thread Building Blocks +# Introduction to Thread Building Blocks MRRTF, 2016-01-25 @@ -6,138 +6,137 @@ Sebastien Binet CNRS/IN2P3/LPC binet@cern.ch -* Recap from last episode +## Recap from last episode -* C++11 std::thread +## C++11 std::thread `C++11` (finally!) standardized threads -- `std::thread` is now part of the standard `C++` library -- `std::thread` is an abstraction and maps to local platform threads (`POSIX`, `Windows(TM)`, ...) + - `std::thread` is now part of the standard `C++` library + - `std::thread` is an abstraction and maps to local platform threads (`POSIX`, `Windows(TM)`, ...) -* C++11 std::thread +## C++11 std::thread .code code/hello-par.cxx - $> g++ -o hello-par --std=c++11 -pthread hello-par.cxx - $> ./hello-par - ** inside thread 139800850654976! - -* Programming style + $> g++ -o hello-par --std=c++11 -pthread hello-par.cxx + $> ./hello-par + ** inside thread 139800850654976! -- Old school: thread functions (what we just saw) -- Middle school: function objects (functors) -- New school: `C++11` lambda functions (aka anonymous functions) +## Programming style -* Types of parallelism + - Old school: thread functions (what we just saw) + - Middle school: function objects (functors) + - New school: `C++11` lambda functions (aka anonymous functions) + +## Types of parallelism Most common types: -- Data -- Task -- Embarrassingly parallel -- Dataflow + - Data + - Task + - Embarrassingly parallel + - Dataflow -* C++ concurrency features +## C++ concurrency features .image figs/cxx-features.png -* Futures +## Futures -*Futures* provide a higher level of abstraction +**Futures** provide a higher level of abstraction -- you start an asynchronous/parallel operation -- you are returned a handle to wait for the result -- thread creation, join and exceptions are handled for you + - you start an asynchronous/parallel operation + - you are returned a handle to wait for the result + - thread creation, join and exceptions are handled for you -* Async tasks and futures +## Async tasks and futures .code code/async.cxx -- access the result via the `get()` method -- one can also use `wait()` to block the thread until the result is available (but doesn't read the result) + - access the result via the `get()` method + - one can also use `wait()` to block the thread until the result is available (but doesn't read the result) -* Intel Threading Building Blocks +## Intel Threading Building Blocks -* Intel TBB - Introduction +## Intel TBB - Introduction Open source project (GPLv2) for supporting scalable parallel programming `C++`. -It is a *higher* *level* toolkit than `C++11` threads. +It is a **higher** **level** toolkit than `C++11` threads. "If it's suitable to your problem, then it's almost certainly a better choice." `TBB` supports common programming patterns for loops: -- `parallel_for` for independent computations on arrays -- `parallel_reduce` for a series of computations across an array -- `parallel_scan` for more general prefix calculations. + - `parallel_for` for independent computations on arrays + - `parallel_reduce` for a series of computations across an array + - `parallel_scan` for more general prefix calculations. -See the [[https://software.intel.com/en-us/node/506140][documentation]] for a complete list. +See the [documentation](https://software.intel.com/en-us/node/506140) for a complete list. `TBB` also provides thread-safe containers, a performant thread safe memory allocator and timing primitives. -* Intel TBB - Introduction (cont'd) - +## Intel TBB - Introduction (cont'd) -`TBB` also allows the construction of *graphs* describing the relationship between different parts of a program's execution and the `TBB` *scheduler* can exploit concurrency where different parts of the workflow are independent. +`TBB` also allows the construction of **graphs** describing the relationship between different parts of a program's execution and the `TBB` **scheduler** can exploit concurrency where different parts of the workflow are independent. -Further, for *task* based workflows, `TBB` allows lower level interaction with its task scheduler, enabling it to be used as an execution engine for a higher level workflow generator. +Further, for **task** based workflows, `TBB` allows lower level interaction with its task scheduler, enabling it to be used as an execution engine for a higher level workflow generator. -*Using* *TBB*. +**Using** **TBB**. -- include the header `"tbb/tbb.h"` -- link against `-ltbb` (and, on `Linux`, against `-lrt`) -- `TBB` identifiers are under the `tbb::` namespace. + - include the header `"tbb/tbb.h"` + - link against `-ltbb` (and, on `Linux`, against `-lrt`) + - `TBB` identifiers are under the `tbb::` namespace. -* Parallel loop algorithms +## Parallel loop algorithms -* Parallel for +## Parallel for Simple parallel operation where the same operation is performed independently over elements of a collection. Serially: - for (size_t i=0; i& r) const { - double *x = my_x; - for(size_t i=r.begin(); i!=r.end(); ++i) - my_func(x[i]); - } - ApplyFunc(double x[]): - my_x{x} - {} - }; +## Parallel for - II + + #include "tbb/tbb.h" + + class ApplyFunc { + double *const my_x; + public: + void operator()(const tbb::blocked_range& r) const { + double *x = my_x; + for(size_t i=r.begin(); i!=r.end(); ++i) + my_func(x[i]); + } + ApplyFunc(double x[]): + my_x{x} + {} + }; The `tbb::blocked_range` parameter is used by `TBB` to instruct threads to operate on a certain chunk of the `my_x` array. -* Parallel for - III +## Parallel for - III With `ApplyFunc` defined, one can now invoke `tbb::parallel_for` like so: - #include "tbb/tbb.h" - - void ParallelApplyFunc(double x[], size_t n) { - tbb::parallel_for(tbb::blocked_range(0, n), ApplyFunc(x)); - } + #include "tbb/tbb.h" + + void ParallelApplyFunc(double x[], size_t n) { + tbb::parallel_for(tbb::blocked_range(0, n), ApplyFunc(x)); + } In this case `parallel_for` is instructed to run over the range `[0,n)` on the `ApplyFunc(x)` class instance. Note that `parallel_for` will take care of setting up the `TBB` thread pool for us (in earlier versions of `TBB`, one needed to call `tbb::task_scheduler_init`, which is still available if you need to setup the `TBB` thread pool with special options). -* Parallel for - IV +## Parallel for - IV As for the `std::thread` case, one can also use a lambda function instead of a functor: @@ -145,36 +144,36 @@ As for the `std::thread` case, one can also use a lambda function instead of a f Using the lambda which copies by value `[=]` satisfies all the criteria needed by `parallel_for` and makes for a very succinct declaration. -* Parallel reduce +## Parallel reduce -`parallel_for` is great for when operations applied to elements of a collection are *independent*. +`parallel_for` is great for when operations applied to elements of a collection are **independent**. But, there are cases where the operation needs input from the previous steps. -If the computation can still be broken down into pieces, we can still parallelize this operation, which is called a *reduction*. +If the computation can still be broken down into pieces, we can still parallelize this operation, which is called a **reduction**. Here's a simple example: if one needs to sum up 10 numbers one can do it like this - (((((((((1+2)+3)+4)+5)+6)+7)+8)+9)+10) + (((((((((1+2)+3)+4)+5)+6)+7)+8)+9)+10) which is sequential. But one could also do it like this: - ((1+2) + (3+4)) + ((5+6) + ((7+8) + (9+10))) + ((1+2) + (3+4)) + ((5+6) + ((7+8) + (9+10))) where it's now much easier to see that the calculation can be parallelized. There are some extra steps needed here: -- how to *split* the whole collection into smaller chunks -- how to *join* the results of the smaller chunks into aggregated results. + - how to **split** the whole collection into smaller chunks + - how to **join** the results of the smaller chunks into aggregated results. -* Parallel reduce - II +## Parallel reduce - II .code code/tbb-parallel-reduce.cxx -* Parallel reduce - III +## Parallel reduce - III Notice the two new methods that were needed -- A special constructor that takes a reference to an existing `parallel_sum` instance and a special dummy parameter `tbb::split`. This is the *splitting* *constructor* that `TBB` uses to generate new parts of the *reduction* that will run on other threads. (The dummy variable distinguishes this method from the class's copy constructor.) -- A `join` method that tells `TBB` how to *combine* the results from one fragment of the reduction with another (in this case, just adding up the partial sums). + - A special constructor that takes a reference to an existing `parallel_sum` instance and a special dummy parameter `tbb::split`. This is the **splitting** **constructor** that `TBB` uses to generate new parts of the **reduction** that will run on other threads. (The dummy variable distinguishes this method from the class's copy constructor.) + - A `join` method that tells `TBB` how to **combine** the results from one fragment of the reduction with another (in this case, just adding up the partial sums). Once the class is ready, we invoke the reduction by calling `parallel_reduce`: @@ -182,7 +181,7 @@ Once the class is ready, we invoke the reduction by calling `parallel_reduce`: Note that because of the extra methods that are needed for `parallel_reduce` it's less easy to use a lambda here. -* Different blocked range templates +## Different blocked range templates `blocked_range` is designed for `1D` collections. However, many times we need to deal with more complicated objects than that. @@ -194,11 +193,11 @@ So a fragment of a matrix multiply would be something like: .code code/tbb-matrix-mult.cxx -* Containers +## Containers `STL` containers are not thread safe. Reading them in parallel is fine. -But as soon as one thread modifies a container while other threads access it: *corruption*. +But as soon as one thread modifies a container while other threads access it: **corruption**. `STL` containers need to be guarded with a `std::mutex` for concurrent r/w access. `STL` containers thus are usually bottlenecks in multithreaded applications. @@ -206,36 +205,36 @@ But as soon as one thread modifies a container while other threads access it: *c `TBB` provides concurrency friendly containers with most of the features one can expect of their `STL` counterparts. There are two strategies to make such containers: -- Use lock-free techniques, which correct for any interference between threads automatically. -- Use fine grained locking, where only the part of the container which is being touched is wrapped in a mutex. + - Use lock-free techniques, which correct for any interference between threads automatically. + - Use fine grained locking, where only the part of the container which is being touched is wrapped in a mutex. -* Concurrent vector +## Concurrent vector `concurrent_vector`: -- allows dynamic growth of an array of type `T`. + - allows dynamic growth of an array of type `T`. -- can be grown safely while other threads are accessing it, including growing it themselves. + - can be grown safely while other threads are accessing it, including growing it themselves. .code code/vec-add.cxx -- clearing or destroying the vector and accessing it is *unsafe*. -- `size()` returns the size of all elements, even those being constructed, so accessing those elements might be problematic. -- unlike an `std::vector`, elements in a concurrent vector are not guaranteed to be contiguous in memory. + - clearing or destroying the vector and accessing it is **unsafe**. + - `size()` returns the size of all elements, even those being constructed, so accessing those elements might be problematic. + - unlike an `std::vector`, elements in a concurrent vector are not guaranteed to be contiguous in memory. -* Other containers +## Other containers `TBB` provides many other concurrent containers, including hash maps and queues. We won't cover them here, but having practiced with the concurrent vector you should find them pretty easy to use. -- `tbb::concurrent_unordered_map` -- `tbb::concurrent_unordered_set` -- `tbb::concurrent_hash_map` -- `tbb::concurrent_queue` + - `tbb::concurrent_unordered_map` + - `tbb::concurrent_unordered_set` + - `tbb::concurrent_hash_map` + - `tbb::concurrent_queue` -* Pipelines and graphs +## Pipelines and graphs -Many real-world problems can be addressed with `TBB` `parallel_for`, `parallel_reduce` and more [[https://software.intel.com/en-us/node/506140][sophisticated patterns]]. -But `TBB` also allows for parallelization based on *task* *workflows*. +Many real-world problems can be addressed with `TBB` `parallel_for`, `parallel_reduce` and more [sophisticated patterns](https://software.intel.com/en-us/node/506140). +But `TBB` also allows for parallelization based on **task** **workflows**. _Pipelines_ are a simple execution concept. - Tasks to be executed come in a linear sequence, much like an assembly line. @@ -243,40 +242,39 @@ _Pipelines_ are a simple execution concept. - Serial process for any data element. - Parallelism arises because we can have many pieces of data moving through the pipeline at the one time. -* Pipelines - - +---------+ - | Start | <- data1, data2, data3, ..., dataN - +---------+ - | - V - +---------+ - | Stage 1 | - +---------+ - | - V - +---------+ - | Stage 2 | - +---------+ - | - V - +---------+ - | Stop | -> outN, ... , out3, out2, out1 - +---------+ - -* Pipelines - -- Each stage can either be serial or parallel. -- No more than one element will be processed by serial stages at a time. -- Serial stages can be in order or out of order. - -- Multiple elements can be processed by parallel stages at the same time -- Ordering is obviously not guaranteed. +## Pipelines + + +---------+ + | Start | <- data1, data2, data3, ..., dataN + +---------+ + | + V + +---------+ + | Stage 1 | + +---------+ + | + V + +---------+ + | Stage 2 | + +---------+ + | + V + +---------+ + | Stop | -> outN, ... , out3, out2, out1 + +---------+ + +## Pipelines + + - Each stage can either be serial or parallel. + - No more than one element will be processed by serial stages at a time. + - Serial stages can be in order or out of order. + + - Multiple elements can be processed by parallel stages at the same time + - Ordering is obviously not guaranteed. Considering the performance one might hope to get with a pipeline, obviously the serial stages are bottlenecks (especially serial in order), so keeping these stages short will help a lot. - -* Defining a Pipeline +## Defining a Pipeline Pipelines are defined as a series of _filters_, each of which takes some input data and produces some transformed output data. The first filter's input data is `void`, as is the last filter's output data. @@ -285,7 +283,7 @@ The first filter's input data is `void`, as is the last filter's output data. `ntoken` prevents data from piling up at the choke point of the pipeline and consuming resources needlessly. -* How a Pipeline runs +## How a Pipeline runs `TBB` actually runs the pipeline by calling the `()` `operator` of each filter for the data element that is going to be processed. @@ -295,7 +293,7 @@ e.g., our `Transform` class might look like this: Note that as the `()` `operator` might be called on a copy of the original class instance it needs to be `const` to ensure it does not make changes to the body which would be lost in the copy. -* How a Pipeline runs - II +## How a Pipeline runs - II Similarly, the `DataWriter` could look like: @@ -304,52 +302,52 @@ Similarly, the `DataWriter` could look like: The first stage is a bit more complicated as it needs a mechanism to tell the pipeline there is no more data to process. This is what the special `tbb::flow_control` class is for. -* How a Pipeline runs - III +## How a Pipeline runs - III .code code/pipeline-datareader.cxx -* TBB Graphs +## TBB Graphs The most general kind of task based workflow is a graph, where tasks are nodes and edges are data that flows between the nodes. - +---------+ - | Start | <- data1, data2, data3, ..., dataN - +---------+-------------- - | \ \ - V \ \ - +---------+ \| \ - | Node 1 | +--------+ +--------+ - +---------+ | Node 2 | | Node 4 | - | +--------+ +--------+ - V | | - +---------+ | | - | Node 3 | | | - +---------+ | | - | | | - | +----------+-------------+ - | | - V V - +---------+ - | Node 5 | -> out1, out2, out3, ..., outN - +---------+ - -* TBB Graphs - II + +---------+ + | Start | <- data1, data2, data3, ..., dataN + +---------+-------------- + | \ \ + V \ \ + +---------+ \| \ + | Node 1 | +--------+ +--------+ + +---------+ | Node 2 | | Node 4 | + | +--------+ +--------+ + V | | + +---------+ | | + | Node 3 | | | + +---------+ | | + | | | + | +----------+-------------+ + | | + V V + +---------+ + | Node 5 | -> out1, out2, out3, ..., outN + +---------+ + +## TBB Graphs - II Graphs are a very flexible way to execute tasks in parallel. `TBB` has many node types that allow for processing, splitting, joining, queueing etc. Note also that although the ASCII art example there is a `DAG` (Directed Acyclic Graph), `TBB` can implement graphs with cycles as well. -The Intel [[https://software.intel.com/en-us/node/517340][documentation]] has a good introduction. +The Intel [documentation](https://software.intel.com/en-us/node/517340) has a good introduction. -* TBB Graph Basics +## TBB Graph Basics -* Setting Up a Graph +## Setting Up a Graph A graph in `TBB` is an object of type `tbb::flow::graph`, and you need to include the header `tbb/flow_graph.h`: .code code/tbb-graph.cxx -* Attaching nodes to your graph +## Attaching nodes to your graph In `TBB`, nodes may have different types. A simple case is to use the helper `function_node`: @@ -358,80 +356,80 @@ A simple case is to use the helper `function_node`: This creates a basic data processing node that: -- ingests a certain type of data and then -- produces an output data type. + - ingests a certain type of data and then + - produces an output data type. _Note:_ in `TBB` speak, nodes exchange _messages_, but these messages can be meaningful pieces of data. -* Making Edges +## Making Edges Once nodes have been connected, one needs to connect them _via_ edges so the graph can do useful work. This is done with the `tbb::flow::make_edge` function. It takes an input and an output node and connects them together: - tbb::flow::make_edge(n1, n2); + tbb::flow::make_edge(n1, n2); This connects node `n1` to `n2`. -* Getting The Graph Started +## Getting The Graph Started You can push data into the graph by using the `try_put` method of a node. This pushes a piece of data into the node and will then trigger all of the associated data flows. - n1.try_put(1.0); + n1.try_put(1.0); Note that `try_put` returns a boolean to say if the node accepted the data. -* At the End +## At the End Before the graph object goes out of scope, one needs to wait for all data processing to finish by calling the `wait_for_all()` method: - g.wait_for_all() + g.wait_for_all() -* Simple example +## Simple example .code code/tbb-graph-example.cxx / START/,/ END/ -* Simple example - II - - $> ./tbb-graph-example - node n 0 - node n 1 - node n 2 - node n 3 - node m 0 - node n 4 - node n 9 - node n 5 - node m 1 - node n 6 - node m 4 - node n 7 - node n 8 - node m 9 - node m 16 - node m 81 - node m 25 - node m 36 - node m 49 - node m 64 - -* Simple example - III - -- all messages passed between nodes are *copied* -- if large pieces of data need to be percolated through nodes, use `std::shared_ptr` (`std::unique_ptr` is not copyable) - - tbb::flow::function_node< shared_ptr, shared_ptr > - generator( g, tbb::flow::unlimited, [](const shared_ptr v) { // note the const - shared_ptr v2 = make_shared (some_function(*v)); - return v2; - } ); - -- if no data needs to be passed down, one can use the lightweight message `tbb::flow::continue_msg()`. - -* Other node types - -* Broadcast nodes +## Simple example - II + + $> ./tbb-graph-example + node n 0 + node n 1 + node n 2 + node n 3 + node m 0 + node n 4 + node n 9 + node n 5 + node m 1 + node n 6 + node m 4 + node n 7 + node n 8 + node m 9 + node m 16 + node m 81 + node m 25 + node m 36 + node m 49 + node m 64 + +## Simple example - III + + - all messages passed between nodes are **copied** + - if large pieces of data need to be percolated through nodes, use `std::shared_ptr` (`std::unique_ptr` is not copyable) + + tbb::flow::function_node< shared_ptr, shared_ptr > + generator( g, tbb::flow::unlimited, [](const shared_ptr v) { // note the const + shared_ptr v2 = make_shared (some_function(*v)); + return v2; + } ); + + - if no data needs to be passed down, one can use the lightweight message `tbb::flow::continue_msg()`. + +## Other node types + +## Broadcast nodes A `broadcast` node sends its input to any output node to which it's connected. It doesn't do any processing. @@ -439,36 +437,36 @@ It's a bit like a plumbing `T` pipe. .code code/tbb-graph-nodes.cxx -* Source Node +## Source Node A source node takes no input, but generates output internally, passing it out to its connected nodes. A source node needs to provide a callable that accepts a reference to its data type and sets the value of the reference to the data to be passed to other nodes. The call interface itself returns a boolean: `true` if more data might be available, `false` if not. -* Source Node - II +## Source Node - II This source node provides the lines from a file as messages, one by one: .code code/tbb-graph-src-node.cxx -* Combining parallelism +## Combining parallelism One of the nice features of `TBB` is that the internal thread pool is managed between all types of parallelisms in an efficient way. This means that you can (and _should_) use parallelism within a `TBB` graph node, if that's possible. In this example a node that processes an array of doubles into another array of doubles uses `parallel_for` to exploit the concurrency available in this operation. - function_node< double *, double * > n1( g, unlimited, [&]( double *a ) -> double * { - double *b = new double[N]; - parallel_for( 0, N, [&](int i) { - b[i] = f1(a[i]); - } ); - return b; - } ); + function_node< double *, double * > n1( g, unlimited, [&]( double *a ) -> double * { + double *b = new double[N]; + parallel_for( 0, N, [&](int i) { + b[i] = f1(a[i]); + } ); + return b; + } ); _Note:_ lambdas can be nested. -* Conclusions +## Conclusions `TBB` is a nice high level parallelism toolkit. Applicable to a lot of HENP real world problems and use cases. diff --git a/2016/20160204-hep-strikes-back/hep-sw.slide b/2016/20160204-hep-strikes-back/hep-sw.slide index 5819e67..4d1f921 100644 --- a/2016/20160204-hep-strikes-back/hep-sw.slide +++ b/2016/20160204-hep-strikes-back/hep-sw.slide @@ -1,154 +1,157 @@ -HEP s/w: parallelism strikes back +# HEP s/w: parallelism strikes back LPC Seminars, 2016-02-04 Sebastien Binet CNRS/IN2P3/LPC -* Parallelism: why? +## Parallelism: why? .image _figs/cpu-free-lunch.png 550 550 -* A bit of history: 1950-2000s +## A bit of history: 1950-2000s -- *1954:* computers were valve-based -- *1956:* first magnetic disk system sold (IBM RAMAC), `FORTRAN` under development -- *1959:* IBM-1401 shipped. Transistorised. Punched card input. -- *1960:* `PDP-1` launched (18-bit words) -- *1964:* `PDP-8` launched (12-bit words) -- *1964:* `System/360` launched (4*8-bit byte words, 8-64-256 `kB` of RAM) + - **1954:** computers were valve-based + - **1956:** first magnetic disk system sold (IBM RAMAC), `FORTRAN` under development + - **1959:** IBM-1401 shipped. Transistorised. Punched card input. + - **1960:** `PDP-1` launched (18-bit words) + - **1964:** `PDP-8` launched (12-bit words) + - **1964:** `System/360` launched (4\*8-bit byte words, 8-64-256 `kB` of RAM) + .image _figs/cern-computers.png 290 _ - -* A bit of history: 1950-2000s (at CERN) - -- *1963:* `IBM-7090` (`x4` `CERN` total computing capacity at the time) -- *1965:* `CDC-6600` (1 `MFLOPs`, `x15` `CERN` capacity) -- *1972-1984:* `CDC-7600`, `IBM-370/168` -- *1982:* VAX 750s,780s,8600s -- *1988-1993:* `Cray` -- *1996:* mainframes replaced by `UNIX` and `PC` servers. - + +## A bit of history: 1950-2000s (at CERN) + + - **1963:** `IBM-7090` (`x4` `CERN` total computing capacity at the time) + - **1965:** `CDC-6600` (1 `MFLOPs`, `x15` `CERN` capacity) + - **1972-1984:** `CDC-7600`, `IBM-370/168` + - **1982:** VAX 750s,780s,8600s + - **1988-1993:** `Cray` + - **1996:** mainframes replaced by `UNIX` and `PC` servers. + + .image _figs/cern-computers.png 290 _ -* +## .image _figs/cern-weekly-interactive-users.png _ 800 - -* (a brief) History of software in HEP -* 50's-90's: FORTRAN77 +## (a brief) History of software in HEP + +## 50's-90's: FORTRAN77 + +//.code \_code/hello.f -#.code _code/hello.f .play _code/hello.f.go /START OMIT/,/END OMIT/ - $ gfortran -c hello.f && gfortran -o hello hello.o - $ ./hello - Hello from FORTRAN + $ gfortran -c hello.f && gfortran -o hello hello.o + $ ./hello + Hello from FORTRAN + + - `FORTRAN77` is the **king** + - 1964: **CERNLIB** + - REAP (paper tape measurements), THRESH (geometry reconstruction) + - SUMX, **HBOOK** (statistical analysis chain) + - ZEBRA (memory management, I/O, ...) + - GEANT3, **PAW** -- `FORTRAN77` is the *king* -- 1964: *CERNLIB* -- REAP (paper tape measurements), THRESH (geometry reconstruction) -- SUMX, *HBOOK* (statistical analysis chain) -- ZEBRA (memory management, I/O, ...) -- GEANT3, *PAW* +## 90's-...: C++ -* 90's-...: C++ +//.code \_code/hello.cxx -#.code _code/hello.cxx .play _code/hello.cxx.go /START OMIT/,/END OMIT/ - $ c++ -o hello hello.cxx && ./hello - Hello from C++ - + $ c++ -o hello hello.cxx && ./hello + Hello from C++ + .image _figs/my-root6splash.png 190 190 -- object-oriented programming (OOP) is the cool kid on the block -- *ROOT*, POOL, LHC++, AIDA, *Geant4* -- `C++` takes roots in HEP + - object-oriented programming (OOP) is the cool kid on the block + - **ROOT**, POOL, LHC++, AIDA, **Geant4** + - `C++` takes roots in HEP + +## 00's-...: python -* 00's-...: python +//.code \_code/hello.py -#.code _code/hello.py .play _code/hello.py.go /START OMIT/,/END OMIT/ - $ python ./hello.py - Hello from python - + $ python ./hello.py + Hello from python + .image _figs/my-python-logo.png 100 250 -- `python` becomes the _de_ _facto_ scripting language in HEP -- framework data-cards -- analysis glue, (whole) analyses in `python` -- *PyROOT*, rootpy -- numpy, scipy, matplotlib, *IPython/Jupyter* + - `python` becomes the _de_ _facto_ scripting language in HEP + - framework data-cards + - analysis glue, (whole) analyses in `python` + - **PyROOT**, rootpy + - numpy, scipy, matplotlib, **IPython/Jupyter** -* Current software in a nutshell +## Current software in a nutshell -- *Generators*: generation of true particles from fondamental physics first principles -- *Full* *Simulation*: tracking of all stable particles in magnetic field through the detector simulating interaction, recording energy deposition (*CPU* *intensive*) -- *Reconstruction*: from real data, or from `Monte-Carlo` simulation data as above -- *Fast* *Simulation*: parametric simulation, faster, coarser -- *Analysis*: daily work of physicists, running on output of reconstruction to derive analysis specific information (*I/O* *intensive*) -- everything in the same `C++` offline control framework (except analysis) + - **Generators**: generation of true particles from fondamental physics first principles + - **Full** **Simulation**: tracking of all stable particles in magnetic field through the detector simulating interaction, recording energy deposition (**CPU** **intensive**) + - **Reconstruction**: from real data, or from `Monte-Carlo` simulation data as above + - **Fast** **Simulation**: parametric simulation, faster, coarser + - **Analysis**: daily work of physicists, running on output of reconstruction to derive analysis specific information (**I/O** **intensive**) + - everything in the same `C++` offline control framework (except analysis) .image _figs/data-flux-summary-all.png 210 800 +## -* + - `C++`: **slow** (very slow?) to compile/develop, **fast** to execute + - `python`: **fast** development cycle (no compilation), **slow** to execute -- `C++`: *slow* (very slow?) to compile/develop, *fast* to execute -- `python`: *fast* development cycle (no compilation), *slow* to execute -# (can be mitigated if leveraging/rewriting(parts in) `C++`. more work) +// (can be mitigated if leveraging/rewriting(parts in) `C++`. more work) .image _figs/xkcd-compiling.png 400 400 Are those our only options ? - -* Moore's law +## Moore's law .image _figs/cpu-free-lunch.png 550 550 -* The hardware/software contract +## The hardware/software contract .image _figs/par-prog-old-days.png _ 850 -* Hardware diversity: combining building blocks +## Hardware diversity: combining building blocks .image _figs/par-prog-heterogeneous.png -* Moore's law +## Moore's law -- Moore's law still observed at the hardware level -- *However* the _effective_ perceived computing power is mitigated + - Moore's law still observed at the hardware level + - **However** the _effective_ perceived computing power is mitigated _"Easy_ _life"_ during the last 20-30 years: -- Moore's law translated into *doubling* compute capacity every ~18 months (_via_ clock frequency) -- *Concurrency* and *parallelism* necessary to efficiently harness the compute power of our new multi-core CPU architectures. + - Moore's law translated into **doubling** compute capacity every ~18 months (_via_ clock frequency) + - **Concurrency** and **parallelism** necessary to efficiently harness the compute power of our new multi-core CPU architectures. _But_ our current software isn't prepared for parallel/concurrent environments. - -* Free lunch is over +## Free lunch is over .image _figs/head-on.png _ 900 -* Interlude: concurrency & parallelism +## Interlude: concurrency & parallelism -* Interlude: concurrency & parallelism +## Interlude: concurrency & parallelism -- *Concurrency* is about _dealing_ with lots of things at once. -- *Parallelism* is about _doing_ lots of things at once. -- Not the same, but related. -- Concurrency is about _structure_, parallelism is about _execution_. + - **Concurrency** is about _dealing_ with lots of things at once. + - **Parallelism** is about _doing_ lots of things at once. + - Not the same, but related. + - Concurrency is about _structure_, parallelism is about _execution_. .image _figs/conc-para.png 200 600 Concurrency is a way to structure a program by breaking it into pieces that can be executed independently. Communication is the means to coordinate the independent executions. -* Concurrency vs Parallelism +## Concurrency vs Parallelism _Concurrency:_ programming as the composition of independently executing processes/tasks. @@ -156,51 +159,48 @@ _Parallelism:_ programming as the simultaneous execution of (possibly related) c .image _figs/conc-vs-par.png 350 _ -* Concurrency vs Parallelism +## Concurrency vs Parallelism Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once. -Concurrency is about (program) *structure*. -Parallelism is about (program) *execution*. +Concurrency is about (program) **structure**. +Parallelism is about (program) **execution**. .image _figs/conc-vs-par-prog.png 350 _ - -* Concurrency in HEP software +## Concurrency in HEP software .image _figs/conc-level.png 600 400 -* Concurrency in HEP software - II +## Concurrency in HEP software - II .image _figs/levels-of-conc.png -# Graal of concurrent software: +// Graal of concurrent software: .image _figs/gaudi-hive-2.png 250 350 -* +## .image _figs/conc-para-mt-mp.png 600 1000 - -* Multi-processing +## Multi-processing Launch _N_ instances of an application on a node with _N_ cores -- re-use pre-existing code -- _a_ _priori_ no required modification of pre-existing code -- satisfactory _scalability_ with the number of cores - -*But:* + - re-use pre-existing code + - _a_ _priori_ no required modification of pre-existing code + - satisfactory _scalability_ with the number of cores -- resource requirements increase with the number of processes -- memory footprint *increases* -- as do other O/S (limited) resources (file descriptors, network sockets, ...) -- scalability of *I/O* debatable when number of cores > ~100 +**But:** + - resource requirements increase with the number of processes + - memory footprint **increases** + - as do other O/S (limited) resources (file descriptors, network sockets, ...) + - scalability of **I/O** debatable when number of cores > ~100 -* Multi-threading +## Multi-threading .image _figs/mt-cxx.png @@ -209,109 +209,107 @@ Launch _N_ instances of an application on a node with _N_ cores - (Intel) Threading Building Blocks - ... -* Time for a new language ? +## Time for a new language ? .image _figs/new-lang.png 600 800 -* Candidates +## Candidates -- python/pypy -- FORTRAN-2008 -- Vala -- Swift -- Rust -- Go -- Chapel -- Scala -- Haskell -- Clojure + - python/pypy + - FORTRAN-2008 + - Vala + - Swift + - Rust + - Go + - Chapel + - Scala + - Haskell + - Clojure -* Why not Go ? +## Why not Go ? -# .image _figs/hello-go.png 600 900 +// .image \_figs/hello-go.png 600 900 .play _code/hello.go - $ go run hello.go - Hello from Go + $ go run hello.go + Hello from Go A nice language with a nice mascot. .image _figs/golang-logo.png 200 400 -* Go in a nutshell +## Go in a nutshell -[[https://golang.org][Go]] is a new, general-purpose programming language. +[Go](https://golang.org) is a new, general-purpose programming language. -- Compiled -- Statically typed -- Concurrent -- Simple -- Productive + - Compiled + - Statically typed + - Concurrent + - Simple + - Productive "Go is a wise, clean, insightful, fresh thinking approach to the greatest-hits subset of the well understood." - Michael T. Jones +## History -* History + - Project starts at Google in 2007 (by Griesemer, Pike, Thompson) + - Open source release in November 2009 + - More than 550 contributors have joined the project + - Version 1.0 release in March 2012 + - Version 1.1 release in May 2013 + - Version 1.2 release in December 2013 + - Version 1.3 release in June 2014 + - Version 1.4 release in December 2014 + - Version 1.5 release in August 2015 + - Version 1.6: slated for ~mid-February 2016 -- Project starts at Google in 2007 (by Griesemer, Pike, Thompson) -- Open source release in November 2009 -- More than 550 contributors have joined the project -- Version 1.0 release in March 2012 -- Version 1.1 release in May 2013 -- Version 1.2 release in December 2013 -- Version 1.3 release in June 2014 -- Version 1.4 release in December 2014 -- Version 1.5 release in August 2015 -- Version 1.6: slated for ~mid-February 2016 +// Go was originally built by a team at Google, led by Robert Griesemer, Rob Pike, and Ken Thompson. In November 2010, Go was launched publically as an open source project. Since then, a team at Google and more than 250 contributors from the open source community continued to improve the Go language, libraries, and tools. -# Go was originally built by a team at Google, led by Robert Griesemer, Rob Pike, and Ken Thompson. In November 2010, Go was launched publically as an open source project. Since then, a team at Google and more than 250 contributors from the open source community continued to improve the Go language, libraries, and tools. +// In March 2012, we announced Go 1, a version of the language and libraries that will be supported for years to come. -# In March 2012, we announced Go 1, a version of the language and libraries that will be supported for years to come. +## Elements of Go -* Elements of Go + - Founding fathers: Russ Cox, Robert Griesemer, Ian Lance Taylor, Rob Pike, Ken Thompson -- Founding fathers: Russ Cox, Robert Griesemer, Ian Lance Taylor, Rob Pike, Ken Thompson + - Concurrent, garbage-collected + - An Open-source general progamming language (BSD-3) + - feel of a **dynamic** **language**: limited verbosity thanks to the _type_ _inference_ _system_, map, slices + - safety of a **static** **type** **system** + - compiled down to machine language (so it is fast, goal is ~10% of C) + - **object-oriented** but w/o classes, **builtin** **reflection** + - first-class functions with **closures** + - implicitly satisfied **interfaces** +## Elements of Go - II -- Concurrent, garbage-collected -- An Open-source general progamming language (BSD-3) -- feel of a *dynamic* *language*: limited verbosity thanks to the _type_ _inference_ _system_, map, slices -- safety of a *static* *type* *system* -- compiled down to machine language (so it is fast, goal is ~10% of C) -- *object-oriented* but w/o classes, *builtin* *reflection* -- first-class functions with *closures* -- implicitly satisfied *interfaces* + - available on MacOSX, Linux, Windows,... x86, x64, ARM, ARM64. + - available on _lxplus_: -* Elements of Go - II + $ ssh lxplus + [...] + * LXPLUS Public Login Service + * 2014-09-23 - expect installed + * 2014-10-02 - golang (Go Language) installed + * ******************************************************************** -- available on MacOSX, Linux, Windows,... x86, x64, ARM, ARM64. -- available on _lxplus_: + $ /usr/bin/go version + go version go1.5.1 linux/amd64 - $ ssh lxplus - [...] - * LXPLUS Public Login Service - * 2014-09-23 - expect installed - * 2014-10-02 - golang (Go Language) installed - * ******************************************************************** +## Concurrency - $ /usr/bin/go version - go version go1.5.1 linux/amd64 +## Goroutines -* Concurrency + - The _go_ statement launches a function call as a goroutine -* Goroutines - -- The _go_ statement launches a function call as a goroutine go f() go f(x, y, ...) -- A goroutine runs concurrently (but not necessarily in parallel) -- A goroutine has its own (growable/shrinkable) stack - + - A goroutine runs concurrently (but not necessarily in parallel) + - A goroutine has its own (growable/shrinkable) stack -* A simple example +## A simple example .code _code/concurrency1.go /f START/,/f END/ @@ -319,8 +317,7 @@ Function f is launched as 3 different goroutines, all running concurrently: .play _code/concurrency1.go /main START/,/main END/ - -* Communication via channels +## Communication via channels A channel type specifies a channel value type (and possibly a communication direction): @@ -340,7 +337,7 @@ A channel permits _sending_ and _receiving_ values: Channel operations synchronize the communicating goroutines. -* Communicating goroutines +## Communicating goroutines Each goroutine sends its results via channel ch: @@ -350,263 +347,248 @@ The main goroutine receives (and prints) all results from the same channel: .play _code/concurrency2.go /main START/,/main END/ +// \* Non-elements of Go -# * Non-elements of Go - -# - *no* dynamic libraries (but dynamic loading since `go-1.5`) -# - *no* templates nor generics (maybe for `go-2.0`) -# - *no* operator overloading +// - **no** dynamic libraries (but dynamic loading since `go-1.5`) +// - **no** templates nor generics (maybe for `go-2.0`) +// - **no** operator overloading -* Real-world application? +## Real-world application? -OK, [[https://golang.org][Go]] is great. +OK, [Go](https://golang.org) is great. -And it's being used by [[https://github.com/golang/go/wiki/GoUsers][many companies]] (beside `Google`): Mozilla, New-York Times, CoreOS, Docker Inc., SpaceX, ... +And it's being used by [many companies](https://github.com/golang/go/wiki/GoUsers) (beside `Google`): Mozilla, New-York Times, CoreOS, Docker Inc., SpaceX, ... But what about `HEP`? and `astro/cosmo`? +## fads -* fads - -* fads +## fads `fads` is a "FAst Detector Simulation" toolkit. -- morally a translation of [[https://cp3.irmp.ucl.ac.be/projects/delphes][C++-Delphes]] into Go -- uses [[https://github.com/go-hep/fwk][go-hep/fwk]] to expose, manage and harness concurrency into the usual `HEP` event loop (`initialize` | `process-events` | `finalize`) -- uses [[https://github.com/go-hep/hbook][go-hep/hbook]] for histogramming, [[htpps://github.com/go-hep/hepmc][go-hep/hepmc]] for `HepMC` input/output + - morally a translation of [C++-Delphes](https://cp3.irmp.ucl.ac.be/projects/delphes) into Go + - uses [go-hep/fwk](https://github.com/go-hep/fwk) to expose, manage and harness concurrency into the usual `HEP` event loop (`initialize` | `process-events` | `finalize`) + - uses [go-hep/hbook](https://github.com/go-hep/hbook) for histogramming, [go-hep/hepmc](htpps://github.com/go-hep/hepmc) for `HepMC` input/output Code is on github (BSD-3): .link https://github.com/go-hep/fwk .link https://github.com/go-hep/fads -Documentation is served by [[https://godoc.org][godoc.org]]: +Documentation is served by [godoc.org](https://godoc.org): .link https://godoc.org/github.com/go-hep/fwk .link https://godoc.org/github.com/go-hep/fads -* go-hep/fads - Installation +## go-hep/fads - Installation As easy as: - $ export GOPATH=$HOME/dev/gocode - $ export PATH=$GOPATH/bin:$PATH - - $ go get github.com/go-hep/fads/... + $ export GOPATH=$HOME/dev/gocode + $ export PATH=$GOPATH/bin:$PATH + + $ go get github.com/go-hep/fads/... Yes, with the ellipsis at the end, to also install sub-packages. -- `go` `get` will recursively download and install all the packages that [[https://github.com/go-hep/fads][go-hep/fads]] depends on. (no `Makefile` needed) + - `go` `get` will recursively download and install all the packages that [go-hep/fads](https://github.com/go-hep/fads) depends on. (no `Makefile` needed) +## go-hep/fwk - Examples -* go-hep/fwk - Examples + $ fwk-ex-tuto-1 -help + Usage: fwk-ex-tuto1 [options] + ex: + $ fwk-ex-tuto-1 -l=INFO -evtmax=-1 - $ fwk-ex-tuto-1 -help - Usage: fwk-ex-tuto1 [options] - - ex: - $ fwk-ex-tuto-1 -l=INFO -evtmax=-1 - - options: - -evtmax=10: number of events to process - -l="INFO": message level (DEBUG|INFO|WARN|ERROR) - -nprocs=0: number of events to process concurrently + options: + -evtmax=10: number of events to process + -l="INFO": message level (DEBUG|INFO|WARN|ERROR) + -nprocs=0: number of events to process concurrently Runs 2 tasks. -#- task 1 stores 2 `int`s under `"t1-ints1"` and `"t2-ints2"`. -#- task 2 retrieves `"t1-ints1"` and stores `"t1-ints1-massaged"` +//- task 1 stores 2 `int`s under `"t1-ints1"` and `"t2-ints2"`. +//- task 2 retrieves `"t1-ints1"` and stores `"t1-ints1-massaged"` .image _figs/fwk-ex1-dflow.png 200 200 -* go-hep/fwk - Examples - - $ fwk-ex-tuto-1 - ::: fwk-ex-tuto-1... - t2 INFO configure... - t2 INFO configure... [done] - t1 INFO configure ... - t1 INFO configure ... [done] - t2 INFO start... - t1 INFO start... - app INFO >>> running evt=0... - t1 INFO proc... (id=0|0) => [10, 20] - t2 INFO proc... (id=0|0) => [10 -> 100] - [...] - app INFO >>> running evt=9... - t1 INFO proc... (id=9|0) => [10, 20] - t2 INFO proc... (id=9|0) => [10 -> 100] - t2 INFO stop... - t1 INFO stop... - app INFO cpu: 654.064us - app INFO mem: alloc: 62 kB - app INFO mem: tot-alloc: 74 kB - app INFO mem: n-mallocs: 407 - app INFO mem: n-frees: 60 - app INFO mem: gc-pauses: 0 ms - ::: fwk-ex-tuto-1... [done] (cpu=788.578us) - - -* go-hep/fwk - Concurrency - -[[https://github.com/go-hep/fwk][fwk]] enables: +## go-hep/fwk - Examples + + $ fwk-ex-tuto-1 + ::: fwk-ex-tuto-1... + t2 INFO configure... + t2 INFO configure... [done] + t1 INFO configure ... + t1 INFO configure ... [done] + t2 INFO start... + t1 INFO start... + app INFO >>> running evt=0... + t1 INFO proc... (id=0|0) => [10, 20] + t2 INFO proc... (id=0|0) => [10 -> 100] + [...] + app INFO >>> running evt=9... + t1 INFO proc... (id=9|0) => [10, 20] + t2 INFO proc... (id=9|0) => [10 -> 100] + t2 INFO stop... + t1 INFO stop... + app INFO cpu: 654.064us + app INFO mem: alloc: 62 kB + app INFO mem: tot-alloc: 74 kB + app INFO mem: n-mallocs: 407 + app INFO mem: n-frees: 60 + app INFO mem: gc-pauses: 0 ms + ::: fwk-ex-tuto-1... [done] (cpu=788.578us) + +## go-hep/fwk - Concurrency + +[fwk](https://github.com/go-hep/fwk) enables: - event-level concurrency - tasks-level concurrency -[[https://github.com/go-hep/fwk][fwk]] relies on [[https://golang.org][Go]]'s runtime to properly schedule _goroutines_. +[fwk](https://github.com/go-hep/fwk) relies on [Go](https://golang.org)'s runtime to properly schedule _goroutines_. -For sub-task concurrency, users are by construction required to use [[https://golang.org][Go]]'s constructs (_goroutines_ and _channels_) so everything is consistent *and* the _runtime_ has the *complete* *picture*. +For sub-task concurrency, users are by construction required to use [Go](https://golang.org)'s constructs (_goroutines_ and _channels_) so everything is consistent **and** the _runtime_ has the **complete** **picture**. -- *Note:* [[https://golang.org][Go]]'s runtime isn't yet _NUMA-aware_. A proposal for _Go-1.5_ _(June-2015)_ is in the [[https://docs.google.com/document/d/1d3iI2QWURgDIsSR6G2275vMeQ_X7w-qxM2Vp7iGwwuM/pub][works]]. (delayed as of Feb-2016) + - **Note:** [Go](https://golang.org)'s runtime isn't yet _NUMA-aware_. A proposal for _Go-1.5_ _(June-2015)_ is in the [works](https://docs.google.com/document/d/1d3iI2QWURgDIsSR6G2275vMeQ_X7w-qxM2Vp7iGwwuM/pub). (delayed as of Feb-2016) +## go-hep/fads - real world use case -* go-hep/fads - real world use case + - translated [C++-Delphes](https://cp3.irmp.ucl.ac.be/projects/delphes)' ATLAS data-card into Go + - [go-hep/fads-app](https://github.com/go-hep/fads/blob/master/cmd/fads-app/main.go) + - installation: -- translated [[https://cp3.irmp.ucl.ac.be/projects/delphes][C++-Delphes]]' ATLAS data-card into Go -- [[https://github.com/go-hep/fads/blob/master/cmd/fads-app/main.go][go-hep/fads-app]] -- installation: + $ go get github.com/go-hep/fads/cmd/fads-app + $ fads-app -help + Usage: fads-app [options] - $ go get github.com/go-hep/fads/cmd/fads-app - $ fads-app -help - Usage: fads-app [options] - - ex: - $ fads-app -l=INFO -evtmax=-1 ./testdata/hepmc.data - - options: - -cpu-prof=false: enable CPU profiling - -evtmax=-1: number of events to process - -l="INFO": log level (DEBUG|INFO|WARN|ERROR) - -nprocs=0: number of concurrent events to process + ex: + $ fads-app -l=INFO -evtmax=-1 ./testdata/hepmc.data -* go-hep/fads - components + options: + -cpu-prof=false: enable CPU profiling + -evtmax=-1: number of events to process + -l="INFO": log level (DEBUG|INFO|WARN|ERROR) + -nprocs=0: number of concurrent events to process -- a `HepMC` converter -- particle propagator -- calorimeter simulator -- energy rescaler, momentum smearer -- isolation -- b-tagging, tau-tagging -- jet-finder (reimplementation of FastJet in Go: [[https://github.com/go-hep/fastjet][go-hep/fastjet]]) -- histogram service (from [[https://github.com/go-hep/fwk][go-hep/fwk]]) - -Caveats: +## go-hep/fads - components -- no real persistency to speak of (_i.e.:_ `JSON`, `ASCII` and `Gob`) -- jet clustering limited to N^3 (slowest and dumbest scheme of `C++-FastJet`) + - a `HepMC` converter + - particle propagator + - calorimeter simulator + - energy rescaler, momentum smearer + - isolation + - b-tagging, tau-tagging + - jet-finder (reimplementation of FastJet in Go: [go-hep/fastjet](https://github.com/go-hep/fastjet)) + - histogram service (from [go-hep/fwk](https://github.com/go-hep/fwk)) +Caveats: -* + - no real persistency to speak of (_i.e.:_ `JSON`, `ASCII` and `Gob`) + - jet clustering limited to N^3 (slowest and dumbest scheme of `C++-FastJet`) +## .image _figs/fads-dflow.png 600 600 +## Results - testbenches -* Results - testbenches - -- Linux: Intel(R) Core(TM)2 Duo CPU @ 2.53GHz, 4GB RAM, 2 cores -- MacOSX-10.6: Intel(R) Xeon(R) CPU @ 2.27GHz, 172GB RAM, 16 cores -- Linux: Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz, 40 cores + - Linux: Intel(R) Core(TM)2 Duo CPU @ 2.53GHz, 4GB RAM, 2 cores + - MacOSX-10.6: Intel(R) Xeon(R) CPU @ 2.27GHz, 172GB RAM, 16 cores + - Linux: Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz, 40 cores -* Linux (40 cores) testbench: memory +## Linux (40 cores) testbench: memory .image _figs/lhcb3-rss.png 550 800 -* Linux (40 cores) testbench: CPU +## Linux (40 cores) testbench: CPU .image _figs/lhcb3-cpu.png 550 800 -* Linux (40 cores) testbench: event throughput +## Linux (40 cores) testbench: event throughput .image _figs/lhcb3-hz.png 550 800 +## Results & Conclusions -* Results & Conclusions + - good RSS scaling + - good CPU scaling -- good RSS scaling -- good CPU scaling - -- bit-by-bit matching physics results wrt `Delphes` (up to calorimetry) + - bit-by-bit matching physics results wrt `Delphes` (up to calorimetry) Also addresses `C++` and `python` deficiencies: -- code distribution -- code installation -- compilation/development speed -- runtime speed -- simple language - -* Prospects + - code distribution + - code installation + - compilation/development speed + - runtime speed + - simple language -- proper persistency package (in the works: [[https://github.com/go-hep/rio][go-hep/rio]]) -- histograms + n-tuples: [[https://github.com/go-hep/hbook][go-hep/hbook]], [[https://github.com/go-hep/hplot][go-hep/hplot]] -- performance improvements (cpu-profiling via `go` `tool` `pprof`) -- implement more of `go-fastjet` combination schemes and strategies -- more end-user oriented documentation +## Prospects -Join the fun: [[https://groups.google.com/d/forum/go-hep][go-hep forum]] + - proper persistency package (in the works: [go-hep/rio](https://github.com/go-hep/rio)) + - histograms + n-tuples: [go-hep/hbook](https://github.com/go-hep/hbook), [go-hep/hplot](https://github.com/go-hep/hplot) + - performance improvements (cpu-profiling via `go` `tool` `pprof`) + - implement more of `go-fastjet` combination schemes and strategies + - more end-user oriented documentation -[[http://go-hep.github.io/][go-hep]] has a website too: [[http://go-hep.github.io]] +Join the fun: [go-hep forum](https://groups.google.com/d/forum/go-hep) -Have also a look at [[https://github.com/astrogo][astrogo]] for tools and libraries for astro/cosmo (_e.g.:_ a reader/writer for `FITS` files, [[https://github.com/astrogo/fitsio][astrogo/fitsio]]) +[go-hep](http://go-hep.github.io/) has a website too: [go-hep.github.io](http://go-hep.github.io) +Have also a look at [astrogo](https://github.com/astrogo) for tools and libraries for astro/cosmo (_e.g.:_ a reader/writer for `FITS` files, [astrogo/fitsio](https://github.com/astrogo/fitsio)) -* Acknowledgements / resources +## Acknowledgements / resources .link http://talks.golang.org/2012/tutorial.slide - .link http://talks.golang.org/2014/taste.slide - .link http://tour.golang.org -* That's all ! - - -* Backup - -* go-hep/fwk - configuration & steering - -- use regular [[https://golang.org][Go]] to configure and steer. -- still on the fence on a DSL-based configuration language (`YAML`, `HCL`, `Toml`, ...) -- probably *not* `Python` though - - // job is the scripting interface to 'fwk' - import "github.com/go-hep/fwk/job" - - func main() { - // create a default fwk application, with some properties - app := job.New(job.P{ - "EvtMax": 10, - "NProcs": 2, - }) - - // ... cont'd on next page... - -* go-hep/fwk - configuration & steering - - // create a task that reads integers from some location - // and publish the square of these integers under some other location - app.Create(job.C{ - Type: "github.com/go-hep/fwk/testdata.task2", - Name: "t2", - Props: job.P{ - "Input": "t1-ints1", - "Output": "t1-ints1-massaged", - }, - }) - // create a task that publish integers to some location(s) - // create after the consummer task to exercize the automatic data-flow scheduling. - app.Create(job.C{ - Type: "github.com/go-hep/fwk/testdata.task1", - Name: "t1", - Props: job.P{ - "Ints1": "t1-ints1", - "Ints2": "t2-ints2", - "Int1": int64(10), // initial value for the Ints1 - "Int2": int64(20), // initial value for the Ints2 - }, - }) - app.Run() - +## That's all ! + +## Backup + +## go-hep/fwk - configuration & steering + + - use regular [Go](https://golang.org) to configure and steer. + - still on the fence on a DSL-based configuration language (`YAML`, `HCL`, `Toml`, ...) + - probably **not** `Python` though + + // job is the scripting interface to 'fwk' + import "github.com/go-hep/fwk/job" + + func main() { + // create a default fwk application, with some properties + app := job.New(job.P{ + "EvtMax": 10, + "NProcs": 2, + }) + + // ... cont'd on next page... + +## go-hep/fwk - configuration & steering + + // create a task that reads integers from some location + // and publish the square of these integers under some other location + app.Create(job.C{ + Type: "github.com/go-hep/fwk/testdata.task2", + Name: "t2", + Props: job.P{ + "Input": "t1-ints1", + "Output": "t1-ints1-massaged", + }, + }) + // create a task that publish integers to some location(s) + // create after the consummer task to exercize the automatic data-flow scheduling. + app.Create(job.C{ + Type: "github.com/go-hep/fwk/testdata.task1", + Name: "t1", + Props: job.P{ + "Ints1": "t1-ints1", + "Ints2": "t2-ints2", + "Int1": int64(10), // initial value for the Ints1 + "Int2": int64(20), // initial value for the Ints2 + }, + }) + app.Run() diff --git a/2016/20160208-info-in2p3-nouveaux-langages/new-languages.slide b/2016/20160208-info-in2p3-nouveaux-langages/new-languages.slide index a847cb9..c0e220d 100644 --- a/2016/20160208-info-in2p3-nouveaux-langages/new-languages.slide +++ b/2016/20160208-info-in2p3-nouveaux-langages/new-languages.slide @@ -1,4 +1,4 @@ -Nouveaux langages +# Nouveaux langages Réunion annuelle Info IN2P3, 2016-02-08 Sébastien Binet @@ -9,68 +9,66 @@ Christophe Meessen CNRS/IN2P3/CPPM meessen@cppm.in2p3.fr -* Nouveaux langages +## Nouveaux langages -Working group dedicated to the study of new languages: [[http://informatique.in2p3.fr/?q=nouveaux-langages][website]] +Working group dedicated to the study of new languages: [website](http://informatique.in2p3.fr/?q=nouveaux-langages) For now, focused on the study of: -- [[http://dlang.org/][D]]: a better `C++` -- [[https://golang.org/][Go]]: a better `C`, with builtin support for concurrency + - [D](http://dlang.org/): a better `C++` + - [Go](https://golang.org/): a better `C`, with builtin support for concurrency ChM investigated `D`. SB investigated `Go`. -* D +## D -- workshop at [[https://indico.in2p3.fr/event/9954/other-view?view=standard][JI-2014]] ([[https://indico.in2p3.fr/event/9954/session/5/contribution/0/material/slides/0.pdf][intro]], [[https://indico.in2p3.fr/event/9954/session/13/contribution/87/material/0/0.pdf][instructions]]) + - workshop at [JI-2014](https://indico.in2p3.fr/event/9954/other-view?view=standard) ([intro](https://indico.in2p3.fr/event/9954/session/5/contribution/0/material/slides/0.pdf), [instructions](https://indico.in2p3.fr/event/9954/session/13/contribution/87/material/0/0.pdf)) -* Go +## Go -Developed a few proof-of-concepts for the HEP community, available from the [[https://github.com/go-hep][go-hep]] organization: +Developed a few proof-of-concepts for the HEP community, available from the [go-hep](https://github.com/go-hep) organization: -- [[https://github.com/go-hep/hbook][go-hep/hbook]]: histograms handling -- [[https://github.com/go-hep/hplot][go-hep/hplot]]: histograms plotting -- [[https://github.com/go-hep/rio][go-hep/rio]]: building blocks for `I/O` persistency -- [[https://github.com/go-hep/fwk][go-hep/fwk]]: a concurrent control framework a-la-Gaudi -- [[https://github.com/go-hep/fads][go-hep/fads]]: a fast detector simulation a-la-Delphes -- etc... + - [go-hep/hbook](https://github.com/go-hep/hbook): histograms handling + - [go-hep/hplot](https://github.com/go-hep/hplot): histograms plotting + - [go-hep/rio](https://github.com/go-hep/rio): building blocks for `I/O` persistency + - [go-hep/fwk](https://github.com/go-hep/fwk): a concurrent control framework a-la-Gaudi + - [go-hep/fads](https://github.com/go-hep/fads): a fast detector simulation a-la-Delphes + - etc... -Also did the same kind of work for the Astro/Cosmo community with the [[https://github.com/astrogo][astrogo]] organization: +Also did the same kind of work for the Astro/Cosmo community with the [astrogo](https://github.com/astrogo) organization: -- [[https://github.com/astrogo/fitsio][astrogo/fitsio]]: `FITS` file format r/w handling -- [[https://github.com/astrogo/asdf][astrogo/asdf]]: `ASDF` file format + - [astrogo/fitsio](https://github.com/astrogo/fitsio): `FITS` file format r/w handling + - [astrogo/asdf](https://github.com/astrogo/asdf): `ASDF` file format - -* Go - II +## Go - II Investigated application of `Go` for: -- `HEP` physics analyses (c.f. `go-hep/...` packages) -- `astro` physics analyses (c.f. [[https://github.com/astrogo/snfusion][astrogo/snfusion]]) -- slow control: [[https://github.com/go-lsst/ncs][go-lsst/ncs]], the "nimble control system" + - `HEP` physics analyses (c.f. `go-hep/...` packages) + - `astro` physics analyses (c.f. [astrogo/snfusion](https://github.com/astrogo/snfusion)) + - slow control: [go-lsst/ncs](https://github.com/go-lsst/ncs), the "nimble control system" About to start the investigation of its suitability for the "embedded" (IoT, ...) space. -* Go - talks, confs, etc... - -- [[http://inspirehep.net/record/1089636?ln=en][CHEP-2010]] -- [[http://inspirehep.net/record/1194222?ln=en][ACAT-2011]] -- [[http://inspirehep.net/record/1215013?ln=en][CHEP-2012]] -- workshop at [[https://indico.in2p3.fr/event/9954/other-view?view=standard][JI-2014]] ([[https://github.com/sbinet/ji-2014-go][slides]]) -- [[https://github.com/go-hep/talks/tree/master/2015/20150121-binet-fads-hsf][GDR TeraScale-2014]] -- workshop at [[http://www.devatlr.univ-montp2.fr/Atelier-Go][Dev@LR 2015-06-15]] ([[https://github.com/sbinet/dev-lr-2015-go][slides]]) +## Go - talks, confs, etc... -* Go usage + - [CHEP-2010](http://inspirehep.net/record/1089636?ln=en) + - [ACAT-2011](http://inspirehep.net/record/1194222?ln=en) + - [CHEP-2012](http://inspirehep.net/record/1215013?ln=en) + - workshop at [JI-2014](https://indico.in2p3.fr/event/9954/other-view?view=standard) ([slides](https://github.com/sbinet/ji-2014-go)) + - [GDR TeraScale-2014](https://github.com/go-hep/talks/tree/master/2015/20150121-binet-fads-hsf) + - workshop at [Dev@LR 2015-06-15](http://www.devatlr.univ-montp2.fr/Atelier-Go) ([slides](https://github.com/sbinet/dev-lr-2015-go)) -- SB. -- [[https://github.com/airnandez/cluefs][ClueFS: A tool for tracing I/O activity at the file system level]] -- [[https://github.com/lhcb-org/lbpkr][lbpkr: Go-based installer for RPMs and Dockers]] (used by `LHCb`) -- [[https://gitlab.in2p3.fr/avirm/analysis-go][analysis tools for AVIRM]] (Application of the interaction of radiation with material) +## Go usage -* Go - prospects, ideas + - SB. + - [ClueFS: A tool for tracing I/O activity at the file system level](https://github.com/airnandez/cluefs) + - [lbpkr: Go-based installer for RPMs and Dockers](https://github.com/lhcb-org/lbpkr) (used by `LHCb`) + - [analysis tools for AVIRM](https://gitlab.in2p3.fr/avirm/analysis-go) (Application of the interaction of radiation with material) -- *starting* to get some momentum -- pondering on a L2/L3-level lecture (+hands-on) in [[https://golang.org][Go]] at Universite Blaise Pascal -- perhaps another workshop+hands-on session at JI-2016 ? +## Go - prospects, ideas + - **starting** to get some momentum + - pondering on a L2/L3-level lecture (+hands-on) in [Go](https://golang.org) at Universite Blaise Pascal + - perhaps another workshop+hands-on session at JI-2016 ? diff --git a/2016/20160211-lpc-svc-info/lpc.slide b/2016/20160211-lpc-svc-info/lpc.slide index f25281d..d8819f2 100644 --- a/2016/20160211-lpc-svc-info/lpc.slide +++ b/2016/20160211-lpc-svc-info/lpc.slide @@ -1,4 +1,4 @@ -Go && Polymer && astro +# Go && Polymer && astro Meeting Svc-info-LPC, 2016-02-11 Sébastien Binet @@ -6,119 +6,117 @@ CNRS/IN2P3/LPC binet@clermont.in2p3.fr -* Go -You know already about [[https://golang.org][Go]]. +## Go -- Concurrency -- Python-like, C-like -- Development speed, runtime speed -- Garbage collector +You know already about [Go](https://golang.org). -* Polymer + - Concurrency + - Python-like, C-like + - Development speed, runtime speed + - Garbage collector + +## Polymer """ -[[https://www.polymer-project.org][Polymer]] is designed to make it easier and faster for developers to create great, reusable components for the modern web. +[Polymer](https://www.polymer-project.org) is designed to make it easier and faster for developers to create great, reusable components for the modern web. """ With custom elements, you can extend the vocabulary of HTML with your own elements. Elements that provide sophisticated UI. Elements that are as easy to use as