af4c933efe6d42a17e5caa3835187976066d6aab — Sebastien Binet 7 months ago b813dae main
2023-02-23-lpc-dev-varda: first import

Signed-off-by: Sebastien Binet <binet@cern.ch>
A 2023/2023-02-23-lpc-dev-varda/_code/client.go => 2023/2023-02-23-lpc-dev-varda/_code/client.go +19 -0
@@ 0,0 1,19 @@
package varda // import "sbinet.org/x/varda"

import (

// Client provides access to a Varda server. After use, a Client should be
// closed via its Close method.
type Client struct {
	// Has unexported fields.

func NewClient(root string) (*Client, error)
func (c *Client) Close() error
func (c *Client) Fetch(ctx context.Context, v Ref) (io.ReadCloser, error)
func (c *Client) IterRef(ctx context.Context, f func(v Ref) error) error
func (c *Client) NumRef(ctx context.Context) (int64, error)
func (c *Client) Upload(ctx context.Context, fname string, r io.Reader) error

A 2023/2023-02-23-lpc-dev-varda/_code/meta.go => 2023/2023-02-23-lpc-dev-varda/_code/meta.go +19 -0
@@ 0,0 1,19 @@
package varda // import "sbinet.org/x/varda"

import "time"

// Meta holds metadata about a Varda resource
type Meta struct {
	ID     string
	Name   string
	MIME   string
	Size   int64
	Width  int64
	Height int64
	Lat    float64
	Lon    float64
	Time   time.Time

func MetaFrom(fname string) (Meta, error)
func (m Meta) IsImage() bool

A 2023/2023-02-23-lpc-dev-varda/_code/ref.go => 2023/2023-02-23-lpc-dev-varda/_code/ref.go +14 -0
@@ 0,0 1,14 @@
package varda // import "sbinet.org/x/varda"

import (

// Ref is a reference to a Varda resource: a photo or video.
type Ref struct {
	// Has unexported fields.

func (r Ref) Hash() hash.Hash
func (r Ref) Meta() Meta
func (r Ref) Thumb() []byte

A 2023/2023-02-23-lpc-dev-varda/_figs/Lets-leave-planet-GAFAM-NATU-BATX_by-David-Revoy.jpg => 2023/2023-02-23-lpc-dev-varda/_figs/Lets-leave-planet-GAFAM-NATU-BATX_by-David-Revoy.jpg +0 -0
A 2023/2023-02-23-lpc-dev-varda/_figs/gioui-logo.png => 2023/2023-02-23-lpc-dev-varda/_figs/gioui-logo.png +0 -0
A 2023/2023-02-23-lpc-dev-varda/_figs/gioui-logo_svg.png => 2023/2023-02-23-lpc-dev-varda/_figs/gioui-logo_svg.png +0 -0
A 2023/2023-02-23-lpc-dev-varda/_figs/perkeep.png => 2023/2023-02-23-lpc-dev-varda/_figs/perkeep.png +0 -0
A 2023/2023-02-23-lpc-dev-varda/_figs/restic.png => 2023/2023-02-23-lpc-dev-varda/_figs/restic.png +0 -0
A 2023/2023-02-23-lpc-dev-varda/_figs/screen-0.png => 2023/2023-02-23-lpc-dev-varda/_figs/screen-0.png +0 -0
A 2023/2023-02-23-lpc-dev-varda/_figs/screen-1.png => 2023/2023-02-23-lpc-dev-varda/_figs/screen-1.png +0 -0
A 2023/2023-02-23-lpc-dev-varda/_figs/screen-2.png => 2023/2023-02-23-lpc-dev-varda/_figs/screen-2.png +0 -0
A 2023/2023-02-23-lpc-dev-varda/_figs/screen-3.png => 2023/2023-02-23-lpc-dev-varda/_figs/screen-3.png +0 -0
A 2023/2023-02-23-lpc-dev-varda/_figs/syncthing.png => 2023/2023-02-23-lpc-dev-varda/_figs/syncthing.png +0 -0
A 2023/2023-02-23-lpc-dev-varda/_figs/varda.png => 2023/2023-02-23-lpc-dev-varda/_figs/varda.png +0 -0
A 2023/2023-02-23-lpc-dev-varda/talk.pdf => 2023/2023-02-23-lpc-dev-varda/talk.pdf +0 -0
A 2023/2023-02-23-lpc-dev-varda/talk.slide => 2023/2023-02-23-lpc-dev-varda/talk.slide +242 -0
@@ 0,0 1,242 @@
# Varda
23 Feb 2023

Sebastien Binet

## Photo application

_"Mz Michu"_ wants to:

- take photos from their smartphone
- automatically transfer/save/backup these photos on the cloud
- create an album off those photos
- share albums or photos with friends and family.

` `

What should they do ?

_Use `$PHOTOAPP` from `$SOME_GAFAM`_, _"of course"_.

` `

` `


` `


## De-GAFAM / De-Google-ify

The web giants are so powerful that they exercise technical, economic, cultural and political domination over our societies.

These dominations pose many problems for our freedoms:

- Surveillance capitalism
- Democratic drift
- Closure to a single vision of society
- Centralization of data and attention

.link https://degooglisons-internet.org

.image _figs/Lets-leave-planet-GAFAM-NATU-BATX_by-David-Revoy.jpg 400 _


Even if they are _"so convenient"_, **I** didn't want my friends and family to be stuck with these evil entities.

` `

So I went on the open source market to find software that could fulfill these end-user criterions:

- take photos from their smartphone
- automatically transfer/save/backup these photos to some remote location
- create an album off those photos
- share albums or photos with friends and family.

` `

` `

Together with:

- minimize ways to loose photos
- ability to easily migrate from one system to the other
- "easily" self hosted setup

## Yes-FOSS: transfering data

Data movement from smartphone to some data storage end-point was easy enough.

` `

I've setup [Syncthing](https://syncthing.net/):

- install `syncthing` on some server
- install `syncthing` on (multiple) client(s):
	- my smartphone,
	- my laptop,
	- ...
- configure the folder(s) that should be replicated
- replication can be unidirectional or bidirectional

.image _figs/syncthing.png 150 _

## Yes-FOSS: backing up data

Backup is performed on the `syncthing` server with [restic](https://restic.net/)

.image _figs/restic.png 200 _

- list the folders to be backed up
- configure the frequency at which they should be backed up
- configure the frequency they should be discarded
- I went with the following time profile:
	- keep the 7 last days
	- keep one per month
	- keep six per year
	- keep two per decade
- configure the backup storage(s):
	- 2 separate physical locations (self managed)
	- 1 encrypted output location on a 3rd-party storage ([Gandi](https://www.gandi.net))

## Yes-FOSS: photos album

Initially, I went with [Perkeep](https://perkeep.org/).

.image _figs/perkeep.png 150 _

[Perkeep](https://perkeep.org/) can do a bunch of things:

- transfer and replicate data
- present data:
	- expose a [net/http](https://pkg.go.dev/net/http) server with some javascript to interact with the store's content

and it's written in [Go](https://golang.org).

` `

- it was a bit heavy for the machines I wanted to deploy it on
	- _(an old refurbished `Linux` machine at my Mom's)_
- it didn't allow for multi-user setups
	- one had to run one [perkeep](https://perkeep.org) instance per user

So I went with writing my own thing.

## Varda

.image _figs/varda.png 400 _

.link https://fr.wikipedia.org/wiki/Agnes_Varda

` `

_Agnès Varda was a Belgian-born French film director, screenwriter, photographer, and artist._

## Varda

Enter [Varda](https://git.sr.ht/~sbinet/varda):

- written in [Go](https://golang.org)
- heavily inspired from [Perkeep](https://perkeep.org)
- command-line based tools to manage data
	- ingest data
	- list data
- components:
	- a set of [Go](https://golang.org)-based packages
	- a [net/http](https://pkg.go.dev/net/http) server, `varda-srv`, where the cmd-line tools can publish data
	- a [Gio](https://gioui.org)-based GUI to present data

## Gio

[Gio](https://gioui.org) can compile to:

- Windows, **Linux**, Darwin,
- Android, iOS
- [WASM](https://webassembly.org/) (so a web-based GUI is possible, eventually)

.image _figs/gioui-logo.png 100 _

[Varda](https://git.sr.ht/~sbinet/varda) has only been tested on `Linux` so far.

## Varda Mechanics

- data is uploaded to a user account on some `Varda` server (via `http`)
	- `vrd-put` can take a (set of) file(s) or a (set of) directories
	- `varda-gui` can do the same
- upon arrival, data is [content-addressed](https://en.wikipedia.org/wiki/Content-addressable_storage)
	- right now, a simple [SHA-224](https://fr.wikipedia.org/wiki/SHA-2)
	- investigating whether using [IPFS](https://docs.ipfs.tech/concepts/content-addressing/) would make sense
	- simplifies backing up data: a simple `/bin/cp` suffies, w/o the risk to overwrite and loose data
		- first collision w/ `SHA-224` is for insertion of `7.34 e+33` elements
		- probability `p > 1e-18` to have a collision is north of `1.1 e+20` elements (`SHA-192`) and
		- probability `p > 1e-18` to have a collision is south of `4.8 e+29` elements (`SHA-256`)
		- `SHA-224` are also partitioned by year (so this improves on the collisions risks)
- `varda-srv` stores the content-addressed data on some local repository
- `varda-srv` stores the metadata (size, filename, MIME type, date, EXIF (orientation, lat/long), thumbnails, ...) in a [SQLite](https://sqlite.org) database

## Varda interface

.code _code/client.go

## Varda interface

.code _code/ref.go

## Varda interface

.code _code/meta.go

## Varda GUI

.image _figs/screen-0.png 600 _

## Varda GUI

.image _figs/screen-1.png 600 _

## Varda GUI

.image _figs/screen-2.png 600 _

## Varda GUI

.image _figs/screen-3.png 600 _

## Current setup

- [syncthing](https://https://syncthing.net/) transfers data to the storage server
- `systemd`-based daemon launches `vrd-put` when new data appears (`fanotify(7)`)
	- daemon removes old data (older than 6 months)
	- with the `syncthing` configuration I have setup, this automatically removes the data on the original device
- `varda-srv` collects data
- [restic](https://restic.net/) monitors the `varda-srv` repository
- `varda-gui` presents data


- add support for video files
- add support for audio files
- add support for photo edition
	- libraries to crop/resize/gamma-correct/... already available
	- _"just"_ need to plug them in
	- plug in a red-eyes removal library
	- plan is to store the photo modification operations (crop/resize/...) at the database level and leave the original untouched
- add albums creation
	- will be done at the database level, adding a unique tag per album
- add photo/album sharing
	- needs a web-based `Varda GUI` backend
	- needs a bit more design work (+security scrutiny)
- add support for internationalization
- add proper support for authentication/authorization