Damn deploy
===========

**Divide and map. Now.** -- the damn project -- helps mappers by dividing some
big area into smaller squares that a human can map.

The `damn-deploy` repository contains the complete `docker-compose`-based
deployment, including this deployment guide.

License
-------

The `damndb` and `upkeep` directories are published under [AGPLv3 License][].
The rest of this repository is published under [MIT License][].

[MIT License]: ./LICENSE
[AGPLv3 License]: ./damndb/LICENSE

Discussion
----------

Use the [mailing list][] for the discussion.

[mailing list]: https://lists.sr.ht/~qeef/damn-project


How to deploy
=============

Prerequisities for deployment are `git`, `docker` and `docker-compose`. The
guide is tested on Debian 10. You need some domain name directing to the
server.

Then, you need to clone the damn deploy repository:

    git clone https://git.sr.ht/~qeef/damn-deploy
    cd damn-deploy

There are two config files: `http.yml` specify all HTTP service images,
`gen.yml` specify images that generate content.

If you don't want to run some image (e.g. `chat` from `http.yml`,) remove the
section from the config file. You may later reset back with `git checkout -p`.

See _Autostart with `systemd`_ section to run the damn services automatically.
See _How to upgrade_ section for the upgrade process.

Core services
-------------

There are HTTP services within the `http.yml` docker compose file. `db`,
`traefik`, and `api` are the core services and must be always present.

1. Set environment variables in `.env` file. The following is the meaning:

    - `POSTGRES_PASSWORD`: Password to PostgreSQL database.
    - `JWT_SECRET`: Secret for generating, signing, and decrypting JSON Web
      Tokens.
    - `SESSION_SECRET`: Secret for cookies (I think).
    - `OAUTH_CONSUMER_KEY`: API key, get from osm.org.
    - `OAUTH_CONSUMER_SECRET`: API secret, get from osm.org.

    - `DOMAIN_NAME`: Use your domain.
    - `WWW_REPO`: Link to a docker git repository with web pages.

    The following are optinal settings. You don't need to change them. The
    defaults work good.

    - `SERVER_SUB`: The subdomain for the server.
    - `DAMN_SERVER_VERSION`: Choose the version of the server.
    - `DAMN_CLIENTS`: Allow origins clients.
    - `DB_HOST`: Alias of `db` docker service, keep that value.
    - `WWW_SUB`: The subdomain for the web pages.

2. Set proper email address in `traefik.yml` file.

3. Create `acme.json` file and set `600` permissions:

       touch acme.json
       chmod 600 acme.json

__NOTE:__ Feel free to tune `db` service parameters in `command` -- the values
here are for 1 GB RAM. See PostgreSQL documentation [Resource Consumption][]
and [Write Ahead Log][].

[Resource Consumption]: https://www.postgresql.org/docs/11/runtime-config-resource.html#GUC-SHARED-BUFFERS
[Write Ahead Log]: https://www.postgresql.org/docs/11/runtime-config-wal.html

Other services
--------------

These are other support services. Feel free to keep the default values.

1. Set environment variables in `.env` file. The following is the meaning:

    - `CHAT_SUB`: The subdomain for the chat.
    - `OSMSM_SUB`: The subdomain for background images used in mappy client.
    - `RSS_SUB`: The subdomain for RSS.
    - `FINISHED_SUB`: The subdomain for finished areas.

Web clients
-----------

There are HTTP clients in `http.yml`. The following are possible settings.

1. Set environment variables in `.env` file. The following is the meaning:

    - `DEFAULT_LANG`: The default language.
    - `DEFAULT_WEB_EDITOR`: Set the default editor to `id` or `rapid`.

    The following are optinal settings. You don't need to change them. The
    defaults work good.

    - `DAMN_MANAGER_REPO`: Link to the manager git repository.
    - `DAMN_MANAGER_VERSION`: Version of the manager.
    - `MANAGER_SUB`: The subdomain for the manager.

    - `DAMN_CLIENT_REPO`: Link to client git repository.
    - `DAMN_CLIENT_VERSION`: Version of the client.
    - `CLIENT_SUB`: The subdomain for the client.
    - `PANEL_SUB`: The subdomain for the panel client.
    - `MAPPY_SUB`: The subdomain for the mappy client.


Autostart with `systemd`
========================

Let's assume the damn deploy repository is cloned to `/root/damn-deploy/`.

1. Create systemd unit for `http.yml` services. This is the default unit. It's
   used on the damn-project.org.

       cat << EOF > /etc/systemd/system/damn.service
       [Unit]
       Description=Damn server app
       After=network.target docker.service

       [Service]
       Type=simple
       WorkingDirectory=/root/damn-deploy
       ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/http.yml up
       ExecStop=/usr/bin/docker-compose -f /root/damn-deploy/http.yml down

       [Install]
       WantedBy=multi-user.target
       EOF

2. Enable and run damn service:

       systemctl start damn.service
       systemctl enable damn.service

Damn upkeep
-----------

Damn upkeep docker image contains scripts that run periodically. Periodic run
must be set with `cron` or `systemd`. The following example uses `systemd`.

An example of upkeep script is square unlock when there is no activity in two
hours.

1. Create `systemd` unit and timer for damn upkeep:

       cat << EOF > /etc/systemd/system/damn_upkeep.service
       [Unit]
       Description=Run damn upkeep

       [Service]
       WorkingDirectory=/root/damn-deploy
       ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/gen.yml run --rm upkeep
       EOF
       cat << EOF > /etc/systemd/system/damn_upkeep.timer
       [Unit]
       Description=Run damn upkeep every 15 minutes

       [Timer]
       OnCalendar=*:0/15

       [Install]
       WantedBy=timers.target
       EOF

2. Enable and run damn upkeep service and timer:

       systemctl enable damn_upkeep.timer
       systemctl enable damn_upkeep.service

       systemctl start damn_upkeep.timer

The following is the description of the damn upkeep scripts.

- Automatically unlock square

  The `unlock_square.sql` upkeep script unlocks a square that is locked for
  more than two hours.

- Update areas statistics

  The `update_areas_statistics.sql` script updates _squares to map_, _squares
  to review_ and _squares done_ fields of all _active areas_ (areas with a
  priority greater than zero.)

- Delete archived areas

  When an area has priority `-10`, the `delete_areas_with_priority_-10.sql`
  script deletes such an area including all its squares and commits. This
  script is moved from `upkeep/scripts` to `upkeep` directory to avoid
  automatic run.  For deleting areas with priority `-10`, you must run the
  following command:

      psql -1 -f upkeep/delete_areas_with_priority_-10.sql "postgresql://damnuser:$(cat .env | sed -n 's/^POSTGRES_PASSWORD=\(.\+\)$/\1/p')@$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)/damndb"

  **NOTE:** It works only for the areas with priority `-10`. For other archived
  areas (areas with priority less than zero,) there is no special upkeep script
  yet.

  **NOTE:** The command works for user `damnuser` and database `damndb`. Update
  the command according to the needs.

Static Map background images
----------------------------

For _mappy_ client, the background image is used. The background image is
provided by `osmsm_www` service (set in `http.yml` file.) Generating of the
background images is done by `osmsm_gen` service (of the `gen.yml` file.)

`osmsm_gen` service should be run periodically, similar to `upkeep` service.
You may set up the `systemd` config files for running `osmsm_gen` every 30
minutes as the following:

1. Create systemd unit and timer for damn osmsm gen:

       cat << EOF > /etc/systemd/system/damn_osmsm_gen.service
       [Unit]
       Description=Run damn osmsm gen

       [Service]
       WorkingDirectory=/root/damn-deploy
       ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/gen.yml run --rm osmsm_gen
       EOF
       cat << EOF > /etc/systemd/system/damn_osmsm_gen.timer
       [Unit]
       Description=Run damn osmsm gen every 30 minutes

       [Timer]
       OnCalendar=*:0/30

       [Install]
       WantedBy=timers.target
       EOF

2. Enable and run damn upkeep service and timer:

       systemctl enable damn_osmsm_gen.timer
       systemctl enable damn_osmsm_gen.service

       systemctl start damn_osmsm_gen.timer

RSS feeds
---------

It's possible to generate and provide RSS feeds of last week changes.

1. Create systemd unit and timer for damn RSS gen:

       cat << EOF > /etc/systemd/system/damn_rss_gen.service
       [Unit]
       Description=Run damn RSS gen

       [Service]
       WorkingDirectory=/root/damn-deploy
       ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/gen.yml run --rm rss_gen
       EOF
       cat << EOF > /etc/systemd/system/damn_rss_gen.timer
       [Unit]
       Description=Run damn RSS gen every day at 3:33.

       [Timer]
       OnCalendar=03:33

       [Install]
       WantedBy=timers.target
       EOF

2. Enable and run damn upkeep service and timer:

       systemctl enable damn_rss_gen.timer
       systemctl enable damn_rss_gen.service

       systemctl start damn_rss_gen.timer


Intersecting areas check
------------------------

Periodically check and update intersecting areas, generate web page showing them.

1. Create systemd unit and timer for generation of intersecting areas:

       cat << EOF > /etc/systemd/system/damn_intersecting_gen.service
       [Unit]
       Description=Run generation of intersecting areas

       [Service]
       WorkingDirectory=/root/damn-deploy
       ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/gen.yml run --rm intersecting_gen
       EOF
       cat << EOF > /etc/systemd/system/damn_intersecting_gen.timer
       [Unit]
       Description=Run generation of intersecting areas every 30 minutes

       [Timer]
       OnCalendar=*:0/30

       [Install]
       WantedBy=timers.target
       EOF

2. Enable and run generation of intersecting area service and timer:

       systemctl enable damn_intersecting_gen.timer
       systemctl enable damn_intersecting_gen.service

       systemctl start damn_intersecting_gen.timer


How to upgrade
==============

1. If use `systemd` autostart, stop the damn service first:

       systemctl stop damn.service

2. Stash the configuration:

       git stash

3. Pull new version of the damn deploy repository:

       git pull

4. Pop stashed changes (renew the configuration):

       git stash pop

5. Maybe rebuild some docker images, but it depends on the upgrade changes:

       docker-compose -f http.yml build --no-cache api
       docker-compose -f http.yml build --no-cache www
       docker-compose -f http.yml build --no-cache manager
       docker-compose -f http.yml build --no-cache client
       docker-compose -f http.yml build --no-cache panel
       docker-compose -f http.yml build --no-cache mappy
       docker-compose -f http.yml build --no-cache finished_www

       docker-compose -f gen.yml build --no-cache upkeep
       docker-compose -f gen.yml build --no-cache osmsm_gen
       docker-compose -f gen.yml build --no-cache rss_gen

6. Remove old docker images:

       docker image prune

7. Start the damn service again:

       systemctl start damn.service


__NOTE:__ When the start of the service fails, it may be needed to delete all
the docker containers and start it again:

    systemctl stop damn.service
    for i in `docker ps -aq`; do docker stop $i; docker rm $i; done
    systemctl start damn.service


Upgrade the database structure
------------------------------

The database structure is rarely to change but it may happen. For new
deployments, there is nothing to do.

However, if you are upgrading from an existing database, you need to run the
upgrade script manually.

All the scripts related to the database structure are stored in `damndb`
directory, starting at `70_...`. Which script should be run when upgrading the
database structure is always noted in the annotated tag of the corresponding
version.

Upgrade to `v0.6.0`
-------------------

Stop the damn service and update the repository:

    systemctl stop damn.service
    git stash
    git pull
    git stash pop

Run the database:

    docker-compose up -d db

Run the database upgrade script (you will be asked for the password stored in
`.env` file):

    export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
    psql -1 -h $DB_HOST -d damndb -U damnuser < damndb/71_areas_squares_stats.sql

Stop the database container:

    docker-compose down

Finally, there is no need to build the database container again. The database
is created only once anyway, so run the damn service again:

    systemctl start damn.service

Upgrade to `v0.7.0`
-------------------

When the upgrade is finished, update the `current_commits` in the database.
Connect to the database (the password is in `.env` file):

    export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
    psql -h $DB_HOST -d damndb -U damnuser

Run the update query:

    update current_commits set type='splitted' where message='The square was splitted';

Upgrade to `v0.8.0`
-------------------

This upgrade introduce change of the repository source and directories
renaming. The best is to stop the running service first:

    systemctl stop damn.service

Make sure all the containers are stopped:

    cd damn_deploy
    docker-compose -f docker-compose.yml stop
    docker-compose -f docker-compose-clients.yml stop
    docker-compose -f docker-compose-chat.yml stop
    cd ..

And then follow the *How to deploy* section (I assume the current directory is
`/root/`:)

    git clone https://git.sr.ht/~qeef/damn-deploy
    cd damn-deploy

Then copy secrets and configuration from `../damn_deploy/.env` to `.env` file.
(The `.env` changed a bit!) Also, update `traefik.yml` and copy
`../damn_deploy/acme.json` to `acme.json`.

The next step is to rewrite systemd config files. From the *Autostart with
`systemd`* section, run the command to *Create systemd unit for
`docker-compose`*:

    cat << EOF > /etc/systemd/system/damn.service
    ...

And from the *Damn upkeep* section, run the command to *Create systemd unit and
timer for damn upkeep*:

    cat << EOF > /etc/systemd/system/damn_upkeep.service
    ...

Rebuild all the docker images with:

    docker-compose -f http.yml build --no-cache api
    docker-compose -f http.yml build --no-cache upkeep
    docker-compose -f clients.yml build --no-cache client
    docker-compose -f clients.yml build --no-cache manager
    docker-compose -f chat.yml build --no-cache chat

Copy old database volume to the new database volume:

    docker volume create --name damn-deploy_damndb-volume
    docker run --rm -v damn_deploy_damndb-volume:/from -v damn-deploy_damndb-volume:/to alpine ash -c 'cd /from ; cp -av . /to'

Remove `damndb` container and rebuild it. It does **nothing** to the database.

    docker rm damndb
    docker-compose -f http.yml build --no-cache db

Reload systemd units and start the service:

    systemctl daemon-reload
    systemctl start damn.service

Now, if everything works and everything is tested, you may remove old database
volume:

    docker volume rm damn_deploy_damndb-volume

Upgrade to `v0.12.0`
--------------------

This upgrade introduces custom square width and height when creating areas. The
function is stored in `damndb/72_st_area_split_custom_square_size.sql` file.
The steps to upgrade follow.

Stop the damn service and update the repository:

    systemctl stop damn.service
    git stash
    git pull
    git stash pop

Run the database:

    docker-compose -f http.yml up -d db

Import the new function into the database (you will be asked for the password
stored in `.env` file):

    export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
    psql -1 -h $DB_HOST -d damndb -U damnuser < damndb/72_st_area_split_custom_square_size.sql

Stop the database container:

    docker-compose -f http.yml down

Finally, there is no need to build the database container again. The database
is created only once anyway. However, `server`, `client`, and `manager` need to
be updated:

    docker-compose -f http.yml build --no-cache api
    docker-compose -f clients.yml build --no-cache client
    docker-compose -f clients.yml build --no-cache manager

Finally, run the damn service again:

    systemctl start damn.service

Upgrade to `v0.17.0`
-------------------

There is new `st_divide` function in `damndb/73_st_divide.sql` file. Follow the
steps to put new function into the database. Stop the damn service and update
the repository:

    systemctl stop damn.service
    git stash
    git pull
    git stash pop

Run the database:

    docker-compose -f http.yml up -d db

Run the database upgrade script (you will be asked for the password stored in
`.env` file):

    export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
    psql -1 -h $DB_HOST -d damndb -U damnuser < damndb/73_st_divide.sql

Stop the database container:

    docker-compose -f http.yml down

Follow the common upgrade process in the _How to upgrade_ section. (You need to
re-build docker images for server and manager as these two were changed.)

Upgrade to v0.17.7
------------------

The `st_divide` function is changed a little bit, so it's needed to re-run the
steps from _Ugrade to v0.17.0_ section.

Upgrade to v0.20.0
------------------

There are few improvements to database in `v0.20.0`. Add `author` field to
`current_areas`. (Until now, the `author` of the area is got by checking the
first commit of the first square.) And improve the performance by indexing. The
steps to upgrade follow.

Stop the damn service and update the repository:

    systemctl stop damn.service
    git stash
    git pull
    git stash pop

Run the database:

    docker-compose -f http.yml up -d db

Update the database structure. Note that you will be asked for the database
password stored in `.env` file:

    psql -1 -h $(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb) -d damndb -U damnuser < damndb/74_current_areas_author.sql
    psql -1 -h $(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb) -d damndb -U damnuser < damndb/75_add_indexes.sql

Stop the database container:

    docker-compose -f http.yml down

Rebuild changed images:

    docker-compose -f http.yml build --no-cache api
    docker-compose -f http.yml build --no-cache manager
    docker-compose -f gen.yml build --no-cache osmsm_gen

Finally, run the damn service again:

    systemctl start damn.service

Upgrade to v0.20.1
------------------

The `75_add_indexes.sql` is changed a little bit, so it's needed to re-run the
steps from _Upgrade to v0.20.0_ for `75_add_indexes.sql`.

Upgrade to v0.21.0
------------------

The `st_divide` function is changed a little bit, so it needs to be updated.

Stop the damn service and update the repository:

    systemctl stop damn.service
    git stash
    git pull
    git stash pop

Run the database container:

    docker-compose -f http.yml up -d db

Update the database:

    psql -1 -f damndb/73_st_divide.sql "postgresql://damnuser:$(cat .env | sed -n 's/^POSTGRES_PASSWORD=\(.\+\)$/\1/p')@$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)/damndb"

Stop the database container:

    docker-compose -f http.yml down

Rebuild changed images:

    docker-compose -f http.yml build --no-cache api
    docker-compose -f http.yml build --no-cache manager
    docker-compose -f http.yml build --no-cache client
    docker-compose -f http.yml build --no-cache panel
    docker-compose -f http.yml build --no-cache mappy

Finally, run the damn service again:

    systemctl start damn.service

Upgrade to v0.22.0
------------------

There is a new `square_from_already_divided` function, so database needs to be
updated.

Stop the damn service and update the repository:

    systemctl stop damn.service
    git stash
    git pull
    git stash pop

Run the database container:

    docker-compose -f http.yml up -d db

Update the database:

    psql -1 -f damndb/76_square_from_already_divided.sql "postgresql://damnuser:$(cat .env | sed -n 's/^POSTGRES_PASSWORD=\(.\+\)$/\1/p')@$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)/damndb"

Stop the database container:

    docker-compose -f http.yml down

Rebuild changed images:

    docker-compose -f http.yml build --no-cache api

Finally, run the damn service again:

    systemctl start damn.service