Damn deploy
===========

**Divide and map. Now.** -- the damn project -- helps mappers by dividing some
big area into smaller squares that a human can map.

The `damn-deploy` repository contains the complete `docker-compose`-based
deployment, including this deployment guide.

License
-------

The `damndb` and `upkeep` directories are published under [AGPLv3 License][].
The rest of this repository is published under [MIT License][].

[MIT License]: ./LICENSE
[AGPLv3 License]: ./damndb/LICENSE

Discussion
----------

Use the [mailing list][] for the discussion.

[mailing list]: https://lists.sr.ht/~qeef/damn-project


How to deploy
=============

Prerequisities for deployment are `git`, `docker` and `docker-compose`. The
guide is tested on Debian 10. You need some domain name directing to the
server.

Then, you need to clone the damn deploy repository:

    git clone https://git.sr.ht/~qeef/damn-deploy
    cd damn-deploy

The most common usecase is *autostart server, clients, and chat*. The following
sections consider *server* setup, setup for *clients* (client and manager),
*chat*, and *autostart* with `systemd`. Then, there is a *How to upgrade*
section that discuss the upgrade/update process. Finally, see the *Damn upkeep*
section for the database periodic upkeep scripts.

Server setup
------------

There are few services within the `server.yml` docker compose file. There is
`db` and `api` services. These are the core services. `db` serves the database
and the `api` is the JSON REST (Fast)API to the database. Then, there is the
`upkeep` that runs periodic tasks on the database. (See the *Damn upkeep*
section.) The next is the `www` service serving the web pages. Finally, the
`traefik` service provides the gate to all the services above.

1. Set environment variables in `.env` file. The following is the meaning:

    - `POSTGRES_PASSWORD`: Password to PostgreSQL database.
    - `JWT_SECRET`: Secret for generating, signing, and decrypting JSON Web Tokens.
    - `SESSION_SECRET`: Secret for cookies (I think).
    - `OAUTH_CONSUMER_KEY`: API key, get from osm.org.
    - `OAUTH_CONSUMER_SECRET`: API secret, get from osm.org.

    - `DOMAIN_NAME`: Use your domain.
    - `WWW_REPO`: Link to a docker git repository with web pages.

    The following are optinal settings. You don't need to change them. The
    defaults work good.

    - `SERVER_SUB`: The subdomain for the server.
    - `DAMN_SERVER_VERSION`: Choose the version of the server.
    - `DAMN_CLIENTS`: Allow origins clients.
    - `DB_HOST`: Alias of `db` docker service, keep that value.
    - `WWW_SUB`: The subdomain for the web pages.

2. Set proper email address in `traefik.yml` file.

3. Create `acme.json` file and set `600` permissions:

       touch acme.json
       chmod 600 acme.json

4. You may run the damn server now.

       docker-compose -f server.yml up

Setup for clients
-----------------

In addition to *Server setup*, you may want to run a client and a manager, too.

1. Set environment variables in `.env` file. The following are optinal
    settings. You don't need to change them. The defaults work good. There is
    the meaning of the customization settings:

    - `DEFAULT_LANG`: The default language.
    - `DEFAULT_AREAS_VIEW`: Show areas on first page by default as `grid` or
      `list`?
    - `DEFAULT_WEB_EDITOR`: Set the default editor to `id` or `rapid-esri`.

    And the meaning of the optional settings for the clients:

    - `CLIENT_SUB`: The subdomain for the client.
    - `DAMN_CLIENT_REPO`: Link to client git repository.
    - `DAMN_CLIENT_VERSION`: Version of the client.
    - `MANAGER_SUB`: The subdomain for the manager.
    - `DAMN_MANAGER_REPO`: Link to the manager git repository.
    - `DAMN_MANAGER_VERSION`: Version of the manager.
    - `PANEL_SUB`: The subdomain for the panel client.
    - `DAMN_PANEL_REPO`: Link to panel client git repository.
    - `DAMN_PANEL_VERSION`: Version of the panel client.

2. You may run the damn server with clients now.

       docker-compose -f server.yml -f clients.yml up

Setup for chat
--------------

In addition to *Server setup*, you may want to run a chat server.

1. Set environment variable in `.env` file.

    - `CHAT_SUB`: The subdomain for the chat.

2. Run the chat server with the server:

       docker-compose -f server.yml -f chat.yml up

   or also with clients:

       docker-compose -f server.yml -f clients.yml -f chat.yml up

Autostart with `systemd`
------------------------

I assume that the damn deploy repository is cloned to `/root/damn-deploy/`.

1. Create systemd unit for `docker-compose`:

       cat << EOF > /etc/systemd/system/damn.service
       [Unit]
       Description=Damn server app
       After=network.target docker.service

       [Service]
       Type=simple
       WorkingDirectory=/root/damn-deploy
       ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/server.yml -f /root/damn-deploy/clients.yml -f /root/damn-deploy/chat.yml up
       ExecStop=/usr/bin/docker-compose -f /root/damn-deploy/server.yml -f /root/damn-deploy/clients.yml -f /root/damn-deploy/chat.yml down

       [Install]
       WantedBy=multi-user.target
       EOF

   Remove ` -f /root/damn-deploy/clients.yml` when running server only.

2. Enable and run damn service:

       systemctl start damn.service
       systemctl enable damn.service


How to upgrade
==============

1. If use `systemd` autostart, stop the damn service first:

       systemctl stop damn.service

2. Stash the configuration:

       git stash

3. Pull new version of the damn deploy repository:

       git pull

4. Pop stashed changes (renew the configuration):

       git stash pop

5. Maybe rebuild some docker images, but it depends on the upgrade changes:

       docker-compose -f server.yml build --no-cache api
       docker-compose -f server.yml build --no-cache upkeep
       docker-compose -f clients.yml build --no-cache client
       docker-compose -f clients.yml build --no-cache manager
       docker-compose -f clients.yml build --no-cache panel
       docker-compose -f chat.yml build --no-cache chat

6. Start the damn service again:

       systemctl start damn.service

Upgrade the database structure
------------------------------

The database structure is rarely to change but it may happen. For new
deployments, there is nothing to do.

However, if you are upgrading from an existing database, you need to run the
upgrade script manually.

All the scripts related to the database structure are stored in `damndb`
directory, starting at `70_...`. Which script should be run when upgrading the
database structure is always noted in the [changelog][].

Upgrade to `v0.6.0`
-------------------

Stop the damn service and update the repository:

    systemctl stop damn.service
    git stash
    git pull
    git stash pop

Run the database:

    docker-compose up -d db

Run the database upgrade script (you will be asked for the password stored in
`.env` file):

    export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
    psql -h $DB_HOST -d damndb -U damnuser < damndb/71_areas_squares_stats.sql

Stop the database container:

    docker-compose down

Finally, there is no need to build the database container again. The database
is created only once anyway, so run the damn service again:

    systemctl start damn.service

Upgrade to `v0.7.0`
-------------------

When the upgrade is finished, update the `current_commits` in the database.
Connect to the database (the password is in `.env` file):

    export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
    psql -h $DB_HOST -d damndb -U damnuser

Run the update query:

    update current_commits set type='splitted' where message='The square was splitted';

Upgrade to `v0.8.0`
-------------------

This upgrade introduce change of the repository source and directories
renaming. The best is to stop the running service first:

    systemctl stop damn.service

Make sure all the containers are stopped:

    cd damn_deploy
    docker-compose -f docker-compose.yml stop
    docker-compose -f docker-compose-clients.yml stop
    docker-compose -f docker-compose-chat.yml stop
    cd ..

And then follow the *How to deploy* section (I assume the current directory is
`/root/`:)

    git clone https://git.sr.ht/~qeef/damn-deploy
    cd damn-deploy

Then copy secrets and configuration from `../damn_deploy/.env` to `.env` file.
(The `.env` changed a bit!) Also, update `traefik.yml` and copy
`../damn_deploy/acme.json` to `acme.json`.

The next step is to rewrite systemd config files. From the *Autostart with
`systemd`* section, run the command to *Create systemd unit for
`docker-compose`*:

    cat << EOF > /etc/systemd/system/damn.service
    ...

And from the *Damn upkeep* section, run the command to *Create systemd unit and
timer for damn upkeep*:

    cat << EOF > /etc/systemd/system/damn_upkeep.service
    ...

Rebuild all the docker images with:

    docker-compose -f server.yml build --no-cache api
    docker-compose -f server.yml build --no-cache upkeep
    docker-compose -f clients.yml build --no-cache client
    docker-compose -f clients.yml build --no-cache manager
    docker-compose -f chat.yml build --no-cache chat

Copy old database volume to the new database volume:

    docker volume create --name damn-deploy_damndb-volume
    docker run --rm -v damn_deploy_damndb-volume:/from -v damn-deploy_damndb-volume:/to alpine ash -c 'cd /from ; cp -av . /to'

Remove `damndb` container and rebuild it. It does **nothing** to the database.

    docker rm damndb
    docker-compose -f server.yml build --no-cache db

Reload systemd units and start the service:

    systemctl daemon-reload
    systemctl start damn.service

Now, if everything works and everything is tested, you may remove old database
volume:

    docker volume rm damn_deploy_damndb-volume

Upgrade to `v0.12.0`
--------------------

This upgrade introduces custom square width and height when creating areas. The
function is stored in `damndb/72_st_area_split_custom_square_size.sql` file.
The steps to upgrade follow.

Stop the damn service and update the repository:

    systemctl stop damn.service
    git stash
    git pull
    git stash pop

Run the database:

    docker-compose -f server.yml up -d db

Import the new function into the database (you will be asked for the password
stored in `.env` file):

    export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
    psql -h $DB_HOST -d damndb -U damnuser < damndb/72_st_area_split_custom_square_size.sql

Stop the database container:

    docker-compose -f server.yml down

Finally, there is no need to build the database container again. The database
is created only once anyway. However, `server`, `client`, and `manager` need to
be updated:

    docker-compose -f server.yml build --no-cache api
    docker-compose -f clients.yml build --no-cache client
    docker-compose -f clients.yml build --no-cache manager

Finally, run the damn service again:

    systemctl start damn.service


Damn upkeep
===========

Damn upkeep docker image contains scripts that run periodically. Periodic run
must be set with `cron` or `systemd`. The following example uses `systemd`.

An example of upkeep script is square unlock when there is no activity in two
hours.

1. Create systemd unit and timer for damn upkeep:

       cat << EOF > /etc/systemd/system/damn_upkeep.service
       [Unit]
       Description=Run damn upkeep

       [Service]
       WorkingDirectory=/root/damn-deploy
       ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/server.yml run --rm upkeep
       EOF
       cat << EOF > /etc/systemd/system/damn_upkeep.timer
       [Unit]
       Description=Run damn upkeep every 15 minutes

       [Timer]
       OnCalendar=*:0/15

       [Install]
       WantedBy=timers.target
       EOF

2. Enable and run damn upkeep service and timer:

       systemctl enable damn_upkeep.timer
       systemctl enable damn_upkeep.service

       systemctl start damn_upkeep.timer

Automatically unlock square
---------------------------

The `unlock_square.sql` upkeep script unlocks a square that is locked for more
than two hours.

Update areas statistics
-----------------------

The `update_areas_statistics.sql` script updates *squares to map*, *squares to
review* and *squares done* fields of all *current* areas (areas with a priority
greater than zero.)

Delete archived areas
---------------------

When an area has priority `-10`, the `delete_areas_with_priority_-10.sql`
script deletes such an area including all its squares and commits during the
upkeep.

**NOTE:** It works only for the areas with priority `-10`. For other archived
areas (areas with priority less than zero,) there is no special upkeep script
yet.