~qeef/damn-deploy: README.md

#Damn deploy

Divide and map. Now. -- the damn project -- helps mappers by dividing some big area into smaller squares that a human can map.

The damn-deploy repository contains the complete docker-compose-based deployment, including this deployment guide.

#License

The damndb and upkeep directories are published under AGPLv3 License. The rest of this repository is published under MIT License.

#Discussion

Use the mailing list for the discussion.

#How to deploy

Prerequisities for deployment are git, docker and docker-compose. The guide is tested on Debian 10. You need some domain name directing to the server.

Then, you need to clone the damn deploy repository:

git clone https://git.sr.ht/~qeef/damn-deploy
cd damn-deploy

You probably want to run server, clients, osmsm, www, and upkeep with systemd. Follow the following sub-sections to see the docker-compose configs. The last sub-section is about systemd.

See How to upgrade section for the upgrade process, Damn upkeep section for the database periodic upkeep scripts, and Static Map background images section for serving background images used in the mappy client.

#Server setup

There are few services within the server.yml docker compose file. There is db and api services. These are the core services. db serves the database and the api is the JSON REST (Fast)API to the database. Then, there is the upkeep that runs periodic tasks on the database. (See the Damn upkeep section.) The next is the www service serving the web pages. Finally, the traefik service provides the gate to all the services above.

Set environment variables in .env file. The following is the meaning:
- POSTGRES_PASSWORD: Password to PostgreSQL database.
- JWT_SECRET: Secret for generating, signing, and decrypting JSON Web Tokens.
- SESSION_SECRET: Secret for cookies (I think).
- OAUTH_CONSUMER_KEY: API key, get from osm.org.
- OAUTH_CONSUMER_SECRET: API secret, get from osm.org.
- DOMAIN_NAME: Use your domain.
- WWW_REPO: Link to a docker git repository with web pages.
The following are optinal settings. You don't need to change them. The defaults work good.
- SERVER_SUB: The subdomain for the server.
- DAMN_SERVER_VERSION: Choose the version of the server.
- DAMN_CLIENTS: Allow origins clients.
- DB_HOST: Alias of db docker service, keep that value.
- WWW_SUB: The subdomain for the web pages.
Set proper email address in traefik.yml file.
Create acme.json file and set 600 permissions:
```
touch acme.json
chmod 600 acme.json
```
You may run the damn server now.
```
docker-compose -f server.yml up
```

#Setup for clients

In addition to Server setup, you may want to run the clients and the manager.

Set environment variables in .env file. The following are optinal settings. You don't need to change them. The defaults work good. There is the meaning of the customization settings:
- DEFAULT_LANG: The default language.
- DEFAULT_WEB_EDITOR: Set the default editor to id or rapid.
And the meaning of the optional settings for the clients:
- CLIENT_SUB: The subdomain for the client.
- DAMN_CLIENT_REPO: Link to client git repository.
- DAMN_CLIENT_VERSION: Version of the client.
- PANEL_SUB: The subdomain for the panel client.
- MAPPY_SUB: The subdomain for the mappy client.
- OSMSM_SUB: The subdomain for background images used in mappy client.
- MANAGER_SUB: The subdomain for the manager.
- DAMN_MANAGER_REPO: Link to the manager git repository.
- DAMN_MANAGER_VERSION: Version of the manager.

You may run the damn server with clients now.

docker-compose -f server.yml -f clients.yml up

#Setup for chat

In addition to Server setup, you may want to run the chat server.

Set environment variable in .env file.
- CHAT_SUB: The subdomain for the chat.

Run the chat server with the server:

docker-compose -f server.yml -f chat.yml up

or also with clients:

docker-compose -f server.yml -f clients.yml -f chat.yml up

#Autostart with `systemd`

Let's assume the damn deploy repository is cloned to /root/damn-deploy/.

Create systemd unit for docker-compose:

cat << EOF > /etc/systemd/system/damn.service
[Unit]
Description=Damn server app
After=network.target docker.service

[Service]
Type=simple
WorkingDirectory=/root/damn-deploy
ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/server.yml -f /root/damn-deploy/clients.yml -f /root/damn-deploy/chat.yml -f /root/damn-deploy/osmsm.ymlup
ExecStop=/usr/bin/docker-compose -f /root/damn-deploy/server.yml -f /root/damn-deploy/clients.yml -f /root/damn-deploy/chat.yml -f /root/damn-deploy/osmsm.yml down

[Install]
WantedBy=multi-user.target
EOF

The unit above is the default used on the damn-project.org. Feel free to remove some -f LINK/TO/FILE.yml for services you don't want to run.

Enable and run damn service:

systemctl start damn.service
systemctl enable damn.service

#How to upgrade

If use systemd autostart, stop the damn service first:
```
systemctl stop damn.service
```
Stash the configuration:
```
git stash
```
Pull new version of the damn deploy repository:
```
git pull
```
Pop stashed changes (renew the configuration):
```
git stash pop
```

Maybe rebuild some docker images, but it depends on the upgrade changes:

docker-compose -f server.yml build --no-cache api
docker-compose -f server.yml build --no-cache upkeep
docker-compose -f server.yml build --no-cache www
docker-compose -f clients.yml build --no-cache client
docker-compose -f clients.yml build --no-cache panel
docker-compose -f clients.yml build --no-cache mappy
docker-compose -f clients.yml build --no-cache manager

Start the damn service again:
```
systemctl start damn.service
```

#Upgrade the database structure

The database structure is rarely to change but it may happen. For new deployments, there is nothing to do.

However, if you are upgrading from an existing database, you need to run the upgrade script manually.

All the scripts related to the database structure are stored in damndb directory, starting at 70_.... Which script should be run when upgrading the database structure is always noted in the [changelog][].

#Upgrade to `v0.6.0`

Stop the damn service and update the repository:

systemctl stop damn.service
git stash
git pull
git stash pop

Run the database:

docker-compose up -d db

Run the database upgrade script (you will be asked for the password stored in .env file):

export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
psql -h $DB_HOST -d damndb -U damnuser < damndb/71_areas_squares_stats.sql

Stop the database container:

docker-compose down

Finally, there is no need to build the database container again. The database is created only once anyway, so run the damn service again:

systemctl start damn.service

#Upgrade to `v0.7.0`

When the upgrade is finished, update the current_commits in the database. Connect to the database (the password is in .env file):

export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
psql -h $DB_HOST -d damndb -U damnuser

Run the update query:

update current_commits set type='splitted' where message='The square was splitted';

#Upgrade to `v0.8.0`

This upgrade introduce change of the repository source and directories renaming. The best is to stop the running service first:

systemctl stop damn.service

Make sure all the containers are stopped:

cd damn_deploy
docker-compose -f docker-compose.yml stop
docker-compose -f docker-compose-clients.yml stop
docker-compose -f docker-compose-chat.yml stop
cd ..

And then follow the How to deploy section (I assume the current directory is /root/:)

git clone https://git.sr.ht/~qeef/damn-deploy
cd damn-deploy

Then copy secrets and configuration from ../damn_deploy/.env to .env file. (The .env changed a bit!) Also, update traefik.yml and copy ../damn_deploy/acme.json to acme.json.

The next step is to rewrite systemd config files. From the Autostart with systemd section, run the command to Create systemd unit for docker-compose:

cat << EOF > /etc/systemd/system/damn.service
...

And from the Damn upkeep section, run the command to Create systemd unit and timer for damn upkeep:

cat << EOF > /etc/systemd/system/damn_upkeep.service
...

Rebuild all the docker images with:

docker-compose -f server.yml build --no-cache api
docker-compose -f server.yml build --no-cache upkeep
docker-compose -f clients.yml build --no-cache client
docker-compose -f clients.yml build --no-cache manager
docker-compose -f chat.yml build --no-cache chat

Copy old database volume to the new database volume:

docker volume create --name damn-deploy_damndb-volume
docker run --rm -v damn_deploy_damndb-volume:/from -v damn-deploy_damndb-volume:/to alpine ash -c 'cd /from ; cp -av . /to'

Remove damndb container and rebuild it. It does nothing to the database.

docker rm damndb
docker-compose -f server.yml build --no-cache db

Reload systemd units and start the service:

systemctl daemon-reload
systemctl start damn.service

Now, if everything works and everything is tested, you may remove old database volume:

docker volume rm damn_deploy_damndb-volume

#Upgrade to `v0.12.0`

This upgrade introduces custom square width and height when creating areas. The function is stored in damndb/72_st_area_split_custom_square_size.sql file. The steps to upgrade follow.

Stop the damn service and update the repository:

systemctl stop damn.service
git stash
git pull
git stash pop

Run the database:

docker-compose -f server.yml up -d db

Import the new function into the database (you will be asked for the password stored in .env file):

export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
psql -h $DB_HOST -d damndb -U damnuser < damndb/72_st_area_split_custom_square_size.sql

Stop the database container:

docker-compose -f server.yml down

Finally, there is no need to build the database container again. The database is created only once anyway. However, server, client, and manager need to be updated:

docker-compose -f server.yml build --no-cache api
docker-compose -f clients.yml build --no-cache client
docker-compose -f clients.yml build --no-cache manager

Finally, run the damn service again:

systemctl start damn.service

#Upgrade to `v0.17.0`

There is new st_divide function in damndb/73_st_divide.sql file. Follow the steps to put new function into the database. Stop the damn service and update the repository:

systemctl stop damn.service
git stash
git pull
git stash pop

Run the database:

docker-compose -f server.yml up -d db

Run the database upgrade script (you will be asked for the password stored in .env file):

export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
psql -h $DB_HOST -d damndb -U damnuser < damndb/73_st_divide.sql

Stop the database container:

docker-compose -f server.yml down

Follow the common upgrade process in the How to upgrade section. (You need to re-build docker images for server and manager as these two were changed.)

#Upgrade to v0.17.7

The st_divide function is changed a little bit, so it's needed to re-run the steps from Ugrade to v0.17.0 section.

#Damn upkeep

Damn upkeep docker image contains scripts that run periodically. Periodic run must be set with cron or systemd. The following example uses systemd.

An example of upkeep script is square unlock when there is no activity in two hours.

Create systemd unit and timer for damn upkeep:

cat << EOF > /etc/systemd/system/damn_upkeep.service
[Unit]
Description=Run damn upkeep

[Service]
WorkingDirectory=/root/damn-deploy
ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/server.yml run --rm upkeep
EOF
cat << EOF > /etc/systemd/system/damn_upkeep.timer
[Unit]
Description=Run damn upkeep every 15 minutes

[Timer]
OnCalendar=*:0/15

[Install]
WantedBy=timers.target
EOF

Enable and run damn upkeep service and timer:

systemctl enable damn_upkeep.timer
systemctl enable damn_upkeep.service

systemctl start damn_upkeep.timer

#Automatically unlock square

The unlock_square.sql upkeep script unlocks a square that is locked for more than two hours.

#Update areas statistics

The update_areas_statistics.sql script updates squares to map, squares to review and squares done fields of all current areas (areas with a priority greater than zero.)

#Delete archived areas

When an area has priority -10, the delete_areas_with_priority_-10.sql script deletes such an area including all its squares and commits. This script is moved from upkeep/scripts to upkeep directory to avoid automatic run. For deleting areas with priority -10, you must run the following command:

psql -f upkeep/delete_areas_with_priority_-10.sql "postgresql://damnuser:$(cat .env | sed -n 's/^POSTGRES_PASSWORD=\(.\+\)$/\1/p')@$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)/damndb"

NOTE: It works only for the areas with priority -10. For other archived areas (areas with priority less than zero,) there is no special upkeep script yet.

NOTE: The command works for user damnuser and database damndb. Update the script according to the needs.

#Static Map background images

For mappy client, the background image is used. The background image is provided by osmsm_www service (set in osmsm.yml file.) Generating of the background images is done by osmsm_gen service (of the same file.)

osmsm_www service is similar to www service from server.yml. To run the osmsm_www service along with the rest of the damn services automatically, you need to add -f /root/damn-deploy/osmsm.yml to ExecStart and ExecStop of /etc/systemd/system/damn.service.

osmsm_gen service should be run periodically, similar to upkeep service. You may set up the systemd config files for running osmsm_gen every day at 12:12 as the following:

Create systemd unit and timer for damn osmsm gen:

cat << EOF > /etc/systemd/system/damn_osmsm_gen.service
[Unit]
Description=Run damn osmsm gen

[Service]
WorkingDirectory=/root/damn-deploy
ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/osmsm.yml run --rm osmsm_gen
EOF
cat << EOF > /etc/systemd/system/damn_osmsm_gen.timer
[Unit]
Description=Run damn osmsm gen every day at 12:12.

[Timer]
OnCalendar=12:12

[Install]
WantedBy=timers.target
EOF

Enable and run damn upkeep service and timer:

systemctl enable damn_osmsm_gen.timer
systemctl enable damn_osmsm_gen.service

systemctl start damn_osmsm_gen.timer

#RSS feeds

It's possible to generate and provide RSS feeds of last week changes.

Create systemd unit and timer for damn RSS gen:

cat << EOF > /etc/systemd/system/damn_rss_gen.service
[Unit]
Description=Run damn RSS gen

[Service]
WorkingDirectory=/root/damn-deploy
ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/rss.yml run --rm rss_gen
EOF
cat << EOF > /etc/systemd/system/damn_rss_gen.timer
[Unit]
Description=Run damn RSS gen every day at 3:33.

[Timer]
OnCalendar=03:33

[Install]
WantedBy=timers.target
EOF

Enable and run damn upkeep service and timer:

systemctl enable damn_rss_gen.timer
systemctl enable damn_rss_gen.service

systemctl start damn_rss_gen.timer

~qeef/damn-deploy

#Damn deploy

#License

#Discussion

#How to deploy

#Server setup

#Setup for clients

#Setup for chat

#Autostart with systemd

#How to upgrade

#Upgrade the database structure

#Upgrade to v0.6.0

#Upgrade to v0.7.0

#Upgrade to v0.8.0

#Upgrade to v0.12.0

#Upgrade to v0.17.0

#Upgrade to v0.17.7

#Damn upkeep

#Automatically unlock square

#Update areas statistics

#Delete archived areas

#Static Map background images

#RSS feeds

#Autostart with `systemd`

#Upgrade to `v0.6.0`

#Upgrade to `v0.7.0`

#Upgrade to `v0.8.0`

#Upgrade to `v0.12.0`

#Upgrade to `v0.17.0`