~qeef/damn-deploy

Divide and map. Now. -- the damn project -- helps mappers by dividing a big area into smaller squares that people can map together.

damn-deploy contains docker-compose-based deployment, including this deployment guide.

The damndb and upkeep directories are published under AGPLv3 License. The rest of this repository is published under MIT License.

#Discussion

Use the mailing list for the discussion.

#How to deploy

Prerequisities for deployment are git, docker and docker-compose. The guide is tested on Debian 10. You need some domain name directing to the server.

Then, you need to clone the damn deploy repository:

git clone https://git.sr.ht/~qeef/damn-deploy
cd damn-deploy

There are two config files: http.yml specify all HTTP service images, gen.yml specify images that generate content.

If you don't want to run some image (e.g. chat from http.yml,) remove the section from the config file. You may later reset back with git checkout -p.

See Autostart with systemd section to run the damn services automatically. See How to upgrade section for the upgrade process.

#Core services

There are HTTP services within the http.yml docker compose file. db, traefik, and api are the core services and must be always present.

Set environment variables in .env file. The following is the meaning:
- POSTGRES_PASSWORD: Password to PostgreSQL database.
- JWT_SECRET: Secret for generating, signing, and decrypting JSON Web Tokens.
- SESSION_SECRET: Secret for cookies (I think).
- OAUTH_CONSUMER_KEY: API key, get from osm.org.
- OAUTH_CONSUMER_SECRET: API secret, get from osm.org.
- DOMAIN_NAME: Use your domain.
- WWW_REPO: Link to a docker git repository with web pages.
The following are optinal settings. You don't need to change them. The defaults work good.
- SERVER_SUB: The subdomain for the server.
- DAMN_SERVER_VERSION: Choose the version of the server.
- DAMN_CLIENTS: Allow origins clients.
- DB_HOST: Alias of db docker service, keep that value.
Set proper email address in traefik.yml file.
Create acme.json file and set 600 permissions:
```
touch acme.json
chmod 600 acme.json
```

NOTE: Feel free to tune db service parameters in command -- the values here are for 1 GB RAM. See PostgreSQL documentation Resource Consumption and Write Ahead Log.

#Other services

These are other support services. Feel free to keep the default values.

Set environment variables in .env file. The following is the meaning:
- CHAT_SUB: The subdomain for the chat.

#Web clients

There are HTTP clients in http.yml. The following are possible settings.

Set environment variables in .env file. The following is the meaning:
- DEFAULT_LANG: The default language.
- DEFAULT_WEB_EDITOR: Set the default editor to id or rapid.
The following are optinal settings. You don't need to change them. The defaults work good.
- DAMN_MANAGER_REPO: Link to the manager git repository.
- DAMN_MANAGER_VERSION: Version of the manager.
- MANAGER_SUB: The subdomain for the manager.
- DAMN_CLIENT_REPO: Link to client git repository.
- DAMN_CLIENT_VERSION: Version of the client.
- CLIENT_SUB: The subdomain for the client.
- PANEL_SUB: The subdomain for the panel client.
- MAPPY_SUB: The subdomain for the mappy client.

#Autostart with `systemd`

Let's assume the damn deploy repository is cloned to /root/damn-deploy/.

Create systemd unit for http.yml services. This is the default unit. It's used on the damn-project.org.

cat << EOF > /etc/systemd/system/damn.service
[Unit]
Description=Damn server app
After=network.target docker.service

[Service]
Type=simple
WorkingDirectory=/root/damn-deploy
ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/http.yml up
ExecStop=/usr/bin/docker-compose -f /root/damn-deploy/http.yml down

[Install]
WantedBy=multi-user.target
EOF

Enable and run damn service:

systemctl start damn.service
systemctl enable damn.service

#Damn upkeep

Damn upkeep docker image contains scripts that run periodically. Periodic run must be set with cron or systemd. The following example uses systemd.

An example of upkeep script is square unlock when there is no activity in two hours.

Create systemd unit and timer for damn upkeep:

cat << EOF > /etc/systemd/system/damn_upkeep.service
[Unit]
Description=Run damn upkeep

[Service]
WorkingDirectory=/root/damn-deploy
ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/gen.yml run --rm upkeep
EOF
cat << EOF > /etc/systemd/system/damn_upkeep.timer
[Unit]
Description=Run damn upkeep every 15 minutes

[Timer]
OnCalendar=*:0/15

[Install]
WantedBy=timers.target
EOF

Enable and run damn upkeep service and timer:

systemctl enable damn_upkeep.timer
systemctl enable damn_upkeep.service

systemctl start damn_upkeep.timer

The following is the description of the damn upkeep scripts.

Automatically unlock square

The unlock_square.sql upkeep script unlocks a square that is locked for more than two hours.
Update areas statistics

The update_areas_statistics.sql script updates squares to map, squares to review and squares done fields of all active areas (areas with a priority greater than zero.)
Delete archived areas

When an area has priority -10, the delete_areas_with_priority_-10.sql script deletes such an area including all its squares and commits. This script is moved from upkeep/scripts to upkeep directory to avoid automatic run. For deleting areas with priority -10, you must run the following command:
```
psql -1 -f upkeep/delete_areas_with_priority_-10.sql "postgresql://damnuser:$(cat .env | sed -n 's/^POSTGRES_PASSWORD=$.\+$$/\1/p')@$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)/damndb"
```
NOTE: It works only for the areas with priority -10. For other archived areas (areas with priority less than zero,) there is no special upkeep script yet.

NOTE: The command works for user damnuser and database damndb. Update the command according to the needs.

#Static Map background images

For mappy client, the background image is used. The background image is provided by osmsm_www service (set in http.yml file.) Generating of the background images is done by osmsm_gen service (of the gen.yml file.)

osmsm_gen service should be run periodically, similar to upkeep service. You may set up the systemd config files for running osmsm_gen every 30 minutes as the following:

Create systemd unit and timer for damn osmsm gen:

cat << EOF > /etc/systemd/system/damn_osmsm_gen.service
[Unit]
Description=Run damn osmsm gen

[Service]
WorkingDirectory=/root/damn-deploy
ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/gen.yml run --rm osmsm_gen
EOF
cat << EOF > /etc/systemd/system/damn_osmsm_gen.timer
[Unit]
Description=Run damn osmsm gen every 30 minutes

[Timer]
OnCalendar=*:0/30

[Install]
WantedBy=timers.target
EOF

Enable and run damn upkeep service and timer:

systemctl enable damn_osmsm_gen.timer
systemctl enable damn_osmsm_gen.service

systemctl start damn_osmsm_gen.timer

#RSS feeds

It's possible to generate and provide RSS feeds of last week changes.

Create systemd unit and timer for damn RSS gen:

cat << EOF > /etc/systemd/system/damn_rss_gen.service
[Unit]
Description=Run damn RSS gen

[Service]
WorkingDirectory=/root/damn-deploy
ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/gen.yml run --rm rss_gen
EOF
cat << EOF > /etc/systemd/system/damn_rss_gen.timer
[Unit]
Description=Run damn RSS gen every day at 3:33.

[Timer]
OnCalendar=03:33

[Install]
WantedBy=timers.target
EOF

Enable and run damn upkeep service and timer:

systemctl enable damn_rss_gen.timer
systemctl enable damn_rss_gen.service

systemctl start damn_rss_gen.timer

#Intersecting areas check

Periodically check and update intersecting areas, generate web page showing them.

Create systemd unit and timer for generation of intersecting areas:

cat << EOF > /etc/systemd/system/damn_intersecting_gen.service
[Unit]
Description=Run generation of intersecting areas

[Service]
WorkingDirectory=/root/damn-deploy
ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/gen.yml run --rm intersecting_gen
EOF
cat << EOF > /etc/systemd/system/damn_intersecting_gen.timer
[Unit]
Description=Run generation of intersecting areas every 30 minutes

[Timer]
OnCalendar=*:0/30

[Install]
WantedBy=timers.target
EOF

Enable and run generation of intersecting area service and timer:

systemctl enable damn_intersecting_gen.timer
systemctl enable damn_intersecting_gen.service

systemctl start damn_intersecting_gen.timer

#How to upgrade

If use systemd autostart, stop all the damn service first:

systemctl stop damn-abandoned-areas.timer
systemctl stop damn-finish-areas.timer
systemctl stop damn_osmsm_gen.timer
systemctl stop damn-rss-feeds.timer
systemctl stop damn-upkeep.timer
systemctl stop damn.service

Stash the configuration:
```
git stash
```
Pull new version of the damn deploy repository:
```
git pull
```
Pop stashed changes (renew the configuration):
```
git stash pop
```

Maybe rebuild some docker images, but it depends on the upgrade changes:

docker-compose -f http.yml build --no-cache api
docker-compose -f http.yml build --no-cache www
docker-compose -f http.yml build --no-cache manager
docker-compose -f http.yml build --no-cache client
docker-compose -f http.yml build --no-cache panel
docker-compose -f http.yml build --no-cache mappy
docker-compose -f http.yml build --no-cache finished_www

docker-compose -f gen.yml build --no-cache upkeep
docker-compose -f gen.yml build --no-cache osmsm_gen
docker-compose -f gen.yml build --no-cache rss_gen

Remove old docker images:
```
docker image prune
```

Start the damn service again:

systemctl start damn.service
systemctl start damn-abandoned-areas.timer
systemctl start damn-finish-areas.timer
systemctl start damn_osmsm_gen.timer
systemctl start damn-rss-feeds.timer
systemctl start damn-upkeep.timer

NOTE: When the start of the service fails, it may be needed to delete all the docker containers and start it again. To delete all containers:

for i in `docker ps -aq`; do docker stop $i; docker rm $i; done

NOTE: When the service started sucessfully but it does not work, recreating traefik may help:

docker-compose -f http.yml up --force-recreate -d traefik

NOTE: It may happen that some upkeep service fails with for example endpoint with name damn-deploy_upkeep_run_1 already exists in network damn-deploy_default. This is probably when periodic upkeep is run while deploying new version. In such a situation:

docker network disconnect --force damn-deploy_default damn-deploy_upkeep_run_1

NOTE: When www is updated, it is not necessary to restart all the damn services. Instead, just run:

docker-compose -f http.yml up --build --force-recreate --no-deps -d www

#Upgrade the database structure

The database structure is rarely to change but it may happen. For new deployments, there is nothing to do.

However, if you are upgrading from an existing database, you need to run the upgrade script manually.

All the scripts related to the database structure are stored in damndb directory, starting at 70_.... Which script should be run when upgrading the database structure is always noted in the annotated tag of the corresponding version.

#Upgrade to `v0.6.0`

Stop the damn service and update the repository:

systemctl stop damn.service
git stash
git pull
git stash pop

Run the database:

docker-compose up -d db

Run the database upgrade script (you will be asked for the password stored in .env file):

export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
psql -1 -h $DB_HOST -d damndb -U damnuser < damndb/71_areas_squares_stats.sql

Stop the database container:

docker-compose down

Finally, there is no need to build the database container again. The database is created only once anyway, so run the damn service again:

systemctl start damn.service

#Upgrade to `v0.7.0`

When the upgrade is finished, update the current_commits in the database. Connect to the database (the password is in .env file):

export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
psql -h $DB_HOST -d damndb -U damnuser

Run the update query:

update current_commits set type='splitted' where message='The square was splitted';

#Upgrade to `v0.8.0`

This upgrade introduce change of the repository source and directories renaming. The best is to stop the running service first:

systemctl stop damn.service

Make sure all the containers are stopped:

cd damn_deploy
docker-compose -f docker-compose.yml stop
docker-compose -f docker-compose-clients.yml stop
docker-compose -f docker-compose-chat.yml stop
cd ..

And then follow the How to deploy section (I assume the current directory is /root/:)

git clone https://git.sr.ht/~qeef/damn-deploy
cd damn-deploy

Then copy secrets and configuration from ../damn_deploy/.env to .env file. (The .env changed a bit!) Also, update traefik.yml and copy ../damn_deploy/acme.json to acme.json.

The next step is to rewrite systemd config files. From the Autostart with systemd section, run the command to Create systemd unit for docker-compose:

cat << EOF > /etc/systemd/system/damn.service
...

And from the Damn upkeep section, run the command to Create systemd unit and timer for damn upkeep:

cat << EOF > /etc/systemd/system/damn_upkeep.service
...

Rebuild all the docker images with:

docker-compose -f http.yml build --no-cache api
docker-compose -f http.yml build --no-cache upkeep
docker-compose -f clients.yml build --no-cache client
docker-compose -f clients.yml build --no-cache manager
docker-compose -f chat.yml build --no-cache chat

Copy old database volume to the new database volume:

docker volume create --name damn-deploy_damndb-volume
docker run --rm -v damn_deploy_damndb-volume:/from -v damn-deploy_damndb-volume:/to alpine ash -c 'cd /from ; cp -av . /to'

Remove damndb container and rebuild it. It does nothing to the database.

docker rm damndb
docker-compose -f http.yml build --no-cache db

Reload systemd units and start the service:

systemctl daemon-reload
systemctl start damn.service

Now, if everything works and everything is tested, you may remove old database volume:

docker volume rm damn_deploy_damndb-volume

#Upgrade to `v0.12.0`

This upgrade introduces custom square width and height when creating areas. The function is stored in damndb/72_st_area_split_custom_square_size.sql file. The steps to upgrade follow.

Stop the damn service and update the repository:

systemctl stop damn.service
git stash
git pull
git stash pop

Run the database:

docker-compose -f http.yml up -d db

Import the new function into the database (you will be asked for the password stored in .env file):

export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
psql -1 -h $DB_HOST -d damndb -U damnuser < damndb/72_st_area_split_custom_square_size.sql

Stop the database container:

docker-compose -f http.yml down

Finally, there is no need to build the database container again. The database is created only once anyway. However, server, client, and manager need to be updated:

docker-compose -f http.yml build --no-cache api
docker-compose -f clients.yml build --no-cache client
docker-compose -f clients.yml build --no-cache manager

Finally, run the damn service again:

systemctl start damn.service

#Upgrade to `v0.17.0`

There is new st_divide function in damndb/73_st_divide.sql file. Follow the steps to put new function into the database. Stop the damn service and update the repository:

systemctl stop damn.service
git stash
git pull
git stash pop

Run the database:

docker-compose -f http.yml up -d db

Run the database upgrade script (you will be asked for the password stored in .env file):

export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
psql -1 -h $DB_HOST -d damndb -U damnuser < damndb/73_st_divide.sql

Stop the database container:

docker-compose -f http.yml down

Follow the common upgrade process in the How to upgrade section. (You need to re-build docker images for server and manager as these two were changed.)

#Upgrade to v0.17.7

The st_divide function is changed a little bit, so it's needed to re-run the steps from Ugrade to v0.17.0 section.

#Upgrade to v0.20.0

There are few improvements to database in v0.20.0. Add author field to current_areas. (Until now, the author of the area is got by checking the first commit of the first square.) And improve the performance by indexing. The steps to upgrade follow.

Stop the damn service and update the repository:

systemctl stop damn.service
git stash
git pull
git stash pop

Run the database:

docker-compose -f http.yml up -d db

Update the database structure. Note that you will be asked for the database password stored in .env file:

psql -1 -h $(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb) -d damndb -U damnuser < damndb/74_current_areas_author.sql
psql -1 -h $(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb) -d damndb -U damnuser < damndb/75_add_indexes.sql

Stop the database container:

docker-compose -f http.yml down

Rebuild changed images:

docker-compose -f http.yml build --no-cache api
docker-compose -f http.yml build --no-cache manager
docker-compose -f gen.yml build --no-cache osmsm_gen

Finally, run the damn service again:

systemctl start damn.service

#Upgrade to v0.20.1

The 75_add_indexes.sql is changed a little bit, so it's needed to re-run the steps from Upgrade to v0.20.0 for 75_add_indexes.sql.

#Upgrade to v0.21.0

The st_divide function is changed a little bit, so it needs to be updated.

Stop the damn service and update the repository:

systemctl stop damn.service
git stash
git pull
git stash pop

Run the database container:

docker-compose -f http.yml up -d db

Update the database:

psql -1 -f damndb/73_st_divide.sql "postgresql://damnuser:$(cat .env | sed -n 's/^POSTGRES_PASSWORD=\(.\+\)$/\1/p')@$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)/damndb"

Stop the database container:

docker-compose -f http.yml down

Rebuild changed images:

docker-compose -f http.yml build --no-cache api
docker-compose -f http.yml build --no-cache manager
docker-compose -f http.yml build --no-cache client
docker-compose -f http.yml build --no-cache panel
docker-compose -f http.yml build --no-cache mappy

Finally, run the damn service again:

systemctl start damn.service

#Upgrade to v0.22.0

There is a new square_from_already_divided function, so database needs to be updated.

Stop the damn service and update the repository:

systemctl stop damn.service
git stash
git pull
git stash pop

Run the database container:

docker-compose -f http.yml up -d db

Update the database:

psql -1 -f damndb/76_square_from_already_divided.sql "postgresql://damnuser:$(cat .env | sed -n 's/^POSTGRES_PASSWORD=\(.\+\)$/\1/p')@$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)/damndb"

Stop the database container:

docker-compose -f http.yml down

Rebuild changed images:

docker-compose -f http.yml build --no-cache api

Finally, run the damn service again:

systemctl start damn.service

refs

clone

#Discussion

#How to deploy

#Core services

#Other services

#Web clients

#Autostart with systemd

#Damn upkeep

#Static Map background images

#RSS feeds

#Intersecting areas check

#How to upgrade

#Upgrade the database structure

#Upgrade to v0.6.0

#Upgrade to v0.7.0

#Upgrade to v0.8.0

#Upgrade to v0.12.0

#Upgrade to v0.17.0

#Upgrade to v0.17.7

#Upgrade to v0.20.0

#Upgrade to v0.20.1

#Upgrade to v0.21.0

#Upgrade to v0.22.0

#Autostart with `systemd`

#Upgrade to `v0.6.0`

#Upgrade to `v0.7.0`

#Upgrade to `v0.8.0`

#Upgrade to `v0.12.0`

#Upgrade to `v0.17.0`