Divide and map. Now. -- the damn project -- helps mappers by dividing some big area into smaller squares that a human can map.
The damn-deploy
repository contains the complete docker-compose
-based
deployment, including this deployment guide.
The damndb
and upkeep
directories are published under AGPLv3 License.
The rest of this repository is published under MIT License.
Use the mailing list for the discussion.
Prerequisities for deployment are git
, docker
and docker-compose
. The
guide is tested on Debian 10. You need some domain name directing to the
server.
Then, you need to clone the damn deploy repository:
git clone https://git.sr.ht/~qeef/damn-deploy
cd damn-deploy
You probably want to run server, clients, osmsm, www, and upkeep with
systemd
. Follow the following sub-sections to see the docker-compose
configs. The last sub-section is about systemd
.
See How to upgrade section for the upgrade process, Damn upkeep section for the database periodic upkeep scripts, and Static Map background images section for serving background images used in the mappy client.
There are few services within the server.yml
docker compose file. There is
db
and api
services. These are the core services. db
serves the database
and the api
is the JSON REST (Fast)API to the database. Then, there is the
upkeep
that runs periodic tasks on the database. (See the Damn upkeep
section.) The next is the www
service serving the web pages. Finally, the
traefik
service provides the gate to all the services above.
Set environment variables in .env
file. The following is the meaning:
POSTGRES_PASSWORD
: Password to PostgreSQL database.
JWT_SECRET
: Secret for generating, signing, and decrypting JSON Web
Tokens.
SESSION_SECRET
: Secret for cookies (I think).
OAUTH_CONSUMER_KEY
: API key, get from osm.org.
OAUTH_CONSUMER_SECRET
: API secret, get from osm.org.
DOMAIN_NAME
: Use your domain.
WWW_REPO
: Link to a docker git repository with web pages.
The following are optinal settings. You don't need to change them. The defaults work good.
SERVER_SUB
: The subdomain for the server.DAMN_SERVER_VERSION
: Choose the version of the server.DAMN_CLIENTS
: Allow origins clients.DB_HOST
: Alias of db
docker service, keep that value.WWW_SUB
: The subdomain for the web pages.Set proper email address in traefik.yml
file.
Create acme.json
file and set 600
permissions:
touch acme.json
chmod 600 acme.json
You may run the damn server now.
docker-compose -f server.yml up
In addition to Server setup, you may want to run the clients and the manager.
Set environment variables in .env
file. The following are optinal
settings. You don't need to change them. The defaults work good. There is
the meaning of the customization settings:
DEFAULT_LANG
: The default language.DEFAULT_WEB_EDITOR
: Set the default editor to id
or rapid
.And the meaning of the optional settings for the clients:
CLIENT_SUB
: The subdomain for the client.DAMN_CLIENT_REPO
: Link to client git repository.DAMN_CLIENT_VERSION
: Version of the client.PANEL_SUB
: The subdomain for the panel client.MAPPY_SUB
: The subdomain for the mappy client.OSMSM_SUB
: The subdomain for background images used in mappy client.MANAGER_SUB
: The subdomain for the manager.DAMN_MANAGER_REPO
: Link to the manager git repository.DAMN_MANAGER_VERSION
: Version of the manager.You may run the damn server with clients now.
docker-compose -f server.yml -f clients.yml up
In addition to Server setup, you may want to run the chat server.
Set environment variable in .env
file.
CHAT_SUB
: The subdomain for the chat.Run the chat server with the server:
docker-compose -f server.yml -f chat.yml up
or also with clients:
docker-compose -f server.yml -f clients.yml -f chat.yml up
systemd
Let's assume the damn deploy repository is cloned to /root/damn-deploy/
.
Create systemd unit for docker-compose
:
cat << EOF > /etc/systemd/system/damn.service
[Unit]
Description=Damn server app
After=network.target docker.service
[Service]
Type=simple
WorkingDirectory=/root/damn-deploy
ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/server.yml -f /root/damn-deploy/clients.yml -f /root/damn-deploy/chat.yml -f /root/damn-deploy/osmsm.ymlup
ExecStop=/usr/bin/docker-compose -f /root/damn-deploy/server.yml -f /root/damn-deploy/clients.yml -f /root/damn-deploy/chat.yml -f /root/damn-deploy/osmsm.yml down
[Install]
WantedBy=multi-user.target
EOF
The unit above is the default used on the damn-project.org. Feel free to
remove some -f LINK/TO/FILE.yml
for services you don't want to run.
Enable and run damn service:
systemctl start damn.service
systemctl enable damn.service
If use systemd
autostart, stop the damn service first:
systemctl stop damn.service
Stash the configuration:
git stash
Pull new version of the damn deploy repository:
git pull
Pop stashed changes (renew the configuration):
git stash pop
Maybe rebuild some docker images, but it depends on the upgrade changes:
docker-compose -f server.yml build --no-cache api
docker-compose -f server.yml build --no-cache upkeep
docker-compose -f server.yml build --no-cache www
docker-compose -f clients.yml build --no-cache client
docker-compose -f clients.yml build --no-cache panel
docker-compose -f clients.yml build --no-cache mappy
docker-compose -f clients.yml build --no-cache manager
Start the damn service again:
systemctl start damn.service
The database structure is rarely to change but it may happen. For new deployments, there is nothing to do.
However, if you are upgrading from an existing database, you need to run the upgrade script manually.
All the scripts related to the database structure are stored in damndb
directory, starting at 70_...
. Which script should be run when upgrading the
database structure is always noted in the [changelog][].
v0.6.0
Stop the damn service and update the repository:
systemctl stop damn.service
git stash
git pull
git stash pop
Run the database:
docker-compose up -d db
Run the database upgrade script (you will be asked for the password stored in
.env
file):
export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
psql -h $DB_HOST -d damndb -U damnuser < damndb/71_areas_squares_stats.sql
Stop the database container:
docker-compose down
Finally, there is no need to build the database container again. The database is created only once anyway, so run the damn service again:
systemctl start damn.service
v0.7.0
When the upgrade is finished, update the current_commits
in the database.
Connect to the database (the password is in .env
file):
export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
psql -h $DB_HOST -d damndb -U damnuser
Run the update query:
update current_commits set type='splitted' where message='The square was splitted';
v0.8.0
This upgrade introduce change of the repository source and directories renaming. The best is to stop the running service first:
systemctl stop damn.service
Make sure all the containers are stopped:
cd damn_deploy
docker-compose -f docker-compose.yml stop
docker-compose -f docker-compose-clients.yml stop
docker-compose -f docker-compose-chat.yml stop
cd ..
And then follow the How to deploy section (I assume the current directory is
/root/
:)
git clone https://git.sr.ht/~qeef/damn-deploy
cd damn-deploy
Then copy secrets and configuration from ../damn_deploy/.env
to .env
file.
(The .env
changed a bit!) Also, update traefik.yml
and copy
../damn_deploy/acme.json
to acme.json
.
The next step is to rewrite systemd config files. From the Autostart with
systemd
section, run the command to Create systemd unit for
docker-compose
:
cat << EOF > /etc/systemd/system/damn.service
...
And from the Damn upkeep section, run the command to Create systemd unit and timer for damn upkeep:
cat << EOF > /etc/systemd/system/damn_upkeep.service
...
Rebuild all the docker images with:
docker-compose -f server.yml build --no-cache api
docker-compose -f server.yml build --no-cache upkeep
docker-compose -f clients.yml build --no-cache client
docker-compose -f clients.yml build --no-cache manager
docker-compose -f chat.yml build --no-cache chat
Copy old database volume to the new database volume:
docker volume create --name damn-deploy_damndb-volume
docker run --rm -v damn_deploy_damndb-volume:/from -v damn-deploy_damndb-volume:/to alpine ash -c 'cd /from ; cp -av . /to'
Remove damndb
container and rebuild it. It does nothing to the database.
docker rm damndb
docker-compose -f server.yml build --no-cache db
Reload systemd units and start the service:
systemctl daemon-reload
systemctl start damn.service
Now, if everything works and everything is tested, you may remove old database volume:
docker volume rm damn_deploy_damndb-volume
v0.12.0
This upgrade introduces custom square width and height when creating areas. The
function is stored in damndb/72_st_area_split_custom_square_size.sql
file.
The steps to upgrade follow.
Stop the damn service and update the repository:
systemctl stop damn.service
git stash
git pull
git stash pop
Run the database:
docker-compose -f server.yml up -d db
Import the new function into the database (you will be asked for the password
stored in .env
file):
export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
psql -h $DB_HOST -d damndb -U damnuser < damndb/72_st_area_split_custom_square_size.sql
Stop the database container:
docker-compose -f server.yml down
Finally, there is no need to build the database container again. The database
is created only once anyway. However, server
, client
, and manager
need to
be updated:
docker-compose -f server.yml build --no-cache api
docker-compose -f clients.yml build --no-cache client
docker-compose -f clients.yml build --no-cache manager
Finally, run the damn service again:
systemctl start damn.service
v0.17.0
There is new st_divide
function in damndb/73_st_divide.sql
file. Follow the
steps to put new function into the database. Stop the damn service and update
the repository:
systemctl stop damn.service
git stash
git pull
git stash pop
Run the database:
docker-compose -f server.yml up -d db
Run the database upgrade script (you will be asked for the password stored in
.env
file):
export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)
psql -h $DB_HOST -d damndb -U damnuser < damndb/73_st_divide.sql
Stop the database container:
docker-compose -f server.yml down
Follow the common upgrade process in the How to upgrade section. (You need to re-build docker images for server and manager as these two were changed.)
The st_divide
function is changed a little bit, so it's needed to re-run the
steps from Ugrade to v0.17.0 section.
Damn upkeep docker image contains scripts that run periodically. Periodic run
must be set with cron
or systemd
. The following example uses systemd
.
An example of upkeep script is square unlock when there is no activity in two hours.
Create systemd unit and timer for damn upkeep:
cat << EOF > /etc/systemd/system/damn_upkeep.service
[Unit]
Description=Run damn upkeep
[Service]
WorkingDirectory=/root/damn-deploy
ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/server.yml run --rm upkeep
EOF
cat << EOF > /etc/systemd/system/damn_upkeep.timer
[Unit]
Description=Run damn upkeep every 15 minutes
[Timer]
OnCalendar=*:0/15
[Install]
WantedBy=timers.target
EOF
Enable and run damn upkeep service and timer:
systemctl enable damn_upkeep.timer
systemctl enable damn_upkeep.service
systemctl start damn_upkeep.timer
The unlock_square.sql
upkeep script unlocks a square that is locked for more
than two hours.
The update_areas_statistics.sql
script updates squares to map, squares to
review and squares done fields of all current areas (areas with a priority
greater than zero.)
When an area has priority -10
, the delete_areas_with_priority_-10.sql
script deletes such an area including all its squares and commits. This script
is moved from upkeep/scripts
to upkeep
directory to avoid automatic run.
For deleting areas with priority -10
, you must run the following command:
psql -f upkeep/delete_areas_with_priority_-10.sql "postgresql://damnuser:$(cat .env | sed -n 's/^POSTGRES_PASSWORD=\(.\+\)$/\1/p')@$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)/damndb"
NOTE: It works only for the areas with priority -10
. For other archived
areas (areas with priority less than zero,) there is no special upkeep script
yet.
NOTE: The command works for user damnuser
and database damndb
. Update
the script according to the needs.
For mappy client, the background image is used. The background image is
provided by osmsm_www
service (set in osmsm.yml
file.) Generating of the
background images is done by osmsm_gen
service (of the same file.)
osmsm_www
service is similar to www
service from server.yml
. To run the
osmsm_www
service along with the rest of the damn services automatically, you
need to add -f /root/damn-deploy/osmsm.yml
to ExecStart
and ExecStop
of
/etc/systemd/system/damn.service
.
osmsm_gen
service should be run periodically, similar to upkeep
service.
You may set up the systemd config files for running osmsm_gen
every day at
12:12 as the following:
Create systemd unit and timer for damn osmsm gen:
cat << EOF > /etc/systemd/system/damn_osmsm_gen.service
[Unit]
Description=Run damn osmsm gen
[Service]
WorkingDirectory=/root/damn-deploy
ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/osmsm.yml run --rm osmsm_gen
EOF
cat << EOF > /etc/systemd/system/damn_osmsm_gen.timer
[Unit]
Description=Run damn osmsm gen every day at 12:12.
[Timer]
OnCalendar=12:12
[Install]
WantedBy=timers.target
EOF
Enable and run damn upkeep service and timer:
systemctl enable damn_osmsm_gen.timer
systemctl enable damn_osmsm_gen.service
systemctl start damn_osmsm_gen.timer
It's possible to generate and provide RSS feeds of last week changes.
Create systemd unit and timer for damn RSS gen:
cat << EOF > /etc/systemd/system/damn_rss_gen.service
[Unit]
Description=Run damn RSS gen
[Service]
WorkingDirectory=/root/damn-deploy
ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/rss.yml run --rm rss_gen
EOF
cat << EOF > /etc/systemd/system/damn_rss_gen.timer
[Unit]
Description=Run damn RSS gen every day at 3:33.
[Timer]
OnCalendar=03:33
[Install]
WantedBy=timers.target
EOF
Enable and run damn upkeep service and timer:
systemctl enable damn_rss_gen.timer
systemctl enable damn_rss_gen.service
systemctl start damn_rss_gen.timer