Damn deploy =========== **Divide and map. Now.** -- the damn project -- helps mappers by dividing some big area into smaller squares that a human can map. The `damn-deploy` repository contains the complete `docker-compose`-based deployment, including this deployment guide. License ------- The `damndb` and `upkeep` directories are published under [AGPLv3 License][]. The rest of this repository is published under [MIT License][]. [MIT License]: ./LICENSE [AGPLv3 License]: ./damndb/LICENSE Discussion ---------- Use the [mailing list][] for the discussion. [mailing list]: https://lists.sr.ht/~qeef/damn-project How to deploy ============= Prerequisities for deployment are `git`, `docker` and `docker-compose`. The guide is tested on Debian 10. You need some domain name directing to the server. Then, you need to clone the damn deploy repository: git clone https://git.sr.ht/~qeef/damn-deploy cd damn-deploy There are two config files: `http.yml` specify all HTTP service images, `gen.yml` specify images that generate content. If you don't want to run some image (e.g. `chat` from `http.yml`,) remove the section from the config file. You may later reset back with `git checkout -p`. See _Autostart with `systemd`_ section to run the damn services automatically. See _How to upgrade_ section for the upgrade process. Core services ------------- There are HTTP services within the `http.yml` docker compose file. `db`, `traefik`, and `api` are the core services and must be always present. 1. Set environment variables in `.env` file. The following is the meaning: - `POSTGRES_PASSWORD`: Password to PostgreSQL database. - `JWT_SECRET`: Secret for generating, signing, and decrypting JSON Web Tokens. - `SESSION_SECRET`: Secret for cookies (I think). - `OAUTH_CONSUMER_KEY`: API key, get from osm.org. - `OAUTH_CONSUMER_SECRET`: API secret, get from osm.org. - `DOMAIN_NAME`: Use your domain. - `WWW_REPO`: Link to a docker git repository with web pages. The following are optinal settings. You don't need to change them. The defaults work good. - `SERVER_SUB`: The subdomain for the server. - `DAMN_SERVER_VERSION`: Choose the version of the server. - `DAMN_CLIENTS`: Allow origins clients. - `DB_HOST`: Alias of `db` docker service, keep that value. - `WWW_SUB`: The subdomain for the web pages. 2. Set proper email address in `traefik.yml` file. 3. Create `acme.json` file and set `600` permissions: touch acme.json chmod 600 acme.json __NOTE:__ Feel free to tune `db` service parameters in `command` -- the values here are for 1 GB RAM. See PostgreSQL documentation [Resource Consumption][] and [Write Ahead Log][]. [Resource Consumption]: https://www.postgresql.org/docs/11/runtime-config-resource.html#GUC-SHARED-BUFFERS [Write Ahead Log]: https://www.postgresql.org/docs/11/runtime-config-wal.html Other services -------------- These are other support services. Feel free to keep the default values. 1. Set environment variables in `.env` file. The following is the meaning: - `CHAT_SUB`: The subdomain for the chat. - `OSMSM_SUB`: The subdomain for background images used in mappy client. - `RSS_SUB`: The subdomain for RSS. - `FINISHED_SUB`: The subdomain for finished areas. Web clients ----------- There are HTTP clients in `http.yml`. The following are possible settings. 1. Set environment variables in `.env` file. The following is the meaning: - `DEFAULT_LANG`: The default language. - `DEFAULT_WEB_EDITOR`: Set the default editor to `id` or `rapid`. The following are optinal settings. You don't need to change them. The defaults work good. - `DAMN_MANAGER_REPO`: Link to the manager git repository. - `DAMN_MANAGER_VERSION`: Version of the manager. - `MANAGER_SUB`: The subdomain for the manager. - `DAMN_CLIENT_REPO`: Link to client git repository. - `DAMN_CLIENT_VERSION`: Version of the client. - `CLIENT_SUB`: The subdomain for the client. - `PANEL_SUB`: The subdomain for the panel client. - `MAPPY_SUB`: The subdomain for the mappy client. Autostart with `systemd` ======================== Let's assume the damn deploy repository is cloned to `/root/damn-deploy/`. 1. Create systemd unit for `http.yml` services. This is the default unit. It's used on the damn-project.org. cat << EOF > /etc/systemd/system/damn.service [Unit] Description=Damn server app After=network.target docker.service [Service] Type=simple WorkingDirectory=/root/damn-deploy ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/http.yml up ExecStop=/usr/bin/docker-compose -f /root/damn-deploy/http.yml down [Install] WantedBy=multi-user.target EOF 2. Enable and run damn service: systemctl start damn.service systemctl enable damn.service Damn upkeep ----------- Damn upkeep docker image contains scripts that run periodically. Periodic run must be set with `cron` or `systemd`. The following example uses `systemd`. An example of upkeep script is square unlock when there is no activity in two hours. 1. Create `systemd` unit and timer for damn upkeep: cat << EOF > /etc/systemd/system/damn_upkeep.service [Unit] Description=Run damn upkeep [Service] WorkingDirectory=/root/damn-deploy ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/gen.yml run --rm upkeep EOF cat << EOF > /etc/systemd/system/damn_upkeep.timer [Unit] Description=Run damn upkeep every 15 minutes [Timer] OnCalendar=*:0/15 [Install] WantedBy=timers.target EOF 2. Enable and run damn upkeep service and timer: systemctl enable damn_upkeep.timer systemctl enable damn_upkeep.service systemctl start damn_upkeep.timer The following is the description of the damn upkeep scripts. - Automatically unlock square The `unlock_square.sql` upkeep script unlocks a square that is locked for more than two hours. - Update areas statistics The `update_areas_statistics.sql` script updates _squares to map_, _squares to review_ and _squares done_ fields of all _active areas_ (areas with a priority greater than zero.) - Delete archived areas When an area has priority `-10`, the `delete_areas_with_priority_-10.sql` script deletes such an area including all its squares and commits. This script is moved from `upkeep/scripts` to `upkeep` directory to avoid automatic run. For deleting areas with priority `-10`, you must run the following command: psql -1 -f upkeep/delete_areas_with_priority_-10.sql "postgresql://damnuser:$(cat .env | sed -n 's/^POSTGRES_PASSWORD=\(.\+\)$/\1/p')@$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)/damndb" **NOTE:** It works only for the areas with priority `-10`. For other archived areas (areas with priority less than zero,) there is no special upkeep script yet. **NOTE:** The command works for user `damnuser` and database `damndb`. Update the command according to the needs. Static Map background images ---------------------------- For _mappy_ client, the background image is used. The background image is provided by `osmsm_www` service (set in `http.yml` file.) Generating of the background images is done by `osmsm_gen` service (of the `gen.yml` file.) `osmsm_gen` service should be run periodically, similar to `upkeep` service. You may set up the `systemd` config files for running `osmsm_gen` every 30 minutes as the following: 1. Create systemd unit and timer for damn osmsm gen: cat << EOF > /etc/systemd/system/damn_osmsm_gen.service [Unit] Description=Run damn osmsm gen [Service] WorkingDirectory=/root/damn-deploy ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/gen.yml run --rm osmsm_gen EOF cat << EOF > /etc/systemd/system/damn_osmsm_gen.timer [Unit] Description=Run damn osmsm gen every 30 minutes [Timer] OnCalendar=*:0/30 [Install] WantedBy=timers.target EOF 2. Enable and run damn upkeep service and timer: systemctl enable damn_osmsm_gen.timer systemctl enable damn_osmsm_gen.service systemctl start damn_osmsm_gen.timer RSS feeds --------- It's possible to generate and provide RSS feeds of last week changes. 1. Create systemd unit and timer for damn RSS gen: cat << EOF > /etc/systemd/system/damn_rss_gen.service [Unit] Description=Run damn RSS gen [Service] WorkingDirectory=/root/damn-deploy ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/gen.yml run --rm rss_gen EOF cat << EOF > /etc/systemd/system/damn_rss_gen.timer [Unit] Description=Run damn RSS gen every day at 3:33. [Timer] OnCalendar=03:33 [Install] WantedBy=timers.target EOF 2. Enable and run damn upkeep service and timer: systemctl enable damn_rss_gen.timer systemctl enable damn_rss_gen.service systemctl start damn_rss_gen.timer Intersecting areas check ------------------------ Periodically check and update intersecting areas, generate web page showing them. 1. Create systemd unit and timer for generation of intersecting areas: cat << EOF > /etc/systemd/system/damn_intersecting_gen.service [Unit] Description=Run generation of intersecting areas [Service] WorkingDirectory=/root/damn-deploy ExecStart=/usr/bin/docker-compose -f /root/damn-deploy/gen.yml run --rm intersecting_gen EOF cat << EOF > /etc/systemd/system/damn_intersecting_gen.timer [Unit] Description=Run generation of intersecting areas every 30 minutes [Timer] OnCalendar=*:0/30 [Install] WantedBy=timers.target EOF 2. Enable and run generation of intersecting area service and timer: systemctl enable damn_intersecting_gen.timer systemctl enable damn_intersecting_gen.service systemctl start damn_intersecting_gen.timer How to upgrade ============== 1. If use `systemd` autostart, stop the damn service first: systemctl stop damn.service 2. Stash the configuration: git stash 3. Pull new version of the damn deploy repository: git pull 4. Pop stashed changes (renew the configuration): git stash pop 5. Maybe rebuild some docker images, but it depends on the upgrade changes: docker-compose -f http.yml build --no-cache api docker-compose -f http.yml build --no-cache www docker-compose -f http.yml build --no-cache manager docker-compose -f http.yml build --no-cache client docker-compose -f http.yml build --no-cache panel docker-compose -f http.yml build --no-cache mappy docker-compose -f http.yml build --no-cache finished_www docker-compose -f gen.yml build --no-cache upkeep docker-compose -f gen.yml build --no-cache osmsm_gen docker-compose -f gen.yml build --no-cache rss_gen 6. Remove old docker images: docker image prune 7. Start the damn service again: systemctl start damn.service __NOTE:__ When the start of the service fails, it may be needed to delete all the docker containers and start it again: systemctl stop damn.service for i in `docker ps -aq`; do docker stop $i; docker rm $i; done systemctl start damn.service Upgrade the database structure ------------------------------ The database structure is rarely to change but it may happen. For new deployments, there is nothing to do. However, if you are upgrading from an existing database, you need to run the upgrade script manually. All the scripts related to the database structure are stored in `damndb` directory, starting at `70_...`. Which script should be run when upgrading the database structure is always noted in the annotated tag of the corresponding version. Upgrade to `v0.6.0` ------------------- Stop the damn service and update the repository: systemctl stop damn.service git stash git pull git stash pop Run the database: docker-compose up -d db Run the database upgrade script (you will be asked for the password stored in `.env` file): export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb) psql -1 -h $DB_HOST -d damndb -U damnuser < damndb/71_areas_squares_stats.sql Stop the database container: docker-compose down Finally, there is no need to build the database container again. The database is created only once anyway, so run the damn service again: systemctl start damn.service Upgrade to `v0.7.0` ------------------- When the upgrade is finished, update the `current_commits` in the database. Connect to the database (the password is in `.env` file): export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb) psql -h $DB_HOST -d damndb -U damnuser Run the update query: update current_commits set type='splitted' where message='The square was splitted'; Upgrade to `v0.8.0` ------------------- This upgrade introduce change of the repository source and directories renaming. The best is to stop the running service first: systemctl stop damn.service Make sure all the containers are stopped: cd damn_deploy docker-compose -f docker-compose.yml stop docker-compose -f docker-compose-clients.yml stop docker-compose -f docker-compose-chat.yml stop cd .. And then follow the *How to deploy* section (I assume the current directory is `/root/`:) git clone https://git.sr.ht/~qeef/damn-deploy cd damn-deploy Then copy secrets and configuration from `../damn_deploy/.env` to `.env` file. (The `.env` changed a bit!) Also, update `traefik.yml` and copy `../damn_deploy/acme.json` to `acme.json`. The next step is to rewrite systemd config files. From the *Autostart with `systemd`* section, run the command to *Create systemd unit for `docker-compose`*: cat << EOF > /etc/systemd/system/damn.service ... And from the *Damn upkeep* section, run the command to *Create systemd unit and timer for damn upkeep*: cat << EOF > /etc/systemd/system/damn_upkeep.service ... Rebuild all the docker images with: docker-compose -f http.yml build --no-cache api docker-compose -f http.yml build --no-cache upkeep docker-compose -f clients.yml build --no-cache client docker-compose -f clients.yml build --no-cache manager docker-compose -f chat.yml build --no-cache chat Copy old database volume to the new database volume: docker volume create --name damn-deploy_damndb-volume docker run --rm -v damn_deploy_damndb-volume:/from -v damn-deploy_damndb-volume:/to alpine ash -c 'cd /from ; cp -av . /to' Remove `damndb` container and rebuild it. It does **nothing** to the database. docker rm damndb docker-compose -f http.yml build --no-cache db Reload systemd units and start the service: systemctl daemon-reload systemctl start damn.service Now, if everything works and everything is tested, you may remove old database volume: docker volume rm damn_deploy_damndb-volume Upgrade to `v0.12.0` -------------------- This upgrade introduces custom square width and height when creating areas. The function is stored in `damndb/72_st_area_split_custom_square_size.sql` file. The steps to upgrade follow. Stop the damn service and update the repository: systemctl stop damn.service git stash git pull git stash pop Run the database: docker-compose -f http.yml up -d db Import the new function into the database (you will be asked for the password stored in `.env` file): export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb) psql -1 -h $DB_HOST -d damndb -U damnuser < damndb/72_st_area_split_custom_square_size.sql Stop the database container: docker-compose -f http.yml down Finally, there is no need to build the database container again. The database is created only once anyway. However, `server`, `client`, and `manager` need to be updated: docker-compose -f http.yml build --no-cache api docker-compose -f clients.yml build --no-cache client docker-compose -f clients.yml build --no-cache manager Finally, run the damn service again: systemctl start damn.service Upgrade to `v0.17.0` ------------------- There is new `st_divide` function in `damndb/73_st_divide.sql` file. Follow the steps to put new function into the database. Stop the damn service and update the repository: systemctl stop damn.service git stash git pull git stash pop Run the database: docker-compose -f http.yml up -d db Run the database upgrade script (you will be asked for the password stored in `.env` file): export DB_HOST=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb) psql -1 -h $DB_HOST -d damndb -U damnuser < damndb/73_st_divide.sql Stop the database container: docker-compose -f http.yml down Follow the common upgrade process in the _How to upgrade_ section. (You need to re-build docker images for server and manager as these two were changed.) Upgrade to v0.17.7 ------------------ The `st_divide` function is changed a little bit, so it's needed to re-run the steps from _Ugrade to v0.17.0_ section. Upgrade to v0.20.0 ------------------ There are few improvements to database in `v0.20.0`. Add `author` field to `current_areas`. (Until now, the `author` of the area is got by checking the first commit of the first square.) And improve the performance by indexing. The steps to upgrade follow. Stop the damn service and update the repository: systemctl stop damn.service git stash git pull git stash pop Run the database: docker-compose -f http.yml up -d db Update the database structure. Note that you will be asked for the database password stored in `.env` file: psql -1 -h $(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb) -d damndb -U damnuser < damndb/74_current_areas_author.sql psql -1 -h $(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb) -d damndb -U damnuser < damndb/75_add_indexes.sql Stop the database container: docker-compose -f http.yml down Rebuild changed images: docker-compose -f http.yml build --no-cache api docker-compose -f http.yml build --no-cache manager docker-compose -f gen.yml build --no-cache osmsm_gen Finally, run the damn service again: systemctl start damn.service Upgrade to v0.20.1 ------------------ The `75_add_indexes.sql` is changed a little bit, so it's needed to re-run the steps from _Upgrade to v0.20.0_ for `75_add_indexes.sql`. Upgrade to v0.21.0 ------------------ The `st_divide` function is changed a little bit, so it needs to be updated. Stop the damn service and update the repository: systemctl stop damn.service git stash git pull git stash pop Run the database container: docker-compose -f http.yml up -d db Update the database: psql -1 -f damndb/73_st_divide.sql "postgresql://damnuser:$(cat .env | sed -n 's/^POSTGRES_PASSWORD=\(.\+\)$/\1/p')@$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)/damndb" Stop the database container: docker-compose -f http.yml down Rebuild changed images: docker-compose -f http.yml build --no-cache api docker-compose -f http.yml build --no-cache manager docker-compose -f http.yml build --no-cache client docker-compose -f http.yml build --no-cache panel docker-compose -f http.yml build --no-cache mappy Finally, run the damn service again: systemctl start damn.service Upgrade to v0.22.0 ------------------ There is a new `square_from_already_divided` function, so database needs to be updated. Stop the damn service and update the repository: systemctl stop damn.service git stash git pull git stash pop Run the database container: docker-compose -f http.yml up -d db Update the database: psql -1 -f damndb/76_square_from_already_divided.sql "postgresql://damnuser:$(cat .env | sed -n 's/^POSTGRES_PASSWORD=\(.\+\)$/\1/p')@$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' damndb)/damndb" Stop the database container: docker-compose -f http.yml down Rebuild changed images: docker-compose -f http.yml build --no-cache api Finally, run the damn service again: systemctl start damn.service