~mediagoblin/mediagoblin

ref: 9bcc05bd2dc3a5241ce1fd2ed7616f7de00873fd mediagoblin/docs/source/siteadmin/deploying.rst -rw-r--r-- 21.9 KiB
9bcc05bdBen Sturmfels Fix virtualenv permissions. a month ago
                                                                                
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
.. MediaGoblin Documentation

   Written in 2011, 2012, 2013, 2020 by MediaGoblin contributors

   To the extent possible under law, the author(s) have dedicated all
   copyright and related and neighboring rights to this software to
   the public domain worldwide. This software is distributed without
   any warranty.

   You should have received a copy of the CC0 Public Domain
   Dedication along with this software. If not, see
   <http://creativecommons.org/publicdomain/zero/1.0/>.

.. _deploying-chapter:

=====================
Deploying MediaGoblin
=====================

This deployment guide will take you step-by-step through
setting up your own instance of MediaGoblin.

MediaGoblin most likely isn't yet available from your operating
system's package manager, however, a basic install isn't too complex in
and of itself. We recommend a setup that combines MediaGoblin,
virtualenv and Nginx on a .deb or .rpm-based GNU/Linux distribution.

Experts may of course choose other deployment options, including
Apache. See our `Deployment wiki page
<https://web.archive.org/web/20200817190402/https://wiki.mediagoblin.org/Deployment>`_ for for more details.
Please note that we are not able to provide support for these
alternative deployment options.

.. note::

   These tools are for site administrators wanting to deploy a fresh
   install.  If you want to join in as a contributor, see our
   `Hacking HOWTO <https://web.archive.org/web/20200817190402/https://wiki.mediagoblin.org/HackingHowto>`_ instead.

.. note::

    Throughout the documentation we use the ``sudo`` command to indicate that
    an instruction requires elevated user privileges to run. You can issue
    these commands as the ``root`` user if you prefer.
    
    If you need help configuring ``sudo``, see the
    `Debian wiki <https://wiki.debian.org/sudo/>`_ or the
    `Fedora Project wiki <https://fedoraproject.org/wiki/Configuring_Sudo/>`_. 


Prepare System
--------------

Dependencies
~~~~~~~~~~~~

MediaGoblin has the following core dependencies:

- Python 3.6+
- `lxml <http://lxml.de/>`_
- `git <http://git-scm.com/>`_
- `SQLite <http://www.sqlite.org/>`_ or `PostgreSQL <http://www.postgresql.org/>`_
- `Python Imaging Library <http://www.pythonware.com/products/pil/>`_
  (PIL or Pillow)
- `virtualenv <http://www.virtualenv.org/>`_
- `Node.js <https://nodejs.org>`_

These instructions have been tested on Debian 11 and Fedora 33. These
instructions should approximately translate to recent Debian
derivatives such as Ubuntu and Trisquel, and to relatives of Fedora
such as CentOS, but we haven't explicitly tested these options.

Issue the following commands:

.. code-block:: bash

    # Debian 11
    sudo apt update
    sudo apt install automake git nodejs npm python3-dev \
    python3-gst-1.0 python3-lxml python3-pil virtualenv

    # Fedora 33
    sudo dnf install automake gcc git-core make nodejs npm \
    libffi-devel python3-devel python3-lxml python3-pillow \
    virtualenv

For a production deployment, you'll also need Nginx as frontend web
server and RabbitMQ to store the media processing queue::

    # Debian
    sudo apt install nginx-light rabbitmq-server

    # Fedora
    sudo dnf install nginx rabbitmq-server

..
   .. note::

      You might have to enable additional repositories under Fedora
      because rabbitmq-server might be not included in official
      repositories. That looks like this for CentOS::

        sudo dnf config-manager --set-enabled centos-rabbitmq-38
        sudo dnf config-manager --set-enabled PowerTools
        sudo dnf install rabbitmq-server
        sudo systemctl enable rabbitmq-server.service
        # TODO: Celery repeatedly disconnects from RabbitMQ on CentOS 8.

      As an alternative, you can try installing redis-server and
      configure it as celery broker.

Configure PostgreSQL
~~~~~~~~~~~~~~~~~~~~

.. note::

   MediaGoblin currently supports PostgreSQL and SQLite. The default
   is a local SQLite database. This will "just work" for small
   deployments. For medium to large deployments we recommend
   PostgreSQL. If you don't want/need PostgreSQL, skip this section.

These are the packages needed for PostgreSQL::

    # Debian
    sudo apt install postgresql python3-psycopg2

    # Fedora
    sudo dnf install postgresql postgresql-server python3-psycopg2

Fedora also requires that you initialize and start the
PostgreSQL database with a few commands. The following commands are
not needed on a Debian-based platform, however::

    # Fedora
    sudo /usr/bin/postgresql-setup initdb
    sudo systemctl enable postgresql
    sudo systemctl start postgresql

The installation process will create a new *system* user named ``postgres``,
which will have privileges sufficient to manage the database. We will create a
new database user with restricted privileges and a new database owned by our
restricted database user for our MediaGoblin instance.

In this example, the database user will be ``mediagoblin`` and the database
name will be ``mediagoblin`` too. We'll first at the user::

    sudo --login --user=postgres createuser --no-createdb mediagoblin

Then we'll create the database where all of our MediaGoblin data will be stored::

    sudo --login --user=postgres createdb --encoding=UTF8 --owner=mediagoblin mediagoblin

.. caution:: Where is the password?

    These steps enable you to authenticate to the database in a password-less
    manner via local UNIX authentication provided you run the MediaGoblin
    application as a user with the same name as the user you created in
    PostgreSQL.

    More on this in :ref:`Drop Privileges for MediaGoblin <drop-privileges-for-mediagoblin>`.


.. _drop-privileges-for-mediagoblin:

Drop Privileges for MediaGoblin
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

MediaGoblin does not require special permissions or elevated
access to run. As such, the preferred way to run MediaGoblin is to
create a dedicated, unprivileged system user for the sole purpose of running
MediaGoblin. Running MediaGoblin processes under an unprivileged system user
helps to keep it more secure. 

The following command will create a system account with a username of
``mediagoblin``.

If you are using a Debian-based system, enter this command::

    # Debian
    sudo useradd --system --create-home --home-dir /var/lib/mediagoblin \
    --group www-data --comment 'GNU MediaGoblin system account' mediagoblin

    # Fedora
    sudo useradd --system --create-home --home-dir /var/lib/mediagoblin \
    --group nginx --comment 'GNU MediaGoblin system account' mediagoblin

This will create a ``mediagoblin`` user and assign it to a group that is
associated with the web server. This will ensure that the web server can
read the media files that users upload (images, videos, etc.)

Many operating systems will automatically create a group
``mediagoblin`` to go with the new user ``mediagoblin``, but just to
be sure::
  
    sudo groupadd --force mediagoblin
    sudo usermod --append --groups mediagoblin mediagoblin
       
No password will be assigned to this account, and you will not be able
to log in as this user. To switch to this account, enter::

    sudo su mediagoblin --shell=/bin/bash

To return to your regular user account after using the system account, type
``exit`` or ``Ctrl-d``.

.. _create-mediagoblin-directory:

Create a MediaGoblin Directory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You should create a working directory for MediaGoblin. This document
assumes your local git repository will be located at 
``/srv/mediagoblin.example.org/mediagoblin/``.
Substitute your preferred local deployment path as needed.

Setting up the working directory requires that we first create the directory
with elevated privileges, and then assign ownership of the directory
to the unprivileged system account.

To do this, enter the following commands, changing the defaults to suit your
particular requirements::

    # Debian
    sudo mkdir --parents /srv/mediagoblin.example.org
    sudo chown --no-dereference --recursive mediagoblin:www-data /srv/mediagoblin.example.org
    sudo find /srv/mediagoblin.example.org -type d -exec chmod 755 {} \;
    sudo find /srv/mediagoblin.example.org -type f -exec chmod 644 {} \;
    sudo find /srv/mediagoblin.example.org/mediagoblin/user_dev/crypto -type d -exec chmod 750 {} \;
    sudo find /opt/mediagoblin.example.org/mediagoblin/user_dev/crypto -type f -exec chmod 640 {} \;
    sudo find /opt/mediagoblin.example.org/mediagoblin/bin -type f -exec chmod 750 {} \;

    # Fedora
    sudo mkdir --parents /srv/mediagoblin.example.org
    sudo chown --no-dereference --recursive mediagoblin:nginx /srv/mediagoblin.example.org
    sudo find /srv/mediagoblin.example.org -type d -exec chmod 755 {} \;
    sudo find /srv/mediagoblin.example.org -type f -exec chmod 644 {} \;
    sudo find /srv/mediagoblin.example.org/mediagoblin/user_dev/crypto -type d -exec chmod 750 {} \;
    sudo find /opt/mediagoblin.example.org/mediagoblin/user_dev/crypto -type f -exec chmod 640 {} \;
    sudo find /opt/mediagoblin.example.org/mediagoblin/bin -type f -exec chmod 750 {} \;


Install MediaGoblin and Virtualenv
----------------------------------

We will now switch to our 'mediagoblin' system account, and then set up
our MediaGoblin source code repository and its necessary services.
You should modify these commands to suit your own environment.

Switch to the ``mediagoblin`` unprivileged user and change to the
MediaGoblin directory that you just created::

    sudo su mediagoblin --shell=/bin/bash
    $ cd /srv/mediagoblin.example.org

.. note::

    Unless otherwise noted, the remainder of this document assumes that all
    operations are performed using the unprivileged ``mediagoblin``
    account, indicated by the ``$`` prefix.

Clone the MediaGoblin repository and set up the git submodules::

    $ git clone --depth=1 https://git.savannah.gnu.org/git/mediagoblin.git \
      --branch stable --recursive
    $ cd mediagoblin

Set up the environment::

    $ ./bootstrap.sh
    $ ./configure
    $ make

Create and set the proper permissions on the ``user_dev`` directory.
This directory will be used to store uploaded media files::

    $ mkdir --mode=2750 user_dev

This concludes the initial configuration of the MediaGoblin
environment. In the future, you can upgrade MediaGoblin according to
the ":doc:`upgrading`" documentation.


Configure Mediagoblin
---------------------

Edit site configuration
~~~~~~~~~~~~~~~~~~~~~~~

Edit ``mediagoblin.ini`` and update ``email_sender_address`` to the
address you wish to be used as the sender for system-generated emails.
You'll find more details in ":doc:`configuration`".

.. note::

   If you're changing the MediaGoblin directories or URL prefix, you
   may need to edit ``direct_remote_path``, ``base_dir``, and
   ``base_url``.


Configure MediaGoblin to use the PostgreSQL database
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you are using PostgreSQL, edit the ``[mediagoblin]`` section in your
``mediagoblin.ini`` and remove the ``#`` prefix on the line containing::

    sql_engine = postgresql:///mediagoblin

This assumes you are running the MediaGoblin application under the
same system account and database account; both ``mediagoblin``.


Update database data structures
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Before you start using the database, you need to run::

    $ ./bin/gmg dbupdate

to populate the database with the MediaGoblin data structures.


Create an admin account
~~~~~~~~~~~~~~~~~~~~~~~

Create a MediaGoblin account with full administration access. Provide
your own email address and enter a secure password when prompted::

    $ ./bin/gmg adduser --username you --email you@example.com
    $ ./bin/gmg makeadmin you


Test the Server
~~~~~~~~~~~~~~~

At this point MediaGoblin should be properly installed.  You can
test the deployment with the following command::

    $ ./lazyserver.sh --server-name=broadcast

You should be able to connect to the machine on port 6543 in your
browser to confirm that the service is operable. You should also be
able to log in with the admin username and password.

Type ``Ctrl-c`` to exit the above server test.

The next series of commands will need to be run as a privileged user.
To return to your regular user account after using the system account,
type ``exit`` or ``Ctrl-d``.


Deploy MediaGoblin
------------------

.. _webserver-config:

Nginx as a reverse-proxy
~~~~~~~~~~~~~~~~~~~~~~~~

This configuration example will use Nginx, however, you may use any
webserver of your choice. If you do not already have a web server,
consider Nginx, as the configuration files may be more clear than the
alternatives.

Create a configuration file at
``/srv/mediagoblin.example.org/nginx.conf`` and create a symbolic link
into a directory that will be included in your ``nginx`` configuration
(e.g. "``/etc/nginx/sites-enabled`` or ``/etc/nginx/conf.d``) with the
following commands::

    # Debian
    sudo ln --symbolic /srv/mediagoblin.example.org/nginx.conf /etc/nginx/sites-enabled/mediagoblin.conf
    sudo rm --force /etc/nginx/sites-enabled/default
    sudo systemctl enable nginx

    # Fedora
    sudo ln -s /srv/mediagoblin.example.org/nginx.conf /etc/nginx/conf.d/mediagoblin.conf
    sudo systemctl enable nginx

You can modify these commands and locations depending on your
preferences and the existing configuration of your Nginx instance. The
contents of this ``/srv/mediagoblin.example.org/nginx.conf`` file
should be modeled on the following::

    server {
     #################################################
     # Stock useful config options, but ignore them :)
     #################################################
     include /etc/nginx/mime.types;

     autoindex off;
     default_type  application/octet-stream;
     sendfile on;

     # Gzip
     gzip on;
     gzip_min_length 1024;
     gzip_buffers 4 32k;
     gzip_types text/plain application/x-javascript text/javascript text/xml text/css;

     #####################################
     # Mounting MediaGoblin stuff
     # This is the section you should read
     #####################################

     # Change this to allow your users to upload larger files. If
     # you enable audio or video you will need to increase this. This
     # is essentially a security setting to prevent *extremely* large
     # files being uploaded. Example settings include 500m and 1g.
     client_max_body_size 100m;

     # prevent attacks (someone uploading a .txt file that the browser
     # interprets as an HTML file, etc.)
     add_header X-Content-Type-Options nosniff;

     server_name mediagoblin.example.org www.mediagoblin.example.org;
     access_log /var/log/nginx/mediagoblin.example.access.log;
     error_log /var/log/nginx/mediagoblin.example.error.log;

     # MediaGoblin's stock static files: CSS, JS, etc.
     location /mgoblin_static/ {
        alias /srv/mediagoblin.example.org/mediagoblin/mediagoblin/static/;
     }

     # Instance specific media:
     location /mgoblin_media/ {
        alias /srv/mediagoblin.example.org/mediagoblin/user_dev/media/public/;
     }

     # Theme static files (usually symlinked in)
     location /theme_static/ {
        alias /srv/mediagoblin.example.org/mediagoblin/user_dev/theme_static/;
     }

     # Plugin static files (usually symlinked in)
     location /plugin_static/ {
        alias /srv/mediagoblin.example.org/mediagoblin/user_dev/plugin_static/;
     }

     # Forward requests to the MediaGoblin app server.
     location / {
        proxy_pass http://127.0.0.1:6543;
     }
    }

The first four ``location`` directives instruct Nginx to serve the
static and uploaded files directly rather than through the MediaGoblin
process. This approach is faster and requires less memory.

.. note::

   The user who owns the Nginx process, normally ``www-data`` or ``nginx``,
   requires execute permission on the directories ``static``,
   ``public``, ``theme_static`` and ``plugin_static`` plus all their
   parent directories. This user also requires read permission on all
   the files within these directories. This is normally the default.

Nginx is now configured to serve the MediaGoblin application. Perform a quick
test to ensure that this configuration works::

    sudo nginx -t

If you encounter any errors, review your Nginx configuration files, and try to
resolve them. If you do not encounter any errors, you can start your Nginx
server (may vary depending on your operating system)::

    sudo systemctl restart nginx

Now start MediaGoblin to test your Nginx configuration::

    sudo su mediagoblin --shell=/bin/bash
    $ cd /srv/mediagoblin.example.org/mediagoblin/
    $ ./lazyserver.sh --server-name=main

You should be able to connect to the machine on port 80 in your
browser to confirm that the service is operable. If this is the
machine in front of you, visit <http://localhost/> or if it is a
remote server visit the URL or IP address provided to you by your
hosting provider. You should see MediaGoblin; this time via Nginx!

Try logging in and uploading an image. If after uploading you see any
"Forbidden" errors from Nginx or your image doesn't show up, you may
need to update the permissions on the new directories MediaGoblin has
created::

    # Debian
    sudo chown --no-dereference --recursive mediagoblin:www-data /srv/mediagoblin.example.org

    # Fedora
    sudo chown --no-dereference --recursive mediagoblin:nginx /srv/mediagoblin.example.org

.. note::
   
   If you see an Nginx placeholder page, you may need to remove the
   Nginx default configuration, or explictly set a ``server_name``
   directive in the Nginx config.

Type ``Ctrl-c`` to exit the above server test and ``exit`` or
``Ctrl-d`` to exit the mediagoblin shell.


.. _systemd-service-files:

Run MediaGoblin as a system service
-----------------------------------

To ensure MediaGoblin is automatically started and restarted in case
of problems, we need to run it as system services. If your operating
system uses Systemd, you can use Systemd ``service files`` to manage
both the Celery and Paste processes as described below.

In the Systemd configuration below, MediaGoblin log files are kept in
the ``/var/log/mediagoblin`` directory. Create the directory and give
it the proper permissions::

    sudo mkdir --parents /var/log/mediagoblin
    sudo chown --no-dereference --recursive mediagoblin:mediagoblin /var/log/mediagoblin

Place the following service files in the ``/etc/systemd/system/``
directory. The first file should be named
``mediagoblin-paster.service``. Be sure to modify it to suit your
environment's setup:

.. code-block:: bash

    # Set the WorkingDirectory and Environment values to match your environment.
    [Unit]
    Description=Mediagoblin

    [Service]
    Type=simple
    User=mediagoblin
    Group=mediagoblin
    Environment=CELERY_ALWAYS_EAGER=false
    WorkingDirectory=/srv/mediagoblin.example.org/mediagoblin
    ExecStart=/srv/mediagoblin.example.org/mediagoblin/bin/paster serve \
                /srv/mediagoblin.example.org/mediagoblin/paste.ini \
                --log-file=/var/log/mediagoblin/mediagoblin.log \
                --server-name=main

    [Install]
    WantedBy=multi-user.target

The second file should be named ``mediagoblin-celeryd.service``:

.. code-block:: bash

    # Set the WorkingDirectory and Environment values to match your environment.
    [Unit]
    Description=MediaGoblin Celery
    After=rabbitmq-server.service

    [Service]
    User=mediagoblin
    Group=mediagoblin
    Type=simple
    WorkingDirectory=/srv/mediagoblin.example.org/mediagoblin
    Environment=MEDIAGOBLIN_CONFIG=/srv/mediagoblin.example.org/mediagoblin/mediagoblin.ini \
                CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery
    ExecStart=/srv/mediagoblin.example.org/mediagoblin/bin/celery worker \
                --logfile=/var/log/mediagoblin/celery.log \
                --loglevel=INFO

    [Install]
    WantedBy=multi-user.target

For details on this approach with a separate Celery process, see
:ref:`background-media-processing`.

Enable these processes to start at boot by entering::

    sudo systemctl enable mediagoblin-paster.service
    sudo systemctl enable mediagoblin-celeryd.service


Start the processes for the current session with::

    sudo systemctl start mediagoblin-paster.service
    sudo systemctl start mediagoblin-celeryd.service

If either command above gives you an error, you can investigate the cause of
the error by entering either of::

    sudo systemctl status mediagoblin-celeryd.service
    sudo systemctl status mediagoblin-paster.service

Or view the full logs with::

    sudo journalctl -u mediagoblin-paster.service -f
    sudo journalctl -u mediagoblin-celeryd.service -f

The above ``systemctl status`` command is also useful if you ever want to
confirm that a process is still running.

Assuming the above was successful, you should now have a MediaGoblin
server that will continue to operate, even after being restarted.
Great job!

If you have a moment, please send us an `email
<mailto:~mediagoblin/mediagoblin@todo.sr.ht?subject=MediaGoblin&20installation%20report>`_
about your experience installing MediaGoblin. We'd love to know what
worked well, what didn't work so well and anything that could be
improved.


.. _restarting mediagoblin:

Restarting MediaGoblin
----------------------

To restart MediaGoblin after making configuration changes, run::

    sudo systemctl restart mediagoblin-celeryd.service
    sudo systemctl restart mediagoblin-paster.service

If you make any changes to the ".service" files, you must first issue
a `daemon-reload` command to refresh Systemd and then restart
MediaGoblin with::

    sudo systemctl daemon-reload
    sudo systemctl restart mediagoblin-celeryd.service
    sudo systemctl restart mediagoblin-paster.service


What next?
----------

This configuration supports upload of images only, but MediaGoblin
also supports other types of media, such as audio, video, PDFs and 3D
models. For details, see ":doc:`media-types`".

See ":doc:`production-deployments`" for more information and other
issues you may want to consider.

For other settings and configuration options, see
":doc:`configuration`".

To enable and configure plugins, see ":doc:`plugins`".

..
   Local variables:
   fill-column: 70
   End: