~sirn/fanboi2

8e2e4e30a16786fb13393753f96eff9eefc3cdfb — Kridsada Thanabulpong 7 years ago 025ad9d + 83da769
Merge pull request #6 from pxfs/feature/new-provision

Simplify development box provisioning with Vagrant
* Closes #5.
27 files changed, 218 insertions(+), 749 deletions(-)

M .gitignore
M .travis.yml
M README.rst
M Vagrantfile
A alembic.ini.sample
R provisioning/files/srv/settings.ini.j2 => development.ini.sample
M fanboi2/__init__.py
M fanboi2/formatters.py
M fanboi2/tests/__init__.py
M fanboi2/tests/test_formatters.py
A production.ini.sample
D provisioning/development_hosts
D provisioning/files/etc/memcached.conf.j2
D provisioning/files/etc/nginx/nginx.conf.j2
D provisioning/files/etc/postgresql/9.2/main/pg_hba.conf.j2
D provisioning/files/etc/postgresql/9.2/main/postgresql.conf.j2
D provisioning/files/etc/redis/redis.conf.j2
D provisioning/files/etc/supervisor/conf.d/brunch.conf.j2
D provisioning/files/etc/supervisor/conf.d/celery.conf.j2
D provisioning/files/etc/supervisor/conf.d/uwsgi.conf.j2
D provisioning/files/etc/uwsgi/fanboi2.ini.j2
D provisioning/files/srv/alembic.ini.j2
D provisioning/files/srv/fanboi2.wsgi.j2
D provisioning/genconfig.py
D provisioning/group_vars/all.yml
D provisioning/site.yml
M setup.py
M .gitignore => .gitignore +1 -6
@@ 32,10 32,5 @@ logs/
*.gz

# User settings
settings.ini
development.ini
alembic.ini

# Provisioning
provisioning/*_hosts
provisioning/host_vars
!provisioning/development_hosts

M .travis.yml => .travis.yml +20 -5
@@ 1,9 1,24 @@
language: python
python: "3.2"
script: nosetests
install: "pip install . --use-mirrors"
before_script: 'psql -c "create database fanboi2;" -U postgres'
env: POSTGRESQL_TEST_DATABASE=postgresql://postgres@localhost:5432/fanboi2

python:
  - "3.2"
  - "pypy3"

# PyPy is trying to create a ctype_config_cache, but the directory is not
# writeable. This is a hacky workaround to make it work.
before_install:
  - 'sudo mkdir -p /opt/python/pypy3-2.3.1/lib_pypy/ctypes_config_cache/__pycache__'
  - 'sudo chmod a+w /opt/python/pypy3-2.3.1/lib_pypy/ctypes_config_cache/__pycache__'

install:
  - 'pip install -e .'

before_script:
  - 'psql -c "create database fanboi2;" -U postgres'

env:
  - POSTGRESQL_TEST_DATABASE=postgresql+pg8000://postgres@localhost:5432/fanboi2

notifications:
    email: false
  email: false

M README.rst => README.rst +42 -52
@@ 1,7 1,7 @@
Fanboi2 |ci|
============

Board engine behind `Fanboi Channel <http://fanboi.ch/>`_ written in Python.
Board engine behind `Fanboi Channel <https://fanboi.ch/>`_ written in Python.

.. |ci| image:: https://api.travis-ci.org/pxfs/fanboi2.png?branch=develop
        :target: https://travis-ci.org/pxfs/fanboi2


@@ 9,69 9,50 @@ Board engine behind `Fanboi Channel <http://fanboi.ch/>`_ written in Python.
Getting Started
---------------

There are two ways of getting the app up and running for development. Using `Vagrant <http://vagrantup.com/>`_ (*The Better Way*) or installing everything manually (*The Adventurous Way*). The recommened method is to use Vagrant as it closely replicates the production environment.

The Better Way
~~~~~~~~~~~~~~

We use `Vagrant <http://www.vagrantup.com/>`_ for development environment provisioning. You must first `install Vagrant <http://docs.vagrantup.com/v2/installation/>`_ (and `VirtualBox <https://www.virtualbox.org/>`_ or any other available `providers <http://docs.vagrantup.com/v2/providers/index.html>`_) and `Ansible <http://www.ansibleworks.com/docs/gettingstarted.html#via-pip>`_ (for Windows users, please use `Cygwin <http://www.cygwin.com/>`_)::

    $ pip install ansible
    $ vagrant up

That's it! You can now visit http://localhost:8080/ and proceed on development. To clean up the VM, you can run either ``vagrant destroy`` to completely remove the VM or ``vagrant halt`` to shutdown the VM. See also `Teardown <http://docs.vagrantup.com/v2/getting-started/teardown.html>`_ section of Vagrant documentation. After you've confirmed the app is running, please see **Management Scripts** section below.

The Adventurous Way
~~~~~~~~~~~~~~~~~~~

If you don't want to use Vagrant, you can manually install everything. Start with dependencies:

- `Python 3.2 <http://www.python.org/>`_
- `PostgreSQL 9.1 <http://www.postgresql.org/>`_
- `Redis <http://redis.io>`_
- `node.js <http://nodejs.org>`_ with `brunch <http://brunch.io/>`_

After all prerequisites are installed, you can now create the database and run setup::
The easiest way to get the app running is to run `Vagrant`_. You can follow these steps to get the app running:

    $ createuser -P fanboi2                # Create fanboi2 user. Please set "fanboi2" as password.
    $ createdb -O fanboi2 fanboi2          # Create main database.
    $ createdb -O fanboi2 fanboi2_test     # Create test database (required for testing).
    $ python3 setup.py develop             # Setup app in develop mode.
    $ pip install jinja2                   # Install Jinja2 for Genconfig.
    $ python3 provisioning/genconfig.py    # Generate settings.ini and alembic.ini.
    $ alembic upgrade head                 # Migrate database.
1. Install `Vagrant`_ of your preferred platform.
2. Install `VirtualBox <https://www.virtualbox.org/>`_ or other `providers <http://docs.vagrantup.com/v2/providers/index.html>`_ supported by Vagrant.
3. Run ``vagrant up`` and read `Getting Started <http://docs.vagrantup.com/v2/getting-started/index.html>`_ while waiting.
4. Run ``vagrant ssh`` to SSH into the development machine.

It is recommended to run tests and see if all tests passed::
Once the development box is up and running, you may now run the server::

    $ pip install nose
    $ nosetests

If all tests passed, you can now run the application. You must first setup assets compilation and do an initial compile before running::

    $ npm install
    $ brunch build
    $ vagrant ssh
    $ cd /vagrant
    $ pserve development.ini

Then you have to make sure Celery is running::
Now you're done! You can now proceed to the Management Scripts section below.

    $ fb2_celery development.ini worker
The Adventurous Way
~~~~~~~~~~~~~~~~~~~

And now you can run the application::
If you don't really want to use Vagrant, you can also install everything using your preferred methods:

    $ pserve development.ini --reload
1. `PyPy3 2.3.1 <http://pypy.org/download.html#default-with-a-jit-compiler>`_.
2. `PostgreSQL 9.2 <http://www.postgresql.org/>`_.
3. `Redis 2.8 <http://redis.io/>`_.
4. `Memcached 1.4 <http://www.memcached.org/>`_.
5. `Node.js 0.10 <http://nodejs.org/>`_ with `Brunch <http://brunch.io/>`_.

You may also found ``brunch watch`` useful for automatic assets compilation::
After the package above are up and running, you may now setup the application::

    $ brunch watch
    $ cp development.ini.sample development.ini
    $ cp alembic.ini.sample alembic.ini
    $ pypy setup.py develop
    $ alembic upgrade head
    $ pserve development.ini

You should now be able to visit http://localhost:6543/ and proceed on development. After you've confirmed the app is running, please see **Management Scripts** section below.
And you're done! You can now proceed to the Management Scripts section below.

Management Scripts
------------------

We currently uses CLI to manage board settings. If you use Vagrant, you will need to SSH into the development box and run the following commands before begin::

    $ vagrant ssh
    $ cd /vagrant
    $ source /srv/http/fanboi2/env/bin/activate

After you've setup the environment, the first thing you want to do is to create a new board::

    $ fb2_create_board development.ini --title Lounge --slug lounge


@@ 81,7 62,18 @@ Above commands will create a board named "Lounge" and "Demo" at ``/lounge`` and 

    $ fb2_update_board development.ini -s lounge -f description

Slug is used here to identify which board to edit. All database fields in board are editable this way. Some field, such as ``settings`` must be a **valid JSON**. Both commands also accepts ``--help`` which will display some available options.
Slug is used here to identify which board to edit. All database fields in board are editable this way. Some field, such as ``settings`` must be a **valid JSON**. Both commands also accepts ``--help`` which will display some available options. Apart from the above two scripts, there are many other commands you might be interested in, such as:

1. ``pserve development.ini`` to run the development server with `Waitress <http://waitress.readthedocs.org/en/latest/>`_.
2. ``pshell development.ini`` to get into Python console with the app loaded.
3. ``fb2_celery development.ini worker`` to start a `Celery <http://www.celeryproject.org/>`_ worker.
4. ``alembic upgrade head`` to update the database to latest version with `Alembic <http://alembic.readthedocs.org/en/latest/>`_.
5. ``brunch build`` to build assets with `Brunch <http://brunch.io/>`_ (or ``brunch watch`` to do it automatically).

Deployment
----------

For provisioning, `Fanboi Channel <https://fanboi.ch/>`_ is currently using `Docker <https://www.docker.com/>`_ for production deployment. Please see the `fanboi2-docker <https://github.com/pxfs/fanboi2-docker/>`_ for more information.

Contributing
------------


@@ 92,14 84,12 @@ We use `git-flow <https://github.com/nvie/gitflow>`_ as primary branching model.
2. Start a new feature with ``git flow feature start feature-name``.
3. After you've done, open a pull request against **develop** branch of this repo.

Please make sure that test coverage is 100% and everything passed. It's also a good idea to open a bug ticket for feature you want to implement before starting.

We have development IRC channel at `irc.freenode.net#fanboi <irc://irc.freenode.net/#fanboi>`_. Although if you want to submit patch anonymously you can also create git patch and post it to `support thread <https://fanboi.ch/lounge/1/>`_ as well.
Please make sure that test coverage is 100% and everything passed. It's also a good idea to open a bug ticket for feature you want to implement before starting. We have development IRC channel at `irc.freenode.net#fanboi <irc://irc.freenode.net/#fanboi>`_. Although if you want to submit patch anonymously you can also create git patch and post it to `support board <https://fanboi.ch/meta/>`_ as well.

License
-------

| Copyright (c) 2013, Kridsada Thanabulpong
| Copyright (c) 2013-2014, Kridsada Thanabulpong
| All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

M Vagrantfile => Vagrantfile +60 -9
@@ 1,11 1,62 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.box = "hashicorp/precise64"
  config.vm.network :forwarded_port, :guest => 80, :host => 8080
  config.vm.network :private_network, :ip => "192.168.200.100"

  config.vm.provision :ansible do |ansible|
    ansible.limit = "all"
    ansible.playbook = "provisioning/site.yml"
    ansible.inventory_path = "provisioning/development_hosts"
  end
  config.vm.box = "phusion/ubuntu-14.04-amd64"
  config.vm.network :forwarded_port, guest: 6543, host: 6543

  config.vm.provision :shell, privileged: true, inline: <<-EOF
    sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 7FCC7D46ACCC4CF8
    sudo apt-get clean
    sudo rm -rf /var/lib/apt/lists/*
    sudo rm -rf /var/lib/apt/lists/partial/*
    sudo apt-get clean

    sudo apt-get -y update
    sudo apt-get -y install curl
    sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
    curl -sL https://deb.nodesource.com/setup | sudo bash -
    sudo add-apt-repository ppa:rwky/redis
    sudo apt-get -y update

    sudo apt-get -y install git-core
    sudo apt-get -y install build-essential zlib1g-dev
    sudo apt-get -y install postgresql-9.2 postgresql-client-9.2 libpq-dev
    sudo apt-get -y install redis-server
    sudo apt-get -y install memcached
    sudo apt-get -y install nodejs

    sudo -u postgres createuser -ds vagrant || true
    sudo -u postgres createuser -ds fanboi2 || true
    sudo sh -c 'echo "local all all trust" > /etc/postgresql/9.2/main/pg_hba.conf'
    sudo sh -c 'echo "host all all 127.0.0.1/32 trust" >> /etc/postgresql/9.2/main/pg_hba.conf'
    sudo sh -c 'echo "host all all ::1/128 trust" >> /etc/postgresql/9.2/main/pg_hba.conf'
    sudo service postgresql restart
    sudo npm install -g brunch
    sudo chown -R vagrant:vagrant $HOME/.npm
  EOF

  config.vm.provision :shell, privileged: false, inline: <<-EOF
    cd /tmp
    rm -rf $HOME/pypy3
    curl -sL https://bitbucket.org/pypy/pypy/downloads/pypy3-2.3.1-linux64.tar.bz2 | tar -xjf -
    mv pypy3*/ $HOME/pypy3
    curl -sL https://bootstrap.pypa.io/ez_setup.py | $HOME/pypy3/bin/pypy
    curl -sL https://bootstrap.pypa.io/get-pip.py | $HOME/pypy3/bin/pypy
    echo '. "$HOME/.bashrc"' > $HOME/.profile
    echo 'export PATH="$HOME/pypy3/bin:$HOME/bin:$PATH"' >> $HOME/.profile

    psql template1 -c "CREATE DATABASE fanboi2_development;"
    psql template1 -c "CREATE DATABASE fanboi2_test;"

    cd /vagrant
    rm -rf fanboi2.egg-info
    rm -rf node_modules
    cp development.ini.sample development.ini
    cp alembic.ini.sample alembic.ini
    $HOME/pypy3/bin/pypy setup.py develop
    $HOME/pypy3/bin/alembic upgrade head
    npm install
    brunch build
  EOF
end

A alembic.ini.sample => alembic.ini.sample +3 -0
@@ 0,0 1,3 @@
[alembic]
script_location = %(here)s/migration
pyramid_configuration = %(here)s/development.ini

R provisioning/files/srv/settings.ini.j2 => development.ini.sample +14 -35
@@ 1,7 1,6 @@
[app:main]
use = egg:fanboi2

{% if development %}
pyramid.reload_templates = true
pyramid.debug_authorization = false
pyramid.debug_notfound = false


@@ 12,25 11,16 @@ pyramid.includes =
    pyramid_tm

debugtoolbar.hosts = 0.0.0.0/0
{% else %}
pyramid.reload_templates = false
pyramid.debug_authorization = false
pyramid.debug_notfound = false
pyramid.debug_routematch = false
pyramid.default_locale_name = en
pyramid.includes =
    pyramid_tm
{% endif %}

mako.directories = fanboi2:templates
sqlalchemy.url = postgresql://{{ db_user }}:{{ db_pass }}@{{ ansible_default_ipv4.address }}:5432/{{ db_name }}
redis.url = redis://{{ ansible_default_ipv4.address }}:6379/0
celery.broker = redis://{{ ansible_default_ipv4.address }}:6379/1
akismet.key = {{ akismet_key }}
dnsbl.providers = {{ dnsbl_providers }}
sqlalchemy.url = postgresql+pg8000://vagrant:@127.0.0.1:5432/fanboi2_development
redis.url = redis://127.0.0.1:6379/0
celery.broker = redis://127.0.0.1:6379/1
akismet.key =
dnsbl.providers =

dogpile.backend = dogpile.cache.memcached
dogpile.arguments.url = {{ ansible_default_ipv4.address }}:11211
dogpile.arguments.url = 127.0.0.1:11211
dogpile.arguments.distributed_lock = true

session.type = file


@@ 38,10 28,15 @@ session.data_dir = %(here)s/tmp/session/data
session.lock_dir = %(here)s/tmp/session/lock
session.key = _session
session.httponly = true
session.secret = {{ csrf_secret }}
session.secret = DEVELOPMENT_USE_ONLY_CHANGE_ME_IN_PROD

app.timezone = Asia/Bangkok
app.secret = DEVELOPMENT_USE_ONLY_CHANGE_ME_IN_PROD

app.timezone = {{ timezone }}
app.secret = {{ session_secret }}
[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 6543

[loggers]
keys = root, fanboi2, sqlalchemy


@@ 52,7 47,6 @@ keys = console
[formatters]
keys = generic

{% if development %}
[logger_root]
level = INFO
handlers = console


@@ 66,21 60,6 @@ qualname = fanboi2
level = INFO
handlers =
qualname = sqlalchemy.engine
{% else %}
[logger_root]
level = WARN
handlers = console

[logger_fanboi2]
level = WARN
handlers =
qualname = fanboi2

[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
{% endif %}

[handler_console]
class = StreamHandler

M fanboi2/__init__.py => fanboi2/__init__.py +1 -1
@@ 95,7 95,7 @@ def configure_components(cfg):  # pragma: no cover
    #
    # This bug only applies to Python 3.2.3 only.
    from .tasks import celery, configure_celery
    engine = engine_from_config(cfg, 'sqlalchemy.', client_encoding='utf8')
    engine = engine_from_config(cfg, 'sqlalchemy.')
    DBSession.configure(bind=engine)
    Base.metadata.bind = engine
    redis_conn.from_url(cfg['redis.url'])

M fanboi2/formatters.py => fanboi2/formatters.py +3 -2
@@ 1,12 1,12 @@
import html
import isodate
import misaka
import pytz
import re
import urllib
import urllib.parse as urlparse
from collections import OrderedDict
from html.parser import HTMLParser
from markdown import Markdown
from markupsafe import Markup




@@ 165,7 165,8 @@ def format_markdown(context, request, text):
    :rtype: Markup
    """
    if text is not None:
        return Markup(misaka.html(str(text)))
        markdown = Markdown()
        return Markup(markdown.convert(str(text)))


RE_ANCHOR = re.compile(r'%s(\d+)(\-)?(\d+)?' % html.escape('>>'))

M fanboi2/tests/__init__.py => fanboi2/tests/__init__.py +1 -1
@@ 7,7 7,7 @@ from sqlalchemy import create_engine

DATABASE_URI = os.environ.get(
    'POSTGRESQL_TEST_DATABASE',
    'postgresql://fanboi2:fanboi2@localhost:5432/fanboi2_test')
    'postgresql+pg8000://fanboi2:@localhost:5432/fanboi2_test')


class DummyRedis(object):

M fanboi2/tests/test_formatters.py => fanboi2/tests/test_formatters.py +4 -4
@@ 169,10 169,10 @@ class TestFormatters(unittest.TestCase):
        from markupsafe import Markup
        request = self._makeRequest()
        tests = [
            ('**Hello, world!**', '<p><strong>Hello, world!</strong></p>\n'),
            ('<b>Foobar</b>', '<p><b>Foobar</b></p>\n'),
            ('Split\n\nParagraph', '<p>Split</p>\n\n<p>Paragraph</p>\n'),
            ('Split\nlines', '<p>Split\nlines</p>\n'),
            ('**Hello, world!**', '<p><strong>Hello, world!</strong></p>'),
            ('<b>Foobar</b>', '<p><b>Foobar</b></p>'),
            ('Split\n\nParagraph', '<p>Split</p>\n<p>Paragraph</p>'),
            ('Split\nlines', '<p>Split\nlines</p>'),
        ]
        for source, target in tests:
            self.assertEqual(format_markdown(None, request, source),

A production.ini.sample => production.ini.sample +67 -0
@@ 0,0 1,67 @@
[app:main]
use = egg:fanboi2

pyramid.reload_templates = false
pyramid.debug_authorization = false
pyramid.debug_notfound = false
pyramid.debug_routematch = false
pyramid.default_locale_name = en
pyramid.includes = pyramid_tm

mako.directories = fanboi2:templates
sqlalchemy.url =
redis.url =
celery.broker =
akismet.key =
dnsbl.providers =

dogpile.backend = dogpile.cache.memcached
dogpile.arguments.url =
dogpile.arguments.distributed_lock = true

session.type = file
session.data_dir = %(here)s/tmp/session/data
session.lock_dir = %(here)s/tmp/session/lock
session.key = _session
session.httponly = true
session.secret =

app.timezone =
app.secret =

[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 6543

[loggers]
keys = root, fanboi2, sqlalchemy

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = WARN
handlers = console

[logger_fanboi2]
level = WARN
handlers =
qualname = fanboi2

[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s

D provisioning/development_hosts => provisioning/development_hosts +0 -2
@@ 1,2 0,0 @@
[webservers]
192.168.200.100  development=1  user=vagrant

D provisioning/files/etc/memcached.conf.j2 => provisioning/files/etc/memcached.conf.j2 +0 -6
@@ 1,6 0,0 @@
-d
logfile /var/log/memcached.log
-m 64
-p 11211
-u memcache
-l {{ ansible_default_ipv4.address }}

D provisioning/files/etc/nginx/nginx.conf.j2 => provisioning/files/etc/nginx/nginx.conf.j2 +0 -85
@@ 1,85 0,0 @@
user www-data;
worker_processes 4;
pid /var/run/nginx.pid;
error_log /var/log/nginx/error.log;

events {
  worker_connections 512;
}

http {
  include mime.types;
  default_type application/octet-stream;
  server_names_hash_bucket_size 64;
  types_hash_max_size 2048;
  keepalive_timeout 120;
  tcp_nodelay on;
  tcp_nopush on;
  sendfile on;

  gzip on;
  gzip_buffers 16 8k;
  gzip_comp_level 9;
  gzip_min_length 16;

  client_max_body_size 32m;

  {% if not development %}
  set_real_ip_from 199.27.128.0/21;
  set_real_ip_from 173.245.48.0/20;
  set_real_ip_from 103.21.244.0/22;
  set_real_ip_from 103.22.200.0/22;
  set_real_ip_from 103.31.4.0/22;
  set_real_ip_from 141.101.64.0/18;
  set_real_ip_from 108.162.192.0/18;
  set_real_ip_from 190.93.240.0/20;
  set_real_ip_from 188.114.96.0/20;
  set_real_ip_from 197.234.240.0/22;
  set_real_ip_from 198.41.128.0/17;
  set_real_ip_from 162.158.0.0/15;
  set_real_ip_from 104.16.0.0/12;
  real_ip_header CF-Connecting-IP;
  {% endif %}

  upstream backend {
    server {{ ansible_default_ipv4.address }}:{{ app_port }};
  }

  server {
    listen 80;
    access_log /var/log/nginx/access.log combined;

    {% if ssl %}
    listen 443 ssl;
    ssl_certificate /etc/nginx/certs/cert.crt;
    ssl_certificate_key /etc/nginx/certs/cert.key;
    {% endif %}

    location /static {
      alias {{ static }};
      {% if not development %}
      expires max;
      {% endif %}
    }

    location / {
      include uwsgi_params;
      uwsgi_pass backend;
      expires 0;

      include /etc/nginx/location.d/*.conf;
    }
  }

  server {
    listen 127.0.0.1:80;
    server_name 127.0.0.1;
    location /nginx_status {
      stub_status on;
      allow 127.0.0.1;
      deny all;
    }
  }

  include /etc/nginx/conf.d/*.conf;
}

D provisioning/files/etc/postgresql/9.2/main/pg_hba.conf.j2 => provisioning/files/etc/postgresql/9.2/main/pg_hba.conf.j2 +0 -7
@@ 1,7 0,0 @@
# TYPE  DATABASE        USER            ADDRESS                 METHOD
local   all             postgres                                peer
host    all             all             127.0.0.1/32            md5
host    all             all             ::1/128                 md5

# Allow connection to fanboi2 databases for app servers.
host {{ db_name }} {{ db_user }} {{ ansible_default_ipv4.address }}/32 md5

D provisioning/files/etc/postgresql/9.2/main/postgresql.conf.j2 => provisioning/files/etc/postgresql/9.2/main/postgresql.conf.j2 +0 -53
@@ 1,53 0,0 @@
data_directory = '/var/lib/postgresql/9.2/main'
hba_file = '/etc/postgresql/9.2/main/pg_hba.conf'
ident_file = '/etc/postgresql/9.2/main/pg_ident.conf'
external_pid_file = '/var/run/postgresql/9.2-main.pid'


#------------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#------------------------------------------------------------------------------

# - Connection Settings -

listen_addresses = 'localhost,{{ ansible_default_ipv4.address }}'
max_connections = 100
unix_socket_directory = '/var/run/postgresql'
port = 5432

# - Security and Authentication -
ssl = true
ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem'
ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key'


#------------------------------------------------------------------------------
# RESOURCE USAGE (except WAL)
#------------------------------------------------------------------------------

# - Memory -
shared_buffers = 24MB


#------------------------------------------------------------------------------
# ERROR REPORTING AND LOGGING
#------------------------------------------------------------------------------

# - What to Log -
log_line_prefix = '%t '
log_timezone = 'localtime'


#------------------------------------------------------------------------------
# CLIENT CONNECTION DEFAULTS
#------------------------------------------------------------------------------

# - Locale and Formatting -
datestyle = 'iso, mdy'
timezone = 'localtime'
client_encoding = 'utf8'
lc_messages = 'en_US.UTF-8'
lc_monetary = 'en_US.UTF-8'
lc_numeric = 'en_US.UTF-8'
lc_time = 'en_US.UTF-8'
default_text_search_config = 'pg_catalog.english'

D provisioning/files/etc/redis/redis.conf.j2 => provisioning/files/etc/redis/redis.conf.j2 +0 -49
@@ 1,49 0,0 @@
daemonize yes
pidfile /var/run/redis/redis-server.pid

# We need to allow remote connection.
bind {{ ansible_default_ipv4.address }}
port 6379
timeout 0

loglevel notice
logfile /var/log/redis/redis-server.log

databases 3
save 900 1
save 300 10
save 60 10000

stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb

dir /var/lib/redis
slave-serve-stale-data yes
slave-read-only yes
slave-priority 100

appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000

slowlog-log-slower-than 10000
slowlog-max-len 128

hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64

activerehashing yes

client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

D provisioning/files/etc/supervisor/conf.d/brunch.conf.j2 => provisioning/files/etc/supervisor/conf.d/brunch.conf.j2 +0 -6
@@ 1,6 0,0 @@
[program:brunch]
command = brunch watch
autorestart = true
startretries = 65535
user = {{ user }}
directory = {{ root }}

D provisioning/files/etc/supervisor/conf.d/celery.conf.j2 => provisioning/files/etc/supervisor/conf.d/celery.conf.j2 +0 -6
@@ 1,6 0,0 @@
[program:celery]
command = {{ virtualenv }}/bin/fb2_celery {{ base }}/settings.ini worker
autorestart = true
startretries = 65535
user = {{ user }}
directory = {{ root }}

D provisioning/files/etc/supervisor/conf.d/uwsgi.conf.j2 => provisioning/files/etc/supervisor/conf.d/uwsgi.conf.j2 +0 -5
@@ 1,5 0,0 @@
[program:uwsgi]
command = uwsgi --master --emperor /etc/uwsgi
autorestart = true
startretries = 65535
user = root

D provisioning/files/etc/uwsgi/fanboi2.ini.j2 => provisioning/files/etc/uwsgi/fanboi2.ini.j2 +0 -15
@@ 1,15 0,0 @@
[uwsgi]
virtualenv = {{ virtualenv }}
wsgi-file = {{ base }}/fanboi2.wsgi
uid = {{ user }}
gid = {{ user }}
socket = {{ ansible_default_ipv4.address }}:{{ app_port }}
max-requests = 1000
{% if development %}
python-autoreload = 3
{% else %}
cheaper = {{ min_workers }}
workers = {{ max_workers }}
cheaper-initial = {{ min_workers }}
cheaper-algo = spare
{% endif %}

D provisioning/files/srv/alembic.ini.j2 => provisioning/files/srv/alembic.ini.j2 +0 -13
@@ 1,13 0,0 @@
[alembic]
# path to migration scripts
script_location = {{ root }}/migration

# path to Pyramid configuration to initialize
pyramid_configuration = {{ base }}/settings.ini

# template used to generate migration files
# file_template = %%(rev)s_%%(slug)s

# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false

D provisioning/files/srv/fanboi2.wsgi.j2 => provisioning/files/srv/fanboi2.wsgi.j2 +0 -4
@@ 1,4 0,0 @@
from pyramid.paster import get_app, setup_logging
ini_path = '{{ base }}/settings.ini'
setup_logging(ini_path)
application = get_app(ini_path, 'main')

D provisioning/genconfig.py => provisioning/genconfig.py +0 -36
@@ 1,36 0,0 @@
import os
from jinja2 import Environment, FileSystemLoader

pvdir = os.path.dirname(os.path.abspath(__file__))
basedir = os.path.abspath(os.path.join(pvdir, os.path.pardir))
jinja_env = Environment(loader=FileSystemLoader(pvdir))

default_config = {
    'development': True,
    'db_user': 'fanboi2',
    'db_pass': 'fanboi2',
    'db_name': 'fanboi2',
    'csrf_secret': 'changeme',
    'session_secret': 'changeme',
    'timezone': 'Asia/Bangkok',

    # Path stubs.
    'base': basedir,
    'root': os.path.join(basedir, 'app'),

    # Ansible stubs.
    'ansible_default_ipv4': {'address': '127.0.0.1'},
}


def gen(source, dest):
    tmpl = jinja_env.get_template(source)
    output = tmpl.render(default_config)
    with open(os.path.join(basedir, dest), 'wb') as f:
        f.write(bytes(output.encode('utf-8')))
    print('%s written.' % dest)


if __name__ == '__main__':
    gen('files/srv/settings.ini.j2', 'settings.ini')
    gen('files/srv/alembic.ini.j2', 'alembic.ini')

D provisioning/group_vars/all.yml => provisioning/group_vars/all.yml +0 -27
@@ 1,27 0,0 @@
---
development: 0
ssl: 0

# Application config
timezone: "Asia/Bangkok"
csrf_secret: "changeme"
session_secret: "changeme"
dnsbl_providers: "zen.spamhaus.org"
akismet_key: ""

# Application paths
base: "/srv/http/fanboi2"
root: "{{base}}/app"
static: "{{root}}/fanboi2/static"
virtualenv: "{{base}}/env"

# PostgreSQL
db_name: "fanboi2"
db_test: "fanboi2_test"
db_user: "fanboi2"
db_pass: "fanboi2"

# Worker
app_port: 9000
min_workers: 2
max_workers: 4

D provisioning/site.yml => provisioning/site.yml +0 -317
@@ 1,317 0,0 @@
---
- hosts: all
  sudo: yes
  handlers:
    - name: restart nginx
      service: name=nginx state=restarted

    - name: restart redis
      service: name=redis-server state=restarted

    - name: restart supervisor
      service: name=supervisor state=restarted

    - name: restart memcached
      service: name=memcached state=restarted

  vars:
    - python32: "{{virtualenv}}/bin/python3"
    - pip32: "{{virtualenv}}/bin/pip-3.2"

  tasks:
    - name: ensure package index is latest
      apt: update_cache=yes upgrade=safe cache_valid_time=604800

    - name: install prerequisites packages
      apt: "pkg={{item}} state=latest"
      with_items:
        - python-software-properties
        - git-core

    # Nginx setup.
    # ---------------------------------------------------------------------

    - name: add ppa:nginx/stable for nginx installation
      apt_repository: repo=ppa:nginx/stable update_cache=yes

    - name: install nginx
      apt: pkg=nginx state=latest

    - name: setup nginx configuration
      template: src=files/etc/nginx/nginx.conf.j2 dest=/etc/nginx/nginx.conf
      notify: restart nginx

    - name: create location configuration directory
      file: path=/etc/nginx/location.d owner=root group=root state=directory

    - name: create nginx certs directory
      file: path=/etc/nginx/certs owner=root group=root state=directory
      when: ssl|int == 1

    - name: cleanup default nginx configuration file
      file: dest=/etc/nginx/sites-enabled/default state=absent
      notify: restart nginx

    # PostgreSQL setup.
    # ---------------------------------------------------------------------

    - name: add official postgresql ubuntu repository key
      apt_key:
        id: ACCC4CF8
        url: https://www.postgresql.org/media/keys/ACCC4CF8.asc

    - name: add official postgresql ubuntu repository
      apt_repository:
        repo: deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main
        update_cache: yes

    - name: install postgresql and its related packages
      apt: "pkg={{item}} state=latest"
      with_items:
        - postgresql-9.2
        - libpq-dev
        - python-psycopg2

    - name: update postgresql configuration file
      template:
        src: files/etc/postgresql/9.2/main/postgresql.conf.j2
        dest: /etc/postgresql/9.2/main/postgresql.conf

    - name: update pg_hba to allow appservers
      template:
        src: files/etc/postgresql/9.2/main/pg_hba.conf.j2
        dest: /etc/postgresql/9.2/main/pg_hba.conf
      register: pgconfig_updated

    - name: restart postgresql server for new configuration
      service: name=postgresql state=restarted
      when: pgconfig_updated.changed

    # Redis setup.
    # ---------------------------------------------------------------------

    - name: add ppa:chris-lea/redis-server for redis installation
      apt_repository: repo=ppa:chris-lea/redis-server update_cache=yes

    - name: install redis
      apt: pkg=redis-server state=latest

    - name: update redis configuration file
      template:
        src: files/etc/redis/redis.conf.j2
        dest: /etc/redis/redis.conf
      notify: restart redis

    # Memcached setup.
    # ---------------------------------------------------------------------

    - name: install memcached
      apt: pkg=memcached state=latest

    - name: update memcached configuration file
      template:
        src: files/etc/memcached.conf.j2
        dest: /etc/memcached.conf
      notify: restart memcached

    # Nodejs setup.
    # ---------------------------------------------------------------------

    - name: add ppa:chris-lea/node.js for node.js installation
      apt_repository: repo=ppa:chris-lea/node.js update_cache=yes

    - name: install node.js
      apt: pkg=nodejs state=latest

    # Python setup.
    # ---------------------------------------------------------------------

    - name: install python3 and related packages
      apt: "pkg={{item}} state=latest"
      with_items:
        - python3.2
        - python3.2-dev
        - python3-setuptools
        - build-essential
        - libpq-dev
        - git

    # pip 1.5 and virtualenv 1.11 is incompatible with system setuptools.
    - name: install python3 requisites
      command: "easy_install3 {{item}}"
      args:
        creates: "/usr/local/bin/{{item}}-3.2"
      with_items:
        - pip==1.4.1
        - virtualenv==1.10.1

    # uWSGI setup.
    # ---------------------------------------------------------------------

    - name: install uwsgi to system python
      pip: name=uwsgi

    - name: ensure that uwsgi configuration dir exists
      file: path=/etc/uwsgi state=directory

    # TODO: Remove me.
    - name: remove uwsgi startup files
      file: dest=/etc/init/uwsgi.conf state=absent

    # Workspace setup.
    # ---------------------------------------------------------------------

    - name: ensure that user to run the app exists
      user: "name={{user}} state=present"

    # In case we use www-data as {{user}}, /var/www isn't properly chown'ed.
    - name: fix /var/www permission
      file: "path=/var/www owner={{user}} group=root state=directory"

    - name: create a working directory
      file: "path={{base}} owner={{user}} state=directory"

    - name: link vagrant path to server directory for development
      sudo_user: "{{user}}"
      file: "src=/vagrant dest={{root}} state=link"
      when: development|int == 1

    - name: clone application to root directory
      sudo_user: "{{user}}"
      register: fanboi2_cloned
      when: development|int == 0
      git:
        repo: https://github.com/pxfs/fanboi2.git
        dest: "{{root}}"
        version: "feature/experiment-view2"

    # Application setup.
    # ---------------------------------------------------------------------

    - name: setup virtualenv for application directory
      sudo_user: "{{user}}"
      command: "virtualenv -p python3 {{virtualenv}}"
      args:
        creates: "{{virtualenv}}"

    - name: install application in development mode
      sudo_user: "{{user}}"
      command: |
        env LANG=en_US.UTF-8 {{pip32}} install -e {{root}} --use-mirrors
        chdir={{root}}
      when: development|int == 1 or fanboi2_cloned.changed

    - name: create configuration file for application
      sudo_user: "{{user}}"
      template: "src=files/srv/settings.ini.j2 dest={{base}}/settings.ini"

    - name: create configuration file for alembic
      sudo_user: "{{user}}"
      template: "src=files/srv/alembic.ini.j2 dest={{base}}/alembic.ini"

    - name: create uwsgi file for application
      sudo_user: "{{user}}"
      template: "src=files/srv/fanboi2.wsgi.j2 dest={{base}}/fanboi2.wsgi"

    - name: create database user for application
      sudo_user: postgres
      postgresql_user:
        user: "{{db_user}}"
        password: "{{db_pass}}"

    - name: create database for application
      sudo_user: postgres
      postgresql_db:
        db: "{{item}}"
        owner: "{{db_user}}"
        encoding: 'UTF-8'
        lc_collate: 'en_US.UTF-8'
        lc_ctype: 'en_US.UTF-8'
      with_items:
        - "{{db_name}}"
        - "{{db_test}}"

    - name: sync database to latest version
      sudo_user: "{{user}}"
      command: "{{virtualenv}}/bin/alembic upgrade head chdir={{base}}"
      when: development|int == 1 or fanboi2_cloned.changed

    - name: startup application with uwsgi
      template: src=files/etc/uwsgi/fanboi2.ini.j2 dest=/etc/uwsgi/fanboi2.ini

    # Assets compilation.
    # ---------------------------------------------------------------------

    # Minification broken in 1.7.12:
    # https://github.com/brunch/brunch/issues/750
    - name: install brunch for assets compilation
      npm: name=brunch@1.7.10 global=yes

    - name: install fanboi2 brunch requirements
      sudo_user: "{{user}}"
      npm: "path={{root}}"

    - name: compile fanboi2 assets
      sudo_user: "{{user}}"
      command: "brunch build --production chdir={{root}}"
      when: development|int == 1 or fanboi2_cloned.changed

    # Processes.
    # ---------------------------------------------------------------------

    - name: install supervisor for process management
      apt: pkg=supervisor state=latest

    - name: enable brunch watch for fanboi2
      template:
        src: files/etc/supervisor/conf.d/brunch.conf.j2
        dest: /etc/supervisor/conf.d/brunch.conf
      notify: restart supervisor
      when: development|int == 1

    - name: enable uwsgi for fanboi2
      template:
        src: files/etc/supervisor/conf.d/uwsgi.conf.j2
        dest: /etc/supervisor/conf.d/uwsgi.conf
      notify: restart supervisor

    - name: enable celery for fanboi2
      template:
        src: files/etc/supervisor/conf.d/celery.conf.j2
        dest: /etc/supervisor/conf.d/celery.conf
      notify: restart supervisor

    - name: restart celery on changes
      command: supervisorctl restart celery
      when: development|int == 1 or fanboi2_cloned.changed

    - name: restart uwsgi on changes
      command: supervisorctl restart uwsgi
      when: development|int == 1 or fanboi2_cloned.changed

    # Firewall configuration.
    # ---------------------------------------------------------------------

    - name: install ufw
      apt: pkg=ufw state=latest

    - name: check installation status for ufw
      register: ufw_setup
      shell: grep ENABLED=yes /etc/ufw/ufw.conf && echo '1' || echo '0'
      changed_when: ufw_setup.stdout == '0'

    - name: allow ssh via ufw
      command: ufw allow 22/tcp
      when: ufw_setup.stdout == '0'

    - name: allow http via ufw
      command: ufw allow 80/tcp
      when: ufw_setup.stdout == '0'

    - name: allow https via ufw
      command: ufw allow 443/tcp
      when: ufw_setup.stdout == '0'

    - name: enable ufw
      shell: echo 'y' | ufw enable
      when: ufw_setup.stdout == '0'

M setup.py => setup.py +2 -3
@@ 20,10 20,9 @@ requires = [
    'alembic >=0.6.2, <0.7',
    'celery >=3.1, <3.2',
    'transaction',
    'psycopg2',
    'pg8000',
    'zope.sqlalchemy',
    'redis',
    'hiredis',
    'dogpile.cache',
    'python3-memcached',
    'pytz',


@@ 32,7 31,7 @@ requires = [

    # Frontend
    'isodate',
    'misaka',
    'Markdown',  # Deprecate me.

    # Tests
    'nose',