~sirn/fanboi2

834edf0edc5dd633c0ecea16231b6ed2d728476d — Kridsada Thanabulpong 3 years ago 9879622
Massive cleanup in preparation for 0.30 (#25)

* Only use environment variable to configure the application.

  Closes #17

  In order to simplify deployment on containerized or 12-factor centric
  environment (such as Docker or Heroku) where writing configuration
  file is not preferrable.

  We also prefer to have only a single way to configure application,
  which means Pyramid-style ini configurations are now dropped in favor
  for environment variable.

* Refactor post utils into post filters

  Closes #23

  Akismet, DNSBL and proxy checking are now refactored into a 'filter'
  which can be easily extended.

* Replaced Beaker with PyNaCl-based session storage.

  This change switched session storage from Memcached-based session
  storage to a cookie-based session storage with strong encryption
  using PyNaCl.

* Replaced app-specific settings with setting model.

  Application specific settings are now stored in the database instead
  of configuration file as these configurations are not required
  for running the app and may require a bit more complicate data
  structure.

* Add services layer

  Closes #18

  We've added services layer and successfully migrated the application
  to use services rather than calling components directly. This change
  will make it easier to extend as well as optimizing the application
  in the future.

  This change also allowed us to remove complexity from model, such as
  that post-commit hooks are now done within the services layer.
  Caching is also implemented in some services, such as settings.

* Explicitly setting up routes.

  We will add authentication and authorization, which requires
  flexibility on the routing configuration, which means the helper in
  its current form is no longer preferrable.

* Use Pyramid native CSRF implementation.

  We have now switched to Pyramid's native CSRF implementation instead
  of using home-grown SecureForm. This change resulted in a simplified
  form handling throughout the application.

* Replaced fb2 maintenance scripts.

  As we will have a proper admin interface, these scripts are no
  longer necessary and its usage are discouraged. Due to this, we are
  dropping all the maintenance scripts in favor for a more specific
  fbctl.

* Remove PyCharm-style type annotations.

  To be replaced with Python 3.5-style type annotations in order to
  incorporate type checking with MyPy.

* Consolidate assets directory for concerns separation.

  Move all assets-related files into assets/ so it's clear that which
  files are responsible for compiling assets (and thus purgable after
  assets are compiled when building images).

* Convert JsonType field to native JSON field.

  SQLALchemy has since supported the native JSON field in PostgreSQL
  and it is no longer necessary to use our own implementation of JSON
  field that was a wrapper over Text.

* Cleanup migration files

  Styling fixes per Flake8.

* Update dependencies.

  Make sure dependencies are up-to-date.
128 files changed, 9734 insertions(+), 7132 deletions(-)

M .gitignore
M .travis.yml
M CHANGES.rst
M README.rst
M Vagrantfile
R examples/alembic.ini.sample => alembic.ini
R gulpfile.js => assets/gulpfile.js
R package.json => assets/package.json
R tsconfig.json => assets/tsconfig.json
R typings.json => assets/typings.json
R yarn.lock => assets/yarn.lock
D examples/README.rst
D examples/development.ini.sample
D examples/production.ini.sample
M fanboi2/__init__.py
M fanboi2/cache.py
A fanboi2/cmd/__init__.py
A fanboi2/cmd/celery.py
A fanboi2/cmd/ctl.py
M fanboi2/errors.py
A fanboi2/filters/__init__.py
A fanboi2/filters/akismet.py
R fanboi2/{utils/dnsbl.py => filters/dnsbl.py}
R fanboi2/{utils/proxy.py => filters/proxy.py}
M fanboi2/forms.py
A fanboi2/geoip.py
M fanboi2/helpers/formatters.py
M fanboi2/helpers/partials.py
A fanboi2/interfaces.py
M fanboi2/models/__init__.py
M fanboi2/models/_base.py
D fanboi2/models/_identity.py
D fanboi2/models/_redis_proxy.py
M fanboi2/models/_versioned.py
M fanboi2/models/board.py
M fanboi2/models/page.py
M fanboi2/models/post.py
M fanboi2/models/rule.py
M fanboi2/models/rule_ban.py
D fanboi2/models/rule_override.py
A fanboi2/models/setting.py
M fanboi2/models/topic.py
A fanboi2/redis.py
D fanboi2/scripts/__init__.py
D fanboi2/scripts/board_create.py
D fanboi2/scripts/board_update.py
D fanboi2/scripts/celery.py
D fanboi2/scripts/topic_sync.py
M fanboi2/serializers.py
A fanboi2/services/__init__.py
A fanboi2/services/board.py
A fanboi2/services/filter_.py
A fanboi2/services/identity.py
A fanboi2/services/page.py
A fanboi2/services/post.py
A fanboi2/services/rate_limiter.py
A fanboi2/services/rule.py
A fanboi2/services/setting.py
A fanboi2/services/task.py
A fanboi2/services/topic.py
D fanboi2/tasks.py
A fanboi2/tasks/__init__.py
A fanboi2/tasks/_base.py
A fanboi2/tasks/_result_proxy.py
A fanboi2/tasks/post.py
A fanboi2/tasks/topic.py
M fanboi2/templates/api/_other.mako
A fanboi2/templates/bad_request.mako
M fanboi2/templates/boards/_subheader.mako
D fanboi2/templates/boards/error_ban.mako
D fanboi2/templates/boards/error_dnsbl.mako
D fanboi2/templates/boards/error_proxy.mako
D fanboi2/templates/boards/error_rate.mako
D fanboi2/templates/boards/error_spam.mako
M fanboi2/templates/boards/new.mako
R fanboi2/templates/boards/{error_status.mako => new_error.mako}
M fanboi2/templates/partials/_layout.mako
M fanboi2/templates/topics/_subheader.mako
D fanboi2/templates/topics/error_ban.mako
D fanboi2/templates/topics/error_dnsbl.mako
D fanboi2/templates/topics/error_proxy.mako
D fanboi2/templates/topics/error_rate.mako
D fanboi2/templates/topics/error_spam.mako
M fanboi2/templates/topics/show.mako
R fanboi2/templates/topics/{error_status.mako => show_error.mako}
M fanboi2/tests/__init__.py
M fanboi2/tests/test_app.py
M fanboi2/tests/test_cache.py
M fanboi2/tests/test_errors.py
A fanboi2/tests/test_filters.py
M fanboi2/tests/test_forms.py
M fanboi2/tests/test_helpers.py
A fanboi2/tests/test_integrations.py
M fanboi2/tests/test_models.py
M fanboi2/tests/test_serializers.py
A fanboi2/tests/test_services.py
M fanboi2/tests/test_tasks.py
D fanboi2/tests/test_utils.py
D fanboi2/tests/test_views.py
D fanboi2/utils/__init__.py
D fanboi2/utils/akismet.py
D fanboi2/utils/checklist.py
D fanboi2/utils/geoip.py
D fanboi2/utils/rate_limiter.py
D fanboi2/utils/request.py
M fanboi2/version.py
M fanboi2/views/api.py
M fanboi2/views/boards.py
M fanboi2/views/pages.py
A fanboi2/wsgi.py
D migration/README.rst
M migration/env.py
M migration/script.py.mako
A migration/versions/05314e264e76_change_string_to_inet.py
M migration/versions/0d0f281cc4ec_create_versioning_tables.py
A migration/versions/0df7bc63ddf1_delete_rule_override_table.py
M migration/versions/28d3c8870c89_create_topic_meta_table.py
M migration/versions/38f5ad30fe6f_create_initial_table.py
A migration/versions/53ab606666d3_change_jsontype_to_json.py
A migration/versions/57b184a0bac9_create_setting_table.py
M migration/versions/6af2b8c6dc3a_add_scope_column_to_rule.py
M migration/versions/7224deb8bfa9_create_page_table.py
M migration/versions/84a168aadc17_add_versioning_columns.py
M migration/versions/a6f20e3c63c2_create_rule_tables.py
M migration/versions/bfbd6a58775c_add_status_column_to_board.py
M migration/versions/c71cae24d111_add_bumped_column.py
M setup.cfg
M setup.py
M .gitignore => .gitignore +13 -21
@@ 14,28 14,20 @@ tmp/
*.dump

# Assets
typings/*
!typings/vendor/
/assets/*/
!/assets/app/
!/assets/vendor/
!/assets/legacy/

# IDE stuff
out/
fanboi2.iml
.idea

# Testing leftovers
# Development
/out/
/venv/
*.iml
*.ini
!alembic.ini
*.gz
.dir-locals.el
.coverage
.noseids

# Assets
node_modules/
results/
logs/
*.min.js

# Vagrant stuff
.idea
.vagrant
*.gz

# User settings
development.ini
alembic.ini

M .travis.yml => .travis.yml +1 -1
@@ 2,7 2,7 @@ language: python
script: nosetests

python:
  - "3.5"
  - "3.6"

install:
  - 'pip install -e .'

M CHANGES.rst => CHANGES.rst +10 -2
@@ 1,5 1,13 @@
Next
====
r30
======

- [Change] Major refactoring to utilizes `pyramid_services <https://github.com/mmerickel/pyramid_services>`.
- [Change] Application now uses environment variable as a primary means for configuration.
- [Change] Switch to use `Pyramid's native CSRF checking <https://docs.pylonsproject.org/projects/pyramid/en/latest/api/csrf.html>`.
- [Change] Switch to use `PyNaCl <https://github.com/Pylons/pyramid_nacl_session/>`_ for session factory.

0.10.2
------

- [Add] Allow post filter to be configured per country.
- [Add] A ``fb2_topic_sync`` script for syncing topic's bumped timestamp.

M README.rst => README.rst +48 -66
@@ 1,93 1,72 @@
Fanboi2 |ci|
============
=======
Fanboi2
=======

Board engine behind `Fanboi Channel <https://fanboi.ch/>`_ written in Python.
|py| |ci|

.. |ci| image:: https://img.shields.io/travis/pxfs/fanboi2.svg?style=flat-square
        :target: https://travis-ci.org/pxfs/fanboi2
Board engine behind `Fanboi Channel`_ written in Python.

Getting Started
---------------
.. |py| image::
        https://img.shields.io/badge/python-3.6-blue.svg
        :target: https://docs.python.org/3/whatsnew/3.6.html

There are two ways of getting the app up and running for development. Using `Vagrant`_ (*The Better Way*) or installing everything manually (*The Adventurous Way*). The recommened method is to use Vagrant as it closely replicates the production environment.
.. |ci| image::
        https://img.shields.io/travis/forloopend/fanboi2.svg
        :target: https://travis-ci.org/forloopend/fanboi2

The Better Way
~~~~~~~~~~~~~~

The easiest way to get the app running is to run `Vagrant`_. You can follow these steps to get the app running:

1. Install `Vagrant`_ of your preferred platform.
2. Install `VirtualBox <https://www.virtualbox.org/>`_ or other `providers <http://docs.vagrantup.com/v2/providers/index.html>`_ supported by Vagrant.
3. Run ``vagrant up`` and read `Getting Started <http://docs.vagrantup.com/v2/getting-started/index.html>`_ while waiting.
4. Run ``vagrant ssh`` to SSH into the development machine.

Once the development box is up and running, you may now run the server::

    $ vagrant ssh
    $ cd /vagrant
    $ pserve development.ini

Now you're done! You can now proceed to the Management Scripts section below.

The Adventurous Way
~~~~~~~~~~~~~~~~~~~

If you don't really want to use Vagrant, you can also install everything using your preferred methods:

1. `Python 3.6 <https://www.python.org/downloads/>`_.
2. `PostgreSQL 9.5 <http://www.postgresql.org/>`_.
3. `Redis 3.0 <http://redis.io/>`_.
4. `Memcached 1.4 <http://www.memcached.org/>`_.
5. `Node.js 6.2 <http://nodejs.org/>`_ with `Yarn`_.

After the package above are up and running, you may now setup the application::
Installation
------------

    $ cp examples/development.ini.sample development.ini
    $ cp examples/alembic.ini.sample alembic.ini
    $ python3.6 setup.py develop
    $ alembic upgrade head
    $ pserve development.ini
Fanboi2 requires that the following softwares are installed.

And you're done! You can now proceed to the Management Scripts section below.
- `Python 3.6 <https://www.python.org/downloads/>`_
- `PostgreSQL 9.6 <http://www.postgresql.org/>`_
- `Redis 4.0 <http://redis.io/>`_
- `NodeJS 9.10 <http://nodejs.org/>`_ with `Yarn 1.5 <https://yarnpkg.com/>`_

Assets
------
After package mentioned are installed or started, you may now clone the application::

The application doesn't come with assets compiled by default and are done externally via `Gulp`_ and `Typings`_. To compile assets, make sure Yarn packages are up-to-date and run ``typings`` and ``gulp`` accordingly::
  $ git clone https://github.com/forloopend/fanboi2.git fanboi2

    $ yarn
    $ yarn run typings install
    $ yarn run gulp
Then setup the application::

Once these commands are run, assets will be compiled to ``fanboi2/static`` in which you should point the web server to it. You should do this on every deploy.
  $ cd fanboi2/
  $ pip3 install -e .
  $ alembic upgrade head

Management Scripts
------------------
  $ cd assets/
  $ yarn
  $ yarn run typings install
  $ yarn run gulp

After you've setup the environment, the first thing you want to do is to create a new board::
  $ fbctl serve

    $ fb2_board_create development.ini --title Lounge --slug lounge
    $ fb2_board_create development.ini --title Demo --slug demo
And you're done! Please visit `http://localhost:6543/panel <http://localhost:6543/panel>`_ to perform initial configuration.

Above commands will create a board named "Lounge" and "Demo" at ``/lounge`` and ``/demo`` respectively. Now if you want to update something such as description, you can now do::
Development
-----------

    $ fb2_board_update development.ini -s lounge -f description
To develop Fanboi2, it is highly recommended to use `Vagrant`_ as it is currently replicating the production environment of `Fanboi Channel`_. You can follow these steps to get the app running:

Slug is used here to identify which board to edit. All database fields in board are editable this way. Some field, such as ``settings`` must be a **valid JSON**. Both commands also accepts ``--help`` which will display some available options. Apart from the above two scripts, there are many other commands you might be interested in, such as:
- Install `Vagrant`_ of your preferred platform.
- Install `VirtualBox`_ or other providers supported by Vagrant.
- Run `vagrant up` and read Getting Started while waiting.
- Run `vagrant ssh` to SSH into the development machine.

1. ``pserve development.ini`` to run the development server with `Waitress <http://waitress.readthedocs.org/en/latest/>`_.
2. ``pshell development.ini`` to get into Python console with the app loaded.
3. ``fb2_celery development.ini worker`` to start a `Celery <http://www.celeryproject.org/>`_ worker.
4. ``alembic upgrade head`` to update the database to latest version with `Alembic <http://alembic.readthedocs.org/en/latest/>`_.
Once the development box is up and running, you can now run the server (inside the development machine)::

Celery worker is required to be run if you want to enable posting features.
    $ cd /vagrant
    $ fbctl serve

Contributing
------------

It's a good idea to open a bug ticket for feature you want to implement before starting and please make sure the code is tested. We have development IRC channel at `irc.fanboi.ch#fanboi <irc://irc.fanboi.ch/#fanboi>`_. Although if you want to submit patch anonymously you can also create git patch and post it to `support board <https://fanboi.ch/meta/>`_ as well.
Fanboi2 is open to any contributors, whether you're learning Python or an expert. Simply open a `pull request <https://github.com/forloopend/fanboi2/pulls>`_ against the **master** branch. Our reviewer will review and merge the pull request as soon as possible. It would be much appreciated if you could follow the following guidelines:

After you have done, simply open a pull request against **master** branch.
- It's always a good idea to open `an issue <https://github.com/forloopend/fanboi2/issues>`_ prior to starting.
- No need for 100% coverage but please make sure new features has bare minimum tests.
- Remember to run `flake8 <https://pypi.python.org/pypi/flake8>`_ and fix any styling issues.
- After done, simply open a `pull request <https://github.com/forloopend/fanboi2/pulls>`_ against **master** branch.

License
-------


@@ 102,7 81,10 @@ Redistribution and use in source and binary forms, with or without modification,

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

.. _Fanboi Channel: https://fanboi.ch/
.. _Waitress: https://docs.pylonsproject.org/projects/waitress/en/latest/
.. _Vagrant: https://www.vagrantup.com/
.. _VirtualBox: https://www.virtualbox.org/
.. _Yarn: https://yarnpkg.com/
.. _Gulp: http://gulpjs.com/
.. _Typings: https://github.com/typings/typings

M Vagrantfile => Vagrantfile +11 -4
@@ 52,16 52,23 @@ Vagrant.configure("2") do |config|
    echo 'PATH="$HOME/bin:$PATH"' >> $HOME/.profile
    echo 'export PATH' >> $HOME/.profile

    psql template1 -c "CREATE DATABASE fanboi2_development;"
    psql template1 -c "CREATE DATABASE fanboi2_dev;"
    psql template1 -c "CREATE DATABASE fanboi2_test;"

    cd /vagrant
    cp examples/development.ini.sample development.ini
    cp examples/alembic.ini.sample alembic.ini
    echo 'CELERY_BROKER_URL="redis://127.0.0.1:6379/1"; export CELERY_BROKER_URL' >> $HOME/.profile
    echo 'DATABASE_URL="postgresql://vagrant:@127.0.0.1:5432/fanboi2_dev"; export DATABASE_URL' >> $HOME/.profile
    echo 'MEMCACHED_URL="127.0.0.1:11211"; export MEMCACHED_URL' >> $HOME/.profile
    echo 'REDIS_URL="redis://127.0.0.1:6379/0"; export REDIS_URL' >> $HOME/.profile
    echo 'SERVER_DEV=true; export SERVER_DEV' >> $HOME/.profile
    echo 'SERVER_HOST="0.0.0.0"; export SERVER_HOST' >> $HOME/.profile
    echo 'SERVER_PORT=6543; export SERVER_PORT' >> $HOME/.profile
    echo "SESSION_SECRET=$(openssl rand -hex 32); export SESSION_SECRET" >> $HOME/.profile

    cd /vagrant
    $HOME/python3.6/bin/pip3 install -e .
    $HOME/python3.6/bin/alembic upgrade head

    cd /vagrant/assets
    env PYTHON=/usr/local/bin/python2.7 $HOME/yarn/bin/yarn
    $HOME/yarn/bin/yarn run typings install
    $HOME/yarn/bin/yarn run gulp

R examples/alembic.ini.sample => alembic.ini +1 -2
@@ 1,3 1,2 @@
[alembic]
script_location = %(here)s/migration
pyramid_configuration = %(here)s/development.ini
script_location = %(here)s/migration
\ No newline at end of file

R gulpfile.js => assets/gulpfile.js +14 -14
@@ 20,38 20,38 @@ var paths = {

    /* Path for storing application-specific assets. */
    app: {
        assets: 'assets/app/assets/*',
        assets: 'app/assets/*',
        stylesheets: [
            'assets/app/stylesheets/app.scss',
            'assets/app/stylesheets/*.scss',
            'assets/app/stylesheets/themes/*.scss'
            'app/stylesheets/app.scss',
            'app/stylesheets/*.scss',
            'app/stylesheets/themes/*.scss'
        ],
        javascripts: {
            glob: 'assets/app/javascripts/**/*.ts',
            base: 'assets/app/javascripts/',
            glob: 'app/javascripts/**/*.ts',
            base: 'app/javascripts/',
            typings: 'typings/index.d.ts',
            entry: 'assets/app/javascripts/app.ts'
            entry: 'app/javascripts/app.ts'
        }
    },

    /* Path for storing third-party assets. */
    vendor: {
        assets: 'assets/vendor/assets/*',
        stylesheets: 'assets/vendor/stylesheets/**/*.css',
        assets: 'vendor/assets/*',
        stylesheets: 'vendor/stylesheets/**/*.css',
        javascripts: [
            'assets/vendor/javascripts/**/*.js'
            'vendor/javascripts/**/*.js'
        ]
    },

    /* Path for storing compatibility assets. */
    legacy: {
        assets: 'assets/legacy/assets/*',
        stylesheets: 'assets/legacy/stylesheets/**/*.css',
        javascripts: 'assets/legacy/javascripts/**/*.js'
        assets: 'legacy/assets/*',
        stylesheets: 'legacy/stylesheets/**/*.css',
        javascripts: 'legacy/javascripts/**/*.js'
    },

    /* Path to output compiled assets to. */
    dest: 'fanboi2/static'
    dest: '../fanboi2/static'
};



R package.json => assets/package.json +2 -2
@@ 2,8 2,8 @@
  "author": "Kridsada Thanabulpong",
  "name": "fanboi2",
  "description": "Assets for Fanboi2 board app",
  "version": "0.10.1",
  "repository": "https://github.com/pxfs/fanboi2.git",
  "version": "0.30.0",
  "repository": "https://github.com/forloopend/fanboi2.git",
  "license": "BSD-3-Clause",
  "dependencies": {
    "autoprefixer": "^6.0.3",

R tsconfig.json => assets/tsconfig.json +0 -0
R typings.json => assets/typings.json +0 -0
R yarn.lock => assets/yarn.lock +0 -0
D examples/README.rst => examples/README.rst +0 -9
@@ 1,9 0,0 @@
Examples
========

This directory contains example configuration for the project. You usually want to copy these files to the root directory according to environment.::

    $ cp alembic.ini.sample ../
    $ cp development.ini.sample ../

Alembic configuration by default will read configuration from ``development.ini``. You may need to change this to point to ``production.ini`` in production environment.
\ No newline at end of file

D examples/development.ini.sample => examples/development.ini.sample +0 -77
@@ 1,77 0,0 @@
[app:main]
use = egg:fanboi2

pyramid.reload_templates = true
pyramid.debug_authorization = false
pyramid.debug_notfound = false
pyramid.debug_routematch = false
pyramid.default_locale_name = en
pyramid.includes =
    pyramid_debugtoolbar
    pyramid_tm

debugtoolbar.hosts = 0.0.0.0/0

mako.directories = fanboi2:templates
sqlalchemy.url = postgresql://vagrant:@127.0.0.1:5432/fanboi2_development
redis.url = redis://127.0.0.1:6379/0
celery.broker = redis://127.0.0.1:6379/1

dogpile.backend = dogpile.cache.memcached
dogpile.arguments.url = 127.0.0.1:11211
dogpile.arguments.distributed_lock = true

session.type = ext:memcached
session.key = _session
session.url = 127.0.0.1:11211
session.httponly = true
session.secret = DEVELOPMENT_USE_ONLY_CHANGE_ME_IN_PROD

app.timezone = Asia/Bangkok
app.secret = DEVELOPMENT_USE_ONLY_CHANGE_ME_IN_PROD
app.akismet_key =
app.dnsbl_providers =
app.proxy_detect.providers =
app.proxy_detect.blackbox.url =
app.proxy_detect.getipintel.url =
app.proxy_detect.getipintel.email =
app.proxy_detect.getipintel.flags =
app.geoip2_database =
app.checklist = */*

[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 6543

[loggers]
keys = root, fanboi2, sqlalchemy

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = INFO
handlers = console

[logger_fanboi2]
level = DEBUG
handlers =
qualname = fanboi2

[logger_sqlalchemy]
level = INFO
handlers =
qualname = sqlalchemy.engine.base.Engine

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s

D examples/production.ini.sample => examples/production.ini.sample +0 -73
@@ 1,73 0,0 @@
[app:main]
use = egg:fanboi2

pyramid.reload_templates = false
pyramid.debug_authorization = false
pyramid.debug_notfound = false
pyramid.debug_routematch = false
pyramid.default_locale_name = en
pyramid.includes = pyramid_tm

mako.directories = fanboi2:templates
sqlalchemy.url =
redis.url =
celery.broker =

dogpile.backend = dogpile.cache.memcached
dogpile.arguments.url =
dogpile.arguments.distributed_lock = true

session.type = ext:memcached
session.key = _session
session.url =
session.httponly = true
session.secret =

app.timezone =
app.secret =
app.akismet_key =
app.dnsbl_providers =
app.proxy_detect.providers =
app.proxy_detect.blackbox.url =
app.proxy_detect.getipintel.url =
app.proxy_detect.getipintel.email =
app.proxy_detect.getipintel.flags =
app.geoip2_database =
app.checklist = */*

[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 6543

[loggers]
keys = root, fanboi2, sqlalchemy

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = WARN
handlers = console

[logger_fanboi2]
level = WARN
handlers =
qualname = fanboi2

[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine.base.Engine

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s

M fanboi2/__init__.py => fanboi2/__init__.py +101 -147
@@ 1,44 1,47 @@
import copy
import binascii
import hashlib
import logging
import os
from functools import lru_cache
from ipaddress import ip_address

from pyramid.config import Configurator
from pyramid.csrf import SessionCSRFStoragePolicy
from pyramid.path import AssetResolver
from pyramid.settings import aslist
from sqlalchemy.engine import engine_from_config
from fanboi2.cache import cache_region
from fanboi2.models import DBSession, Base, redis_conn, identity
from fanboi2.tasks import celery, configure_celery
from fanboi2.utils import akismet, dnsbl, geoip, proxy_detector, checklist
from pyramid.settings import asbool
from pyramid_nacl_session import EncryptedCookieSessionFactory


def remote_addr(request):
    """Similar to Pyramid's :attr:`request.remote_addr` but will fallback
    to ``HTTP_X_FORWARDED_FOR`` when ``REMOTE_ADDR`` is either a private
    address or a loopback address. If multiple forwarded IPs are given
    in ``HTTP_X_FORWARDED_FOR``, only the first one will be returned.
class NoValue(object):  # pragma: no cover
    """Base class for representing ``NO_VALUE`` to differentiate from
    absent of value and :type:`None`, e.g. in settings, where :type:`None`
    might be an acceptable value, but absent of value may not.
    """

    :param request: A :class:`pyramid.request.Request` object.
    def __repr__(self):
        return 'NO_VALUE'

    :type request: pyramid.request.Request
    :rtype: str
    """
    ipaddr = ip_address(request.environ.get('REMOTE_ADDR', '255.255.255.255'))
    if ipaddr.is_private:
        return request.environ.get('HTTP_X_FORWARDED_FOR', str(ipaddr)).\
            split(",")[0].\
            strip()
    return str(ipaddr)

NO_VALUE = NoValue()


ENV_SETTINGS_MAP = (
    ('CELERY_BROKER_URL', 'celery.broker', NO_VALUE, None),
    ('DATABASE_URL', 'sqlalchemy.url', NO_VALUE, None),
    ('MEMCACHED_URL', 'dogpile.arguments.url', NO_VALUE, None),
    ('GEOIP_PATH', 'geoip.path', None, None),
    ('REDIS_URL', 'redis.url', NO_VALUE, None),
    ('SERVER_DEV', 'server.development', False, asbool),
    ('SESSION_SECRET', 'session.secret', NO_VALUE, None),
)

LOGGING_FMT = '%(asctime)s %(levelname)6s %(name)s[%(process)d] %(message)s'
LOGGING_DATEFMT = '%H:%M:%S'


def route_name(request):
    """Returns :attr:`name` of current :attr:`request.matched_route`.

    :param request: A :class:`pyramid.request.Request` object.

    :type request: pyramid.request.Request
    :rtype: str
    """
    if request.matched_route:
        return request.matched_route.name


@@ 49,9 52,6 @@ def _get_asset_hash(path):
    """Returns an MD5 hash of the given assets path.

    :param path: An asset specification to the asset file.

    :type param: str
    :rtype: str
    """
    if ':' in path:
        package, path = path.split(':')


@@ 74,137 74,91 @@ def tagged_static_path(request, path, **kwargs):
    :param request: A :class:`pyramid.request.Request` object.
    :param path: An asset specification to the asset file.
    :param kwargs: Arguments to pass to :meth:`request.static_path`.

    :type request: pyramid.request.Request
    :type path: str
    :type kwargs: dict
    :rtype: str
    """
    kwargs['_query'] = {'h': _get_asset_hash(path)[:8]}
    return request.static_path(path, **kwargs)


def normalize_settings(settings, _environ=os.environ):
    """Normalize settings to the correct format and merge it with environment
    equivalent if relevant key exists.

    :param settings: A settings :type:`dict`.
def settings_from_env(settings_map=ENV_SETTINGS_MAP, environ=os.environ):
    """Reads environment variable into Pyramid-style settings."""
    settings = {}
    for env, rkey, default, fn in settings_map:
        value = environ.get(env, default)
        if value is NO_VALUE:
            raise RuntimeError('{} is not set'.format(env))
        if fn is not None:
            value = fn(value)
        settings[rkey] = value
    return settings


def tm_maybe_activate(request):
    """Returns whether should the transaction manager be activated."""
    return not request.path_info.startswith('/static/')


def setup_logger(settings):  # pragma: no cover
    """Setup logger per configured in settings."""
    if settings['server.development']:
        import coloredlogs
        coloredlogs.install(
            level=logging.DEBUG,
            fmt=LOGGING_FMT,
            datefmt=LOGGING_DATEFMT)
    else:
        logging.basicConfig(
            level=logging.WARN,
            format=LOGGING_FMT,
            datefmt=LOGGING_DATEFMT)


def make_config(settings):  # pragma: no cover
    """Returns a Pyramid configurator."""
    config = Configurator(settings=settings)
    config.add_settings({
        'mako.directories': 'fanboi2:templates',
        'dogpile.backend': 'dogpile.cache.memcached',
        'dogpile.arguments.distributed_lock': True,
        'tm.activate_hook': tm_maybe_activate})

    if config.registry.settings['server.development']:
        config.add_settings({
            'pyramid.reload_templates': True,
            'pyramid.debug_authorization': True,
            'pyramid.debug_notfound': True,
            'pyramid.default_locale_name': 'en',
            'debugtoolbar.hosts': '0.0.0.0/0'})
        config.include('pyramid_debugtoolbar')

    :type settings: dict
    :rtype: dict
    """
    def _cget(env_key, settings_key):
        settings_val = settings.get(settings_key, '')
        return _environ.get(env_key, settings_val)

    sqlalchemy_url = _cget('SQLALCHEMY_URL', 'sqlalchemy.url')
    redis_url = _cget('REDIS_URL', 'redis.url')
    celery_broker_url = _cget('CELERY_BROKER_URL', 'celery.broker')
    dogpile_url = _cget('DOGPILE_URL', 'dogpile.arguments.url')
    session_url = _cget('SESSION_URL', 'session.url')
    session_secret = _cget('SESSION_SECRET', 'session.secret')

    app_timezone = _cget('APP_TIMEZONE', 'app.timezone')
    app_secret = _cget('APP_SECRET', 'app.secret')
    app_akismet_key = _cget('APP_AKISMET_KEY', 'app.akismet_key')
    app_dnsbl_providers = _cget('APP_DNSBL_PROVIDERS', 'app.dnsbl_providers')

    app_proxy_detect_providers = _cget(
        'APP_PROXY_DETECT_PROVIDERS',
        'app.proxy_detect.providers')

    app_proxy_detect_blackbox_url = _cget(
        'APP_PROXY_DETECT_BLACKBOX_URL',
        'app.proxy_detect.blackbox.url')

    app_proxy_detect_getipintel_url = _cget(
        'APP_PROXY_DETECT_GETIPINTEL_URL',
        'app.proxy_detect.getipintel.url')

    app_proxy_detect_getipintel_email = _cget(
        'APP_PROXY_DETECT_GETIPINTEL_EMAIL',
        'app.proxy_detect.getipintel.email')

    app_proxy_detect_getipintel_flags = _cget(
        'APP_PROXY_DETECT_GETIPINTEL_FLAGS',
        'app.proxy_detect.getipintel.flags')

    app_geoip2_database = _cget('APP_GEOIP2_DATABASE', 'app.geoip2_database')
    app_checklist = _cget('APP_CHECKLIST', 'app.checklist')

    if app_dnsbl_providers is not None:
        app_dnsbl_providers = aslist(app_dnsbl_providers)

    if app_proxy_detect_providers is not None:
        app_proxy_detect_providers = aslist(app_proxy_detect_providers)

    if app_checklist is not None:
        app_checklist = aslist(app_checklist)

    _settings = copy.deepcopy(settings)
    _settings.update({
        'sqlalchemy.url': sqlalchemy_url,
        'redis.url': redis_url,
        'celery.broker': celery_broker_url,
        'dogpile.arguments.url': dogpile_url,
        'session.url': session_url,
        'session.secret': session_secret,
        'app.timezone': app_timezone,
        'app.secret': app_secret,
        'app.akismet_key': app_akismet_key,
        'app.dnsbl_providers': app_dnsbl_providers,
        'app.proxy_detect.providers': app_proxy_detect_providers,
        'app.proxy_detect.blackbox.url': app_proxy_detect_blackbox_url,
        'app.proxy_detect.getipintel.url': app_proxy_detect_getipintel_url,
        'app.proxy_detect.getipintel.email': app_proxy_detect_getipintel_email,
        'app.proxy_detect.getipintel.flags': app_proxy_detect_getipintel_flags,
        'app.geoip2_database': app_geoip2_database,
        'app.checklist': app_checklist,
    })

    return _settings


def main(global_config, **settings):  # pragma: no cover
    """This function returns a Pyramid WSGI application.

    :param global_config: A :type:`dict` containing global config.
    :param settings: A :type:`dict` containing values from INI.

    :type global_config: dict
    :type settings: dict
    :rtype: pyramid.router.Router
    """
    config = Configurator(settings=normalize_settings(settings))
    config.include('pyramid_mako')
    config.include('pyramid_beaker')

    engine = engine_from_config(config.registry.settings, 'sqlalchemy.')
    DBSession.configure(bind=engine)
    Base.metadata.bind = engine

    cache_region.configure_from_config(config.registry.settings, 'dogpile.')
    redis_conn.from_url(config.registry.settings['redis.url'])
    celery.config_from_object(configure_celery(config.registry.settings))
    identity.configure_tz(config.registry.settings['app.timezone'])
    akismet.configure_key(config.registry.settings['app.akismet_key'])
    dnsbl.configure_providers(config.registry.settings['app.dnsbl_providers'])
    geoip.configure_geoip2(config.registry.settings['app.geoip2_database'])
    checklist.configure_checklist(config.registry.settings['app.checklist'])
    proxy_detector.configure_from_config(
        config.registry.settings,
        'app.proxy_detect.')

    config.set_request_property(remote_addr)
    config.include('pyramid_services')

    session_secret_hex = config.registry.settings['session.secret'].strip()
    session_secret = binascii.unhexlify(session_secret_hex)
    session_factory = EncryptedCookieSessionFactory(
        session_secret,
        cookie_name='_session',
        timeout=3600,
        httponly=True)

    config.set_session_factory(session_factory)
    config.set_csrf_storage_policy(SessionCSRFStoragePolicy(key='_csrf'))
    config.set_request_property(route_name)
    config.add_request_method(tagged_static_path)
    config.add_route('robots', '/robots.txt')

    config.include('fanboi2.cache')
    config.include('fanboi2.filters')
    config.include('fanboi2.geoip')
    config.include('fanboi2.models')
    config.include('fanboi2.redis')
    config.include('fanboi2.serializers')
    config.include('fanboi2.views.pages', route_prefix='/pages')
    config.include('fanboi2.services')
    config.include('fanboi2.tasks')

    config.include('fanboi2.views.api', route_prefix='/api')
    config.include('fanboi2.views.pages', route_prefix='/pages')
    config.include('fanboi2.views.boards', route_prefix='/')
    config.add_static_view('static', 'static', cache_max_age=3600)
    config.scan()

    return config.make_wsgi_app()
    return config

M fanboi2/cache.py => fanboi2/cache.py +13 -11
@@ 1,20 1,22 @@
import hashlib
import logging
from dogpile.cache import make_region


log = logging.getLogger(__name__)
from dogpile.cache import make_region


def _key_mangler(key):
def key_mangler(key):
    """Retrieve cache keys as a long concatenated strings and turn them into
    an MD5 hash.
    an SHA256 hash.

    :param key: A cache key :type:`str`.

    :type key: str
    :rtype: str
    """
    return hashlib.md5(bytes(key.encode('utf8'))).hexdigest()
    return hashlib.sha256(bytes(key.encode('utf8'))).hexdigest()


def includeme(config):  # pragma: no cover
    cache_region = make_region(key_mangler=key_mangler)
    cache_region.configure_from_config(config.registry.settings, 'dogpile.')

    def cache_region_factory(context, request):
        return cache_region

cache_region = make_region(key_mangler=_key_mangler)
    config.register_service_factory(cache_region_factory, name='cache')

A fanboi2/cmd/__init__.py => fanboi2/cmd/__init__.py +0 -0
A fanboi2/cmd/celery.py => fanboi2/cmd/celery.py +15 -0
@@ 0,0 1,15 @@
import sys

from celery.bin.celery import main as celery_main

from .. import settings_from_env, setup_logger, make_config


def main(argv=sys.argv):
    """Run Celery with application environment."""
    settings = settings_from_env()
    setup_logger(settings)
    config = make_config(settings)
    config.make_wsgi_app()

    celery_main(argv)

A fanboi2/cmd/ctl.py => fanboi2/cmd/ctl.py +76 -0
@@ 0,0 1,76 @@
import argparse
import code
import sys

from ..version import __VERSION__, __PYRAMID__


SHELL_BANNER = """\
      ___  ___
    / /     / /
   /_/_   _/_/    fanboi2 %s
  / /   / /       pyramid %s
 /_/   /_/__

Loaded locals:

    app           Fanboi2 WSGI application
    config        Pyramid configurator
    registry      Pyramid registry
    request       Pyramid request context
    root          Pyramid root
    root_factory  Pyramid root factory
"""


def shell(args):
    """Run the interactive shell for the application."""
    from ..wsgi import app, config
    import pyramid.scripting
    with pyramid.scripting.prepare() as env:
        code.interact(
            SHELL_BANNER % (__VERSION__, __PYRAMID__),
            local={
                "app": app,
                "config": config,
                **env
            })


def serve(args):
    """Run the web server for the application."""
    from ..wsgi import app
    from waitress import serve as waitress_serve
    waitress_serve(app, host=args.host, port=args.port)
    sys.exit(0)


def gensecret(args):
    """Generates a NaCl secret."""
    from pyramid_nacl_session import generate_secret
    print(generate_secret(as_hex=True).decode('utf-8'))
    sys.exit(0)


def main():
    """Parse the command line arguments."""
    parser = argparse.ArgumentParser()
    subparsers = parser.add_subparsers(dest='subparser', title='subcommands')

    pshell = subparsers.add_parser('shell')
    pshell.set_defaults(func=shell)

    pserve = subparsers.add_parser('serve')
    pserve.add_argument('--port', type=int, default=6543)
    pserve.add_argument('--host', default='0.0.0.0')
    pserve.set_defaults(func=serve)

    gsecret = subparsers.add_parser('gensecret')
    gsecret.set_defaults(func=gensecret)

    args = parser.parse_args()
    if args.subparser is None:
        parser.print_usage()
        sys.exit(1)

    args.func(args)

M fanboi2/errors.py => fanboi2/errors.py +26 -34
@@ 1,19 1,3 @@
def serialize_error(type, *args):
    """Serialize the given error type string into an error class.
    If :param:`attrs` is provided, it will be passed to the class on
    initialization.
    """
    return {
        'rate_limited': RateLimitedError,
        'params_invalid': ParamsInvalidError,
        'spam_rejected': SpamRejectedError,
        'dnsbl_rejected': DnsblRejectedError,
        'ban_rejected': BanRejectedError,
        'status_rejected': StatusRejectedError,
        'proxy_rejected': ProxyRejectedError,
    }.get(type, BaseError)(*args)


class BaseError(Exception):
    """A base :class:`Exception` class that provides hint for reporting
    errors as JSON response.


@@ 26,23 10,17 @@ class BaseError(Exception):
        serializable types.

        :param request: A :class:`pyramid.request.Request` object.
        :type request: pyramid.request.Request
        :rtype: object
        """
        return 'An exception error occurred.'

    @property
    def name(self):
        """The short globally recognizable name of this error.
        :rtype: str
        """
        """The short globally recognizable name of this error."""
        return 'unknown'

    @property
    def http_status(self):
        """The HTTP status code to response as.
        :rtype: str
        """
        """The HTTP status code to response as."""
        return '500 Internal Server Error'




@@ 52,7 30,6 @@ class RateLimitedError(BaseError):
    be accessed from :property:`timeleft`.

    :param timeleft: An :type:`int` in seconds until user is unblocked.
    :type timeleft: int
    """

    def __init__(self, timeleft):


@@ 76,7 53,6 @@ class ParamsInvalidError(BaseError):
    inside :property:`messages`.

    :param messages: An :type:`dict` of :type:`list` of field errors.
    :type messages: str
    """

    def __init__(self, messages):


@@ 94,31 70,32 @@ class ParamsInvalidError(BaseError):
        return '400 Bad Request'


class SpamRejectedError(BaseError):
class AkismetRejectedError(BaseError):
    """An :class:`Exception` class that will be raised if user request was
    blocked due user request failed a spam check.
    """

    def message(self, request):
        return 'The request has been identified as spam and therefore rejected.'
        return 'The request has been identified as spam ' +\
            'by Akismet and therefore rejected.'

    @property
    def name(self):
        return 'spam_rejected'
        return 'akismet_rejected'

    @property
    def http_status(self):
        return '422 Unprocessable Entity'


class DnsblRejectedError(BaseError):
class DNSBLRejectedError(BaseError):
    """An :class:`Exception` class that will be raised if user request was
    blocked due user IP address failed an DNSBL check.
    """

    def message(self, request):
        return 'The IP address is being listed in one of DNSBL databases ' +\
               'and therefore rejected.'
            'and therefore rejected.'

    @property
    def name(self):


@@ 136,7 113,7 @@ class BanRejectedError(BaseError):

    def message(self, request):
        return 'The IP address is being listed in the ban list ' +\
               'and therefore rejected.'
            'and therefore rejected.'

    @property
    def name(self):


@@ 153,7 130,6 @@ class StatusRejectedError(BaseError):
    the lock could be retrieved from :property:`status`.

    :param status: A :type:`str` of the status that caused the block.
    :type status: str
    """

    def __init__(self, status):


@@ 178,7 154,7 @@ class ProxyRejectedError(BaseError):

    def message(self, request):
        return 'The IP address has been identified as an open proxy ' +\
               'or VPN service and therefore rejected.'
            'or VPN service and therefore rejected.'

    @property
    def name(self):


@@ 187,3 163,19 @@ class ProxyRejectedError(BaseError):
    @property
    def http_status(self):
        return '422 Unprocessable Entity'


_ERRORS = {
    'rate_limited': RateLimitedError,
    'params_invalid': ParamsInvalidError,
    'akismet_rejected': AkismetRejectedError,
    'dnsbl_rejected': DNSBLRejectedError,
    'ban_rejected': BanRejectedError,
    'status_rejected': StatusRejectedError,
    'proxy_rejected': ProxyRejectedError,
}


def deserialize_error(type_):
    """Deserialize the given error type string into an error class."""
    return _ERRORS.get(type_, BaseError)

A fanboi2/filters/__init__.py => fanboi2/filters/__init__.py +16 -0
@@ 0,0 1,16 @@
import venusian


def register_filter(name):  # pragma: no cover
    def _wrapped(cls):
        def callback(scanner, obj_name, obj):
            registry = scanner.config.registry
            registry['filters'].append((name, obj))
        venusian.attach(cls, callback)
        return cls
    return _wrapped


def includeme(config):  # pragma: no cover
    config.registry.setdefault('filters', [])
    config.scan()

A fanboi2/filters/akismet.py => fanboi2/filters/akismet.py +45 -0
@@ 0,0 1,45 @@
import requests

from ..version import __VERSION__
from . import register_filter


@register_filter(name='akismet')
class Akismet(object):
    """Basic integration between Pyramid and Akismet."""

    __default_settings__ = None

    def __init__(self, key, services={}):
        self.key = key

    def _api_post(self, name, data=None):
        """Make a request to Akismet API and return the response.

        :param name: A :type:`str` of API method name to request.
        :param data: A :type:`dict` payload.
        """
        return requests.post(
            'https://%s.rest.akismet.com/1.1/%s' % (self.key, name),
            headers={'User-Agent': "fanboi2/%s" % __VERSION__},
            data=data,
            timeout=2)

    def should_reject(self, payload):
        """Returns :type:`True` if the message is spam. Returns :type:`False`
        if the message was not a spam or Akismet was not configured.

        :param payload: A filter payload.
        """
        if self.key:
            try:
                return self._api_post('comment-check', data={
                    'comment_content': payload['body'],
                    'blog': payload['application_url'],
                    'user_ip': payload['ip_address'],
                    'user_agent': payload['user_agent'],
                    'referrer': payload['referrer'],
                }).content == b'true'
            except (KeyError, requests.Timeout):
                pass
        return False

R fanboi2/utils/dnsbl.py => fanboi2/filters/dnsbl.py +24 -18
@@ 1,31 1,37 @@
import socket
from ipaddress import ip_interface, ip_network

from . import register_filter

class Dnsbl(object):

@register_filter(name='dnsbl')
class DNSBL(object):
    """Utility class for checking IP address against DNSBL providers."""

    def __init__(self):
        self.providers = []
    __default_settings__ = (
        'proxies.dnsbl.sorbs.net',
        'xbl.spamhaus.org',
    )

    def configure_providers(self, providers):
        if isinstance(providers, str):
            providers = providers.split()
        self.providers = providers
    def __init__(self, providers, services={}):
        if not providers:
            providers = tuple()
        self.providers = tuple(providers)

    def listed(self, ip_address):
    def should_reject(self, payload):
        """Returns :type:`True` if the given IP address is listed in the
        DNSBL providers. Returns :type:`False` if not listed or no DNSBL
        providers present.

        :param payload: A filter payload.
        """
        if self.providers:
            for provider in self.providers:
                try:
                    check = '.'.join(reversed(ip_address.split('.')))
                    res = socket.gethostbyname("%s.%s." % (check, provider))
                    ipaddr = ip_interface("%s/255.0.0.0" % (res,))
                    if ipaddr.network == ip_network('127.0.0.0/8'):
                        return True
                except (socket.gaierror, ValueError):
                    continue
        for provider in self.providers:
            try:
                check = '.'.join(reversed(payload['ip_address'].split('.')))
                res = socket.gethostbyname("%s.%s." % (check, provider))
                ipaddr = ip_interface("%s/255.0.0.0" % (res,))
                if ipaddr.network == ip_network('127.0.0.0/8'):
                    return True
            except (socket.gaierror, ValueError):
                continue
        return False

R fanboi2/utils/proxy.py => fanboi2/filters/proxy.py +53 -66
@@ 1,15 1,16 @@
import requests

from ..version import __VERSION__
from ..cache import cache_region as cache_region_
from . import register_filter


class BlackBoxProxyDetector(object):
    """Provides integration with Black Block Proxy Block service."""

    def __init__(self, config):
        self.url = config.get('url')
    def __init__(self, **kwargs):
        self.url = kwargs.get('url')
        if not self.url:
            self.url = 'http://www.shroomery.org/ythan/proxycheck.php'
            self.url = 'http://proxy.mind-media.com/block/proxycheck.php'

    def check(self, ip_address):
        """Request for IP evaluation and return raw results. Return the


@@ 17,14 18,11 @@ class BlackBoxProxyDetector(object):
        an error code returned from Black Box Proxy Block.

        :param ip_address: An :type:`str` IP address.

        :type ip_address: str
        :rtype: str or None
        """
        try:
            result = requests.get(
                self.url,
                headers={'User-Agent': "Fanboi2/%s" % __VERSION__},
                headers={'User-Agent': "fanboi2/%s" % __VERSION__},
                params={'ip': ip_address},
                timeout=2)
        except requests.Timeout:


@@ 37,9 35,6 @@ class BlackBoxProxyDetector(object):
        :type:`True` if evaluation result is 'Y', i.e. a proxy.

        :param result: A result from evaluation request.

        :type result: str
        :rtype: bool
        """
        if result == b'Y':
            return True


@@ 49,10 44,10 @@ class BlackBoxProxyDetector(object):
class GetIPIntelProxyDetector(object):
    """Provides integration with GetIPIntel proxy detection service."""

    def __init__(self, config):
        self.url = config.get('url')
        self.flags = config.get('flags')
        self.email = config.get('email')
    def __init__(self, **kwargs):
        self.url = kwargs.get('url')
        self.flags = kwargs.get('flags')
        self.email = kwargs.get('email')
        if not self.url:
            self.url = 'http://check.getipintel.net/check.php'
        if not self.email:


@@ 64,9 59,6 @@ class GetIPIntelProxyDetector(object):
        positive.

        :param ip_address: An :type:`str` IP address.

        :type ip_address: str
        :rtype: str or None
        """
        params = {'contact': self.email, 'ip': ip_address}
        if self.flags:


@@ 74,7 66,7 @@ class GetIPIntelProxyDetector(object):
        try:
            result = requests.get(
                self.url,
                headers={'User-Agent': "Fanboi2/%s" % __VERSION__},
                headers={'User-Agent': "fanboi2/%s" % __VERSION__},
                params=params,
                timeout=5)
        except requests.Timeout:


@@ 88,9 80,6 @@ class GetIPIntelProxyDetector(object):
        probability higher than ``0.99`` (for example, ``0.994120``).

        :param result: A result from evaluation request.

        :type result: str
        :rtype: bool
        """
        if float(result) > 0.99:
            return True


@@ 103,52 92,50 @@ DETECTOR_PROVIDERS = {
}


@register_filter(name='proxy')
class ProxyDetector(object):
    """Base class for dispatching proxy detection into multiple providers."""

    def __init__(self, cache_region=cache_region_):
        self.providers = []
        self.instances = {}
        self.cache_region = cache_region

    def configure_from_config(self, config, key=None):
        """Configure and initialize proxy detectors. The configuration dict
        may contains provider-specific configuration using the same dotted-
        name as the provider itself, for example ``getipintel.url``.

        If the configuration key is prefixed with other dotted names, ``key``
        may be given to extract from that prefix.

        :param config: Configuration :type:`dict`.
        :param key: Key prefix to extract configuration.
        """
        if key is None:
            key = ''
        self.providers = config.get('%sproviders' % (key,), [])
        for provider in self.providers:
            class_ = DETECTOR_PROVIDERS[provider]
            provider_key = "%s%s." % (key, provider)
            provider_config = {}
            for k, v in config.items():
                if k.startswith(provider_key):
                    provider_config[k[len(provider_key):]] = v
            self.instances[provider] = class_(provider_config)

    def detect(self, ip_address):
        """Detect if the given ``ip_address`` is a proxy using providers
        configured via :meth:``configure_from_config``.

        :param ip_address: An IP address to perform a proxy check against.
        :type ip_address: str
        :rtype: bool
    __use_services__ = ('cache',)
    __default_settings__ = {
        'blackbox': {
            'enabled': False,
            'url': 'http://proxy.mind-media.com/block/proxycheck.php',
        },
        'getipintel': {
            'enabled': False,
            'url':  'http://check.getipintel.net/check.php',
            'email': None,
            'flags': None,
        },
    }

    def __init__(self, settings=None, services={}):
        if not settings:
            settings = {}
        self.settings = settings
        self.cache_region = services['cache']

    def _get_cache_key(self, provider, ip_address):
        return 'filters.proxy:provider=%s,ip_address=%s' % (
            provider,
            ip_address)

    def should_reject(self, payload):
        """Returns :type:`True` if the given IP address is identified as
        proxy by one of the providers. Returns :type:`False` if the IP
        address was not identified as a proxy or no providers configured.

        :param payload: A filter payload.
        """
        for provider in self.providers:
            detector = self.instances[provider]
            result = self.cache_region.get_or_create(
                'proxy:%s:%s' % (provider, ip_address),
                lambda: detector.check(ip_address),
                should_cache_fn=lambda v: v is not None,
                expiration_time=21600)
            if result is not None and detector.evaluate(result):
                return True
        for provider, settings in self.settings.items():
            if settings['enabled']:
                detector = DETECTOR_PROVIDERS[provider](**settings)
                result = self.cache_region.get_or_create(
                    self._get_cache_key(provider, payload['ip_address']),
                    lambda: detector.check(payload['ip_address']),
                    should_cache_fn=lambda v: v is not None,
                    expiration_time=21600)
                if result is not None and detector.evaluate(result):
                    return True
        return False

M fanboi2/forms.py => fanboi2/forms.py +0 -49
@@ 1,8 1,4 @@
import hmac
import os
from hashlib import sha1
from wtforms import TextField, TextAreaField, Form, BooleanField
from wtforms.ext.csrf.fields import CSRFTokenField
from wtforms.validators import Length as _Length
from wtforms.validators import Required, ValidationError



@@ 34,43 30,6 @@ class Length(_Length):
            raise ValidationError(message % dict(min=self.min, max=self.max))


class SecureForm(Form):
    """Generate CSRF token based based on randomly generated string token."""
    csrf_token = CSRFTokenField()

    def __init__(self, formdata=None, obj=None, prefix='', request=None):
        super(SecureForm, self).__init__(formdata, obj, prefix)
        self.request = request
        self.csrf_token.current_token = self.generate_csrf_token()

    def _generate_hmac(self, message):
        secret = self.request.registry.settings['app.secret']
        return hmac.new(
            bytes(secret.encode('utf8')),
            bytes(message.encode('utf8')),
            digestmod=sha1,
        ).hexdigest()

    def generate_csrf_token(self):
        if 'csrf' not in self.request.session:
            self.request.session['csrf'] = sha1(os.urandom(64)).hexdigest()
        self.csrf_token.csrf_key = self.request.session['csrf']
        return self._generate_hmac(self.request.session['csrf'])

    def validate_csrf_token(self, field):
        if not field.data:
            raise ValidationError('CSRF token missing.')
        hmac_compare = self._generate_hmac(field.csrf_key)
        if not hmac.compare_digest(field.data, hmac_compare):
            raise ValidationError('CSRF token mismatched.')

    @property
    def data(self):
        d = super(SecureForm, self).data
        d.pop('csrf_token')
        return d


class TopicForm(Form):
    """A :class:`Form` for creating new topic. This form should be populated
    to two objects, :attr:`title` to :class:`Topic` and :attr:`body` to


@@ 80,17 39,9 @@ class TopicForm(Form):
    body = TextAreaField('Body', validators=[Required(), Length(5, 4000)])


class SecureTopicForm(SecureForm, TopicForm, Form):
    pass


class PostForm(Form):
    """A :class:`Form` for replying to a topic. The :attr:`body` field should
    be populated to :class:`Post`.
    """
    body = TextAreaField('Body', validators=[Required(), Length(5, 4000)])
    bumped = BooleanField('Bump this topic', default=True)


class SecurePostForm(SecureForm, PostForm, Form):
    pass

A fanboi2/geoip.py => fanboi2/geoip.py +14 -0
@@ 0,0 1,14 @@
import geoip2.database as geoip_db


def includeme(config):  # pragma: no cache
    geoip_path = config.registry.settings['geoip.path']
    if not geoip_path:
        return

    geoip = geoip_db.Reader(geoip_path)

    def geoip_factory(context, request):
        return geoip

    config.register_service_factory(geoip, name='geoip')

M fanboi2/helpers/formatters.py => fanboi2/helpers/formatters.py +12 -55
@@ 1,14 1,17 @@
import html
import isodate
import misaka
import pytz
import re
import urllib
import urllib.parse as urlparse
from collections import OrderedDict
from html.parser import HTMLParser

import isodate
import misaka
import pytz
from markupsafe import Markup

from ..interfaces import ISettingQueryService


RE_PARAGRAPH = re.compile(r'(?:(?P<newline>\r\n|\n|\r)(?P=newline)+)')
RE_THUMBNAILS = (


@@ 64,13 67,10 @@ def extract_thumbnail(text):
    links creation.

    :param text: A :type:`str` to extract URL from.

    :type text: str
    :rtype: list
    """
    thumbnails = OrderedDict()
    for re, thumb, url in RE_THUMBNAILS:
        for item in re.findall(text):
    for r, thumb, url in RE_THUMBNAILS:
        for item in r.findall(text):
            try:
                if not isinstance(item, tuple):
                    item = [item]


@@ 96,9 96,6 @@ def url_fix(string):
    Ported from ``werkzeug.urls.url_fix``.

    :param string: A :type:`str` containing URL to fix.

    :type string: str
    :rtype: str
    """
    scheme, netloc, path, qs, anchor = urlparse.urlsplit(string)
    path = urlparse.quote(path, '/%')


@@ 116,10 113,6 @@ def format_text(text, shorten=None):
    :param shorten: An :type:`int` that specifies approximate length of text
                    to be displayed. The full post will be shown if
                    :type:`None` is given.

    :type text: str
    :type shorten: int or None
    :rtype: PostMarkup
    """
    output = []
    thumbs = []


@@ 170,11 163,6 @@ def format_markdown(context, request, text):
    :param context: A :class:`mako.runtime.Context` object.
    :param request: A :class:`pyramid.request.Request` object.
    :param text: A :type:`str` containing unformatted Markdown text.

    :type context: mako.runtime.Context or None
    :type request: pyramid.request.Request
    :type text: str or None
    :rtype: Markup
    """
    if text is not None:
        return Markup(misaka.html(str(text)))


@@ 220,14 208,7 @@ def format_post(context, request, post, shorten=None):
    :param context: A :class:`mako.runtime.Context` object.
    :param request: A :class:`pyramid.request.Request` object.
    :param post: A :class:`fanboi2.models.Post` object.
    :param shorten: An :type:`int` or :type:`None` that gets passed to
                    :func:`format_text`.

    :type context: mako.runtime.Context or None
    :type request: pyramid.request.Request
    :type post: fanboi2.models.Post
    :type shorten: int or None
    :rtype: Markup
    :param shorten: An :type:`int` or :type:`None`.
    """
    text = format_text(post.body, shorten)



@@ 289,11 270,6 @@ def format_page(context, request, page):
    :param context: A :class:`mako.runtime.Context` object.
    :param request: A :class:`pyramid.request.Request` object.
    :param page: A :class:`fanboi2.models.Page` object.

    :type context: mako.runtime.Context or None
    :type request: pyramid.request.Request
    :type page: fanboi2.models.Page
    :rtype: Markup
    """
    if page.formatter == 'markdown':
        return format_markdown(context, request, page.body)


@@ 308,16 284,10 @@ def format_datetime(context, request, dt):
    :param context: A :class:`mako.runtime.Context` object.
    :param request: A :class:`pyramid.request.Request` object.
    :param dt: A :class:`datetime.datetime` object.

    :type context: mako.runtime.Context or None
    :type request: pyramid.request.Request
    :type dt: datetime.datetime
    :rtype: str
    """
    settings = request.registry.settings
    assert isinstance(settings, dict)
    timezone = pytz.timezone(settings['app.timezone'])
    return dt.astimezone(timezone).strftime('%b %d, %Y at %H:%M:%S')
    setting_query_svc = request.find_service(ISettingQueryService)
    tz = pytz.timezone(setting_query_svc.value_from_key('app.time_zone'))
    return dt.astimezone(tz).strftime('%b %d, %Y at %H:%M:%S')


def format_isotime(context, request, dt):


@@ 326,11 296,6 @@ def format_isotime(context, request, dt):
    :param context: A :class:`mako.runtime.Context` object.
    :param request: A :class:`pyramid.request.Request` object.
    :param dt: A :class:`datetime.datetime` object.

    :type context: mako.runtime.Context or None
    :type request: pyramid.request.Request
    :type dt: datetime.datetime
    :rtype: str
    """
    return isodate.datetime_isoformat(dt.astimezone(pytz.utc))



@@ 340,10 305,6 @@ def unquoted_path(context, request, *args, **kwargs):

    :param context: A :class:`mako.runtime.Context` object.
    :param request: A :class:`pyramid.request.Request` object.

    :type context: mako.runtime.Context or None
    :type request: pyramid.request.Request
    :rtype: str
    """
    return urllib.parse.unquote(request.route_path(*args, **kwargs))



@@ 362,10 323,6 @@ def user_theme(context, request, cookie='_theme'):

    :param context: A :class:`mako.runtime.Context` object.
    :param request: A :class:`pyramid.request.Request` object.

    :type context: mako.runtime.Context or None
    :type request: pyramid.request.Request
    :rtype: str
    """
    user_theme = request.cookies.get(cookie)
    if user_theme is None or user_theme not in THEMES:

M fanboi2/helpers/partials.py => fanboi2/helpers/partials.py +41 -34
@@ 1,78 1,85 @@
from markupsafe import Markup
from fanboi2.helpers.formatters import format_markdown
from fanboi2.models import DBSession, Page
from fanboi2.cache import cache_region as cache_region_
from sqlalchemy.orm.exc import NoResultFound

from ..interfaces import IPageQueryService
from .formatters import format_markdown

def _get_internal_page(slug, cache_region=cache_region_):

def _get_cache_key(slug):
    """Returns a cache key for given partial.

    :param slug: An internal page slug.
    """
    return 'helpers.partials:slug=%s' % (slug,)


def get_partial(request, slug):
    """Returns a content of internal page.

    :param request: A :class:`pyramid.request.Request` object.
    :param slug: An internal page slug.
    :type slug: String
    :rtype: String or None
    """
    page_query_svc = request.find_service(IPageQueryService)
    cache_region = request.find_service(name='cache')

    def _creator():
        page = DBSession.query(Page).filter_by(
                namespace='internal',
                slug=slug).\
            first()
        try:
            page = page_query_svc.internal_page_from_slug(slug)
        except NoResultFound:
            return
        if page:
            return page.body

    return cache_region.get_or_create(
        'partial:%s' % (slug,),
        _get_cache_key(slug),
        _creator,
        expiration_time=43200)


def global_css(context, request, cache_region=cache_region_):
def reload_partial(request, slug):
    """Delete the key for the given slug from the cache to force
    reloading of partials the next time it is accessed.

    :param request: A :class:`pyramid.request.Request` object.
    :param slug: An internal page slug.
    """
    cache_region = request.find_service(name='cache')
    cache_key = _get_cache_key(slug)
    return cache_region.delete(cache_key)


def global_css(context, request):
    """Returns a string of inline global custom CSS for site-wide CSS override.
    This custom CSS is the content of ``internal:global/css`` page.

    :param context: A :class:`mako.runtime.Context` object.
    :param request: A :class:`pyramid.request.Request` object.
    :param cache_region: Optional cache region to cache this partial.

    :type context: mako.runtime.Context or None
    :type request: pyramid.request.Request
    :type cache_region: dogpile.cache.region.CacheRegion
    :rtype: Markup or None
    """
    page = _get_internal_page('global/css', cache_region)
    page = get_partial(request, 'global/css')
    if page:
        return Markup(page)


def global_appendix(context, request, cache_region=cache_region_):
def global_appendix(context, request):
    """Returns a HTML of global appendix content. This appendix content is the
    content of ``internal:global/appendix`` page.

    :param context: A :class:`mako.runtime.Context` object.
    :param request: A :class:`pyramid.request.Request` object.
    :param cache_region: Optional cache region to cache this partial.

    :type context: mako.runtime.Context or None
    :type request: pyramid.request.Request
    :type cache_region: dogpile.cache.region.CacheRegion
    :rtype: Markup or None
    """
    page = _get_internal_page('global/appendix', cache_region)
    page = get_partial(request, 'global/appendix')
    if page:
        return format_markdown(context, request, page)


def global_footer(context, request, cache_region=cache_region_):
def global_footer(context, request):
    """Returns a HTML of global footer content. This footer content is the
    content of ``internal:global/footer`` page.

    :param context: A :class:`mako.runtime.Context` object.
    :param request: A :class:`pyramid.request.Request` object.
    :param cache_region: Optional cache region to cache this partial.

    :type context: mako.runtime.Context or None
    :type request: pyramid.request.Request
    :type cache_region: dogpile.cache.region.CacheRegion
    :rtype: Markup or None
    """
    page = _get_internal_page('global/footer', cache_region)
    page = get_partial(request, 'global/footer')
    if page:
        return Markup(page)

A fanboi2/interfaces.py => fanboi2/interfaces.py +94 -0
@@ 0,0 1,94 @@
from zope.interface import Interface


class IBoardQueryService(Interface):
    def list_active():
        pass

    def board_from_slug(board_slug):
        pass


class IFilterService(Interface):
    def evaluate(payload={}):
        pass


class IIdentityService(Interface):
    def identity_for(**kwargs):
        pass


class IPageQueryService(Interface):
    def list_public():
        pass

    def public_page_from_slug(page_slug):
        pass

    def internal_page_from_slug(page_slug):
        pass


class IPostCreateService(Interface):
    def enqueue(topic_id, body, bumped, ip_address, payload={}):
        pass

    def create(topic_id, body, bumped, ip_address, payload={}):
        pass


class IPostQueryService(Interface):
    def list_from_topic_id(topic_id, query=None):
        pass

    def was_recently_seen(ip_address):
        pass


class IRateLimiterService(Interface):
    def limit_for(seconds, **kwargs):
        pass

    def is_limited(**kwargs):
        pass

    def time_left(**kwargs):
        pass


class IRuleBanQueryService(Interface):
    def is_banned(ip_address, scopes):
        pass


class ISettingQueryService(Interface):
    def value_from_key(key):
        pass

    def reload(key):
        pass


class ITaskQueryService(Interface):
    def result_from_uid(task_uid):
        pass


class ITopicCreateService(Interface):
    def enqueue(board_slug, title, body, ip_address, payload={}):
        pass

    def create(board_slug, title, body, ip_address, payload={}):
        pass


class ITopicQueryService(Interface):
    def list_from_board_slug(board_slug):
        pass

    def list_recent_from_board_slug(board_slug):
        pass

    def topic_from_id(topic_id):
        pass

M fanboi2/models/__init__.py => fanboi2/models/__init__.py +73 -53
@@ 1,69 1,89 @@
from sqlalchemy import event
from sqlalchemy.sql import desc, func, select
from ._base import DBSession, Base, JsonType
from ._identity import Identity
from ._redis_proxy import RedisProxy
from ._versioned import make_versioned
import logging

from sqlalchemy.engine import engine_from_config
from sqlalchemy.orm import sessionmaker
import zope.sqlalchemy

from ._base import Base
from ._versioned import make_history_event, setup_versioned
from .board import Board
from .topic import Topic
from .topic_meta import TopicMeta
from .post import Post
from .page import Page
from .post import Post
from .rule import Rule
from .rule_ban import RuleBan
from .rule_override import RuleOverride
from .setting import Setting
from .topic import Topic
from .topic_meta import TopicMeta


__all__ = [
    'Base',
    'Board',
    'Page',
    'Post',
    'Rule',
    'RuleBan',
    'Setting',
    'Topic',
    'TopicMeta',
]


_MODELS = {
    'board': Board,
    'topic': Topic,
    'topic_meta': TopicMeta,
    'page': Page,
    'post': Post,
    'rule': Rule,
    'rule_ban': RuleBan,
    'rule_override': RuleOverride,
    'setting': Setting,
    'topic': Topic,
    'topic_meta': TopicMeta,
}

def serialize_model(type_):

# Versioned need to be setup after all models are initialized otherwise
# SQLAlchemy won't be able to locate relation tables due to import order.
setup_versioned()


def deserialize_model(type_):
    """Deserialize the given model type string into a model class."""
    return _MODELS.get(type_)


redis_conn = RedisProxy()
identity = Identity(redis=redis_conn)
make_versioned(DBSession)


@event.listens_for(DBSession, 'before_flush')
def _create_topic_meta(session, context, instances):
    """Assign a new topic meta to a topic on creation."""
    for topic in filter(lambda m: isinstance(m, Topic), session.new):
        if topic.meta is None:
            topic.meta = TopicMeta(post_count=0)


@event.listens_for(DBSession, 'before_flush')
def _update_topic_meta_states(session, context, instance):
    """Update topic metadata and related states when new posts are made."""
    for post in filter(lambda m: isinstance(m, Post), session.new):
        topic = post.topic
        board = topic.board
        if topic in session.new:
            topic_meta = topic.meta
        else:
            topic_meta = session.query(TopicMeta).\
                         filter_by(topic=topic).\
                         with_for_update().\
                         one()

        topic_meta.post_count = post.number = topic_meta.post_count + 1
        topic_meta.posted_at = post.created_at or func.now()
        if post.bumped is None or post.bumped:
            topic_meta.bumped_at = topic_meta.posted_at

        if topic.status == 'open' and \
           topic_meta.post_count >= board.settings['max_posts']:
            topic.status = 'archived'

        session.add(topic_meta)
        session.add(topic)
        session.add(post)
def init_dbsession(dbsession, tm=None):  # pragma: no cover
    """Initialize SQLAlchemy ``dbsession`` with application defaults.

    :param dbsession: A :class:`sqlalchemy.orm.session.Session` object.
    :param tm: A Zope transaction manager.
    """
    zope.sqlalchemy.register(dbsession, transaction_manager=tm)


def configure_sqlalchemy(settings):  # pragma: no cover
    """Configure SQLAlchemy with the given settings."""
    engine = engine_from_config(settings, 'sqlalchemy.')
    Base.metadata.bind = engine
    dbmaker = sessionmaker()
    dbmaker.configure(bind=engine)
    return dbmaker


def includeme(config):  # pragma: no cover
    config.include('pyramid_tm')
    dbmaker = configure_sqlalchemy(config.registry.settings)
    make_history_event(dbmaker)

    log_level = logging.WARN
    if config.registry.settings['server.development']:
        log_level = logging.INFO

    logger = logging.getLogger('sqlalchemy.engine.base.Engine')
    logger.setLevel(log_level)

    def dbsession_factory(context, request):
        dbsession = dbmaker()
        init_dbsession(dbmaker, tm=request.tm)
        return dbsession

    config.register_service_factory(dbsession_factory, name='db')

M fanboi2/models/_base.py => fanboi2/models/_base.py +8 -35
@@ 1,32 1,7 @@
import json
import re
from sqlalchemy.ext.declarative import declarative_base, declared_attr
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.schema import MetaData
from sqlalchemy.sql.sqltypes import Text
from sqlalchemy.sql.type_api import TypeDecorator
from zope.sqlalchemy import ZopeTransactionExtension
from ._versioned import make_versioned_class


RE_FIRST_CAP = re.compile('(.)([A-Z][a-z]+)')
RE_ALL_CAP = re.compile('([a-z0-9])([A-Z])')


class JsonType(TypeDecorator):
    """Serializable field for storing data as JSON text. If the field is
    ``NULL`` in the database, a default value of empty :type:`dict` is
    returned on retrieval.
    """
    impl = Text

    def process_bind_param(self, value, dialect):
        return json.dumps(value)

    def process_result_value(self, value, dialect):
        if not value:
            return {}
        return json.loads(value)
from ._versioned import make_versioned_class


class BaseModel(object):


@@ 37,15 12,13 @@ class BaseModel(object):
            setattr(self, key, value)


Versioned = make_versioned_class()

metadata = MetaData(naming_convention={
  "ix": 'ix_%(column_0_label)s',
  "uq": "uq_%(table_name)s_%(column_0_name)s",
  "ck": "ck_%(table_name)s_%(constraint_name)s",
  "fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
  "pk": "pk_%(table_name)s"
    "ix": 'ix_%(column_0_label)s',
    "uq": "uq_%(table_name)s_%(column_0_name)s",
    "ck": "ck_%(table_name)s_%(constraint_name)s",
    "fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
    "pk": "pk_%(table_name)s"
})

DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
Base = declarative_base(metadata=metadata, cls=BaseModel)
Versioned = make_versioned_class()

D fanboi2/models/_identity.py => fanboi2/models/_identity.py +0 -63
@@ 1,63 0,0 @@
import datetime
import hashlib
import pytz
import random
import string


class Identity(object):
    """Generates a unique user identity for each user based on IP address."""
    STRINGS = string.ascii_letters + string.digits + "+/."

    def __init__(self, redis=None):
        self.timezone = pytz.utc
        self.redis = redis

    def configure_tz(self, timezone):
        """Configure timezone to use for key generation.

        :param timezone: A timezone :type:`str`.

        :type timezone: str
        :rtype: None
        """
        self.timezone = pytz.timezone(timezone)

    def _key(self, ip_address, namespace="default"):
        """Generate a unique key for each :attr:`ip_address` under namespace
        :attr:`namespace`. Generated key will contain the current date in
        the configured timezone to ensure key is unique to each day.

        :param ip_address: An IP address :type:`str`.
        :param namespace: A namespace :type:`str` to generate key in.
        :type ip_address: str
        :type namespace: str
        :rtype: str
        """
        today = datetime.datetime.now(self.timezone).strftime("%Y%m%d")
        return "ident:%s:%s:%s" % (today,
                                   namespace,
                                   hashlib.md5(ip_address.encode('utf8')).
                                       hexdigest())

    def get(self, *args, **kwargs):
        """Retrieve user ident from Redis or generate a new one if it does
        not already exists. Ident is generated from a random string and
        expired every 24 hours.

        :param args: Arguments that will be passed to :meth:`_key`.
        :param kwargs: Keyword arguments that will be passed to :meth:`_key`.

        :type args: list
        :type kwargs: dict
        :rtype: str
        """
        key = self._key(*args, **kwargs)
        ident = self.redis.get(key)
        if ident is None:
            ident = ''.join(random.choice(self.STRINGS) for x in range(9))
            self.redis.setnx(key, ident)
            self.redis.expire(key, 86400)
        else:
            ident = ident.decode('utf-8')
        return ident

D fanboi2/models/_redis_proxy.py => fanboi2/models/_redis_proxy.py +0 -21
@@ 1,21 0,0 @@
import redis


class RedisProxy(object):
    """Wrapper around :class:`redis.StrictRedis` to allow late binding of
    Redis object. This wrapper will proxy all method calls to Redis object
    if initialized.
    """

    def __init__(self, cls=redis.StrictRedis):
        self._cls = cls
        self._redis = None

    def from_url(self, *args, **kwargs):
        self._redis = self._cls.from_url(*args, **kwargs)

    def __getattr__(self, name):
        if self._redis is not None:
            return self._redis.__getattribute__(name)
        raise RuntimeError("{} is not initialized".
                           format(repr(self._cls.__name__)))

M fanboi2/models/_versioned.py => fanboi2/models/_versioned.py +16 -51
@@ 1,4 1,5 @@
from collections import OrderedDict

from sqlalchemy import event
from sqlalchemy.ext.declarative import declared_attr
from sqlalchemy.orm import attributes, object_mapper, mapper


@@ 17,10 18,6 @@ def _is_fk_column(column, table):

    :param column: SQLAlchemy column.
    :param table: SQLAlchemy table.

    :type column: sqlalchemy.sql.schema.Column
    :type table: sqlalchemy.sql.schema.Table
    :rtype: bool
    """
    for fk in column.foreign_keys:
        if fk.references(table):


@@ 32,9 29,6 @@ def _is_versioning_column(column):
    """Returns :type:`True` if column belongs to versioning table.

    :param column: SQLAlchemy column to check.

    :type column: sqlalchemy.sql.schema.Column
    :rtype: bool
    """
    return "version_meta" in column.info



@@ 45,9 39,6 @@ def _copy_history_column(column):
    and its unique constraint removed.

    :param column: SQLAlchemy column to copy.

    :type column: sqlalchemy.sql.schema.Column
    :rtype: sqlalchemy.sql.schema.Column
    """
    new_column = column.copy()
    new_column.unique = False


@@ 62,9 53,6 @@ def _history_mapper(model_mapper):
    """Configure SQLAlchemy mapper and enable history support.

    :param model_mapper: SQLAlchemy mapper object for a model.

    :type model_mapper: sqlalchemy.orm.mapper.Mapper
    :rtype: sqlalchemy.orm.mapper.Mapper
    """
    model_class = model_mapper.class_
    model_table = model_mapper.local_table


@@ 179,23 167,15 @@ def _history_mapper(model_mapper):
        model_mapper.add_property('version', model_table.c.version)


def make_versioned(session, retrieve=None):
    """Enable the versioned mapper. If ``retrieve`` is given, the function
    will be used for retrieving a list of mapper objects (see also
    :func:`make_versioned_class`).
def setup_versioned(retrieve=None):
    """Enable the versioned for the mapper. If ``retrieve`` is given
    the function will be used for retrieving a list of mapper objects
    (see also :func:`make_versioned_class`).

    :param session: SQLAlchemy session object.
    :param retrieve: Function for retrieve a list of mapper objects.

    :type session: sqlalchemy.orm.session.Session
    :type retrieve: function | None

    :rtype: sqlalchemy.orm.session.Session
    """
    _make_history_event(session)

    if retrieve is None:
        retrieve = lambda: _versioned_mappers
        retrieve = lambda: _versioned_mappers  # noqa: E731

    for model_mapper in retrieve():
        _history_mapper(model_mapper)


@@ 204,15 184,12 @@ def make_versioned(session, retrieve=None):
def make_versioned_class(register=None):
    """Factory for creating a Versioned mixin. If ``register`` is given, the
    function will be used for registering the mapper object (see also
    :func:`make_versioned`).
    :func:`setup_versioned`).

    :param register: Function for registering a mapper object.
    :type register: function | None

    :rtype: class
    """
    if register is None:
        register = lambda m: _versioned_mappers.append(m)
        register = lambda m: _versioned_mappers.append(m)  # noqa: E731

    class _Versioned(object):
        """Mixin for enabling versioning for a model."""


@@ 232,9 209,6 @@ def _is_versioned_object(obj):
    """Returns `True` if object is version-enabled.

    :param obj: SQLAlchemy model object.

    :type obj: sqlalchemy.ext.declarative.api.Base
    :rtype: bool
    """
    return hasattr(obj, '__history_mapper__')



@@ 246,11 220,6 @@ def _create_version(obj, session, type_=None, force=False):
    :param session: SQLAlchemy session object.
    :param type_: Type of a change.
    :param force: Flag to always create version.

    :type obj: sqlalchemy.ext.declarative.api.Base
    :type session: sqlalchemy.orm.scoping.scoped_session
    :type type_: string
    :type force: bool
    """
    obj_mapper = object_mapper(obj)
    history_mapper = obj.__history_mapper__


@@ 282,14 251,16 @@ def _create_version(obj, session, type_=None, force=False):
            if prop.key not in obj_state.dict:
                getattr(obj, prop.key)

            added_, unchanged_, deleted_ = attributes.get_history(obj, prop.key)
            added_, unchanged_, deleted_ = attributes.get_history(
                obj,
                prop.key)

            if deleted_:
                attr[prop.key] = deleted_[0]
                obj_changed = True
            elif unchanged_:
                attr[prop.key] = unchanged_[0]
            elif added_:
            elif added_:  # pragma: no cover
                obj_changed = True

    if not obj_changed:


@@ 306,7 277,7 @@ def _create_version(obj, session, type_=None, force=False):
                if obj_changed is True:
                    break

    if not obj_changed and not force:
    if not obj_changed and not force:  # pragma: no cover
        return

    attr['version'] = obj.version


@@ 324,8 295,6 @@ def _create_history_dirty(session, objs):

    :param session: SQLAlchemy sesion object.
    :param objs: Dirty objects usually obtained by ``session.dirty``
    :type session: sqlalchemy.orm.session.Session
    :type objs: list[sqlalchemy.ext.declarative.api.Base]
    """
    for obj in objs:
        if _is_versioned_object(obj):


@@ 339,8 308,6 @@ def _create_history_deleted(session, objs):

    :param session: SQLAlchemy sesion object.
    :param objs: Dirty objects usually obtained by ``session.deleted``
    :type session: sqlalchemy.orm.session.Session
    :type objs: list[sqlalchemy.ext.declarative.api.Base]
    """
    for obj in objs:
        if _is_versioned_object(obj):


@@ 352,7 319,7 @@ def _create_history_deleted(session, objs):
        related = None

        def _create_cascade_version(target_obj):
            if not target_obj in objs:
            if target_obj not in objs:
                _create_version(target_obj,
                                session,
                                type_='update.cascade',


@@ 374,16 341,14 @@ def _create_history_deleted(session, objs):
                try:
                    for target_obj in related:
                        _create_cascade_version(target_obj)
                except TypeError:
                except TypeError:  # pragma: no cover
                    _create_cascade_version(related)


def _make_history_event(session):
def make_history_event(session):
    """Registers an event for recording versioned object.

    :param session: SQLAlchemy session object.
    :type session: sqlalchemy.orm.session.Session
    :rtype: sqlalchemy.orm.session.Session
    """
    @event.listens_for(session, 'before_flush')
    def update_history(session_, context, instances):

M fanboi2/models/board.py => fanboi2/models/board.py +20 -13
@@ 2,9 2,10 @@ from sqlalchemy.ext.declarative import declared_attr
from sqlalchemy.orm import synonym
from sqlalchemy.sql import func
from sqlalchemy.sql.schema import Column
from sqlalchemy.sql.sqltypes import Integer, DateTime, String, Text, \
    Unicode, Enum
from ._base import Base, JsonType, Versioned
from sqlalchemy.sql.sqltypes import DateTime, Enum, Integer, String
from sqlalchemy.sql.sqltypes import Text, Unicode, JSON

from ._base import Base, Versioned


DEFAULT_BOARD_CONFIG = {


@@ 15,6 16,14 @@ DEFAULT_BOARD_CONFIG = {
}


BoardStatusEnum = Enum(
    'open',
    'restricted',
    'locked',
    'archived',
    name='board_status')


class Board(Versioned, Base):
    """Model class for board. This model serve as a category to topic and
    also holds settings regarding how posts are created and displayed. It


@@ 28,18 37,14 @@ class Board(Versioned, Base):
    updated_at = Column(DateTime(timezone=True), onupdate=func.now())
    slug = Column(String(64), unique=True, nullable=False)
    title = Column(Unicode(255), nullable=False)
    _settings = Column('settings', JsonType, nullable=False, default={})
    _settings = Column('settings', JSON, nullable=False, default={})
    agreements = Column(Text, nullable=True)
    description = Column(Text, nullable=True)
    status = Column(Enum('open',
                         'restricted',
                         'locked',
                         'archived',
                         name='board_status'),
                    default='open',
                    nullable=False)
    status = Column(BoardStatusEnum, default='open', nullable=False)

    def get_settings(self):
        if self._settings is None:
            return DEFAULT_BOARD_CONFIG
        settings = DEFAULT_BOARD_CONFIG.copy()
        settings.update(self._settings)
        return settings


@@ 49,5 54,7 @@ class Board(Versioned, Base):

    @declared_attr
    def settings(cls):
        return synonym('_settings', descriptor=property(cls.get_settings,
                                                        cls.set_settings))
        return synonym('_settings',
                       descriptor=property(
                           cls.get_settings,
                           cls.set_settings))

M fanboi2/models/page.py => fanboi2/models/page.py +2 -1
@@ 1,6 1,7 @@
from sqlalchemy.sql import func
from sqlalchemy.sql.schema import Column, UniqueConstraint
from sqlalchemy.sql.sqltypes import Integer, DateTime, String, Text, Unicode
from sqlalchemy.sql.sqltypes import DateTime, Integer, String, Text, Unicode

from ._base import Base, Versioned



M fanboi2/models/post.py => fanboi2/models/post.py +3 -19
@@ 1,8 1,9 @@
from sqlalchemy import event
from sqlalchemy.dialects.postgresql import INET
from sqlalchemy.orm import backref, relationship
from sqlalchemy.sql import func
from sqlalchemy.sql.schema import Column, ForeignKey, UniqueConstraint
from sqlalchemy.sql.sqltypes import Integer, DateTime, String, Text, Boolean

from ._base import Base, Versioned




@@ 19,7 20,7 @@ class Post(Versioned, Base):
    created_at = Column(DateTime(timezone=True), default=func.now())
    updated_at = Column(DateTime(timezone=True), onupdate=func.now())
    topic_id = Column(Integer, ForeignKey('topic.id'), nullable=False)
    ip_address = Column(String, nullable=False)
    ip_address = Column(INET, nullable=False)
    ident = Column(String(32), nullable=True)
    number = Column(Integer, nullable=False)
    name = Column(String, nullable=False)


@@ 31,20 32,3 @@ class Post(Versioned, Base):
                                         lazy='dynamic',
                                         cascade='all,delete',
                                         order_by='Post.number'))


@event.listens_for(Post.__mapper__, 'before_insert')
def populate_post_name(mapper, connection, target):
    """Populate :attr:`Post.name` using name set within :attr:`Post.settings`
    if name is empty otherwise use user input.
    """
    if target.name is None:
        target.name = target.topic.board.settings['name']


@event.listens_for(Post.__mapper__, 'before_insert')
def populate_post_ident(mapper, connection, target):
    from . import identity
    board = target.topic.board
    if board.settings['use_ident']:
        target.ident = identity.get(target.ip_address, board.slug)

M fanboi2/models/rule.py => fanboi2/models/rule.py +6 -5
@@ 1,7 1,8 @@
from sqlalchemy.dialects.postgresql import INET
from sqlalchemy.sql import func, desc, and_, or_
from sqlalchemy.sql import func, and_, or_
from sqlalchemy.sql.schema import Column
from sqlalchemy.sql.sqltypes import Integer, DateTime, Boolean, String, Unicode
from sqlalchemy.sql.sqltypes import Boolean, DateTime, Integer, String, Unicode

from ._base import Base




@@ 27,12 28,12 @@ class Rule(Base):

    @classmethod
    def listed(cls, ip_address, scopes=None):
        scope_q = cls.scope == None
        scope_q = cls.scope == None  # noqa: E712
        if scopes is not None:
            scope_q = or_(scope_q, cls.scope.in_(scopes))
        return and_(
            scope_q,
            cls.active == True,
            cls.active == True,  # noqa: E712
            cls.ip_address.op('>>=')(ip_address),
            or_(cls.active_until == None,
            or_(cls.active_until == None,  # noqa: E712
                cls.active_until >= func.now()))

M fanboi2/models/rule_ban.py => fanboi2/models/rule_ban.py +2 -2
@@ 1,6 1,6 @@
from sqlalchemy.orm import column_property
from sqlalchemy.sql.schema import Column, ForeignKey
from sqlalchemy.sql.sqltypes import Integer, String
from sqlalchemy.sql.sqltypes import Integer

from .rule import Rule



D fanboi2/models/rule_override.py => fanboi2/models/rule_override.py +0 -15
@@ 1,15 0,0 @@
from sqlalchemy.orm import column_property
from sqlalchemy.sql.schema import Column, ForeignKey
from sqlalchemy.sql.sqltypes import Integer, String
from ._base import JsonType
from .rule import Rule


class RuleOverride(Rule):
    """Model class that provides a settings override on top of rule model."""

    __tablename__ = 'rule_override'
    __mapper_args__ = {'polymorphic_identity': 'override'}

    rule_id = Column(Integer, ForeignKey('rule.id'), primary_key=True)
    override = Column(JsonType, default={})

A fanboi2/models/setting.py => fanboi2/models/setting.py +19 -0
@@ 0,0 1,19 @@
from sqlalchemy.sql.schema import Column
from sqlalchemy.sql.sqltypes import String, JSON

from ._base import Base, Versioned


DEFAULT_SETTINGS = {
    'app.time_zone': 'UTC',
    'app.ident_size': 10,
}


class Setting(Versioned, Base):
    """Model class for various site settings."""

    __tablename__ = 'setting'

    key = Column(String, nullable=False, primary_key=True)
    value = Column(JSON)

M fanboi2/models/topic.py => fanboi2/models/topic.py +5 -3
@@ 1,8 1,10 @@
import re

from sqlalchemy.orm import backref, relationship
from sqlalchemy.sql import desc, func, select
from sqlalchemy.sql.schema import Column, ForeignKey
from sqlalchemy.sql.sqltypes import Integer, DateTime, Enum, Unicode

from ._base import Base, Versioned
from .post import Post
from .topic_meta import TopicMeta


@@ 30,9 32,9 @@ class Topic(Versioned, Base):
                                         lazy='dynamic',
                                         cascade='all,delete',
                                         order_by=desc(func.coalesce(
                                             select([TopicMeta.bumped_at]).\
                                                where(TopicMeta.topic_id==id).\
                                                as_scalar(),
                                             select([TopicMeta.bumped_at]).
                                             where(TopicMeta.topic_id == id).
                                             as_scalar(),
                                             created_at))))

    QUERY = (

A fanboi2/redis.py => fanboi2/redis.py +11 -0
@@ 0,0 1,11 @@
from redis import StrictRedis


def includeme(config):  # pragma: no cache
    redis_url = config.registry.settings['redis.url']
    redis_conn = StrictRedis.from_url(redis_url)

    def redis_conn_factory(context, request):
        return redis_conn

    config.register_service_factory(redis_conn_factory, name='redis')

D fanboi2/scripts/__init__.py => fanboi2/scripts/__init__.py +0 -1
@@ 1,1 0,0 @@
# package

D fanboi2/scripts/board_create.py => fanboi2/scripts/board_create.py +0 -42
@@ 1,42 0,0 @@
import optparse
import sys
import transaction
from pyramid.paster import setup_logging, get_appsettings
from sqlalchemy import engine_from_config
from ..models import DBSession, Board


DESCRIPTION = "Insert a new board into the database."
USAGE = "Usage: %prog config arguments"


def main(argv=sys.argv):
    parser = optparse.OptionParser(usage=USAGE, description=DESCRIPTION)
    parser.add_option('-t', '--title', dest='title', type='string')
    parser.add_option('-s', '--slug', dest='slug', type='string')

    if not argv or len(argv) < 2:
        parser.print_help()
        sys.exit(1)

    config_uri = argv[1]
    argv = argv[2:]

    options, args = parser.parse_args(argv)
    if options.title is None:
        parser.error('You must provide at least --title')

    slug = options.slug
    if slug is None:
        slug = options.title.lower().replace(' ', '_')

    setup_logging(config_uri)
    settings = get_appsettings(config_uri)
    engine = engine_from_config(settings, 'sqlalchemy.')
    DBSession.configure(bind=engine)
    with transaction.manager:
        board = Board(title=options.title, slug=slug)
        DBSession.add(board)
        DBSession.flush()
        print(("Successfully added %s (slug: %s)" %
               (board.title, board.slug)))

D fanboi2/scripts/board_update.py => fanboi2/scripts/board_update.py +0 -83
@@ 1,83 0,0 @@
import json
import optparse
import os
import sys
import tempfile
import transaction
from pyramid.paster import setup_logging, get_appsettings
from sqlalchemy import engine_from_config
from sqlalchemy.orm.exc import NoResultFound
from subprocess import call
from fanboi2 import DBSession
from fanboi2.models import Board, JsonType


DESCRIPTION = "Update board settings."
USAGE = "Usage: %prog config arguments"


def main(argv=sys.argv):
    parser = optparse.OptionParser(usage=USAGE, description=DESCRIPTION)
    parser.add_option('-f', '--field', dest='field', type='string')
    parser.add_option('-s', '--slug', dest='slug', type='string')

    if not argv or len(argv) < 2:
        parser.print_help()
        sys.exit(1)

    config_uri = argv[1]
    argv = argv[2:]

    options, args = parser.parse_args(argv)
    if options.field is None:
        parser.error('You must provide --field')
    if options.slug is None:
        parser.error('You must provide --slug')

    setup_logging(config_uri)
    settings = get_appsettings(config_uri)
    engine = engine_from_config(settings, 'sqlalchemy.')
    DBSession.configure(bind=engine)
    with transaction.manager:
        original = None
        modified = None
        board = None
        serialized = False

        try:
            board = DBSession.query(Board).filter_by(slug=options.slug).one()
        except NoResultFound:
            print('No board with %s slug found.' % options.slug)
            sys.exit(1)

        try:
            original = getattr(board, options.field)
        except AttributeError:
            print('No field %s found in board.' % options.field)
            sys.exit(1)

        with tempfile.NamedTemporaryFile(suffix='.tmp') as tmp:
            if original is not None:
                if isinstance(getattr(Board, options.field).type, JsonType):
                    serialized = True
                    dumps = json.dumps(original, indent=4)
                    tmp.write(bytes(dumps.encode('utf8')))
                else:
                    tmp.write(bytes(str(original).encode('utf8')))
                tmp.flush()
            call([os.environ.get('EDITOR', 'vi'), tmp.name])
            modified = open(tmp.name, "r").read()

        if serialized:
            modified = json.loads(modified)
        else:
            original = original if original is not None else str()
            modified = modified.rstrip("\r\n")

        if modified == original:
            print('Not modified.')
        else:
            setattr(board, options.field, modified)
            DBSession.add(board)
            print('Successfully updated %s for %s.' %
                  (options.field, board.title))

D fanboi2/scripts/celery.py => fanboi2/scripts/celery.py +0 -14
@@ 1,14 0,0 @@
import os
import sys
from fanboi2.tasks import celery
from pyramid.paster import bootstrap


def main(argv=sys.argv):
    if not len(argv) >= 2:
        sys.stderr.write("Usage: %s config\n" % os.path.basename(argv[0]))
        sys.stderr.write("Configuration file not present.\n")
        sys.exit(1)

    bootstrap(argv[1])
    celery.start(argv[:1] + argv[2:])  # Remove argv[1].

D fanboi2/scripts/topic_sync.py => fanboi2/scripts/topic_sync.py +0 -29
@@ 1,29 0,0 @@
import os
import sys
import transaction
import sqlalchemy as sa
from zope.sqlalchemy import mark_changed
from fanboi2.models import DBSession, TopicMeta, Post
from pyramid.paster import bootstrap


def main(argv=sys.argv):
    if not len(argv) >= 2:
        sys.stderr.write("Usage: %s config\n" % os.path.basename(argv[0]))
        sys.stderr.write("Configuration file not present.\n")
        sys.exit(1)

    bootstrap(argv[1])

    query = TopicMeta.__table__.update().\
            values(bumped_at=sa.select([Post.created_at]).\
                   where(Post.topic_id == TopicMeta.topic_id).\
                   where(Post.bumped).\
                   order_by(sa.desc(Post.created_at)).\
                   limit(1))

    with transaction.manager:
        session = DBSession()
        results = session.execute(query)
        mark_changed(session)
        print("Successfully synced topic.")

M fanboi2/serializers.py => fanboi2/serializers.py +26 -39
@@ 1,6 1,24 @@
import re
import datetime
import pytz
from fanboi2.helpers.formatters import format_post, format_page

from .helpers.formatters import format_post, format_page
from .interfaces import ISettingQueryService


TRUTHY_RE = re.compile('^Y|y|T|t|[1-9]')


def _truthy(request, key):
    """Check that ``key`` in the request params is truty value.

    :param request: A :class:`pyramid.request.Request` object.
    :param key: A :type:`str` to lookup for.
    """
    if key not in request.params:
        return False
    val = str(request.params.get(key, 'false'))
    return bool(TRUTHY_RE.match(val))


def _datetime_adapter(obj, request):


@@ 8,13 26,9 @@ def _datetime_adapter(obj, request):

    :param obj: A :class:`datetime.datetime` object.
    :param request: A :class:`pyramid.request.Request` object.

    :type obj: datetime.datetime
    :type request: pyramid.request.Request
    """
    settings = request.registry.settings
    assert isinstance(settings, dict)
    tz = pytz.timezone(settings['app.timezone'])
    setting_query_svc = request.find_service(ISettingQueryService)
    tz = pytz.timezone(setting_query_svc.value_from_key('app.time_zone'))
    return obj.astimezone(tz).isoformat()




@@ 23,9 37,6 @@ def _sqlalchemy_query_adapter(obj, request):

    :param obj: An iterable SQLAlchemy's :class:`sqlalchemy.orm.Query` object.
    :param request: A :class:`pyramid.request.Request` object.

    :type obj: sqlalchemy.orm.Query
    :type request: pyramid.request.Request
    """
    return [item for item in obj]



@@ 35,10 46,6 @@ def _board_serializer(obj, request):

    :param obj: A :class:`fanboi2.models.Board` object.
    :param request: A :class:`pyramid.request.Request` object.

    :type obj: fanboi2.models.Board
    :type request: pyramid.request.Request
    :rtype: dict
    """
    result = {
        'type': 'board',


@@ 51,7 58,7 @@ def _board_serializer(obj, request):
        'title': obj.title,
        'path': request.route_path('api_board', board=obj.slug),
    }
    if request.params.get('topics') and not 'board' in request.params:
    if _truthy(request, 'topics') and not _truthy(request, 'board'):
        result['topics'] = obj.topics.limit(10)
    return result



@@ 61,10 68,6 @@ def _topic_serializer(obj, request):

    :param obj: A :class:`fanboi2.models.Topic` object.
    :param request: A :class:`pyramid.request.Request` object.

    :type obj: fanboi2.models.Topic
    :type request: pyramid.request.Request
    :rtype: dict
    """
    result = {
        'type': 'topic',


@@ 78,9 81,9 @@ def _topic_serializer(obj, request):
        'title': obj.title,
        'path': request.route_path('api_topic', topic=obj.id),
    }
    if request.params.get('board'):
    if _truthy(request, 'board'):
        result['board'] = obj.board
    if request.params.get('posts') and not 'topic' in request.params:
    if _truthy(request, 'posts') and not _truthy(request, 'topic'):
        result['posts'] = obj.recent_posts()
    return result



@@ 112,7 115,7 @@ def _post_serializer(obj, request):
            query=obj.number,
        ),
    }
    if request.params.get('topic'):
    if _truthy(request, 'topic'):
        result['topic'] = obj.topic
    return result



@@ 122,10 125,6 @@ def _page_serializer(obj, request):

    :param obj: A :class:`fanboi2.models.Page` object.
    :param request: A :class:`pyramid.request.Request` object.

    :type obj: fanboi2.models.Page
    :type request: pyramid.request.Request
    :rtype: dict
    """
    return {
        'type': 'page',


@@ 149,10 148,6 @@ def _result_proxy_serializer(obj, request):

    :param obj: A :class:`fanboi2.tasks.ResultProxy` object.
    :param request: A :class:`pyramid.request.Request` object.

    :type obj: fanboi2.tasks.ResultProxy
    :type request: pyramid.request.Request
    :rtype: dict
    """
    result = {
        'type': 'task',


@@ 161,7 156,7 @@ def _result_proxy_serializer(obj, request):
        'path': request.route_path('api_task', task=obj.id),
    }
    if obj.success():
        result['data'] = obj.object
        result['data'] = obj.deserialize(request)
    return result




@@ 170,10 165,6 @@ def _async_result_serializer(obj, request):

    :param obj: A :class:`celery.result.AsyncResult` object.
    :param request: A :class:`pyramid.request.Request` object.

    :type obj: celery.result.AsyncResult
    :type request: pyramid.request.Request
    :rtype: dict
    """
    return {
        'type': 'task',


@@ 189,10 180,6 @@ def _base_error_serializer(obj, request):

    :param obj: A :class:`fanboi2.errors.BaseError` object.
    :param request: A :class:`pyramid.request.Request` object.

    :type obj: fanboi2.errors.BaseError
    :type request: pyramid.request.Request
    :rtype: dict
    """
    return {
        'type': 'error',

A fanboi2/services/__init__.py => fanboi2/services/__init__.py +157 -0
@@ 0,0 1,157 @@
from ..interfaces import \
    IBoardQueryService,\
    IFilterService,\
    IIdentityService,\
    IPageQueryService,\
    IPostCreateService,\
    IPostQueryService,\
    IRateLimiterService,\
    IRuleBanQueryService, \
    ISettingQueryService,\
    ITaskQueryService,\
    ITopicCreateService,\
    ITopicQueryService

from .board import BoardQueryService
from .filter_ import FilterService
from .identity import IdentityService
from .page import PageQueryService
from .post import PostCreateService, PostQueryService
from .rate_limiter import RateLimiterService
from .rule import RuleBanQueryService
from .setting import SettingQueryService
from .task import TaskQueryService
from .topic import TopicCreateService, TopicQueryService


def includeme(config):  # pragma: no cover

    # Board Query

    def board_query_factory(context, request):
        dbsession = request.find_service(name='db')
        return BoardQueryService(dbsession)

    config.register_service_factory(
        board_query_factory,
        IBoardQueryService)

    # Filter

    def filter_factory(context, request):
        filters = request.registry['filters']

        def service_query_fn(*a, **k):
            return request.find_service(*a, **k)

        return FilterService(filters, service_query_fn)

    config.register_service_factory(
        filter_factory,
        IFilterService)

    # Identity

    def identity_factory(context, request):
        redis_conn = request.find_service(name='redis')
        setting_query_svc = request.find_service(ISettingQueryService)
        ident_size = setting_query_svc.value_from_key('app.ident_size')
        return IdentityService(redis_conn, ident_size)

    config.register_service_factory(
        identity_factory,
        IIdentityService)

    # Page Query

    def page_query_factory(context, request):
        dbsession = request.find_service(name='db')
        return PageQueryService(dbsession)

    config.register_service_factory(
        page_query_factory,
        IPageQueryService)

    # Post Create

    def post_create_factory(context, request):
        dbsession = request.find_service(name='db')
        identity_svc = request.find_service(IIdentityService)
        setting_query_svc = request.find_service(ISettingQueryService)
        return PostCreateService(dbsession, identity_svc, setting_query_svc)

    config.register_service_factory(
        post_create_factory,
        IPostCreateService)

    # Post Query

    def post_query_factory(context, request):
        dbsession = request.find_service(name='db')
        return PostQueryService(dbsession)

    config.register_service_factory(
        post_query_factory,
        IPostQueryService)

    # Rate Limiter

    def rate_limiter_factory(context, request):
        redis_conn = request.find_service(name='redis')
        return RateLimiterService(redis_conn)

    config.register_service_factory(
        rate_limiter_factory,
        IRateLimiterService)

    # RuleBan query

    def rule_ban_query_factory(context, request):
        dbsession = request.find_service(name='db')
        return RuleBanQueryService(dbsession)

    config.register_service_factory(
        rule_ban_query_factory,
        IRuleBanQueryService)

    # Setting Query

    def setting_query_factory(context, request):
        dbsession = request.find_service(name='db')
        cache_region = request.find_service(name='cache')
        return SettingQueryService(dbsession, cache_region)

    config.register_service_factory(
        setting_query_factory,
        ISettingQueryService)

    # Task Query

    def task_query_factory(context, request):
        return TaskQueryService()

    config.register_service_factory(
        task_query_factory,
        ITaskQueryService)

    # Topic Create

    def topic_create_factory(context, request):
        dbsession = request.find_service(name='db')
        identity_svc = request.find_service(IIdentityService)
        setting_query_svc = request.find_service(ISettingQueryService)
        return TopicCreateService(dbsession, identity_svc, setting_query_svc)

    config.register_service_factory(
        topic_create_factory,
        ITopicCreateService)

    # Topic Query

    def topic_query_factory(context, request):
        dbsession = request.find_service(name='db')
        return TopicQueryService(dbsession)

    config.register_service_factory(
        topic_query_factory,
        ITopicQueryService)

A fanboi2/services/board.py => fanboi2/services/board.py +26 -0
@@ 0,0 1,26 @@
from ..models import Board


class BoardQueryService(object):
    """Board query service provides a service for querying a board
    or a collection of boards from the database.
    """

    def __init__(self, dbsession):
        self.dbsession = dbsession

    def list_active(self):
        """Query all boards that are not archived."""
        return list(
            self.dbsession.query(Board).
            order_by(Board.title).
            filter(Board.status != 'archived'))

    def board_from_slug(self, board_slug):
        """Query a board from the given board slug.

        :param board_slug: The slug :type:`str` identifying a board.
        """
        return self.dbsession.query(Board).\
            filter_by(slug=board_slug).\
            one()

A fanboi2/services/filter_.py => fanboi2/services/filter_.py +48 -0
@@ 0,0 1,48 @@
from collections import namedtuple

from ..interfaces import ISettingQueryService, IPostQueryService


FilterResult = namedtuple('FilterResult', ('rejected_by', 'filters'))


class FilterService(object):
    """Filter service provides a service for evaluating content using
    predefined sets of pre-posting filters.
    """

    def __init__(self, filters, service_query_fn):
        self.filters = filters
        self.service_query_fn = service_query_fn

    def evaluate(self, payload):
        """Evaluate the given payload with filters.

        :param payload: A filter payload to verify.
        """
        filters_chain = []
        setting_query_svc = self.service_query_fn(ISettingQueryService)

        if 'ip_address' in payload:
            post_query_svc = self.service_query_fn(IPostQueryService)
            if post_query_svc.was_recently_seen(payload['ip_address']):
                return FilterResult(rejected_by=None, filters=[])

        for name, cls in self.filters:
            services = {}
            filters_chain.append(name)
            if hasattr(cls, '__use_services__'):
                for s in cls.__use_services__:
                    services[s] = self.service_query_fn(name=s)

            settings_name = "ext.filters.%s" % (name,)
            settings_default = getattr(cls, '__default_settings__', None)
            settings = setting_query_svc.value_from_key(settings_name)
            if settings is None:
                settings = settings_default

            f = cls(settings, services)
            if f.should_reject(payload):
                return FilterResult(rejected_by=name, filters=filters_chain)

        return FilterResult(rejected_by=None, filters=filters_chain)

A fanboi2/services/identity.py => fanboi2/services/identity.py +37 -0
@@ 0,0 1,37 @@
import string
import random


STRINGS = string.ascii_letters + string.digits + "+/."


class IdentityService(object):
    """Identity service provides a service for querying an identity
    for a user given by a payload from the database or generate a new
    one if not already exists.
    """

    def __init__(self, redis_conn, ident_size):
        self.redis_conn = redis_conn
        self.ident_size = ident_size

    def _get_key(self, **kwargs):
        return 'services.identity.%s' % (
            (','.join('%s=%s' % (k, v) for k, v in sorted(kwargs.items()))),)

    def identity_for(self, **kwargs):
        """Query the identity for user matching :param:`kwargs` payload
        or generate a new one if not exists.

        :param payload: Payload to identify this rate limit.
        """
        key = self._get_key(**kwargs)
        ident = self.redis_conn.get(key)

        if ident is not None:
            return ident.decode('utf-8')

        ident = ''.join(random.choice(STRINGS) for x in range(self.ident_size))
        self.redis_conn.setnx(key, ident)
        self.redis_conn.expire(key, 86400)
        return ident

A fanboi2/services/page.py => fanboi2/services/page.py +35 -0
@@ 0,0 1,35 @@
from ..models import Page


class PageQueryService(object):
    """Page query service provides a service for querying a page
    or a collection of pages from the database.
    """

    def __init__(self, dbsession):
        self.dbsession = dbsession

    def list_public(self):
        """Query all public pages."""
        return list(
            self.dbsession.query(Page).
            order_by(Page.title).
            filter_by(namespace='public'))

    def public_page_from_slug(self, page_slug):
        """Query a public page from the given page slug.

        :param page_slug: A slug :type:`str` identifying the page.
        """
        return self.dbsession.query(Page).\
            filter_by(namespace='public', slug=page_slug).\
            one()

    def internal_page_from_slug(self, page_slug):
        """Query an internal page from the given page slug.

        :param page_slug: A slug :type:`str` identifying the page.
        """
        return self.dbsession.query(Page).\
            filter_by(namespace='internal', slug=page_slug).\
            one()

A fanboi2/services/post.py => fanboi2/services/post.py +136 -0
@@ 0,0 1,136 @@
import datetime

from sqlalchemy.sql import func
import pytz

from ..errors import StatusRejectedError
from ..models import Post, Topic
from ..tasks import add_post


class PostCreateService(object):
    """Post create service provides a service for creating a post."""

    def __init__(self, dbsession, identity_svc, setting_query_svc):
        self.dbsession = dbsession
        self.identity_svc = identity_svc
        self.setting_query_svc = setting_query_svc

    def enqueue(self, topic_id, body, bumped, ip_address, payload={}):
        """Enqueues the post creation to the posting queue. Posts that are
        queued will be processed with pre-posting filters using the given
        :param:`payload`.

        :param topic_id: A topic ID :type:`int` to lookup the post.
        :param body: A :type:`str` topic body.
        :param bumped: A :type:`bool` whether to bump the topic.
        :param ip_address: An IP address of the topic creator.
        :param payload: A request payload containing request metadata.
        """
        return add_post.delay(
            topic_id,
            body,
            bumped,
            ip_address,
            payload=payload)

    def create(self, topic_id, body, bumped, ip_address):
        """Creates a post and associate related metadata. Unlike ``enqueue``,
        this method performs the actual creation of the topic.

        :param topic_id: A topic ID :type:`int` to lookup the post.
        :param body: A :type:`str` topic body.
        :param bumped: A :type:`bool` whether to bump the topic.
        :param ip_address: An IP address of the topic creator.
        """

        # Preflight

        topic = self.dbsession.query(Topic).\
            with_for_update().\
            get(topic_id)

        topic_meta = topic.meta
        board = topic.board

        if topic.status != 'open':
            raise StatusRejectedError(topic.status)

        if board.status not in ('open', 'restricted'):
            raise StatusRejectedError(board.status)

        # Update topic meta

        topic_meta.post_count = topic_meta.post_count + 1
        topic_meta.posted_at = func.now()
        if bumped is None or bumped:
            topic_meta.bumped_at = func.now()

        self.dbsession.add(topic_meta)

        # Update topic

        max_posts = board.settings['max_posts']

        if topic.status == 'open' and topic_meta.post_count >= max_posts:
            topic.status = 'archived'
            self.dbsession.add(topic)

        # Create post

        ident = None
        if board.settings['use_ident']:
            time_zone = self.setting_query_svc.value_from_key('app.time_zone')
            tz = pytz.timezone(time_zone)
            timestamp = datetime.datetime.now(tz).strftime("%Y%m%d")
            ident = self.identity_svc.identity_for(
                board=board.slug,
                ip_address=ip_address,
                timestamp=timestamp)

        post = Post(
            topic=topic,
            number=topic_meta.post_count,
            body=body,
            bumped=bumped,
            name=board.settings['name'],
            ident=ident,
            ip_address=ip_address)

        self.dbsession.add(post)

        # Finalize

        self.dbsession.flush()
        return post


class PostQueryService(object):
    """Post query service provides a service for querying a collection
    of posts from the database.
    """

    def __init__(self, dbsession):
        self.dbsession = dbsession

    def list_from_topic_id(self, topic_id, query=None):
        """Query posts for the given topic matching query.

        :param topic_id: A topic ID :type:`int` to lookup the post.
        :param query: A query :type:`string` for scoping the post.
        """
        topic = self.dbsession.query(Topic).filter_by(id=topic_id).one()
        return list(topic.scoped_posts(query))

    def was_recently_seen(self, ip_address):
        """Returns whether the given IP address was recently seen.

        :param ip_address: An :type:`str` IP address to lookup.
        """
        anchor = datetime.datetime.now() - datetime.timedelta(days=3)
        q = self.dbsession.query(Post).\
            filter(
                Post.created_at >= anchor,
                Post.ip_address == ip_address).\
            exists()
        return self.dbsession.query(q).scalar()

A fanboi2/services/rate_limiter.py => fanboi2/services/rate_limiter.py +39 -0
@@ 0,0 1,39 @@
class RateLimiterService(object):
    """Rate limiter service provides a service for querying whether
    the user given by a payload should be rate-limited.
    """

    def __init__(self, redis_conn):
        self.redis_conn = redis_conn

    def _get_key(self, **kwargs):
        return 'services.rate_limiter:%s' % (
            (','.join('%s=%s' % (k, v) for k, v in sorted(kwargs.items()))),)

    def limit_for(self, expiration=10, **kwargs):
        """Set the rate limited key for the user and expire at least
        the given :param:`expiration` in seconds.

        :param expiration: A number of seconds to rate limited for.
        :param kwargs: Payload to identify this rate limit.
        """
        key = self._get_key(**kwargs)
        self.redis_conn.set(key, 1)
        self.redis_conn.expire(key, expiration)

    def is_limited(self, **kwargs):
        """Returns :type:`True` if the given :param:`kwargs` is rate limited.

        :param kwargs: Payload to identify this rate limit.
        """
        key = self._get_key(**kwargs)
        return self.redis_conn.exists(key)

    def time_left(self, **kwargs):
        """Returns the number of seconds left until the given payload
        is no longer rate limited.

        :param kwargs: Payload to identify this rate limit
        """
        key = self._get_key(**kwargs)
        return self.redis_conn.ttl(key)

A fanboi2/services/rule.py => fanboi2/services/rule.py +19 -0
@@ 0,0 1,19 @@
from ..models import RuleBan


class RuleBanQueryService(object):
    """Rule ban query service provides a service for query the ban list."""

    def __init__(self, dbsession):
        self.dbsession = dbsession

    def is_banned(self, ip_address, scopes=None):
        """Verify whether the IP address is in the ban list.

        :param ip_address: An IP address :type:`str` to lookup for.
        :param scopes: A scope :type:`str` to lookup for.
        """
        q = self.dbsession.query(RuleBan).\
            filter(RuleBan.listed(ip_address, scopes)).\
            exists()
        return self.dbsession.query(q).scalar()

A fanboi2/services/setting.py => fanboi2/services/setting.py +45 -0
@@ 0,0 1,45 @@
from ..models.setting import DEFAULT_SETTINGS, Setting


class SettingQueryService(object):
    """Setting query service provides a service for querying a runtime
    application settings that are not part of startup process, such as
    timezone.
    """

    def __init__(self, dbsession, cache_region):
        self.dbsession = dbsession
        self.cache_region = cache_region

    def _get_cache_key(self, key):
        """Returns a cache key for given key.

        :param key: The setting key.
        """
        return 'services.settings:key=%s' % (key,)

    def value_from_key(self, key, _default=DEFAULT_SETTINGS):
        """Returns a setting value either from a cache, from a database
        or the default value. The value returned by this method will be
        cached for 3600 seconds (1 hour).

        :param key: The setting key.
        """
        def _creator_fn():
            setting = self.dbsession.query(Setting).filter_by(key=key).first()
            if not setting:
                return _default.get(key, None)
            return setting.value
        return self.cache_region.get_or_create(
            self._get_cache_key(key),
            _creator_fn,
            expiration_time=3600)

    def reload(self, key):
        """Delete the given key from the cache to force reloading
        of the settings the next time it is accessed.

        :param key: The setting key.
        """
        cache_key = self._get_cache_key(key)
        return self.cache_region.delete(cache_key)

A fanboi2/services/task.py => fanboi2/services/task.py +13 -0
@@ 0,0 1,13 @@
from ..tasks import celery, ResultProxy


class TaskQueryService(object):
    """Task query service provides a service for querying a task status."""

    def result_from_uid(self, task_uid):
        """Query a task from the given task UID.

        :param task_uid: A UID :type:`str` for a task.
        """
        task = celery.AsyncResult(task_uid)
        return ResultProxy(task)

A fanboi2/services/topic.py => fanboi2/services/topic.py +153 -0
@@ 0,0 1,153 @@
import datetime

from sqlalchemy.orm import contains_eager
from sqlalchemy.sql import or_, and_, func, desc
import pytz

from ..errors import StatusRejectedError
from ..models import Board, Topic, TopicMeta, Post
from ..tasks import add_topic


class TopicCreateService(object):
    """Topic create service provides a service for creating a topic."""

    def __init__(self, dbsession, identity_svc, setting_query_svc):
        self.dbsession = dbsession
        self.identity_svc = identity_svc
        self.setting_query_svc = setting_query_svc

    def enqueue(self, board_slug, title, body, ip_address, payload={}):
        """Enqueues the topic creation to the posting queue. Topics that are
        queued will be processed with pre-posting filters using the given
        :param:`payload`.

        :param board_slug: The slug :type:`str` identifying a board.
        :param title: A :type:`str` topic title.
        :param body: A :type:`str` topic body.
        :param ip_address: An IP address of the topic creator.
        :param payload: A request payload containing request metadata.
        """
        return add_topic.delay(
            board_slug,
            title,
            body,
            ip_address,
            payload=payload)

    def create(self, board_slug, title, body, ip_address):
        """Creates a topic and associate related metadata. Unlike ``enqueue``,
        this method performs the actual creation of the topic.

        :param board_slug: The slug :type:`str` identifying a board.
        :param title: A :type:`str` topic title.
        :param body: A :type:`str` topic body.
        :param ip_address: An IP address of the topic creator.
        """

        # Preflight

        board = self.dbsession.query(Board).\
            filter(Board.slug == board_slug).\
            one()

        if board.status != 'open':
            raise StatusRejectedError(board.status)

        # Create topic

        topic = Topic(
            board=board,
            title=title,
            created_at=func.now(),
            updated_at=func.now(),
            status='open')

        self.dbsession.add(topic)

        # Create topic meta

        topic_meta = TopicMeta(
            topic=topic,
            post_count=1,
            posted_at=func.now(),
            bumped_at=func.now())

        self.dbsession.add(topic_meta)

        # Create post

        ident = None
        if board.settings['use_ident']:
            time_zone = self.setting_query_svc.value_from_key('app.time_zone')
            tz = pytz.timezone(time_zone)
            timestamp = datetime.datetime.now(tz).strftime("%Y%m%d")
            ident = self.identity_svc.identity_for(
                board=topic.board.slug,
                ip_address=ip_address,
                timestamp=timestamp)

        post = Post(
            topic=topic,
            number=topic_meta.post_count,
            body=body,
            bumped=True,
            name=board.settings['name'],
            ident=ident,
            ip_address=ip_address)

        self.dbsession.add(post)

        # Finalize

        self.dbsession.flush()
        return topic


class TopicQueryService(object):
    """Topic query service provides a service for querying a topic or
    a collection of topics from the database.
    """

    def __init__(self, dbsession):
        self.dbsession = dbsession

    def _list_q(self, board_slug):
        """Internal method for querying topic list.

        :param board_slug: The slug :type:`str` identifying a board.
        """
        anchor = datetime.datetime.now() - datetime.timedelta(days=7)
        return self.dbsession.query(Topic).\
            join(Topic.board, Topic.meta).\
            options(contains_eager(Topic.meta)).\
            filter(and_(Board.slug == board_slug,
                        or_(Topic.status == "open",
                            and_(Topic.status != "open",
                                 TopicMeta.bumped_at >= anchor)))).\
            order_by(desc(func.coalesce(
                TopicMeta.bumped_at,
                Topic.created_at)))

    def list_from_board_slug(self, board_slug):
        """Query topics for the given board slug.

        :param board_slug: The slug :type:`str` identifying a board.
        """
        return list(self._list_q(board_slug))

    def list_recent_from_board_slug(self, board_slug):
        """Query recent 10 topics for the given board slug.

        :param board_slug: The slug :type:`str` identifying a board.
        """
        return list(self._list_q(board_slug).limit(10))

    def topic_from_id(self, topic_id):
        """Query a topic from the given topic ID.

        :param topic_id: The ID :type:`int` identifying a topic.
        """
        return self.dbsession.query(Topic).\
            filter_by(id=topic_id).\
            one()

D fanboi2/tasks.py => fanboi2/tasks.py +0 -204
@@ 1,204 0,0 @@
import transaction
from celery import Celery, states
from sqlalchemy.exc import IntegrityError
from fanboi2.errors import serialize_error
from fanboi2.models import DBSession, Post, Topic, Board, \
    RuleBan, RuleOverride, serialize_model
from fanboi2.utils import akismet, dnsbl, proxy_detector, geoip, checklist

celery = Celery()


def configure_celery(settings):  # pragma: no cover
    """Returns a Celery configuration object.

    :param settings: A settings :type:`dict`.

    :type settings: dict
    :rtype: dict
    """
    return {
        'BROKER_URL': settings['celery.broker'],
        'CELERY_RESULT_BACKEND': settings['celery.broker'],
        'CELERY_ACCEPT_CONTENT': ['json'],
        'CELERY_TASK_SERIALIZER': 'json',
        'CELERY_RESULT_SERIALIZER': 'json',
        'CELERY_EVENT_SERIALIZER': 'json',
        'CELERY_TIMEZONE': settings['app.timezone'],
    }


class ResultProxy(object):
    """A proxy class for :class:`celery.result.AsyncResult` that provide
    results serialization using :func:`fanboi2.errors.serialize_error` and
    :func:`fanboi2.models.serialize_model`.

    :param result: A result of :class:`celery.AsyncResult`.
    :type result: celery.result.AsyncResult
    """

    def __init__(self, result):
        self._result = result
        self._object = None

    @property
    def object(self):
        """Serializing the result into Python object.

        :rtype: object
        """
        if self._object is None:
            obj, id_, *args = self._result.get()
            if obj == 'failure':
                self._object = serialize_error(id_, *args)
            else:
                class_ = serialize_model(obj)
                if class_ is not None:
                    self._object = DBSession.query(class_).\
                        filter_by(id=id_).\
                        one()
        return self._object

    def success(self):
        """Returns true if result was successfully processed.

        :rtype: bool
        """
        return self._result.state == states.SUCCESS

    def __getattr__(self, name):
        return self._result.__getattribute__(name)


@celery.task()
def add_topic(request, board_id, title, body):
    """Insert a topic to the database.

    :param request: A serialized request :type:`dict` returned from
                    :meth:`fanboi2.utils.serialize_request`.
    :param board_id: An :type:`int` referencing board ID.
    :param title: A :type:`str` topic title.
    :param body: A :type:`str` topic body.

    :type request: dict
    :type board_id: int
    :type title: str
    :type body: str
    :rtype: tuple
    """
    ip_address = request['remote_addr']
    country_code = geoip.country_code(ip_address)
    country_scope = 'country:%s' % (str(country_code).lower())

    with transaction.manager:
        board = DBSession.query(Board).get(board_id)
        board_scope = 'board:%s' % (board.slug,)

        if DBSession.query(RuleBan).\
           filter(RuleBan.listed(ip_address, scopes=(board_scope,))).\
           count() > 0:
            return 'failure', 'ban_rejected'

        override = {}
        rule_override = DBSession.query(RuleOverride).filter(
            RuleOverride.listed(ip_address, scopes=(board_scope,))).\
            first()

        if rule_override is not None:
            override = rule_override.override

        board_status = override.get('status', board.status)
        if board_status != 'open':
            return 'failure', 'status_rejected', board_status

        if checklist.enabled(country_scope, 'akismet') and \
           akismet.spam(request, body):
            return 'failure', 'spam_rejected'

        if checklist.enabled(country_scope, 'dnsbl') and \
           dnsbl.listed(ip_address):
            return 'failure', 'dnsbl_rejected'

        if checklist.enabled(country_scope, 'proxy_detect') and \
           proxy_detector.detect(ip_address):
            return 'failure', 'proxy_rejected'

        post = Post(body=body, ip_address=ip_address)
        post.topic = Topic(board=board, title=title)
        DBSession.add(post)
        DBSession.flush()
        return 'topic', post.topic_id


@celery.task(bind=True, max_retries=4)  # 5 total.
def add_post(self, request, topic_id, body, bumped):
    """Insert a post to a topic.

    :param self: A :class:`celery.Task` object.
    :param request: A serialized request :type:`dict` returned from
                    :meth:`fanboi2.utils.serialize_request`.
    :param topic_id: An :type:`int` referencing topic ID.
    :param body: A :type:`str` post body.
    :param bumped: A :type:`bool` specifying bump status.

    :type self: celery.Task
    :type request: dict
    :type topic_id: int
    :type body: str
    :type bumped: bool
    :rtype: tuple
    """
    ip_address = request['remote_addr']
    country_code = geoip.country_code(ip_address)
    country_scope = 'country:%s' % (str(country_code).lower())

    with transaction.manager:
        topic = DBSession.query(Topic).get(topic_id)
        board = topic.board
        board_scope = 'board:%s' % (board.slug,)

        if DBSession.query(RuleBan).\
           filter(RuleBan.listed(ip_address, scopes=(board_scope,))).\
           count() > 0:
            return 'failure', 'ban_rejected'

        if topic.status != 'open':
            return 'failure', 'status_rejected', topic.status

        override = {}
        rule_override = DBSession.query(RuleOverride).filter(
            RuleOverride.listed(ip_address, scopes=(board_scope,))).\
            first()

        if rule_override is not None:
            override = rule_override.override

        board_status = override.get('status', board.status)
        if not board_status in ('open', 'restricted'):
            return 'failure', 'status_rejected', board_status

        if checklist.enabled(country_scope, 'akismet') and \
           akismet.spam(request, body):
            return 'failure', 'spam_rejected'

        if checklist.enabled(country_scope, 'dnsbl') and \
           dnsbl.listed(ip_address):
            return 'failure', 'dnsbl_rejected'

        if checklist.enabled(country_scope, 'proxy_detect') and \
           proxy_detector.detect(ip_address):
            return 'failure', 'proxy_rejected'

        post = Post(
            topic=topic,
            body=body,
            bumped=bumped,
            ip_address=ip_address)

        try:
            DBSession.add(post)
            DBSession.flush()
        except IntegrityError as e:
            raise self.retry(exc=e)

        return 'post', post.id

A fanboi2/tasks/__init__.py => fanboi2/tasks/__init__.py +23 -0
@@ 0,0 1,23 @@
from ._base import celery
from ._result_proxy import ResultProxy
from .post import add_post
from .topic import add_topic


__all__ = [
    'ResultProxy',
    'add_post',
    'add_topic',
    'celery',
]


def configure_celery(settings):  # pragma: no cover
    """Configure Celery with the given settings."""
    celery.config_from_object({
        'broker_url': settings['celery.broker'],
        'result_backend': settings['celery.broker']})


def includeme(config):  # pragma: no cover
    configure_celery(config.registry.settings)

A fanboi2/tasks/_base.py => fanboi2/tasks/_base.py +25 -0
@@ 0,0 1,25 @@
import transaction
from celery import Celery
from celery import Task as BaseTask


celery = Celery()


class ModelTask(BaseTask):  # pragma: no cover
    """Provides a base class that automatically commit a transaction when
    the task is success, or abort when the task is a failure or will be
    retried.
    """

    def on_failure(self, exc, task_id, args, kwargs, einfo):
        """Task failure handler."""
        transaction.abort()

    def on_retry(self, exc, task_id, args, kwargs, einfo):
        """Task retry handler."""
        transaction.abort()

    def on_success(self, retval, task_id, args, kwargs):
        """Task success handler."""
        transaction.commit()

A fanboi2/tasks/_result_proxy.py => fanboi2/tasks/_result_proxy.py +39 -0
@@ 0,0 1,39 @@
from celery import states

from ..errors import deserialize_error
from ..models import deserialize_model


class ResultProxy(object):
    """A proxy class for :class:`celery.result.AsyncResult` that provide
    results serialization using :func:`fanboi2.errors.deserialize_error` and
    :func:`fanboi2.models.deserialize_model`.

    :param result: A result of :class:`celery.AsyncResult`.
    """

    def __init__(self, result):
        self._result = result
        self._object = None

    def deserialize(self, request):
        """Deserializing the result into Python object."""
        if self._object is None:
            obj, id_, *args = self._result.get()
            if obj == 'failure':
                class_ = deserialize_error(id_)
                if class_ is not None:
                    self._object = class_(*args)
            else:
                dbsession = request.find_service(name='db')
                class_ = deserialize_model(obj)
                if class_ is not None:
                    self._object = dbsession.query(class_).get(id_)
        return self._object

    def success(self):
        """Returns true if result was successfully processed."""
        return self._result.state == states.SUCCESS

    def __getattr__(self, name):
        return self._result.__getattribute__(name)

A fanboi2/tasks/post.py => fanboi2/tasks/post.py +49 -0
@@ 0,0 1,49 @@
import pyramid.scripting

from ..errors import StatusRejectedError
from ..interfaces import IFilterService, IPostCreateService
from ._base import celery, ModelTask


@celery.task(base=ModelTask, bind=True)
def add_post(
        self,
        topic_id,
        body,
        bumped,
        ip_address,
        payload={},
        _request=None,
        _registry=None):
    """Insert a post to a topic.

    :param self: A :class:`celery.Task` object.
    :param topic_id: The ID of a topic to add a post to.
    :param body: Content of the post as submitted by the user.
    :param bumped: A :type:`bool` whether to bump the topic.
    :param ip_address: An IP address of the poster.
    :param payload: A request payload containing request metadata.
    """
    with pyramid.scripting.prepare(
            request=_request,
            registry=_registry) as env:
        request = env['request']

        filter_svc = request.find_service(IFilterService)
        filter_result = filter_svc.evaluate(payload={
            'body': body,
            'ip_address': ip_address,
            **payload})
        if filter_result.rejected_by:
            return 'failure', "%s_rejected" % (filter_result.rejected_by,)

        post_create_svc = request.find_service(IPostCreateService)
        try:
            post = post_create_svc.create(
                topic_id,
                body,
                bumped,
                ip_address)
        except StatusRejectedError as e:
            return 'failure', e.name, e.status
        return 'post', post.id

A fanboi2/tasks/topic.py => fanboi2/tasks/topic.py +48 -0
@@ 0,0 1,48 @@
import pyramid.scripting

from ..errors import StatusRejectedError
from ..interfaces import IFilterService, ITopicCreateService
from ._base import celery, ModelTask


@celery.task(base=ModelTask, bind=True)
def add_topic(
        self,
        board_slug,
        title,
        body,
        ip_address,
        payload={},
        _request=None,
        _registry=None):
    """Insert a topic to the database.

    :param board_slug: The slug :type:`str` identifying a board.
    :param title: A :type:`str` topic title.
    :param body: A :type:`str` topic body.
    :param ip_address: An IP address of the topic creator.
    :param payload: A request payload containing request metadata.
    """
    with pyramid.scripting.prepare(
            request=_request,
            registry=_registry) as env:
        request = env['request']

        filter_svc = request.find_service(IFilterService)
        filter_result = filter_svc.evaluate(payload={
            'body': body,
            'ip_address': ip_address,
            **payload})
        if filter_result.rejected_by:
            return 'failure', "%s_rejected" % (filter_result.rejected_by,)

        topic_create_svc = request.find_service(ITopicCreateService)
        try:
            topic = topic_create_svc.create(
                board_slug,
                title,
                body,
                ip_address)
        except StatusRejectedError as e:
            return 'failure', e.name, e.status
        return 'topic', topic.id

M fanboi2/templates/api/_other.mako => fanboi2/templates/api/_other.mako +1 -2
@@ 126,7 126,7 @@
                                </ul>
                                <p>In case of post rejection, the following statuses are returned:</p>
                                <ul>
                                    <li><strong>spam_rejected</strong> — the post or topic has been identified as spam.</li>
                                    <li><strong>akismet_rejected</strong> — the post or topic has been identified as spam by Akismet.</li>
                                    <li><strong>dnsbl_rejected</strong> — the IP address is listed in one of DNSBL databases.</li>
                                    <li><strong>ban_rejected</strong> — the IP address is listed in the ban list.</li>
                                    <li><strong>proxy_rejected</strong> — the IP address has been identified as an open proxy or public VPN.</li>


@@ 149,4 149,3 @@
        </div>
    </div>
</div>


A fanboi2/templates/bad_request.mako => fanboi2/templates/bad_request.mako +18 -0
@@ 0,0 1,18 @@
<%namespace name='formatters' module='fanboi2.helpers.formatters' />
<%inherit file='partials/_layout.mako' />
<%def name='title()'>Bad Request</%def>
<header class="subheader">
    <div class="container">
        <h2 class="subheader-title">400 Bad Request</h2>
        <div class="subheader-body"><p>The request is invalid.</p></div>
    </div>
</header>
<div class="sheet">
    <div class="container">
        <h2 class="sheet-title">I will take the ring to Mordor.</h2>
        <div class="sheet-body">
            <p>Though I do not know the way.</p>
            <p><em>Please retry the previous operation.</em></p>
        </div>
    </div>
</div>
\ No newline at end of file

M fanboi2/templates/boards/_subheader.mako => fanboi2/templates/boards/_subheader.mako +1 -1
@@ 7,7 7,7 @@
            <ul class="actions">
                <li class="actions-item"><a class="button${' brand' if request.route_name == 'board' else ''} static" href="${request.route_path('board', board=board.slug)}">Recent topics</a></li>
                <li class="actions-item"><a class="button${' brand' if request.route_name == 'board_all' else ''} static" href="${request.route_path('board_all', board=board.slug)}">All topics</a></li>
                % if override.get('status', board.status) == 'open':
                % if board.status == 'open':
                    <li class="actions-item"><a class="button${' brand' if request.route_name == 'board_new' else ''} static" href="${request.route_path('board_new', board=board.slug)}">New topic</a></li>
                % endif
            </ul>

D fanboi2/templates/boards/error_ban.mako => fanboi2/templates/boards/error_ban.mako +0 -12
@@ 1,12 0,0 @@
<%include file='_subheader.mako' />
<%inherit file='../partials/_layout.mako' />
<%def name='title()'>New topic - ${board.title}</%def>
<div class="sheet">
    <div class="container">
        <h2 class="sheet-title">You cannot pass.</h2>
        <div class="sheet-body">
            <p>The dark fire will not avail you. Go back to the Shadow!</p>
            <p><em>Your IP address is being listed in the ban list.</em></p>
        </div>
    </div>
</div>
\ No newline at end of file

D fanboi2/templates/boards/error_dnsbl.mako => fanboi2/templates/boards/error_dnsbl.mako +0 -12
@@ 1,12 0,0 @@
<%include file='_subheader.mako' />
<%inherit file='../partials/_layout.mako' />
<%def name='title()'>New topic - ${board.title}</%def>
<div class="sheet">
    <div class="container">
        <h2 class="sheet-title">A strange game.</h2>
        <div class="sheet-body">
            <p>The only winning move is not to play. How about a nice game of chess?</p>
            <p><em>Your IP address is listed in DNSBL and therefore rejected.</em></p>
        </div>
    </div>
</div>

D fanboi2/templates/boards/error_proxy.mako => fanboi2/templates/boards/error_proxy.mako +0 -12
@@ 1,12 0,0 @@
<%include file='_subheader.mako' />
<%inherit file='../partials/_layout.mako' />
<%def name='title()'>New topic - ${board.title}</%def>
<div class="sheet">
    <div class="container">
        <h2 class="sheet-title">You look surprised to see me, again, Mr. Anderson.</h2>
        <div class="sheet-body">
            <p>That's the difference between us. I've been expecting you.</p>
            <p><em>Your IP address has been identified as an open proxy or public VPN.</em></p>
        </div>
    </div>
</div>
\ No newline at end of file

D fanboi2/templates/boards/error_rate.mako => fanboi2/templates/boards/error_rate.mako +0 -12
@@ 1,12 0,0 @@
<%include file='_subheader.mako' />
<%inherit file='../partials/_layout.mako' />
<%def name='title()'>New topic - ${board.title}</%def>
<div class="sheet">
    <div class="container">
        <h2 class="sheet-title">Nothing travels faster than light.</h2>
        <div class="sheet-body">
            <p>With the possible exception of bad news, which obeys its own special laws.</p>
            <p><em>You're posting too fast. Please wait <strong>${timeleft} seconds</strong> before trying again.</em></p>
        </div>
    </div>
</div>

D fanboi2/templates/boards/error_spam.mako => fanboi2/templates/boards/error_spam.mako +0 -12
@@ 1,12 0,0 @@
<%include file='_subheader.mako' />
<%inherit file='../partials/_layout.mako' />
<%def name='title()'>New topic - ${board.title}</%def>
<div class="sheet">
    <div class="container">
        <h2 class="sheet-title">I'm sorry, Dave. I'm afraid I can't do that.</h2>
        <div class="sheet-body">
            <p>This mission is too important for me to allow you to jeopardize it.</p>
            <p><em>Your topic has been identified as spam and therefore rejected.</em></p>
        </div>
    </div>
</div>

M fanboi2/templates/boards/new.mako => fanboi2/templates/boards/new.mako +1 -1
@@ 13,7 13,7 @@
    </div>
</div>
<form class="form" action="${request.route_path('board_new', board=board.slug)}" method="post">
    ${form.csrf_token}
    <input type="hidden" name="csrf_token" value="${get_csrf_token()}">
    <div class="container">
        <div class="form-item${' error' if form.title.errors else ''}">
            <label class="form-item-label" for="${form.title.id}">Topic</label>

R fanboi2/templates/boards/error_status.mako => fanboi2/templates/boards/new_error.mako +45 -13
@@ 3,24 3,56 @@
<%def name='title()'>New topic - ${board.title}</%def>
<div class="sheet">
    <div class="container">
    % if status == 'restricted':
        <h2 class="sheet-title">Is that your two cents worth, Worth?</h2>
    % if name == 'akismet_rejected':
        <h2 class="sheet-title">I'm sorry, Dave. I'm afraid I can't do that.</h2>
        <div class="sheet-body">
            <p>For what it’s worth…</p>
            <p><em>Board is currently disallow creation of new topic.</em></p>
            <p>This mission is too important for me to allow you to jeopardize it.</p>
            <p><em>Your topic has been identified as spam by Akismet and therefore rejected.</em></p>
        </div>
    % elif status == 'locked':
        <h2 class="sheet-title">Too much garbage in your face?</h2>
    % elif name == 'ban_rejected':
        <h2 class="sheet-title">You cannot pass.</h2>
        <div class="sheet-body">
            <p>There's plenty of space out in space!</p>
            <p><em>Board has been locked by moderator.</em></p>
            <p>The dark fire will not avail you. Go back to the Shadow!</p>
            <p><em>Your IP address is being listed in the ban list.</em></p>
        </div>
    % elif status == 'archived':
        <h2 class="sheet-title">Hasta la vista, baby.</h2>
    % elif name == 'dnsbl_rejected':
        <h2 class="sheet-title">A strange game.</h2>
        <div class="sheet-body">
            <p>You gotta listen to the way people talk.</p>
            <p><em>Board has been archived.</em></p>
            <p>The only winning move is not to play. How about a nice game of chess?</p>
            <p><em>Your IP address is listed in DNSBL and therefore rejected.</em></p>
        </div>
    % endif
    % elif name == 'proxy_rejected':
        <h2 class="sheet-title">You look surprised to see me, again, Mr. Anderson.</h2>
        <div class="sheet-body">
            <p>That's the difference between us. I've been expecting you.</p>
            <p><em>Your IP address has been identified as an open proxy or public VPN.</em></p>
        </div>
    % elif name == 'rate_limited':
        <h2 class="sheet-title">Nothing travels faster than light.</h2>
        <div class="sheet-body">
            <p>With the possible exception of bad news, which obeys its own special laws.</p>
            <p><em>You're posting too fast. Please wait <strong>${time_left} seconds</strong> before trying again.</em></p>
        </div>
    % elif name == 'status_rejected':
        % if status == 'restricted':
            <h2 class="sheet-title">Is that your two cents worth, Worth?</h2>
            <div class="sheet-body">
                <p>For what it’s worth…</p>
                <p><em>Board is currently disallow creation of new topic.</em></p>
            </div>
        % elif status == 'locked':
            <h2 class="sheet-title">Too much garbage in your face?</h2>
            <div class="sheet-body">
                <p>There's plenty of space out in space!</p>
                <p><em>Board has been locked by moderator.</em></p>
            </div>
        % elif status == 'archived':
            <h2 class="sheet-title">Hasta la vista, baby.</h2>
            <div class="sheet-body">
                <p>You gotta listen to the way people talk.</p>
                <p><em>Board has been archived.</em></p>
            </div>
        % endif
    % end
    </div>
</div>
\ No newline at end of file

M fanboi2/templates/partials/_layout.mako => fanboi2/templates/partials/_layout.mako +1 -1
@@ 58,7 58,7 @@ ${self.body()}
        </div>
        <ul class="footer-links">
            <li class="footer-links-item"><a href="${request.route_path('api_root')}">API documentation</a></li>
            <li class="footer-links-item"><a href="https://github.com/pxfs/fanboi2">Source code</a></li>
            <li class="footer-links-item"><a href="https://github.com/forloopend/fanboi2">Source code</a></li>
        </ul>
    </div>
</footer>

M fanboi2/templates/topics/_subheader.mako => fanboi2/templates/topics/_subheader.mako +1 -1
@@ 10,7 10,7 @@
            <ul class="actions">
                <li class="actions-item"><a class="button static" href="${request.route_path('board', board=board.slug)}">Back</a></li>
                <li class="actions-item"><a class="button brand static" href="${request.route_path('topic', board=board.slug, topic=topic.id)}">Show topic</a></li>
                % if topic.status == 'open' and override.get('status', board.status) in ('open', 'restricted'):
                % if topic.status == 'open' and board.status in ('open', 'restricted'):
                    <li class="actions-item"><a class="button green static" href="#reply">Reply</a></li>
                % endif
            </ul>

D fanboi2/templates/topics/error_ban.mako => fanboi2/templates/topics/error_ban.mako +0 -12
@@ 1,12 0,0 @@
<%include file='_subheader.mako' />
<%inherit file='../partials/_layout.mako' />
<%def name='title()'>${topic.title} - ${board.title}</%def>
<div class="sheet">
    <div class="container">
        <h2 class="sheet-title">Toto, I've a feeling we're not in Kansas any more.</h2>
        <div class="sheet-body">
            <p>We must be over the rainbow!</p>
            <p><em>Your IP address is being listed in the ban list.</em></p>
        </div>
    </div>
</div>
\ No newline at end of file

D fanboi2/templates/topics/error_dnsbl.mako => fanboi2/templates/topics/error_dnsbl.mako +0 -12
@@ 1,12 0,0 @@
<%include file='_subheader.mako' />
<%inherit file='../partials/_layout.mako' />
<%def name='title()'>${topic.title} - ${board.title}</%def>
<div class="sheet">
    <div class="container">
        <h2 class="sheet-title">A strange game.</h2>
        <div class="sheet-body">
            <p>The only winning move is not to play. How about a nice game of chess?</p>
            <p><em>Your IP address is listed in DNSBL and therefore rejected.</em></p>
        </div>
    </div>
</div>
\ No newline at end of file

D fanboi2/templates/topics/error_proxy.mako => fanboi2/templates/topics/error_proxy.mako +0 -12
@@ 1,12 0,0 @@
<%include file='_subheader.mako' />
<%inherit file='../partials/_layout.mako' />
<%def name='title()'>${topic.title} - ${board.title}</%def>
<div class="sheet">
    <div class="container">
        <h2 class="sheet-title">You look surprised to see me, again, Mr. Anderson.</h2>
        <div class="sheet-body">
            <p>That's the difference between us. I've been expecting you.</p>
            <p><em>Your IP address has been identified as an open proxy or public VPN.</em></p>
        </div>
    </div>
</div>
\ No newline at end of file

D fanboi2/templates/topics/error_rate.mako => fanboi2/templates/topics/error_rate.mako +0 -12
@@ 1,12 0,0 @@
<%include file='_subheader.mako' />
<%inherit file='../partials/_layout.mako' />
<%def name='title()'>${topic.title} - ${board.title}</%def>
<div class="sheet">
    <div class="container">
        <h2 class="sheet-title">Nothing travels faster than light.</h2>
        <div class="sheet-body">
            <p>With the possible exception of bad news, which obeys its own special laws.</p>
            <p><em>You're posting too fast. Please wait <strong>${timeleft} seconds</strong> before trying again.</em></p>
        </div>
    </div>
</div>

D fanboi2/templates/topics/error_spam.mako => fanboi2/templates/topics/error_spam.mako +0 -12
@@ 1,12 0,0 @@
<%include file='_subheader.mako' />
<%inherit file='../partials/_layout.mako' />
<%def name='title()'>${topic.title} - ${board.title}</%def>
<div class="sheet">
    <div class="container">
        <h2 class="sheet-title">I'm sorry, Dave. I'm afraid I can't do that.</h2>
        <div class="sheet-body">
            <p>This mission is too important for me to allow you to jeopardize it.</p>
            <p><em>Your post has been identified as spam and therefore rejected.</em></p>
        </div>
    </div>
</div>

M fanboi2/templates/topics/show.mako => fanboi2/templates/topics/show.mako +3 -3
@@ 43,7 43,7 @@
            </div>
        </div>
    </div>
% elif override.get('status', board.status) == 'locked':
% elif board.status == 'locked':
    <div class="sheet">
        <div class="container">
            <h2 class="sheet-title">Board locked</h2>


@@ 53,7 53,7 @@
            </div>
        </div>
    </div>
% elif override.get('status', board.status) == 'archived':
% elif board.status == 'archived':
    <div class="sheet">
        <div class="container">
            <h2 class="sheet-title">Board archived</h2>


@@ 65,7 65,7 @@
    </div>
% else:
    <form class="form" id="reply" action="${request.route_path('topic', board=board.slug, topic=topic.id)}" method="post" data-topic-inline-reply="true">
        ${form.csrf_token}
        <input type="hidden" name="csrf_token" value="${get_csrf_token()}">
        <div class="container">
            <div class="form-item${' error' if form.body.errors else ''}">
                <label class="form-item-label" for="${form.body.id}">Reply</label>

R fanboi2/templates/topics/error_status.mako => fanboi2/templates/topics/show_error.mako +33 -1
@@ 3,6 3,37 @@
<%def name='title()'>${topic.title} - ${board.title}</%def>
<div class="sheet">
    <div class="container">
    % if name == 'akismet_rejected':
        <h2 class="sheet-title">I'm sorry, Dave. I'm afraid I can't do that.</h2>
        <div class="sheet-body">
            <p>This mission is too important for me to allow you to jeopardize it.</p>
            <p><em>Your post has been identified as spam and therefore rejected.</em></p>
        </div>
    % elif name == 'ban_rejected':
        <h2 class="sheet-title">Toto, I've a feeling we're not in Kansas any more.</h2>
        <div class="sheet-body">
            <p>We must be over the rainbow!</p>
            <p><em>Your IP address is being listed in the ban list.</em></p>
        </div>
    % elif name == 'dnsbl_rejected':
        <h2 class="sheet-title">A strange game.</h2>
        <div class="sheet-body">
            <p>The only winning move is not to play. How about a nice game of chess?</p>
            <p><em>Your IP address is listed in DNSBL and therefore rejected.</em></p>
        </div>
    % elif name == 'proxy_rejected':
        <h2 class="sheet-title">You look surprised to see me, again, Mr. Anderson.</h2>
        <div class="sheet-body">
            <p>That's the difference between us. I've been expecting you.</p>
            <p><em>Your IP address has been identified as an open proxy or public VPN.</em></p>
        </div>
    % elif name == 'rate_limited':
        <h2 class="sheet-title">Nothing travels faster than light.</h2>
        <div class="sheet-body">
            <p>With the possible exception of bad news, which obeys its own special laws.</p>
            <p><em>You're posting too fast. Please wait <strong>${time_left} seconds</strong> before trying again.</em></p>
        </div>
    % elif name == 'status_rejected':
        % if status == 'locked':
            <h2 class="sheet-title">Don't panic.</h2>
            <div class="sheet-body">


@@ 24,5 55,6 @@
                % endif
            </div>
        % endif
    % end
    </div>
</div>
</div>
\ No newline at end of file

M fanboi2/tests/__init__.py => fanboi2/tests/__init__.py +59 -215
@@ 1,28 1,40 @@
import logging
import os
import transaction
import unittest
from pyramid import testing
from sqlalchemy import create_engine
from sqlalchemy.orm import Query
from webob.multidict import MultiDict
from fanboi2.models import DBSession, Base, redis_conn

from sqlalchemy.engine import create_engine
from sqlalchemy.orm import sessionmaker

logging.basicConfig()
logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)
from ..models import make_history_event


DATABASE_URI = os.environ.get(
DATABASE_URL = os.environ.get(
    'POSTGRESQL_TEST_DATABASE',
    'postgresql://fanboi2:@localhost:5432/fanboi2_test')

engine = create_engine(DATABASE_URL)
dbmaker = sessionmaker()
make_history_event(dbmaker)

class DummyRedis(object):

    @classmethod
    def from_url(cls, *args, **kwargs):
        return cls()
def make_cache_region(store=None):
    from dogpile.cache import make_region
    if store is None:
        store = {}
    return make_region().configure(
        'dogpile.cache.memory',
        arguments={'cache_dict': store})


def mock_service(request, mappings={}):
    def _find_service(iface=None, name=None):
        for l in (iface, name):
            if l in mappings:
                return mappings[l]

    request.find_service = _find_service
    return request


class DummyRedis(object):

    def __init__(self):
        self._store = {}


@@ 51,166 63,12 @@ class DummyRedis(object):
    def ttl(self, key):
        return self._expire.get(key, 0)

    def ping(self):
        return True


class _ModelInstanceSetup(object):

    def _newBoard(self, **kwargs):
        from fanboi2.models import Board
        return Board(**kwargs)

    def _newTopic(self, **kwargs):
        from fanboi2.models import Topic
        return Topic(**kwargs)

    def _newTopicMeta(self, **kwargs):
        from fanboi2.models import TopicMeta
        if not kwargs.get('post_count'):
            kwargs['post_count'] = 0
        return TopicMeta(**kwargs)

    def _newPost(self, **kwargs):
        from fanboi2.models import Post
        if not kwargs.get('ip_address'):
            kwargs['ip_address'] = '0.0.0.0'
        return Post(**kwargs)

    def _newPage(self, **kwargs):
        from fanboi2.models import Page
        return Page(**kwargs)

    def _newRule(self, **kwargs):
        from fanboi2.models import Rule
        return Rule(**kwargs)

    def _newRuleBan(self, **kwargs):
        from fanboi2.models import RuleBan
        return RuleBan(**kwargs)

    def _newRuleOverride(self, **kwargs):
        from fanboi2.models import RuleOverride
        return RuleOverride(**kwargs)

    def _makeBoard(self, **kwargs):
        board = self._newBoard(**kwargs)
        DBSession.add(board)
        DBSession.flush()
        return board

    def _makeTopic(self, **kwargs):
        topic = self._newTopic(**kwargs)
        DBSession.add(topic)
        DBSession.flush()
        return topic

    def _makeTopicMeta(self, **kwargs):
        topic_meta = self._newTopicMeta(**kwargs)
        DBSession.add(topic_meta)
        DBSession.flush()
        return topic_meta

    def _makePost(self, **kwargs):
        post = self._newPost(**kwargs)
        DBSession.add(post)
        DBSession.flush()
        return post

    def _makePage(self, **kwargs):
        page = self._newPage(**kwargs)
        DBSession.add(page)
        DBSession.flush()
        return page

    def _makeRule(self, **kwargs):
        rule = self._newRule(**kwargs)
        DBSession.add(rule)
        DBSession.flush()
        return rule

    def _makeRuleBan(self, **kwargs):
        rule_ban = self._newRuleBan(**kwargs)
        DBSession.add(rule_ban)
        DBSession.flush()
        return rule_ban

    def _makeRuleOverride(self, **kwargs):
        rule_override = self._newRuleOverride(**kwargs)
        DBSession.add(rule_override)
        DBSession.flush()
        return rule_override

class ModelMixin(_ModelInstanceSetup, unittest.TestCase):

    @classmethod
    def tearDownClass(cls):
        super(ModelMixin, cls).tearDownClass()
        Base.metadata.bind = None
        DBSession.remove()

    @classmethod
    def setUpClass(cls):
        super(ModelMixin, cls).setUpClass()
        engine = create_engine(DATABASE_URI)
        DBSession.configure(bind=engine)
        Base.metadata.bind = engine

    def setUp(self):
        super(ModelMixin, self).setUp()
        redis_conn._redis = DummyRedis()
        Base.metadata.drop_all()
        Base.metadata.create_all()
        transaction.begin()

    def tearDown(self):
        super(ModelMixin, self).tearDown()
        transaction.abort()
        Base.metadata.drop_all()
        redis_conn._redis = None

    def assertSAEqual(self, first, second, msg=None):
        if isinstance(first, Query):
            return self.assertListEqual(list(first), second, msg)
        else:
            return self.assertEqual(first, second, msg)


class RegistryMixin(unittest.TestCase):

    def tearDown(self):
        super(RegistryMixin, self).tearDown()
        testing.tearDown()

    def _makeConfig(self, request=None, registry=None):
        return testing.setUp(
            request=request,
            registry=registry)

    def _makeRequest(self, **kw):
        """:rtype: pyramid.request.Request"""
        request = testing.DummyRequest(**kw)
        request.user_agent = kw.get('user_agent', 'Mock/1.0')
        request.remote_addr = kw.get('remote_addr', '127.0.0.1')
        request.referrer = kw.get('referrer')
        request.content_type = 'application/x-www-form-urlencoded'
        request.params = MultiDict(kw.get('params') or {})
        return request

    def _makeRegistry(self, **kw):
        """:rtype: pyramid.registry.Registry"""
        from pyramid.registry import Registry
        registry = Registry()
        registry.settings = {
            'app.timezone': 'Asia/Bangkok',
            'app.secret': 'demo',
        }
        registry.settings.update(kw)
        return registry
    def _reset(self):
        self._store = {}
        self._expire = {}


class DummyAsyncResult(object):

    def __init__(self, id_, status, result=None):
        self._id = id_
        self._status = status


@@ 233,54 91,40 @@ class DummyAsyncResult(object):
        return self._result


class TaskMixin(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        from fanboi2.tasks import celery
        super(TaskMixin, cls).setUpClass()
        celery.config_from_object({'CELERY_ALWAYS_EAGER': True})
class ModelTransactionEngineMixin(object):

    @classmethod
    def tearDownClass(cls):
        from fanboi2.tasks import celery
        super(TaskMixin, cls).tearDownClass()
        celery.config_from_object({'CELERY_ALWAYS_EAGER': False})


class ViewMixin(ModelMixin, RegistryMixin, unittest.TestCase):
    def setUp(self):
        super(ModelTransactionEngineMixin, self).setUp()
        self.connection = engine.connect()
        self.tx = self.connection.begin()

    def _make_csrf(self, request):
        import hmac
        import os
        from hashlib import sha1
        request.session['csrf'] = sha1(os.urandom(64)).hexdigest()
        request.params['csrf_token'] = hmac.new(
            bytes(request.registry.settings['app.secret'].encode('utf8')),
            bytes(request.session['csrf'].encode('utf8')),
            digestmod=sha1,
        ).hexdigest()
        return request
    def tearDown(self):
        super(ModelTransactionEngineMixin, self).tearDown()
        self.tx.rollback()
        self.connection.close()

    def _POST(self, data=None):
        request = self._makeRequest(params=data)
        request.method = 'POST'
        return request

    def _GET(self, data=None):
        request = self._makeRequest(params=data)
        return request
class ModelSessionMixin(ModelTransactionEngineMixin, object):

    def _json_POST(self, data=None):
        request = self._makeRequest()
        request.content_type = 'application/json'
        request.json_body = data
        return request
    def setUp(self):
        super(ModelSessionMixin, self).setUp()
        from sqlalchemy import event
        from ..models import Base
        self.dbsession = dbmaker(bind=self.connection)
        Base.metadata.bind = self.connection
        Base.metadata.create_all()
        self.dbsession.begin_nested()

        @event.listens_for(self.dbsession, "after_transaction_end")
        def restart_savepoint(dbsession, transaction):
            if transaction.nested and not transaction._parent.nested:
                dbsession.expire_all()
                dbsession.begin_nested()

class CacheMixin(unittest.TestCase):
    def tearDown(self):
        super(ModelSessionMixin, self).tearDown()
        self.dbsession.close()

    def _getRegion(self, store=None):
        from dogpile.cache import make_region
        return make_region().configure('dogpile.cache.memory', arguments={
            'cache_dict': store if store is not None else {}})
    def _make(self, model_obj):
        self.dbsession.add(model_obj)
        return model_obj

M fanboi2/tests/test_app.py => fanboi2/tests/test_app.py +88 -306
@@ 2,70 2,31 @@ import unittest
from pyramid import testing


class TestRemoteAddr(unittest.TestCase):

    def _getFunction(self):
        from fanboi2 import remote_addr
        return remote_addr

    def _makeRequest(self, ipaddr, forwarded=None):
        request = testing.DummyRequest()
        request.environ = {'REMOTE_ADDR': ipaddr}
        if forwarded:
            request.environ['HTTP_X_FORWARDED_FOR'] = forwarded
        return request

    def test_remote_addr(self):
        request = self._makeRequest("171.100.10.1")
        self.assertEqual(self._getFunction()(request), "171.100.10.1")

    def test_private_fallback(self):
        request = self._makeRequest("10.0.1.1", "171.100.10.1, 127.0.0.1")
        self.assertEqual(self._getFunction()(request), "171.100.10.1")

    def test_loopback_fallback(self):
        request = self._makeRequest("127.0.0.1", "171.100.10.1")
        self.assertEqual(self._getFunction()(request), "171.100.10.1")

    def test_private_without_fallback(self):
        request = self._makeRequest("10.0.1.1")
        self.assertEqual(self._getFunction()(request), "10.0.1.1")

    def test_loopback_without_fallback(self):
        request = self._makeRequest("127.0.0.1")
        self.assertEqual(self._getFunction()(request), "127.0.0.1")

    def test_remote_fallback(self):
        request = self._makeRequest("171.100.10.1", "8.8.8.8")
        self.assertEqual(self._getFunction()(request), "171.100.10.1")


class TestRouteName(unittest.TestCase):

    def _getFunction(self):
        from fanboi2 import route_name
        return route_name
    def setUp(self):
        self.config = testing.setUp()
        self.request = testing.DummyRequest()
        self.request.registry = self.config.registry

    def _makeRequest(self, name):
        request = testing.DummyRequest()
    def tearDown(self):
        testing.tearDown()

        class MockMatchedRoute(object):
    def _get_function(self):
        from .. import route_name
        return route_name

    def test_route_name(self):
        class _MockMatchedRule(object):
            def __init__(self, name):
                self.name = name

        request.matched_route = None
        if name is not None:
            request.matched_route = MockMatchedRoute(name)
        return request

    def test_route_name(self):
        request = self._makeRequest("foobar")
        self.assertEqual(self._getFunction()(request), "foobar")
        self.request.matched_route = _MockMatchedRule('foobar')
        self.assertEqual(self._get_function()(self.request), 'foobar')

    def test_route_name_not_exists(self):
        request = self._makeRequest(None)
        self.assertEqual(self._getFunction()(request), None)
        self.request.matched_route = None
        self.assertEqual(self._get_function()(self.request), None)


class DummyStaticURLInfo:


@@ 81,15 42,9 @@ class DummyStaticURLInfo:
class TestTaggedStaticUrl(unittest.TestCase):

    def setUp(self):
        self.config = testing.setUp()

    def tearDown(self):
        testing.tearDown()

    def _makeRequest(self):
        from pyramid.url import URLMethodsMixin

        class Request(URLMethodsMixin):
        class _MockRequest(URLMethodsMixin):
            application_url = 'http://example.com:5432'
            script_name = ''



@@ 97,15 52,18 @@ class TestTaggedStaticUrl(unittest.TestCase):
                self.environ = environ