Art crawler
f05026bd — Louis Solofrizzo 4 years ago
web: Skip empty keys for rendering
099618c3 — Louis Solofrizzo 4 years ago
Tree: Add go.mod & go.sum for more recent golang versions
e740b731 — Louis Solofrizzo 4 years ago
README: Add help for binary and packaged installation


browse  log 



You can also use your local clone with git send-email.



Art crawler - Live Version

#Table of Contents

#About the project

This collections of binaries helps to crawl art around the internet, mainly from subreddits. There are stored in S3, where you can then sort them, and expose them in any way you want.

#Getting Started


For the worker, you'll need two libraries, which you can install via pip3:

  • imgurpython
  • gfycat

In order to sort the images, two external programs are needed:

  • feh
  • vlc

Please refer to your distro package manager on how to install these two.

You will need the following API keys:

  • reddit
  • imgur
  • gfycat
  • S3

Please refer to the documentation on each of those services on how to get a key. You are not required to use Amazon's S3, any provider that is S3 compatible will work too.



Archlinux users can install the software through the kart-git AUR package


You can directly download the binaires here


In order to compile, you will need the following programs:

  • golang
  • cmake

Clone the repository:

git clone https://git.mobley.ne02ptzero.me/~louis/kart

cd & create the build directory:

cd kart && mkdir -p build && cd build && cmake ..

Compile everything:




One configuration file is used by all the services:

  id: XXX
  secret: XXX
  user_agent: linux:XXX:0.0.1 (by /u/User)
  username: User
  password: Password
    - pixelart
    - VoxelArt

  id: XXX
  secret: XXX

  id: XXX
  secret: XXX

  endpoint: s3.fr-par.scw.cloud
  region: fr-par
  id: XXX
  secret: XXX
  bucket: my-bucket
  prefix: non-sorted/
  good_prefix: good/

  directory: /tmp/
  database: /var/db/kart.db.sqlite

  directory: /usr/local/sbin/

  port: 8080
  views: /usr/share/kart/views
  static: /usr/share/kart/static
  database: /var/db/kart-web.db.sqlite
  desc: Some description


The worker is programs that crawl the subreddits, indefinetely. You can run it with:

kart-worker -conf /etc/kart-conf.yml

It will get the posts of subreddits, look for images, gif or videos, and then upload them in S3, if the images are new.


The sorter is the progam meant to be run by a human in order to sort the images from the worker.

kart-sorter -conf /etc/kart-conf.yml

It will launch a 'sort session' where the user is asked to say "Yes or No" to each images the worker has crawled. If the user wants them, the content is moved to another prefix in the S3 bucket, in order to be consumed.


The web binary is the web server that I use to display the images. It is launched with:

kart-web -conf /etc/kart-conf.yml

And it is meant to have a reverse proxy in front of it. Here's the one I use:

server {
    server_name  kart.ne02ptzero.me;

    location / {
        proxy_pass   ;

    location /cdn {
        rewrite                /cdn/(.*) /good/$1  break;
        proxy_pass             https://art.s3.fr-par.scw.cloud/;


If you want to contribute, please send your patches to ~louis/kart@lists.mobley.ne02ptzero.me You can look at git-send-email.io if you need to setup your mail client.

#Bug Report

Please report your bugs at todo/~louis/kart You can also send an email to ~louis/kart@todo.mobley.ne02ptzero.me