~n0mn0m/airflow-docker

973f449a6b64ccc96f3d1f9cdcbdfa49b5707323 — Alex Hagerman 1 year, 5 months ago fe8dd67
Documentation updates and docker link location updates.
4 files changed, 20 insertions(+), 22 deletions(-)

M Dockerfile
M LICENSE
M README.md
M config/airflow.cfg
M Dockerfile => Dockerfile +3 -3
@@ 1,8 1,8 @@
# VERSION 1.10.1
# AUTHOR: Matthieu "Puckel_" Roisil
# AUTHOR: Alex Hagerman
# DESCRIPTION: Basic Airflow container
# BUILD: docker build --rm -t puckel/docker-airflow .
# SOURCE: https://github.com/puckel/docker-airflow
# BUILD: docker build --rm -t alexhagerman/docker-airflow .
# SOURCE: https://github.com/alexhagerman/docker-airflow

FROM python:3.6-slim
LABEL maintainer="Alex Hagerman"

M LICENSE => LICENSE +1 -1
@@ 186,7 186,7 @@
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

   Copyright 2017 Matthieu "Puckel_" Roisil
   Copyright 2018 Matthieu "Puckel_" Roisil and Alex Hagerman

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.

M README.md => README.md +15 -17
@@ 1,12 1,10 @@
# docker-airflow
[![CircleCI](https://circleci.com/gh/puckel/docker-airflow/tree/master.svg?style=svg)](https://circleci.com/gh/puckel/docker-airflow/tree/master)
[![Docker Build Status](https://img.shields.io/docker/build/puckel/docker-airflow.svg)]()

[![Docker Hub](https://img.shields.io/badge/docker-ready-blue.svg)](https://hub.docker.com/r/puckel/docker-airflow/)
[![Docker Pulls](https://img.shields.io/docker/pulls/puckel/docker-airflow.svg)]()
[![Docker Stars](https://img.shields.io/docker/stars/puckel/docker-airflow.svg)]()
[![Docker Hub](https://img.shields.io/badge/docker-ready-blue.svg)](https://hub.docker.com/r/alexhagerman/docker-airflow/)
[![Docker Pulls](https://img.shields.io/docker/pulls/alexhagerman/docker-airflow.svg)]()
[![Docker Stars](https://img.shields.io/docker/stars/alexhagerman/docker-airflow.svg)]()

This repository contains **Dockerfile** of [apache-airflow](https://github.com/apache/incubator-airflow) for [Docker](https://www.docker.com/)'s [automated build](https://registry.hub.docker.com/u/puckel/docker-airflow/) published to the public [Docker Hub Registry](https://registry.hub.docker.com/).
This repository contains **Dockerfile** of [apache-airflow](https://github.com/apache/incubator-airflow) for [Docker](https://www.docker.com/)'s [automated build](https://registry.hub.docker.com/u/alexhagerman/docker-airflow/) published to the public [Docker Hub Registry](https://registry.hub.docker.com/).

## Informations



@@ 19,26 17,26 @@ This repository contains **Dockerfile** of [apache-airflow](https://github.com/a

Pull the image from the Docker repository.

    docker pull puckel/docker-airflow
    docker pull alexhagerman/docker-airflow

## Build

Optionally install [Extra Airflow Packages](https://airflow.incubator.apache.org/installation.html#extra-package) and/or python dependencies at build time :

    docker build --rm --build-arg AIRFLOW_DEPS="datadog,dask" -t puckel/docker-airflow .
    docker build --rm --build-arg PYTHON_DEPS="flask_oauthlib>=0.9" -t puckel/docker-airflow .
    docker build --rm --build-arg AIRFLOW_DEPS="datadog,dask" -t alexhagerman/docker-airflow .
    docker build --rm --build-arg PYTHON_DEPS="flask_oauthlib>=0.9" -t alexhagerman/docker-airflow .

or combined

    docker build --rm --build-arg AIRFLOW_DEPS="datadog,dask" --build-arg PYTHON_DEPS="flask_oauthlib>=0.9" -t puckel/docker-airflow .
    docker build --rm --build-arg AIRFLOW_DEPS="datadog,dask" --build-arg PYTHON_DEPS="flask_oauthlib>=0.9" -t alexhagerman/docker-airflow .

Don't forget to update the airflow images in the docker-compose files to puckel/docker-airflow:latest.
Don't forget to update the airflow images in the docker-compose files to alexhagerman/docker-airflow:latest.

## Usage

By default, docker-airflow runs Airflow with **SequentialExecutor** :

    docker run -d -p 8080:8080 puckel/docker-airflow webserver
    docker run -d -p 8080:8080 alexhagerman/docker-airflow webserver

If you want to run another executor, use the other docker-compose.yml files provided in this repository.



@@ 54,7 52,7 @@ NB : If you want to have DAGs example loaded (default=False), you've to set the 

`LOAD_EX=n`

    docker run -d -p 8080:8080 -e LOAD_EX=y puckel/docker-airflow
    docker run -d -p 8080:8080 -e LOAD_EX=y alexhagerman/docker-airflow

If you want to use Ad hoc query, make sure you've configured connections:
Go to Admin -> Connections and Edit "postgres_default" set this values (equivalent to values in airflow.cfg/docker-compose*.yml) :


@@ 65,7 63,7 @@ Go to Admin -> Connections and Edit "postgres_default" set this values (equivale

For encrypted connection passwords (in Local or Celery Executor), you must have the same fernet_key. By default docker-airflow generates the fernet_key at startup, you have to set an environment variable in the docker-compose (ie: docker-compose-LocalExecutor.yml) file to set the same key accross containers. To generate a fernet_key :

    docker run puckel/docker-airflow python -c "from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)"
    docker run alexhagerman/docker-airflow python -c "from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)"

## Configurating Airflow



@@ 111,7 109,7 @@ This can be used to scale to a multi node setup using docker swarm.

If you want to run other airflow sub-commands, such as `list_dags` or `clear` you can do so like this:

    docker run --rm -ti puckel/docker-airflow airflow list_dags
    docker run --rm -ti alexhagerman/docker-airflow airflow list_dags

or with your docker-compose set up like this:



@@ 119,8 117,8 @@ or with your docker-compose set up like this:

You can also use this to run a bash shell or any other command in the same environment that airflow would be run in:

    docker run --rm -ti puckel/docker-airflow bash
    docker run --rm -ti puckel/docker-airflow ipython
    docker run --rm -ti alexhagerman/docker-airflow bash
    docker run --rm -ti alexhagerman/docker-airflow ipython

# Wanna help?


M config/airflow.cfg => config/airflow.cfg +1 -1
@@ 500,7 500,7 @@ authenticate = False
# Optional Docker Image to run on slave before running the command
# This image should be accessible from mesos slave i.e mesos slave
# should be able to pull this docker image before executing the command.
# docker_image_slave = puckel/docker-airflow
# docker_image_slave = alexhagerman/docker-airflow

[kerberos]
ccache = /tmp/airflow_krb5_ccache