~shulhan/haminer

e31c6a2b3dac9f06bd15b4834a28b545999c7dab — Shulhan 1 year, 7 months ago 4b59013
_doc: add some introduction in the README and index

Show what the HTTP log looks like, how it stored, and how it can be used.
2 files changed, 95 insertions(+), 4 deletions(-)

M README
M _doc/index.adoc
M README => README +46 -1
@@ 7,8 7,53 @@

Library and program to parse and forward HAProxy logs.

Supported forwarder: Influxdb, QuestDB.
The HTTP logs is HTTP request that received by HAProxy frontend and forwarded
to backend.
In default format, it looks like these (split into multi lines, for
readability):

----
<158>Sep  4 17:08:47 haproxy[109530]: 185.83.144.103:46376
  [04/Sep/2022:17:08:47.264] www~ be_kilabit/kilabit-0.0/0/1/2/3 200 89 - -
  ---- 5/5/0/0/0 0/0 "GET / HTTP/1.1"
----

See
https://www.haproxy.com/documentation/hapee/1-8r1/onepage/#8.2.3[HTTP log format documentation]
for more information.

Currently, there are two supported database where haminer can forward the
parsed log: influxdb and questdb.
Haminer support Influxdb v1 and v2.

----
 +---------+  UDP  +---------+      +-----------+
 | HAProxy |------>| haminer |----->| influxdb  |
 +---------+       +---------+      | / questdb |
                                    +-----------+
----

In Influxdb, the log are stored as measurement called `haproxy`.
In Questdb, the log are stored as table called `haproxy`.

The following fields are stored as tags (in Influxdb) or symbol (in Questdb):
host, server, backend, frontend, http_method, http_url, http_query,
http_proto, http_status, term_state, client_ip, client_port.

And the following fields are stored as fields (in Influxdb) or values (in
Questdb): time_req, time_wait, time_connect, time_rsp, time_all,
conn_active, conn_frontend, conn_backend, conn_server, conn_retries,
queue_server, queue_backend, bytes_read.

Once the log has been accumulated, we can query the data.
For example, with Questdb we can count each visited URL using the following
query,

----
select backend, http_url, count(*) as visit from 'haproxy'
group by backend, http_url
order by visit desc;
----

==  Installation


M _doc/index.adoc => _doc/index.adoc +49 -3
@@ 3,10 3,56 @@
:sectanchors:
:sectlinks:

Welcome to haminer.
haminer is a library and program to parse and forward HAProxy HTTP logs.

The HTTP logs is HTTP request that received by HAProxy frontend and forwarded
to backend.
In default format, it looks like these (split into multi lines, for
readability):

----
<158>Sep  4 17:08:47 haproxy[109530]: 185.83.144.103:46376
  [04/Sep/2022:17:08:47.264] www~ be_kilabit/kilabit-0.0/0/1/2/3 200 89 - -
  ---- 5/5/0/0/0 0/0 "GET / HTTP/1.1"
----

See
https://www.haproxy.com/documentation/hapee/1-8r1/onepage/#8.2.3[HTTP log format documentation]
for more information.

Currently, there are two supported database where haminer can forward the
parsed log: influxdb and questdb.
Haminer support Influxdb v1 and v2.

----
 +---------+  UDP  +---------+      +-----------+
 | HAProxy |------>| haminer |----->| influxdb  |
 +---------+       +---------+      | / questdb |
                                    +-----------+
----

In Influxdb, the log are stored as measurement called `haproxy`.
In Questdb, the log are stored as table called `haproxy`.

The following fields are stored as tags (in Influxdb) or symbol (in Questdb):
host, server, backend, frontend, http_method, http_url, http_query,
http_proto, http_status, term_state, client_ip, client_port.

And the following fields are stored as fields (in Influxdb) or values (in
Questdb): time_req, time_wait, time_connect, time_rsp, time_all,
conn_active, conn_frontend, conn_backend, conn_server, conn_retries,
queue_server, queue_backend, bytes_read.

Once the log has been accumulated, we can query the data.
For example, with Questdb we can count each visited URL using the following
query,

----
select backend, http_url, count(*) as visit from 'haproxy'
group by backend, http_url
order by visit desc;
----

haminer is a library and a program to write static web server with embedded
files using AsciiDoc markup format.

==  Documentation