~rumpelsepp/homepage

ref: 2a0829d7d5c82198e590e856efc864471be97a18 homepage/_posts/2016-08-03-use-ssh-blacklists.adoc -rw-r--r-- 9.1 KiB
2a0829d7Stefan Tatschner add degoogle | A huge list of alternatives to Google products. Privacy tips, tricks, and links. 5 months ago
                                                                                
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
= Use SSH Blacklists on FreeBSD

NOTE: openbl has been discontinued. So, I leave this article for the archive...

Since I am a very lazy person, I am quoting the introduction of openbl.org:

[quote, openbl.org]
The OpenBL.org project (formerly known as the SSH blacklist) is about
detecting, logging and reporting various types of internet abuse. Currently our
hosts monitor ports 21 (FTP), 22 (SSH), 23 (TELNET), 25 (SMTP), 110 (POP3), 143
(IMAP), 587 (Submission), 993 (IMAPS) and 995 (POP3S) for bruteforce login
attacks as well as scans on ports 80 (HTTP) and 443 (HTTPS) for vulnerable
installations of phpMyAdmin and other web applications.

The traditional method to protect a server from SSH bruteforce attacks is `fail2ban`;
I hate `fail2ban`:

* It is a very ugly (they call it legacy) Python script which runs as root; 24/7.
* It parses logfiles in order to derive and apply firewall rules.
* It eats ressources, as they were burgers.
* It is ugly to setup, due to its legacyness
* ...

Since I run my own mailserver (under FreeBSD of course), I am familiar with so called
public blacklists. That is some list, were admins (= some script) report abusers
automatically. These lists are updated frequently and can be used by poor guys
like me, who want to benefit from the gathered data (= IP addresses of abusers).
The question was, are there any of those blacklists especially for SSH brute-forcers?

There are. I have found two which are usable:

* `https://www.openbl.org/lists/base_all.txt` *Update: This link is dead now!!*
* https://www.spamhaus.org/drop/edrop.txt

The task is, to add these IP addresses to a firewall block rule and update it, let's
say, once a day. Since I failed in verifying ip addresses in POSIX shell, I wrote
a little, sweet and tiny (= ugly) Python script, which downloads the ip list
asynchronously (just because I can!). It concatenates the two lists together,
verifies each ip address to be valid, and adds them to my `ipfw` firewall rules.

IMPORTANT: It is very, _very_, *very* important to verify, that the downloaded strings
           are actual ip addresses. Imagine you download shellcode, e.g. `; rm -rf /*`,
           which is being executed on your server...

== Python Script

Enough bullshitting now. Let's dive into the code... Here is the script, which is
explained using the really awesome `asciidoctor` features.

[source, python]
----
#!/usr/bin/env python3

import argparse
import asyncio
import logging
import shlex
from subprocess import run
from ipaddress import ip_address, ip_network
import aiohttp  # <1>


DEFAULT_BLs = (  # <2>
    "https://www.openbl.org/lists/base_all.txt",
    "https://www.spamhaus.org/drop/edrop.txt",
)
FIREWALL = 'ipfw'  # <3>
IPFW_TABLE = '5'


async def ips_fetch(session, url):  # <4>
    logging.info('Reading BL: {}'.format(url))
    async with session.get(url) as r:
        if r.status != 200:
            return False
        return await r.text()


async def bls_fetch(bls, session):
    results = await asyncio.gather(
        *[ips_fetch(session, bl) for bl in bls]
    )
    return results


def ip_verify(addr):  # <5>
    try:
        return ip_address(addr)
    except ValueError:
        try:
            return ip_network(addr)
        except ValueError:
            return False


def fw_ip_add(ip, args):  # <6>
    if args.f == 'ipfw':
        run(['ipfw', '-qf', 'table', '5', 'add', shlex.quote(str(ip))])
    elif args.f == 'dummy':
        pass
    else:
        print('Firewall not supported')
        exit(1)


def fw_prepare(args):  # <7>
    if args.f == 'ipfw':
        run(['ipfw', '-qf', 'add', 'deny', 'ip', 'from', 'table(5)', 'to', 'me'])
        run(['ipfw', '-qf', 'table', shlex.quote(str(args.ipfw_table)), 'flush'])
    elif args.f == 'dummy':
        pass
    else:
        print('Firewall not supported')
        exit(1)


def logging_init(loglevel):  # <8>
    # From python docs. No magic stackoverflow involved. :)
    # https://docs.python.org/3/howto/logging.html#logging-to-a-file
    numeric_level = getattr(logging, loglevel.upper(), None)
    if not isinstance(numeric_level, int):
        print('Invalid log level: "{}"'.format(loglevel))
        exit(1)

    logging.basicConfig(
        format='%(asctime)s %(levelname)s: %(message)s',
        level=numeric_level,
    )


def parse_args():  # <9>
    parser = argparse.ArgumentParser()
    parser.add_argument(
        '-d',
        action='store_true',
        help='Dry run, does not alter firewall rules'
    )
    parser.add_argument(
        '-b',
        metavar='URL',
        nargs='+',
        default=DEFAULT_BLs,
        help='Specify blacklists to process'
    )
    parser.add_argument(
        '-f',
        metavar='FIREWALL',
        default=FIREWALL,
        help='Specify firewall backend [default: ipfw]'
    )
    parser.add_argument(
        '--ipfw-table',
        metavar='TABLE',
        default=IPFW_TABLE,
        help='Specify table for ipfw [default: 5]'
    )
    parser.add_argument(
        '-l',
        metavar='LEVEL',
        type=str,
        default='INFO',
        help='CRITICAL, ERROR, WARNING, INFO [default], DEBUG'
    )

    return parser.parse_args()


def main():
    args = parse_args()
    logging_init(args.l)

    if not args.d:
        logging.debug('Preparing firewall tables')
        fw_prepare(args)

    # Async magic!
    loop = asyncio.get_event_loop()  # <10>
    with aiohttp.ClientSession(loop=loop) as session:
        ips = loop.run_until_complete(
            bls_fetch(args.b, session)
        )
    loop.close()

    ips = ('\n'.join(ips)).splitlines()  # <11>
    # Poor man's comment remover
    ips = map(lambda ip: ip.split('#')[0].split(';')[0].strip(), ips)

    ctr = 0  # <12>

    for ip in ips:  # <13>
        ip_verified = ip_verify(ip)
        if ip_verified is False:
            logging.debug('Invalid ip address: "{}"'.format(ip))
            logging.debug('Continuing with next ip')
            continue
        if not args.d:
            logging.debug('Blocking "{}"'.format(ip_verified))
            fw_ip_add(ip_verified, args)
            ctr += 1
        else:
            logging.debug('Would block "{}"'.format(ip_verified))
    logging.info('{} ips blocked'.format(ctr))


if __name__ == '__main__':
    main()
----

Thanks to `asciidoctor` and the `jekyll-asciidoc` plugin, I am now able to
comment on selected lines of code, which is awesome. Thanks to the creators
of `asciidoctor`, keep on the awesome work!

<1> Load the https://aiohttp.readthedocs.io[`aiohttp` library]. There are no
    technical reasons against `requests` or the stdlib. I just wanted to try
    the new asynchronous capabilities of Python 3.5. One can replace it safely
    with the http library of choice; but the script has to be adapted then...
<2> These are the blacklists which I use per default. Since I like Python
    programming a lot, I decided to make the blacklists configurable, and
    overwritable by the commandline options.
<3> My collegues still use `iptables`, so I designed the script to be extendible
    to other firewall backends. I use `ipfw` on FreeBSD, it should not be
    difficult to adapt the script to, e.g. `iptables`.
<4> To use the `async def` methods, a recent Python implementation has to be
    used, I use 3.5 currently. These two methods download the ip list from the
    configured blacklists. I have mostly copied the code from the `aiohttp`
    documentation. In the end, the `bls_fetch` method returns a string that
    contains the concatenated content of all configured blacklists.
<5> That's the actual part why I have chosen to implement this script in
    Python. The gathered ip addresses (and address ranges) can simply be
    verified by instantiating an ipaddress object. When it fails, an exception
    is raised. I trust the Python stdlib to be correct.
<6> Just add the given ip address to the firewall table by calling the `ipfw`
    binary. Please note, that all given strings MUST be sanitized using the
    `shlex` module when calling a binary!!
<7> Do some preparations, add the lookup table, flush it, make coffe, ...
<8> Initialize logging framework. I want logging, when this crap is run by `cron`.
<9> That's the argument parser, most variables can be overwritten on the commandline.
<10> That's needed because of `aiohttp`; it might be possible to replace it with
     a simple call to `requests.get()` or something...
<11> String post processing. The downloaded string has to be converted to the proper
     Python data types, and most importantly, comments should be removed in advance.
<12> A simple counter, to count how many ip addresses have been added to the block list.
<13> That's the main loop of my script. It iterates over the list of ip addresses,
     calls the verify methods and, if true, adds them to the firewall block table.

And finally add this script to cron, e.g. like this:

./etc/crontab
----
# ...
@daily  root    nice -n  5 /usr/local/bin/openbl.py
----

I have tested the script for two days now. It seems to work quite well; it indeed
blocks potential abusers:

----
$ ipfw -cde list
....
01200   417   25127 deny ip from table(5) to me
....
----

So, I hope this was useful, and I hope my server is now more secure, unless someone
hacks openbl.org and locks me out of my server... :D