A froghat.ca/2023/12/selinux/index.rst => froghat.ca/2023/12/selinux/index.rst +120 -0
@@ 0,0 1,120 @@
+VPS migration & selinux resentment
+==================================
+
+:summary: Rant about the one time I forgot to disable selinux and it prevented sshd from starting.
+:date: Dec 22 2023
+
+.. aside::
+
+ `skip to selinux rant <#selinux-rant>`_
+
+Earlier this year I went to renew my VPS at OVH. As well as a domain with Gandi.
+
+The price to renew the domain with Gandi went up from $51.15 to $72.59 while it was in my cart. I asked them what that was about and they replied, paraphrasing, "ya we raised the price 🖕". So instead I transferred my domain to porkbun.com for about $25.
+
+----
+
+At OVH it was going to cost $194 to renew my VPS for one year. I'm not sure why, but they wouldn't offer a discount for paying for an entire year upfront as they typically do. For $82, I switched to their cheapest AMD EPYC VPS. Same specs except less memory, which I didn't need that much of anyway.
+
+The service on that machine collects a lot of images and serves them on a website. Since, I didn't get around to making a good way of removing old files to free up space for new ones, migrating it to a new machine was an opportunity for housekeeping.
+
+This was as straightforward as you might expect:
+
+- I started the service on the new machine to begin collecting data.
+- Exported data I wanted to retain from the old machine up until the timestamp of the first event on the new machine. (Timestamps were in messages from an external service that both connected to, rather than wall clock which could differ between machines.)
+- Imported that historical data into the SQLite database on the new machine and copied corresponding images on disk with `tar` and `ssh`.
+
+After testing the new site in my browser, I updated the DNS records with porkbun to the new host's address.
+
+Even though the DNS record had a ten minute expiry, the old server still had connections when it was shut down about a day later. Those clients reconnected to the new host pretty quickly on their own.
+
+.. aside::
+
+ A stacked graph showing the number of connections to the :hl-yellow:`old` and :hl-purple:`new` servers over time.
+
+.. picture:: switchy-lm.png
+
+ .. source::
+ :srcset: switchy-dm.png
+ :media: (prefers-color-scheme: dark)
+
+It all went pretty smoothly and, in the end, I got that nice chart out of it.
+
+----
+
+.. _selinux rant:
+
+Fedora has a running prank of installing selinux when you install Fedora. I disabled selinux on my new VPS using `setenforce`.
+
+About a month after setting up the machine, I tried rebooting it to troubleshoot a networking issue. After a few minutes, the webserver did not come up and attempts to `ssh` into the machine timed out. Yet, the OVH dashboard reported that the machine was running.
+
+I ship application and system logs and metrics using `vector <https://vector.dev/>`__ which are then stored in `loki <https://grafana.com/oss/loki/>`__. By some miracle of science, vector was shipping logs during this time that the machine was not accepting ssh connections.
+
+.. class:: full-width reflow
+.. code::
+
+ [audit] AVC avc: denied { read } for pid=818 comm="sshd" name="sshd_config" dev="sda5" ino=974214 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=0
+ [sshd] /etc/ssh/sshd_config: Permission denied
+
+Evidently, selinux was keeping my system secure by preventing sshd from reading its own configuration file in order to start up. At that moment I realized I had only temporarily disabled selinux with `setenforce` and forgotten to disable it across reboots.
+
+.. aside::
+
+ I have to hand it to 'em here, requiring users to disable selinux twice in two different ways, once for the current boot and another for subsequent boots, is really very clever. Good one, Fedora, you got me!
+
+I used a recovery boot thingy that OVH provides to boot my VPS. OVH emails me SSH login credentials. I log in and disable selinux persistently. Now the machine boots normally.
+
+----
+
+What really got to me, is that I put a fair bit of my own time into reducing downtime in my service. I went so far as to use `sd_pid_notify_with_fds <https://www.freedesktop.org/software/systemd/man/252/sd_pid_notify_with_fds.html>`__ and `sd_listen_fds <https://www.freedesktop.org/software/systemd/man/252/sd_listen_fds.html>`__ to prevent missing messages broadcast by an external event source between service restarts. A number of times, the service upgraded without closing the socket it uses to read incoming events. Events would be buffered and read when the service comes back up; instead of being missed entirely while the connection was down.
+
+It worked. And even though this was a side-project of no real significance to anyone, I felt happy that I had no unplanned downtime in just over two years of running it.
+
+By the time I finished fumbling my way through the repair and the system came back online, it had been down for almost an hour. I understand that I forgot to disable selinux, and that selinux would prevent my service from starting because I didn't write policies for it. That's my fault, so that's fair I guess. And nginx wouldn't start either, as far as I know, because Fedora doesn't ship a policy that lets it bind to port 80, whatever. But I don't know why sshd would not be allowed to read its own configuration file at `/etc/ssh/sshd_config`. This makes no sense.
+
+My first guess was, if you run with selinux disabled temporarily and you upgrade your system, then packages and files or services (like sshd) won't be installed with selinux contexts. So when you reboot, and selinux is enabled again, it misbehaves because of missing contexts.
+
+On the other hand, `setenforce 0` doesn't actually *disable* selinux, according to the man page, it just sets it to permissive mode. And permissive mode is practically the same as selinux being enabled (in enforcing mode) in the sense that it runs and evaluates denials based on policies. The key difference being it doesn't *act* on those denials and just reports them. As far as I know, you are meant to be able to go from enforcing mode to permissive mode and back without issue -- for the purpose of troubleshooting or something. So if being in permissive mode messes up selinux contexts, that wouldn't make sense.
+
+Alternatively, it's possible I copied `/etc/ssh/sshd_config` over from the old machine with `tar` and, by default, it doesn't copy extended attributes or selinux context stuff. If I had done this, I would expect to see it in my shell history next to where I did this for a bunch of other files. But I don't see that. And the file creation timestamp doesn't suggest it was copied from the old machine either.
+
+So I have no idea how this happened. I tried to ask about it on #fedora on irc.libera.chat. To be upfront, I was pretty snarky after wasting an hour of my Sunday afternoon on this piece of shit software. I was advised; to enable sshd.service, don't ssh in as root, and that I screwed it up because it "just works".
+
+.. aside::
+
+ Long ago, I had spent some time trying to use selinux properly. It might be better now, but most of the documentation was high level overviews. It was time consuming to sort through that junk to find actual solutions to problems, like what you needed to run to configure selinux or even what the relevant packages were in Fedora.
+
+ Some of the resources are just memes, like `stopdisablingselinux.com <https://stopdisablingselinux.com/>`__ and a `colouring book`_ with illustrations that look like something out of a fever dream.
+
+ .. picture:: wtf-lm.png
+
+ .. source::
+ :srcset: wtf-dm.png
+ :media: (prefers-color-scheme: dark)
+
+ This isn't some random thing, it's the first item under *Additional resources* on the `Getting started with SELinux`_ page at docs.fedoraproject.org.
+
+ To be absolutely clear, *I am not trying to bully the illustrator here*. But whenever I've gone to look up documentation on selinux, it's because selinux has interrupted something else I'd rather be doing. A colouring book could be cute in some contexts. selinux holding my system hostage is not one of them.
+
+ .. _`colouring book`: https://people.redhat.com/duffy/selinux/selinux-coloring-book_A4-Stapled.pdf
+ .. _`Getting started with SELinux`: https://docs.fedoraproject.org/en-US/quick-docs/selinux-getting-started/#_additional_resources
+
+I clarified I wasn't asking whether or not I "screwed it up" but *how* -- so I can not do that in the future. I was pretty sarcastic and, predictably, the conversation in this internet chat room was pointless as nobody learned anything and everyone was rude to everyone else.
+
+In the end, I am surprised that sshd was not allowed to read its own configuration file. And this doesn't inspire confidence that selinux won't produce further surprises. How is an selinux enjoyer meant to minimize the surprises that might prevent their machine from booting?
+
+Presumably you can set the system to permissive, restart a service, and check the audit logs for denials that would have prevented it from starting in enforcing mode. Then do that for NetworkManager, systemd-networkd, systemd-resolved, or whatever other services you can think of that are important that selinux could possibly interfere with?
+
+But if your system is undergoing change as part of a reboot, as with `dnf system-upgrade`, will changes in that upgrade invite the wrath of the machine spirit? I have no idea. Do you? *(If so, please elaborate in the comments section below -- the engagement helps promote my content on this advertising platform!)*
+
+If Fedora shipped selinux permissive by default and allowed me to opt-in specific services to run under enforcing mode, that would be interesting to me. At least that way, I could voluntarily waste my time with it on my own terms instead of on a Sunday afternoon when I'd rather be trying to unfuck whatever ufw did to my nftables.
+
+As far as I can tell, the security selinux provides is job security for system administrators using it to create fragile complex systems that require specialized knowledge to maintain.
+
+.. .. class:: full-width
+
+.. figure:: sleepyfox.jpg
+
+ slepey fox -- `photographed by Mark Dumont <https://www.flickr.com/photos/wcdumonts/40602952543>`__ -- did you know this fox has never heard of selinux? look how soundly he snoozes :nowrap:`🦊💤`
+
+In conclusion, selinux is a great addition to any Linux based operating system. My VPS was never safer or more secure than on the day that none of its services were running.
A froghat.ca/2023/12/selinux/sleepyfox.jpg => froghat.ca/2023/12/selinux/sleepyfox.jpg +0 -0
A froghat.ca/2023/12/selinux/switchy-dm.png => froghat.ca/2023/12/selinux/switchy-dm.png +0 -0
A froghat.ca/2023/12/selinux/switchy-lm.png => froghat.ca/2023/12/selinux/switchy-lm.png +0 -0
A froghat.ca/2023/12/selinux/wtf-dm.png => froghat.ca/2023/12/selinux/wtf-dm.png +0 -0
A froghat.ca/2023/12/selinux/wtf-lm.png => froghat.ca/2023/12/selinux/wtf-lm.png +0 -0
M fucko/__main__.py => fucko/__main__.py +0 -24
@@ 99,27 99,3 @@ def maybe_pdb_context(args):
if __name__ == "__main__":
main()
-
-
-# import operator
-
-# class Partial(object):
-# def __init__(self, fn):
-# self.fn = fn
-
-# def __call__(self, v):
-# return self.fn(v)
-
-# def __getitem__(self, item):
-# return Partial(lambda v: self.fn(v)[item])
-
-# def __getattr__(self, attr):
-# return Partial(lambda v: getattr(self.fn(v), attr))
-
-# def __eq__(self, other):
-# return Partial(lambda v: self.fn(v) == other)
-
-# # def __rshift__(self, other):
-# # breakpoint()
-
-# _ = Partial(lambda v: v)
M fucko/boingo.py => fucko/boingo.py +6 -3
@@ 251,10 251,13 @@ def start_httpd(host, port, srv, rebuild_continuum):
if self.path.startswith("/_watch/"):
self.send_watch_events_forever()
else:
- super().do_GET()
+ try:
+ super().do_GET()
+ except FileNotFoundError:
+ self.send_error(404)
- def send_response(self, code):
- super().send_response(code)
+ def send_response(self, code, message=None):
+ super().send_response(code, message=message)
if code == 301:
# Stupid hack to prevent the client from waiting for a response
# body forever in http/1.1.
M fucko/rstext.py => fucko/rstext.py +1 -1
@@ 277,7 277,7 @@ def build_MemeHTMLTranslator(cls):
self.body.append("</picture>\n")
def visit_source(self, node):
- attrs = {a: node.get(a) for a in Source.option_spec.keys()}
+ attrs = {a: node.get(a) for a in Source.option_spec.keys() if a in node}
self.body.append(self.starttag(node, "source", **attrs))
def depart_source(self, node):