0c026b75133c60def332c74311bc3b665f14164a — Ben Fiedler 2 months ago eec54e4
Draft of Bundlewrap post
2 files changed, 349 insertions(+), 0 deletions(-)

A content/blog/bundlewrap-first-impressions.md
A content/blog/img/bundlewrap-plot.svg
A content/blog/bundlewrap-first-impressions.md => content/blog/bundlewrap-first-impressions.md +271 -0
@@ 0,0 1,271 @@
title: "Bundlewrap: First Impressions"
date: 2021-01-25T00:00:00+01:00
draft: true
tags: [homelab]

I have spent quite some time with configuration management for my home infra
setup, and I have recently come across a new tool that I'm excited to share with
you. It's called [Bundlewrap](https://bundlewrap.org), and it is a flexible,
small-scale[^2] configuration management solution. Bundlewrap is written in
Python 3[^1], and is unique in the sense that the infrastructure configuration
is also written in Python. I was first introduced to Bundlewrap by
[@kunsi](https://chaos.social/@kunsi) in December 2020, who gave a brief
presentation at cozy conference.

My current setup uses [Ansible](https://ansible.com) and the oldest parts are
about two years old, so it's mostly written for Ansible 2.4 and later. Most
comparisons done are against Ansible.

# Brief overview

Bundlewrap manages nodes via so-called *bundles*, which roughly correspond to
Ansible roles. A bundle describes a desired state on the target nodes,
comprising of one or more *items*, which would be tasks in Ansible.

Nodes are defined in `nodes.py`. Each node has a separate dictionary
called `metadata` associated with it, which holds all the node's
configuration data and can be read and written by the bundles.  Machines can be
grouped and group metadata can be applied to all members by specifying it in

Let's look at an example: the bundle `ssh-server` should install and configure
an SSH server on our node `test`. To do so we have to specify the *items* we
want to use, here [`pkg_apt`](https://docs.bundlewrap.org/items/pkg_apt/),
[`file`](https://docs.bundlewrap.org/items/file/) and `svc_systemd`. An entire
repository could look something like this:

├── bundles                  # All bundles are in the directory bundles/
│   └── ssh-server           # Our ssh-server bundle
│       ├── files            # File templates for the ssh-server bundle
│       │   └── sshd_config  # SSH server configuration template (omitted)
│       ├── items.py         # Item definition file, Ansible: tasks/main.yml
│       └── metadata.py      # Metadata definition file, Ansible: defaults/main.yml
├── groups.py                # Group configuration
└── nodes.py                 # Node configuration

# nodes.py
nodes = {
    'test': {
        'hostname': '',
        'bundles': {'ssh-server'},
        'metadata': {'nodevar': 'string'},

# groups.py
groups = {
    'group': {
        'members': {'test'},
        'metadata': {'groupvar': 5},

# items.py for bundle ssh-server
pkg_apt = {
    'openssh-server': {'installed': True},

files = {
    '/etc/ssh/sshd_config': {
        'source': 'sshd_config',
        'triggers': {'svc_systemd:sshd:restart'},

svc_systemd = {
    'sshd': {
        'enabled': True,
        'running': True,
        'needs': {'pkg_apt:openssh-server'},

# metadata.py for bundle ssh-server
defaults = {
    'bundlevar': ['abc', 'def'],

Configuration is done via Python dictionaries, and since the files are literal
Python code, you can embed arbitrary logic in these files. Bundlewrap is
[well-documented](https://docs.bundlewrap.org), and I encourage you to read the
docs if you want to figure out what an item does.

Now that we have seen the general structure of bundles, let us come to the
feature comparison.

# The Good

## Agentless, push-based, no Python required on guest

Just like Ansible, Bundlewrap is both agentless and push-based. Managed nodes
are accessed via SSH, which is my preferred way. Contrary to Ansible though,
Bundlewrap does not require Python to be installed on the managed hosts, and
instead relies on [common

Privilege escalation must work noninteractively. Since I anyway hate entering
passwords, this not a problem for me. The exact privilege escalation method used
is configurable, so BSDs for example have the option to use `doas` instead of
`sudo` (the default).

Bundlewrap does *not* do SSH multiplexing by default, but it is possible to pass
arbitrary arguments to the underlying ssh invocation via an environment

## Automatic metadata merging, metadata generation

In the example above you might have noticed that we have defined metadata for
both the node and the group it belongs to. Generally in Bundlewrap,
non-collection metadata follows a strict hierarchy: node metadata overrides
group metadata overrides bundle metadata. Collections are merged recursively,
which is one of the best features Bundlewrap has. We can instruct Bundlewrap to
display the metadata associated with `test`. The output is color-coded[^4] according
to where the key comes from (group/node/bundle), which is very helpful.

% bw metadata test
    "bundlevar": [       # Colored blue = from bundle
    "groupvar": 5,       # Colored yellow = from group
    "nodevar": "string"  # Colored red    = from node

This is one of the features I miss most from Ansible. I have a ton of roles
which would like to have their variables merged. One example is my Prometheus
setup: My monitoring server has to know about every exporter that a node has
installed in order to scrape all of them. Ideally I'd just have a list for each
node which has `(exporter, port)` pairs and each exporter role appends a pair to
this node, thus allowing the monitioring role to work independently of the
available exporters. However, since Ansible does not allow appending to an
existing variable, I am stuck hardcoding every possible exporter into the main
prometheus role.

Bundlewrap also allows for generating new metadata from existing metadata, using
a concept called *metadata reactors*. These are defined at the bundle level and
are extremely powerful. You can, for example, ensure that every virtual host
automatically also gets issued a letsencrypt certificate, while still separating
the webhost and letsencrypt bundles.

## Secret derivation

Ansible has secrets, which allow you to store encrypted data and decrypt it with
a static key. Bundlewrap can also do this, but additionally it allows you to
generate secrets dynamically, which you can extract on demand. This is
especially useful for automatic password generation for user accounts or when
connecting a service to a DB user account: In both cases I don't really care
*what* the secret is, only that 1. it is a *secret* known only to the correct
parties and 2. I can recover it if needed. Additionally, the secrets
can easily be rotated by replacing the key used for secret derivation! Of
course, now anyone in possession of the Bundlewrap master secret can derive all
your passwords, so be sure to secure it well.

## Offline testing

This one's huge: Bundlewrap supports sensible offline testing. Bundlewrap tests
involve assembling all metadata for all nodes, checking that all items are
well-formed, all templates instantiate without errors, and so on. This is a
feature I sorely miss from Ansible. While Ansible has the `--check` parameter, it
still simulates each step by connecting to the target node, which is *really*
slow compared to local execution. Plus, you can run Bundlewrap tests as part of
your CI pipeline (even works for secrets without the decryption/generation

% bw test
✓ No reactors violated their declared keys
✓ group  has no subgroup loops
✓ test  has no metadata conflicts
✓ test  ssh-server  file:/etc/ssh/sshd_config
✓ test  ssh-server  pkg_apt:openssh-server
✓ test  ssh-server  svc_systemd:sshd
✓ test  ssh-server  svc_systemd:sshd:restart
✓ test  ssh-server  svc_systemd:sshd:reload

## Small core

Bundlewrap has an extremely small "standard library" of items, and prides itself
on staying that way. Personally, I value scope-restriction a lot in projects, so
this is a good thing. On the other hand it means that, more often than not, you
have to write the code for new items yourself, e.g. support for a new package
manager. Fortunately, the code is quite accessible, and the methods you need to
implement are well-documented.

# The Neutral

## Python dicts

Python dictionaries look much more like JSON than YAML, however in my opinion
this does not impact readability. Writing Python dicts is slightly more pleasant
than raw JSON, since it allows the use of single quotes for string
identifiers[^3]. Formatting is taken care of by any linter, which is nicer
than YAML, where indentation cannot be automatically inferred. Of course, this
is true for raw Python code as well.

## Statistics and dependency graphs

This is undoubtedly a cool feature: Bundlewrap can output graphs (in graphviz
format) visualizing the item/bundle dependencies on a node, or your
repositories' group relationships. And it also keeps track of statistics such as
the number of items, nodes, groups, bundles and so on. These features don't have
a downside, however I also haven't (yet) discovered  clear upsides other than "ooh,

{{< figure class="invertable resizable" src="/blog/img/bundlewrap-plot.svg" caption="Output of bw plot test" >}}

# The Bad

## Python

I really, really, really dislike Python. Mainly because it is interpreted
and dynamically typed, which means that most errors will occur at runtime when
it is too late to fix stuff. Working with Bundlewrap snippets is even worse,
since some variables are passed "automagically", which confuses my poor language
server (and `mypy` as well), so any possibility of static type checking is
chucked right out of the window.

Bundlewraps excellent local testing feature alleviates this issue somewhat.

## Turing-complete config language

Having all the flexibility and power of Python also means having more footguns
available to shoot yourself with. Bundlewrap relies much more on the user for
constraining the bundle complexity. Personally, I think that for small
infrastructures (such as what I run at home) this is fine, however I would be
wary of this power for bigger deployments.

# Conclusion

I've spent the last month thinking about and testing configuration management
systems, and believe I have found a hidden gem in Bundlewrap. The only other
notable mention I tried was [cdist](https://cdi.st), however it has its fair
share of oddities, most notably being a 100% sh-based solution. Of course, this
doesn't mean that it's not good for you! Go check it out if that premise excites

I hope you got a brief overview of Bundlewrap and it's features. Personally, I
think it's a better solution for my usecase than Ansible, and I'm going to
slowly port my Ansible roles to Bundlewrap this year. If you're interested and
want to see more configurations/examples check out
[my](https://git.sr.ht/~bfiedler/bundlewrap) and (especially)
[Franziska's](https://git.kunsmann.eu/kunsi/bundlewrap) repositories.

If you have any questions or comments feel free to reach out to me via my
[public inbox](https://lists.sr.ht/~bfiedler/public-inbox) or toot
[@bfiedler](https://mastodon.3fx.ch/@bfiedler) on Mastodon.

[^1]: Quick reminder that Python 2 went EOL in January 2020.
[^2]: think <300 nodes/machines
[^3]: Laugh at me all you want, that is my biggest gripe when writing JSON by hand.
[^4]: on your terminal, at least

A content/blog/img/bundlewrap-plot.svg => content/blog/img/bundlewrap-plot.svg +78 -0
@@ 0,0 1,78 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
<!-- Generated by graphviz version 2.43.0 (0)
<!-- Title: bundlewrap Pages: 1 -->
<svg width="385pt" height="279pt"
 viewBox="0.00 0.00 385.00 279.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 275)">
<polygon fill="white" stroke="transparent" points="-4,4 -4,-275 381,-275 381,4 -4,4"/>
<g id="clust1" class="cluster">
<path fill="none" stroke="#303030" stroke-width="2" stroke-dasharray="5,2" d="M20,-8C20,-8 357,-8 357,-8 363,-8 369,-14 369,-20 369,-20 369,-143 369,-143 369,-149 363,-155 357,-155 357,-155 20,-155 20,-155 14,-155 8,-149 8,-143 8,-143 8,-20 8,-20 8,-14 14,-8 20,-8"/>
<text text-anchor="middle" x="188.5" y="-139.8" font-family="Helvetica,sans-Serif" font-size="14.00">ssh&#45;server</text>
<!-- pkg_apt:openssh&#45;server -->
<g id="node1" class="node">
<path fill="#303030" stroke="#303030" d="M188.5,-52C188.5,-52 27.5,-52 27.5,-52 21.5,-52 15.5,-46 15.5,-40 15.5,-40 15.5,-28 15.5,-28 15.5,-22 21.5,-16 27.5,-16 27.5,-16 188.5,-16 188.5,-16 194.5,-16 200.5,-22 200.5,-28 200.5,-28 200.5,-40 200.5,-40 200.5,-46 194.5,-52 188.5,-52"/>
<text text-anchor="middle" x="108" y="-30.3" font-family="Helvetica,sans-Serif" font-size="14.00" fill="white">pkg_apt:openssh&#45;server</text>
<!-- file:/etc/ssh/sshd_config -->
<g id="node2" class="node">
<path fill="#303030" stroke="#303030" d="M348.5,-124C348.5,-124 189.5,-124 189.5,-124 183.5,-124 177.5,-118 177.5,-112 177.5,-112 177.5,-100 177.5,-100 177.5,-94 183.5,-88 189.5,-88 189.5,-88 348.5,-88 348.5,-88 354.5,-88 360.5,-94 360.5,-100 360.5,-100 360.5,-112 360.5,-112 360.5,-118 354.5,-124 348.5,-124"/>
<text text-anchor="middle" x="269" y="-102.3" font-family="Helvetica,sans-Serif" font-size="14.00" fill="white">file:/etc/ssh/sshd_config</text>
<!-- svc_systemd:sshd -->
<g id="node3" class="node">
<path fill="#303030" stroke="#303030" d="M147.5,-124C147.5,-124 28.5,-124 28.5,-124 22.5,-124 16.5,-118 16.5,-112 16.5,-112 16.5,-100 16.5,-100 16.5,-94 22.5,-88 28.5,-88 28.5,-88 147.5,-88 147.5,-88 153.5,-88 159.5,-94 159.5,-100 159.5,-100 159.5,-112 159.5,-112 159.5,-118 153.5,-124 147.5,-124"/>
<text text-anchor="middle" x="88" y="-102.3" font-family="Helvetica,sans-Serif" font-size="14.00" fill="white">svc_systemd:sshd</text>
<!-- svc_systemd:sshd&#45;&gt;pkg_apt:openssh&#45;server -->
<g id="edge1" class="edge">
<path fill="none" stroke="#c24948" stroke-width="2" d="M92.94,-87.7C95.17,-79.9 97.85,-70.51 100.33,-61.83"/>
<polygon fill="#c24948" stroke="#c24948" stroke-width="2" points="103.11,-52.1 104.69,-62.96 102.22,-57.05 100.85,-61.86 100.37,-61.72 99.89,-61.58 101.26,-56.77 96.04,-60.48 103.11,-52.1 103.11,-52.1"/>
<!-- svc_systemd:sshd:reload -->
<g id="node4" class="node">
<path fill="#303030" stroke="#303030" d="M233.5,-271C233.5,-271 66.5,-271 66.5,-271 60.5,-271 54.5,-265 54.5,-259 54.5,-259 54.5,-247 54.5,-247 54.5,-241 60.5,-235 66.5,-235 66.5,-235 233.5,-235 233.5,-235 239.5,-235 245.5,-241 245.5,-247 245.5,-247 245.5,-259 245.5,-259 245.5,-265 239.5,-271 233.5,-271"/>
<text text-anchor="middle" x="150" y="-249.3" font-family="Helvetica,sans-Serif" font-size="14.00" fill="white">svc_systemd:sshd:reload</text>
<!-- svc_systemd:sshd:reload&#45;&gt;svc_systemd:sshd -->
<g id="edge2" class="edge">
<path fill="none" stroke="#c24948" stroke-width="2" d="M132.51,-234.85C123.58,-225.13 113.29,-212.28 107,-199 97.26,-178.45 92.49,-153 90.17,-134.08"/>
<polygon fill="#c24948" stroke="#c24948" stroke-width="2" points="89.09,-124.05 94.64,-133.51 90.13,-128.96 90.66,-133.94 90.16,-133.99 89.67,-134.04 89.13,-129.07 85.69,-134.47 89.09,-124.05 89.09,-124.05"/>
<!-- svc_systemd:sshd:restart -->
<g id="node5" class="node">
<path fill="#303030" stroke="#303030" d="M298,-199C298,-199 128,-199 128,-199 122,-199 116,-193 116,-187 116,-187 116,-175 116,-175 116,-169 122,-163 128,-163 128,-163 298,-163 298,-163 304,-163 310,-169 310,-175 310,-175 310,-187 310,-187 310,-193 304,-199 298,-199"/>
<text text-anchor="middle" x="213" y="-177.3" font-family="Helvetica,sans-Serif" font-size="14.00" fill="white">svc_systemd:sshd:restart</text>
<!-- svc_systemd:sshd:reload&#45;&gt;svc_systemd:sshd:restart -->
<g id="edge3" class="edge">
<path fill="none" stroke="#c24948" stroke-width="2" d="M165.57,-234.7C173.2,-226.22 182.52,-215.86 190.88,-206.58"/>
<polygon fill="#c24948" stroke="#c24948" stroke-width="2" points="197.61,-199.1 194.26,-209.55 194.63,-203.16 191.29,-206.87 190.92,-206.54 190.54,-206.2 193.89,-202.49 187.57,-203.53 197.61,-199.1 197.61,-199.1"/>
<!-- svc_systemd:sshd:restart&#45;&gt;file:/etc/ssh/sshd_config -->
<g id="edge5" class="edge">
<path fill="none" stroke="#6bb753" stroke-width="2" d="M226.27,-162.7C233.25,-153.61 241.92,-142.3 249.63,-132.25"/>
<polygon fill="#6bb753" stroke="#6bb753" stroke-width="2" points="255.82,-124.18 253.31,-134.85 253.18,-128.45 250.13,-132.41 249.74,-132.11 249.34,-131.81 252.38,-127.84 246.17,-129.37 255.82,-124.18 255.82,-124.18"/>
<!-- svc_systemd:sshd:restart&#45;&gt;svc_systemd:sshd -->
<g id="edge4" class="edge">
<path fill="none" stroke="#c24948" stroke-width="2" d="M181.94,-162.92C177.25,-160.29 172.49,-157.59 168,-155 153.73,-146.76 138.17,-137.49 124.6,-129.31"/>
<polygon fill="#c24948" stroke="#c24948" stroke-width="2" points="116,-124.11 126.89,-125.43 120.54,-126.27 124.82,-128.86 124.56,-129.29 124.3,-129.71 120.02,-127.13 122.23,-133.14 116,-124.11 116,-124.11"/>