~craftyguy/blog.craftyguy.net

5e945955e2c3c1c0843786a31e890f5bfbe82ffb — Clayton Craft 12 days ago cc68d32 main
content/2022-01-07-ipxe-aarch64: new post
A content/2022-01-07-ipxe-aarch64.md => content/2022-01-07-ipxe-aarch64.md +161 -0
@@ 0,0 1,161 @@
+++
title = "Network booting an aarch64 SBC with u-boot and iPXE"
date = 2022-01-07
[taxonomies]
tags = ["linux", "embedded"]
+++

I recently started trying to figure out network booting for aarch64 single
board computers (SBC), such as the Raspberry Pi, for a new CI I've been
helping out with at my Igalia day job. For one reason or another, I never
participated in the Raspberry Pi "fad" (maybe because they use Broadcom chips,
which are [or were] notoriously unfriendly on Linux? I don't recall why... But
I digress...)

But, I do have a quite capable aarch64 SBC just laying around, literally
collecting dust... the Purism Librem 5 DevKit!

{{ resize_image_fit_width(path="../static/images/2022-01-07-sweeping-off-devkit.jpg", width=300, caption="sweeping off the devkit for recommissioning with a tiny broom") }}

While not exactly a Raspberry Pi, I believe many of the concepts pre-Linux boot
are similar and this should serve as a decent replacement until the Great Chip
Shortage of 2020-???? is over and those things are available for purchase again.

The general idea is that u-boot will execute [iPXE](https://ipxe.org/start),
which will be responsible establishing a network connection and booting
whatever the DHCP server on the other end tells it to boot. The end goal is to
have it load/boot the Linux kernel and an initfs based on 
[boot2container](https://gitlab.freedesktop.org/mupuf/boot2container).

This is the first in a series of posts to get there. The focus of this initial
post is building/setting up iPXE, the DHCP server, and doing a test boot from
u-boot.

## Building iPXE and configuring the devkit

The first step is to build iPXE, since I want to embed a script for it to run
automatically on boot. I did the compilation on the devkit, since iPXE is a
relatively small program and it didn't take too long to compile on this CPU:

```sh
$ git clone git://git.ipxe.org/ipxe.git
$ cd ipxe/src

## needed so that ipxe doesn't lock up if you want to C-b to enter the cmdline
$ cat << EOF > config/local/nap.h
#undef NAP_EFIX86
#undef NAP_EFIARM
#define NAP_NULL
EOF

## and create a simple ipxe script that will be executed when ipxe runs:
$ cat << EOF > ipxescript
#!ipxe

:retry_dhcp
echo Acquiring an IP
dhcp || goto retry_dhcp

echo Got the IP: $${netX/ip} / $${netX/netmask}

:retry_boot
echo Booting from DHCP...
autoboot || goto retry_boot
EOF

## build/install:
$ make bin-arm64-efi/snp.efi -j4 EMBED=ipxescript
$ doas cp bin-arm64-efi/snp.efi /boot/ipxe.efi
```

Loading things in u-boot is quite tedious, since you have to specify memory
addresses to load files into, and the correct *load command to read files into
memory. I already have an existing install of postmarketOS on my devkit, so I
used the `/boot` partition (formatted as `ext2`) as a home for the iPXE binary.
I created the following U-boot helper script for loading iPXE, since typing all
of these in becomes tiresome very quickly:

```sh
$ cat << EOF > /tmp/ipxe 
echo ===== Loading iPXE =====
ext2load mmc 0:1 $kernel_addr_r ipxe.efi
ext2load mmc 0:1 $fdt_addr_r imx8mq-librem5-devkit.dtb
fdt addr $fdt_addr_r
fdt resize
echo ===== Running iPXE =====
bootefi $kernel_addr_r $fdt_addr_r
EOF
```

I'm not entirely sure if we need to specify/load the dtb, but it doesn't seem
to hurt! Also note that this u-boot script is using `bootefi` to load the iPXE
app. That'll be important later on when we try to boot a kernel.

The u-boot script must be compiled before u-boot can execute it:

```sh
$ mkimage -A arm64 -C none -O linux -T script -d /tmp/ipxe /tmp/ipxe.scr
$ doas cp /tmp/ipxe.scr /boot
```

In that last step, I copy it to `/boot` since I'm performing these steps on the
devkit, and `/boot` is the `ext2` partition I'll run iPXE from when booted into
u-boot.

## Configuring dnsmasq for BOOTP/DHCP
Now that all the necessary pieces are setup/installed on the devkit, the last
step is to run dnsmasq on a host to provide BOOTP service to the devkit. This
should be good enough for our purposes:

```sh
$ export workdir=/path/to/some/dir

## set to the network interface that is on the same physical LAN as the devkit that dnsmasq will bind to
$ export iface=eth0

## needs to run as root since it binds to ports < 1000
$ doas dnsmasq \
    --port=0 \
    --dhcp-hostsfile="$workdir"/hosts.dhcp \
    --dhcp-optsfile="$workdir"/options.dhcp \
    --dhcp-leasefile="$workdir"/dnsmasq.leases \
    --dhcp-boot=grubnetaa64.efi \
    --dhcp-range=10.42.0.10,10.42.0.100 \
    --dhcp-script=/bin/echo \
    --enable-tftp="$iface" \
    --tftp-root="$workdir"/tftp \
    --log-queries=extra \
    --conf-file=/dev/null \
    --log-debug \
    --no-daemon \
    --interface="$iface"
```

Note that the boot option sent to the client is `grubnetaa64.efi`. This is a
binary I pulled from some Debian build of grub2 for aarch64, since it was
annoying to have to build grub myself just for a quick smoke test.

Grub isn't necessary for booting the Linux kernel, but it is a small
application that serves as a good test to make sure that u-boot, iPXE, and
dnsmasq are happy.

If you're like me and run firewalls everywhere, you'll need to punch some holes
in it for bootp / tftp to work.

Once dnsmasq is started, the devkit is reset and the u-boot script to run iPXE
is executed:

```
Hit any key to stop autoboot:  0
u-boot=> env set boot_scripts ipxe.scr
u-boot=> boot
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /ipxe.scr
294 bytes read in 1 ms (287.1 KiB/s)
```

{{ resize_image_fit_width(path="../static/images/2022-01-07-first-grub-boot.png", width=300, caption="PXE booting to grub") }}

In Part 2, I'll cover booting the Linux kernel... Stay tuned!

A static/images/2022-01-07-first-grub-boot.png => static/images/2022-01-07-first-grub-boot.png +0 -0
A static/images/2022-01-07-sweeping-off-devkit.jpg => static/images/2022-01-07-sweeping-off-devkit.jpg +0 -0