~chrichri/RockPro64_u-boot_SATA_software_RAID_howto

Howto move an armbian system running off eMMc/sdcard onto a sata SoftwareRaid.
b283fdcc — Chris Vogel 1 year, 4 months ago
Flash Sector Size could be wrong
52ee04d2 — Chris Vogel 1 year, 4 months ago
finally worked, ready to publish
b751d221 — Chris Vogel 1 year, 4 months ago
more url

refs

main
browse  log 

clone

read-only
https://git.sr.ht/~chrichri/RockPro64_u-boot_SATA_software_RAID_howto
read/write
git@git.sr.ht:~chrichri/RockPro64_u-boot_SATA_software_RAID_howto

You can also use your local clone with git send-email.

#Howto move an armbian system running off eMMc/sdcard onto a sata SoftwareRaid

I'm running a yunohost based on Armbian on a RockPro64 in a nas case. When I started out it just has been a test about how well yunohost works and whether the hardware would be sufficient. I started out with two SSDs attached, but I didn't configure them initially.

I just wanted to learn about yunohost and ran the whole system on a 16GB eMM. After actually using it for about a year I decided it would be time to make use of the SSDs configured as a raid1 (mirroring).

To avoid confusion and complexity I decided that I'd want to boot of the SSDs.

I had to configure it without physical access to the host. This is what I wrote down in preparation to installing #uboot into the SPI of my yunohost remotely via ssh and make it boot of the #SoftwareRAID on the two SSDs.

To get this how-to I tried all the steps in a testing environment, then wrote an outline of the right order, reseted my environment to what I describe and started over capturing my terminal. Then I filled in the relevant terminal snippets with some explanations.

I ran my how-to against my productive yunohost extending and correcting it once along the way ...

... and stumbled over not using exactly the same hardware and not checking the raid headers. With the added information this shouldn't happen to you.

Thanks to all those wonderful FLOSS people out there who made this possible with their software and sharing of knowledge!

#my environment

I'm Running #Armbian Bullseye from eMMC (sdcard should work the same) on a #Pine64 #RockPro64 (4GB). The #SBC is build into a Pine64 NAS case. Two SSDs are connected via a PCI sata controller I also bought from Pine64.

 ____            _    ____               __   _  _   
|  _ \ ___   ___| | _|  _ \ _ __ ___    / /_ | || |  
| |_) / _ \ / __| |/ / |_) | '__/ _ \  | '_ \| || |_ 
|  _ < (_) | (__|   <|  __/| | | (_) | | (_) |__   _|
|_| \_\___/ \___|_|\_\_|   |_|  \___/   \___/   |_|  
                                                     
Welcome to Armbian 23.8.1 Bullseye with Linux 4.4.213-legacy-rockchip64

System load:   8%               Up time:       1:06     
Memory usage:  3% of 3.72G      IP:            192.168.2.169
CPU temp:      41°C             Usage of /:    15% of 28G       

Last login: Wed Sep 20 09:27:35 2023 from 192.168.2.20
root@rockpro64:~# cat /etc/armbian-release /etc/debian_version 
# PLEASE DO NOT EDIT THIS FILE
BOARD=rockpro64
BOARD_NAME="RockPro 64"
BOARDFAMILY=rockchip64
BUILD_REPOSITORY_URL=https://github.com/armbian/build
BUILD_REPOSITORY_COMMIT=ac829eb
LINUXFAMILY=rockchip64
ARCH=arm64
IMAGE_TYPE=stable
BOARD_TYPE=conf
INITRD_ARCH=arm64
KERNEL_IMAGE_TYPE=Image
FORCE_BOOTSCRIPT_UPDATE=
VENDOR=Armbian
BOOTSCRIPT_FORCE_UPDATE="no"
BOOTSCRIPT_DST="boot.cmd"
VERSION=23.8.1
REVISION=23.8.1
BRANCH=legacy
11.7
root@rockpro64:~# lspci
00:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd RK3399 PCI Express Root Port
01:00.0 SATA controller: JMicron Technology Corp. JMB58x AHCI SATA controller
root@rockpro64:~# fdisk -l /dev/sda
Disk /dev/sda: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: ST2000VN000-1HJ1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
root@rockpro64:~# fdisk -l /dev/sdb
Disk /dev/sdb: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WDC WD20EARX-00P
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

The harddisk shown above are the ones I used in my testing environment while writing this guide.

#hardware known not to work

#DeLOCK 90498 5-port SATA controller.

The controller can only be used by the kernel if it had not been initialized by u-boots pci command. There are some details in my notes.md.

#preparations to recover from failure

#install software

root@rockpro64:~# apt-get install mtd-utils libubootenv-tool
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libiniparser1 libubootenv0.1
The following NEW packages will be installed:
  libiniparser1 libubootenv-tool libubootenv0.1 mtd-utils
[0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
[...]

These two packages contain tools to manipulate the u-boot image and environment we'll need later to write into the SPI.

Instead of compiling a u-boot myself I decided to keep it simple and found that Debian Trixie does offer a package for u-boot newer or equal than 2023.04 (in some forum or manual I had read that starting with this release it would be possible to boot of sata).

Get the deb file and extract it.

root@rockpro64:~# wget http://ftp.de.debian.org/debian/pool/main/u/u-boot/u-boot-rockchip_2023.07+dfsg-1_arm64.deb
--2023-09-20 09:36:19--  http://ftp.de.debian.org/debian/pool/main/u/u-boot/u-boot-rockchip_2023.07+dfsg-1_arm64.deb
Resolving ftp.de.debian.org (ftp.de.debian.org)... 141.76.2.4
Connecting to ftp.de.debian.org (ftp.de.debian.org)|141.76.2.4|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1887440 (1.8M) [application/vnd.debian.binary-package]
Saving to: ‘u-boot-rockchip_2023.07+dfsg-1_arm64.deb’

u-boot-rockchip_2023.07+dfsg- 100%[===============================================>]   1.80M  --.-KB/s    in 0.1s    

2023-09-20 09:36:19 (14.5 MB/s) - ‘u-boot-rockchip_2023.07+dfsg-1_arm64.deb’ saved [1887440/1887440]

root@rockpro64:~# mkdir u-boot-rockchip_2023.07+dfsg-1_arm64.deb.d
root@rockpro64:~# dpkg-deb -X u-boot-rockchip_2023.07+dfsg-1_arm64.deb u-boot-rockchip_2023.07+dfsg-1_arm64.deb.d/
./
./usr/
[...]
./usr/lib/u-boot/rockpro64-rk3399/
./usr/lib/u-boot/rockpro64-rk3399/idbloader.img
./usr/lib/u-boot/rockpro64-rk3399/rk3399-rockpro64.dtb
./usr/lib/u-boot/rockpro64-rk3399/u-boot-nodtb.bin
./usr/lib/u-boot/rockpro64-rk3399/u-boot-spl.bin
./usr/lib/u-boot/rockpro64-rk3399/u-boot-tpl.bin
./usr/lib/u-boot/rockpro64-rk3399/u-boot.bin
./usr/lib/u-boot/rockpro64-rk3399/u-boot.img
./usr/lib/u-boot/rockpro64-rk3399/u-boot.itb
./usr/lib/u-boot/rockpro64-rk3399/uboot.elf
[...]
./usr/share/lintian/overrides/u-boot-rockchip

There's a subdirectory containing the files you'll need to prepare the u-boot image to copy into the RockPro64s SPI.

The SPI is a small flash memory that can be used to boot from. It is completely independend of sdcard, eMMC or other connected memories.)

#prepare harddisks

Your harddisks need to be partitioned for software raid and the software raid needs to be configured. On the software raids md devices you'll need to create filesystems and mount these to copy your running system onto the harddisks.

If you're just interested in how-to boot your RockPro64 of a software raid on sata devices - skip ahead to install u-boot to SPI

#partition haddisks

Create partitions on your harddisk for the software raid with the partitions you'll need. I used created three for swap, the root filesystem and a bigger data partition.

Be careful to use the right device node for your setup. You'd need to replace /dev/sda and /dev/sdb by whatever your system uses.

root@rockpro64:~# fdisk /dev/sda

Welcome to fdisk (util-linux 2.36.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xa6985dc8.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-3907029167, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-3907029167, default 3907029167): +14G

Created a new partition 1 of type 'Linux' and of size 14 GiB.

Command (m for help): t
Selected partition 1
Hex code or alias (type L to list all): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'.

Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (2-4, default 2): 
First sector (29362176-3907029167, default 29362176): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (29362176-3907029167, default 3907029167): +4G

Created a new partition 2 of type 'Linux' and of size 4 GiB.

Command (m for help): t
Partition number (1,2, default 2): 
Hex code or alias (type L to list all): fd

Changed type of partition 'Linux' to 'Linux raid autodetect'.

Command (m for help): n
Partition type
   p   primary (2 primary, 0 extended, 2 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (3,4, default 3): 
First sector (37750784-3907029167, default 37750784): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (37750784-3907029167, default 3907029167): +200G

Created a new partition 3 of type 'Linux' and of size 200 GiB.

Command (m for help): t
Partition number (1-3, default 3): 
Hex code or alias (type L to list all): fd

Changed type of partition 'Linux' to 'Linux raid autodetect'.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

You'd need to do the same for your second harddrive. Let's take a shortcut by copying the MBR containing the partition table over to the second drive:

root@rockpro64:~# dd if=/dev/sda of=/dev/sdb bs=512 count=1
1+0 records in
1+0 records out
512 bytes copied, 0.00244502 s, 209 kB/s

This works only if your system only uses primary partitions. If you use extended partitions you'll need to do use fdisk on the second drive, also (or find out how to copy the partition data for your extended partition).

Checking the partition table of the two drives they should be identical:

root@rockpro64:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: ST2000VN000-1HJ1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xa6985dc8

Device     Boot    Start       End   Sectors  Size Id Type
/dev/sda1           2048  29362175  29360128   14G fd Linux raid autodetect
/dev/sda2       29362176  37750783   8388608    4G fd Linux raid autodetect
/dev/sda3       37750784 457181183 419430400  200G fd Linux raid autodetect


Disk /dev/sdb: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: WDC WD20EARX-00P
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xa6985dc8

Device     Boot    Start       End   Sectors  Size Id Type
/dev/sdb1           2048  29362175  29360128   14G fd Linux raid autodetect
/dev/sdb2       29362176  37750783   8388608    4G fd Linux raid autodetect
/dev/sdb3       37750784 457181183 419430400  200G fd Linux raid autodetect

#create software raids

To create the software raid devices composed of the partitions you just made you'll need the kernel module available:

root@rockpro64:~# lsmod | grep 'md_mod\|raid1 '
raid1                  36864  3
md_mod                143360  6 raid456,raid0,raid1,multipath,linear,raid10

You should take care that there are not already software raids running on md devices you might use during the following steps or that your harddisks contained any data that the driver interpreted and started as software raids:

root@rockpro64:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
unused devices: <none>

The device for your root filesystem should use the older version 1.0 header format which is written at the end of the partition (-e 1.0). By doing so you'll not need to worry whether u-boot would be able to understand the software raid.

root@rockpro64:~# mdadm -C /dev/md1 -e 1.0 -l1 -n2 /dev/sda1 /dev/sdb1 
mdadm: Fail create md1 when using /sys/module/md_mod/parameters/new_array
mdadm: array /dev/md1 started.

If you get that error message and you're worried or if you do not get it and you wonder why: mdadm-error.md.

The rest of the arrays is created using the latest header format:

root@rockpro64:~# mdadm -C /dev/md2 -l1 -n2 --run /dev/sda2 /dev/sdb2
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: Fail create md2 when using /sys/module/md_mod/parameters/new_array
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.
root@rockpro64:~# mdadm -C /dev/md3 -l1 -n2 --run /dev/sda3 /dev/sdb3
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: Fail create md3 when using /sys/module/md_mod/parameters/new_array
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md3 started.

To make sure the creation of the software raids worked, look at /proc/mdstat:

root@rockpro64:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md3 : active raid1 sdb3[1] sda3[0]
      209715072 blocks super 1.0 [2/2] [UU]
      	resync=DELAYED
      bitmap: 2/2 pages [8KB], 65536KB chunk

md2 : active raid1 sdb2[1] sda2[0]
      4194240 blocks super 1.0 [2/2] [UU]
      	resync=DELAYED
      
md1 : active raid1 sdb1[1] sda1[0]
      14679936 blocks super 1.0 [2/2] [UU]
      [=======>.............]  resync = 36.2% (5319616/14679936) finish=1.4min speed=110873K/sec
      
unused devices: <none>

If you've been fast enough you can still see the software raids syncing.

After the software raids are synced they should look like this:

root@rockpro64:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md3 : active raid1 sdb3[1] sda3[0]
      209583104 blocks super 1.2 [2/2] [UU]
      bitmap: 0/2 pages [0KB], 65536KB chunk

md2 : active raid1 sdb2[1] sda2[0]
      4189184 blocks super 1.2 [2/2] [UU]
      
md1 : active raid1 sdb1[1] sda1[0]
      14679936 blocks super 1.0 [2/2] [UU]
      
unused devices: <none>

#check your software-raid

When I followed this howto I already had configured my software raids a year ago. I didn't check them and failed, because they had all the same name in their header and that led to a broken mdadm.conf created by mkconf for the initramfs.

Look at your raids:

root@rock64pro:~# for D in /dev/md[0-9]*; do mdadm -Db $D; done
ARRAY /dev/md1 metadata=1.0 name=/dev/md/1 UUID=a33a64c9:298aa4cc:8f2c834f:32eb8840
ARRAY /dev/md2 metadata=1.2 name=ups_wrong UUID=843a093d:01b6194a:dce59dfd:ddd419fd
ARRAY /dev/md3 metadata=1.2 name=ups_wrong UUID=cca47d3b:a9ea246b:fa2e39a8:ee9a7f51

Make sure all name entries for your arrays differ from each other. If you find two arrays with the same name like above /dev/md2 and /dev/md3 both named ups_wrong your system will most probably not boot.

Start over making your arrays or rename them to give them different names.

#make filesystems

To use the new software raids you'll need to create filesystems on them. To mount them during boot process automatically you'll need to identify them. You'll put a label on the block devices to use during the boot process.

There is no reason I know off using UUID or paths shouldn't work. I just decided to go by the most self-explaining and simple option.

#filesystems

Create you swap partition:

root@rockpro64:~# mkswap -L swap /dev/md2
Setting up swapspace version 1, size = 4 GiB (4289720320 bytes)
LABEL=swap, UUID=5674cbfd-e8f7-478a-8660-fb72c1e26cfd

Create the filesystems for the rest of your partitions. Make sure that you use the /dev/mdX device with header version 1.0 (in cat /proc/mdstat the device with super 1.0) for your root or boot partition that will contain /boot/.

root@rockpro64:~# mkfs.ext4 -L root /dev/md1
mke2fs 1.46.2 (28-Feb-2021)
Creating filesystem with 3669984 4k blocks and 917504 inodes
Filesystem UUID: 9cb9e959-f232-4a9b-93cf-24be379edef3
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done   

root@rockpro64:~# mkfs.ext4 -L data /dev/md3
mke2fs 1.46.2 (28-Feb-2021)
Creating filesystem with 52395776 4k blocks and 13099008 inodes
Filesystem UUID: a9924914-3532-4c2c-993b-596f5f6e37cf
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done     
#labels

Check that your partitions contain the labels used further down (or make sure you know what you're doing with paths or UUIDs).

root@rockpro64:~# blkid /dev/md*
/dev/md1: LABEL="root" UUID="9cb9e959-f232-4a9b-93cf-24be379edef3" BLOCK_SIZE="4096" TYPE="ext4"
/dev/md2: LABEL="swap" UUID="5674cbfd-e8f7-478a-8660-fb72c1e26cfd" TYPE="swap"
/dev/md3: LABEL="data" UUID="a9924914-3532-4c2c-993b-596f5f6e37cf" BLOCK_SIZE="4096" TYPE="ext4"

If you're missing labels you can add them to a formated partition by using swaplabel or e2label.

#mount software raids

Mount your filesystems on the software raid to /mnt.

I decided on a root partition for the system and a data partition containing the directories /opt, /var and /home to seperate operating system and user data, but at the same time to not segement the available space in too many partitions.

To use the data partition for the three directories under / I bind-mounted subdirectiories of /data to these directories. Initially I tried with symlinks on my yunohost, but that led to misfunctions.

root@rockpro64:~# mount LABEL=root /mnt
root@rockpro64:~# mkdir /mnt/opt /mnt/var /mnt/home /mnt/data
root@rockpro64:~# mount LABEL=data /mnt/data
root@rockpro64:~# mkdir /mnt/data/opt /mnt/data/var /mnt/data/home
root@rockpro64:~# mount --bind /mnt/data/home /mnt/home
root@rockpro64:~# mount --bind /mnt/data/var /mnt/var 
root@rockpro64:~# mount --bind /mnt/data/opt /mnt/opt

Check whether your new filesystems are mounted and sized as expected:

root@rockpro64:~# mount | grep /dev/md
/dev/md1 on /mnt type ext4 (rw,relatime,data=ordered)
/dev/md3 on /mnt/data type ext4 (rw,relatime,data=ordered)
/dev/md3 on /mnt/home type ext4 (rw,relatime,data=ordered)
/dev/md3 on /mnt/var type ext4 (rw,relatime,data=ordered)
/dev/md3 on /mnt/opt type ext4 (rw,relatime,data=ordered)
root@rockpro64:~# df -h /mnt /mnt/data
Filesystem      Size  Used Avail Use% Mounted on
/dev/md1         14G   40K   13G   1% /mnt
/dev/md3        196G   40K  186G   1% /mnt/data

#copy emmc to harddisk

Copy your running system to the partitions mounted on /mnt. If you add the option -v you'll get messages for every file that has been copied.

root@rockpro64:/mnt# rsync -aH --exclude='/dev/**' --exclude='/proc/**'  --exclude='/run/**'  --exclude='/sys/**' --exclude='/mnt/**' / /mnt

For the rsync command expect to get some error messages. You're trying to copy a running system and some files are opened, changing and/or locked.

You need to read the errors if you get some and decide for yourself.

If you're not sure you could disable services before rerunning the rsync-command which then will only copy changed files.

This way you can work your way forward to a copy that doesn't contain errors related to files you think are important.

Even if you do not get any error during rsync services might break if they write to their different files while rsync copies these at different points of time.

#check

Check whether the size of your data matches more or less:

root@rockpro64:/mnt# df -h / /mnt /mnt/data
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk1p1   28G  3.9G   24G  15% /
/dev/md1         14G  2.8G   11G  21% /mnt
/dev/md3        196G  1.2G  185G   1% /mnt/data

I had 3.9G on the eMMC I copied to a new / and /data partition. The new partitions add up to more or less the same amount of data.

Another way to check:

root@rockpro64:/mnt# du -xs / /mnt /mnt/data
4059992	/
2838648	/mnt
1220820	/mnt/data
root@rockpro64:/mnt# bc
bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 
2838648+1220820
4059468

#new fstab for your software raid

Write a new fstab to your new partitions:

root@rockpro64:~# echo 'LABEL=root / ext4 defaults,errors=remount-ro 0 1
LABEL=data /data ext4 defaults 0 2
/data/opt /opt none defaults,bind 0 0
/data/var /var none defaults,bind 0 0
/data/home /home none defaults,bind 0 0
LABEL=swap swap swap defaults 0 4              
tmpfs /tmp tmpfs defaults,nosuid 0 0' > /mnt/etc/fstab

Make sure that your fstab contains only labels you can find using blkid on your software raid partitions.

If you changed the partition layout during this guide your fstab needs reflect these changes.

#armbianEnv.txt

armbianEnv.txt is used by the boot script for u-boot to set the u-boot environment. It contains a reference to the root partition to generate the kernel command lines parameter root=<partition>.

You need to change rootdev to your software raid root partition using your favourite editor.

After changing it my armbianEnv.txt looked like this:

root@rockpro64:~# cat /mnt/boot/armbianEnv.txt
bootlogo=false
overlay_prefix=rockchip
rootdev=LABEL=root
rootfstype=ext4
console=none
verbosity=4
usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u

Take care that the content of rootdev really points to your new root partition in case you changed the partitioning or naming used during this guide.

#install u-boot to SPI

I tried with various guides and different u-boot versions without success. In the end I learned more about u-boot and put together snippets of information from different howtos to get to the following. In my notes you can find stuff I tried. The comments are in german, but the terminal cut'n'paste speaks for itself in many places. At the end of those notes there are the websites I used to gather the knowledge for the following process.

#create image to copy to SPI

The u-boot image copied to SPI is different from the image used for eMMC or sdcard. It is not contained in the debian package we unpacked earlier.

You'll need to put it together from these three pieces: u-boot-tpl.bin, u-boot-spl.bin and u-boot.itb:

root@rockpro64:~# mkimage -n rk3399 -T rkspi -d /root/u-boot-rockchip_2023.07+dfsg-1_arm64.deb.d/usr/lib/u-boot/rockpro64-rk3399/u-boot-tpl.bin:/root/u-boot-rockchip_2023.07+dfsg-1_arm64.deb.d/usr/lib/u-boot/rockpro64-rk3399/u-boot-spl.bin idbloader-spi.bin
Image Type:   Rockchip RK33 (SPI) boot image
Init Data Size: 71680 bytes
Boot Data Size: 114688 bytes
root@rockpro64:~# cp idbloader-spi.bin spi.img

To get some background, why you had to copy a spl (secondary program loader) and a tpl (tertiary program loader) you could read 3 minutes into this presentation.

After that there's a gap that is filled with 0xff. You need to append these (0x00 though works the same) to your spi.img:

root@rockpro64:~# yes $'\xff' | tr -d $'\x0a' | dd of=spi.img bs=1024 seek=368 count=16
16+0 records in
16+0 records out
16384 bytes (16 kB, 16 KiB) copied, 0.00142332 s, 11.5 MB/s

Now you need to add in some firmware and information about the hardware:

root@rockpro64:~# dd if=/root/u-boot-rockchip_2023.07+dfsg-1_arm64.deb.d/usr/lib/u-boot/rockpro64-rk3399/u-boot.itb of=spi.img bs=1024 seek=384
1048+0 records in
1048+0 records out
1073152 bytes (1.1 MB, 1.0 MiB) copied, 0.0311938 s, 34.4 MB/s

You can look at the data contained in u-boot.itb using dumpimage -l /root/u-boot-rockchip_2023.07+dfsg-1_arm64.deb.d/usr/lib/u-boot/rockpro64-rk3399/u-boot.itb

Since you started with the same data like me and we followed the same steps, we should have the same spi.img file:

root@rockpro64:~# sha256sum spi.img 
3d85ed0dacad20c48fb6938813eb80829175113f7f97e4a8012dbaa89f8f56bc  spi.img

#copy image to SPI

First erase your SPI that shouldn't contain any meaningful data if you didn't use it, yet. If you're unsure, make a backup copy using dd if=/dev/mtd0 of=spi.backup.img bs=1024.

root@rockpro64:~# flash_erase /dev/mtd0 0 0
Erasing 4 Kibyte @ 3f7000 -- 100 % complete

Check if you really emptied it:

root@rockpro64:~# dd if=/dev/mtd0 bs=1024 | hexdump
0000000 ffff ffff ffff ffff ffff ffff ffff ffff
*
03f8000
4064+0 records in
4064+0 records out
4161536 bytes (4.2 MB, 4.0 MiB) copied, 4.96204 s, 839 kB/s

Copy your spi.img to SPI:

root@rockpro64:~# flashcp -v spi.img /dev/mtd0
Erasing blocks: 358/358 (100%)
Writing data: 1432k/1432k (100%)
Verifying data: 1432k/1432k (100%)

#create u-boot environment

You would be ready to go if that eMMC/sdcard wouldn't still be in your RockPro64.

In the standard configuration of your u-boot version eMMC and sdcard take precedence over the harddrives. If you want to shutdown remove your sdcard/eMMC from the RockPro64 it should start from harddisks already.

If you prefer not to touch your hardware or you simply can't (like me), then you'll need to prepare an environment for u-boot which gets written to the SPI as well.

The only thing you'll need to change compared to the default environment is boot_targets. scsi needs to precede mmc. The default looks like this (copied from the u-boot cli):

=> env print
arch=arm
baudrate=1500000
board=rockpro64_rk3399
board_name=rockpro64_rk3399
boot_targets=mmc1 mmc0 nvme scsi usb pxe dhcp spi
bootcmd=bootflow scan
bootdelay=2
cpu=armv8
cpuid#=5447465333332e30300000000009098a
eth1addr=06:93:08:b5:fc:e7
ethaddr=06:93:08:b5:fc:e6
fdt_addr_r=0x01f00000
fdtcontroladdr=f1f10300
fdtfile=rockchip/rk3399-rockpro64.dtb
fdtoverlay_addr_r=0x02000000
kernel_addr_r=0x02080000
kernel_comp_addr_r=0x08000000
kernel_comp_size=0x2000000
loadaddr=0x800800
partitions=uuid_disk=${uuid_gpt_disk};name=loader1,start=32K,size=4000K,uuid=${uuid_gpt_loader1};name=loader2,start=8MB,size=4MB,uuid=${u;
preboot=usb start
pxefile_addr_r=0x00600000
ramdisk_addr_r=0x06000000
script_offset_f=0xffe000
script_size_f=0x2000
scriptaddr=0x00500000
serial#=109cb483ce05242c
soc=rk3399
stderr=serial,vidconsole
stdin=serial,usbkbd
stdout=serial,vidconsole
vendor=pine64

Environment size: 1043/32764 bytes

Initially I thought it would be enough to write just the changed boot_targets to the environment, but then only a few variables are filled with valued that differ from board to board (like mac or serial number). This wouldn't boot, because of a missiing bootcmd.

You need to create a txt file containing your environment like this:

root@rockpro64:~# cat env.txt 
arch=arm
baudrate=1500000
board=rockpro64_rk3399
board_name=rockpro64_rk3399
boot_targets=scsi mmc1 mmc0 nvme usb pxe dhcp spi
bootcmd=bootflow scan
bootdelay=2
cpu=armv8
fdt_addr_r=0x01f00000
fdtfile=rockchip/rk3399-rockpro64.dtb
fdtoverlay_addr_r=0x02000000
kernel_addr_r=0x02080000
kernel_comp_addr_r=0x08000000
kernel_comp_size=0x2000000
loadaddr=0x800800
partitions=uuid_disk=${uuid_gpt_disk};name=loader1,start=32K,size=4000K,uuid=${uuid_gpt_loader1};name=loader2,start=8MB,size=4MB,uuid=${uuid_gpt_loader2};name=trust,size=4M,uuid=${uuid_gpt_atf};name=boot,size=112M,bootable,uuid=${uuid_gpt_boot};name=rootfs,size=-,uuid=B921B045-1DF0-41C3-AF44-4C6F280D3FAE;
preboot=usb start
pxefile_addr_r=0x00600000
ramdisk_addr_r=0x06000000 
script_offset_f=0xffe000
script_size_f=0x2000
scriptaddr=0x00500000
soc=rk3399 
stderr=serial,vidconsole
stdin=serial,usbkbd
stdout=serial,vidconsole
vendor=pine64

The only value changed is boot_targets. The missing values are automatically filled in by u-boot during boot.

From your env.txt file you need to create an image you can write to the environment flash:

root@rockpro64:~# mkenvimage -s 0x8000 -o env.img env.txt

Again: You used the same software and data so your image should be the same like mine:

root@rockpro64:~# sha256sum env.img
fc4a6aebbb805008ee92981e8893748d925931f1758df384dbe5ccef0d40cc8e  env.img

#copy u-boot environment to SPI

In this configuration the SPI area for the u-boot environment is presented by the OS as /dev/mtd1.

You need to copy your env.img to that device:

root@rockpro64:~# flashcp -v env.img /dev/mtd1
Erasing blocks: 8/8 (100%)
Writing data: 32k/32k (100%)
Verifying data: 32k/32k (100%)

Now that you've initialized your u-boot environment you can check and correct it using fw_printenv and fw_setenv. First you need to make a configuration file for these tools.

root@rockpro64:~# echo '# MTD device name       Device offset   Env. size       Flash sector size       Number of sectors
/dev/mtd1               0x0             0x8000' > /etc/fw_env.config

Check that the content is correct:

root@rockpro64:~# cat /etc/fw_env.config
# MTD device name       Device offset   Env. size       Flash sector size       Number of sectors
/dev/mtd1               0x0             0x8000

Now you can use fw_printenv to show the environment saved in SPI or fw_setenv to change it:

root@rockpro64:~# fw_printenv 
arch=arm
baudrate=1500000
board=rockpro64_rk3399
board_name=rockpro64_rk3399
boot_targets=mmc1 mmc0 nvme usb pxe dhcp spi
bootcmd=bootflow scan
bootdelay=2
cpu=armv8
fdt_addr_r=0x01f00000
fdtfile=rockchip/rk3399-rockpro64.dtb
fdtoverlay_addr_r=0x02000000
kernel_addr_r=0x02080000
kernel_comp_addr_r=0x08000000
kernel_comp_size=0x2000000
loadaddr=0x800800
partitions=uuid_disk=${uuid_gpt_disk};name=loader1,start=32K,size=4000K,uuid=${uuid_gpt_loader1};name=loader2,start=8MB,size=4MB,uuid=${uuid_gpt_loader2};name=trust,size=4M,uuid=${uuid_gpt_atf};name=boot,size=112M,bootable,uuid=${uuid_gpt_boot};name=rootfs,size=-,uuid=B921B045-1DF0-41C3-AF44-4C6F280D3FAE;
preboot=usb start
pxefile_addr_r=0x00600000
ramdisk_addr_r=0x06000000 
script_offset_f=0xffe000
script_size_f=0x2000
scriptaddr=0x00500000
soc=rk3399 
stderr=serial,vidconsole
stdin=serial,usbkbd
stdout=serial,vidconsole
vendor=pine64

Uhm, I'd better correct that one.

root@rockpro64:~# fw_setenv boot_targets "scsi mmc1 mmc0 nvme usb pxe dhcp spi"
root@rockpro64:~# fw_printenv | grep boot_targets
boot_targets=scsi mmc1 mmc0 nvme usb pxe dhcp spi

This is the way you can change your u-boot environment without access to the serial console and without writing each time a new environment image to flash.

Done. That's it.

#reboot into the harddisk installation

Reboot.

root@rockpro64:~# reboot
root@rockpro64:~# Connection to RockPro64Armbian closed by remote host.

Now you should check that you really run off the harddisks.

root@rockpro64:~# mount | grep /dev/md
/dev/md127 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/md126 on /data type ext4 (rw,relatime,data=ordered)
/dev/md126 on /home type ext4 (rw,relatime,data=ordered)
/dev/md126 on /opt type ext4 (rw,relatime,data=ordered)
/dev/md126 on /var type ext4 (rw,relatime,data=ordered)
/dev/md126 on /var/log.hdd type ext4 (rw,relatime,data=ordered)
/dev/md126 on /data/var/log.hdd type ext4 (rw,relatime,data=ordered)

All mounts for my system are coming from /dev/md*. What about your eMMC or your sdcard?

root@rockpro64:~# fdisk -l /dev/mmcblk1
/dev/mmcblk1       /dev/mmcblk1boot0  /dev/mmcblk1boot1  /dev/mmcblk1p1     /dev/mmcblk1rpmb   
root@rockpro64:~# fdisk -l /dev/mmcblk1
Disk /dev/mmcblk1: 29.12 GiB, 31268536320 bytes, 61071360 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc40ef599

Device         Boot Start      End  Sectors  Size Id Type
/dev/mmcblk1p1      32768 60456959 60424192 28.8G 83 Linux
root@rockpro64:~# mount | grep mmcblk1
root@rockpro64:~# 

It is not used.

#Questions, improvements, remarks

If you

  • find errors or mistakes
  • think you have an improvement
  • want to add something
  • have a question

Please feel free to open an issue on sourcehut.

Merge request are welcome!

#License of this howto

Creative Commons Attribution 3.0

Do not follow this link