Lessons Learnt on Breaking and Reinstalling Arch Linux on Raspberry Pi

Posted on Views:

I'm a big fan of Arch Linux for its simplicity, and ease to use. However, sometimes it can be the source of headache. This week, I upgraded the Arch Linux on my Raspberry Pi 3 with latest packages with the following command yay -Syu --noconfirm (WARNING: you shouldn't be doing this). When I finished upgrade, my Raspberry Pi 3 stopped working. And I had to spend quite a lot of time getting things back to normal.

Initial Investigation and Thoughts

Since I don't have any clue on what happened at all. I first tried whether SSH still works... And I got can't find zsh. That is indeed a bad sign, I can't do anything 😞. Next thing I did was reboot. And things got even worse -- I can't even SSH now, refused to connect at port 22 😢.

At this point, I had no choice but to connect the Pi to an external monitor. It didn't help. While I could see the bootloader start screen, the kernel seems broken, so machine did not start.

Attempted Solution with Chroot

Next, my plan is to pull the MicroSD card out and try if I can get chroot to work. I am expecting this to be difficult cause Raspberry Pi is ARM based, and my laptop is not. It turned out to be really difficult because I wasn't familiar with how chroot and mounting works.

To chroot into an ARM-based machine (in this case aarch64 architecture), you would normally need an ARM based machine. If you really want to do it from a x86_64 architecture machine, you would have to setup some kind of emulation. In this case, I'm using QEMU, which is a pretty standard tool for emulation on Linux. Here's the steps I followed:

  • Install QEMU related packages:
    1
    sudo pacman -S qemu-user-static qemu-user-static-binfmt
    These two packages are both necessary. According to ArchLinux Wiki, the first qemu-user-static provides user mode emulation, this would avoid the need to setup a full blown virtual machine to run the binaries. Instead, it would run the binary with static translation.
  • You would also need to install some cross-compiling libraries:
    1
    sudo pacman -S aarch64-linux-gnu-glibc
  • Insert your MicroSD card to your machine, mount the /boot partition and the / (root) partition, can be something like below, you would need to run as root:
    1
    2
    sudo mount --mkdir /dev/sdX2 /mnt/sdcard/
    sudo mount --mkdir /dev/sdX1 /mnt/sdcard/boot
  • Once this is done, you can follow the steps here. I went with the arch-chroot route:
    1
    sudo arch-chroot /mnt/sdcard
    However, I kept getting /lib/ld-linux-aarch64.so.1 does not exist, when I ran the command above. Initially, I was thinking it was my host machine that has the issue, so I copied the file ld-linux-aarch64.so.1 from /usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1. And it still didn't work. I had spent almost one hour on this before I realized, the issue lies with my chroot and not my host machine. I had to do create a symbolic link from /usr/lib to /lib in the chroot (i.e. the sdcard file system). This time, I got segmentation fault when I did the chroot.

At this point, I might as well reinstall the whole ArchLinux. But before that I did a backup using the rsync:

1
2
# backup is important!
sudo rsync -av /mnt/sdcard ~/rpi_bak

Reinstallation of ArchLinux on Raspberry

Reinstalling the ArchLinux on Raspberry Pi 3 actually felt easier than on x86_64 machine(because it is actually ArchLinux Arm which is a port of ArchLinux for Arm devices). You can follow the guide from ArchLinux Arm. I would recommend setting the boot partition to be more than 200MB, 400MB would be ideal.

Once you are done with installation, I would recommend not to put your MicroSD card back to your Raspberry yet. Instead, you should chroot into the mounted folder and start installing some packages. If you are following the guide from ArchLinux Arm, you can do:

1
2
3
4
5
# mount boot and root again if you have unmounted
sudo mount /dev/sdX2 ./root
# this ensures the boot drive is properly mounted.
sudo mount /dev/sdX1 ./root/boot
sudo arch-chroot ./root

You are now acting as the root user for the newly initiated ArchLinux installation. You can start by populating the keys:

1
2
pacman-key --init
pacman-key --populate archlinuxarm

Once you are done with that, you would have the chance to install some necessary software, I would recommend at least install sudo, git, wget, zsh. For me, I copied all my files from my previous installation, so:

1
rsync -av ~/rpi_bak/sdcard/home/alarm/ ~/rpi-staging/root/home/alarm/

Once this is done, I thought I'm ready to put this MicroSD to my Raspberry Pi and things would get back to normal. However, things still didn't work as expected: the machine booted, but it had no Ethernet or WiFi access. I also noticed the lights were blinking wrongly. This made me think the whole situation had something to do with kernel. So, I had to plug the MicroSD back to my laptop and switch the kernel and install the linux-rpi package.

1
sudo pacman -S linux-rpi

This package is RPi Foundation patched Linux kernel and modules which turned out to solve the issue. You would be asked if you want to replace the current kernel and the u-boot. Go ahead and say yes. Now, if you had followed the official guide from ArchLinux Arm, you would get an error, because 200MB on the boot drive is not enough for installing a second kernel package when the current kernel is present (this is the reason I recommend using more than 200MB on boot partition). For me, I had to delete some files from the /boot/dtbs folder (which I don't recommend doing). This folder is for device tree blobs, so there might be unexpected consequences for deleting random files.

After rebooting and start using the kernel from RPi Foundation, everything would work as expected. I continued with some housekeeping tasks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# install cron
sudo pacman -S cronie && sudo systemctl enable --now cronie

# set the hostname
hostnamectl hostname raspberrypi

# set timezone
sudo timedatectl set-timezone Pacific/Auckland

# add the default user to sudoer
sudo visudo
# Add the following lines:
# alarm ALL=(ALL:ALL) ALL
# Defaults:alarm !authenticate

# set locale to utf-8
localectl set-locale LANG=en_US.UTF-8

I also compiled the aur helper from my laptop and transferred it RPi -- it just takes forever to compile things on RPi. For doing this you would need to get the PKGBUILD file for the package you want to build and run: CARCH=aarch64 makepkg -Cf

Lessons Learnt

Honestly, I would rather not go through this process again. I did learnt a lot though:

  • Don't ever run things with --no-confirm, it is so dangerous, especially on a rolling release distro like Arch
  • I was surprised I can run chroot on the aarch64 with x86_64 machine, this turned out to be the magic of qemu-user-static and qemu-user-static-binfmt. The binfmt is particularly interesting, Wikipedia would provide better explanation than I do. Basically, Linux knows to use aarch64 related emulation when you are inside the chroot -- there's a mapping from the binary format to the interpreter at runtime. For example, I had this /usr/lib/binfmt.d/qemu-aarch64-static.conf file which contains
    :qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:FP
    This would allow QEMU execute programs for other processor architectures as if they were native binaries.
  • Setting up a system through chroot is much easier than doing things over SSH, this is especially true if you are dealing with Raspberry Pi or something that's not powerful enough.

References:

https://wiki.archlinux.org/title/QEMU#Details_on_packages_available_in_Arch_Linux
https://wiki.archlinux.org/title/QEMU#Chrooting_into_arm/arm64_environment_from_x86_64
https://archlinuxarm.org/platforms/armv8/broadcom/raspberry-pi-3
https://en.wikipedia.org/wiki/Binfmt_misc
https://github.com/qemu/qemu/blob/f31160c7d1b89cfb4dd4001a23575b42141cb0ec/qemu-doc.texi#L2590