Feature #447

Need upgrade path

Added by Hammel over 4 years ago. Updated 9 months ago.

Status:In ProgressStart date:15 Jun 2015
Priority:ImmediateDue date:
Assignee:Hammel% Done:


Target version:1.1.0 - Upgrades
Severity:03 - Medium


There is no way to upgrade a system. I need one.

Plan A

Create 4 partitions:
  1. p0: FAT/boot
  2. p1: EXT4/Minimal update
  3. p2: EXT4/Runtime
  4. p3: EXT4/User Data w/update files area

An update causes a reboot into p1 which does the update to p2 and then reboots into P2. This takes the system out of action during the update but probably not for long as long as the update image has already been downloaded.

This also assumes that the update is a complete rewrite of the boot partition. That may not be desirable.

Plan B

Always use opkg's, even for files already on the rootfs.

Create 3 partitions:
  1. p0: FAT/boot
  2. p1: EXT4/Runtime
  3. p2: EXT4/User Data w/update files area

For example, a kernel update would be downloaded as an opkg and installed over the existing kernel (essentially). This requires postinst processing to switch to the new kernel (copy to p0).

The question is how will the system behave when not all files are initially package controlled. Replacing system files with opkg files may present problems.

Overview of the image and updates

The image build (re: rootfs) sits on a two partitions. The first is a vfat partition required by the bootloader. The second is the rootfs. Both are images copied byte-for-byte to their respective partitions in order to create the boot SD card. An update process can either update specific packages (of which there are none for the base platform) or it can replace the partition image. The latter is often required in embedded systems and might be more appropriate for PiBox unless PiBox switches to a fully package managed system.

An update to the rootfs can be done in place. A secondary rootfs partition can be installed and the boot process pointed at it so the next boot uses it. This requires a mechanism to fallback to the original partition if something fails in the new partition. A key sequence or other extenal impetus at boot time would be required to force the fallback.

The boot partition can't use an alternate. So files have to be where we expect them. The boot partition also doesn't use symlinks so we can't place alternates in directories and just update the symlinks. This implies that we have two directories, current and new, that contain the data to boot. Within that data is a kernel with an initramfs capable of replacing the booting kernel with the older version if the boot process fails.

The rootfs update is not overly difficult. It requires an extra boot script that validates the running system and resets if anything goes wrong. This verification tests that all components are online correctly, specifically wireless and USB support.

The boot partition update is more complex since it requires an error checking mechanism built into the initramfs. This error checking would be to verify that all drivers are available and can load without error.

Associated revisions

Revision d7a71139
Added by Hammel 7 months ago

RM #447: Bump Busybox as part of initramfs generation in preparation for upgrade management changes.


#1 Updated by Hammel almost 3 years ago

  • Description updated (diff)
  • Category set to Support
  • Assignee set to Hammel
  • Priority changed from Normal to Immediate
  • Target version set to 0.12.0

#3 Updated by Hammel over 1 year ago

  • Description updated (diff)
  • Status changed from New to In Progress
  • % Done changed from 0 to 10

There are a couple of related solutions to consider.

Bank Switching

This is like Plan A except we use full images in each bank. Bank A is the initial install. Bank B is the first update. We download the update into Bank B, while Bank A is running, and validate it. When ready, we swap to Bank B. Swapping could be done with a reboot but it would be faster (and less error prone?) to do a kexec to a new kernel (which might be in the second bank). Note that since the new kernel is being loaded without booting from the VFAT partition (as Pi's demand) we'd need to copy in required files to the VFAT partition after the swap, but only after additional/final validation says this will be okay.

Overlay Filesystems

This would be layers of data, with the lowest being the core image and higher layers holding, in order from bottom to top:
  • Graveyard - deleted files
  • RW - updates and modifications to the core image, including end user customizations
  • presentation - what we see at run time

This also requires a kexec and has the same issues with the VFAT partition as the Bank Switching solution.

Additional Comments

  • Validation requires checksuming downloaded data to start, but goes as far as verifying operational status after kexec'ing and restarting the platform via the init processing.
  • Rollback capability is required in case the update fails.
  • There may need to be a small kernel/initramfs specifically designed to handle the update process only. This allows us to have a constant tool for upgrades irrespective of operational kernel updates. The update kernel/initramfs would, after update, kexec into the latest kernel image.

#4 Updated by Hammel 9 months ago

  • Target version changed from 0.12.0 to 1.1.0 - Upgrades

#5 Updated by Hammel 9 months ago

To get started using overlays with squashfs root file system images I need to add an initramfs to the boot process. This requires
  1. Building an initramfs
  2. Configuring the boot to use it.

Building the initramfs

The kernel build could do this for us but it's probably easier to generate a basic Busybox and populate it with a custom init script that mounts partitions and sets up the overlays. This would need to be added as a new Busybox target in the build system and the new target added to the default set of top level targets.

This design requires the kernel to support squashfs and overlays. These can be compiled into the kernel for simplicity, otherwise the drivers need to be stuffed into the initramfs. Currently both are enabled as modules in the kernel build.

The root file system will need to be converted from tar to squashfs format. This is supported by Buildroot in the Filesystem Images submenu.

Configuring boot

The config.txt needs a new option: initramfs. This points to the file to use and the memory location where it should be loaded. To simplify, you can specify to load it after the kernel.

 initramfs <filename> followkernel

The initramfs image will need to be stored in the FAT partition on the SD card with the Linux kernel.

Filesystem layout

The root Linux partition will now be divided into the following structure:
  1. /overlay - Where overlay directories live.
    1. /overlay/presentation - what we see; the combined view
    2. /overlay/upper - where runtime root file system changes are saved; userdata (application runtime data) is not stored in the root filesystem but rather in /userdata.
    3. /overlay/work - used by the overlay driver
    4. /overlay/backups - used for upgrades and resets
  2. /squashfs - Where the squashfs root filesystem images are stored
  3. /utils - Utilities required to manage bank switching/overlays/squashfs
  4. /userdata - Application save data is stored here. It is bind mounted into the presentation layer.

Bank switching would imply two separate upper directories, one for each bank. This might be the best solution. A new release gets a new squashfs and a copy of the old upper directory. If the new version fails self-check (which needs to be implemented as an init script function) then we swap back to the old version. The alternative is to backup the current upper directory and, if the new version has runtime failures, restore it and revert to the old version.

Filesystem images need to be signed. Signing verification tools go in /utils. I'm not sure how to do this yet. Other utilities, including release-specific tools will be placed in the utils directory. This will allow the init script to be updated on the fly by dropping new utilities specific to the new release in /utils and having the init script find and then run them as part of its processing. Anything dropped in /utils has to be signed and is probably an archive that is unpacked by the init script for use.

Also available in: Atom PDF