Feature #447

Need upgrade path

Added by Hammel about 5 years ago. Updated 9 months ago.

Status:ClosedStart date:15 Jun 2015
Priority:ImmediateDue date:
Assignee:Hammel% Done:


Target version:1.1.0 - Upgrades
Severity:03 - Medium


There is no way to upgrade a system. I need one.

Plan A

Create 4 partitions:
  1. p0: FAT/boot
  2. p1: EXT4/Minimal update
  3. p2: EXT4/Runtime
  4. p3: EXT4/User Data w/update files area

An update causes a reboot into p1 which does the update to p2 and then reboots into P2. This takes the system out of action during the update but probably not for long as long as the update image has already been downloaded.

This also assumes that the update is a complete rewrite of the boot partition. That may not be desirable.

Plan B

Always use opkg's, even for files already on the rootfs.

Create 3 partitions:
  1. p0: FAT/boot
  2. p1: EXT4/Runtime
  3. p2: EXT4/User Data w/update files area

For example, a kernel update would be downloaded as an opkg and installed over the existing kernel (essentially). This requires postinst processing to switch to the new kernel (copy to p0).

The question is how will the system behave when not all files are initially package controlled. Replacing system files with opkg files may present problems.

Overview of the image and updates

The image build (re: rootfs) sits on a two partitions. The first is a vfat partition required by the bootloader. The second is the rootfs. Both are images copied byte-for-byte to their respective partitions in order to create the boot SD card. An update process can either update specific packages (of which there are none for the base platform) or it can replace the partition image. The latter is often required in embedded systems and might be more appropriate for PiBox unless PiBox switches to a fully package managed system.

An update to the rootfs can be done in place. A secondary rootfs partition can be installed and the boot process pointed at it so the next boot uses it. This requires a mechanism to fallback to the original partition if something fails in the new partition. A key sequence or other extenal impetus at boot time would be required to force the fallback.

The boot partition can't use an alternate. So files have to be where we expect them. The boot partition also doesn't use symlinks so we can't place alternates in directories and just update the symlinks. This implies that we have two directories, current and new, that contain the data to boot. Within that data is a kernel with an initramfs capable of replacing the booting kernel with the older version if the boot process fails.

The rootfs update is not overly difficult. It requires an extra boot script that validates the running system and resets if anything goes wrong. This verification tests that all components are online correctly, specifically wireless and USB support.

The boot partition update is more complex since it requires an error checking mechanism built into the initramfs. This error checking would be to verify that all drivers are available and can load without error.

Associated revisions

Revision d7a71139
Added by Hammel over 1 year ago

RM #447: Bump Busybox as part of initramfs generation in preparation for upgrade management changes.

Revision b9dfcb6b
Added by Hammel 9 months ago

RM #447: Fix mksd and mkinstall to properly create an image file that can be used with QEMU.

Revision 8a1fdc5e
Added by Hammel 9 months ago

RM #447: Add qemu startup script. It's just a work in progress, since QEMU doesn't seem to work right with RPi2.

Revision 68bfffa2
Added by Hammel 9 months ago

RM #447: Add squashfs to generated rootfs image types. Add -o option to generate overlay structured rootfs on image device or file. Add qemu.sh script for testing images under QEMU.

Revision 97bf777d
Added by Hammel 9 months ago

RM #447: Minor updates to qemu script. This just doesn't work with PiBox yet for unknown reasons.

Revision a316a927
Added by Hammel 9 months ago

RM #447: Build busybox to be static so it can be used in initramfs. Update qemu.sh to use initrd. Integrate initial pass at init script for overlay setup.

Revision f37a18cb
Added by Hammel 9 months ago

RM #447: Update initramfs to properly mount the squashfs into an overlay. Removed skeleton since it isn't necessary (we don't use it for Buildroot anymore).

Revision 429d7e71
Added by Hammel 9 months ago

RM #447: Make xterm privileged by default because it's almost always needed that way for now.

Revision b8078c99
Added by Hammel 9 months ago

RM #447: Quiet blockhandler output to console (still goes to logs).

Revision efba5ab1
Added by Hammel 9 months ago

RM #447: Separate busybox configs for dev platform vs initramfs.

Revision 9281f16d
Added by Hammel 9 months ago

RM #447: Extend install scripts (and qemu test tool) to support installing to file or SD card and either traditional rootfs or overlay structured.

Revision fed2a272
Added by Hammel 9 months ago

RM #447: Allow stopping at the end of firstboot instead of rebooting by adding firstboot=noreboot to kernel command line.

Revision 85b2fa70
Added by Hammel 9 months ago

RM #447: Run mdev in firstboot so that pibox-config can be properly configured for the display device.

Revision c121e687
Added by Hammel 9 months ago

RM #447: Add pdebug, pstep to allow console output and stepping in init script. Add missing migration from initramfs to rootfs of /dev, /proc, /sys.


#1 Updated by Hammel over 3 years ago

  • Description updated (diff)
  • Category set to Support
  • Assignee set to Hammel
  • Priority changed from Normal to Immediate
  • Target version set to 0.12.0

#3 Updated by Hammel over 2 years ago

  • Description updated (diff)
  • Status changed from New to In Progress
  • % Done changed from 0 to 10

There are a couple of related solutions to consider.

Bank Switching

This is like Plan A except we use full images in each bank. Bank A is the initial install. Bank B is the first update. We download the update into Bank B, while Bank A is running, and validate it. When ready, we swap to Bank B. Swapping could be done with a reboot but it would be faster (and less error prone?) to do a kexec to a new kernel (which might be in the second bank). Note that since the new kernel is being loaded without booting from the VFAT partition (as Pi's demand) we'd need to copy in required files to the VFAT partition after the swap, but only after additional/final validation says this will be okay.

Overlay Filesystems

This would be layers of data, with the lowest being the core image and higher layers holding, in order from bottom to top:
  • Graveyard - deleted files
  • RW - updates and modifications to the core image, including end user customizations
  • presentation - what we see at run time

This also requires a kexec and has the same issues with the VFAT partition as the Bank Switching solution.

Additional Comments

  • Validation requires checksuming downloaded data to start, but goes as far as verifying operational status after kexec'ing and restarting the platform via the init processing.
  • Rollback capability is required in case the update fails.
  • There may need to be a small kernel/initramfs specifically designed to handle the update process only. This allows us to have a constant tool for upgrades irrespective of operational kernel updates. The update kernel/initramfs would, after update, kexec into the latest kernel image.

#4 Updated by Hammel over 1 year ago

  • Target version changed from 0.12.0 to 1.1.0 - Upgrades

#5 Updated by Hammel over 1 year ago

To get started using overlays with squashfs root file system images I need to add an initramfs to the boot process. This requires
  1. Building an initramfs
  2. Configuring the boot to use it.

Building the initramfs

The kernel build could do this for us but it's probably easier to generate a basic Busybox and populate it with a custom init script that mounts partitions and sets up the overlays. This would need to be added as a new Busybox target in the build system and the new target added to the default set of top level targets.

This design requires the kernel to support squashfs and overlays. These can be compiled into the kernel for simplicity, otherwise the drivers need to be stuffed into the initramfs. Currently both are enabled as modules in the kernel build.

The root file system will need to be converted from tar to squashfs format. This is supported by Buildroot in the Filesystem Images submenu.

Configuring boot

The config.txt needs a new option: initramfs. This points to the file to use and the memory location where it should be loaded. To simplify, you can specify to load it after the kernel.

 initramfs <filename> followkernel

The initramfs image will need to be stored in the FAT partition on the SD card with the Linux kernel.

Filesystem layout

The root Linux partition will now be divided into the following structure:
  1. /overlay - Where overlay directories live.
    1. /overlay/current - Symlink to the currently active image
    2. /overlay/next - Symlink to the next (upgrade) image
    3. /overlay/image_id - a specific release image
      1. /overlay/image_id/presentation - what we see; the combined view
      2. /overlay/image_id/upper - where runtime root file system changes are saved; userdata (application runtime data) is not stored in the root filesystem but rather in /userdata.
      3. /overlay/image_id/work - used by the overlay driver
      4. /overlay/image_id/backups - temporary storage used during upgrades and resets
      5. /overlay/image_id/squashfs - The squashfs root filesystem for this image release
      6. /overlay/image_id/utils - Utilities required to manage bank switching/overlays/squashfs
  2. /userdata - Application save data is stored here. It is bind mounted into the presentation layer of the current image. Application data has to be migrated to new formats, as needed, by the applications on first boot.

Bank switching would imply two banks: current and next. A new release gets a new squashfs and a copy of the old upper directory, which may have package updates or similar that we want to propagate to the next release. If the new version fails self-check (which needs to be implemented as an init script function) then we swap back to the old version.

Filesystem images need to be signed. Signing verification tools go in /utils. I'm not sure how to do this yet. Other utilities, including release-specific tools will be placed in the utils directory. This will allow the init script to be updated on the fly by dropping new utilities specific to the new release in /utils and having the init script find and then run them as part of its processing. Anything dropped in /utils has to be signed and is probably an archive that is unpacked by the init script for use.

Boot Process

The initramfs mounts raw partitions under /boot (vfat partition) and /root (ext partition) from the SD card. The /root partition is structured with the overlay directories. The initramfs uses these directories to mount the currently active image as the runtime environment.

The real root and boot partitions are made accessible through a bind mount into the overlay's presentation layer by the initramfs.

Image Upgrades

An image can be downloaded and upacked to a directory under /overlay on the real root partition. Symlinks are updated so the next boot will start the new image. The image must have a self-test included on firstboot. If the self-test fails then the boot sets up a fallback to the old image and reboots.

If an image boots successfully it will check for a next tree and clean it up.

#6 Updated by Hammel 9 months ago

Minor update: I tried to get QEMU to work the RPi2 build. There are instructions for running QEMU specific to RPi2 which I've scripted. It boots up to a login prompt but I can't get any keyboard input.

I would really like to get QEMU working with this because it will make testing the initramfs easier than having to write an SD card and install it to a board. The latter isn't that hard but then I need power, a display, a keyboard, etc. With QEMU I can do it from my dev box.

Still reviewing options here. There are other generic ARM types I may be able to use. I know, however, that the aarch64 QEMU doesn't work with my RPi2 build even though that build does run on real hardware just fine.

#7 Updated by Hammel 9 months ago

  • % Done changed from 10 to 20

I've written an initial init script with all the basic functionality required for mounting the overlay, and I've also updated the mksd/mkinstall script to properly generate an image with the overlay directory structure along with the newly generated squashfs rootfs image. Without the initramfs I can boot to a login with the image. I was able to build an initramfs with it and boot qemu with it far enough to tell what's not working in the early stages of the script. I still don't have keyboard input in qemu so this kind of testing will be of limited value, but it will save some time before I need to move to real hardware.

Current updates are committed and pushed. Next up is to begin debugging the init script early stages. Using "step" as a kernel parameter lets me step through the script but there are some early function errors that need to be cleaned up before I will want to do that. Once I get the early function fixes in I will need to switch to real hardware in order to be able to type at the keyboard.

#8 Updated by Hammel 9 months ago

Last night I got the init script working to the point of mounting the overlay and booting PiBox. The boot process stops before it gets to a login for some reason but I'm not particularly worried about that yet. It may be an artifact of how I'm booting in QEMU. I'll know more when I try on real hardware, which is pretty much required now that I have the overlay mounting cleanly.

There is one more step in the init script to complete: bind mounting real partitions into the overlay. I have two choices here.

First, I can keep the mounts I have in the init script and bind them to the overlay, then disable their mounting by mdev in the rootfs.

Second, I can skip the boot and userdata mounts in the init script (they aren't needed to get the overlay running) and let mdev mount the partitions as it currently does.

The second method seems simpler so I'll try that first.

#9 Updated by Hammel 9 months ago

Quick test of the image on real hardware: it boots fine up to getting a login. Login does work. Mount points look okay - userdata is under mmcblk0p3 as it should be. However, X does not start. I'm not sure why yet. Running "S99UI start" manually doesn't do anything. Then again this is a dev platform so maybe xinit has nothing to do for some reason.

More testing soon.

#10 Updated by Hammel 9 months ago

Firstboot doesn't set pibox-config (by calling tvservice) correctly. That's why X.org didn't start. If I run it manually after I've logged in genConfig (in the functions script) properly creates the pibox-config so S99UI starts X correctly.

So something is fishy with how firstboot is working using an initramfs.

#11 Updated by Hammel 9 months ago

Found it. The tvservice program fails because vchiq is not loaded during firstboot. This is because S10mdev has not been run. So I need to add that to firstboot and then I think setup of pibox-config will complete correctly. This is still to be tested though I've run it manually and it seems fine.

#12 Updated by Hammel 9 months ago

  • % Done changed from 20 to 60

Tested with media apps. Everything seems to be working fine now with one exception: picam. I get no picture. This is probably not due to the switch to overlay/squashfs but I need to verify that. It's likely a power issue related to the camera plugged into the USB with a bunch of media sticks at the same time.

Once I verify and/or fix the picam problem I can call this one good. I don't actually have a way to upgrade - that will take some additional scripting to swap banks and I'd need a way of knowing that needs to be done such as an "On demand upgrade" app or something. Plus I need some validation checks on the new bank. But any app work is outside the scope of the dev platform so the work on the "upgrade path" is just infrastructure to put things in place for future development.

A side note: installation to an SD card is faster with this because I don't dd all the rootfs. Instead I just copy over the compressed squashfs. I also extended the installation scripts to install to a file so a dd-based method could still be used for distribution purposes.

#13 Updated by Hammel 9 months ago

Well, I can't figure out why mjpeg-streamer won't open /dev/video0 under the new overlays. The device file is there, the mjpeg-streamer uvc_video.so module is there and all modules are loaded. But I keep getting a "No such file or directory" error.

Worse, if I don't do the overlay/squashfs (re: copy the rootfs directly to disk) then it works perfectly.

So this can't be used until I figure out what's wrong with video, specifically v4l. It might be that I didn't move /dev from the initramfs into the switch_root. But I kinda doubt it.

#14 Updated by Hammel 9 months ago

  • % Done changed from 60 to 90

Ah. Fixed. The problem was that the init script didn't unmount /proc and /sys and didn't migrate /dev to the newroot. Adding this bit of code was all that was required to get picam working in the overlay/squashfs.


I need to commit and push the changes. Then I can close this issue and open a new one for doing upgrades.

#15 Updated by Hammel 9 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 90 to 100
Code is pushed. Closing issue. I'll open two more for future work.
  1. Add upgrade tools
  2. Automate upgrades

Also available in: Atom PDF