Feature #447
closed
Need upgrade path
100%
Description
There is no way to upgrade a system. I need one.
Plan A¶
Create 4 partitions:- p0: FAT/boot
- p1: EXT4/Minimal update
- p2: EXT4/Runtime
- p3: EXT4/User Data w/update files area
An update causes a reboot into p1 which does the update to p2 and then reboots into P2. This takes the system out of action during the update but probably not for long as long as the update image has already been downloaded.
This also assumes that the update is a complete rewrite of the boot partition. That may not be desirable.
Plan B¶
Always use opkg's, even for files already on the rootfs.
Create 3 partitions:- p0: FAT/boot
- p1: EXT4/Runtime
- p2: EXT4/User Data w/update files area
For example, a kernel update would be downloaded as an opkg and installed over the existing kernel (essentially). This requires postinst processing to switch to the new kernel (copy to p0).
The question is how will the system behave when not all files are initially package controlled. Replacing system files with opkg files may present problems.
Overview of the image and updates¶
The image build (re: rootfs) sits on a two partitions. The first is a vfat partition required by the bootloader. The second is the rootfs. Both are images copied byte-for-byte to their respective partitions in order to create the boot SD card. An update process can either update specific packages (of which there are none for the base platform) or it can replace the partition image. The latter is often required in embedded systems and might be more appropriate for PiBox unless PiBox switches to a fully package managed system.
An update to the rootfs can be done in place. A secondary rootfs partition can be installed and the boot process pointed at it so the next boot uses it. This requires a mechanism to fallback to the original partition if something fails in the new partition. A key sequence or other extenal impetus at boot time would be required to force the fallback.
The boot partition can't use an alternate. So files have to be where we expect them. The boot partition also doesn't use symlinks so we can't place alternates in directories and just update the symlinks. This implies that we have two directories, current and new, that contain the data to boot. Within that data is a kernel with an initramfs capable of replacing the booting kernel with the older version if the boot process fails.
The rootfs update is not overly difficult. It requires an extra boot script that validates the running system and resets if anything goes wrong. This verification tests that all components are online correctly, specifically wireless and USB support.
The boot partition update is more complex since it requires an error checking mechanism built into the initramfs. This error checking would be to verify that all drivers are available and can load without error.
Updated by Hammel almost 8 years ago
- Description updated (diff)
- Category set to Support
- Assignee set to Hammel
- Priority changed from Normal to Immediate
- Target version set to 0.12.0
Updated by Hammel over 6 years ago
Updated by Hammel over 6 years ago
- Description updated (diff)
- Status changed from New to In Progress
- % Done changed from 0 to 10
There are a couple of related solutions to consider.
Bank Switching¶
This is like Plan A except we use full images in each bank. Bank A is the initial install. Bank B is the first update. We download the update into Bank B, while Bank A is running, and validate it. When ready, we swap to Bank B. Swapping could be done with a reboot but it would be faster (and less error prone?) to do a kexec to a new kernel (which might be in the second bank). Note that since the new kernel is being loaded without booting from the VFAT partition (as Pi's demand) we'd need to copy in required files to the VFAT partition after the swap, but only after additional/final validation says this will be okay.
Overlay Filesystems¶
This would be layers of data, with the lowest being the core image and higher layers holding, in order from bottom to top:- Graveyard - deleted files
- RW - updates and modifications to the core image, including end user customizations
- presentation - what we see at run time
This also requires a kexec and has the same issues with the VFAT partition as the Bank Switching solution.
Additional Comments¶
- Validation requires checksuming downloaded data to start, but goes as far as verifying operational status after kexec'ing and restarting the platform via the init processing.
- Rollback capability is required in case the update fails.
- There may need to be a small kernel/initramfs specifically designed to handle the update process only. This allows us to have a constant tool for upgrades irrespective of operational kernel updates. The update kernel/initramfs would, after update, kexec into the latest kernel image.
Updated by Hammel almost 6 years ago
- Target version changed from 0.12.0 to 1.1.0 - Upgrades
Updated by Hammel almost 6 years ago
- Building an initramfs
- Configuring the boot to use it.
Building the initramfs¶
The kernel build could do this for us but it's probably easier to generate a basic Busybox and populate it with a custom init script that mounts partitions and sets up the overlays. This would need to be added as a new Busybox target in the build system and the new target added to the default set of top level targets.
This design requires the kernel to support squashfs and overlays. These can be compiled into the kernel for simplicity, otherwise the drivers need to be stuffed into the initramfs. Currently both are enabled as modules in the kernel build.
The root file system will need to be converted from tar to squashfs format. This is supported by Buildroot in the Filesystem Images submenu.
Configuring boot¶
The config.txt needs a new option: initramfs. This points to the file to use and the memory location where it should be loaded. To simplify, you can specify to load it after the kernel.
initramfs <filename> followkernel
The initramfs image will need to be stored in the FAT partition on the SD card with the Linux kernel.
Filesystem layout¶
The root Linux partition will now be divided into the following structure:- /overlay - Where overlay directories live.
- /overlay/current - Symlink to the currently active image
- /overlay/next - Symlink to the next (upgrade) image
- /overlay/image_id - a specific release image
- /overlay/image_id/presentation - what we see; the combined view
- /overlay/image_id/upper - where runtime root file system changes are saved; userdata (application runtime data) is not stored in the root filesystem but rather in /userdata.
- /overlay/image_id/work - used by the overlay driver
- /overlay/image_id/backups - temporary storage used during upgrades and resets
- /overlay/image_id/squashfs - The squashfs root filesystem for this image release
- /overlay/image_id/utils - Utilities required to manage bank switching/overlays/squashfs
- /userdata - Application save data is stored here. It is bind mounted into the presentation layer of the current image. Application data has to be migrated to new formats, as needed, by the applications on first boot.
Bank switching would imply two banks: current and next. A new release gets a new squashfs and a copy of the old upper directory, which may have package updates or similar that we want to propagate to the next release. If the new version fails self-check (which needs to be implemented as an init script function) then we swap back to the old version.
Filesystem images need to be signed. Signing verification tools go in /utils. I'm not sure how to do this yet. Other utilities, including release-specific tools will be placed in the utils directory. This will allow the init script to be updated on the fly by dropping new utilities specific to the new release in /utils and having the init script find and then run them as part of its processing. Anything dropped in /utils has to be signed and is probably an archive that is unpacked by the init script for use.
Boot Process¶
The initramfs mounts raw partitions under /boot (vfat partition) and /root (ext partition) from the SD card. The /root partition is structured with the overlay directories. The initramfs uses these directories to mount the currently active image as the runtime environment.
The real root and boot partitions are made accessible through a bind mount into the overlay's presentation layer by the initramfs.
Image Upgrades¶
An image can be downloaded and upacked to a directory under /overlay on the real root partition. Symlinks are updated so the next boot will start the new image. The image must have a self-test included on firstboot. If the self-test fails then the boot sets up a fallback to the old image and reboots.
If an image boots successfully it will check for a next tree and clean it up.
Updated by Hammel almost 5 years ago
Minor update: I tried to get QEMU to work the RPi2 build. There are instructions for running QEMU specific to RPi2 which I've scripted. It boots up to a login prompt but I can't get any keyboard input.
I would really like to get QEMU working with this because it will make testing the initramfs easier than having to write an SD card and install it to a board. The latter isn't that hard but then I need power, a display, a keyboard, etc. With QEMU I can do it from my dev box.
Still reviewing options here. There are other generic ARM types I may be able to use. I know, however, that the aarch64 QEMU doesn't work with my RPi2 build even though that build does run on real hardware just fine.
Updated by Hammel almost 5 years ago
- % Done changed from 10 to 20
I've written an initial init script with all the basic functionality required for mounting the overlay, and I've also updated the mksd/mkinstall script to properly generate an image with the overlay directory structure along with the newly generated squashfs rootfs image. Without the initramfs I can boot to a login with the image. I was able to build an initramfs with it and boot qemu with it far enough to tell what's not working in the early stages of the script. I still don't have keyboard input in qemu so this kind of testing will be of limited value, but it will save some time before I need to move to real hardware.
Current updates are committed and pushed. Next up is to begin debugging the init script early stages. Using "step" as a kernel parameter lets me step through the script but there are some early function errors that need to be cleaned up before I will want to do that. Once I get the early function fixes in I will need to switch to real hardware in order to be able to type at the keyboard.
Updated by Hammel almost 5 years ago
Last night I got the init script working to the point of mounting the overlay and booting PiBox. The boot process stops before it gets to a login for some reason but I'm not particularly worried about that yet. It may be an artifact of how I'm booting in QEMU. I'll know more when I try on real hardware, which is pretty much required now that I have the overlay mounting cleanly.
There is one more step in the init script to complete: bind mounting real partitions into the overlay. I have two choices here.
First, I can keep the mounts I have in the init script and bind them to the overlay, then disable their mounting by mdev in the rootfs.
Second, I can skip the boot and userdata mounts in the init script (they aren't needed to get the overlay running) and let mdev mount the partitions as it currently does.
The second method seems simpler so I'll try that first.
Updated by Hammel almost 5 years ago
Quick test of the image on real hardware: it boots fine up to getting a login. Login does work. Mount points look okay - userdata is under mmcblk0p3 as it should be. However, X does not start. I'm not sure why yet. Running "S99UI start" manually doesn't do anything. Then again this is a dev platform so maybe xinit has nothing to do for some reason.
More testing soon.
Updated by Hammel almost 5 years ago
Firstboot doesn't set pibox-config (by calling tvservice) correctly. That's why X.org didn't start. If I run it manually after I've logged in genConfig (in the functions script) properly creates the pibox-config so S99UI starts X correctly.
So something is fishy with how firstboot is working using an initramfs.
Updated by Hammel almost 5 years ago
Found it. The tvservice program fails because vchiq is not loaded during firstboot. This is because S10mdev has not been run. So I need to add that to firstboot and then I think setup of pibox-config will complete correctly. This is still to be tested though I've run it manually and it seems fine.
Updated by Hammel almost 5 years ago
- % Done changed from 20 to 60
Tested with media apps. Everything seems to be working fine now with one exception: picam. I get no picture. This is probably not due to the switch to overlay/squashfs but I need to verify that. It's likely a power issue related to the camera plugged into the USB with a bunch of media sticks at the same time.
Once I verify and/or fix the picam problem I can call this one good. I don't actually have a way to upgrade - that will take some additional scripting to swap banks and I'd need a way of knowing that needs to be done such as an "On demand upgrade" app or something. Plus I need some validation checks on the new bank. But any app work is outside the scope of the dev platform so the work on the "upgrade path" is just infrastructure to put things in place for future development.
A side note: installation to an SD card is faster with this because I don't dd all the rootfs. Instead I just copy over the compressed squashfs. I also extended the installation scripts to install to a file so a dd-based method could still be used for distribution purposes.
Updated by Hammel almost 5 years ago
Well, I can't figure out why mjpeg-streamer won't open /dev/video0 under the new overlays. The device file is there, the mjpeg-streamer uvc_video.so module is there and all modules are loaded. But I keep getting a "No such file or directory" error.
Worse, if I don't do the overlay/squashfs (re: copy the rootfs directly to disk) then it works perfectly.
So this can't be used until I figure out what's wrong with video, specifically v4l. It might be that I didn't move /dev from the initramfs into the switch_root. But I kinda doubt it.
Updated by Hammel almost 5 years ago
- % Done changed from 60 to 90
Ah. Fixed. The problem was that the init script didn't unmount /proc and /sys and didn't migrate /dev to the newroot. Adding this bit of code was all that was required to get picam working in the overlay/squashfs.
Bingo.
I need to commit and push the changes. Then I can close this issue and open a new one for doing upgrades.
Updated by Hammel almost 5 years ago
- Status changed from In Progress to Closed
- % Done changed from 90 to 100
- Add upgrade tools
- Automate upgrades