Feature #732

PiBox with Docker

Added by Hammel about 1 year ago. Updated about 1 month ago.

Status:In ProgressStart date:30 Nov 2019
Priority:NormalDue date:
Assignee:Hammel% Done:

40%

Category:-
Target version:PiBox - 3.0 - Corrino
Severity:03 - Medium

Description

Getting containers on PiBox isn't very straight forward. I can compile container engines and clients but cross compiling them (because many of them use Go) might not be so simple. This may limit me, initially, to existing container engines and clients such as Docker (though I'd prefer to use LXD/LXC).

Docker has static ARM binaries that should work for RPi. I need to download them and test on the dev platform to see what happens.

A first test of this would be to try an existing container built for ARM, like BusyBox or Bash or httpd - something simple.

If I want to get really clever I can create my own container infrastructure.

ck.txt Magnifier - Output from check-config.sh on running PiBox Development Platform. (4.66 KB) Hammel, 10 Aug 2020 16:30

Associated revisions

Revision 99f7f7ac
Added by Hammel 4 months ago

RM #732: Fix up bashsetup to match project.

Revision 91086c4b
Added by Hammel 4 months ago

RM #732: Add init script for docker daemon. Add check-config script for validating kernel for Docker use. Remove deprecated .mk file.

History

#1 Updated by Hammel about 1 year ago

  • Description updated (diff)

#2 Updated by Hammel 4 months ago

  • Priority changed from Normal to Immediate
  • Severity changed from 03 - Medium to 01 - Critical

#3 Updated by Hammel 4 months ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 20

There is a pidock project locally (not pushed). This has a docker component that pulls premade docker images from upstream repositories. I haven't run this in awhile so I need to retest this to make sure it still works, then check it in.

Before I do that I'm upgrading Buildroot to the latest long term release (2020.02.4). This has Docker support in it and can build Docker manually. I'd like to use that instead of pulling the upstream binaries. I may leave both options in the tree.

I'm also stripping PiBox of most of its rootfs constructs because I need very little in the base overlay other than what is required to run Docker. Then Docker can be used to run upstream containers.

#4 Updated by Hammel 4 months ago

  • Tracker changed from Bug to Feature

Building Docker from Buildroot fails right now because of a problem downloading the libsodium archive. I can get to the archive from my browser but not via Buildroot.

I've disabled the Docker build in Buildroot to let it complete. Then I'll build the Docker component separately which will download the upstream pre-built binaries.

#5 Updated by Hammel 4 months ago

Building Docker from Buildroot failed. And after some experimentation I've changed my mind about how to do this.

I've integrated docker into the PiBox build twice now. The first merged lots of changes all at once and just got confusing. The second was cleaner - more commits of specific incremental changes - but in the end I didn't like doing it that way. The development platform doesn't need Docker. It shouldn't be a new hardware type either. It's just another user space component.

The correct way to do this is to make Docker an opkg, just like any other app. So I'm going to redo this as a 3rd party build, like omxplayer or mjpeg-streamer, and simply build an opkg with the appropriate bits. This is easier, and cleaner, than trying to merge it into the PiBox development platform build anyway.

Some of the changes I made in my 2nd attempt above are needed for other RM's however, so I'll cherry pick those as patches. No sense wasting that work.

#6 Updated by Hammel 4 months ago

  • % Done changed from 20 to 40

Created and pushed the first version of the pidock opkg build to pibox/pidock.

Now I need to do the following:
  1. Add init script to start docker daemon
  2. Test package install
  3. Test docker operation on RPi2 and RPi 3.

#7 Updated by Hammel 4 months ago

  • Project changed from PiBox to pidock
  • Category deleted (04 - Applications)

#8 Updated by Hammel 4 months ago

Problems found in testing.

  1. check-config.sh doesn't work because the dev platform doesn't have bash.
  2. cgroups are not fully enabled in the kernel so the init script doesn't work.
  3. /proc/config.gz is not enabled by default. Use "modprobe configs" to load it - add that to modules.conf

I think enabling Bash and running check-config.sh would help.

#9 Updated by Hammel 4 months ago

Attached the output from check-config.sh. There are a number of kernel options to add to make this work cleanly.

Things to do:

#10 Updated by Hammel 4 months ago

Looks like the version of docker I downloaded, while a static arm binary, doesn't work on the RPi3. I get an "illegal instruction" from it. But then, I also get a segfault with curl. So I'm wondering if there is some other weird problem here.

Update: curl works if just retrieving the front page to google.com.

Ugh. This might be the reason:

bash-4.4# file /usr/bin/dockerd
/usr/bin/dockerd: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, Go BuildID=jHK_QBChLPFK5ZRuGt5x/_ghDqTccjM27JRHTjOh1/POv6fCAkLuZNd7OEiA8l/pKPAHl62MhHNeEAscq-H, BuildID[sha1]=adf3a13c53a3141edf105d06a95e88398d2f0227, not stripped

bash-4.4# file /usr/bin/pibox-network-config
/usr/bin/pibox-network-config: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 4.14.99, stripped

The "for GNU/Linux" bit is what's different. Because I use a newer Linux in the toolchain I may need to build Docker in Buildroot. Because of this I need to modify PiBox, not just create an opkg for Docker.

Bummer.

Lowering priority/severity. I don't think I'll be coming back to this any time soon.

#11 Updated by Hammel about 1 month ago

  • Priority changed from Immediate to Normal
  • Severity changed from 01 - Critical to 03 - Medium

Also available in: Atom PDF