Feature #732
open
PiBox with Docker
40%
Description
Getting containers on PiBox isn't very straight forward. I can compile container engines and clients but cross compiling them (because many of them use Go) might not be so simple. This may limit me, initially, to existing container engines and clients such as Docker (though I'd prefer to use LXD/LXC).
Docker has static ARM binaries that should work for RPi. I need to download them and test on the dev platform to see what happens.
A first test of this would be to try an existing container built for ARM, like BusyBox or Bash or httpd - something simple.
If I want to get really clever I can create my own container infrastructure.
Files
Updated by Hammel over 4 years ago
- Priority changed from Normal to Immediate
- Severity changed from 03 - Medium to 01 - Critical
Updated by Hammel over 4 years ago
- Status changed from New to In Progress
- % Done changed from 0 to 20
There is a pidock project locally (not pushed). This has a docker component that pulls premade docker images from upstream repositories. I haven't run this in awhile so I need to retest this to make sure it still works, then check it in.
Before I do that I'm upgrading Buildroot to the latest long term release (2020.02.4). This has Docker support in it and can build Docker manually. I'd like to use that instead of pulling the upstream binaries. I may leave both options in the tree.
I'm also stripping PiBox of most of its rootfs constructs because I need very little in the base overlay other than what is required to run Docker. Then Docker can be used to run upstream containers.
Updated by Hammel over 4 years ago
- Tracker changed from Bug to Feature
Building Docker from Buildroot fails right now because of a problem downloading the libsodium archive. I can get to the archive from my browser but not via Buildroot.
I've disabled the Docker build in Buildroot to let it complete. Then I'll build the Docker component separately which will download the upstream pre-built binaries.
Updated by Hammel over 4 years ago
Building Docker from Buildroot failed. And after some experimentation I've changed my mind about how to do this.
I've integrated docker into the PiBox build twice now. The first merged lots of changes all at once and just got confusing. The second was cleaner - more commits of specific incremental changes - but in the end I didn't like doing it that way. The development platform doesn't need Docker. It shouldn't be a new hardware type either. It's just another user space component.
The correct way to do this is to make Docker an opkg, just like any other app. So I'm going to redo this as a 3rd party build, like omxplayer or mjpeg-streamer, and simply build an opkg with the appropriate bits. This is easier, and cleaner, than trying to merge it into the PiBox development platform build anyway.
Some of the changes I made in my 2nd attempt above are needed for other RM's however, so I'll cherry pick those as patches. No sense wasting that work.
Updated by Hammel over 4 years ago
- % Done changed from 20 to 40
Created and pushed the first version of the pidock opkg build to pibox/pidock.
Now I need to do the following:Add init script to start docker daemon- Test package install
- Test docker operation on RPi2 and RPi 3.
Updated by Hammel over 4 years ago
- Project changed from PiBox to pidock
- Category deleted (
04 - Applications)
Updated by Hammel over 4 years ago
Problems found in testing.
- check-config.sh doesn't work because the dev platform doesn't have bash.
- cgroups are not fully enabled in the kernel so the init script doesn't work.
- /proc/config.gz is not enabled by default. Use "modprobe configs" to load it - add that to modules.conf
I think enabling Bash and running check-config.sh would help.
Updated by Hammel over 4 years ago
Attached the output from check-config.sh. There are a number of kernel options to add to make this work cleanly.
Things to do:- Enable modules
- cls_cgroup
- veth
- bridge
- Mount cgroupfs
Updated by Hammel over 4 years ago
Looks like the version of docker I downloaded, while a static arm binary, doesn't work on the RPi3. I get an "illegal instruction" from it. But then, I also get a segfault with curl. So I'm wondering if there is some other weird problem here.
Update: curl works if just retrieving the front page to google.com.
Ugh. This might be the reason:
bash-4.4# file /usr/bin/dockerd
/usr/bin/dockerd: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, Go BuildID=jHK_QBChLPFK5ZRuGt5x/_ghDqTccjM27JRHTjOh1/POv6fCAkLuZNd7OEiA8l/pKPAHl62MhHNeEAscq-H, BuildID[sha1]=adf3a13c53a3141edf105d06a95e88398d2f0227, not stripped
bash-4.4# file /usr/bin/pibox-network-config
/usr/bin/pibox-network-config: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 4.14.99, stripped
The "for GNU/Linux" bit is what's different. Because I use a newer Linux in the toolchain I may need to build Docker in Buildroot. Because of this I need to modify PiBox, not just create an opkg for Docker.
Bummer.
Lowering priority/severity. I don't think I'll be coming back to this any time soon.
Updated by Hammel about 4 years ago
- Priority changed from Immediate to Normal
- Severity changed from 01 - Critical to 03 - Medium