https://redmine.graphics-muse.org/https://redmine.graphics-muse.org/favicon.ico?16278048512019-02-25T22:17:11ZGraphics Muse Issue TrackerPiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=24462019-02-25T22:17:11ZHammelmjhammel@graphics-muse.org
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Immediate</i></li></ul> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=24472019-02-25T22:17:36ZHammelmjhammel@graphics-muse.org
<ul><li><strong>Target version</strong> changed from <i>1.0 - Atreides</i> to <i>1.1.0 - Upgrades</i></li></ul> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=24712019-03-03T05:06:36ZHammelmjhammel@graphics-muse.org
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>10</i></li></ul><p>Other options:</p>
<ol>
<li><a href="https://www.slideshare.net/righiandr/linux-boottime-23817352" class="external">Slides with general boot optimizations</a></li>
</ol>
<p>This is mostly the same as the link in the issue description. Some of these would be handled with RM <a class="issue tracker-2 status-5 priority-7 priority-highest closed" title="Feature: Need upgrade path (Closed)" href="https://redmine.graphics-muse.org/issues/447">#447</a> by using squashfs and overlays that are managed by an initramfs init script for booting.</p> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=24732019-03-03T18:33:43ZHammelmjhammel@graphics-muse.org
<ul></ul><p>From RM#231:</p>
<p>Looking through dmesg to see boot times I found the following things:</p>
<pre><code>A bunch of eth drivers are compiled in. Only smsc95xx needs to be enabled and even that could be a loadable module.<br /> bcm2708_spi takes 3.5 seconds to load<br /> sda1 (ext4) takes 9 seconds to load<br /> wlan0 takes 8 seconds to load<br /> X (and the fbtft) takes 17 seconds to load</code></pre> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=24832019-03-09T19:56:12ZHammelmjhammel@graphics-muse.org
<ul><li><strong>% Done</strong> changed from <i>10</i> to <i>20</i></li></ul><p>Added lpj to kernel boot args via a firstboot test and updates cmdline.txt. <br />Changes pushed.</p> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=24972019-03-17T16:27:54ZHammelmjhammel@graphics-muse.org
<ul></ul><p>Getting from power on to mounting the SD card takes about 6 seconds on a Pi 2.</p>
<pre>
[ 5.844178] EXT4-fs (mmcblk0p2): recovery complete
[ 5.853241] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 5.853328] VFS: Mounted root (ext3 filesystem) readonly on device 179:2.
[ 5.863535] devtmpfs: mounted
[ 5.870637] Freeing unused kernel memory: 1024K
[ 5.870975] Run /sbin/init as init process
</pre>
<p>Nearly all of this is getting USB and SD access setup. Then getting the RNG takes another 5 seconds.</p>
<pre>
[ 6.156952] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 9.468533] FAT-fs (sda1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[ 9.486838] FAT-fs (sdb1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[ 9.737072] EXT4-fs (mmcblk0p3): mounting ext3 file system using the ext4 subsystem
[ 9.758496] EXT4-fs (mmcblk0p3): recovery complete
[ 9.758531] EXT4-fs (mmcblk0p3): mounted filesystem with ordered data mode. Opts: (null)
[ 9.979002] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[ 10.917545] random: crng init done
</pre>
<p>This can be fixed with preseeding. The rest of the time goes to loading modules. By the time the Bluetooth init script is run,<br />which is the last one that shows up in dmesg, we're at close to 30 seconds.</p>
<pre>
[ 28.846777] Bluetooth: BNEP socket layer initialized
</pre>
<a name="Things-I-can-do"></a>
<h1 >Things I can do<a href="#Things-I-can-do" class="wiki-anchor">¶</a></h1>
<ul>
<li>Call log() in all my custom scripts so I can tell which ones are taking a long time. Or at least call log() from rcS to say which script is being called. That will get more info into dmesg.</li>
</ul>
<ul>
<li>Put network startup in the background. Nothing needs it up front to start, I don't think.</li>
</ul>
<ul>
<li>Add RNG seeding early in the boot sequence.</li>
</ul>
<ul>
<li>I see this with at least some wifi drivers: unknown parameter '11n_disable' ignored. I should take those out from the drivers I've seen that message from (rt2800usb, carl9170).</li>
</ul>
<ul>
<li>There is something happening here that's taking 8 seconds - find out what it is:</li>
</ul>
<pre>
[ 11.725594] usbcore: registered new interface driver rt2800usb
[ 19.922011] snd_bcm2835: module is from the staging directory, the quality is unknown, you have been warned.
</pre>
<p>I think this is module loading from /etc/modules.conf. I might be able to slurp just the module names from the file and feed them as one request to modprobe.</p>
<p>I feel confident I can get this down to 15 seconds, maybe less, to get to the UI. And with the changes to bootsplash it will seem much quicker.</p> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=25012019-03-19T02:20:53ZHammelmjhammel@graphics-muse.org
<ul></ul><p>Module loading was a bottleneck. I fixed the init script to scarf all the module names into a single variable and pass them all at once to modprobe. I also switched from depmod to depmod -A which helped a bunch too. This shaves about 7 seconds off the boot time.</p>
<p>The next biggest bottleneck is mdev. It takes 9 seconds for that to complete. One possible solution is to write a C program that handles all the configuration of usbhandler and friends (from nldev). Still, mdev has to spawn that C program for each device it finds so it's hard to say how much time this would save. I can start by writing usbhandler in C and seeing if that helps.</p>
<p>After a little googling I found that mdev has its own init script that I've been overwriting. If I use it, which has the following:</p>
<blockquote>
<p> echo /sbin/mdev >/proc/sys/kernel/hotplug<br /> /sbin/mdev -s
# coldplug modules<br /> find /sys/ -name modalias -print0 | xargs -0 sort -u | tr '\n' '\0' | xargs -0 modprobe -abq</p>
</blockquote>
<p>then it seems to boot about 6 seconds faster overall, but it also loads a lot more modules (due to that last line). In fact, it loaded the module for the Pi Zero W wifi even though I'd commented it out of the modules.conf and the node came up using that wifi instead of the usb dongle I had connected (which came up as wlan1). So it seems I should drop my mdev script, and possibly drop loading my modules as this takes care of it, and much faster.</p>
<p>Network bring up is about 5.5 seconds. I can put that into the background and hang dependent services off it, like smb.</p>
<p>smb startup, on its own, takes almost 9 seconds. Combined with the network that's almost 15 seconds that could be done in the background so the UI can be up faster.</p> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=25022019-03-19T03:53:46ZHammelmjhammel@graphics-muse.org
<ul></ul><p>I commented out all the modules in /etc/modules.conf and tried again. I found the following modules were not loaded automatically by the mdev changes:</p>
<p>snd-bcm2835<br />spi_bcm2835<br />uvcvideo</p>
<p>However, there are alot of new drivers loaded. One of them is hid_multitouch. I'm not certain yet but I may need to blacklist that in order for tslib to work. That said, I ran the following and it looks okay:</p>
<ol>
<li>evtest <br />No device specified, trying to scan all of /dev/input/event*<br />Available devices:<br />/dev/input/event0: ILITEK ILITEK-TP Mouse<br />/dev/input/event1: HID 6901:2701<br />/dev/input/event2: HID 6901:2701 Mouse<br />/dev/input/event3: HID 6901:2701 Consumer Control<br />/dev/input/event4: HID 6901:2701 System Control<br />/dev/input/event5: ILITEK ILITEK-TP</li>
</ol>
<p>Despite having hid_multitouch loaded, event5 is mapped correctly to the touchscreen and shows up correctly so maybe I don't need to blacklist that driver. More testing is required.</p> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=25042019-03-23T20:15:55ZHammelmjhammel@graphics-muse.org
<ul><li><strong>% Done</strong> changed from <i>20</i> to <i>30</i></li></ul><p>Modules loading is cleaned up for both S10mdev (which now handles many of the module loading duties) and S11dev which does manual modules.conf module loading. It's a bit faster now.<br />Changes tested, committed and pushed.</p>
<p>log() can't be called by rcS directly (by adding it to the script) because we still need to source the functions script, which in turn sources package specific functions like bumpSplash. Getting rid of psplash (see RM <a class="issue tracker-9 status-5 priority-7 priority-highest closed" title="Action Item: Clean up boot splash (Closed)" href="https://redmine.graphics-muse.org/issues/689">#689</a>) will help here.</p>
<p>So now we're down to this:</p>
<pre>
[ 37.781344] Running /etc/init.d/S97stopsplash
</pre>
<p>That's the last init process running before the UI gets going. I still think there are things I can do to speed this up, especially if I can offload network startup stuff to run in parallel with UI startup.</p> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=25102019-03-24T21:45:10ZHammelmjhammel@graphics-muse.org
<ul></ul><a name="Compressing-kernelimg"></a>
<h1 >Compressing kernel.img<a href="#Compressing-kernelimg" class="wiki-anchor">¶</a></h1>
<p>The defconfig for RPi uses gzip for compression and I wasn't even grabbing that image - I was grabbing the uncompressed image.<br />I've switched to LZ4 compression and am now copying the compressed image to the package tree.</p>
<p>Changes tested, committed and pushed.</p>
<a name="Using-initramfs"></a>
<h1 >Using initramfs<a href="#Using-initramfs" class="wiki-anchor">¶</a></h1>
<p>The following could be used with a stripped down initramfs to get a bootsplash faster. This would then switch to the animated splash in the init processing.</p>
<pre>
/etc/init.d/start.sh:
#!/bin/sh
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mount -t devtmpfs devtmpfs /dev
# Mount RFS / do some critical stuff
mount /dev/mmcblk0p1 /media
fbsplash -s /media/splash.ppm -d /dev/fb0
mount -o move /proc /media/proc
mount -o move /sys /media/sys
mount -o move /dev /media/dev
# Switch to production system
exec switch_root /media /linuxrc
</pre>
<a name="Improving-RNG-setup"></a>
<h1 >Improving RNG setup<a href="#Improving-RNG-setup" class="wiki-anchor">¶</a></h1>
<p>Getting RNG setup early is partly done, but there is the usual problem of not just seeding, but increasing the count of available entropy. The former is already done. The latter is not, because that requires an ioctl call for RNDADDTOENTCNT.</p>
<p>I can add a C program to piboxlib/ptools that handles this. It's a simply utility for bumping the available entropy - nothing more - using the ioctl. I can then add that to S20urandom (which I'll have to dup in my buildroot skeleton from the buildroot target). And I can generate a seed file with 4096 bits of entropy and add it to the initial build.</p>
<p>But S20urandom will need to change to read the whole 4096 bytes - or however large the entropy pool actually is. Right now it just reads 512 bytes from /etc/random-seed (which is where the new initialized data should go during the build).</p>
See:
<ul>
<li>/proc/sys/kernel/random/entropy_avail</li>
<li>/proc/sys/kernel/random/pool_size</li>
<li>man urandom</li>
</ul>
<p>The man page for urandom explains how to do all of this.</p> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=25142019-03-25T02:58:41ZHammelmjhammel@graphics-muse.org
<ul></ul><p>urandom updates are ready for testing. I've added a custom S20urandom and generate the seed file in Buildroot's postbuild.sh script. Both were verified with shellcheck.</p>
<p>I still need to write the C program to do the ioctl and place it in libpibox.</p> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=25152019-03-26T05:25:54ZHammelmjhammel@graphics-muse.org
<ul><li><strong>% Done</strong> changed from <i>30</i> to <i>60</i></li></ul><p>RNG updates completed and tested, committed and pushed. This includes the C program, pbsetentropy, added to libpibox.</p>
<p>The last speedup I want to try is networking: how can I startup the network interfaces and then all the subsystems that depend on it in the background while the UI comes up?</p> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=25162019-03-26T16:21:58ZHammelmjhammel@graphics-muse.org
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/2516/diff?detail_id=2885">diff</a>)</li><li><strong>% Done</strong> changed from <i>60</i> to <i>30</i></li></ul> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=25172019-03-28T02:08:53ZHammelmjhammel@graphics-muse.org
<ul><li><strong>% Done</strong> changed from <i>30</i> to <i>60</i></li></ul><p>Along with networking I also want to run bootchard to get an idea of where exactly the slow processing is at.</p> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=25182019-03-28T03:45:45ZHammelmjhammel@graphics-muse.org
<ul><li><strong>File</strong> <a href="/attachments/127">bootchart.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/127/bootchart.png">bootchart.png</a> added</li></ul><p>I ran bootchard tonight. Here is the image.<br /><img src="https://redmine.graphics-muse.org/attachments/download/127/bootchart.png" title="Bootchart of boot processon Pi Zero W." alt="Bootchart of boot processon Pi Zero W." /><br />What surprised me is that it's not really networking interface startup that's the bottleneck. There are two bottlenecks. The first is the depmod command, which may just be superfluous (it could be run on firstboot if necessary) and can be outright dropped. The second is Samba startup. That could be handled by placing the daemons (smb/nmb) in the background. I can start them first in a subshell and then background them to fully daemonize their startup.</p>
<p>Together, that may shave 20 seconds on a 40 second startup!</p> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=25192019-03-28T04:10:28ZHammelmjhammel@graphics-muse.org
<ul><li><strong>File</strong> <a href="/attachments/128">bootchart-2.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/128/bootchart-2.png">bootchart-2.png</a> added</li></ul><p>I was right. Those two things shave about 15 seconds off the boot time.<br /><img src="https://redmine.graphics-muse.org/attachments/download/128/bootchart-2.png" title="Boot time afte rminor adjustments." alt="Boot time afte rminor adjustments." /><br />Next would be parallelizing processes depending on an IP (nptd, sshd, httpd, etc.). That could save a few more seconds, plus there are up to 9 seconds worth of artificial sleeps in the init scripts. So I could probably drop another 10 seconds! Woohoo!</p> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=25212019-03-30T17:51:25ZHammelmjhammel@graphics-muse.org
<ul><li><strong>File</strong> <a href="/attachments/129">bootchart-rpi3.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/129/bootchart-rpi3.png">bootchart-rpi3.png</a> added</li><li><strong>Priority</strong> changed from <i>Immediate</i> to <i>High</i></li><li><strong>% Done</strong> changed from <i>60</i> to <i>70</i></li></ul><p>Those previous tests were on a single-core Raspberry Pi Zero W. I tried the same build on a quad-core Raspberry Pi 3 and got even better results.<br />----<br /><img src="https://redmine.graphics-muse.org/attachments/download/129/bootchart-rpi3.png" title="Boot times for RPi 3 with onboard Wifi enabled." alt="Boot times for RPi 3 with onboard Wifi enabled." /><br />----<br />I've got it down to about just under 12 seconds! And at least 1 second of that is getting bootchard started. And another second on psplash getting started, which I could put in the background (or will replace later). So I could get down to under 10 seconds boot on the RPi 3.</p>
Again, the only things left to improve that will have a big impact are
<ol>
<li>Wifi setup</li>
<li>modprobe processing</li>
</ol>
<p>The former I can do some tricks with (there is an artificial sleep in there that probably could be sync'd better). The latter I'm not sure what else I can do. I have tried generating the modules list on first boot so it doesn't have to get rebuilt on each boot, but that doesn't help modprobe process each module.</p>
<p>I think this is good enough for now. I want to move on to other things and will return to this later to try and squeeze even more out of the boot times.</p> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=28292020-07-23T04:46:58ZHammelmjhammel@graphics-muse.org
<ul><li><strong>Priority</strong> changed from <i>High</i> to <i>Immediate</i></li><li><strong>Severity</strong> changed from <i>03 - Medium</i> to <i>04 - Low</i></li></ul> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=32312022-04-14T21:55:34ZHammelmjhammel@graphics-muse.org
<ul><li><strong>Severity</strong> changed from <i>04 - Low</i> to <i>05 - Very Low</i></li></ul> PiBox - Action Item #472: Boot time speedupshttps://redmine.graphics-muse.org/issues/472?journal_id=35622023-05-21T03:37:33ZHammelmjhammel@graphics-muse.org
<ul><li><strong>Target version</strong> changed from <i>1.1.0 - Upgrades</i> to <i>3.0 - Corrino</i></li></ul><p>This might benefit from writing some C apps that handle initial setup instead of trying to create lots of nodes manually via sh, eg.</p>