Project

General

Profile

Actions

Testing #893

open

System Integration Tests for Harkonnen Release

Added by Hammel 7 months ago. Updated 2 days ago.

Status:
In Progress
Priority:
Immediate
Assignee:
Category:
09 - Testing
Target version:
Start date:
28 Feb 2022
Due date:
% Done:

20%

Estimated time:
Severity:
03 - Medium

Description

Feature development is essentially complete for the PiBox 2.0 - Harkonnen release. However, full integration tests need to be run to validate existing functionality passes regression and new features are at least stable, if not bug free.

The following systems need to be fully tested. A test plan need to be written for each one. Separate issues will be written for each testplan and test status will be tracked here. Regression testing implies previous functionality is still working. Feature testing validates new components. New systems will have no regression tests and existing systems may not have new functionality. Platforms represent the devices on which the system was tested.

Test Suite Platforms
System Test Plan Regression Status Feature Status RPi 1 RPi 2 RPi 3 Zero (G1) Zero W Desktop
Dev Platform DevPlatform Pass Pass Pass Pass (1) Fail (1) n/a
Media System MediaSystem n/a n/a n/a n/a
Media Player Media_Player Fail (2) Fail (1) n/a
Kiosk Kiosk Fail (2) Fail (1) n/a
Auto Kiosk Auto_Kiosk Fail (2) Fail (1) n/a
PiSentry PiSentry n/a n/a n/a n/a
PiStore PiStore n/a n/a n/a n/a
Desktop (VideoLib) VideoLib n/a n/a n/a n/a n/a

General Notes

  • This is the first full test suite run on system builds, as system builds are new to this release. This means there really isn't any regression testing and all tests are new feature testing.
  • These tests are being run in September 2022. Supply chain issues make acquisition of new boards nearly impossible unless you get on a wait list. Replacing dead boards won't happen for awhile, I think.
  • Only WiFi networking is tested. Ethernet support is not a supported component of any PiBox system build.
  • G1: The Zero is fairly useless for PiBox without a WiFI dongle, which tend to pull too much power from that board.

Notes on Failed Tests

  1. Zero W: X.org flashes slowly on HDMI. Removing WiFi and keyboard/mouse dongles does not fix problem. Possibly bad board. Unit cannot be tested further.
  2. Zero: WiFi dongle not stable. Unit cannot be tested further.

Notes on Passed Tests

  1. Caveats listed in comment #note-8.
Actions #1

Updated by Hammel 6 months ago

  • Description updated (diff)
  • Status changed from New to In Progress
  • % Done changed from 0 to 10
Actions #2

Updated by Hammel 13 days ago

  • Description updated (diff)
Actions #3

Updated by Hammel 13 days ago

  • Description updated (diff)
Actions #4

Updated by Hammel 13 days ago

Ready to test
  1. Dev platform, rpi1 pizero/w
  2. Dev platform, rpi2 model B
Then make and test
  1. Dev platform, rpi1 model B+
  2. Dev platform, rpi3 Model B+
Actions #5

Updated by Hammel 12 days ago

  • Description updated (diff)
Actions #6

Updated by Hammel 12 days ago

  • Description updated (diff)
Actions #7

Updated by Hammel 12 days ago

  • Description updated (diff)
Actions #8

Updated by Hammel 12 days ago

  • Description updated (diff)

Pi Zero, Dev Platform

Subsystem Feature Test Status Notes
Display: Desktop Boot to UI Pass
Display: 7" Touchscreen Boot to UI N/A Only tested against HDMI, not touchscreen
UI X org Boot to UI Pass
Shell Command History Pass Up-arrow navigation of history works using a wired keyboard's keypad with numlock turned off.
Utilities HW ID Examine /etc/pibox-config, /etc/pibox-version Pass
Input Keyboard FAVI keyboard keys Pass Arrow keys do not work in history.
Mouse FAVI keyboard touchpad Pass
Touchscreen RPi official touchscreen N/A
Net Config UI Verify UI Pass
Interface Static IP Pass
DHCP Pass Network does not come up; dmesg suggests dwc3 problems, possibly with power draw
Access Point Fail Connected to client but no dhcp handed out; dmesg suggests dwc3 problems, possibly with power draw
Scanner Visual display of local WiFi signals Pass

Notes

  • Boot time for the Pi Zero is about 60 seconds to the UI.
Actions #9

Updated by Hammel 12 days ago

  • Description updated (diff)
Actions #10

Updated by Hammel 12 days ago

  • Description updated (diff)
Actions #11

Updated by Hammel 12 days ago

  • Description updated (diff)
Actions #12

Updated by Hammel 12 days ago

RPi 2 Model B, Dev Platform

Subsystem Feature Test Status Notes
Display: Desktop Boot to UI Pass
Display: 7" Touchscreen Boot to UI N/A Only tested against HDMI, not touchscreen
UI X org Boot to UI Pass
Shell Command History Pass Up-arrow navigation of history works using a wired keyboard's keypad with numlock turned off.
Utilities HW ID Examine /etc/pibox-config, /etc/pibox-version Pass
Input Keyboard FAVI keyboard keys Pass Arrow keys do not work in history.
Mouse FAVI keyboard touchpad Pass
Touchscreen RPi official touchscreen N/A
Net Config UI Verify UI Pass
Interface Static IP Pass (1)
DHCP Fail (1) Network does not come up; dmesg suggests authentication but denied association
Access Point Pass
Scanner Visual display of local WiFi signals Pass

Notes on Passed Tests

  1. Multiple empty nameserver entries written to /etc/resolv.conf. See #RM 901
  2. Network restart does not bring up eth0 as dhcp, but eth0 is unsupported by PiBox.

Notes on Failed Tests

  1. DHCP setup is correct but no wifi connection is made. dmesg reports "denied association (code=27)", which appears to mean Disabled because STA does not support HT. dmesg also reports "disabling HT/VTE/HE due to WEP/TKIP use". The router is setup to use TKIP+AES. Changing the STA (re: Pi) to use CCMP (re: AES) doesn't seem to fix the problem. So this is a configuration error with respect to the router, not PiBox. That said, the use of TKIP is discouraged in favor of CCMP only. That's because CCMP is used with WPA2 which is what we're configuring by default on PiBox. So pnc should be defaulting to CCMP, not TKIP.
Actions #13

Updated by Hammel 12 days ago

  • Description updated (diff)
Actions #14

Updated by Hammel 11 days ago

  • Description updated (diff)
Actions #15

Updated by Hammel 11 days ago

RPi 1 Model B+, Dev Platform

Subsystem Feature Test Status Notes
Display: Desktop Boot to UI Pass
Display: 7" Touchscreen Boot to UI N/A Only tested against HDMI, not touchscreen
UI X org Boot to UI Pass
Shell Command History Pass Up-arrow navigation of history works using a wired keyboard's keypad with numlock turned off.
Utilities HW ID Examine /etc/pibox-config, /etc/pibox-version Pass
Input Keyboard FAVI keyboard keys Pass Arrow keys do not work in history.
Mouse FAVI keyboard touchpad Pass
Touchscreen RPi official touchscreen N/A
Net Config UI Verify UI Pass
Interface Static IP Pass
DHCP Fail Network does not come up; dmesg suggests authentication but denied association
Access Point Pass
HW Wired Pass
WiFi Fail Multiple dongles fail to connect to router
Scanner Visual display of local WiFi signals Pass

Notes on Passed Tests

Notes on Failed Tests

Actions #16

Updated by Hammel 8 days ago

RPi 3 Model B+, Dev Platform

Subsystem Feature Test Status Notes
Display: Desktop Boot to UI Pass
Display: 7" Touchscreen Boot to UI Pass
UI X org Boot to UI Pass
Shell Command History Pass Up-arrow navigation of history works using a wired keyboard's keypad with numlock turned off.
Utilities HW ID Examine /etc/pibox-config, /etc/pibox-version Pass
Input Keyboard FAVI keyboard keys Pass Arrow keys do not work in history.
Mouse FAVI keyboard touchpad Pass
Touchscreen RPi official touchscreen Pass Works poorly on dev platform because touchscreen is not calibrated.
Net Config UI Verify UI Pass
Interface Static IP N/A Previously tested. Interface should be a unit test, not a system test.
DHCP N/A Previously tested. Interface should be a unit test, not a system test.
Access Point Pass Dongle only - brcm doesn't work well enough yet.
HW Wired Pass
WiFi Pass Dongle only - brcm doesn't work well enough yet. AP only.
Scanner Visual display of local WiFi signals N/A Previously tested. Interface should be a unit test, not a system test.

Notes on Passed Tests

Notes on Failed Tests

  • Before configuration the IP field has the address 192.168.101.100 but the network is not configured for that. Don't know where that address comes from. It's not in the source code.
Actions #17

Updated by Hammel 8 days ago

  • Description updated (diff)
  • % Done changed from 10 to 20
Next to test
  1. Media System, rpi2 model B (just add system packages to SD)
  2. Media System, rpi3 model B+ (just add system packages to SD)
  3. Media System, rpi3 model B+, touchscreen (just add system packages to SD)
Then make and test
  1. Media Player System, rpi1 model B+
  2. Media Player System, rpi2 model B+
  3. Media Player System, rpi3 Model B+
  4. Media Player System, rpi3 Model B+, touchscreen
Actions #18

Updated by Hammel 2 days ago

RPi 2 Model B+, Media System

Subsystem Feature Test Status Notes
Splash Sequential update during boot Pass See RM #905
UI Matchbox Launcher displays, overlaid by selected app Fail xterm does not go full screen, overlaying the launcher; see RM #907
Launcher Display x rows by y columns Fail layout is larger than display; see RM #906
Navigation Use keyboard arrow keys to navigation app selection
App Manager Start app via launcher
VideoFE Selection Arrow keys
Search
Playback Start video
FF
Rewind
Exit
MusicFE Selection by title Arrow Keys
Selection by Artist Arrow Keys
Playback Start audio
FF
Rewind
Exit
PiClock Clock Check time, over multiple seconds
Calendar Check date, including manually changing
PiCam Video Display Start video
Video Display Exit video
PiXM Navigation By channel id
By category
Output Playback start
Playback stop
PiPics Navigation, keyboard FF
Rewind
Navigation, touchscreen FF
Rewind
Net Config UI Verify UI
Interface Static IP
DHCP
Access Point Pi accepts WiFi connections
Scanner Visual display of local WiFi signals
WebUI Users Login
Add
Delete
Password Change
WebCam
Net Config Interface
Access Point
Actions #19

Updated by Hammel 2 days ago

RPi 3 Model B+, Media System

Subsystem Feature Test Status Notes
Splash Sequential update during boot Fail See RM #905
UI Matchbox Launcher displays, overlaid by selected app Fail xterm does not go full screen, overlaying the launcher; see RM #907
Launcher Display x rows by y columns Fail layout is larger than display; see RM #906
Navigation Use keyboard arrow keys to navigation app selection
App Manager Start app via launcher
VideoFE Selection Arrow keys
Search
Playback Start video
FF
Rewind
Exit
MusicFE Selection by title Arrow Keys
Selection by Artist Arrow Keys
Playback Start audio
FF
Rewind
Exit
PiClock Clock Check time, over multiple seconds
Calendar Check date, including manually changing
PiCam Video Display Start video
Video Display Exit video
PiXM Navigation By channel id
By category
Output Playback start
Playback stop
PiPics Navigation, keyboard FF
Rewind
Navigation, touchscreen FF
Rewind
Net Config UI Verify UI
Interface Static IP
DHCP
Access Point Pi accepts WiFi connections
Scanner Visual display of local WiFi signals
WebUI Users Login
Add
Delete
Password Change
WebCam
Net Config Interface
Access Point
Actions

Also available in: Atom PDF